AI red teaming is maturing, and security leaders need to rethink how they test AIGenerative AI is rapidly moving from experimentation to product. AI-enabled applications now support customer interactions, internal workflows, analytics, and automation across the organization. As adoption accelerates, security leaders face a growing challenge: understanding how these systems behave under real-world adversarial conditions. A recent report from Forrester makes it clear that AI red teaming is becoming a necessary security practice, but one that differs significantly from traditional penetration testing and red team engagement. Why AI red teaming is differentAccording to Forrester, AI red teaming blends established offensive security techniques with new testing approaches designed specifically for AI-enabled systems. Traditional red teams focus on infrastructure, applications, APIs, and the SDLC. AI red teaming must also evaluate risks unique to AI, including bias, toxicity, safety failures, data exposure, and unintended behavior. These challenges are compounded by the probabilistic nature of AI. Models retrain, responses vary, and integrations evolve quickly. Static, point-in-time testing loses relevance fast. Given this, Forrester emphasizes the importance of evaluating the entire AI application stack, not just the model itself. A fragmented AI red teaming landscapeOne of the report’s central observations is how fragmented the AI red teaming market has become. Security teams generally encounter two primary approaches: Traditional offensive security providers extending pen testing and red team services to AI-enabled environments. AI and ML security vendors offering continuous, automated prompt-based testingEach approach brings value, but neither is sufficient on its own. Prompt saturation can identify patterns at scale but often lacks context. Manual testing provides depth but struggles to keep pace with rapidly evolving AI systems. Forrester’s conclusion is pragmatic: the most effective AI red teaming programs combine human-led testing with continuous, adaptive, and agentic techniques. This hybrid model more closely reflects real adversary behavior and produces findings that are both actionable and relevant. Why prompt testing alone falls shortPrompt injection and jailbreaks tend to dominate AI security discussions, but Forrester is clear that they represent only part of the overall risk picture. Many of the most significant vulnerabilities exist in systems surrounding AI: Application logic that routes prompts and responsesAPIs and integrations connecting models to enterprise dataSource code repositories and CI/CD pipelinesIdentity, access, and data controls governing AI usageIn short, AI is still software, just software with new failure modes. Red teaming that focuses only on prompts leaves critical blind spots. Early AI red teaming is imperfect but necessaryMany organizations are testing AI systems earlier than they would prefer, often driven by regulatory requirements, audits, or customer scrutiny. Forrester acknowledges that early AI red team engagements may be imperfect, but they still provide value by uncovering systemic issues, informing governance decisions, and demonstrating due diligence. The key shift is moving from one-time assessment to ongoing AI red teaming programs that evolve alongside the technology. From testing to operational AI securityThe Forrester report points to a broader shift: AI red teaming is increasingly connected to how organizations operationalize security day to day. Testing alone is not enough. Security teams need continuous visibility into where AI is being used, how it is accessed, and how risk is introduced across applications, users, and data. As AI becomes embedded into SaaS platforms, custom applications, and internal workflows, the attack surface expands rapidly. Without a consistent way to discover AI usage, assess risks, and enforce controls, many organizations are left stitching together point solutions, each addressing only part of the problem. Forrester highlights how providers such as SPLX are pushing AI red teaming beyond isolated assessments toward scalable, continuous evaluation of AI-enabled systems. This reflects a growing recognition that AI security must be built on foundational security principles; continuous verification, least-privilege access, and strong data protections, rather than implicit trust. Applying zero trust principles to AI securityWhile the report does not frame AI red teaming as a zero trust exercise explicitly, many of its recommendations align closely with zero trust principles. AI systems should not be trusted by default, whether they are public AI services, embedded SaaS features, or internally developed models and agents. Applying zero trust thinking to AI means continuously validating access to AI systems, tightly controlling how AI interacts with enterprise data, and monitoring behaviour across users, applications, and integrations. When paired with continuous AI red teaming, this approach helps organizations reduce risk while still enabling rapid AI adoption. Rather than adding more disconnected tools, security leaders are increasingly looking to unify AI discovery, adversarial testing, and runtime controls, and governance into a cohesive security architecture, one that scales as AI usage grows. What security leaders should do nextAI red teaming is still maturing, but the direction is clear. Based on Forrester’s research, security leaders should: Expand AI testing beyond prompt to include applications, integrations, and data flowsCombine human expertise with continuous, adaptive testing techniquesApply zero trust principles to AI access, data exposure, and runtime behaviourTreat AI red teaming as an ongoing security capability, not a one-time eventAI will continue to move fast. The organizations that succeed will be the ones that can scale AI securely, without increasing complexity and losing visibility. Learn moreDownload the full Forrester report on AI red teaming to explore testing approaches, engagement models, and best practices for securing AI-enabled applications.
[#item_full_content] AI red teaming is maturing, and security leaders need to rethink how they test AIGenerative AI is rapidly moving from experimentation to product. AI-enabled applications now support customer interactions, internal workflows, analytics, and automation across the organization. As adoption accelerates, security leaders face a growing challenge: understanding how these systems behave under real-world adversarial conditions. A recent report from Forrester makes it clear that AI red teaming is becoming a necessary security practice, but one that differs significantly from traditional penetration testing and red team engagement. Why AI red teaming is differentAccording to Forrester, AI red teaming blends established offensive security techniques with new testing approaches designed specifically for AI-enabled systems. Traditional red teams focus on infrastructure, applications, APIs, and the SDLC. AI red teaming must also evaluate risks unique to AI, including bias, toxicity, safety failures, data exposure, and unintended behavior. These challenges are compounded by the probabilistic nature of AI. Models retrain, responses vary, and integrations evolve quickly. Static, point-in-time testing loses relevance fast. Given this, Forrester emphasizes the importance of evaluating the entire AI application stack, not just the model itself. A fragmented AI red teaming landscapeOne of the report’s central observations is how fragmented the AI red teaming market has become. Security teams generally encounter two primary approaches: Traditional offensive security providers extending pen testing and red team services to AI-enabled environments. AI and ML security vendors offering continuous, automated prompt-based testingEach approach brings value, but neither is sufficient on its own. Prompt saturation can identify patterns at scale but often lacks context. Manual testing provides depth but struggles to keep pace with rapidly evolving AI systems. Forrester’s conclusion is pragmatic: the most effective AI red teaming programs combine human-led testing with continuous, adaptive, and agentic techniques. This hybrid model more closely reflects real adversary behavior and produces findings that are both actionable and relevant. Why prompt testing alone falls shortPrompt injection and jailbreaks tend to dominate AI security discussions, but Forrester is clear that they represent only part of the overall risk picture. Many of the most significant vulnerabilities exist in systems surrounding AI: Application logic that routes prompts and responsesAPIs and integrations connecting models to enterprise dataSource code repositories and CI/CD pipelinesIdentity, access, and data controls governing AI usageIn short, AI is still software, just software with new failure modes. Red teaming that focuses only on prompts leaves critical blind spots. Early AI red teaming is imperfect but necessaryMany organizations are testing AI systems earlier than they would prefer, often driven by regulatory requirements, audits, or customer scrutiny. Forrester acknowledges that early AI red team engagements may be imperfect, but they still provide value by uncovering systemic issues, informing governance decisions, and demonstrating due diligence. The key shift is moving from one-time assessment to ongoing AI red teaming programs that evolve alongside the technology. From testing to operational AI securityThe Forrester report points to a broader shift: AI red teaming is increasingly connected to how organizations operationalize security day to day. Testing alone is not enough. Security teams need continuous visibility into where AI is being used, how it is accessed, and how risk is introduced across applications, users, and data. As AI becomes embedded into SaaS platforms, custom applications, and internal workflows, the attack surface expands rapidly. Without a consistent way to discover AI usage, assess risks, and enforce controls, many organizations are left stitching together point solutions, each addressing only part of the problem. Forrester highlights how providers such as SPLX are pushing AI red teaming beyond isolated assessments toward scalable, continuous evaluation of AI-enabled systems. This reflects a growing recognition that AI security must be built on foundational security principles; continuous verification, least-privilege access, and strong data protections, rather than implicit trust. Applying zero trust principles to AI securityWhile the report does not frame AI red teaming as a zero trust exercise explicitly, many of its recommendations align closely with zero trust principles. AI systems should not be trusted by default, whether they are public AI services, embedded SaaS features, or internally developed models and agents. Applying zero trust thinking to AI means continuously validating access to AI systems, tightly controlling how AI interacts with enterprise data, and monitoring behaviour across users, applications, and integrations. When paired with continuous AI red teaming, this approach helps organizations reduce risk while still enabling rapid AI adoption. Rather than adding more disconnected tools, security leaders are increasingly looking to unify AI discovery, adversarial testing, and runtime controls, and governance into a cohesive security architecture, one that scales as AI usage grows. What security leaders should do nextAI red teaming is still maturing, but the direction is clear. Based on Forrester’s research, security leaders should: Expand AI testing beyond prompt to include applications, integrations, and data flowsCombine human expertise with continuous, adaptive testing techniquesApply zero trust principles to AI access, data exposure, and runtime behaviourTreat AI red teaming as an ongoing security capability, not a one-time eventAI will continue to move fast. The organizations that succeed will be the ones that can scale AI securely, without increasing complexity and losing visibility. Learn moreDownload the full Forrester report on AI red teaming to explore testing approaches, engagement models, and best practices for securing AI-enabled applications.
