Most of what gets sold as AI red teaming is a slide deck of jailbreaks. Someone coaxes a chatbot into saying something it should not, screenshots it, and calls the model tested. That is a demo. It proves a single prompt worked once, on one build, under conditions nobody wrote down. It tells you almost nothing about whether the system is fit for release, and it cannot be run again to check whether a fix held.
The gap between that theater and a real evaluation is the whole subject of this piece. A real evaluation has a scope you can defend, a record you can reproduce, and a decision rule that connects a finding to a release gate. Everything else is content.
Start by fixing what red teaming is
The US Executive Order gives the working definition most standards now build on: AI red teaming is a structured testing effort to find flaws and vulnerabilities in an AI system, often in a controlled environment and in collaboration with the developers, using adversarial methods to surface harmful outputs, undesirable behaviors, and misuse risks, as OpenAI's external red teaming white paper records it. Two words in that definition carry the weight. Structured means the effort has a method behind it rather than a mood. Vulnerabilities means you are looking for conditions an adversary could exploit, which is a different thing from outputs you find distasteful.
CISA sharpens this by placing red teaming inside an older discipline. AI red teaming is a subset of AI Testing, Evaluation, Validation and Verification, and AI TEVV is itself a sub-component of software TEVV, which has been used for more than four decades, CISA argues. The practical consequence is that you do not get to invent a new vocabulary for AI. A finding is a defect, a defect has a severity, a severity feeds a release decision, and a fix gets re-tested. If your red team output does not slot into that machinery, it is not an evaluation. It is a report.
There is a second, quieter warning worth carrying. A critical paper on the practice argues that AI red teaming has drifted from its origin as a critical-thinking exercise into a narrow hunt for model-level flaws, and that this narrowness overlooks the sociotechnical system in which the model runs, as the authors of "Red Teaming AI Red Teaming" put it. Hold that thought. It is the difference between testing a model and testing a system, and it decides most of what follows.
What to test: model, system, and the seams between them
The same critical paper proposes two levels, and the split is the most useful planning tool I know. Micro-level model red teaming examines the model itself. Macro-level system red teaming spans the entire development lifecycle and the interactions between models, users, and environments, the authors write. A scope that stops at the model boundary will miss the failures that reach production, because production failures live in the seams.
For the model layer, the OWASP Top 10 for LLM Applications is the shared vocabulary. The OWASP GenAI project frames its red teaming work as both a test of the model and a test of the model's safeguards, and treats guardrail behavior as a first-class target rather than an afterthought, per the initiative's overview. That distinction matters for scoping. A prompt-injection finding against a bare model and the same injection against a model behind an input filter are two different tests with two different meanings.
For the system layer, OWASP's vendor criteria draw a line most buyers miss. Meaningful adversarial testing spans simple GenAI systems such as chatbots and RAG applications and advanced systems including tool-calling agents, MCP architectures, and multi-agent workflows, and the guide exists partly to separate that from superficial jailbreak-only offerings, OWASP states. A tool-calling agent can read a file, call an API, and act on the result. Testing its refusal behavior on a chat prompt tells you nothing about whether a poisoned document can make it exfiltrate data. Those are different attack surfaces, and a scope document should name which ones are in bounds.
Figure 1: A jailbreak-only offer reaches chatbot refusal in part and leaves the other three surfaces untested. The surfaces come from OWASP's simple and advanced system types. The coverage grades are my own.
The planning question is not "can we break it" but "which surfaces did we agree to test, and which did we knowingly leave out." A scope that names its exclusions is honest. A scope that implies total coverage is the first red flag.
What a real finding looks like: reproducibility is the whole game
Here is the claim the industry keeps failing. Transparency audits of leading generative AI companies found only 25 percent of policy enforcement and evaluation criteria satisfied on average in 2023, and no company provided reproducible evaluations to characterize the effectiveness of its risk mitigations, the Safe Harbor authors report. A result you cannot reproduce is not a finding. It is an anecdote with a timestamp.
Reproducibility is what lets a red team result feed the software machinery CISA describes. To record a finding so it survives contact with an engineer, capture at minimum: the exact input, the model or system version and configuration, the full output, the conditions of access, and the harm or vulnerability class it maps to. OpenAI's white paper is explicit that qualitative red team findings are meant to seed automated evaluations, where new metrics derived from the findings track whether a desired behavior holds as models evolve, OpenAI writes. You cannot build an automated eval from a screenshot. You can build one from a logged input, an expected refusal, and a version stamp.
One structural obstacle deserves naming. The Safe Harbor paper documents that leading companies' terms of service prohibit independent evaluation of the most sensitive flaws, and that researchers doing good-faith work have had accounts suspended without warning, justification, or appeal, the authors write. If your evaluation depends on external researchers, reproducibility requires a legal and technical safe harbor that lets them run the test twice and publish. Without pre-agreed rules of engagement and a vulnerability disclosure policy, your "independent" assessment is running on borrowed permission that can vanish mid-test.
Figure 2: Both bars stand at two. Two of the four questions a jailbreak demo cannot answer, and two only a real evaluation answers. The chart plots no group for a question a demo answers.
What blocks a release: the decision rule you write before you test
Many findings do not need a fix. The critical-practice paper is blunt that organizations do not need to address every risk red teamers identify, but they benefit from comprehensive awareness of failure modes before deployment, the authors write. So the question is never "did we find something." You always find something. The question is which findings cross the line you drew in advance.
Treat the release gate the way software TEVV treats a defect. The same paper calls for a feedback loop that tracks identified issues, prioritizes them by risk assessment, implements mitigations, and verifies the fixes do not introduce new problems, the authors write. A finding blocks a release when its severity clears a threshold you set before testing began, and it stays blocked until the fix is re-tested against the recorded reproduction. Set the threshold after you see the findings and you will move it to hit the ship date every time.
CISA's framing gives the escalation its teeth. The purpose of TEVV is to establish that a system is fit for purpose, CISA notes. A blocking finding is one that shows the system is not fit for its stated purpose, which is a higher bar than embarrassing it. A model that can be coaxed into rude language may be a quality issue. A tool-calling agent that leaks another user's data through an injected document is a fitness failure. Write your gate against fitness for purpose and it will survive the release meeting.
How to tell a real evaluation from a demo
Run the offer through four questions. Did they scope the system or only the model? Can every finding be reproduced from its record? Was the decision rule written before testing? Does each blocking finding tie to fitness for purpose rather than taste? The OWASP vendor criteria exist to help buyers separate meaningful adversarial testing from jailbreak-only offerings, and they frame the same split around green flags and red flags, realistic threat models, evaluation rigor, tooling quality, and governance considerations, OWASP notes.
A demo answers the first two questions with a shrug and never reaches the last two. A real evaluation answers all four in writing, before anyone opens a terminal. That document, not the jailbreak reel, is the deliverable you should be paying for.