Every Major AI Provider Just Shipped Multi-Agent. None of Them Shipped Evaluation for It.

OpenAI added multi-agent orchestration to the Responses API on July 9. GPT-5.6 Sol Ultra can spawn parallel subagents on a single problem. Anthropic shipped multi-agent orchestration in Claude Managed Agents in May. Google's Gemini Enterprise Agent Platform includes agent-to-agent coordination. Every major provider now lets you deploy systems where multiple AI agents coordinate, delegate, and synthesize work across parallel workstreams.

None of them shipped evaluation infrastructure for multi-agent systems.

The deployment tooling is months ahead of the testing tooling. And the gap is about to get expensive.

What Breaks When Agents Talk to Each Other

Single-agent evaluation is a solved problem (or at least a well-understood one). You send a prompt, get a response, score it against criteria. The input-output relationship is linear. The failure modes are predictable: hallucination, off-topic responses, format violations, factual errors.

Multi-agent systems break every assumption that single-agent evaluation relies on.Error cascading. Agent A produces slightly wrong output. Agent B treats it as ground truth and builds on it. Agent C synthesizes both and amplifies the error. By the time the orchestrator returns a final answer, the original mistake has been laundered through three layers of confident-sounding reasoning. A single-agent evaluator scoring the final output might rate it highly because the logic chain looks coherent. The error is buried in step two of a twelve-step workflow.

Coordination failures. Two agents hand a task back and forth indefinitely. An orchestrator spawns subagents for subtasks that overlap, and they produce conflicting outputs that the synthesizer averages into mush. A planner pastes the entire execution trace into the next agent's prompt and exceeds the context window, silently truncating critical information.

Emergent behavior. When you deploy four agents with independent tool access, the system can take execution paths that no individual agent's behavior would predict. Agent interactions create a combinatorial space that static test suites cannot cover. You can unit test each agent individually, pass every test, and still have the multi-agent system fail on real tasks because the failure lives in the interaction, not the components.

Accountability gaps. When a multi-agent system produces a wrong answer, which agent caused it? The orchestrator that decomposed the task poorly? The researcher that retrieved outdated information? The reviewer that failed to catch the error? The synthesizer that misweighted inputs? In a single-agent system, blame is trivial. In a multi-agent system, root cause analysis requires tracing through the full conversation graph, and current observability tools were not built for conversation graphs.

What Evaluation for Multi-Agent Actually Requires

Evaluating multi-agent systems requires testing at three distinct levels, and most tools only operate at one.

Span-level evaluation scores each individual LLM call within the system: did this specific agent call use the right tool? Did it ground its response in the retrieved context? Did the tool call return valid data? This is where current evaluation tools work well. It is also the least useful level for catching multi-agent failures, because span-level success does not imply system-level success.

Trace-level evaluation scores the full multi-agent run as a unit: did the system complete the task? Did it stay within the latency budget? Did the plan the orchestrator created actually match the task requirements? Did subagent outputs get correctly synthesized? This is where the gap is widest. Scoring a multi-agent trace requires understanding the orchestration graph, not just the final output.

Interaction-level evaluation tests the boundaries between agents: when Agent A hands off to Agent B, does B receive sufficient context? When two agents disagree, does the system resolve the conflict correctly or just pick one? When an agent fails, does the system retry, route around, or silently drop the subtask? These interaction patterns are where production failures cluster, and no mainstream evaluation tool tests them systematically.​

The EU AI Act Is Coming (and the Delay Does Not Help You)

The EU AI Act's high-risk AI system obligations were pushed to December 2027 under the Digital Omnibus agreement finalized on June 29, 2026, a 16-month delay from the original August 2026 deadline. Most multi-agent orchestration in high-impact sectors (healthcare, finance, legal, employment) falls under the "high-risk" classification, triggering requirements for human-in-the-loop oversight, immutable audit trails, scenario-based incident testing, and traceability for each component in the system.

The delay gives teams more calendar time. It does not reduce the scope of what compliance requires. If you are deploying multi-agent systems in regulated industries, you need evaluation infrastructure that can attribute decisions to specific agents within the orchestration graph, maintain audit trails across agent interactions, and demonstrate that the system was tested against the failure modes that multi-agent coordination introduces.

"We tested each agent individually" will not satisfy a regulator asking why the multi-agent system produced a dangerous output. The regulation requires system-level assurance, and system-level assurance requires system-level evaluation.

​

Why Single-Agent Eval Tools Cannot Scale to This

Current evaluation platforms, even good ones, were built for single-agent workflows. They model evaluation as: input goes in, output comes out, score the output. The evaluation target is a function with one input and one output.

Multi-agent systems behave as processes with internal state, branching execution paths, inter-agent communication, and emergent behavior. Evaluating them requires:

• Graph-aware scoring that understands orchestration topology, not just final outputs

• Interaction testing that injects failures at agent boundaries to verify error handling

• Cascading error detection that traces how mistakes propagate through the agent chain

• Differential evaluation that compares multi-agent system outputs against single-agent baselines to verify that coordination actually adds value (in many cases, it does not)

This is a fundamentally different evaluation architecture than scoring individual LLM outputs against rubrics.

​

What Should Exist

The evaluation infrastructure for multi-agent systems needs to do four things that nothing on the market does comprehensively today.

Orchestration-aware test generation. Given a multi-agent system's architecture (which agents exist, what tools they have, how they coordinate), automatically generate test cases that target the interaction boundaries. If Agent A does research and Agent B does synthesis, generate cases where the research is subtly wrong and verify that the synthesizer catches it.

Cascade analysis. For any multi-agent failure, automatically trace backward through the agent graph to identify where the error originated, how it propagated, and which agent had the last opportunity to catch it. This is not trace visualization. This is automated root cause attribution across agent boundaries.

Emergent behavior detection. Monitor multi-agent systems in production for execution patterns that were never observed during testing. When the system takes a novel path through the agent graph, flag it for human review before it becomes a recurring failure mode.

Comparative evaluation. Automatically benchmark the multi-agent system against a single-agent baseline on the same tasks. Multi-agent coordination has overhead (cost, latency, error surface area). If the multi-agent system does not measurably outperform a single agent on a task, the coordination is adding risk without adding value.

​

Where This Lands

The providers shipping multi-agent orchestration are building deployment infrastructure. Evaluation infrastructure is a different problem, and it has always lagged deployment by 12 to 18 months. We saw this with single-agent systems: tool use shipped in 2023, evaluation for tool use matured in 2025.

The difference now is that multi-agent systems fail in ways that are harder to detect, harder to diagnose, and harder to fix. And the regulatory environment is not giving teams 18 months to figure it out.

Stratix was built to evaluate complex AI systems, not just individual model outputs. Its multi-generation evaluation architecture uses multiple Natural Language Judges that assess agent traces from different angles, scoring tool usage, trajectory quality, hallucination risk, and compliance independently. That architecture translates directly to evaluating multi-agent production systems, because the evaluation infrastructure already understands how agents interact.

If your stack includes multi-agent orchestration (or will soon), your evaluation infrastructure needs to match the complexity of what you are deploying. Single-agent eval tools applied to multi-agent systems will miss the failures that matter most: the ones that live in the interaction layer.