Stratix Can Now Generate Synthetic Evaluation Data (Public Preview)

By The LayerLens Team

TL;DR

• Stratix can now generate synthetic evaluation traces, in public preview, so a team can build an agent evaluation set before it has any production traffic.

• Start from three places: your own traces or an existing dataset, a built-in industry scenario, or a saved definition you can reuse.

• Preview one sample trace, scored on correctness, coverage, plausibility, and safety, and pick a quality level (Fast, Balanced, or High Quality) before generating the full batch.

• Each generated trace is a full multi-agent run that renders like a production trace and drops straight into evaluation.

• The built-in registry covers 14 industries and 30 scenarios, so a team with no traces can start cold.

• Every generated dataset carries its provenance back to the run and configuration that produced it.

The cold-start problem

Many agent failures come from interactions teams do not see until production traffic exists. Benchmarks and small hand-written suites often miss the messy, multi-step traffic that breaks agents in the field, so the first real test tends to arrive with the first real users. That is an expensive way to find out.

Stratix can now generate synthetic evaluation traces, in public preview. From the Datasets area you create a new dataset and generate traces from three starting points: your own traces or an existing dataset, a built-in industry scenario, or a saved definition you can reuse. The result is a real evaluation dataset, built in minutes, with no instrumentation and no production traffic required.

Most generated test sets are too easy

There is a catch to generating test data, and it is easy to miss. Ask a model for 500 test cases and it hands back 500 that look perfectly reasonable. Look closer and they resemble each other: most describe the ordinary, expected path, worded in slightly different ways. The rare, awkward inputs that actually break an agent in production barely show up. A set like that passes its own checks and still says nothing about how the agent will hold up once real users start doing unpredictable things.

Teams have already seen this gap with AI-written code. 94% of tech leaders rate it higher quality than human code in review, yet 78% report more production incidents once it ships. Looking right in review and holding up in production are two different things, and a synthetic test set is no exception.

Preview one trace before you generate the batch

Before committing a run, you generate a single sample trace and read it in full. It comes back scored on four dimensions: correctness against the scenario's reference answer, coverage of the variation you asked for, plausibility that a real user or system would behave this way, and safety against the scenario's policy rules. You also pick a quality level, Fast, Balanced, or High Quality, which trades speed against rigor. If the sample is thin, you adjust your instruction and generate another. Only a setup that produces a trace you trust earns the full run, and even then it feeds your normal evaluation before it gates a release.

The traces are real multi-agent runs

A generated trace is a full multi-agent run. It captures the distinct agents, the hand-offs between them, their tool calls, and a per-event waterfall timeline. It renders in the trace inspector exactly like a trace ingested from production, and it drops straight into evaluation with no conversion step. The one below was generated from a financial-services scenario: four agents and eighteen events, from wallet intake through risk classification to an incident responder.

Start cold in your industry

If you have no traces at all, you start from the built-in registry: fourteen industries, from healthcare to financial services to insurance, across thirty scenarios, each with a description and a ground-truth reference. A short instruction narrows it to the slice that matters, so a support team targets refund requests that fall outside policy. And a saved definition re-runs a setup a team has already reviewed, so the evaluation set that mattered last month can be regenerated the same way this month.

Every dataset can be traced to its source

Synthetic data is only useful if you can trust where it came from. Every dataset a run produces is tagged with its origin, the exact generation run and the configuration behind it, so you can always trace a set back to how it was made. When a customer or an auditor asks where the evaluation data came from, the answer is already attached to the data.

Key Takeaways

• Synthetic generation removes the cold-start blocker: a team can stand up a realistic evaluation dataset in minutes instead of waiting for production traffic.

• Quality control is built in: a scored sample-trace preview and three quality tiers let you check a setup before spending on a full run.

• The output is real: full multi-agent traces that evaluate exactly like ingested production traces.

• Breadth of sourcing: seed from your own data, from 30 industry scenarios, or from a saved definition.

• Provenance is attached to every generated dataset, so evaluations stay auditable.

Frequently Asked Questions

What is synthetic data generation in Stratix?

A public-preview capability that generates evaluation traces for AI agents from your own data or a built-in industry scenario, so you can build an evaluation set before you have production traffic.

How do I start if I have no traces?

Use the built-in registry of 14 industries and 30 scenarios. Each has a description and a ground-truth reference, and a short instruction narrows it to the cases you care about.

Can I check quality before generating a whole batch?

Yes. Generate one sample trace and read it, scored on correctness, coverage, plausibility, and safety, then choose a quality level before committing the run.

What does a generated trace look like?

A full multi-agent run with distinct agents, hand-offs, tool calls, and a per-event waterfall timeline. It renders like a production trace and is ready to evaluate.

Is it generally available?

It is in public preview. Start from your own traces or an industry scenario on Stratix.

Availability

Synthetic data generation is in public preview on Stratix. Start from your own traces or an industry scenario, generate one sample, and read it before you scale. Start generating an evaluation set on Stratix.