Agent environments

Overview#

An environment is a simulated service your agent can be tested against. It holds its own data, serves its own tools on the same endpoint paths as the real service, and knows the correct answer to every task written against it. A request for a record the environment never created returns a 404, the same as the real service would.

Environments are part of Agent Worlds in LayerLens. Find them in the sidebar under Evaluate. The catalog shows the environment types currently available in your workspace.

The environment workspace organizes generation, tasks, evaluations and results:

• Environment. Generate the data, tools and rules.
• Tasks. Questions drawn from the environment's own answer key, each with a known correct answer, so responses are scored automatically.
• Evaluate. Run supported models against the tasks under a spend cap. Every attempt is saved as a trace.
• Insights. See which tasks no model passes, which every model passes, which ones separate models, and whether a failure belongs to the agent or to the environment.

Generate an environment#

Open Environments and choose Generate. Pick a type, then describe the data you want in one sentence. The sentence fills in the settings, and you can adjust any of them before building.

SEC EDGAR settings include company size, restatement rate, event filings per year, first fiscal year, late share, years of filings, number of companies (1 to 40), and quarterly filings on or off. In one reviewed EDGAR example, a 15-company build produced 6 tools, 287 records and 25 rules.

Tools, records & rules#

The Environment tab shows what was built.

• Tools. Each tool shows its endpoint path with a copy button. SEC EDGAR has six tools on the real EDGAR paths, for example GET /submissions/CIK{cik}.json and /api/xbrl/companyfacts/CIK{cik}.json. Try the agent opens Chat.
• Records. The generated data. For EDGAR, each company has a CIK, exchange, ticker and its filings. Expand a company to see its 10-Q and 10-K rows with period, filed date, accession number and figure count.
• Rules. The checks the data was generated under, so the records stay consistent with each other.

Run an agent#

Open Evaluate and choose Run models.

• Pick one or more models from the catalog. The run dialog shows the models and providers available in your workspace.
• Pick a task set.
• Optionally add a system prompt.
• Set a dollar cap per model. A run stops spending when it reaches the cap.

The model calls the environment's tools directly, the same way it would call the real service.

Tasks#

Tasks come from the environment's answer key. A 15-company EDGAR build produces 45 tasks in six sets: quarter_revenue, quarter_net_income, annual_filing_date, year_end_assets, annual_accession and quarterly_report_count.

Each task shows its known answer and how it is graded. Tasks are graded by exact value: the check comes from the answer key, not from a model. Generate more adds tasks from the same answer key. Add a task lets you write one by hand.

Results#

Results show one row per task and one column per model. Turn on Failures only to narrow the grid. Click a cell to open that attempt.

The attempt panel shows pass or fail, the reason, cost, tokens in and out, tool calls, and the full transcript. From there you can choose Improve the prompt or Open in Traces.

Traces. Open in Traces shows every model call and tool call on a timeline, ending with the grade. Environment traces work like any other trace in LayerLens, so judges and datasets apply to them too.

Chat#

Talk to the same agent the grader runs against the environment's current data, with a dollar cap per turn. You can choose which available provider serves the model.

Insights#

Insights shows the pass rate across graded attempts and splits tasks into three groups: tasks no model passes, tasks every model passes, and tasks that separate models. Attempts that stopped early or could not be graded are counted separately from wrong answers, so a timeout never reads as a wrong answer.

Assets & activity#

• Assets. Download every trace the environment produced as one bundle to hand to a reviewer.
• Activity. Every job with its duration and cost. Jobs that changed the environment's version can be reverted.

Environments FAQ#

What's new#

The most recent additions to the platform.

Compare Runs

Compare two finished runs of the same benchmark, prompt by prompt. Open a benchmark's runs and pick a baseline run and a candidate run. The model can be the same or different as long as the benchmark matches. One click swaps them.

• The top row shows the accuracy change in points and how many prompts improved, regressed or stayed the same.
• Show regressions lists each prompt that got worse, with the baseline answer, the candidate answer and the correct answer.
• Custom scorers can also be picked in the run dialog without attaching them to the benchmark first.

Judge Automation Rules

Every judge has an Automation tab. A rule runs the judge on each new trace that matches a filter.

• New rule: name, filter (agent, framework, status, tag; leave blank to match every new trace in the project), sample rate, and daily cap (UTC).
• Before saving, the dialog shows how many traces matched in the last 24 hours and how many evaluations the rule would run per day.
• Activate this rule turns it on. Leave it off to save the rule without running it.

Matching traces are judged within about a minute, and the verdict is saved on the trace.

Policy Coverage

Upload a policy, regulation or SOP, for example a SOC 2 report, a GDPR mapping or an internal procedure. LayerLens proposes judges and scorers from its clauses and cites the clause behind each one. Nothing is created until you approve it. Approved judges can take automation rules like any other judge.

Find Policy Coverage in the sidebar under Catalog.

Bring Your Own Provider

Run LayerLens evaluation calls on your own AWS Bedrock or Azure OpenAI account.

• Set up. Go to Settings › Providers › Add Provider. Choose AWS Bedrock or Azure OpenAI, enter your credentials and click Test Connection. LayerLens lists the models your account can reach; pick the ones to use and save. Credentials are stored encrypted and can be rotated or deleted on the same page.
• Use. Your models appear wherever LayerLens calls a model: judges, benchmark runs and evaluation spaces. Each evaluation uses one provider; different evaluations can use different providers. Catalog models stay available.
• Data. Results, traces, scores, prompts and completions are stored in LayerLens as usual. The provider setting changes only which account runs the model calls.
• Expired credentials. Evaluations on that provider fail until you update them, and the error shows in the run log.

Release notes#

• September 21, 2026Environments: consistent header actions and breadcrumbs; choose the provider that serves a model in Chat.
• September 18, 2026Environments: models call environment tools natively; fixed verdicts not showing on the Runs grid.
• September 17, 2026Bring Your Own Provider for AWS Bedrock and Azure OpenAI. Sessions stay signed in for about 30 days. New breadcrumbs, notifications and page headers.
• September 9, 2026Compare Runs. Judge Automation Rules. Policy Coverage. Custom scorers in the run dialog. Reveal and copy API keys.
• August 31, 2026Benchmarks show their source, link and code.
• August 10, 2026Python SDK 1.11.0: fixes for durations, metrics and zero-cost records; new ScorerResult model; OpenTelemetry (OpenInference) import.