LayerLens platform guide

Welcome#

LayerLens is the platform for getting AI agents ready for the work that matters. It has three parts.

• Agent Evals. Run models and agents against benchmarks, score them with judges and scorers, evaluate production traces, and compare runs.
• Agent Environments. Test an agent inside a simulated version of the service it will use, with its own data, tools and answer key. See Environments.
• Synthetic Data. Generate realistic multi-agent trace datasets from your own traces or a built-in scenario.

Surfaces

To get started, sign up at app.layerlens.ai, create an API key under Admin › API Keys, and follow one of the quickstarts below.

Quickstart#

Quickstart: SDK

pip install --extra-index-url https://sdk.layerlens.ai/package "layerlens[cli]"
export LAYERLENS_STRATIX_API_KEY="your-api-key"

Run a benchmark evaluation:

from layerlens import Stratix

client = Stratix()
model = client.models.get_by_key("MODEL_KEY_FROM_CATALOG")  # Replace with a current Catalog key
benchmark = client.benchmarks.get_by_key("arc-agi-2")
evaluation = client.evaluations.create(model=model, benchmark=benchmark)
result = client.evaluations.wait_for_completion(evaluation)
print(f"Accuracy: {result.accuracy}")

Create a judge and score a trace:

judge = client.judges.create(
    name="Response Quality Judge",
    evaluation_goal="Rate whether the response is accurate, complete, and well-structured",
)
upload = client.traces.upload("./my_traces.json")
trace_eval = client.trace_evaluations.create(trace_id=upload.trace_ids[0], judge_id=judge.id)
result = client.trace_evaluations.wait_for_completion(trace_eval.id)
print(result.score, result.passed, result.reasoning)

Every method also works with await on AsyncStratix.

Quickstart: web app

1. Sign in at app.layerlens.ai.
2. Open a benchmark in the catalog and choose Run. Select one or more models and any custom scorers. Results appear in Evaluations as rows finish.
3. Open Traces and upload a trace file, or add an adapter to your agent so traces arrive on their own.
4. Open Judges, create a judge with a name and an evaluation goal, and run it on a trace.
5. Open the judge's Automation tab and add a rule so new matching traces are judged automatically.

Glossary#

Agent

The system being tested: a model plus prompts, tools and logic. A model is the underlying LLM, such as GPT, Claude or Gemini.

Benchmark

A dataset plus a way to grade it. LayerLens hosts public benchmarks, and organizations can upload private ones.

Evaluation

One run of a model or agent against a benchmark, scored per row and overall.

Compare Runs

Two finished evaluations of the same benchmark, compared prompt by prompt.

Judge

An LLM grader with an evaluation goal. System judges come with LayerLens; custom judges belong to your organization.

Judge Automation Rule

A filter on a judge so new matching traces are judged as they arrive.

Scorer

A reusable grader used inside an evaluation, either a model with a prompt or a code check.

Trace

The record of one agent run: inputs, outputs, each model call and tool call, timings, cost and errors.

Trace evaluation

Judges run over one trace or a set of traces.

Dataset

A named, versioned set of traces used as input to a trace evaluation.

Synthetic data

Generated traces built from your own traces, a built-in scenario or a saved definition.

Space

A saved leaderboard of chosen models and benchmarks that you can share.

Adapter

SDK code that converts traces from a framework, provider or protocol into the LayerLens format.

Environment

A simulated service with its own data, tools and answer key, used to test an agent.

Policy Coverage

Upload a policy, regulation or SOP and LayerLens proposes judges and scorers from it.

Bring Your Own Provider

Run evaluation calls on your own AWS Bedrock, Azure OpenAI, OpenAI, Anthropic or Google Gemini account.

Evaluations#

An evaluation runs a model or agent against a benchmark or dataset and saves scores per row and overall.

• Create. Open a benchmark and choose Run. Pick one or more models (catalog models, models from your own provider, or your own endpoint), pick judges and scorers, and run. Rows appear as they finish.
• Browse. Filter runs by model, benchmark, scorer or judge, status and date.
• Open a run. Each row shows the prompt, the answer, the expected answer when the benchmark has one, and each grader's verdict.
• Compare. Pick a baseline and a candidate run of the same benchmark. The top row shows the accuracy change and how many prompts improved, regressed or stayed the same. Show regressions lists each prompt that got worse, with both answers and the correct one.

SDK

client.evaluations.create(model=, benchmark=), wait_for_completion(), client.results.get_all(evaluation_id). In Result.metrics, built-in metrics are numbers and custom scorer results are ScorerResult objects; a scorer that failed to run has score=None.

CLI

layerlens evaluate run | list | get, layerlens evaluations run | compare, layerlens bulk eval.

Judges#

A judge is an LLM grader with a name, an evaluation goal and a model that does the judging. Judges run on traces, on evaluation outputs, and automatically through automation rules.

System judges come with every workspace and carry a System badge. Clone one to customize it. Custom judges belong to your organization.

Create in the app

Go to Judges › New judge. Give it a name, an evaluation goal and a judging model, write the rubric, and test it on examples before saving.

Create with the SDK

judge = client.judges.create(
    name="Faithfulness",
    evaluation_goal="Every claim in the output is supported by the context",
)

Test a judge

Run it on one trace from the judge page, or with layerlens judge test --judge-id JUDGE_ID --trace-id TRACE_ID.

Automation rules

Every judge has an Automation tab. A rule has a name, a filter (agent, framework, status, tag), a sample rate and a daily cap. Matching traces are judged within about a minute and the verdict is saved on the trace.

Policy Coverage

Under Catalog, upload a policy, regulation or SOP. LayerLens proposes judges and scorers from its clauses and cites the clause behind each. You approve each one before it is created.

System judges

Scorers#

A scorer is a reusable grader used inside an evaluation. It is either a model with a prompt or a code check such as exact match, regex or JSON schema.

• Create. Go to Scorers › New scorer. Add a name, description, model and prompt. Use placeholders such as {{output}}, {{expected}}, {{context}} and {{input}}, test on examples, and save.
• Use. Pick any number of scorers in the run dialog. Each one adds a verdict per row.
• SDK and CLI. client.scorers.create / get / list / delete; layerlens scorer create | get | list | delete.

Traces#

A trace records one agent run: inputs, outputs, and each model call and tool call with timing, tokens, cost and errors. Production evaluations include a tamper-evident trace record.

Getting traces in

• Upload a JSON or JSONL file from Traces › Upload, or with client.traces.upload("./traces.json").
• Add an adapter to your agent so traces arrive on their own.
• Import OpenTelemetry spans in the OpenInference format, as OTLP JSON or protobuf.

Viewing a trace

See the agents involved, the number of events, duration and cost, a timeline of model and tool calls, and each call's inputs, outputs and errors.

CLI

layerlens trace list | get | search | export | delete.

Trace evaluations

A trace evaluation runs judges and scorers over one trace or a set of traces. Pick the traces (a dataset, a filter or a single trace), pick the judges and scorers, and run. Results show a score per judge or scorer, a verdict per trace, and a link from each failure to the event that caused it.

SDK

client.trace_evaluations.create(trace_id=, judge_id=), wait_for_completion().

Datasets & spaces#

Datasets

A dataset is a named, versioned set of traces, used as input to a trace evaluation.

• Create. Go to Datasets › New Dataset and build one from existing traces, from an uploaded file, or by generating synthetic traces.
• View. Each dataset has a Data Preview, a Schema (traces added later must match it) and a Version History. Synthetic datasets record the generation job that made them.

From a dataset you can add more traces or run an evaluation on every trace in it.

Spaces

A space is a saved evaluation setup: models, dataset or benchmark, and scoring. Rerun it after each change to your agent and track the scores over time.

SDK and CLI

client.evaluation_spaces; layerlens space create | get | list | delete.

Continuous evaluation#

Traces arrive on their own through adapters or OpenTelemetry. Judge Automation Rules judge each new matching trace within about a minute, within the sample rate and daily cap you set. For a fixed regression set, save a dataset and rerun a trace evaluation on it after each change.

Synthetic data#

Generate realistic multi-agent traces so you have an evaluation dataset before real traffic arrives.

Start. Go to Datasets › New Dataset › Generate synthetic traces.

Step 1: pick a source

Start from your own traces or a dataset, from a built-in industry scenario, or from a saved definition.

Step 2: set it up

• Guidance prompt: up to 10,000 characters describing what to generate.
• Quality: Fastest, Balanced or Highest.
• Number of traces: 1 to 10,000.
• Destination: a new dataset, an existing dataset, or loose traces.

Generate test trace makes one preview scored on correctness, coverage, plausibility and safety, so you can check the output before the full run. Save as reusable definition keeps the settings for next time.

Generation runs in the background. Cancel keeps what was already generated and marks the dataset partial.

What you get

Full multi-agent traces with agents, handoffs and tool calls, viewable in Traces like any other trace.

CLI

layerlens synthetic templates, layerlens synthetic generate.

SDK, CLI & adapters#

Install and authenticate

Python 3.8 or higher. Tested on 3.9 to 3.12; some framework extras need 3.10 or later.

pip install --extra-index-url https://sdk.layerlens.ai/package "layerlens[cli]"
export LAYERLENS_STRATIX_API_KEY="your-api-key"

Create the key at Settings › API Keys. Pass Stratix(api_key=...) to use a different key. LAYERLENS_STRATIX_BASE_URL changes the API address (default https://api.layerlens.ai/api/v1). Requests time out after 10 minutes by default; client.with_options(timeout=120.0) changes it for one call.

Clients

• PublicClient: models, benchmarks and head-to-head comparisons, for example pc.comparisons.compare_models(benchmark_key=, model_key_1=, model_key_2=, outcome_filter=).
• Stratix: evaluations, results, traces, judges, scorers, trace evaluations and spaces.
• AsyncStratix and AsyncPublicClient: the same methods with await.

Errors. NotFoundError, AuthenticationError, APIError and APIResponseValidationError, all subclasses of StratixError.

from layerlens import Stratix, NotFoundError, AuthenticationError, APIError

try:
    model = client.models.get_by_id("model-id")
except NotFoundError as e:
    print(f"Not found: {e.message}")
except AuthenticationError as e:
    print(f"Auth failed: {e.message}")
except APIError as e:
    print(f"API error ({e.status_code}): {e.message}")

The full SDK reference is in the docs/ folder of the stratix-python repository.

CLI reference

Installed as both layerlens and stratix. Options on every command: --api-key, --host, --port, --format table|json, --verbose, --version.

Adapters

Adapters convert traces from the tools you already use into one LayerLens format, so the same judges keep working when you switch frameworks. Adapters are in public preview. Install one extra per stack:

pip install --extra-index-url https://sdk.layerlens.ai/package "layerlens[langchain]"

Thirty adapters are in public preview across agent frameworks, model providers, agent platforms, protocols, observability and embeddings. Run layerlens integration list to see the adapters supported by the installed SDK version.

OpenTelemetry. The SDK reads OTLP exports in the OpenInference format and keeps service details such as service.name on the trace.

CI gates

layerlens ci report writes a markdown summary for the CI job. The samples/cicd/ folder has a threshold check (quality_gate.py --threshold 0.85 fails the job on a regression), a pre-commit hook and a GitHub Actions workflow. Any CI system that runs Python can use the same script.

Samples

The samples/ folder in the stratix-python repository has runnable examples for adapters, CI, Claude Code, the CLI, CopilotKit, core SDK calls, industry use cases, instrumentation, integrations and MCP.

Models & providers#

Models and benchmarks

The catalog lists the models and benchmarks LayerLens can evaluate. Each benchmark shows its source, link and code. Compare two models on a benchmark from the catalog or with pc.comparisons.compare_models(...); use outcome_filter to see only the prompts where one model fails.

Bring Your Own Provider

Run evaluation calls on your own AWS Bedrock or Azure OpenAI account. Go to Settings › Providers › Add Provider, enter credentials, click Test Connection, pick your models and save. Each evaluation uses one provider. Results and prompts are stored in LayerLens either way. See the full walkthrough under What's new.

Admin#

Organizations

A user can belong to several organizations, each with its own data and members. API keys work only in the organization that issued them.

Team

Invite teammates by email from Settings › Team and assign each a role. Removing someone keeps their work in the organization.

API keys

Create, reveal, copy, rotate and revoke keys from Settings › API Keys.

Sessions

You stay signed in for about 30 days.

Security#

• Each organization's data is visible only to that organization's members, and API keys are scoped to one organization.
• Provider credentials are stored encrypted.
• Production evaluations include a tamper-evident trace record.

For security questions or a review, contact [email protected].

Support#

Troubleshooting

Contact

Support: [email protected]. Security: [email protected]. Sales: [email protected]. SDK issues: GitHub issues on stratix-python.