New in LayerLens: Bring Your Own Provider for AWS Bedrock and Azure OpenAI

By the LayerLens Team. LayerLens builds the evaluation platform for AI agents and models.

TL;DR

• Bring Your Own Provider is live in LayerLens: connect your AWS Bedrock or Azure OpenAI account under Settings > Providers and run evaluations on your own cloud capacity instead of ours.

• Add a provider connection, test it before saving, and LayerLens discovers your available models automatically. Choose which ones to expose in your workspace. Rotate or delete credentials at any time.

• Teams with existing enterprise agreements on AWS or Azure can now run all LayerLens evaluation workflows (judges, scorers, benchmarks, continuous eval) through their own accounts with their own spend controls.

• Available on all LayerLens plans.

LayerLens evaluations have always run inference to score your agents and models. Until today, that inference ran on LayerLens-managed capacity. For teams with their own AWS Bedrock or Azure OpenAI accounts, that meant evaluation traffic left their infrastructure boundary even when production traffic did not.

Bring Your Own Provider changes that. Connect your cloud account, and LayerLens sends evaluation inference through it.

What Bring Your Own Provider Does

Settings > Providers is a new page in LayerLens. Each provider connection stores your credentials, verifies the connection is live, and lets you select which models from that account are available in your workspace.

Once connected, every LayerLens evaluation workflow that calls a model (judges, scorers, benchmarks, agentic evaluations, continuous evaluation) can route through your provider. Your infrastructure handles the inference. LayerLens handles the evaluation logic, scoring, and results.

[INSERT SCREENSHOT: Settings > Providers page showing the add-connection flow or a saved provider with model list]

How to Use It

1. Open Settings > Providers in your LayerLens workspace.

2. Click Add Provider and select AWS Bedrock or Azure OpenAI.

3. Enter your credentials and click Test Connection. LayerLens verifies it can reach your account before saving.

4. LayerLens discovers your available models automatically and maps them to the LayerLens model registry. Choose which ones to expose in your workspace.

5. Save. Those models are now available anywhere LayerLens runs inference: judge configurations, benchmark runs, evaluation spaces.

Credentials are stored encrypted. You can rotate them from the same page without recreating the connection. Delete a provider connection when you no longer need it.

Who This Is Built For

Teams running evaluations at scale on existing enterprise cloud agreements. The evaluation spend shows up on your AWS or Azure bill, not a separate LayerLens line item.

Teams in regulated environments where data residency or egress restrictions apply to inference traffic. Model calls route through your cloud account.

Teams that want to evaluate fine-tuned or privately hosted models on Bedrock or Azure alongside frontier models already in the LayerLens catalog.

Frequently Asked Questions

Can I use both my own provider and LayerLens-managed models?

Each evaluation runs on one provider. You cannot switch providers mid-evaluation. But you can run one evaluation on AWS Bedrock and the next on LayerLens-managed capacity. Adding a provider connection does not remove access to the models already in the LayerLens catalog.

Which AWS Bedrock models are supported?

Any model available in your Bedrock account. LayerLens discovers your available models automatically and maps them to the LayerLens model registry. You select which ones to expose in your workspace.

Does this change how evaluation results are stored?

No. Evaluation results, traces, and scores are stored in LayerLens the same way regardless of which provider handled the inference. The provider connection only affects where the model calls go.

What happens if my provider credentials expire?

Evaluations that depend on that provider will fail until you update the credentials. LayerLens surfaces the connection error in the evaluation run log. Use Settings > Providers to rotate credentials.

Is Bring Your Own Provider available on all plans?

Yes. BYOP is available on all LayerLens plans.

Will more providers be supported?

Bedrock and Azure OpenAI are the first two. We want customer feedback on which additional providers are needed before committing to a roadmap. If you have a provider you need supported, reach out.

Connect your AWS Bedrock or Azure OpenAI account in Settings > Providers and run your next evaluation on your own infrastructure.