"AI evaluation platform" now describes three different jobs

Author:

The LayerLens Team

Last updated:

Published:

Ten products sell themselves as an AI evaluation platform, and they are built around three different units of work.

Open the product pages for Braintrust, Arize, LangSmith, Langfuse and Galileo side by side and the same four verbs come back: trace, evaluate, monitor, improve. Each one offers LLM-as-judge scoring. Each one puts a pass rate on a dashboard. A team shipping a customer support agent reads those pages, picks the cleanest quickstart, spends two weeks wiring it in, and finds out in month two that it does not answer the question they bought it for.

The comparison misses it because at the feature-grid level the difference does not exist. Every vendor traces. Every vendor scores. What separates them sits underneath, in what each product treats as the unit of work: a single request, a change, or a whole model. A team that needed the third but bought the first ends up with excellent traces and no answer.

TL;DR

  • Ten evaluation products split into three jobs by unit of work: the request (Langfuse, Arize Phoenix, LangSmith), the change (Braintrust, DeepEval, MLflow, Promptfoo, Galileo), and the model (Artificial Analysis, Stratix). The first two are crowded. The third has two occupants here.

  • Three of the ten have been acquired or agreed to be, one of them by a model provider. Two deals have closed and one is still pending. Dates and status are in the table below.

  • Licensing does not split the way the marketing does. The Langfuse core is MIT with its enterprise directories under separate terms, and Promptfoo, DeepEval and MLflow ship OSI-licensed cores. Arize Phoenix is source-available under Elastic License 2.0, free to self-host and not OSI open source. Braintrust, LangSmith and Galileo are commercial, as are Arize's hosted AX tier and Stratix, and Stratix has no self-host path.

  • Four questions settle which job a team needs, and the first one decides most cases: does the model come from inside the building or outside it.

Three jobs behind one AI evaluation platform vocabulary

Teams land in one of three places, and most teams sit in more than one at different times.

Job one, the request. Somebody needs to see what a single agent run actually did. Which tools it called, in what order, where latency went, which step produced the wrong answer. Langfuse, Arize Phoenix and LangSmith are built around this unit. The Langfuse core ships under MIT, with enterprise features held back under separate terms, and self-hosts through Docker Compose or Kubernetes. Phoenix builds on OpenInference and OpenTelemetry, which matters if a team wants instrumentation it can point somewhere else later. LangSmith is commercial and framework-agnostic despite the LangChain name, tracing the OpenAI SDK, the Anthropic SDK and custom clients alike.

Teams reach for this first because it is the most legible problem. An agent did something strange, and somebody wants the receipt.

Job two, the change. A team has a prompt edit, a model swap, or a retrieval tweak, and needs to know whether it made things better before it merges. The unit here is the diff. Braintrust is the clearest expression of it, pairing production tracing with offline experiments and pre-ship regression gating, and it raised an $80M Series B led by ICONIQ in February 2026. DeepEval takes the same job in pytest form, running locally and in CI, with Confident AI as the hosted layer above it. MLflow, Linux Foundation-governed since 2020, carries GenAI evaluation into teams already running it for classical ML. Promptfoo approaches the diff from the security side, red-teaming prompts and agents for exploitable behavior.

Galileo sits here too. Its Luna evaluation models distill LLM-as-judge into smaller scorers so that a team can score high traffic volumes without paying frontier prices per call. Galileo puts Luna-2 at 97% cheaper than GPT-4 evaluation. That is a vendor figure and should be read as one.

Job three, the model. Somebody needs to know how a model behaves when nobody on the team built it, measured the same way every time, so that this month's number means the same thing as last month's. The unit is the model. Artificial Analysis publishes an Intelligence Index composed of nine evaluations, GPQA Diamond and SciCode among them, at v4.1 as of June 2026. It also reports throughput and cost-per-task, and it benchmarks hosting providers alongside the models themselves. LayerLens runs Stratix in this job, re-evaluating a public catalog of 171 active models from 28 companies against 78 public benchmarks spanning 18 categories. The whole catalog is re-scored on the same suite each cycle, so two runs are comparable.

Job three is the one teams discover late. It becomes urgent the first time a provider ships a point release and somebody has to answer whether the agent got worse because of the prompt change or because the model underneath moved.

[INSERT IMAGE: IMG-1 - Matrix mapping ten evaluation products against three jobs, with 2026 ownership changes.]

The ownership question no comparison table has a column for

Product

Status

Announced

Closed

Parent

Langfuse

Completed

January 16, 2026

Announced as closed

ClickHouse

Promptfoo

Pending

March 9, 2026

Not yet closed

OpenAI

Galileo

Completed

April 9, 2026

May 22, 2026

Cisco (Splunk)

ClickHouse announced Langfuse as a closed acquisition rather than an intent, so it carries no separate closing date, and it committed to keeping the MIT license on core features. The other seven products in the roster remain independent.

The practical consequence for a buyer has nothing to do with intentions. A product's roadmap eventually serves its parent's roadmap, and a buyer signing a two-year commitment is signing up for whichever direction that turns out to be. Promptfoo's blog gives one observable data point: its most recent post is dated March 12, 2026, three days after the OpenAI announcement, and nothing has published in the four and a half months since, checked July 28, 2026. That is a fact about publishing cadence, and a buyer can weigh it however they like.

There is a narrower version of this question that only applies to job three, and the Promptfoo deal is the live example. When the organization measuring a model also builds that model, as OpenAI would after acquiring an evaluation tool, the measurement inherits the builder's priorities. That does not make the numbers wrong. It makes them the kind of number a team should reproduce before betting a deployment on it.

Four questions that decide which evaluation platform you need

Who wrote the model? Inside the building, jobs one and two cover it. Outside the building, and especially across several vendors, job three becomes the load-bearing one.

Does the number need to survive a version change? A pass rate compared across weeks needs a fixed harness and a pinned prompt set, scored by a judge that has not silently changed underneath it. That is its own engineering problem. Job two tooling optimizes for the current diff and does not always keep those constant across months.

Who reads the output? A developer debugging a run needs a waterfall. A team lead approving a merge needs a threshold. An auditor needs a record of what ran and under which configuration, with a timestamp on it. Those three consumers rarely accept each other's artifact.

What happens when the vendor's parent reprioritizes? For an open-source core with a real self-host path, the answer is that the team keeps running it. For a commercial product, the answer is a migration.

What this framework does not settle

The three-job split is a purchasing lens, and it has real limits.

It does not tell a team how many tools to run. Most serious deployments end up with two, usually a tracing layer plus something that gates changes, and the integration cost between them is a genuine tax the framework ignores.

It also flattens products that deliberately span jobs. Braintrust does tracing as well as gating. Arize ships Phoenix for job one and AX for parts of job two. Calling either one "a job two tool" is a simplification a procurement team should not take literally.

And it says nothing about the thing most teams actually get wrong: having no evaluation criteria worth automating. A tool that scores against a vague rubric produces a precise number about nothing. Choosing the right category leaves that problem exactly where it was.

Stratix only does job three, and that is the constraint

Because Stratix re-scores its whole public catalog on the same suite every cycle, a Stratix number carries a history: a model can be compared against its own past score rather than only against whatever else shipped that month. That is the property job three exists to provide, and it is the only reason a fixed method is worth its cost. The same infrastructure carries into job two through custom judges, so a team can score its own agent traces against a rubric it wrote and compare the result to the public catalog on a common method.

[INSERT IMAGE: IMG-2 - Stratix comparison view scoring two model evaluations side by side on one benchmark.]

Two limits go with that. A fixed public benchmark suite is contaminable, and a model vendor that wants a number can tune toward a published suite. That is a standing weakness of every fixed-method measurement, including this one. And on breadth of question, Artificial Analysis is ahead, because it prices and times what it measures and Stratix does not. A team whose question is which endpoint to buy should start there.

Stratix also does not replace a tracing layer for a team whose only question is what happened inside one request last Tuesday. Langfuse and Phoenix are both good at that and both free to self-host, the Langfuse core under MIT and Phoenix under Elastic License 2.0. A team already running one should keep it.

Frequently Asked Questions

Can one tool cover all three jobs?

No product currently covers all three well. Job three requires continuously re-running a fixed benchmark suite against models the vendor does not build, an operating cost the job one and job two vendors have not taken on. Artificial Analysis and LayerLens are the two in this list running that workload.

Does a self-hostable core protect against an acquisition?

Partly. A license that permits self-hosting means a team can keep running the version it has if a roadmap turns. It does not guarantee future features land in that build, and the terms on today's release do not bind tomorrow's. Check whether the license is OSI-approved or source-available, because Elastic License 2.0 and MIT give a team different options if a parent company changes direction.

We already run a tracing tool. Do we need a second product?

Usually yes, and that is the most common end state rather than a failure of planning. Tracing answers what happened inside one run, and it holds no opinion about whether this week's build is better than last week's. If the team is making merge decisions or tracking a model it does not own, that second question needs tooling that keeps a harness and a prompt set constant, which a tracing layer is not built to do. Budget for the integration cost between the two, because it is real and no vendor absorbs it.

Does the EU AI Act decide which job a team needs?

For most readers, no. From August 2, 2026, Article 101 gives the European Commission power to fine providers of general-purpose AI models up to 3% of worldwide annual turnover or 15 million euros, whichever is higher. That binds the organization that trained the model. A team deploying somebody else's model sits downstream of it, receiving the provider's documentation rather than producing it. The measurement question stays an engineering one.

Before the next evaluation demo, write down which unit the team is actually arguing about: one request, one change, or one model. Bring that sentence to the call and most of the feature grid stops mattering, because a vendor built around a different unit will start answering a question nobody asked.

Sources

‹ AI regression testing: your quality gate passes while the agent gets worse

The Company That Builds the Agent Should Not Grade Its Homework ›