Where AI Agents Fail on Stripe and Apollo

Stripe and Apollo both ship MCP servers (Stripe, Apollo), which puts billing reports and CRM writes in an agent's hands while leading agents still complete only about 58% of single-turn CRM tasks and at least one agent benchmark grades a write tool's success message as proof the write happened. We reviewed 1,537 attempts by 13 models on a Stripe billing replica and ran 9 models 3 times each on an Apollo CRM replica.

Gemini 3.8 Flash and Claude Opus 5.5 matched the answer key on all 6 "paid in total" questions, while Claude Sonnet 5.5, Claude Haiku 5.5, Gemini 4 Argon, GPT-6 Astra and GPT-6.1 Sol matched none, and on both customers we checked, those 5 subtracted a refunded charge the key counted.

On Apollo, the write task named a CRM stage that doesn't exist. GPT-5.5 and Claude Haiku 5.5 went ahead without asking and passed all 3 attempts, while GPT-6 Astra stopped to ask every time and failed the CRM check. Both results turn on task wording, so write the definition and the fallback into the task and grade the CRM record, since 2 attempts reported a result the CRM didn't show.

Every Stripe Lookup Passed

The Stripe environment is a generated billing account for a fictional company, Synthetic Softworks, with 10 customers and 120 invoices behind 8 tools, including list_invoices. It holds 111 questions, each with a known answer graded by exact value: the grader looks for the known number or ID in an agent's final answer. 13 models have run against it, for 1,403 passes in 1,537 attempts.

Grouping the latest result from each model on each question shows where the misses sit, from lookups such as "What is the total of invoice in_synth00000015?" to totals such as "Which customer has paid the most?":

Every question that needed one record passed on every model. Every miss needed an agent to count a list, add up amounts, or decide what an amount should include.

Refunds Split "Paid in Total"

The hardest Stripe questions ask how much a customer has paid in total. For Northgate Foundry Corporation the known answer is 1,347,732 cents, the sum of its 10 successful charges. One of those charges, $1,077.84, was later refunded in full.

8 of the 13 models that ran it (14 attempts in all) answered exactly 1,239,948, which is the known answer minus the refund. Claude Fable 5.1 laid out the whole calculation: "1,347,732 gross. Less the 107,784 refund → 1,239,948 net paid." It found the right charges, did the arithmetic correctly and failed, because it answered a different question than the answer key.

For GPT-6 Astra and GPT-6.1 Sol, these 6 questions account for 6 of the 7 Stripe questions they did not pass every time.

The same definition moves other answers. Asked which customer has paid the most, a Claude Haiku 5.5 run summed each customer net of refunds and named Halcyon Systems Technologies, while the key names Stonebridge Works Laboratories. Asked how much Synthetic Softworks has collected before fees, GPT-5.5 passed by answering the gross figure, 7,733,880, and adding that the total net of refunds would be 7,154,190. Gemini 4 Argon computed both numbers and gave the net one as its answer, which failed.

Both readings are defensible, and only the task can say which one finance wants.

A Correct "Two" Failed the Grader

Exact-value grading has a gap of its own. Asked how many of Northgate's invoices are not fully paid, Gemini 3.1 Pro Preview answered "Two of Northgate Foundry Corporation's invoices are not fully paid." The known answer is 2, the grader looked for the numeral, and the run failed.

Separately, the environment's own summary flags grader problems. It sorts recent failures by cause, and reports that all 3 it sorted point at grader mismatches, where the grader's check disagrees with an answer the data supports. Until those graders are reviewed, some failures in the table above may be scoring errors.

The Rest Were Counting Errors

After the refund split and the grader gap, what is left looks like ordinary mistakes:

• Miscounts of a complete list. On the Northgate invoice count, a Claude Haiku 4.5 run received all 12 invoices in one list_invoices response and answered 11, and a GPT-5.4 Mini run answered 10.
• Sums that drift. A Claude Haiku 4.5 run added up Northgate's charges to 1,347,728, 4 cents short of the key. A GPT-5.4 Mini run put Northgate's total tax at 80,944 cents against a known 137,000.
• The account mistaken for a customer. Synthetic Softworks is the Stripe account itself. Asked how much it had collected, Claude Haiku 5.5 and Claude Sonnet 5 searched the customer list for a customer by that name, found none, and said they could not answer.

Price Didn't Predict Reliability

The environment's reliability table counts a question as passed only when a model passed every graded attempt on it. Cost per pass is what a model spent divided by the questions it passed:

Claude Sonnet 5.5 passed 217 of 232 across more runs than the others. Denominators differ because models ran different subsets of the question set. The cheapest model per pass in the table, Claude Haiku 5.5 at about a tenth of a cent, lost 6 of its 9 misses on the "paid in total" questions, and the most expensive per pass, Claude Fable 5.1, lost 5 of its 6 misses on those questions.

Apollo Search Passed, Writes Split

The Apollo environment is a generated CRM built from 200 prospects at 40 fintech companies, with 34 tools including search_contacts, create_contact and list_contact_stages. We ran 9 models on its 4 tasks, 3 attempts each from a fresh copy of the CRM, with all 34 tool definitions sent on every call and a $1 spending cap per model:

• 2 search tasks: list the Apollo IDs of VPs of Sales at Austin fintech companies with 50 to 200 employees, and of Boston biotechnology companies with 200 to 1,000 employees that have raised $20M to $60M.
• 1 paraphrase of the first search, generated by the platform, with the same known answer, to check that wording does not change the result.
• 1 write task: add 12 prospects to the CRM at the Cold Outreach stage, where 3 of the 12 already exist. A state check then searches the CRM for each email and expects exactly 1 contact per person, then checks the total contact count.

78 of 107 attempts passed and 8 failed. The other 21 never reached a grade, which we cover below. The search tasks and the paraphrase passed on every graded attempt except 1 from GPT-5.4 Mini, and the other 7 failures all came from the write task.

Asking First Counted as Failing

The Apollo CRM has a stage called "Cold" and no stage called "Cold Outreach". The task asks for the stage that does not exist.

GPT-6 Astra stopped on all 3 attempts and asked: "The CRM has a 'Cold' stage, but no 'Cold Outreach' stage. Should I add all 12 prospects to 'Cold,' updating any existing contacts rather than creating duplicates?" On its last attempt it made a single tool call and changed no data. GPT-6.1 Sol completed the job on its first attempt and asked the same question on its other 2. All 5 attempts that asked failed the state check.

GPT-5.5 and Claude Haiku 5.5 passed all 3 write attempts each. Claude Haiku 5.5 did it for $0.04 across all 12 of its attempts.

Whether asking was the right call depends on the workflow: an overnight batch import has no one to answer, so a question leaves the work undone, while a careful person writing to a CRM the sales team relies on would probably check first. This grader rewards proceeding, so a team reading these results needs to decide which behavior it wants before it reads a pass rate.

A Pass With a False Summary

GPT-5.4 Mini failed its first 2 write attempts. Both reported adding all 12 prospects at the Cold stage, and both failed the CRM check. Its third attempt passed, with the message: "Three already existed and were moved/updated there; the other nine were created in that stage."

The platform's record of what that attempt changed lists 9 added contacts and no updates. The CRM ended up correct and the message described a move that never happened. A team that audited this agent by reading its messages would believe 3 existing prospects had been moved out of their current stage. In an earlier run of the same task with a different tool setup, Claude Haiku 4.5 created all 12 contacts without searching first, left 3 duplicates and reported "I've successfully added all 12."

21 Attempts Never Got a Grade

We capped each model at $1 to keep the batch small, and the cap shaped the results. The platform reports 20 attempts across the environment's runs that stopped because the budget ran out, and a smaller number where the model provider rejected the request before an agent answered. In this batch:

• Claude Sonnet 5.5 and Claude Opus 5.5 passed every search attempt that reached a grade, and neither finished a write attempt: their last write attempts stopped early after adding 7 contacts and 1 contact. Sending all 34 tool definitions with every call made each step expensive: one Claude Sonnet 5.5 write attempt used 86,696 input tokens over 10 tool calls before it stopped.
• Gemini 3.8 Flash, which passed every Stripe question it ran, spent $1.05 here and produced 1 graded answer in 11 attempts.

These attempts produced no answer to grade, so they count as neither passed nor failed, and a team that caps agent spending per task should expect the same thing on write-heavy jobs.

Write the Definition Into the Task

For a finance team putting an agent on Stripe, the refund split matters more than the choice of model. State in the task whether "paid", "collected" and "revenue" include refunds, failed charges and fees, and keep a few questions with known answers in each definition so a change in model or prompt shows up as a changed number.

For a sales ops team putting an agent on Apollo, decide what should happen when a request names a stage, owner or field that does not exist. If your agent should ask, grade asking as a pass. If it should proceed with the closest match, say so in the instructions and check the CRM afterward for duplicates and unexpected stage changes. Either way, grade what changed in the CRM, because 2 of the runs above described changes that did not match what they did.

For anyone building graders, accept numbers written as words, or tell your agent to answer with numerals, and review the environment's own flagged mismatches before a failure counts against a model.

Limits of This Review

• Fictional, generated data. Both environments are built from synthetic records. Real billing accounts carry partial refunds, credits and disputes, which could widen the gross-versus-net gap.
• Stripe attempts were not run as a single batch. The 1,537 Stripe attempts accumulated over several runs, and models ran different subsets of questions, which is why the reliability denominators differ.
• Exact value reads the final answer. A Stripe answer with the right number and wrong reasoning would pass, and a right answer in the wrong format fails.
• The Apollo write task is 1 task. 3 attempts per model shows a consistent behavior on that task, such as GPT-6 Astra asking every time, but it is not a pass rate for CRM writes in general.
• The Apollo cap and tool setup affected results. With tools looked up as needed instead of sent all at once, earlier runs of the Claude models hit provider rejections on the search tasks, and with all tools up front, our $1 cap stopped the most expensive runs. A higher cap would likely turn most of the 21 ungraded attempts into grades.

Next, we are adding answer keys for both readings of the "paid in total" questions so each run can be scored against the definition it chose, adding a version of the Apollo write task that names the "Cold" stage correctly, and re-running the models our cap stopped with a higher budget.

How We Ran It

Both environments are LayerLens Environments: generated replicas of an app's data behind its own tool interface, built so every question has a known answer or a checkable end state. Exact value grading looks for the known answer in an agent's final message. Environment state grading queries the app after the run. Every attempt is recorded as a trace with each tool call, its arguments and result, token counts, cost, and a record of what the attempt changed.

LayerLens runs this kind of review on your own agents and tasks, graded against answers you already know: layerlens.ai