An agent was asked what a supplier invoice should read after the supplier corrected it by email. The thread held 2 amounts: 10,475 USD in the original invoice and 10,625 USD in a correction 2 messages later. The agent ran 1 search, read the thread once and answered 10,625 USD, naming the original amount and the reason for the change. The answer matched the task's known answer and passed.
The run took place in a simulated Gmail inbox built with LayerLens Environments and filled with fictional suppliers, where every task has an answer recorded before any agent runs. The sections below cover what the pass shows, what it leaves open and 2 follow-up tests that would make this check worth relying on.
Where this lookup sits in accounts payable
Accounts payable (AP) is the team that pays a company's suppliers. Before a payment goes out, someone confirms the invoice amount and checks it against the purchase order and what was received.
Invoices change after they arrive. When a supplier leaves off a freight charge or bills the wrong quantity, the fix often arrives as a reply in the same thread as the original, which leaves 2 amounts for 1 invoice in the mail an agent reads.
An agent asked for the amount can return a number that came straight from the supplier and is out of date. Here the original figure is 150 USD lower than the corrected one, so paying it would leave the supplier with an open balance and both companies reconciling it. When a correction lowers the amount, the same mistake becomes an overpayment to recover. No payment was involved in this test; these are the costs of returning the outdated amount.
The test
The environment simulates one buyer's mailbox at a fictional company: 8 threads and 22 messages, 4 of the threads about invoices from 2 suppliers. An agent working in it can call 6 tools, all read operations: searching and opening messages and threads, listing labels and reading the mailbox profile. None of them sends mail, edits an invoice or moves money, and no accounting system is connected.
The agent's job was to answer 1 question in text. It ran on Claude Sonnet 5 with a generic instruction to work against a Gmail API, use its tools and reply with a final answer. The instruction says nothing about invoices or corrections.
The task, as written in the environment:
Sundial Foundry corrected invoice INV-2025-0003 partway through the thread. What amount should it read, in USD?
The known answer is 10,625. The thread behind it has 5 messages:
- The supplier sends invoice INV-2025-0003 for 10,475 USD, for a sensor order under purchase order PO-4027.
- The buyer replies that the invoice has gone to approvals.
- The supplier corrects it to 10,625 USD because a freight line was left off. The due date stays the same.
- The buyer replies that the invoice was updated to 10,625 USD and payment is scheduled.
- The supplier sends a payment reference.
All 5 messages were in the fictional mailbox before the run, and the agent sent nothing. It approved nothing, changed no record and started no payment.
What happened
The saved trace records 2 tool calls. The agent searched the mailbox for "INV-2025-0003" and got back 5 message IDs, all in 1 thread. It then opened that thread, and the response contained all 5 messages in full.
Its final answer:
The corrected amount for INV-2025-0003 is 10,625 USD (up from the original 10,475 USD, due to a freight line that had been omitted). The due date remained 2025-08-26.
Every figure in that answer appears in the thread the agent read, and the freight explanation appears only in the supplier's correction. The run used 7,940 input tokens and 176 output tokens and cost under 2 cents.
The grader was an exact-value check against the known answer. Its recorded explanation reads: looked for "10625" in the agent's answer. The verdict was pass.
What the result establishes
The trace shows that the agent found the right thread with 1 search on the invoice number, retrieved every message in it before answering, and returned the corrected amount in the currency asked for, with the original amount and the supplier's reason for the change. Several things remain untested.
Which message the answer came from. The corrected amount also appears in message 4, the buyer's own confirmation, which is the last message in the thread to carry an amount. A correct answer does not by itself show that the agent relied on the supplier's correction, and the trace records no reasoning between tool calls. The freight explanation in the answer points to message 3, which is the strongest evidence available here.
How often this works. This is 1 saved attempt. The same task has also run on 6 other foundation models, once each, and all 7 runs matched the known answer. A task all 7 runs pass cannot separate one agent from another.
Whether an agent finds the correction unprompted. The question itself says the invoice was corrected partway through the thread. A payment request names the invoice; it does not announce that the invoice was corrected.
Which amount the answer commits to. The check looked for 10625 in the answer text, and this answer also contains 10,475. The recorded explanation does not show whether the check separates the amount an answer presents as current from one it mentions in passing.
Agreement with an accounting record. The environment holds no invoice records outside the mailbox, so the known answer is the value written down when the task was created. Nothing was compared against a ledger.
What to test next
The next 2 tests are proposals and have not been run.
Remove the hint and the confirmation. Take the buyer's confirmation out of the thread and ask what amount is payable on INV-2025-0003, with no mention of a correction. The supplier's correction is then the only place 10,625 USD appears, so a correct answer has to come from reading it.
Grade the amount the answer commits to, and its source. Add a second correction later in the thread, so 3 amounts appear and only the last is current. Then grade the single payable amount the answer states and the message it cites for it. Each attempt is already saved as a trace, and LayerLens can run a rubric-based judge over a trace, which is one way to score the citation as well as the number.
Why controlled records help
Against a live inbox, a team would first have to work out what each invoice should read before judging an agent. Here the mailbox, the correction and the known answer are fixed, so a team can add the harder thread and rerun every candidate agent against identical records, comparing them trace by trace before any of them touches supplier mail.
Method and limits
This is a development-environment example with fictional companies, people and invoices; no customer data or deployment is involved. One saved attempt was inspected end to end: task, run, trace, both tool calls with their full responses, final answer and grade. Results for the other 6 runs come from the task summary and were not inspected trace by trace. No new runs, regrades or task changes were made for this post.
LayerLens Environments simulate services such as Gmail, Stripe and Salesforce, so teams can test agents on records with known answers before giving them live access, and again whenever the agent changes. See how Environments work.