Teams that connect an AI agent to HubSpot (a CRM, the system sales and support teams use to track deals, contacts and tickets) want fast answers to questions like "how many deals are open?" or "how many tickets are still waiting?". Counts like these are among the first questions a team asks a HubSpot agent. Each answer depends on a condition inside the records: which pipeline stages (the steps a deal or ticket moves through) count as closed, or which stage ends a ticket. An agent can return the right number by repeating the total a search reports, without reading the records behind it, and a test that grades only the final number cannot tell whether an agent checked.
Whether a final-number check is enough depends on the question and on what you grade. Questions that a reported total cannot answer, such as the dollar value of the open pipeline, push the evidence into the answer itself. A rubric (a written set of pass conditions) applied to the saved trace, the log of every tool call an agent made and every response it got back, checks the route behind any answer.
The Earlier Test Passed an Agent That Never Checked Its Count
In the earlier test, 7 agents, each on a different foundation model, answered the open-deal question and all 7 passed. We followed 2 of them. Run A, on Claude Sonnet 5.5, took the total from a filtered search and then listed all 10 deals to confirm 8 were open. Run B, on Claude Sonnet 5, took the reported total of 8 from a search that returned 1 row, read that 1 deal and never followed the cursor (the token an API returns for fetching the next page of results). Both traces, with their grade records and a script that recounts the records each agent read, are in the public test pack on Hugging Face.
This follow-up adds 5 new runs to A and B: a rerun of the open-deal question on run A's foundation model, the same kind of question about support tickets, and 3 harder questions, each with its expected answer written from the records before the run.
Northwind Systems Has 8 Open Deals of 10 and 4 Open Tickets of 5
Every run took place in a LayerLens Environment, a simulated version of a service that saves every tool call and response as a trace. This one, hubspot-b2b-pipeline v2 (ID 6ab3e8b5142fb38e8b6b71de), is a fictional sales team's HubSpot account, a portal in HubSpot's terms, named Northwind Systems. It copies the format of HubSpot's API responses, nothing more, and gives agents 20 tools modeled on HubSpot's API, including tools that can change records.
A deal is a potential sale. The account has 1 deal pipeline with 7 stages, and 2 of them, Closed Won and Closed Lost, are flagged closed. Of the 10 deals, 8 are open and worth $1,268,400 together; 1 is Closed Won ($290,900) and 1 is Closed Lost ($59,700). Vantage Materials, Bluepeak Harbor and Northgate Cascade each hold 2 open deals with the same deal name, so the 8 open deals belong to 5 companies.
Tickets sit in a separate support pipeline where the Closed stage ends a ticket. The 5 tickets are spread across New (2), Waiting on contact (1), Waiting on us (1) and Closed (1), so 4 are open.
The 5 new runs used Claude Sonnet 5.5 with no custom prompt, a $0.50 spending cap per run, and the platform's default instruction to end with a line reading ANSWER: <answer>. Tools loaded as needed through a tool_search call instead of all 20 being sent up front. Each run was graded by an exact-value check of that final line against the known answer. We read the route, meaning the tool calls, the records read and whether a cursor was followed, from each trace by hand.
A Rerun Took a New Route to 8, and the Ticket Count Passed at 4
Run C used run A's foundation model, with tools loaded as needed instead of all at once, and reached 8 a different way. It read the pipeline, searched for deals outside the 2 closed stages (total 8), searched for deals inside them (total 2), then listed all 10 deals:
"There are 8 open deals in the portal. I counted them with a search that excludes the two closed stages, Closed Won and Closed Lost. [...] The search found 2 closed deals: one Closed Won and one Closed Lost. A listing of the portal's deals showed 10 in total, which matches 8 open plus 2 closed."
Run D asked how many tickets are not Closed. The agent read the support pipeline, searched for tickets outside the Closed stage and named the 4 it found before answering:
"Four of the five tickets in the portal are not Closed. They sit in the Support Pipeline, the only ticket pipeline. I searched for every ticket whose stage isn't "Closed" and got these four [...] The fifth ticket is the one in the Closed stage. ANSWER: 4"
On 3 Harder Questions, Every Answer Listed the Records Behind It
Picture a sales lead asking an assistant for the open pipeline before a forecast call. "How many deals are open?" can be answered from a search total. "What are the open deals worth?" cannot, because an agent has to read the amount on every open deal. "How many companies have an open deal?" works the same way: a deal total says nothing about which companies the deals belong to.
We ran 3 such questions on Claude Sonnet 5.5 for $0.1475 in total. Run E2 asked for the open pipeline in dollars and listed all 8 open deals with stage and amount to reach $1,268,400. Run E3 asked how many open deals close before 2025-11-01; a search filtered on stage and date could return that count without opening a deal, yet the agent named the 4 deals and their close dates. Run E4 asked how many companies have at least 1 open deal, where counting open deals gives 8, and the agent listed the 5 companies by record ID with each one's open-deal count.
| Run | Question | Expected | Model | HubSpot calls (excl. tool_search) | What the answer rested on | Time | Cost | Run ID |
|---|---|---|---|---|---|---|---|---|
| A | Open deals (not in a closed stage) | 8 | Claude Sonnet 5.5 | 3: list_pipelines, search_objects, search_objects | Filtered search total, then all 10 deals listed | n/a | $0.078 | 6acee5b61bf2d58efc7566e4 |
| B | Open deals (not in a closed stage) | 8 | Claude Sonnet 5 | 5: search_objects, list_pipelines, search_objects, list_owners, search_objects | Reported total 8 from a 1-row page; 1 deal read, cursor not followed | n/a | $0.117 | 6acedf574ac7e762469ea6e7 |
| C (rerun) | Open deals (not in a closed stage) | 8 | Claude Sonnet 5.5 | 4: list_pipelines, search_objects (open, total 8), search_objects (closed, total 2), list_objects (10 deals) | Open total + closed total + full list, 8+2=10 | 8.8s | $0.040 | 6adb9739b7b3674113b2932d |
| D | Tickets not Closed | 4 | Claude Sonnet 5.5 | 3: list_pipelines, search_objects, search_objects | The 4 open tickets named by company | 6.6s | $0.030 | 6adb975f16f91d6a701f8337 |
| E2 | Total amount of open deals, USD | 1,268,400 | Claude Sonnet 5.5 | 4: list_pipelines, list_properties, search_objects, search_objects | All 8 open deals listed with stage and amount | 11.3s | $0.069 | 6adba586d7a7af9c15b99aa4 |
| E3 | Open deals with close date before 2025-11-01 | 4 | Claude Sonnet 5.5 | 2: list_pipelines, search_objects | The 4 deals named with close dates | 6.8s | $0.029 | 6adba5a61a68e4c091e3605a |
| E4 | Companies with at least 1 open deal | 5 | Claude Sonnet 5.5 | 3: list_pipelines, list_objects, search_objects | 5 companies listed by record ID with open-deal counts | 14.4s | $0.049 | 6adba5c01a68e4c091e3605b |
The Grade Scored a Checked Count and an Unchecked Count the Same
All 7 runs examined here passed, and every one used only read calls even though tools that change records were available. On the open-deal question, the exact-value check scored run B's single 1-row search the same as run C's 3-way cross-check. The harder questions produced answers a reviewer can check line by line, but the grade did not read those lines either: it compared the final number.
How to test depends on how a team uses the agent:
- A team whose reps ask bare counts in chat should not rely on the exact-value check alone, since a reported search total can match the known count without an agent reading a record or following a cursor. Grade the trace for the reads behind the number, or require the answer to list the records it counted.
- A team pulling totals for forecasts or pipeline reports should include questions a reported total cannot answer, such as E2's sum and E4's company count. A date-filtered count like E3 can still come from a search total, so its trace needs the same check as a bare count.
- A team whose agent holds write tools should confirm in the trace that a question asking only for information produced only reads, as all 7 runs here did.
This Account Cannot Test Owners, Pipelines or Pagination Past 200
- Deals here have no owner, there is 1 deal pipeline, and all 10 deals fit on 1 page of HubSpot's search, which returns up to 200 results per page. A larger account with owners, several pipelines and more than 200 deals would test whether agents scope by owner or pipeline and follow cursors across pages.
- Runs A and B received all 20 tools up front, while C to E4 loaded tools as needed, so time and cost are compared only among C to E4. Running every question on one setup would make all 7 comparable, and would show whether run C's new route came from the setup.
- Each row is 1 run of 1 configuration. A pass shows the answer matched once; it is not a success rate. More runs per question would show how often a route like run B's appears.
- Runs C to E4 ran in the LayerLens development environment, and their traces are visible to members only. Runs A and B are public in the test pack.
The Next Test Grades the Route From the Saved Traces
LayerLens graders can evaluate saved traces against a rubric. A rubric such as "pass only if the count matches deal records the agent read" has not been written for this task yet. Because every run saved its tool calls, the next step is to write it, apply it to these 7 traces without running the agents again, and report which runs it passes and which it flags.
LayerLens Environments simulate HubSpot, Salesforce, Gmail and other services and save every tool call, so teams can test agents against known answers before go-live and after every change. See how Environments work.