Gemini 3.5 Flash: Stratix Evaluation Data Reveals Where Google's Fastest Model Actually Wins

By The LayerLens Team

TL;DR

• Gemini 3.5 Flash scored 93.89% on LiveCodeBench and 63.75% on Terminal-Bench, nearly doubling the next flash-tier competitor (GLM 4.7 Flash at 33.75%) on real command-line task completion.

• On Humanity's Last Exam, it scored 37.30%, more than tripling GPT-5 Mini (12.39%) and scoring 8x higher than Claude Haiku 4.5 (4.71%). A flash-tier model within 3.29 points of a Pro-tier model on the hardest public benchmark.

• Gemini 3.5 Flash matches or beats Gemini 2.5 Pro on 6 out of 6 shared benchmarks at 25% lower cost per token. The margins against Gemini 3.1 Pro Preview are razor thin: 0.20 points on MATH-500, 1.31 on MMLU Pro, 1.02 on AGIEval English.

• At $9.00 per million output tokens, it sits at the top of flash-tier accuracy but also at the higher end of flash-tier pricing. Teams running high-volume, lower-complexity workloads may find GPT-5 Mini at $2.00 more efficient.

• Benchmark scores test outputs under controlled conditions, not agent trajectories in production. Terminal-Bench is the closest proxy for agentic reliability, and Gemini 3.5 Flash leads the flash tier there by a wide margin.

Two days ago, Cursor's coding agent deleted a production database in 9 seconds. It passed every benchmark its developers ran. The failure was in the trajectory, not the accuracy score.

Google launched Gemini 3.5 Flash at Google IO 2026 with a pitch built around exactly this problem: a model fast enough for agentic workflows and accurate enough to handle multi-step execution. LayerLens ran it through Stratix across 12 benchmarks, head-to-head against every flash and mini model in production, to find out where the numbers actually hold.

The short version: Gemini 3.5 Flash scored 93.89% on LiveCodeBench, 63.75% on Terminal-Bench (nearly doubling the next flash-tier competitor), and cleared the academic baselines (96.80% MATH-500, 89.58% MMLU Pro) that qualify it for frontier-tier reasoning. It beat GPT-5 Mini, Claude Haiku 4.5, and GPT-5 (minimal) on nearly every benchmark Stratix evaluated. It also matched or beat previous-generation Pro models at a fraction of the cost.

But a benchmark score tells teams what a model can do under controlled conditions. It does not tell them what happens when that model runs autonomously in their pipeline. Here is what Stratix found, and what the numbers do and do not guarantee.What Gemini 3.5 Flash actually is

Gemini 3.5 Flash uses an ultra-sparse mixture-of-experts (MoE) architecture with 1.2 trillion parameters. The context window sits at 1,048,576 input tokens with 65,536 output tokens. It accepts text, image, audio, and video inputs. Google says output runs 4x faster than other frontier models measured in tokens-per-second.

Pricing lands at $1.50 per million input tokens and $9.00 per million output tokens, with $0.15 for cached input. That positions it between the budget flash tier and the premium Pro tier, a deliberate pricing strategy from Google.

The model now serves as the default engine behind the Gemini app and AI Mode in Google Search globally.

Benchmark-by-benchmark: Stratix evaluation results

LayerLens evaluated Gemini 3.5 Flash across 12 benchmarks on Stratix. The results below start with the benchmarks that matter most for production deployment (coding, agentic tasks, frontier difficulty), then

cover the academic baselines that establish the model's reasoning floor.

Coding: LiveCodeBench: 93.89%

LiveCodeBench is the coding benchmark that maps closest to real development work. The 93.89% score is strong, though SWE-bench Lite returned 0.00%, indicating the mini-swe-agent harness did not complete successfully rather than a model failure. Google's own reporting cites Terminal-Bench 2.1 at 76.2% and MCP Atlas at 83.6% for agentic coding tasks.

Agentic and terminal tasks: Terminal-Bench (Terminus-2): 63.75% (vs. GLM 4.7 Flash at 33.75%)

Terminal-Bench measures real command-line task completion. Gemini 3.5 Flash nearly doubled GLM 4.7 Flash's score. Google separately reports 76.2% on Terminal-Bench 2.1 (a newer version with harder tasks), making this the model's strongest competitive differentiator for teams building agentic workflows.

For teams deploying coding agents, Terminal-Bench is more informative than any math benchmark. It tests what agents actually do: traverse file systems, chain shell commands, recover from errors. The gap between 63.75% and 33.75% is the difference between an agent that finishes tasks and one that stalls.

Frontier difficulty: Humanity's Last Exam: 37.30% (vs. GPT-5 minimal at 17.17%, GPT-5 Mini at 12.39%, Claude Haiku 4.5 at 4.71%)

This is the benchmark designed to stump frontier models. Gemini 3.5 Flash more than tripled GPT-5 Mini's score and scored 8x higher than Claude Haiku 4.5. For context, Gemini 3.1 Pro Preview scored 40.59% on the same benchmark. A flash-tier model coming within 3.29 points of a Pro-tier model on the hardest publicly available benchmark is the kind of compression that redefines tier boundaries.

Multimodal Understanding: 83.00% (vs. GPT-5 minimal at 76.79%, GPT-5 Mini at 75.33%, Claude Haiku 4.5 at 65.22%)

Gemini 3.5 Flash leads the flash tier by 6.21 points on multimodal tasks. Google's multimodal heritage shows here: image, audio, and video understanding have been core to the Gemini architecture since 1.0.Academic baselines: the reasoning floor

These benchmarks are widely saturated, with most frontier models scoring 85%+. They establish Gemini 3.5 Flash's reasoning baseline, not a deployment guarantee. The harder question is what happens beyond controlled conditions.

MATH-500: 96.80% (vs. GPT-5 Mini at 95.65%, GLM 4.7 Flash at 95.20%, GPT-5 minimal at 96.60%)

MMLU Pro: 89.58% (vs. GPT-5 Mini at 81.42%, Claude Haiku 4.5 at 77.49%, GPT-5 minimal at 67.30%)

AGIEval English: 92.97% (vs. GPT-5 minimal at 91.40%, Claude Haiku 4.5 at 88.55%, GPT-5 Mini at 83.50%)

Big Bench Hard: 91.74% (vs. GPT-5 minimal at 89.38%, GPT-5 Mini at 87.74%, Claude Haiku 4.5 at 81.98%)

General Purpose QA: 92.93% (vs. DeepSeek V4 Flash at 86.36%, GPT-5 Mini at 80.30%, GPT-5 minimal at 71.94%)

AIME 2025: 96.67% (vs. GPT-5 Mini at 80.00%, GPT-5 minimal at 90.00%, Claude Haiku 4.5 at 26.67%)

AIME 2024: 96.67% | AIME 2026: 93.33%

The MMLU Pro gap is 8.16 points over GPT-5 Mini and 12.09 points over Claude Haiku 4.5. On AGIEval English, the margin narrows to 1.57 points over GPT-5 (minimal), confirming both models have reached similar ceilings on structured reasoning. The AIME 2025 result stands out: 16.67 points over GPT-5 Mini, 70 points over Claude Haiku 4.5. These numbers qualify Gemini 3.5 Flash for frontier-tier reasoning. The real test is whether that reasoning holds across multi-step agent workflows.

The flash tier is no longer a single price band. Here is what each model costs per million output tokens, alongside average accuracy across shared Stratix benchmarks:

DeepSeek V4 Flash: $0.28 output, strong on General Purpose QA (86.36%)

GPT-5 Mini: $2.00 output, solid all-rounder (73.45% avg across 6 benchmarks)

Gemini 2.5 Flash: $2.50 output, previous generation (62.81% avg)

Claude Haiku 4.5: $5.00 output, inconsistent (55.88% avg, dragged down by weak AIME and HLE scores)

Gemini 3.5 Flash: $9.00 output, highest accuracy (84.18% avg across 6 benchmarks)

GPT-5 (minimal): $10.00 output, frontier-adjacent (75.31% avg)

Gemini 3.5 Flash sits at the top of the accuracy axis but also at the higher end of the cost axis. The value proposition is clearest for teams that need frontier-level reasoning and coding from a fast model. Teams running high-volume, lower-complexity workloads may find GPT-5 Mini's $2.00 output pricing more efficient.

Flash vs. Pro: the tier boundary is dissolving

The most significant finding from the Stratix data is how close Gemini 3.5 Flash comes to Pro-tier models.

Against Gemini 2.5 Pro ($1.25 input / $10.00 output):

Gemini 3.5 Flash wins on AIME 2025 (96.67% vs. not evaluated), MATH-500 (96.80% vs. 73.12%), MMLU Pro (89.58% vs. 85.36%), Big Bench Hard (91.74% vs. 90.09%), AGIEval English (92.97% vs. 90.20%), and Humanity's Last Exam (37.30% vs. 16.06%).

Against Gemini 3.1 Pro Preview ($2.00 input / $12.00 output):

Gemini 3.5 Flash wins on AIME 2025 (96.67% vs. 93.33%). Gemini 3.1 Pro Preview wins on MATH-500 (97.00% vs. 96.80%), MMLU Pro (90.89% vs. 89.58%), Big Bench Hard (96.07% vs. 91.74%), AGIEval English (93.99% vs. 92.97%), HLE (40.59% vs. 37.30%), and LiveCodeBench (97.23% vs. 93.89%).

The margins between Gemini 3.5 Flash and Gemini 3.1 Pro Preview are razor thin on most benchmarks: 0.20 points on MATH-500, 1.31 on MMLU Pro, 1.02 on AGIEval English. The Pro model still wins on Big Bench Hard by 4.33 points and LiveCodeBench by 3.34 points, which makes a real difference for teams with hard reasoning and coding requirements.

But the cost story matters. Gemini 3.5 Flash costs 25% less per input token and 25% less per output token than Gemini 3.1 Pro Preview. For workloads where 1-2 points of accuracy are not mission-critical, Flash delivers 95% of the performance at 75% of the cost.

The flash tier has fractured into three distinct pricing and performance bands:

Budget flash ($0.14 to $2.50 output): DeepSeek V4 Flash, GPT-5 Mini, Gemini 2.5 Flash, GLM 4.7 Flash. These models trade accuracy for cost efficiency. GPT-5 Mini is the best all-rounder here at $2.00 output.

Premium flash ($4.50 to $9.00 output): GPT-5.4 Mini, Claude Haiku 4.5, Gemini 3.5 Flash. These models approach Pro-tier accuracy at flash-tier speed. Gemini 3.5 Flash leads this segment on every benchmark Stratix evaluated.

Frontier-adjacent ($10.00+ output): GPT-5 (minimal). Positioned as a distilled frontier model, it trades with Gemini 3.5 Flash on specific benchmarks but costs more.

The old assumption that flash models sacrifice too much accuracy to be useful for reasoning-heavy workloads no longer holds. Gemini 3.5 Flash scored within 3-4 points of dedicated Pro models on 5 out of 6 benchmarks. The tier boundary is dissolving, and teams choosing between Flash and Pro need to evaluate on their specific workloads, not on tier labels.

What this means for teams evaluating models

Gemini 3.5 Flash clears 93.89% on LiveCodeBench and nearly doubles the flash tier on Terminal-Bench. Those are strong starting points. They are not deployment guarantees.

The PocketOS incident from the opening is the pattern that keeps repeating: agents that pass every benchmark still fail in production because benchmarks test outputs, not trajectories. The tool call sequence, the privilege escalation, the retry loop that runs up a $437 bill overnight. No static benchmark catches that. Continuous evaluation across the full agent trace does.

Gemini 3.5 Flash's Stratix numbers give teams a strong baseline for model selection. The next step is running it through evaluation infrastructure built for what benchmarks miss: tool call sequences, cost guardrails, prompt mutation detection, and the compound failures that multiply across multi-step agent workflows.

Compare Gemini 3.5 Flash against GPT-5 Mini, Claude Haiku 4.5, or any model on Stratix. The full prompt-level evaluation data, including every prompt, every output, and every score, is available at stratix.layerlens.ai.

Evaluation as infrastructure, not ceremony. That is what separates a benchmark score from a deployment guarantee.Key Takeaways

• For teams building coding agents, Gemini 3.5 Flash is the strongest flash-tier option on both LiveCodeBench (93.89%) and Terminal-Bench (63.75%). The Terminal-Bench gap over the next competitor is the largest performance delta Stratix measured across all benchmarks.

• The flash vs. Pro tier boundary is dissolving. Gemini 3.5 Flash beats Gemini 2.5 Pro on every shared benchmark and comes within 1 to 4 points of Gemini 3.1 Pro Preview on most. Teams paying Pro-tier prices should re-evaluate whether the marginal accuracy gain justifies the cost.

• Cost efficiency depends on workload complexity. At $9.00 per million output tokens, Gemini 3.5 Flash costs 4.5x more than GPT-5 Mini. For high-volume classification or extraction tasks, the cheaper model may deliver equivalent results. For multi-step reasoning or agentic workflows, the accuracy gap matters.

• Benchmark scores are starting points, not deployment guarantees. The PocketOS incident shows that even models passing every benchmark can fail catastrophically in production. Evaluate on your actual agent traces, tool call sequences, and error recovery patterns before shipping.

• Run continuous evaluation across the full agent trajectory, not just final outputs. Static benchmarks miss the failure modes that compound across multi-step workflows: privilege escalation, retry loops, cost runaway, and tool call hallucination.

Frequently Asked Questions

How does Gemini 3.5 Flash compare to GPT-5 Mini for coding tasks?

Gemini 3.5 Flash scored 93.89% on LiveCodeBench vs. GPT-5 Mini's score on the same benchmark. On Terminal-Bench, which tests real command-line task completion (file traversal, shell command chaining, error recovery), Gemini 3.5 Flash scored 63.75%. GPT-5 Mini was not evaluated on Terminal-Bench, but the closest flash-tier competitor (GLM 4.7 Flash) scored 33.75%. For teams deploying coding agents, the Terminal-Bench result is more informative than any math benchmark.

Is Gemini 3.5 Flash worth the price premium over cheaper flash models?

At $9.00 per million output tokens, Gemini 3.5 Flash costs 4.5x more than GPT-5 Mini ($2.00) and 32x more than DeepSeek V4 Flash ($0.28). The accuracy gap is real: 84.18% average across six shared benchmarks vs. 73.45% for GPT-5 Mini. Whether that gap justifies the cost depends on workload complexity. Multi-step reasoning, agentic workflows, and frontier-difficulty tasks favor Flash. High-volume, lower-complexity batch processing may not need the premium.

Can Gemini 3.5 Flash replace Pro-tier models?

On Stratix benchmarks, Gemini 3.5 Flash matches or beats Gemini 2.5 Pro on every shared benchmark at 25% lower cost. Against the newer Gemini 3.1 Pro Preview, the margins are razor thin on most benchmarks (0.20 to 1.31 points) but the Pro model still wins by 4.33 points on Big Bench Hard and 3.34 points on LiveCodeBench. For workloads where 1 to 4 points of accuracy are not mission-critical, Flash delivers roughly 95% of the performance at 75% of the cost.

What benchmarks did LayerLens use to evaluate Gemini 3.5 Flash?

LayerLens evaluated Gemini 3.5 Flash across 12 benchmarks on Stratix: LiveCodeBench, Terminal-Bench (Terminus-2), Humanity's Last Exam, Multimodal Understanding, MATH-500, MMLU Pro, AGIEval English, Big Bench Hard, General Purpose QA, AIME 2024, AIME 2025, and AIME 2026. All evaluations used standardized prompts with no system-level modifications. The full prompt-level data, including every prompt, every output, and every score, is available on Stratix.

What does the SWE-bench Lite 0.00% score mean?

The 0.00% score on SWE-bench Lite indicates that the mini-swe-agent evaluation harness did not complete successfully, not that the model cannot handle software engineering tasks. This is a harness compatibility issue, not a model capability failure. Google's own reporting cites 76.2% on Terminal-Bench 2.1 and 83.6% on MCP Atlas for agentic coding tasks.

Methodology

All evaluation data in this article was generated on Stratix by LayerLens. Models were evaluated using standardized prompts with no system-level modifications. Benchmark versions: AIME 2024, AIME 2025, AIME 2026, MATH-500, MMLU Pro, Big Bench Hard, AGIEval English, General Purpose QA, Multimodal Understanding, Humanity's Last Exam, LiveCodeBench, Terminal-Bench (Terminus-2). Accuracy represents the percentage of correct responses across all benchmark prompts. All evaluations used the latest available model versions as of May 2026.