Unit Test Pass Rate Is Not a Code Quality Signal

TL;DR:

• SWE-Bench measures whether AI-generated code passes unit tests. Not whether it is good code. These are different things.

• A 2025 analysis of SWE-Bench+ found 47.93% of "resolved" issues were incorrectly marked. The patches passed weak test suites but did not correctly solve the underlying problem.

• A new benchmark, Needle in the Repo (2025), addresses this gap directly: it evaluates maintainability in AI-generated repository edits, not just correctness.

• Code quality decomposes into at least four dimensions: correctness (testable), maintainability (not tested by most benchmarks), idiom adherence (not tested), and security posture (tested inconsistently).

• Deterministic code graders handle correctness checks. Custom judges handle the semantic quality dimensions. Both can run on the same evaluation trace.

SWE-Bench Lite presents real GitHub issues. The model generates a patch. If the patch passes the associated tests, it counts as resolved.

The benchmark measures exactly what it was designed to measure: whether the generated code passes the tests that existed at the time the issue was filed.

The problem is that teams use SWE-Bench scores as a proxy for production code quality. Those two things are not the same measurement. The gap between them has a name now.

What the Research Found

A 2025 analysis of SWE-Bench+ found that 47.93% of issues marked as "resolved" were incorrectly classified. The patches passed the test suites, but independent review found the underlying problems were not actually fixed. The test suites were not comprehensive enough to catch the failure.

The SWE-Bench+ finding describes what happens when tests are the only signal: plausible patches that pass weak test suites look identical to correct patches that pass strong ones.

A separate benchmark released in 2025, Needle in the Repo, addresses the gap directly. It evaluates AI-generated repository edits on maintainability: specifically whether the edits integrate well with existing code structure, follow established patterns, and avoid introducing technical debt. The researchers developed it explicitly because existing benchmarks did not cover this dimension.

EvalPlus, which augmented HumanEval with more comprehensive test cases, found significant overestimation of LLM performance in previous evaluations. Models that looked strong on standard benchmarks performed materially worse when the tests were more thorough. More tests revealed more failures the original benchmark missed.

The Dimensions That Don't Get Tested

Code quality decomposes into at least four dimensions.

Correctness: The code produces the right output for the test inputs. SWE-Bench, LiveCodeBench, HumanEval: this is what they measure.

Maintainability: The code is readable by the next engineer. Variable names are descriptive. Functions do one thing. The logic does not require a comment to decode. Needle in the Repo now measures this. Most evaluation setups do not.

Idiom adherence: The code uses the patterns and constructs the language and codebase expect. Python written like C is syntactically valid and passes unit tests. It creates friction for everyone who reads it afterward.

Security posture: The code handles edge cases an attacker could provide. Unit tests catch these when someone thought to write the adversarial test. They often do not, because test authors and code authors share the same blind spots.

The Consequence in Practice

Teams notice the gap not through a dashboard but through behavior: code review cycles do not decrease after adopting AI code generation, even when raw output volume increases. Engineers spend the same review time, just on different problems. Instead of catching logic errors, they catch naming, decomposition, and handling issues the model got wrong in ways the tests did not catch.

The debt accumulates in the untested dimensions. A codebase built with AI assistance and evaluated only on correctness has unknown quality debt in maintainability and security. It does not register as bugs. It registers as "this codebase is hard to work in."

Evaluating Both Dimensions

Two types of checks cover the gap.

Deterministic code graders run checks that do not require LLM inference: does the function adhere to the expected signature, are imports scoped correctly, does the error handling reach the expected branches? These are fast, cheap, and exact.

Custom judges evaluate the semantic dimensions: is this readable? Would it pass review? Are there security concerns the unit tests would not catch? An LLM judge calibrated on examples of high-quality and low-quality code from your actual codebase evaluates this more consistently than ad-hoc peer review.

Running both on the same trace gives you a correctness signal and a quality signal from the same evaluation pass. Two models with nearly identical correctness scores can produce code with meaningfully different quality profiles on the untested dimensions. The only way to see that difference is to measure it.

Frequently Asked Questions

Why don't mainstream benchmarks measure maintainability?

Correctness has a ground truth: the test passes or it does not. Maintainability requires human judgment or a calibrated evaluator. Building benchmark datasets around tasks with automatic ground truth is practical at scale. Quality evaluation is harder to automate and requires per-codebase context.

How do you build a custom judge for code quality?

Start with 30 or more examples from your own codebase: real code a senior engineer reviewed as high quality, and real code that passed tests but failed review. Those labeled examples calibrate the judge to your team's actual quality bar.

Does the quality gap narrow as models improve?

Correctness scores are improving rapidly. SWE-Bench Verified top models are now above 60%. Maintainability research is earlier. Needle in the Repo is a 2025 paper establishing the benchmark. The training signal for quality is less mature than for correctness. The gap is likely to persist even as correctness improves.

What is the signal that your evaluation setup has this gap?

Code review burden stays high or increases post-AI adoption. Engineers describe AI-generated code as technically correct but hard to work with. Refactoring cycles increase in areas with heaviest AI assistance.

Is this specific to certain models?

It is a general pattern across models, with varying quality profiles. Some models produce readable code with weaker security posture. Others produce denser code that handles edge cases better. The variation is real and model-specific. That is why measuring it for your specific task is worth the investment.