The LayerLens Team covers AI model evaluations, benchmark analysis, and the evolving landscape of AI performance. For the latest independent evaluation data, explore $LayerLens/$.
Summary
GPT-5 (high) from OpenAI scored 51.7 on SWE-bench Lite (SWE-agent), placing it top 10 (rank 6 of 45) on this benchmark. This places the model in the below-frontier band for SWE-bench Lite (SWE-agent). Acceptable for cost-sensitive workloads or as part of a multi-model ensemble. Not a default choice for high-stakes routing.
Model details
- Provider: OpenAI
- Model key:
openai/gpt-5-high - Context length: 400,000 tokens
- License: Proprietary
- Open weights: no
Benchmark methodology
Benchmark goal: The benchmark aims to evaluate the capabilities of Large Language Models (LLMs) across a diverse range of tasks, particularly focusing on their proficiency in specific domains and tasks where general-purpose LLMs might struggle.
Scoring metrics:
- Accuracy: The proportion of questions for which the model provides the correct answer, typically expressed as a percentage.
Analysis
Key takeaways:
- The model performs well on tasks requiring direct code modification and logical bug fixes within a single component.
- It struggles with tasks that require a deeper understanding of system-wide implications, object lifecycle, or complex inter-component interactions.
- Improvements are needed in handling diverse data types, especially when explicit type conversions or specific evaluation contexts are involved.
- The model shows potential for feature implementation but needs refinement in anticipating and testing the full scope of requested changes.
Failure modes observed
Common failure modes:
- Misinterpretation of problem context leading to partial or incorrect fixes.
- Failure to fully trace side effects of code changes across interconnected systems.
- Inadequate handling of edge cases or specific data types.
- Difficulty in applying global logical changes that account for different states or configurations.
- Incorrect understanding of internal object representations and how they are handled by printing or serialization functions.
Example: In the task involving HttpResponse and memoryview objects, the model initially failed to convert bytearray objects correctly, leading to incorrect byte representations instead of expected content.
Secondary metrics
- Readability score: 14.9
- Toxicity score: 0.002
- Ethics score: 0.000
Run this evaluation yourself
LayerLens evaluates GPT-5 (high) continuously across 11+ benchmarks. To replicate this SWE-bench Lite (SWE-agent) evaluation on your own model, traces, or a different benchmark configuration, open the model in $LayerLens/$.
_Source: LayerLens evaluation 6912046c61ef3abf3656c486. Updated 2025-11-11._