coding benchmark · included in ECI
SWE-Bench verified
500 human-validated real GitHub issues; the model must produce a patch that passes tests.
Results
33
Random baseline
0.0%
Score ceiling
100%
Released
13 Aug 2024
Score by model release date
Best score by organisation
Leaderboard
| # | Model | Score | Relative | Setting | Released | Run | Evidence |
|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 4.7 Anthropic | 83.5% ±1.7 | max | 16 Apr 2026 | 20 Apr 2026 | Log ↗ | |
| 2 | GPT-5.5 (xhigh) OpenAI | 80.6% ±1.8 | xhigh | 23 Apr 2026 | 24 Apr 2026 | Log ↗ | |
| 3 | Gemini 3.5 Flash Google | 79.3% ±1.8 | high | 19 May 2026 | 1 Jun 2026 | Log ↗ | |
| 4 | Claude Opus 4.6 (no thinking) Anthropic | 78.7% ±1.9 | — | 5 Feb 2026 | 18 Feb 2026 | Log ↗ | |
| 5 | GLM 5.2 Open source Z.ai | 78.7% ±1.9 | max | 16 Jun 2026 | 25 Jun 2026 | Log ↗ | |
| 6 | DeepSeek v4 Pro (high) Open source DeepSeek | 77.6% ±1.9 | max | 24 Apr 2026 | 18 Jun 2026 | Log ↗ | |
| 7 | Qwen3.7 Max Qwen | 77.3% ±1.9 | — | 19 May 2026 | 18 Jun 2026 | Log ↗ | |
| 8 | GPT-5.4 OpenAI | 76.9% ±1.9 | high | 5 Mar 2026 | 6 Mar 2026 | Log ↗ | |
| 9 | Kimi K2.6 Open source MoonshotAI | 76.7% ±1.9 | — | 20 Apr 2026 | 8 May 2026 | Log ↗ | |
| 10 | Qwen 3.6 Max (Preview) Alibaba | 76.7% ±1.9 | — | 20 Apr 2026 | 28 May 2026 | Log ↗ | |
| 11 | Claude Opus 4.5 (no thinking) Anthropic | 76.7% ±1.9 | — | 24 Nov 2025 | 5 Feb 2026 | Log ↗ | |
| 12 | Gemini 3.1 Pro Preview Google | 75.6% ±2.0 | — | 19 Feb 2026 | 24 Feb 2026 | Log ↗ | |
| 13 | Gemini 3 Flash Preview Google | 75.4% ±2.0 | — | 17 Dec 2025 | 18 Feb 2026 | Log ↗ | |
| 14 | Claude Sonnet 4.6 (no thinking) Anthropic | 75.2% ±2.0 | — | 17 Feb 2026 | 21 Feb 2026 | Log ↗ | |
| 15 | GPT-5.3-Codex OpenAI | 74.8% ±2.0 | high | 5 Feb 2026 | 25 Feb 2026 | Log ↗ | |
| 16 | GLM 5.1 Open source Z.ai | 74.2% ±2.0 | — | 7 Apr 2026 | 15 May 2026 | Source ↗ | |
| 17 | Kimi K2.5 Open source MoonshotAI | 73.8% ±2.0 | — | 27 Jan 2026 | 17 Feb 2026 | Log ↗ | |
| 18 | GPT-5.2 OpenAI | 73.8% ±2.0 | high | 11 Dec 2025 | 12 Feb 2026 | Log ↗ | |
| 19 | GPT-5 OpenAI | 73.6% ±2.0 | high | 7 Aug 2025 | 6 Feb 2026 | Log ↗ | |
| 20 | Claude Opus 4.1 Anthropic | 73.3% ±2.0 | — | 5 Aug 2025 | 11 Feb 2026 | Log ↗ | |
| 21 | Gemini 3 Pro Preview Google DeepMind | 72.9% ±2.0 | — | 18 Nov 2025 | 13 Feb 2026 | Log ↗ | |
| 22 | GLM 5 Open source Z.ai | 72.1% ±2.1 | — | 11 Feb 2026 | 15 Feb 2026 | Log ↗ | |
| 23 | GPT-5 (medium) OpenAI | 71.5% ±2.1 | medium | 7 Aug 2025 | 5 Feb 2026 | Log ↗ | |
| 24 | Claude Sonnet 4.5 (no thinking) Anthropic | 71.3% ±2.1 | — | 29 Sep 2025 | 5 Feb 2026 | Log ↗ | |
| 25 | Claude Opus 4 Anthropic | 70.7% ±2.1 | — | 22 May 2025 | 6 Feb 2026 | Log ↗ | |
| 26 | GPT-5.1 OpenAI | 68.0% ±2.1 | high | 13 Nov 2025 | 18 Feb 2026 | Log ↗ | |
| 27 | GPT-5 mini (medium) OpenAI | 64.7% ±2.2 | medium | 7 Aug 2025 | 1 Feb 2026 | Log ↗ | |
| 28 | o3 (medium) OpenAI | 62.3% ±2.2 | medium | 16 Apr 2025 | 12 Feb 2026 | Log ↗ | |
| 29 | Claude 3.7 Sonnet Anthropic | 61.0% ±2.2 | — | 24 Feb 2025 | 4 Feb 2026 | Log ↗ | |
| 30 | Qwen 3.6 Plus (2026-04-02) Alibaba | 57.9% ±2.2 | — | 31 Mar 2026 | 14 May 2026 | Log ↗ | |
| 31 | Gemini 2.5 Pro (Jun 2025) Google DeepMind | 57.6% ±2.3 | — | 5 Jun 2025 | 13 Feb 2026 | Log ↗ | |
| 32 | GPT-4.1 OpenAI | 48.5% ±2.3 | — | 14 Apr 2025 | 8 Feb 2026 | Log ↗ | |
| 33 | GPT-4o (Nov 2024) OpenAI | 31.0% ±2.1 | — | 20 Nov 2024 | 11 Feb 2026 | Log ↗ |
Caveats: settings (reasoning effort, agent scaffold) differ between rows and materially affect scores; the best reported setting per model is shown. Source: Epoch AI, CC BY 4.0.