coding benchmark · included in ECI
WeirdML
Novel, unusual machine-learning tasks solved by writing and running code.
Results
149
Random baseline
0.0%
Score ceiling
100%
Released
16 Jan 2025
Score by model release date
Best score by organisation
Leaderboard
| # | Model | Score | Relative | Setting | Released | Run | Evidence |
|---|---|---|---|---|---|---|---|
| 1 | GPT-6 Astra (pro, max) OpenAI | 93.6% ±0.0 | promax | 3 Sep 2026 | — | Source ↗ | |
| 2 | GPT-6 Astra OpenAI | 93.3% ±0.0 | max | 3 Sep 2026 | — | Source ↗ | |
| 3 | Claude Fable 5.1 Anthropic | 92.9% ±0.0 | max | 1 Sep 2026 | — | Source ↗ | |
| 4 | Claude Fable 5 Anthropic | 91.9% ±0.0 | max | 9 Jun 2026 | — | Source ↗ | |
| 5 | Claude Opus 5 Anthropic | 91.8% ±0.0 | max | 24 Jul 2026 | — | Source ↗ | |
| 6 | GPT-5.6 Sol (pro, max) OpenAI | 89.4% ±0.0 | promax | 9 Jul 2026 | — | Source ↗ | |
| 7 | GPT-5.6 Sol OpenAI | 88.8% ±0.0 | high | 9 Jul 2026 | — | Source ↗ | |
| 8 | GPT-5.5 (xhigh) OpenAI | 84.9% ±0.0 | xhigh | 23 Apr 2026 | — | Source ↗ | |
| 9 | GPT-5.5 OpenAI | 83.9% ±0.0 | high | 23 Apr 2026 | — | Source ↗ | |
| 10 | Claude Opus 4.8 Anthropic | 82.9% ±0.0 | xhigh | 28 May 2026 | — | Source ↗ | |
| 11 | Kimi K3 Open source MoonshotAI | 82.6% ±0.0 | max | 16 Jul 2026 | — | Source ↗ | |
| 12 | GPT-5.3-Codex OpenAI | 79.3% | — | 5 Feb 2026 | — | Source ↗ | |
| 13 | GPT-5.6 Terra OpenAI | 78.3% ±0.0 | high | 9 Jul 2026 | — | Source ↗ | |
| 14 | Claude Opus 4.6 Anthropic | 78.0% ±0.0 | high | 5 Feb 2026 | — | Source ↗ | |
| 15 | Claude Opus 4.6 (unknown thinking) Anthropic | 77.9% | unknown | 5 Feb 2026 | — | Source ↗ | |
| 16 | GPT-5.3 Codex (xhigh) OpenAI | 77.9% ±0.0 | xhigh | 5 Feb 2026 | — | Source ↗ | |
| 17 | GPT-5.4 (xhigh) OpenAI | 77.7% ±0.0 | xhigh | 5 Mar 2026 | — | Source ↗ | |
| 18 | Claude Opus 4.7 Anthropic | 76.4% ±0.0 | high | 16 Apr 2026 | — | Source ↗ | |
| 19 | Claude Opus 4.7 (no thinking) Anthropic | 76.4% ±0.0 | — | 16 Apr 2026 | — | Source ↗ | |
| 20 | Claude Opus 4.7 (unknown) Anthropic | 76.4% | unknown | 16 Apr 2026 | — | Source ↗ | |
| 21 | GLM 5.3 Open source Z.ai | 75.4% ±0.0 | max | 14 Aug 2026 | — | Source ↗ | |
| 22 | GPT-5.2 (xhigh) OpenAI | 72.2% ±0.0 | xhigh | 11 Dec 2025 | — | Source ↗ | |
| 23 | Gemini 3.1 Pro Preview Google | 72.1% ±0.0 | — | 19 Feb 2026 | — | Source ↗ | |
| 24 | GLM 5.2 Open source Z.ai | 70.1% ±0.0 | max | 16 Jun 2026 | — | Source ↗ | |
| 25 | Gemini 3 Pro Preview Google DeepMind | 69.9% ±0.0 | — | 18 Nov 2025 | — | Source ↗ | |
| 26 | Claude Sonnet 5 Anthropic | 68.8% ±0.0 | high | 30 Jun 2026 | — | Source ↗ | |
| 27 | Grok 4.6 SpaceXAI | 67.3% ±0.0 | high | 12 Aug 2026 | — | Source ↗ | |
| 28 | GPT-5.5 (no thinking) OpenAI | 67.2% ±0.0 | none | 23 Apr 2026 | — | Source ↗ | |
| 29 | DeepSeek V4 Pro 0813 Open source DeepSeek | 66.2% ±0.0 | max | 13 Aug 2026 | — | Source ↗ | |
| 30 | Claude Sonnet 4.6 (medium) Anthropic | 66.1% ±0.0 | medium | 17 Feb 2026 | — | Source ↗ | |
| 31 | Claude Opus 4.6 (no thinking) Anthropic | 65.9% ±0.0 | — | 5 Feb 2026 | — | Source ↗ | |
| 32 | Claude Opus 4.5 (16k thinking) Anthropic | 63.7% ±0.0 | 16K | 24 Nov 2025 | — | Source ↗ | |
| 33 | GPT-5.2 (medium) OpenAI | 63.4% ±0.0 | medium | 11 Dec 2025 | — | Source ↗ | |
| 34 | DeepSeek V4 Flash 0731 Open source DeepSeek | 63.0% ±0.0 | max | 31 Jul 2026 | — | Source ↗ | |
| 35 | Gemini 3.5 Flash Google | 62.6% ±0.0 | high | 19 May 2026 | — | Source ↗ | |
| 36 | Gemini 3 Flash Preview Google | 61.6% ±0.0 | — | 17 Dec 2025 | — | Source ↗ | |
| 37 | GPT-5.6 Luna OpenAI | 60.9% ±0.0 | high | 9 Jul 2026 | — | Source ↗ | |
| 38 | GPT-5.1 OpenAI | 60.8% ±0.0 | high | 13 Nov 2025 | — | Source ↗ | |
| 39 | GPT-5 OpenAI | 60.7% ±0.0 | high | 7 Aug 2025 | — | Source ↗ | |
| 40 | GPT-5 Pro OpenAI | 60.4% ±0.0 | high | 7 Oct 2025 | — | Source ↗ | |
| 41 | Muse Spark 1.2 (xhigh) Meta AI | 60.3% ±0.0 | xhigh | 5 Aug 2026 | — | Source ↗ | |
| 42 | GPT-5.4 Mini OpenAI | 60.3% ±0.0 | high | 17 Mar 2026 | — | Source ↗ | |
| 43 | o3 Pro OpenAI | 58.2% ±0.0 | high | 10 Jun 2025 | — | Source ↗ | |
| 44 | GPT-5.4 (none) OpenAI | 57.4% ±0.0 | none | 5 Mar 2026 | — | Source ↗ | |
| 45 | GPT-5.4 Pro (no thinking) OpenAI | 57.4% ±0.0 | none | 5 Mar 2026 | — | Source ↗ | |
| 46 | GLM 5.1 Open source Z.ai | 57.1% ±0.0 | — | 7 Apr 2026 | — | Source ↗ | |
| 47 | Gemini 3.6 Flash Google | 56.1% ±0.0 | high | 21 Jul 2026 | — | Source ↗ | |
| 48 | Kimi K2.6 Open source MoonshotAI | 55.9% ±0.0 | — | 20 Apr 2026 | — | Source ↗ | |
| 49 | GPT-5-codex OpenAI | 54.5% | — | 15 Sep 2025 | — | Source ↗ | |
| 50 | Kimi K2.7 Code Open source MoonshotAI | 54.1% ±0.0 | — | 12 Jun 2026 | — | Source ↗ | |
| 51 | Gemini 2.5 Pro (Jun 2025) Google DeepMind | 54.0% | 16K | 5 Jun 2025 | — | Source ↗ | |
| 52 | GPT-5 Mini OpenAI | 52.7% ±0.0 | high | 7 Aug 2025 | — | Source ↗ | |
| 53 | o4 Mini OpenAI | 52.6% ±0.0 | high | 16 Apr 2025 | — | Source ↗ | |
| 54 | o3 OpenAI | 52.4% ±0.0 | high | 16 Apr 2025 | — | Source ↗ | |
| 55 | Gemma 4 31B Open source Google | 52.3% ±0.0 | — | 2 Apr 2026 | — | Source ↗ | |
| 56 | Grok 4.20 SpaceXAI | 52.3% ±0.0 | — | 17 Feb 2026 | — | Source ↗ | |
| 57 | Gemini 3.1 Flash Lite Google | 52.2% ±0.0 | — | 3 Mar 2026 | — | Source ↗ | |
| 58 | Grok 4.3 Beta xAI | 49.9% ±0.0 | unknown | 17 Apr 2026 | — | Source ↗ | |
| 59 | GPT-5.2 (none) OpenAI | 49.6% ±0.0 | none | 11 Dec 2025 | — | Source ↗ | |
| 60 | GPT-5.2 (low) OpenAI | 49.6% | low | 11 Dec 2025 | — | Source ↗ | |
| 61 | GPT-5.4 Nano OpenAI | 49.2% ±0.0 | high | 17 Mar 2026 | — | Source ↗ | |
| 62 | DeepSeek v4 Pro (high) Open source DeepSeek | 48.9% ±0.0 | max | 24 Apr 2026 | — | Source ↗ | |
| 63 | GLM 5 Open source Z.ai | 48.2% ±0.0 | — | 11 Feb 2026 | — | Source ↗ | |
| 64 | gpt-oss-120b Open source OpenAI | 48.2% | high | 5 Aug 2025 | — | Source ↗ | |
| 65 | Claude Sonnet 4.5 (16k thinking) Anthropic | 47.7% ±0.0 | 16K | 29 Sep 2025 | — | Source ↗ | |
| 66 | o1-preview OpenAI | 47.6% ±0.0 | — | 12 Sep 2024 | — | Source ↗ | |
| 67 | DeepSeek-V3.2-Speciale Open source DeepSeek | 46.7% ±0.0 | — | 1 Dec 2025 | — | Source ↗ | |
| 68 | Claude Sonnet 4.5 (no thinking) Anthropic | 46.7% ±0.0 | — | 29 Sep 2025 | — | Source ↗ | |
| 69 | Grok 4.5 (unknown thinking) xAI | 46.4% ±0.0 | unknown | 8 Jul 2026 | — | Source ↗ | |
| 70 | Claude Sonnet 4 Anthropic | 46.1% ±0.0 | 16K | 22 May 2025 | — | Source ↗ | |
| 71 | o1 OpenAI | 46.1% ±0.0 | high | 17 Dec 2024 | — | Source ↗ | |
| 72 | Claude Opus 4.1 Anthropic | 45.9% ±0.0 | 16K | 5 Aug 2025 | — | Source ↗ | |
| 73 | Grok 4 xAI | 45.7% ±0.0 | — | 9 Jul 2025 | — | Source ↗ | |
| 74 | DeepSeek v4 Flash (max) Open source DeepSeek | 45.6% ±0.0 | max | 24 Apr 2026 | — | Source ↗ | |
| 75 | Kimi K2.5 Open source MoonshotAI | 45.6% ±0.0 | — | 27 Jan 2026 | — | Source ↗ | |
| 76 | Claude Haiku 4.5 Anthropic | 45.4% ±0.0 | — | 15 Oct 2025 | — | Source ↗ | |
| 77 | Claude Opus 4 Anthropic | 43.7% ±0.0 | 16K | 22 May 2025 | — | Source ↗ | |
| 78 | Mistral Medium 3.5 Open source Mistral | 43.7% ±0.0 | — | 28 Apr 2026 | — | Source ↗ | |
| 79 | o3 Mini OpenAI | 43.7% ±0.0 | high | 31 Jan 2025 | — | Source ↗ | |
| 80 | Nemotron 3 Ultra Open source NVIDIA | 43.5% ±0.0 | — | 4 Jun 2026 | — | Source ↗ | |
| 81 | Mercury 2 Inception | 43.2% ±0.0 | — | 4 Mar 2026 | — | Source ↗ | |
| 82 | Grok 4 Fast xAI | 42.9% ±0.0 | — | 19 Sep 2025 | — | Source ↗ | |
| 83 | Kimi K2 Thinking Open source MoonshotAI | 42.8% ±0.0 | — | 6 Nov 2025 | — | Source ↗ | |
| 84 | Grok-3 mini xAI | 42.6% | high | 24 Jun 2025 | — | Source ↗ | |
| 85 | Gemini 2.5 Flash (Sep 2025) Google DeepMind | 41.9% ±0.0 | 16k | 25 Sep 2025 | — | Source ↗ | |
| 86 | DeepSeek-R1 (May 2025) Open source DeepSeek | 41.6% ±0.0 | — | 28 May 2025 | — | Source ↗ | |
| 87 | Qwen3 Coder 480B A35B Open source Qwen | 41.2% ±0.0 | — | 23 Jul 2025 | — | Source ↗ | |
| 88 | Qwen3-235B-A22B-Thinking (Jul 2025) Open source Alibaba | 41.0% | — | 25 Jul 2025 | — | Source ↗ | |
| 89 | Gemini 2.5 Flash (Apr 2025) Google DeepMind | 41.0% ±0.0 | — | 17 Apr 2025 | — | Source ↗ | |
| 90 | Gemini 2.5 Flash (May 2025) Google DeepMind | 40.9% | 16K | 20 May 2025 | — | Source ↗ | |
| 91 | gpt-oss-20b Open source OpenAI | 40.9% ±0.0 | high | 5 Aug 2025 | — | Source ↗ | |
| 92 | GLM-4.5 Thinking Open source Z.ai (Zhipu AI),Tsinghua University | 40.6% ±0.0 | thinking | 5 Aug 2025 | — | Source ↗ | |
| 93 | Claude 3.5 Sonnet (Oct 2024) Anthropic | 40.0% ±0.0 | — | 22 Oct 2024 | — | Source ↗ | |
| 94 | Qwen3.5-27B Open source Qwen | 39.5% ±0.0 | — | 25 Feb 2026 | — | Source ↗ | |
| 95 | DeepSeek V3.2 Exp Open source DeepSeek | 39.5% ±0.0 | thinking | 29 Sep 2025 | — | Source ↗ | |
| 96 | GPT-4.5 Preview (Feb 2025) OpenAI | 39.4% ±0.0 | — | 27 Feb 2025 | — | Source ↗ | |
| 97 | Kimi K2 (Jul 2025) Open source Moonshot | 39.4% ±0.0 | — | 12 Jul 2025 | — | Source ↗ | |
| 98 | Kimi K2 Instruct Open source Moonshot | 39.4% | — | 12 Jul 2025 | — | Source ↗ | |
| 99 | GPT-4.1 OpenAI | 39.0% ±0.0 | — | 14 Apr 2025 | — | Source ↗ | |
| 100 | Gemini 3.5 Flash Lite Google | 39.0% ±0.0 | high | 21 Jul 2026 | — | Source ↗ | |
| 101 | Qwen3 235B A22B Thinking 2507 Open source Qwen | 38.7% ±0.0 | — | 25 Jul 2025 | — | Source ↗ | |
| 102 | Qwen3-235B-A22B-Instruct (Jul 2025) Open source Alibaba | 38.7% | — | 25 Jul 2025 | — | Source ↗ | |
| 103 | DeepSeek V3.1 Open source DeepSeek | 38.4% ±0.0 | thinking | 21 Aug 2025 | — | Source ↗ | |
| 104 | GPT-5 Nano OpenAI | 38.1% ±0.0 | high | 7 Aug 2025 | — | Source ↗ | |
| 105 | Nemotron 3 Super Open source NVIDIA | 38.0% ±0.0 | — | 11 Mar 2026 | — | Source ↗ | |
| 106 | GPT-5.4 nano (no thinking) OpenAI | 38.0% ±0.0 | none | 17 Mar 2026 | — | Source ↗ | |
| 107 | GPT-5.4 mini (none) OpenAI | 37.9% ±0.0 | none | 17 Mar 2026 | — | Source ↗ | |
| 108 | GPT-4.1 Mini OpenAI | 37.6% ±0.0 | — | 14 Apr 2025 | — | Source ↗ | |
| 109 | Qwen3 235B A22B Open source Qwen | 37.3% ±0.0 | — | 28 Apr 2025 | — | Source ↗ | |
| 110 | Grok 3 xAI | 37.2% ±0.0 | — | 9 Apr 2025 | — | Source ↗ | |
| 111 | MiniMax M2.7 Open source MiniMax | 37.0% ±0.0 | — | 18 Mar 2026 | — | Source ↗ | |
| 112 | Kimi K2 (Sep 2025) Open source Moonshot | 36.7% ±0.0 | — | 5 Sep 2025 | — | Source ↗ | |
| 113 | R1 Open source DeepSeek | 36.5% ±0.0 | — | 20 Jan 2025 | — | Source ↗ | |
| 114 | o1-mini (medium) OpenAI | 36.3% ±0.0 | medium | 12 Sep 2024 | — | Source ↗ | |
| 115 | DeepSeek-V3 (Mar 2025) Open source DeepSeek | 36.1% ±0.0 | — | 24 Mar 2025 | — | Source ↗ | |
| 116 | Gemini 2.5 Flash-Lite (Jun 2025) Google DeepMind | 35.2% | 16K | 17 Jun 2025 | — | Source ↗ | |
| 117 | gemini-2.5-flash-lite-preview-06-17 (Default thinking length) Google DeepMind | 35.2% ±0.0 | — | 17 Jun 2025 | — | Source ↗ | |
| 118 | Gemma 4 26B A4B Open source Google | 35.2% ±0.0 | — | 2 Apr 2026 | — | Source ↗ | |
| 119 | grok-code-fast-1 xAI | 35.1% ±0.0 | — | 28 Aug 2025 | — | Source ↗ | |
| 120 | Qwen3.6 35B A3B Open source Qwen | 34.5% ±0.0 | — | 27 Apr 2026 | — | Source ↗ | |
| 121 | Qwen3 Coder Next Open source Qwen | 34.4% ±0.0 | — | 4 Feb 2026 | — | Source ↗ | |
| 122 | Mistral Medium 3.1 Mistral | 33.1% ±0.0 | — | 13 Aug 2025 | — | Source ↗ | |
| 123 | Inkling Open source Thinking Machines | 32.3% ±0.0 | high | 15 Jul 2026 | — | Source ↗ | |
| 124 | Claude 3.5 Sonnet (Jun 2024) Anthropic | 31.0% ±0.0 | — | 20 Jun 2024 | — | Source ↗ | |
| 125 | Claude 3.5 Haiku (Oct 2024) Anthropic | 30.7% ±0.0 | — | 22 Oct 2024 | — | Source ↗ | |
| 126 | Qwen3 30B A3B Open source Qwen | 29.8% | — | 29 Apr 2025 | — | Source ↗ | |
| 127 | GPT-5 nano (low) OpenAI | 25.9% ±0.0 | low | 7 Aug 2025 | — | Source ↗ | |
| 128 | Gemini 2.0 Flash (Feb 2025) Google DeepMind,Google | 25.8% ±0.0 | — | 5 Feb 2025 | — | Source ↗ | |
| 129 | GPT-4o (Nov 2024) OpenAI | 25.1% ±0.0 | — | 20 Nov 2024 | — | Source ↗ | |
| 130 | Gemini 1.5 Flash (Sep 2024) Google DeepMind | 24.9% ±0.0 | — | 24 Sep 2024 | — | Source ↗ | |
| 131 | Llama 4 Maverick Open source Meta | 24.5% ±0.0 | — | 6 Apr 2025 | — | Source ↗ | |
| 132 | Grok-2 (Dec 2024) xAI | 22.2% ±0.0 | — | 12 Dec 2024 | — | Source ↗ | |
| 133 | Gemini 1.5 Pro (Sept 2024) Google DeepMind | 22.2% ±0.0 | — | 24 Sep 2024 | — | Source ↗ | |
| 134 | Llama 3.1-405B Open source Meta AI | 21.4% ±0.0 | — | 23 Jul 2024 | — | Source ↗ | |
| 135 | Claude 3 Opus Anthropic | 19.2% ±0.0 | — | 29 Feb 2024 | — | Source ↗ | |
| 136 | GPT-4.1 Nano OpenAI | 19.0% ±0.0 | — | 14 Apr 2025 | — | Source ↗ | |
| 137 | GPT-4 Turbo (Apr 2024) OpenAI | 18.0% ±0.0 | — | 9 Apr 2024 | — | Source ↗ | |
| 138 | Qwen2.5-72B Open source Alibaba | 16.0% ±0.0 | — | 19 Sep 2024 | — | Source ↗ | |
| 139 | Llama 3.3 70B Open source Meta AI | 14.4% ±0.0 | — | 6 Dec 2024 | — | Source ↗ | |
| 140 | GPT-4 (Jun 2023) OpenAI | 12.4% ±0.0 | — | 13 Jun 2023 | — | Source ↗ | |
| 141 | GPT-4o-mini OpenAI | 11.8% ±0.0 | — | 18 Jul 2024 | — | Source ↗ | |
| 142 | Qwen2-72B Open source Alibaba | 11.3% ±0.0 | — | 7 Jun 2024 | — | Source ↗ | |
| 143 | Claude 3 Sonnet Anthropic | 10.2% ±0.0 | — | 29 Feb 2024 | — | Source ↗ | |
| 144 | Claude 3 Haiku Anthropic | 9.8% ±0.0 | — | 7 Mar 2024 | — | Source ↗ | |
| 145 | Llama 3.1-70B Open source Meta AI | 9.0% ±0.0 | — | 23 Jul 2024 | — | Source ↗ | |
| 146 | Claude 2.1 Anthropic | 7.1% ±0.0 | — | 21 Nov 2023 | — | Source ↗ | |
| 147 | GPT-3.5 Turbo (Jan 2024) OpenAI | 3.5% ±0.0 | — | 25 Jan 2024 | — | Source ↗ | |
| 148 | Mixtral 8x22B Open source Mistral AI | 3.2% ±0.0 | — | 17 Apr 2024 | — | Source ↗ | |
| 149 | Llama 3.1-8B Open source Meta AI | 1.7% ±0.0 | — | 23 Jul 2024 | — | Source ↗ |
Caveats: settings (reasoning effort, agent scaffold) differ between rows and materially affect scores; the best reported setting per model is shown. Source: Epoch AI, CC BY 4.0.