coding benchmark · included in ECI
WeirdML
Novel, unusual machine-learning tasks solved by writing and running code.
Results
49
Random baseline
0.0%
Score ceiling
100%
Released
16 Jan 2025
Score by model release date
Best score by organisation
Leaderboard
| # | Model | Score | Relative | Setting | Released | Run | Evidence |
|---|---|---|---|---|---|---|---|
| 1 | Kimi K3 Open source MoonshotAI | 82.6% ±0.0 | max | 16 Jul 2026 | — | Source ↗ | |
| 2 | GLM 5.3 Open source Z.ai | 75.4% ±0.0 | max | 14 Aug 2026 | — | Source ↗ | |
| 3 | GLM 5.2 Open source Z.ai | 70.1% ±0.0 | max | 16 Jun 2026 | — | Source ↗ | |
| 4 | DeepSeek V4 Pro 0813 Open source DeepSeek | 66.2% ±0.0 | max | 13 Aug 2026 | — | Source ↗ | |
| 5 | DeepSeek V4 Flash 0731 Open source DeepSeek | 63.0% ±0.0 | max | 31 Jul 2026 | — | Source ↗ | |
| 6 | GLM 5.1 Open source Z.ai | 57.1% ±0.0 | — | 7 Apr 2026 | — | Source ↗ | |
| 7 | Kimi K2.6 Open source MoonshotAI | 55.9% ±0.0 | — | 20 Apr 2026 | — | Source ↗ | |
| 8 | Kimi K2.7 Code Open source MoonshotAI | 54.1% ±0.0 | — | 12 Jun 2026 | — | Source ↗ | |
| 9 | Gemma 4 31B Open source Google | 52.3% ±0.0 | — | 2 Apr 2026 | — | Source ↗ | |
| 10 | DeepSeek v4 Pro (high) Open source DeepSeek | 48.9% ±0.0 | max | 24 Apr 2026 | — | Source ↗ | |
| 11 | GLM 5 Open source Z.ai | 48.2% ±0.0 | — | 11 Feb 2026 | — | Source ↗ | |
| 12 | gpt-oss-120b Open source OpenAI | 48.2% | high | 5 Aug 2025 | — | Source ↗ | |
| 13 | DeepSeek-V3.2-Speciale Open source DeepSeek | 46.7% ±0.0 | — | 1 Dec 2025 | — | Source ↗ | |
| 14 | DeepSeek v4 Flash (max) Open source DeepSeek | 45.6% ±0.0 | max | 24 Apr 2026 | — | Source ↗ | |
| 15 | Kimi K2.5 Open source MoonshotAI | 45.6% ±0.0 | — | 27 Jan 2026 | — | Source ↗ | |
| 16 | Mistral Medium 3.5 Open source Mistral | 43.7% ±0.0 | — | 28 Apr 2026 | — | Source ↗ | |
| 17 | Nemotron 3 Ultra Open source NVIDIA | 43.5% ±0.0 | — | 4 Jun 2026 | — | Source ↗ | |
| 18 | Kimi K2 Thinking Open source MoonshotAI | 42.8% ±0.0 | — | 6 Nov 2025 | — | Source ↗ | |
| 19 | DeepSeek-R1 (May 2025) Open source DeepSeek | 41.6% ±0.0 | — | 28 May 2025 | — | Source ↗ | |
| 20 | Qwen3 Coder 480B A35B Open source Qwen | 41.2% ±0.0 | — | 23 Jul 2025 | — | Source ↗ | |
| 21 | Qwen3-235B-A22B-Thinking (Jul 2025) Open source Alibaba | 41.0% | — | 25 Jul 2025 | — | Source ↗ | |
| 22 | gpt-oss-20b Open source OpenAI | 40.9% ±0.0 | high | 5 Aug 2025 | — | Source ↗ | |
| 23 | GLM-4.5 Thinking Open source Z.ai (Zhipu AI),Tsinghua University | 40.6% ±0.0 | thinking | 5 Aug 2025 | — | Source ↗ | |
| 24 | Qwen3.5-27B Open source Qwen | 39.5% ±0.0 | — | 25 Feb 2026 | — | Source ↗ | |
| 25 | DeepSeek V3.2 Exp Open source DeepSeek | 39.5% ±0.0 | thinking | 29 Sep 2025 | — | Source ↗ | |
| 26 | Kimi K2 (Jul 2025) Open source Moonshot | 39.4% ±0.0 | — | 12 Jul 2025 | — | Source ↗ | |
| 27 | Kimi K2 Instruct Open source Moonshot | 39.4% | — | 12 Jul 2025 | — | Source ↗ | |
| 28 | Qwen3 235B A22B Thinking 2507 Open source Qwen | 38.7% ±0.0 | — | 25 Jul 2025 | — | Source ↗ | |
| 29 | Qwen3-235B-A22B-Instruct (Jul 2025) Open source Alibaba | 38.7% | — | 25 Jul 2025 | — | Source ↗ | |
| 30 | DeepSeek V3.1 Open source DeepSeek | 38.4% ±0.0 | thinking | 21 Aug 2025 | — | Source ↗ | |
| 31 | Nemotron 3 Super Open source NVIDIA | 38.0% ±0.0 | — | 11 Mar 2026 | — | Source ↗ | |
| 32 | Qwen3 235B A22B Open source Qwen | 37.3% ±0.0 | — | 28 Apr 2025 | — | Source ↗ | |
| 33 | MiniMax M2.7 Open source MiniMax | 37.0% ±0.0 | — | 18 Mar 2026 | — | Source ↗ | |
| 34 | Kimi K2 (Sep 2025) Open source Moonshot | 36.7% ±0.0 | — | 5 Sep 2025 | — | Source ↗ | |
| 35 | R1 Open source DeepSeek | 36.5% ±0.0 | — | 20 Jan 2025 | — | Source ↗ | |
| 36 | DeepSeek-V3 (Mar 2025) Open source DeepSeek | 36.1% ±0.0 | — | 24 Mar 2025 | — | Source ↗ | |
| 37 | Gemma 4 26B A4B Open source Google | 35.2% ±0.0 | — | 2 Apr 2026 | — | Source ↗ | |
| 38 | Qwen3.6 35B A3B Open source Qwen | 34.5% ±0.0 | — | 27 Apr 2026 | — | Source ↗ | |
| 39 | Qwen3 Coder Next Open source Qwen | 34.4% ±0.0 | — | 4 Feb 2026 | — | Source ↗ | |
| 40 | Inkling Open source Thinking Machines | 32.3% ±0.0 | high | 15 Jul 2026 | — | Source ↗ | |
| 41 | Qwen3 30B A3B Open source Qwen | 29.8% | — | 29 Apr 2025 | — | Source ↗ | |
| 42 | Llama 4 Maverick Open source Meta | 24.5% ±0.0 | — | 6 Apr 2025 | — | Source ↗ | |
| 43 | Llama 3.1-405B Open source Meta AI | 21.4% ±0.0 | — | 23 Jul 2024 | — | Source ↗ | |
| 44 | Qwen2.5-72B Open source Alibaba | 16.0% ±0.0 | — | 19 Sep 2024 | — | Source ↗ | |
| 45 | Llama 3.3 70B Open source Meta AI | 14.4% ±0.0 | — | 6 Dec 2024 | — | Source ↗ | |
| 46 | Qwen2-72B Open source Alibaba | 11.3% ±0.0 | — | 7 Jun 2024 | — | Source ↗ | |
| 47 | Llama 3.1-70B Open source Meta AI | 9.0% ±0.0 | — | 23 Jul 2024 | — | Source ↗ | |
| 48 | Mixtral 8x22B Open source Mistral AI | 3.2% ±0.0 | — | 17 Apr 2024 | — | Source ↗ | |
| 49 | Llama 3.1-8B Open source Meta AI | 1.7% ±0.0 | — | 23 Jul 2024 | — | Source ↗ |
Caveats: settings (reasoning effort, agent scaffold) differ between rows and materially affect scores; the best reported setting per model is shown. Source: Epoch AI, CC BY 4.0.