The Vibe Father
Roundup
Machine learning engineering gets a benchmark break most domains don't. SWE-bench Verified — the suite everyone quotes for real-world coding skill — is built from real Python repositories, and ML lives in Python: PyTorch, JAX, Hugging Face, the whole stack. So a high SWE-bench Verified score is a genuinely strong proxy for ML engineering ability, more so than it is for DevOps or mobile. The top of the board is trustworthy here. We keep the live version at /benchmarks (VCI = SWE 40 / TB 30 / LCB 30).
But ML has its own specific traps that a general score doesn't test, and they're brutal because they fail quietly or slowly: a tensor-shape mismatch that only surfaces three layers deep, a CUDA/device error that costs an hour of GPU time, a training loop that runs and converges to nonsense. So pick a strong reasoner — and stay skeptical of code that "runs."
The ML engineering board, July 2026
Sorted by SWE-bench Verified, which for this domain reads close to direct. Where a score isn't published we say so — we don't invent numbers.
| Model | SWE-bench Verified (strong proxy here) | Notes for ML work |
|---|---|---|
 Claude Fable 5 | 95.0 | Top ceiling for custom layers and training internals |
| Claude Opus 4.8 | 88.6 | The default — elite, half of Fable's price |
| Claude Sonnet 5 | 85.2 | High-volume pipeline and data-loader work |
 GPT-5.5 | 80.6 | Strong all-rounder, good with CLIs and cluster tooling |
 Gemini 3.5 Flash | 79.3 | 167 tok/s — fast experiment iteration |
 DeepSeek V4 Pro | 77.6 | $0.435 / $0.87 per M — cheap high-volume scripting |
 Qwen3.7 Max | 77.3 | 204 tok/s — fastest here, great for rapid runs |
| Gemini 3.1 Pro | 75.6 | Puzzle strength (LCB 88.5) above its repo score |
The traps that no benchmark measures
ML engineering fails in ways SWE-bench doesn't probe. These are the ones that eat your day:
- Tensor shapes. The classic. A transpose in the wrong place, a batch dimension dropped, a broadcast that silently does the wrong thing. The best models track shapes through a forward pass in their head; weaker ones guess and let the runtime error tell them. Ask the model to annotate expected shapes at each step and verify them.
- CUDA and devices. Tensors on the wrong device, dtype mismatches under mixed precision, out-of-memory errors that depend on batch size. These waste real GPU money. A model that reasons about device placement up front saves you the crash on step 900.
- Silent training bugs. The worst kind — the loop runs, the loss goes down, and the model learns garbage because of a leaked label, an unshuffled loader, or a metric computed wrong. No exception, no red text. Only careful reasoning and validation catch these.
- Framework churn. PyTorch, JAX, and the HF libraries move fast; a model may reach for a deprecated API. State your versions.
Why the top of the board matters more here
In a lot of domains a mid-board model is plenty. ML tilts toward the reasoning ceiling because the failures are subtle and expensive. Opus 4.8 at 88.6 is the default — it holds shapes and devices in mind and reasons about training dynamics carefully. Fable 5's 95.0 earns its price on the genuinely hard internals: a custom autograd function, a distributed-training bug, a numerically unstable loss. Gemini 3.1 Pro's LiveCodeBench strength (88.5) makes it a real option for self-contained algorithmic pieces — a custom sampler, a metric implementation.
Picks by ML workload
- Model internals and training loops. Opus 4.8 as the default; Fable 5 for custom layers, distributed training, and numerically delicate code.
- Data loaders and pipelines. Sonnet 5, or DeepSeek V4 Pro for value — capable and cheap for the plumbing around the model.
- Fast experiment iteration. Gemini 3.5 Flash or Qwen3.7 Max. Speed helps when you're tweaking configs and rerunning, and you validate every run anyway.
- Contained algorithmic pieces. Gemini 3.1 Pro is a strong option given its LiveCodeBench strength.
How to actually test on your stack
Do this before you commit GPU budget to a model's output:
- Pick one representative task — a real layer, a real training bug, a real data-loader change. Not a toy MNIST loop.
- Give the same decision-complete task to two or three candidates, versions stated. See prompt engineering for coding agents for how to make it fair.
- Judge on your criteria: do the shapes check out, does it run on your device without a rewrite, and does a tiny training run behave sanely?
- Repeat on a second task. One is a data point; two is a signal.
Bringing your own API keys makes this cheap — swap the model, rerun the task, compare, paying only for tokens. The Vibe Father runs different models against the same job for exactly this reason.
Our honest bottom line for ML engineering
Because ML lives where SWE-bench measures, the board is a real ranking here — trust the top of it. Run Opus 4.8 as your default, Fable 5 for the hard internals, DeepSeek V4 Pro or a fast model for the surrounding plumbing. But the discipline that saves your GPU hours isn't model choice — it's verifying shapes, devices, and training behavior before you trust a run. For the data that feeds the model see the best AI for data science, for the Python underneath it the best AI model for Python, and the whole field in the best coding model roundup. Understand the benchmark this rests on in our SWE-bench explainer. Live numbers at /benchmarks.
