The Vibe Father
Explainer
If you only look at one number before choosing a coding model, make it SWE-bench Verified. It is the closest thing the industry has to a benchmark of actual software engineering — not puzzles, not toy functions, but real bugs in real repositories, graded by the repositories' own tests. It is also the heaviest-weighted component of our Vibe Coding Index at 40%, and this post explains what it measures, why we trust it more than anything else, and where its blind spots are.
What it actually is
SWE-bench started as an academic benchmark built from real GitHub issues: take an actual bug report or feature request from a popular open-source Python project, hand the model the repository as it existed at that moment, and ask it to produce a patch. The patch is then judged the only way that matters — by running the repo's own test suite. Tests that were failing must pass; tests that were passing must not break. No rubric, no human vibes, no partial credit for a nice explanation. The original benchmark lives at swebench.com.
The "Verified" part is the crucial upgrade. The original dataset had a real problem: some tasks were effectively unsolvable — underspecified issues, tests that depended on information not in the issue, environments that would not build. OpenAI worked with human annotators to screen the dataset and released SWE-bench Verified: a 500-task subset where humans confirmed each issue is actually solvable from the information given. That turned a noisy benchmark into a trustworthy one — a failure on Verified means the model failed, not the task.
Why it is the closest proxy for real work
Think about what a single Verified task demands. The model must read an issue written by a human who was annoyed, not writing a spec. It must navigate an unfamiliar codebase and find where the problem actually lives — often several files away from where the issue points. It must write a fix that respects the project's existing conventions, and it must not break anything else, because the whole test suite is watching. That is, almost line for line, the job description of a working software engineer.
This is why the benchmark separates models so brutally. Plenty of models can write a clean function from a clean prompt; far fewer can perform surgery on someone else's living codebase. The current top of our board tells the story: Claude Fable 5 at 95.0 — the first model past 95 — then Opus 4.8 at 88.6 and Sonnet 5 at 85.2. Those gaps are not cosmetic. In our experience, the distance between an 85 and a 95 is the difference between "usually needs a retry on hard tasks" and "usually does not."
The limits, honestly
No benchmark is the job, and Verified has three caveats worth knowing.
First, it is Python-heavy. The tasks come from popular Python repositories, so a model's score tells you the most about Python and progressively less about your Rust services or your TypeScript front end. Scores transfer reasonably well in practice, but "reasonably well" is doing quiet work in that sentence.
Second, the tasks are bounded. Every Verified task is a discrete, human-confirmed-solvable issue with a known-good outcome. Real work includes ambiguous requirements, wrong issue reports, and problems whose correct resolution is "close this and fix the design instead." The benchmark cannot measure judgment about whether the task should be done.
Third — and least appreciated — the agent scaffold affects the score. Models do not take this benchmark bare; they run inside a harness that lets them browse files, run tests, and iterate. Two labs reporting scores with different scaffolds are not running a perfectly controlled experiment, and a better harness can be worth real points. It is one more reason we cite every score on our leaderboard to its public source rather than laundering them into an unsourced table: you should be able to see exactly whose number it is and how it was produced.
How we use it
SWE-bench Verified carries 40% of our Vibe Coding Index — the largest weight of the three suites, precisely because it is the nearest proxy for the work you will actually hand a model. The other 60% exists because Verified alone misses things: shell-driving competence (Terminal-Bench) and contamination-resistant fresh problems (LiveCodeBench). The full weighting logic is in how we rank coding models, and when a lab has not yet published a Verified score — as with Grok 4.5 right now — we renormalize around what exists rather than treating missing as zero.
The live column, every score linked to its source and refreshed nightly, is at /benchmarks. When someone tells you a model is great at coding, this is the number to ask for first.