The Vibe Father
Benchmark explainer
If you have watched coding leaderboards at all in 2026, you have seen a strange thing happen: the top models cleared SWE-bench Verified so thoroughly that a 90-plus score stopped being newsworthy. Claude Fable 5 sits at 95.0 on our own board; Opus 4.8 is at 88.6. When a benchmark's ceiling gets crowded, it stops telling you who is actually better. That is the problem SWE-bench Pro is built to fix — a tougher, less-contaminated variant of the benchmark that made agentic coding measurable in the first place. This is what it is, why scores on it are reported lower, and why that lower number is a feature, not a regression.
Start with what SWE-bench is
SWE-bench takes real issues from real open-source Python repositories and asks a model to produce a patch that resolves the issue. The patch is then run against the project's actual test suite. Either the tests pass or they do not — there is no partial credit, no "looks plausible," no judge to charm. That pass/fail rigor is why it became the reference benchmark for whether a model can do repository-scale engineering rather than answer isolated puzzles. SWE-bench Verified is the human-vetted subset where the tasks are known to be solvable and the tests are known to be fair; it is the version we track, and the one most people mean when they say "SWE-bench." If the term is new to you, we wrote a full SWE-bench Verified explainer that goes deeper on the mechanics.
Why a tougher variant became necessary
Two forces eroded SWE-bench Verified's usefulness as models improved. The first is saturation. When the best models resolve the large majority of a fixed task set, the gaps between them shrink into noise, and a one-point difference can come down to which handful of quirky issues happened to be in the set. A benchmark that everyone nearly aces has stopped discriminating.
The second, more uncomfortable force is contamination. SWE-bench draws from public GitHub repositories, and the issues, discussions, and fixes for those repositories are exactly the kind of text that ends up in training data. When a model may have seen the resolved issue during training, a passing patch no longer cleanly proves it can reason through a novel bug — it might be recalling one. Nobody can fully rule this out for a public, static benchmark, and as the leaderboard climbed, that doubt grew louder.
What makes SWE-bench Pro harder
SWE-bench Pro, which drew significant attention through 2026, responds to both problems by raising the difficulty and lowering the contamination risk. We will describe the design intent rather than quote a scoreboard, because the exact figures move and the point is conceptual:
- Harder tasks. The issues are selected to be more involved — multi-file changes, deeper reasoning chains, and problems where a naive patch that touches one function will not make the tests pass. These are the tasks where the difference between a model that plans and a model that guesses actually shows.
- Less contamination surface. By emphasizing tasks that are harder to have simply memorized — including work drawn from sources less likely to be sitting whole in a training set — Pro tries to measure reasoning rather than recall. No public benchmark can promise zero contamination, but the design pushes hard in the honest direction.
- Same brutal grading. It keeps SWE-bench's defining virtue: the patch either passes the real test suite or it does not. There is no rubric to game.
Why lower scores do not mean worse models
This is the part that trips people up, so we will be blunt about it. When a model that scores in the nineties on SWE-bench Verified posts a much lower number on SWE-bench Pro (as reported across 2026), that drop is not the model getting dumber between Tuesday and Wednesday. It is the same model measured against a harder, less-memorizable set of problems. A test that everyone passes tells you nothing about who is best; a test that separates the field is doing its job precisely because the numbers come down.
Think of it the way you would think of any exam. If a class average is 98%, the exam was too easy to rank the students. Rewrite it so the average is 65% and suddenly the ranking is meaningful again — but nobody got worse at the subject overnight. SWE-bench Pro is that harder exam. The absolute numbers are lower and the spread between models is wider, which is exactly the information a saturated benchmark had stopped providing.
How to read Pro scores in the wild
A few habits keep you from being misled. First, never compare a Pro score against a Verified score as though they are the same scale — they are different tests, and a 55 on one is not "worse than" an 88 on the other. Second, treat any specific Pro percentage you see as reported rather than gospel; the harness, the task subset, and the scaffolding all affect the number, and different write-ups measure slightly different things. Third, weigh the gap between models more than any single figure. On a harder benchmark, a consistent lead across many hard tasks is a stronger signal than a hair's-breadth edge on an easy one.
What we track, and why
Our benchmarks board stays on SWE-bench Verified as its repository-work backbone, alongside Terminal-Bench and LiveCodeBench, because Verified remains the most widely reported, most directly comparable number across the whole field — and because we would rather show a consistent, published, cross-vendor scale than chase every new harness before its methodology settles. That is also why our roundups quote Fable 5 at 95.0 and Opus 4.8 at 88.6: those are Verified figures, comparable to every other row on the board. SWE-bench Pro is a signal we read closely and expect to matter more over time, but we do not fold reported numbers from a moving target into a scale meant to be apples-to-apples.
The healthy way to hold both is this: Verified tells you who can do the broad body of real-world engineering tasks, and it is now so widely passed that it functions more as a floor than a ranking. Pro tells you who is still standing when the tasks get genuinely hard — the frontier separator that a saturated benchmark can no longer be. You want both readings, and you want to keep them on their own axes.
Bottom line
SWE-bench Pro exists because success ruined the original as a ranking tool. It is harder, it is designed to resist memorization, and it keeps the pass/fail honesty that made SWE-bench worth trusting. The scores are lower and that is the whole point — a benchmark that separates the field is more useful than one everybody aces. Read Pro figures as reported, compare gaps rather than absolutes, and keep an eye on the official project at swebench.com. For the benchmark our board runs on, start with SWE-bench Verified explained, and for the models topping it, the best open-weight coding models. Live scores are always at /benchmarks.
