The Vibe Father
There is no such thing as "the" LLM leaderboard, and the single most common mistake in model selection is treating one board's ranking as a general verdict. "llm leaderboard" went breakout in US searches during the Opus 5 launch, so here is the honest map of the field.
The chart below makes the point better than any argument. It is one model — Claude Opus 5 — across seven boards on launch day.
Same model, same day, scores from 30.2 to 79.2. None of these numbers are comparable to each other, because none of these boards measure the same thing.
If a model can score 78 on one board and 30 on another simultaneously, then "what does it score" is an incomplete question. The complete question is "what does it score on the board that resembles my work."
The field, by what it answers
| Leaderboard | Question it answers | Trust for |
|---|---|---|
| Artificial Analysis indices | How capable across a documented weighted suite? | Cross-model comparison |
| Arena preference Elo | Which output do humans prefer? | Subjective quality, taste |
| SWE-bench (Verified / Pro) | Can it fix real repository issues? | Maintenance work |
| DeepSWE | Can it work at repository scale? | Large multi-file changes |
| Terminal-Bench | Can it operate a shell to completion? | CLI and ops automation |
| Frontier-Bench | Can it persist on long-horizon agent work? | Unattended agents |
| OSWorld | Can it drive a GUI? | Computer use |
| ARC-AGI-3 | Can it model a novel system from scratch? | Hard debugging, reverse engineering |
| GDPval-AA | Can it do professional knowledge work? | Specs, analysis, planning |
| Vendor launch charts | What does the vendor want emphasised? | Finding what to verify |
| Your own ticket suite | Does it work on my codebase? | The actual decision |
Notice that the last row is not useless — it is just a different kind of information. A vendor chart tells you which results the lab is confident about, which is a genuine signal. What it cannot do is settle a comparison, because the vendor chose the benchmarks, the effort tier, and the opponents.
The four failure modes
1. Comparing across effort tiers
Modern models expose reasoning-effort settings, and boards publish per configuration. "Opus 5 scores 78" means at max effort. "GPT-5.6 Sol scores 78" means at xhigh. Comparing either to a competitor's default is not a comparison. A score without its tier is unusable.
2. Ignoring confidence intervals
Artificial Analysis states a 95% confidence interval under ±1% for its Intelligence Index. Opus 5 at 61 versus Fable 5 at 60 is therefore a result at the edge of noise, not a gulf. Treat sub-point gaps as ties.
3. Reading a saturated board
When top scores cluster above 90%, a benchmark has stopped discriminating. Opus 5 posts 97.5% on ARC-AGI-1 and 90.4% on ARC-AGI-2 — neither separates frontier models any more. A launch that leads with a saturated board is telling you something about what it would rather not discuss.
4. Assuming a composite matches your weighting
Every composite embeds an opinion. The Artificial Analysis Intelligence Index puts 34% on Agents. Our own Vibe Coding Index puts 45% on Coding and 35% on Agentic because we are ranking for coding work specifically. Neither is wrong, both are choices. If your work is 90% maintenance on a legacy monolith, neither weighting is yours.
Static benchmarks versus human preference
The deepest split in the field is between benchmarks with a correct answer and boards that measure what people prefer.
| Static benchmarks | Preference Elo | |
|---|---|---|
| Measures | Correctness against a known answer | Which output humans pick |
| Strength | Objective, reproducible | Captures taste and usability |
| Weakness | Saturates, can be trained toward | Style can beat substance |
| Fails when | The test set leaks | Voters reward confident formatting |
They disagree regularly, and the disagreement is informative rather than embarrassing. A model that wins static benchmarks but loses preference Elo is often technically correct and unpleasant to work with. A model that wins preference but trails on correctness is often producing confident, well-formatted output that does not survive a test suite.
This is why our Coding dimension blends Arena Code WebDev human-preference Elo at a fixed weight when current votes exist, rather than picking a side. Either signal alone has a known failure mode.
How to spot a cherry-picked chart in ten seconds
Launch materials are not dishonest so much as selectively framed. A short checklist catches most of it.
| Red flag | What it usually hides |
|---|---|
| No effort tier named | The headline used max, your default will not match |
| Only benchmarks the model wins | A loss exists — find it |
| A saturated board leading the deck | Nothing to say on the discriminating ones |
| Opponents at older versions | The current competitor is closer or ahead |
| Y-axis starting above zero | A small gap drawn as a chasm |
| "Internal benchmark" with no definition | Not independent evidence |
| No confidence interval anywhere | Gaps may be inside the noise |
The useful habit is to look for the loss. Every real model loses something — Opus 5 loses DeepSWE v1.1 to GPT-5.6 Sol, 68.8% to 72.7%, and is substantially behind Mythos 5 on exploit development. A launch page with twelve wins and no losses is not describing a model, it is describing a marketing position.
Benchmarks have a life cycle
Every good benchmark eventually stops working, and knowing where one sits in its life cycle is most of reading it correctly.
| Stage | Signs | How to use it |
|---|---|---|
| New | Scores near zero, wide spread | Watch — it may be measuring noise |
| Discriminating | Clear separation, movement between releases | This is where the signal is |
| Maturing | Top cluster tightening | Still useful, read gaps carefully |
| Saturated | Leaders above 90%, differences under a point | Retire it from decisions |
ARC-AGI-1 at 97.5% and ARC-AGI-2 at 90.4% are saturated. ARC-AGI-3, where the best verified result is about 30% against a human baseline of 100%, is wide open. SWE-bench Verified at 96% is nearly done as a discriminator, which is precisely why SWE-bench Pro — where Opus 5 posts 79.2% — exists.
When a lab moves its headline from one benchmark generation to the next, that is the life cycle working, not goalpost-shifting. When a lab keeps quoting the saturated one, that is worth noticing.
The one leaderboard that actually decides
Yours. Twenty real tickets from your own backlog, run through each candidate model in your own harness, scored on the things that pay you.
| Measure | Why it beats a public board |
|---|---|
| Accepted diff rate | Your reviewers, your standards |
| Cost per accepted change | Includes retries, which benchmarks hide |
| Repair rate within 7 days | Catches fluent-but-wrong output |
| Wall-clock to merge | Latency is a real cost |
| Escalation rate to a human | Measures autonomy honestly |
Public leaderboards are for deciding what to test. They narrow a field of forty models to three candidates, which is genuinely valuable and is the job they are good at. They cannot tell you which of the three works on your conventions, your framework versions, and your definition of clean.
A practical reading routine
- Identify the shape of your work. Maintenance, greenfield, agentic, ops, analysis — they map to different boards.
- Pick two boards that match it. One static, one preference-based, if both exist.
- Read the methodology once. Weighting, scoring rule, confidence interval, scope.
- Note the effort tier on every number you write down.
- Shortlist three models, not one.
- Run your twenty tickets. This is the step that decides.
- Re-check quarterly, or on a launch that moves your shortlist.
Step three is the one people skip, and it takes about fifteen minutes per board. It is also what converts a leaderboard from a scoreboard you react to into an instrument you can use.
Building your own internal leaderboard
The twenty-ticket evaluation is worth turning into a standing asset rather than a one-off. Teams that do this stop re-arguing model choice every launch.
| Component | What it looks like |
|---|---|
| Fixed task set | 20–40 real tickets, frozen, spanning bug fix / feature / refactor / migration |
| Fixed harness | Same prompts, same tools, same retry cap for every candidate |
| Automatic scoring | Tests pass or they do not — no human vibes in the primary metric |
| Human review sample | A subset read by a senior engineer for style and safety |
| Cost capture | Tokens in and out per task, including retries |
| Version pinning | Record the exact model revision and effort tier |
Freezing the task set is what makes it a leaderboard rather than an anecdote. If the tasks change between runs, you cannot attribute a score change to the model. Refresh the set deliberately — perhaps quarterly — and treat that refresh as a version bump that resets comparisons.
The payoff arrives on launch day. Instead of a week of debate, someone runs the suite and posts three numbers, accepted rate, cost per accepted change, and repair rate. That is a decision, and it takes an afternoon.
Where our board fits
The Vibe Coding Index is a coding-weighted composite. 45% Coding, 35% Agentic, 20% Intelligence, built from published Artificial Analysis dimensions with Arena Code WebDev blended into Coding when fresh votes exist.
We publish the source dimensions next to the composite specifically so you can disagree with our weighting. If your work is more research than implementation, read the Intelligence column and ignore our headline. If you run unattended agents, weight Agentic higher than we do. A composite you cannot decompose is a composite you should not trust — including ours.
We also leave cells empty rather than estimating them. An em-dash means nobody has published a comparable measurement, which is real information, filling it with a guess would quietly convert an unknown into a ranking.
Common questions
Which LLM leaderboard is the most accurate?
None in general. Each measures something different — the accurate one is whichever resembles your workload. For cross-model comparison with published methodology, Artificial Analysis is the strongest starting point.
Why do leaderboards disagree?
Because they measure different capabilities, at different effort tiers, with different scoring rules. Disagreement between a static benchmark and preference Elo is usually informative rather than an error.
Can benchmarks be gamed?
Static benchmarks can be trained toward, which is why private and semi-private test sets like ARC-AGI-3's exist. Preference boards can be gamed by confident formatting. Diversify your sources.
Which leaderboard should a coding team start with?
The standalone Coding and Agentic boards rather than a general intelligence composite, then a preference board such as Arena Code WebDev for the taste dimension. Use both to build a shortlist of three, and let your own ticket suite pick the winner.
What is a meaningful score gap?
Depends on the confidence interval, which is why you read the methodology first. On the Artificial Analysis Intelligence Index, under a point is a tie and three to five points is worth evaluating. On a preference board, check the vote count and uncertainty before reading any gap at all — preliminary entries move.
Should I switch models when a leaderboard changes?
No. Use it to decide what to test, then let your own twenty tickets decide whether to switch.
How do I know when a benchmark is saturated?
When the leaders cluster above roughly 90% and the differences between them fall inside the confidence interval. At that point the board has stopped separating models and should come out of your decision process.
Are vendor benchmark charts worth reading?
Yes, as a map of what to verify rather than as evidence. They tell you which results a lab is confident about. They cannot settle a comparison, because the lab picked the benchmarks, the effort tier and the opponents.
Sources and further reading
- Artificial Analysis, benchmarking methodology
- ARC Prize. ARC-AGI-3
- The Intelligence Index, explained
- ARC-AGI-3 explained
- Vibe Coding Index methodology
The uncomfortable truth about this whole category is that leaderboards are most useful to people who least need them. If you have twenty tickets, a harness and a habit of measuring cost per accepted change, a leaderboard saves you a week of evaluation by narrowing the field. If you do not have those things, a leaderboard becomes a substitute for judgment — and the model at the top of a board you have not read the methodology for is not a decision, it is a guess with a number attached. The fix is not to distrust leaderboards. It is to use them for the job they are genuinely good at — narrowing forty options to three — and then to do the unglamorous work of running those three against tasks that actually resemble yours.
