The Vibe Father
Reality check
We run a public benchmark board, so this may read as a strange thing to admit: most AI coding leaderboards deserve more skepticism than they get. Not because the people making them are dishonest — usually they aren't — but because the ways a benchmark can quietly mislead you are subtle, structural, and easy to miss if you take the top number at face value. So here's the honest version from people who publish scores for a living: how leaderboards mislead, which signals resist gaming, and how to read a board critically instead of just reading the winner off the top.
Three ways a benchmark misleads you
Contamination. The oldest problem and still the deadliest. If a benchmark's problems — or things very close to them — were in a model's training data, a high score measures memory, not ability. Popular public benchmarks are especially exposed: the more famous the test, the more likely its problems and solutions are scattered across the web the model trained on. A model can post an impressive number by having effectively seen the answer key, and nothing about the score tells you that happened. When a brand-new model debuts near the top of an old, well-known benchmark, contamination is a hypothesis you should hold, not dismiss.
Cherry-picked configurations. A single model can produce wildly different scores depending on the scaffold around it — the agent framework, the prompt, the number of allowed attempts, the tools it's given, how much compute it's allowed to spend. Report the best configuration and you get a headline number that no one running the model in a normal setup will ever reproduce. The score isn't fabricated; it's just measured under conditions engineered to flatter it. Two "SWE-bench" numbers for the same model can differ by a lot purely because one ran with a richer harness, and the leaderboard rarely shows you that context.
Vendor-run evaluations. When a lab measures and publishes its own model's score, the incentives are obvious. It's not that vendors lie — it's that they choose the benchmarks, the configs, and the framing that make their model look strongest, and they're not obligated to publish the ones that don't. A vendor benchmark is marketing that happens to contain a true number. Useful, but read it as a claim to verify, not a fact to accept.
Signals that resist gaming
The good news is that contamination and cherry-picking leave fingerprints, and some benchmarks are built specifically to be hard to game.
Contamination-resistant collection. The cleanest defense is measuring on problems the model couldn't have seen. LiveCodeBench does exactly this: it continuously collects contest problems published after models' training cutoffs, so a high score can't be memorization — the problems didn't exist when the model was trained. That's a fundamentally more trustworthy signal on a brand-new model than any static, famous benchmark, and it's why we weight it in our own index. We break down the mechanism in our LiveCodeBench explainer.
Harder, fresher variants. As models saturate a benchmark, the benchmark stops discriminating — everyone scores high and the ranking goes noisy. The response is tougher variants. SWE-bench Pro is getting attention as a reportedly harder take on real-repository issue-solving, designed to pull the ceiling back up so top models can be told apart again. We frame it as a concept worth watching and mark any specific number as reported rather than treating it as settled; the point is the pattern — when a benchmark gets easy, trust the harder successor more. More in what is SWE-bench Pro.
Multiple suites that cheat-proof each other. No single benchmark is trustworthy alone. A model that has to score well on real repository work, on driving a live shell, and on fresh un-memorizable problems simultaneously has fewer places to hide, because the three measure different skills and a contamination or scaffold trick that inflates one rarely inflates all three.
How we try to earn the trust
Our board tracks a weighted composite of three suites: SWE-bench Verified at 40%, Terminal-Bench at 30%, and LiveCodeBench at 30%. The mix is deliberate — real repository patches, agentic shell operation, and contamination-resistant fresh reasoning triangulate each other, so a model has to be genuinely good to top all three. Two rules keep us honest: every score is cited to its public source with a link, so you can verify whose measurement you're looking at; and missing scores are never invented or counted as zero — when a suite hasn't been published for a model, we renormalize across the suites that exist and mark the gap plainly. The full reasoning is in our index methodology.
How to read any leaderboard critically
You don't need to run benchmarks yourself to read them well. A short checklist keeps you from being fooled:
- Who ran it? Vendor-published scores are claims. Independent, third-party evaluations carry more weight. If you can't tell who measured it, that's a red flag.
- Is it cited? A number without a linked source is a number you can't verify. Boards that hide their sources are asking for trust they haven't earned.
- Could the model have seen it? On old, famous, static benchmarks, assume some contamination risk — especially for models that debuted after the benchmark went public. Prefer post-cutoff signals for new releases.
- What configuration? If the board doesn't say how many attempts, what scaffold, and what compute, the headline number is missing the context that determines whether you'll ever reproduce it.
- Does it agree with itself? Cross-check across independent suites. A model that's #1 on one board and mid-pack on two others is telling you the #1 was probably a favorable configuration or a lucky benchmark, not a general truth.
The honest bottom line
Benchmarks are the best shortlist generator we have, and they're worth using — but a leaderboard is evidence, not a verdict. The trustworthy way to use one is to treat the top few models as a shortlist, check who measured the scores and whether they're cited, lean on contamination-resistant and harder-variant signals for anything new, and then test the shortlist on your own code, on your own stack, in your own workflow. The number that predicts your outcome is the one you generate yourself; the leaderboard just tells you which models are worth the test. That's exactly how we intend ours to be used, and every score on it lives, linked, at /benchmarks.
