The Vibe Father
ARC-AGI-3 is the interactive benchmark where the best frontier model scored 0.37% in March 2026. On July 24, 2026, the ARC Prize Foundation verified Claude Opus 5 at 30.16% on the semi-private evaluation. That is why "arc agi 3" went breakout in US searches alongside the Opus 5 launch.
A jump of that size is either a genuine capability shift or a measurement artefact, and it is worth understanding which. This explains what ARC-AGI-3 actually tests, how its unusual scoring works, what the 30% means in context, and — since they trended in the same window — what "deepswe" and "frontier bench" are too.
Related every Opus 5 benchmark explained and how to rank coding models without fooling yourself.
What ARC-AGI-3 actually is
ARC-AGI-3 is the first fully interactive benchmark in the ARC-AGI series. Instead of static puzzle grids, it presents novel, abstract, turn-based game environments handcrafted by human game designers.
The defining constraint there are no instructions, no rules, and no stated goals. An agent has to explore the environment, work out how it functions, discover what winning even looks like, and carry that understanding forward as later levels introduce new mechanics.
| Property | Detail |
|---|---|
| Format | Interactive, turn-based game environments |
| Public games | 3, released for development |
| Private games | 3, used for final leaderboard rankings |
| Levels per game | 8 to 10, each adding new mechanics |
| Instructions given | None |
| Human performance | 100% of environments solved |
The public/private split is the part that makes the benchmark credible. You can develop against three games, your ranking comes from three you have never seen. That structure makes it materially harder to overfit than a benchmark whose full test set is on the internet.
The scoring is stranger than the format
ARC-AGI-3 does not simply score pass or fail. Evaluation rests on Relative Human Action Efficiency (RHAE), which compares an agent's action count against the second-best first-run human completion for each level.
Scores are quadratic in inefficiency. Using twice as many actions as the human reference yields roughly 1% credit. All levels cap at five times the human action count.
| Agent behaviour | Roughly what it earns |
|---|---|
| Matches human action efficiency | Near full credit |
| Uses 2x the human actions | About 1% credit |
| Uses 5x or more | Capped out |
| Solves the level by brute force | Almost no credit |
This is the crucial detail for interpreting any ARC-AGI-3 number. A model cannot grind its way to a good score by trying everything. Because the penalty is quadratic, an agent that flails and eventually succeeds scores close to zero. The benchmark is measuring whether the model understood the environment, not whether it eventually stumbled into the exit.
It also explains why early scores were so brutal. Scoring 0.25% does not mean a model solved 0.25% of levels — it largely means it solved some levels appallingly inefficiently.
The progression, in context
| Model | Score | When |
|---|---|---|
| Gemini 3.1 Pro | 0.37% | March 2026, at launch |
| GPT-5.4 | 0.26% | March 2026 |
| Claude Opus 4.6 | 0.25% | March 2026 |
| Grok 4.2 | 0.00% | March 2026 |
| GPT-5.6 Sol (max effort) | 7.8% | Before the Opus 5 launch |
| Claude Opus 5 | 30.16% verified at high effort | July 24, 2026 |
| Humans | 100% | — |
Two details worth pausing on. First, the ARC Prize Foundation verified 30.16% on the semi-private evaluation at high effort — not max. Anthropic's own reported figure is 30.2%. Those agree, which is the outcome you want when a lab claim meets an independent verifier.
Second, the progression was not a single leap from 0.37% to 30%. GPT-5.6 Sol had already moved the bar to 7.8%. Opus 5 roughly quadrupled that. The 80x framing against March numbers is technically true and slightly misleading, the honest comparison is against the standing best, and that is a ~4x improvement.
Does this predict anything useful for coding?
Partly — and this is where launch coverage tends to overreach.
| Work type | Does ARC-AGI-3 predict it? |
|---|---|
| Debugging a genuinely novel failure | Reasonably well — no memorized template exists |
| Reverse-engineering an undocumented system | Well — that is close to the task shape |
| Long-horizon agent work with feedback | Moderately — the loop is similar |
| Writing another CRUD endpoint | Not at all — a much cheaper model is fine |
| Large multi-file refactors | Poorly — see DeepSWE instead |
The honest summary. ARC-AGI-3 measures the ability to build a working model of an unfamiliar system from observation alone. That is genuinely what senior debugging feels like, and it is genuinely not what most day-to-day software work is. A 30% score is a strong reason to reach for Opus 5 when you are stuck. It is a poor reason to route your entire ticket queue through the most expensive model you can find.
It is also worth keeping the ceiling in view. Humans solve 100% of these environments. A model at 30% is a large improvement over 0.37% and still nowhere near a competent person. Anyone citing this result as evidence of imminent general intelligence is reading a different chart than the one published.
Why ARC-AGI-1 and ARC-AGI-2 stopped being the interesting ones
The ARC series has repeatedly had to redesign itself as models caught up. Opus 5 posts 97.5% on ARC-AGI-1 and 90.4% on ARC-AGI-2 — numbers high enough that neither board separates frontier models usefully any more.
| Benchmark | Format | Opus 5 | Still discriminating? |
|---|---|---|---|
| ARC-AGI-1 | Static grid puzzles | 97.5% | No — saturated |
| ARC-AGI-2 | Harder static puzzles | 90.4% | Barely |
| ARC-AGI-3 | Interactive environments, no instructions | 30.16% | Yes — wide open |
This is the normal life cycle of a good benchmark, and it is why the third version changed format rather than just adding harder puzzles. Static puzzles can be pattern-matched once enough examples exist in training data. An interactive environment with no stated goal cannot be — you have to actually probe it and update on what you observe.
The lesson generalises beyond ARC. When a benchmark's top scores cluster above 90%, it has stopped being a measurement and become a formality. Watch which benchmarks a lab chooses to report, a release that leads with a saturated board is telling you something.
The competition angle
ARC-AGI-3 is also the basis of the ARC Prize 2026 competition, which is part of why the term spikes whenever a model posts a result. The prize structure matters for interpretation, it creates an incentive for purpose-built agent scaffolds that are optimised specifically for these environments.
That is worth separating from a general-purpose model result. Opus 5's 30.16% is a frontier model evaluated on the semi-private set, not a bespoke solver built to win a competition. Those two categories of result will increasingly diverge, and comparing across them is a mistake — a specialised scaffold beating a general model on ARC-AGI-3 says very little about which you should point at your codebase.
Frontier-Bench, DeepSWE and the rest of the trending benchmark names
Two other benchmark terms went breakout in the same window, so here is the quick orientation.
| Benchmark | What it measures | Opus 5 result |
|---|---|---|
| Frontier-Bench v0.1 | Long-horizon agentic coding | 43.3% — leads (Fable 5 33.7%, GPT-5.6 Sol 34.4%) |
| DeepSWE v1.1 | Repo-scale software engineering | 68.8% — loses to GPT-5.6 Sol at 72.7% |
| SWE-bench Pro | Harder, less-saturated repo tasks | 79.2% — leads (GPT-5.6 Sol 64.6%) |
| OSWorld 2.0 | Computer use and GUI automation | 70.6% — leads (GPT-5.6 Sol 62.6%) |
| GDPval-AA v2 | Professional knowledge work | 1,861 Elo — leads by over 100 |
DeepSWE is the one to weigh if your work is maintenance on a large existing codebase, because it is the benchmark most directly aimed at that shape of task — and Opus 5 loses it. Frontier-Bench is the better proxy if you run agents that work unattended for a long time. The two disagreeing is useful information rather than a contradiction, they measure different jobs, and a model can be excellent at open-ended exploration while trailing on disciplined repository-scale edits. Any single ranking that folds both into one number has thrown that distinction away, which is exactly why we publish the source dimensions alongside the composite instead of only the composite.
If you came here from a search for "llm leaderboard," the practical version of that question is not one ranking but three, a capability board, a price axis, and your own evaluation on your own repository. Our Vibe Coding Index handles the first two and is deliberately transparent about being unable to do the third.
What a genuinely high score would require
It is worth spelling out what the remaining 70 points represent, because "30%" sounds closer to solved than it is.
To score well under RHAE, an agent has to reach the goal in roughly the number of actions a competent human needed on their first run — with no instructions, in an environment it has never seen. That means forming a hypothesis about the mechanics early, testing it cheaply, and committing. Exploration that a human does in eight moves and an agent does in twenty scores close to nothing even though both "solved" it.
| Capability required | Why it is hard for current models |
|---|---|
| Cheap hypothesis testing | Models tend to explore exhaustively rather than selectively |
| Carrying learning across levels | New mechanics arrive, prior rules partly still apply |
| Knowing when to stop exploring | Every extra action is quadratically penalised |
| Inferring the goal at all | Nothing states what winning looks like |
That third row is the interesting one for anyone building agents. The benchmark penalises exactly the behaviour that makes agents expensive in production — thorough, unfocused exploration. A model that scores well on ARC-AGI-3 is a model that decides quickly and correctly, which is also the model that costs less to run.
How to use benchmark news like this without getting played
- Check who verified it. A lab-reported number confirmed by the benchmark's own foundation is much stronger evidence than a slide.
- Check the effort configuration. 30.16% was verified at high effort. Comparing that to another model's default is meaningless.
- Check the metric shape. RHAE is efficiency-weighted and quadratic, it is not a pass rate, and reading it as one will mislead you.
- Check the standing best, not the oldest number. 0.37% to 30% sounds like 80x. Against the actual previous leader it is about 4x.
- Check the human baseline. 100% here. The gap is still enormous.
- Then ignore all of it and run twenty of your own tickets, because none of the above knows your codebase.
Common questions
What is ARC-AGI-3?
An interactive benchmark of novel turn-based game environments with no instructions, rules or stated goals. Agents must explore, infer the goal, model the dynamics and plan efficiently across 8 to 10 levels per game.
What did Claude Opus 5 score on ARC-AGI-3?
The ARC Prize Foundation verified 30.16% on the semi-private evaluation at high effort. Anthropic reported 30.2%.
Why were earlier scores under 1%?
Because scoring is efficiency-based and quadratic. Models could sometimes complete levels but used far more actions than a human, which earns almost no credit.
Is ARC-AGI-3 a coding benchmark?
No. It measures novel problem-solving and environment modelling. It correlates with hard debugging and reverse-engineering, not with routine implementation work.
Has any model beaten humans on it?
No. Humans solve 100% of the environments. The best verified model result is about 30%.
Why does the same model report different ARC-AGI-3 numbers?
Because effort configuration changes the result. The ARC Prize Foundation verified 30.16% at high effort on the semi-private set, Anthropic reported 30.2%. Those agree. A number quoted without its effort tier is not comparable to one that names it.
What is the difference between the public and private games?
Three games are public for development and three are private and used for leaderboard ranking. The split is what stops a strong score being explained by having seen the environment during training.
Sources and further reading
- ARC Prize. Announcing ARC-AGI-3
- arXiv 2603.24621. ARC-AGI-3, a new challenge for frontier agentic intelligence
- DataCamp, the interactive reasoning benchmark
- Vellum. Claude Opus 5 benchmarks
- Vibe Coding Index leaderboard
The reason ARC-AGI-3 is worth following, more than most benchmarks, is that it was built to resist the thing that makes benchmarks stale. Private evaluation games, no instructions, and an efficiency-weighted score together make memorisation nearly worthless. When a number on a benchmark like that moves 4x in four months, it is much more likely to reflect a real capability change than a training-data artefact — which is exactly why the result is interesting, and exactly why the remaining 70-point gap to human performance is the honest headline.
