The Vibe Father
The Artificial Analysis Intelligence Index is a weighted blend of nine evaluations across four categories, and version 4.1 puts more weight on agentic work than on anything else. When you read that Claude Opus 5 scores 61 and Fable 5 scores 60, this is the machinery producing those numbers.
Searches for "artificial analysis" rose sharply alongside the Opus 5 launch, which makes sense — it is the independent source most launch coverage cites, and almost nobody explains what it measures. We have a particular reason to be precise here, the Artificial Analysis indices are the inputs to our own Vibe Coding Index, so if this methodology is wrong, our board is wrong.
The composition
| Category | Weight | Evaluations |
|---|---|---|
| Agents | 34% | GDPval-AA v2 (20%), τ³-Banking (14%) |
| Coding | 24% | Terminal-Bench v2.1 (16%), SciCode (8%) |
| Scientific Reasoning | 24% | HLE (12%), GPQA Diamond (6%), CritPt (6%) |
| General | 18% | AA-Omniscience (12%), AA-LCR (6%) |
The headline design choice is that Agents carries the largest single weight at 34%, with GDPval-AA v2 alone at 20% — the biggest individual component in the whole index. That is a deliberate statement that agentic capability is what a 2026 intelligence measure should emphasise.
It also explains a result that confuses people. A model can lead the Intelligence Index without being the best pure code generator, because raw code generation is only part of a 24% slice. When Opus 5 leads at 61, a meaningful share of that comes from professional knowledge work and banking-agent tasks, not from writing functions.
How the scoring works
Most components use pass@1, the model must produce the correct answer on its first attempt. No best-of-N, no retry credit. That is a stricter and more honest standard than benchmarks that quietly allow multiple samples, and it is why pass@1 numbers look lower than marketing figures you may have seen elsewhere.
GDPval-AA v2 works differently because it has no single correct answer. It uses pairwise comparisons to produce Elo ratings anchored so that human expert performance sits at 1,000, then normalizes with clamp((Elo - 500) / 2000).
That formula is worth sitting with, because it tells you how to read the Elo numbers in launch coverage
| Elo | Normalized | Meaning |
|---|---|---|
| 500 | 0.0 | Floor of the scale |
| 1,000 | 0.25 | Human expert anchor |
| 1,500 | 0.50 | — |
| 1,861 (Opus 5) | ~0.68 | Current top reported |
| 2,500 | 1.0 | Ceiling of the scale |
So Opus 5's widely quoted 1,861 Elo on GDPval-AA v2 is well above the human expert anchor of 1,000 — but on the normalized scale it is around 0.68, not near a ceiling. Elo numbers sound more dramatic than the normalized contribution they actually make.
How precise is it?
Artificial Analysis estimates a 95% confidence interval for the overall Intelligence Index of less than ±1%, based on extensive repeats. Individual component evaluations carry wider intervals.
That number should change how you read the leaderboard. A one-point gap between Opus 5 at 61 and Fable 5 at 60 is at roughly the edge of the stated confidence interval. It is real, it is consistently measured, and it is not a gulf. Anyone describing a one-point Intelligence Index lead as domination has not read the methodology.
| Gap | How to read it |
|---|---|
| Under 1 point | Within measurement noise — treat as a tie |
| 1–2 points | Real but small, will not be visible in daily use |
| 3–5 points | Meaningful, worth evaluating |
| 5+ points | Different capability class |
The Coding and Agentic boards
Alongside the composite, Artificial Analysis publishes standalone capability boards — the Coding Index and the Agentic Index — on the same fixed 0–100 scale. These are the two that matter most for our purposes, and together with Intelligence they are the three dimensions our Vibe Coding Index consumes.
For Claude Opus 5 on launch day those read. Intelligence 61, Coding 78, Agentic 55. Note how differently they scale. A Coding score of 78 and an Agentic score of 55 are not directly comparable to each other as difficulty statements — each board normalizes its own components. Comparing a model's Coding to its Agentic tells you nothing, comparing two models' Coding tells you a great deal.
The nine evaluations, one by one
Knowing what each component actually tests makes the composite far easier to interpret.
| Evaluation | Weight | What it tests |
|---|---|---|
| GDPval-AA v2 | 20% | Real professional knowledge work, scored by pairwise human-anchored Elo |
| τ³-Banking | 14% | Multi-step agent tasks in a banking domain with tools and rules |
| Terminal-Bench v2.1 | 16% | Operating a shell to complete real tasks end to end |
| SciCode | 8% | Scientific programming — code that has to be numerically right |
| HLE | 12% | Extremely hard expert-level questions across disciplines |
| GPQA Diamond | 6% | Graduate-level science questions resistant to search |
| CritPt | 6% | Critical physics-style reasoning |
| AA-Omniscience | 12% | Breadth of factual knowledge, including knowing when it does not know |
| AA-LCR | 6% | Long-context reasoning across large inputs |
A few observations that change how you read a score. Terminal-Bench at 16% is the single largest coding component — meaning the index's idea of "coding" leans toward operating a shell and finishing a task, not toward writing an elegant function. That aligns well with agentic development and less well with, say, algorithmic puzzle solving.
AA-Omniscience at 12% rewards calibration, not just recall. A model that confidently invents an answer is penalised relative to one that declines. For anyone deploying agents, that is arguably more valuable than raw knowledge.
And scientific reasoning carries the same 24% as coding. If you are choosing a model purely for software work, a meaningful quarter of the composite is measuring something adjacent to your use case. That is not a flaw — it is a general intelligence index, not a coding index — but it is a reason to read the standalone Coding board rather than the composite when coding is the job.
The effort-tier problem
This is the single most common misreading of Artificial Analysis data, and it is worth being blunt about.
Modern models expose reasoning-effort settings, and Artificial Analysis publishes results per configuration — you will see entries like "Claude Opus 5 (Adaptive Reasoning, Max Effort)" and "GPT-5.6 Sol (xhigh)" as separate rows. A model's score at max effort is not the score you get at your production default.
| Comparison | Valid? |
|---|---|
| Opus 5 (max) vs GPT-5.6 Sol (max) | Yes — like for like |
| Opus 5 (max) vs GPT-5.6 Sol (xhigh) | Only if you say so explicitly |
| Opus 5 (max) vs your production setup at high | No — different operating point |
| Any score quoted without its tier | Unusable |
When we score a model, we take the tier the board ranks at the top and say which one it is. That is why the Opus 5 scorecard records "adaptive reasoning, max effort" in its evidence metadata rather than just a bare 78.
What the index deliberately does not measure
Artificial Analysis is explicit that the Intelligence Index is a text-only, English-language evaluation suite. Multilingual and multimodal capability are benchmarked separately to keep the measurement consistent.
That is a reasonable design decision with real consequences for buyers
- If you ship in multiple languages, the index says nothing about quality in them. A model can lead here and be mediocre in Japanese.
- If your workload is vision-heavy — screenshots, diagrams, PDFs, UI automation — the index is not measuring it.
- If latency is your constraint, intelligence is orthogonal. The index has no opinion on speed, and reasoning models are slow.
- If your codebase is unusual — an internal DSL, a niche framework, heavy legacy — no public benchmark covers it.
Independent measurement versus vendor reporting
The reason to care about any of this is that vendor-reported and independently measured numbers behave differently, and launch coverage mixes them freely.
| Independent (e.g. Artificial Analysis) | Vendor-reported | |
|---|---|---|
| Who chose the benchmarks | The evaluator, uniformly for all models | The lab, for this launch |
| Harness | One harness across models | The lab's own scaffold |
| Effort tier | Published per configuration | Often unstated |
| Opponent selection | Everyone on the board | Chosen comparisons |
| Reproducible | Methodology is published | Usually not |
Neither is worthless. Vendor numbers frequently arrive first and are usually directionally right — Anthropic reported ARC-AGI-3 at 30.2% and the ARC Prize Foundation independently verified 30.16%, which is the outcome you hope for. But the two categories should never sit in the same table without labels, which is why our scorecards separate measured index dimensions from reported launch claims and only the former feed the ranking.
Why we use it anyway
Given all those caveats, a fair question is why our board depends on it. Three reasons.
It is independent — a third party running its own harness, not a vendor reporting on itself. It is documented, so the weighting and normalization are inspectable rather than a black box. And it is consistent across models, which is the property that makes comparison possible at all, a benchmark that is fair but applied differently per vendor is useless for ranking.
What we do not do is treat it as the whole answer. Our Coding dimension blends Arena Code WebDev human-preference Elo when current votes exist, precisely because static evaluations and human preference disagree in informative ways. And every scorecard shows the source dimensions alongside the composite, so you can disagree with our weighting and read the inputs yourself.
Reading any index responsibly
- Find the weighting. If it is not published, the number is an opinion.
- Check the scoring rule. pass@1 and best-of-N are different sports.
- Check the confidence interval. Gaps inside it are ties.
- Check the effort tier. Untiered numbers are not comparable.
- Check the scope. Text-only and English-only are large exclusions.
- Then test on your own work, because none of it knows your codebase.
Common questions
What is the Artificial Analysis Intelligence Index?
A weighted composite of nine evaluations across Agents (34%), Coding (24%), Scientific Reasoning (24%) and General (18%), reported on a fixed scale so models can be compared consistently.
What does Claude Opus 5 score?
61 on the Intelligence Index at max effort — the top published result at launch — plus 78 on the Coding Index and 55 on the Agentic Index.
Is a one-point lead meaningful?
Barely. The stated 95% confidence interval for the overall index is under ±1%, so a single point sits at the edge of measurement noise. Treat sub-point gaps as ties and reserve real weight for separations of three points or more.
Why does the index weight agents so heavily?
By design — the v4.1 framework emphasises agentic tasks, with GDPval-AA v2 alone carrying 20% of the total, the largest single component in the index.
What does an Elo of 1,861 actually mean?
GDPval-AA v2 anchors human expert performance at 1,000 Elo and normalizes with clamp((Elo - 500) / 2000). So 1,861 sits well above the human anchor but lands around 0.68 on the normalized scale — strong, and not near the ceiling.
Does it measure multilingual or multimodal ability?
No. It is a text-only, English-language suite, those capabilities are benchmarked separately.
What is pass@1 and why does it matter?
The model must be right on its first attempt, with no best-of-N sampling and no retry credit. It is a stricter standard than many published figures use, which is why pass@1 numbers often look lower than marketing charts for the same model.
Why is the Coding Index different from the Intelligence Index?
They are separate boards. Intelligence is a nine-evaluation composite in which coding is a 24% slice, the Coding Index is a standalone capability board. For choosing a model to write software, read the Coding board rather than the composite.
Sources and further reading
- Artificial Analysis, intelligence benchmarking methodology
- Artificial Analysis Coding Index
- Artificial Analysis Agentic Index
- How the Vibe Coding Index is calculated
- How to read every major LLM leaderboard
One last point about why methodology literacy is worth an afternoon. Every model launch produces a wave of confident claims built on numbers whose construction almost nobody checks. Knowing that Agents carries 34%, that scoring is pass@1, that the confidence interval is under a point, and that everything is text-only English turns you from a consumer of those claims into someone who can evaluate them in about thirty seconds. That skill outlasts any individual model, and on the current release cadence it will get used roughly every six weeks. It also changes what you argue about internally. A team that has read the methodology stops debating whether a model is good and starts debating whether the index weighting matches its work — which is a question with an answer, and one that points directly at which standalone board to read instead.
