The Vibe Father
First look
xAI shipped Grok 4.5 yesterday, and for the first time we're treating a Grok release as a serious coding-model launch rather than a curiosity. The headline: 87.4 on LiveCodeBench, which puts it squarely in the frontier pack on contest-style coding, at $2 per million input tokens and $6 per million output — pricing that undercuts most of the models it's now standing next to. Add a 500k context window and 91 tokens per second, and xAI has finally built something coders should actually evaluate instead of politely ignore.
That's the real part. Now the honest part: xAI has published no SWE-bench Verified and no Terminal-Bench results. And that gap is the entire story of this first look.
What one benchmark can and cannot tell you
The three benchmarks we weight on our leaderboard measure genuinely different skills, and conflating them is how people get burned:
- LiveCodeBench measures contest-style, algorithmic problem solving — self-contained problems with clean specs and checkable answers. It's a real signal for raw coding intelligence.
- SWE-bench Verified measures whether a model can resolve actual GitHub issues in real repositories — navigate unfamiliar code, understand context it didn't write, make a change that fixes the bug without breaking the neighborhood.
- Terminal-Bench measures agentic shell work — can the model operate a real terminal over multiple steps to accomplish a task, recovering when commands fail.
History says these skills correlate but do not travel together. Plenty of models crush self-contained problems and then flounder the moment a task requires understanding a 200-file codebase or running a stateful, multi-step session in a shell. That's why our index weights SWE-bench Verified at 40% — repo work is the job — with Terminal-Bench and LiveCodeBench at 30% each.
So the precise, honest claim about Grok 4.5 today is: it's a frontier-class contest coder at a bargain price, and we do not yet know if it can do real repo work. Not "it can't" — we don't know. There's a difference, and until xAI or independent evaluators publish SWE-bench Verified and Terminal-Bench numbers, anyone claiming to know is guessing.
Where it lands in the pack
Here's how the LiveCodeBench score and pricing stack up against the models around it on our board:
| Model | LiveCodeBench | $ in / M | $ out / M |
|---|---|---|---|
| Claude Fable 5 | 89.8 | 10.00 | 50.00 |
| Claude Opus 4.8 | 87.8 | 5.00 | 25.00 |
| Gemini 3.5 Flash | 87.6 | 1.50 | 9.00 |
| DeepSeek V4 Pro | 87.5 | 0.435 | 0.87 |
| Grok 4.5 | 87.4 | 2.00 | 6.00 |
| GPT-5.3 Codex | 87.3 | 1.75 | 14.00 |
| GPT-5.5 | 85.3 | — | — |
Two things jump out. First, the pack is tight — five models within half a point of each other means LiveCodeBench alone can't separate them, which is exactly why the missing benchmarks matter so much. Second, the pricing is genuinely aggressive: Grok 4.5's output tokens cost about a quarter of Claude Opus 4.8's $25. If — big if — its repo-work numbers come in anywhere near its contest numbers, that price makes it one of the most interesting value plays on the board. DeepSeek V4 Pro still owns the absolute floor on price, but Grok is undercutting the Western flagships hard.
How we're actually testing it this week
We don't wait for labs to publish numbers before forming a working opinion — we slot new models into low-risk seats and let real work grade them. Here's the plan for Grok 4.5, and it's a template you can steal for any new release:
- Reviewer seat first. Reviewing is the perfect audition: it's cheap (mostly input tokens, and Grok's $2/M input is friendly), it's low-risk (a bad review wastes minutes, a bad merge wastes days), and it stress-tests exactly the skill LiveCodeBench doesn't — reading unfamiliar code in context and reasoning about what's wrong with it. The 500k context window means it can hold a serious diff plus the surrounding files. If its review notes are sharp, that's early evidence the repo-comprehension gap might not be a gap.
- Then Builder, on a branch, behind a verification gate. If it survives the reviewer audition, it gets a real ticket — on an isolated branch, with the gate running our actual build and our actual test suite before anything is called done. No self-reported success accepted. This is where "contest coder" and "repo worker" definitively part ways, and the gate catches the difference before it costs anything.
- Track the boring numbers. Tokens per completed task, retries needed, files touched versus files that needed touching. A model that's 30% cheaper per token but needs two extra rounds isn't cheaper.
That whole audition costs an afternoon and a few dollars precisely because our workflow isn't married to any one provider — adding Grok 4.5 is a config entry, not a migration. When you can trial a model in a contained seat with independent verification, "should we adopt this?" stops being a debate and becomes a measurement.
The early verdict
Grok 4.5 is real. The LiveCodeBench score earns it a place in the frontier conversation, the pricing earns it a place in the value conversation, and the 500k context is enough for serious work. What it hasn't earned yet is trust with your repository — because nobody has published the evidence, and we won't pretend otherwise. xAI getting serious about code is good for everyone; competition at $6-per-million output puts pressure exactly where we like seeing it.
The moment SWE-bench Verified or Terminal-Bench numbers exist for Grok 4.5, they go on the leaderboard and this post gets a follow-up with a real verdict. Until then: audition it in the cheap seats, gate everything, and don't let a tight LiveCodeBench pack convince you the race is decided. The benchmark that matters most is the one nobody's published.