Introducing Vibe-Coding Arena

The benchmark ranking doesn't hold up

We graded four frontier models, kept anonymous as A–D, on the same 14 tasks. The headline: an outcome benchmark and the process grade disagree. A model can look competitive on a standard benchmark and still come out the weakest once you grade how it actually works, a gap a pass/fail score can't see.

The rest of this note is where those grades come from: why outcome scoring misses this, how the evaluation is built, a full worked example, and where it goes next.

One bit throws away the trajectory

A test suite reduces a whole build to one bit: does the final state match? It says nothing about how the agent got there: whether it planned, verified, and recovered, or thrashed and claimed a success it could not back up. Yet that is most of what separates a good agent from a bad one: two agents can ship the same working app by completely different routes, and an outcome score calls them equal.

Two agents reach the same finish line, a working result, but by very different routes: one goes straight there in a couple of iterations, while the other loops through dead ends and retries before it lands. An outcome score records only that both finished; the arena grades the route each took to get there.

A test suite also does not produce an observation such as "the right pane is blank for every directory." A human notices it and says so, naming a specific fault the model had not caught (information it did not have), and the next round is then graded on whether the model fixes exactly that. This signal, a correction that carries information without disclosing the intended answer, cannot be reconstructed from an outcome-only score. It is the signal the arena is built to capture.

Classify the space, design the tasks, evaluate the process

The evaluation is built in three steps. First we classify the space of agent coding work into a taxonomy, a map of what agents are actually used for. Then domain experts design tasks that cover that map, each modeled on a real product. Then those tasks are run in the arena, where a human corrects the agent round by round and grades the development process.

2.1 · Classify the space of agent work

We map that space from real usage, not assumptions. We take a large corpus of real user trajectories (actual sessions of people building and debugging with coding agents) and classify each along three axes: what kind of work, in what technical area, in what business domain. Counting those labels across the whole corpus turns real usage into a distribution, so the coverage map reflects what people actually do with agents. An LLM assigns the labels, since the judgment is semantic.

Across the substantive sessions (one-shot lookups aside), here is the real work people bring to a coding agent, by kind of work and technical area:

Percentages are over the ~9,100 sessions left after one-shot lookups are removed; each label is the majority of three LLM runs. The grey other slice combines the catch-all with the categories under ~2%.

2.2 · Design the tasks

Guided by that map, domain experts design the tasks, choosing products that fill out the categories and turning each into something an agent must build. Every task is authored around a real, existing product or specification: an app from the App Store or Product Hunt, or a well-known open-source tool. The expert distills what actually makes that product hard to build: its engineering core (the algorithm, state machine, protocol, or systems work), not its surface UI. Tasks with no real core ("render a static screen") are deliberately excluded; every task has business logic, persistence, networking, concurrency, or a numerical / systems kernel that has to be gotten right.

Each task comes in one of two forms. A multi-round task is decomposed into roughly ten natural engineering milestones: each round adds one independent capability and forces a regression check on everything built before, so the corpus rewards sustained, cumulative work rather than one lucky shot. A spec task hands over a complete specification at once and asks for an end-to-end delivery, with each requirement independently checkable.

Most of the design effort goes into keeping a task honest and hard to game:

• One product per task. If a prototype secretly bundles several independent products (a commit-graph viewer that also needs a full graph-layout engine, say), it is split into separate self-contained tasks rather than crammed into one, so every round is a real increment, not a third of three subsystems at once.
• Every requirement is checkable. Each round carries an explicit acceptance checklist: binary facts (“each commit renders as a node”), things a human can judge by eye (“the branch animation is smooth”), and numeric thresholds only where they come from the task's real need (a 16 kHz sample rate, recall ≥ 90%), never a number invented to look rigorous. Soft, game-able wording (“supports basic editing”, “reasonably fast”) is closed off.
• The known shortcuts are banned up front. On tasks with an objective ground truth the prompt rules them out explicitly: no linking a ready-made library, no hardcoding the reference answer, no happy-path “file exists → pass” assertions, no widening a tolerance to hide a bug.

Difficulty is tuned to a discriminative band: hard enough that a frontier model rarely nails a round on the first try, but not so hard that every model fails, since a task everyone flunks separates nothing. Multi-round tasks are capped at ten rounds, no single round is allowed to balloon into a whole subsystem, and any task that would otherwise depend on real hardware is re-based on a desktop-equivalent product or driven by pre-recorded data, so every task is fully testable in a standard environment.

Coverage by domain

The same corpus by application area: ten broad areas (inner ring) fanning out into 61 sub-domains (outer). Nothing dominates: the largest area is dev-tools / DevOps at ~17%, and no single sub-domain tops ~4%. That breadth is what lets one score stand in for general build-and-debug skill rather than fluency in one niche.

A few representative prototypes:

Two things make a task gradeable rather than merely hard: the task-specific rubric dimensions the expert writes (the per-task criteria to grade against, each with an explicit standard) and its round structure, which turns the build into a process the arena can grade round by round. The real-product prototype plays a different role: it keeps the task grounded and non-trivial.

2.3 · Evaluate the process

That process is the arena itself, where the designed tasks are run, round by round, under live human correction:

A human expert scores the process on three layers:

• Per round — a round score on a −2 / −1 / 0 / +1 scale for the state after the round and, on a fix round, a second fix-effect score on a −2 / −1 / 0 scale rating whether that round moved the requested correction forward (0), spun without progress (−1), or regressed (−2); five binary checks (requirement understanding, context preservation, implementation path, constraint adherence, readiness for the next round); and, in prose, both a rationale for the scores and the round's main shortcomings (the specific defects a fix round is then asked to correct).
• Rubric — four fixed capability dimensions and at least four task-specific dimensions, each with an explicit written standard, assessed once over the whole task.
• Behavior — eight patterns scored across the session: planning and decomposition, exploration, self-verification, accuracy of completion claims, root-cause analysis versus repeated trial, error recovery, knowledge-gap handling, and scope adherence.

What makes an agent good or bad on real software work is largely how it behaves across rounds under correction: whether it recovers, whether it takes in a new observation instead of repeating a failed attempt, whether it stops claiming a success it cannot support. That is exactly what a human in the loop captures, and exactly what the three layers turn into a structured, layered grade, a picture of the whole development process rather than a verdict on its final frame.

One spec task, four models, all three layers

Here is one full task graded across the same four models (A–D). It is a demanding, spec-style task with an objective ground truth: implement a GGUF LLM inference engine from scratch in pure Swift (no linking or FFI to llama.cpp / ggml or any existing GGUF library), with correctness defined as bit-level agreement with the official llama.cpp. It breaks into ten independently-checkable deliverables (A–J), from container parsing to forward-pass numerical parity (the core), each judged against an explicit written standard, with the obvious ways to fake it ruled out in the prompt itself.

Overall result (5-point). All four models eventually delivered all ten items, so the outcome bit is identical. The process score is not:

Binary checks met is the share of the five per-round binary checks (requirement understanding, context preservation, implementation path, constraint adherence, and readiness for the next round) that pass, aggregated over all of a model's rounds. It tracks how sound the work was round to round, independent of the final outcome: all four eventually delivered all ten items, but they cleared very different fractions of these checks getting there.

Layer 1 · per round

Each round has a round score (state after the round); fix rounds also have a fix-effect score (progress vs the requested fix). Model A delivered clean in two rounds, base (0) then fix 1 (+1): forward parity correct on the first pass, self-verified against a live llama.cpp golden. Model B took eight rounds, and the scores show exactly where it stalled:

Layer 2 · rubric (4 fixed + 8 task-specific dimensions, over the whole task)

Each dimension carries an explicit written standard (e.g., forward parity: +1 = cosine ≥ 0.999 and quantized greedy bit-identical, first try; 0 = reached after several fixes; −1 = systematically wrong, cosine < 0.99).

Layer 3 · behavior (8 patterns, across the session)

All four models delivered A–J, so an outcome-only benchmark would score them the same: four passes. The process layers do not: they run from 5/5 down to 2/5. Model A reached the 5/5 in two rounds, verifying honestly against a live llama.cpp golden at each step. Model B needed eight: it printed "ALL PASSED" without ever comparing to the golden, spent a full round changing nothing, and blamed various low-level details (thread-safety, accumulation order) for seven rounds before finding the real cause (RoPE). Model C scored lowest on several counts: its first round would not even compile (−2), four of its six rounds were negative (including one that regressed working functionality), and its self-verification only agreed with its own buggy reference rather than the real golden. Telling a clean two-round delivery from an eight-round thrash, or a real check from one that only agrees with itself, is exactly what the process score captures and a single pass/fail bit cannot.

Toward an automated arena

The arena produces the richest signal here, but it rests on a human expert per trajectory. Two parts of that human have to be automated before the signal can run at the scale agents now operate on: the grading, and the correction. The first is well underway; the second is a genuine open problem.

Grading the process without a human

One route we are building is a controlled benchmark that produces the arena's kind of grade automatically: the process signal, without a human in the loop. In it, the model acts as a technical lead on a production-style investigation: it coordinates helper subagents and long-running background jobs, and the run is scored from the event log (the recorded sequence of its actions) together with the artifacts it produced, checked against a hidden reference (the oracle). The core score is a deterministic rule over the events and artifacts, with no LLM in it, so re-scoring a run reproduces the same number exactly; a small LLM-judge layer adds the qualitative dimensions a rule cannot capture, such as tool-use quality and scope discipline.

The design principle is strict: it scores only the decisions the model controls (how it plans, delegates, packages context for a worker, schedules background work, resolves conflicts, orders dependent steps, verifies, and audits), never the harness's own plumbing. The test is simple: if a metric cannot separate "the model did this well" from "the harness did it automatically," it does not enter the score. And the scoring is relational: it is read from the event log rather than by demanding one canonical trajectory (a dependent step counts only if its event follows the one that unlocked it; conflict resolution requires re-reading the primary evidence after a worker reports rather than trusting it blindly), so many sensible ways of working score well while blind trust, premature starts, and skipped verification are caught precisely. The scorer is built and reproducible; what remains is to calibrate it against the arena's human grades and confirm it separates models the way the arena does.

Automating the correction

Harder than automating the grading is the arena's defining act: the human who notices "the right pane is blank for every directory" and hands that back, round after round. That correction is exactly what a read-once score cannot produce, and reproducing it automatically runs into a structural tension between information and measurement validity. A task-agnostic nudge ("check your work again") is safe but carries no new information: it only asks the model to re-examine what it already has, with the very knowledge that produced the error. In practice such a scaffold lifts success rates a little but cannot point the model at the actual fault. Handing over the reference answer is informative but discloses it, voiding the measurement. A workable automated corrector would have to be all of the following at once, and they pull against one another:

• Information-bearing — it must tell the model something it did not already have; a generic "check again" does not.
• Non-disclosing — it must not reveal the intended answer, or the measurement is void.
• Automatic and cross-round — supplied without a human, adapting as the trajectory unfolds.
• Stable — reproducible enough to rank models, despite an LLM in the loop.

The tension is structural: the more informative a corrective signal is, the closer it drifts to disclosing the answer. One direction is suggested by the human correction itself: "the right pane is blank" is an observation about the environment's state, not about the intended answer, so it carries information the model lacked without revealing what the result should be. It leans on a familiar asymmetry: noticing that something is wrong is often easier than knowing the fix. A blank pane, a validation that never ran, or two of the model's own code paths that disagree can all be surfaced without solving the task. Whether an environment-grounded signal like that can stand in for a human corrector, round after round, is exactly the open problem this line of work points to next.