A small model with a review loop matches a larger one at 58% of the cost.

What the loop is for

Coding models are trained to finish fast and sound right, which makes them confident by default. Ask one to build a program and it comes back with something that compiles and a note saying it's done, from the same model that did the work. In the runs below, 100 Luna sessions that had been told to check their own work returned after 4 to 20 minutes with about half of the hidden tests passing. Every one of them stopped on its own, well short of the six hours it had.

A review loop splits the two jobs. A builder works until it believes the task is done; then a checker, in a fresh session that hasn't seen the builder's reasoning, tries to break the result, and if it finds problems the builder gets the list and goes again. The checker is the same model with the same task statement; it just starts without the builder's transcript.

ProgramBench is a good place to measure whether that helps. Each task gives an agent a compiled program (svgbob, xz, samtools, direnv, 16 more) and its documentation, offline, and asks for a reimplementation from observed behaviour alone. Hidden tests then score the rebuild: 262 to 1,425 of them per task here. It's a fair stand-in for everyday work, and on our 20 tasks no published run passes every test; the best passes between 38% and 99%.

What we ran

We ran the loop as a zeroshot graph with two nodes, a builder and a checker; zeroshot is our open-source runner for graphs of coding agents, and this is about the simplest graph it can run. One builder turn is a build, one checker turn a check, and the pair is a round. The builder keeps its session across rounds; every check starts fresh with only the task statement and the workspace. A rejection sends the checker's list of discrepancies back as the builder's feedback, and the loop ends when the checker accepts, after 50 rounds, or at six hours.

We wrote two prompts and froze them before the first run; neither was tuned to any task. Everything else the model sees is ProgramBench's own task statement and the runner's standard instructions for a builder and a checker.

builder

Complete the task described in the input. Keep working and verifying your own work until you believe the result fully satisfies the task. If review feedback is provided, address every point without breaking what already works.

checker

Independently judge whether the current workspace satisfies the task. Derive your own checks from the task statement and whatever it makes available; do not rely on the builder's tests, notes, or claims. Run the checks. Accept only with concrete evidence that nothing is wrong. Otherwise reject and list the most important discrepancies, each with a minimal reproduction.

Luna and Sol are two models in the GPT-5.6 line, Luna the small one and Sol the large one; we ran both through OpenAI's Codex CLI at its xhigh reasoning-effort setting. Luna costs $0.20 per million input tokens and $1.20 per million output tokens, Sol $4 and $20. Sol ran each task as a single shot (one build, no checker) five times. Luna's loop ran five times per task.

1.The loop reaches Sol's score at 58% of Sol's cost

Why this comparison. If you're paying for the work, the question isn't which model scores higher but what a given level of quality costs. Luna costs about a twentieth of Sol per token, and the loop spends tokens on checks and repairs that a single shot never makes, so a loop's bill depends on how many rounds it takes. We asked, per task, what the loop had to spend to reach the score Sol's single shot gets there. The loop's five runs are scored build by build; the cost to match is the smallest budget at which the mean of their last builds within that budget reaches Sol's mean score, and we report it as a share of what Sol's single shot cost on the same task.

What we expected. If the checker catches bugs the builder missed, the loop should reach Sol's score before it has spent what Sol spends, with a wide margin on tasks where a cheap first build already lands close. If the small model can't do a task, no budget gets there. We expected the loop to come in cheaper on most tasks, not all: where a cheap first build lands far from the mark, 50 rounds of a small model may never close the gap.

What we found. On 15 of the 20 tasks the loop reached Sol's mean score for less than Sol's single shot cost, and across the 20 tasks the median is 58% of Sol's spend.¹ On 3 tasks it needed more than Sol spent (130% to 193%), and on 2 (datasurgeon, xz) it hadn't reached Sol's score by its last build. The figure below shows each task.

Fixing the cost instead of the score gives the same picture. With each loop scored at the last build that fits inside Sol's mean cost on the task, 11 to 50 rounds in, the loop scored higher on 15 of the 20 tasks.² On 7 of them (ditaa, parqeye, dstask, direnv, calcurse, gittype, gdu) the 95% intervals of the two means don't overlap; on the 5 tasks Sol leads, they do.³ Averaged over the tasks, the loop passes 69.5% of the hidden tests to Sol's 65.6%. What the loop does pay is wall-clock time: a median 69 minutes to reach Sol's score and 2.3 hours to spend Sol's budget, against 11 minutes for Sol's single shot (medians of the per-task averages).

2.A single session given both roles stops after one build

Why this comparison. The cheapest explanation of the loop's lead is that the checker's instructions are good, and that any session told to build and then check would do as well. If that were true, the separate checker session would add little beyond the prompt.

What we expected. Two outcomes seemed possible. Either a single session given both roles keeps going for about as long as the loop does and lands near the same score, or it judges its own work fine and stops early. We leaned toward the second: the builder prompt already tells the session to verify its own work before it stops, and a second instruction to do the same thing seemed unlikely to change when it stops.

What we ran. We gave Luna the builder's and the checker's text above, verbatim, plus the runner's standard checker instructions and four sentences of glue; we call the whole thing the role prompt:

glue

Work in rounds. In each round, act first as the builder and then as the checker, following the instructions for each role below. If the checker rejects, its list of discrepancies is the review feedback for the builder in the next round. Repeat until the checker accepts, then return.

The model, the task statement, the six-hour limit and the five runs per task were the loop's; the only change is one session instead of two.

What we found. Luna didn't loop. All 100 sessions returned on their own after 4 to 20 minutes (median 9), at about the price of one build. They scored 50.5% on average, 2 points below Luna's plain single shot (the loop's first build). At that spend the loop itself has finished exactly one build in all but one of the runs in the comparison,⁴ so this is a comparison of two prompts for one build, and the plain one wins it 14 tasks to 6. Everything the loop gains over a single shot comes from the extra rounds: 69.5% at Sol's budget, 71.1% after all its rounds.⁵

3.The loop is more reliable

Why this comparison. If you ship agent-built software, the worst run matters as much as the average one. A single shot is one attempt, and attempts vary: the model commits to a design in its first minutes and may misread a flag along the way. A checker should catch the worst of that.

What we expected. The loop's runs should land closer to each other than any single shot's, and Sol's single shots, from the stronger model, should vary less than Luna's.

What we found. The figure below pools the 20 tasks, each run plotted as its score minus its arm's mean on that task; an arm is one of the four setups (Sol's single shot, Luna's single shot, the role-prompt sessions, the loop at Sol's budget). Scored at Sol's budget, 86% of the loop's runs land within 5 points of their task's mean; so do 72% of Sol's single shots, 65% of Luna's single shots and 62% of the role-prompt sessions.⁶

The loop's lead over Sol is in the tails. Four of Sol's 100 runs ended more than 15 points from their task's mean; the loop had one such run in 100, on the high side.⁷ Leave those out and the two arms are equally steady: 85% of Sol's remaining runs land within 5 points, 87% of the loop's. What the loop never did was fail badly: no loop run fell more than 15 points below its task's mean, where Sol's single shot had three such runs, Luna's two and the role prompt's two.

Conclusion

The loop we measured is the simplest thing that can be called a review loop: two nodes, two generic prompts that never changed, and a checker whose only advantage over the builder is a fresh start. On these tasks that was enough for a small model to reach a large model's score at a median 58% of its cost, and to do it without the badly failed runs that every single-shot arm produced. The trade is wall-clock time: reaching that score took the loop about six times as long as Sol's single shot, which matters for some uses and not for others. A prompt asking one session to do both jobs didn't produce the same effect, because the session stopped when it believed it was done.

That is a core hypothesis zeroshot is built on: an independent check catches what a builder's own assessment misses. The two-node loop is its smallest instance. A zeroshot graph can carry more than that, such as several builders working in parallel, checkers that specialize (one runs the tests, one reads the diff, one compares against the spec), verifiers with their own tools, and prompts written for the job at hand. We expect graphs built for a task to bring the cost-to-match number well below what a generic two-node loop gets.

Limitations

• 20 tasks, not the whole benchmark. Five we chose by hand: svgbob and ditaa from a shortlist of programs that are small by code size and rarely solved by published runs, and calcurse, revive and fasttext by a rule (at most 800 hidden tests over more than one test branch, Sol's published run between 15% and 80%, the best published run at least 5 points higher, one task per language not yet covered, and within each language the domain least like the two diagram renderers). The other 15 we drew at random, with a recorded seed and stratified by language, from the 72 ProgramBench tasks that met three conditions: Sol's published single run scores between 10% and 90%, the best published run is at least 5 points higher, and the hidden tests number at most 1,500 spread over more than one test branch.
• One model family, one harness. Luna and Sol are two models in the GPT-5.6 line, driven by the Codex CLI.
• Five runs per arm. The intervals in the figures are 95% bootstrap intervals over those five runs; some of them are fairly wide.

Setup and what we changed

Code, experiment files, every per-task result and the figure scripts are on the benchmark/programbench branch of the zeroshot repository.

Harness

zeroshot 10.8.0 driving Codex CLI 0.155.0 inside the official ProgramBench task image, instead of mini-SWE-agent, the scaffold the benchmark ships with. The task statement is the upstream mini-SWE-agent prompt with only its scaffold mechanics removed (the persona line, the one-command protocol, the submit sentinel, the command examples); our unit tests re-derive it byte for byte.

The reference binary

ProgramBench puts the reference at ./executable, which is also the build target. With the task unmodified, builders overwrote the reference, the checkers then had only a README that contradicts the binary in places and enforced it, and the loop gained nothing. So we moved the reference to a root-owned, execute-only path outside the workspace and pointed the task statement there. Every number in this post comes from that setup, both arms alike. The reference stays a black box that can be run but not read, and instrumenting it through the dynamic loader or ptrace disqualifies a run.

Documentation

As shipped, except on svgbob, where we corrected three README statements that contradict the binary, for both arms.

Network

Tool commands have no route out, as under upstream's --network none. Only Codex reaches the model API, through a per-attempt proxy that allows nothing else; web search is off.

Limits

Six hours per attempt, as upstream; three hours per build, one per check; no step limit.

Scoring

ProgramBench's own evaluator on the task image, hidden tests pinned, identical workspaces scored once. Each task's smoke test re-scores a published submission and asks for agreement within 2 points. Four tasks failed that gate (crowbook, dust, samtools, tree-sitter) and stayed in the study, because the causes touch both arms the same way.⁸ We compare arms within this setup and treat leaderboard numbers as context.

Audits

We scan every run's shell commands. Instrumenting the reference, a tool command calling the model API or reaching the network through Codex's proxy, reading process environments or searching the web disqualifies a run. All 300 runs in these numbers passed. We also diff each check against the preceding snapshot to confirm the checker edited nothing.