Scout: Which Model Should Review Which? The Asymmetry in Cross-Model Code Review

Summary

A controlled experiment on 116 competitive-programming tasks reports that write-then-review pairings between Claude and Codex behave asymmetrically against their own baselines: one ordering lifts pass rates by 18 percentage points, the reverse ordering costs 8.6. That result is narrow — one model pair, one benchmark family, a reviewer forbidden from running tests, and a prompt that forces the reviewer to emit a program whether or not it has anything to say. But the mechanism it points at generalizes further than the numbers do: review helps in proportion to how much catchable error the draft still contains, and a reviewer with no execution oracle and no option to abstain will fill that vacuum with rewrites. The repository-scale evidence, where reviewers can explore the codebase and abstain, reproduces the headroom effect — and reproduces the harm as well, at 7.3 points, whenever the reviewer is too weak for the patches it is judging. The practical conclusion is not a model ranking. It is that two things move the outcome and only one of them is cheap: the reviewer’s permissions — can it execute, can it decline to intervene, can it only comment rather than rewrite — and its competence at the review task itself, which in the one comparison that holds the harness fixed is worth 7.7 points on its own.

Key Findings

The measurement, stated precisely

Zuodong Xiang, Yike Zhang, YueMing Zhang and Hailu Xu ran six conditions over 116 hard and medium LiveCodeBench problems released after 2025, using Claude Opus 4.7 and Codex GPT-5.5 at high reasoning effort (Xiang et al.). The reviewer saw the problem statement and the draft, could not execute tests, and emitted a final program. All figures are the authors’:

Three things in that table get less attention than the headline. First, the harmful direction is the least statistically secure result of the set — it clears the corrected threshold, barely. Second, most of the gain available to Codex drafts is available without a second vendor: self-review carries 12.9 of the 18.1 points. The cross-vendor premium in the best direction is roughly five points, not eighteen. Anyone justifying a second subscription on this study should be pricing five points, not the headline.

Third — and this is the caveat the authors state and most summaries drop — each ordering is separable from its own baseline, but the two orderings are not separable from each other. The direct contrast between them does not survive multiple-comparison correction (p_BH = .1444), and the paper says plainly that it therefore does “not claim that the two cross-model orderings are statistically separable.” The asymmetry in the table is a description of this sample, not an established between-orderings difference.

Mechanism: review pays in proportion to residual catchable error

The authors’ own explanation is a headroom argument — Claude’s heavier first pass leaves less for any second pass to find, and Codex leaves more residual, catchable error behind. The latency asymmetry is consistent with it: 86.2 seconds solo for Claude against 38.5 for Codex (full text).

This part of the story carries further than the rest, because it replicates independently. Zhou, Xu, Zhou, Singh, Gui and Joty measured verification dynamics across 14 open-source models plus GPT-4o and found that holding the verifier fixed while strengthening the generator collapses the ability to detect wrong answers — true-negative rate falling from 0.68 on Llama-3.1-8B solutions to 0.17 on Qwen3-32B solutions, while true-positive rate stays roughly flat (Zhou et al.). The stronger the writer, the harder its remaining errors are to see. A reviewer’s value is not a fixed property of the reviewer.

That framing also explains why the effect is so sensitive to execution. Jason Wei’s account of verification asymmetry lists the properties that make checking cheaper than solving — objective truth, fast verification, scalable verification, low noise, continuous reward — and notes the reverse cases where verification costs more than generation (Wei). A static reviewer looking at a competitive-programming solution with hidden tests has almost none of those properties available. It is operating in precisely the regime where verification is not obviously the easier job.

Mechanism: the harm is action, not blindness

The paper’s qualitative reading of reviewer behaviour transfers further than its pass rates do. Per the authors: “Codex GPT-5.5 as reviewer, when uncertain, tends to discard the writer’s data structure and start over…Claude Opus 4.7 as reviewer tends to keep the writer’s interface and repair one local invariant.” Edits help; rewrites hurt.

Two design choices in the experiment make that disposition maximally expensive. The reviewer always emits a final program, so there is no separate decision about whether to intervene at all — a limitation the authors name. And the reviewer holds write authority over the draft by construction. Combine must produce output, cannot run anything and may rewrite freely, and a reviewer facing a draft population that already passes 91.4% of the time has no safe move available.

Independent work suggests the overcorrection tendency is not specific to one vendor. Haolin Jin and Huaming Chen document what they call systematic overcorrection in requirement-conformance judgement — LLM reviewers flagging conforming code as non-conforming — and report that more elaborate prompts, including demands for explanation and suggested corrections, make the misjudgement rate worse rather than better (Jin and Chen). Production teams reach the same place empirically: G-Research’s account of building an internal review tool describes abandoning single-pass review for a two-call split where the first pass maximizes recall and a second pass filters, with integration thresholds set at 100% recall on mandatory rules and overall precision above 85% (G-Research).

Mechanism: self-preference is the wrong explanation here

The obvious story — models favour their own output, so cross-model review must be better — does not fit the data. Wataoka, Takahashi and Ri quantified self-preference in LLM judges over 33,000 Chatbot Arena dialogues, scoring GPT-4 at 0.520 — the highest among the models they measured — and traced the effect to perplexity: judges rate low-perplexity text more highly than humans do regardless of who wrote it, and self-generated text is low-perplexity by construction (Wataoka et al.). Augment Code builds its case for adversarial review on exactly this evidence, arguing that masking a judge’s ability to recognize its own output reduces the bias, which implicates recognition rather than instruction (Augment Code).

That is a real bias, and it is a good argument for separating maker from checker. It is not what drives this experiment’s self-review conditions. Self-preference predicts a self-reviewer finds less than an independent one would; it does not predict a net loss, so a positive self-review result cannot rule it out. But it also cannot account for the pattern: Codex self-review gained 12.9 points while Claude self-review changed nothing at all, and self-preference offers no reason for the two to differ. Headroom does. Note the limits of the underlying evidence too — the bias was measured on pairwise judging of dialogue, not on the net correctness of a revised program, so neither result isolates it.

The deference literature does not apply — and that is itself informative

Multi-agent debate research documents exactly the failure a cross-vendor gate might be expected to produce. Wynn, Satija and Hadfield find that models shift from correct to incorrect answers in response to peer reasoning more often than the reverse, and that stronger agents flip toward weaker peers more readily than weaker agents learn from stronger ones (Wynn et al.). Bertalanič and Fortuna measured the cost directly: unguided homogeneous debate underperformed isolated self-correction on GSM-Hard and MMLU-Hard while consuming a 2.1×–3.4× token multiplier, with modal adoption of peer answers reaching 85.5% and oracle gaps — correct answers present in the pool but voted away — of up to 32.3 percentage points (Bertalanič and Fortuna). Kasprova and colleagues trace how sycophancy toward a user propagates through a multi-agent panel, recovering 10.5 points of absolute accuracy by supplying priors that rank peers on their tendency to agree with the user’s stance; the authors are explicit that they measure agreement with the user rather than with the shifting peer majority, which distinguishes what they study from conformity (Kasprova et al.).

None of that is what happened in the cross-model review experiment. There is no dialogue: the reviewer never meets the writer, receives no argument, and cannot be talked out of a position. Deference has no channel to travel through. Importing the debate explanation would be a mistake.

The forward-looking version matters more. Every practical review loop that adds a back-and-forth — reviewer raises a finding, writer defends, reviewer reconsiders — reintroduces exactly the channel these papers measure. The controlled study’s single-shot design is the favourable case. A conversational review gate carries a failure mode the study never tested.

The confound nobody has separated: fresh context vs. different vendor

Song Tae-Eun ran review in a session containing only the artifact and the review prompt, with no access to the production conversation, using a single model throughout — Claude Opus 4.6. Reported F1 for cross-context review was 28.6% against 24.6% for self-review, 21.7% for repeated self-review and 23.8% for subagent review, with critical-error detection at 40% versus 29%, all at p < 0.01. The author attributes the benefit to context separation itself rather than to repetition, and names single-model coverage as a limitation (Song).

Which raises the question the cross-vendor literature keeps stepping around. A second-vendor review gate changes two variables at once: the weights and the context. No published experiment appears to vary them factorially. It remains plausible that a meaningful share of what a cross-vendor gate delivers is available from a clean-context pass on the same model — which is cheaper, simpler, and does not require a second billing relationship.

Does it survive into repository maintenance? The harm does too

The authors are explicit that their tasks are self-contained Python programs with hidden tests rather than patches inside a live repository, that 116 paired tasks suffice for a first diagnostic rather than a stable ranking, and that generalization beyond this model pair to families such as Gemini, DeepSeek, Qwen or Grok is untested.

The repository-scale evidence tells a partly different story. Wang, Chen, Wang and colleagues built a generate-review-revise loop where an agentic reviewer explores the repository before deciding, evaluated over 1,384 pull requests derived from 500 SWE-bench Verified issues. Resolve rates rose from 27.5% to 56.9% for the weakest generator, 50.9% to 68.8% for a mid-tier one, and 72.2% to 75.4% for the strongest; agentic review beat single-turn review by 8.5 points on the weakest generator’s patches (Wang et al.). The headroom law replicates cleanly — the weakest writer gains nearly thirty points, the strongest gains three.

The harm replicates too, and that is the part worth sitting with. Three configurations in that table land below their no-review baseline. Two of them are small: SWE-Review-8B on GLM-5 patches, 71.6% against 72.2%, which the authors attribute to the distilled reviewers being “much smaller and less capable than GLM-5 itself.” The third is not small. An untrained Qwen3-30B-A3B used as a reviewer drops GLM-5 patches to 64.9% — a 7.3-point regression, within striking distance of the static setting’s 8.6 — and takes another 1.8 points off the mid-tier generator’s patches. None of the three is reported with a confidence interval.

What separates that reviewer from the one immediately below it in the table is not its permissions. Both explore the repository, both may decline to request changes, both run inside the same loop. The difference is supervised fine-tuning on review trajectories, and it is worth 7.7 points: the same architecture goes from −7.3 to +0.4 on the same split. Its comment rate barely moves (84.2% to 82.0%) while its decision accuracy climbs from 63.2% to 69.0%. That is the harmful configuration stated precisely — a reviewer that intervenes constantly and judges poorly — and handing it a repository to explore does not fix it. What the repository setting buys is a ceiling: the strongest reviewers improve every split. What it does not buy is a floor.

Temper this with the ceiling. Zhang, Pan, Yusuf and co-authors’ review-agent benchmark, built from human review comments over real pull requests and covering PR-agent, Devin, Claude Code and Codex, reports that existing review agents taken together solve only around 40% of its tasks, and that agent reviews frequently emphasize different concerns than human ones (Zhang et al.). Repository-scale review is safer than the competitive-programming result suggests. It is not close to solved.

Giving the reviewer an oracle is the higher-leverage change

The pre-LLM-agent literature already located the binding constraint. Olausson, Inala, Wang, Gao and Solar-Lezama found self-repair gains modest and highly variable once repair cost is accounted for, and identified the bottleneck as the model’s ability to produce accurate feedback on its own code; substituting stronger feedback produced substantially larger gains (Olausson et al.), with the substitution of human feedback for GPT-4’s reported as improving the passing repair rate by a factor of 1.58 (summary). The reviewer slot is where quality binds. That has been true since 2023; the cross-model result is a recent instance of it.

The harness answer is to stop asking the reviewer to be the oracle. Anthropic’s guidance on verification loops frames the target as a repeating cycle in which the agent runs tests, linters and custom checks and fixes what fails before proceeding, with the checks encoded as skills rather than left to the model’s discretion (Anthropic). Rick Hightower’s treatment of the maker/checker split makes the same point structurally: where “good enough” can be expressed in code, the checker should be code — able to run real tests, compile, validate schemas and query databases — and a model checker is reserved for the genuinely subjective residue (Hightower).

The catch is that repositories supply thinner oracles than benchmarks do. Dipongkor, Baral, Lam and Moran examined 4,882 agent-generated pull requests and report that agents include test changes in only 49.6% of PRs touching code under test; that existing tests cover 61.5% of changed executable lines in Java but 27.0% in Python; that 64.8% of Python PRs have no changed line executed by any existing test; and that error-handling constructs are missed at 86.0% in Java and 81.0% in Python (Dipongkor et al.). Let the reviewer run the tests assumes tests that, on most agent-authored Python changes, do not touch the code in question.

The handoff format may move the reviewer more than the vendor does

Every review gate ships the reviewer something besides the diff. Todd Orr ran the experiment that isolates it: six diffs — five with planted bugs, one clean control — packaged four ways and reviewed twice each by Claude Opus 4.7 and Codex GPT-5.5, for 96 reviews. The variable was the implementer’s own self-assessment, framing language and reasoning trace: present in three graduated strengths, stripped entirely in a redacted condition (Orr).

The severity result is the sharpest. Claude caught the planted bug in nearly every condition — detection barely moved — but what it called the bug did: 9 of 10 catches tagged Critical under redaction, against 6–7 of 10 once any implementer framing was attached. On the clean control diff, mean findings fell monotonically with framing strength, from 12.5 redacted to 11.5, 9.5 and 6.0 as confidence language increased. Codex moved further still, averaging 9.4 findings per review on redacted packages against 2.4–2.6 under mild and moderate framing.

Orr is careful about the mechanism, and the caution is worth preserving. He raises anchoring and then declines to claim it, noting that whether LLMs are subject to it “in the same psychological sense humans are is (as far as I know) an open question.” His alternative description is that the reviewer is “freeloading off the implementer’s framing” — accepting all tests pass as a premise rather than as a claim to check. He is equally direct about the limits: two runs per cell is “enough for the catch-rate and severity numbers to mean something… not enough for the small-issue counts to be tight,” strong framing accidentally tipped the reviewer off on two diffs, and the grading is heuristic at roughly 90–95% accuracy by spot-check.

This is one practitioner’s eval rather than a peer-reviewed study, and it should carry that weight. It also varies framing within each reviewer rather than estimating a vendor effect to weigh against — so it establishes that both models are framing-sensitive, not that framing outweighs the vendor. But it varies something no published cross-vendor comparison controls at all. If the packaging moves severity classification by three tags in ten, a vendor A/B needs to hold the handoff format constant and run at more than one setting of it: an effect measured at a single framing may not survive at another.

The tooling default runs in the direction the study warns about

OpenAI’s cross-provider plugin puts Codex inside Claude Code, adding standard review, an adversarial review pass, asynchronous task delegation, and a review gate that can block code from being finalized until Codex has evaluated it (Techstrong.ai). The gate is opt-in rather than on by default — the project’s own documentation describes it as an optional gate, enabled with /codex:setup --enable-review-gate, which installs a Stop hook that blocks completion until Codex has reviewed (openai/codex-plugin-cc). But enabling it is one command, and the direction it then runs in is Claude writes, Codex reviews — the ordering the controlled experiment measures as net-negative in its setting.

That is a caution rather than a verdict, for two reasons. The plugin’s reviewer is not the paper’s reviewer: it explores a real repository rather than a problem statement, which is the variable that separated the harmful setup from the safe one. And harness version is a confound in its own right. Ben Sghaier, Li, Adams and Hassan fixed the model and varied only the harness across 35 sequential releases of a single CLI, evaluated on stratified SWE-bench Verified tasks, and traced quality swings to specific harness pull requests amid release velocities exceeding two per day (Ben Sghaier et al.). Codex reviewing inside a plugin and Codex GPT-5.5 reviewing inside a research design are not the same reviewer, and at the release cadence that work documents, the two drift apart fast.

Practitioners who have built these loops by hand tend to route around the ordering question entirely. Sameer Ahmed Khan’s dual-review setup has both models review the same diff independently, merges the findings tagged by source, treats agreement as near-certain signal, and escalates disagreements to a human with both positions attached (Khan). No model rewrites the other’s code, so the harmful-rewrite pathway never opens.

There is a formal version of that instinct. Ning Liu’s jury approach has independently-trained models each solve the problem once, with the selection signal coming from agreement structure rather than any model scoring another’s work; a cross-family panel of Qwen3-235B, DeepSeek-V3.2, Claude Sonnet 4.6 and Kimi-K2.5 reached 56.7% on AIME-2024 against 36.7% for self-consistency, and four decorrelated models beat 32 resamples of one (Liu). The mechanism is error decorrelation — the same property cross-vendor review is reaching for, harvested without letting one model edit another’s output. The scope caveat is real: that work covers answer selection on reasoning and math benchmarks, not patch revision in a repository.

Practical Implications

Separate flagging from fixing. Denying the reviewer write authority is the most plausible single fix for the measured harm — but no published experiment isolates it, and the authors are careful about this: they “did not measure rewrite frequency directly,” and present the rewrite mechanism as an interpretive reading of the artifact rather than a quantitative result. Adopt it because it is cheap and the failure mode it closes is real, not because it has been shown to be the operative variable. Let the reviewer produce findings; let the original writer, or a deterministic tool, apply them. Augment Code’s operational version is to grant reviewers read-only tooling — read, grep, glob, no write — so nothing changes before a human sees it. In the controlled experiment the reviewer’s rewrite is the entire harm channel.

Add an explicit abstain path. A reviewer required to emit output on every draft will manufacture findings on correct code. Make no change required a first-class, cheap answer, and log how often it fires. If your reviewer never abstains, you are measuring its compliance, not the code.

Buy execution before you buy a second vendor. Ordering matters most when the reviewer is blind — but exploration and an abstain path are not on their own enough, since the repository-scale setting still cost 7.3 points when the reviewer was too weak for the patches it judged. Route the verification budget into deterministic checks the agent can run — the checker that compiles, tests and lints costs no tokens per invocation and never hallucinates a finding — and reserve the model reviewer for the judgement calls code cannot express.

Then check that the oracle exists. Before assuming test execution rescues the reviewer, measure coverage on the paths agents actually touch in your repository. On agent-authored Python changes, the sampled evidence says the odds are better than even that no existing test exercises the changed lines at all. Error-handling paths are worse still.

Strip implementer framing from the handoff. Every review gate ships the reviewer something alongside the diff — the writing agent’s summary, its confidence assessment, its own account of what it changed. In the one experiment that varies it directly, removing it moved Claude’s Critical tags by three in ten, took Claude’s findings on clean code from 6.0 under strong confidence language to 12.5 with none, and took Codex from as low as 2.4 findings per review to 9.4. Stripping it is a plumbing change rather than a procurement decision. Run your loop both ways before concluding the vendor was the variable.

Try a clean-context pass on the same model first. It is the cheapest thing on this list and the least tested against the cross-vendor alternative. If a fresh-session review recovers most of the benefit, the second subscription is buying less than assumed.

If your writer is your strongest model, the burden of proof runs the other way. Adding a weaker reviewer with rewrite authority over a strong draft is the configuration this evidence flags. That does not mean never do it — it means measure before adopting it, and prefer the comment-only form.

Measure your own pairing, cheaply. Retain the pre-review draft. Track how often the reviewer intervenes, and on merged changes, how many interventions fixed a real defect versus introduced one. Two counters and a retained artifact are enough to detect a net-negative gate; nobody needs a paper to know whether their own loop is paying for itself.

Open Questions

Does the direction survive a version bump? The result is measured on Claude Opus 4.7 and Codex GPT-5.5. Both vendors have shipped since. The headroom mechanism predicts the asymmetry tracks whichever model currently leaves less residual error — which means the direction could invert on a release, not merely weaken. The authors themselves expect newer versions to change the numbers.

How much of the cross-vendor benefit is just fresh context? Published work has not yet varied weights and context separation independently within a single design. Until that experiment exists, the cross-vendor premium is an upper bound on the value of the vendor difference.

Does an explicit intervene-or-abstain decision remove the harm entirely, or only shrink it? Not entirely: at repository scale a reviewer with both an abstain path and repository access still cost 7.3 points when it was too weak for the patches. But those results change several variables at once, so how much the abstain option contributes on its own is unmeasured. Isolating it on the original benchmark would settle it.

Is the whole effect an artifact of the no-execution constraint? Re-running the six conditions with a reviewer permitted to execute against sample tests would answer whether static review is the problem or the pairing is.

Is the rewrite disposition a model property or a prompt property? The authors suggest more conservative reviewer instructions might reduce harmful rewrites, and did not test it. A one-line prompt change that neutralizes an 8.6-point regression would be the cheapest fix available, and nobody has published whether it works.

Do third families behave the same way? Gemini, DeepSeek, Qwen and Grok are untested as either role in this design, and a two-model result cannot establish a general ordering rule.

How much does the packaging of the draft change the reviewer’s behaviour? One practitioner eval found both tested reviewers sensitive to framing, Codex markedly so — but it varied framing within each reviewer rather than estimating a vendor effect to compare it against, and it rests on 96 reviews, two runs per cell and heuristic grading. No peer-reviewed work has varied handoff framing as a controlled factor, and most cross-vendor comparisons leave it uncontrolled entirely. Whether framing or vendor is the larger term is, on current evidence, simply unmeasured.

Sources

1. Cross-Model LLM Code Review: Should you use Claude to review Codex or vice versa? — Zuodong Xiang, Yike Zhang, YueMing Zhang, Hailu Xu (abstract)
2. Cross-Model LLM Code Review — full text — conditions, per-condition figures, reviewer-behaviour analysis, limitations
3. Variation in Verification: Understanding Verification Dynamics in Large Language Models — Yefan Zhou, Austin Xu, Yilun Zhou, Janvijay Singh, Jiang Gui, Shafiq Joty
4. Asymmetry of verification and verifier’s law — Jason Wei
5. Is Self-Repair a Silver Bullet for Code Generation? — Theo X. Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, Armando Solar-Lezama
6. Is Self-Repair a Silver Bullet for Code Generation? — summary and reported figures — EmergentMind
7. Are LLMs Reliable Code Reviewers? Systematic Overcorrection in Requirement Conformance Judgement — Haolin Jin, Huaming Chen
8. Self-Preference Bias in LLM-as-a-Judge — Koki Wataoka, Tsubasa Takahashi, Ryokan Ri
9. Talk Isn’t Always Cheap: Understanding Failure Modes in Multi-Agent Debate — Andrea Wynn, Harsh Satija, Gillian Hadfield
10. The Cost of Consensus: Isolated Self-Correction Prevails Over Unguided Homogeneous Multi-Agent Debate — Blaž Bertalanič, Carolina Fortuna
11. Too Polite to Disagree: Understanding Sycophancy Propagation in Multi-Agent Systems — Vira Kasprova, Amruta Parulekar, Abdulrahman AlRabah, Krishna Agaram, Ritwik Garg, Sagar Jha, Nimet Beyza Bozdag, Dilek Hakkani-Tür
12. Cross-Context Review: Improving LLM Output Quality by Separating Production and Review Sessions — Song Tae-Eun
13. SWE-Review: Closing the Loop on Issue Resolution with Agentic Code Review — Ruoyu Wang, Jierun Chen, Shaowei Wang, Chaofan Tao, Sidi Yang, Yuxin Jiang, Kim-Hui Yap, Lifeng Shang, Xiaohui Li, Haoli Bai
14. Code Review Agent Benchmark — Yuntong Zhang, Zhiyuan Pan, Imam Nur Bani Yusuf, Haifeng Ruan, Ridwan Shariffdeen, Abhik Roychoudhury
15. Test Coverage Analysis of Agentic Pull Requests — Atish Kumar Dipongkor, Talank Baral, Wing Lam, Kevin Moran
16. Don’t Blame the Large Language Model: How Agent Harness Evolution Shapes Coding Agent Quality — Oussama Ben Sghaier, Hao Li, Bram Adams, Ahmed E. Hassan
17. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code — Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, Ion Stoica
18. LLMs as a Jury: Cross-Model Consensus Can Outperform Process Reward Models for LLM Reasoning — Ning Liu
19. Building verification loops in Claude Code with skills — Anthropic
20. Harness Engineering: The Maker/Checker Split, Three Ways — Rick Hightower
21. Building a code review tool: The LLM patterns that actually work — G-Research
22. Adversarial Code Review: Why the Maker Shouldn’t Grade the Checker — Augment Code
23. Claude Code and Codex Together: Dual AI Review — Sameer Ahmed Khan
24. OpenAI Challenges Claude Code With Cross-Platform Codex Plugin Push — Techstrong.ai
25. What I Found When Claude Reviewed Codex’s Work — Todd Orr (96-review handoff-framing experiment, May 2026)
26. openai/codex-plugin-cc — OpenAI (plugin documentation; optional review gate and its Stop hook)