Scout: When the Harness Regresses: Attributing Coding-Agent Quality to Scaffolding, Not the Model

Summary

When a coding agent starts producing worse patches, the reflex explanation is that the model got dumber. A controlled 2026 study that held the model fixed across 35 sequential releases of one open-source agent shows how much of that decline the scaffolding alone can drive: task-resolution rate wandered between roughly 23% and 39% with no real upward trend, token spend per task climbed more than 70%, and the swings tracked specific scaffolding changes — context-management rewrites, tool wiring — while the weights stayed fixed. Anthropic’s own April 2026 postmortem is the same shape at production scale: six weeks of complaints that Claude Code “felt less intelligent” resolved to three harness and operating-instruction changes, with the model and API untouched. The problem is that a pass/fail resolve rate can’t tell you which layer moved — the evidence lives one level down, in the trajectory, the sequence of searches, reads, edits, tool calls, and validations a run actually performed. This briefing maps the misattribution problem and the diagnostic instruments — trajectory taxonomies and automated failure diagnosis — that make a regression legible enough to pin on the scaffolding or the model.

Key Findings

The reflex is to blame the model; the scaffolding changed more

“Scaffolding” and “harness” are the same thing under two names: the middleware between a developer and the model. Ben Sghaier, Li, Adams, and Hassan define it as a layer “that orchestrates system prompts, tool execution, context management, and iterative reasoning loops” — everything wrapped around the weights. Their controlled longitudinal study is the empirical anchor for the attribution question. Across five open-source agents (Codex, Qwen Code, Gemini CLI, OpenCode, OpenHands) and a deep dive into 35 sequential releases of one of them, they held the underlying LLM constant and re-ran a stratified slice of SWE-bench Verified tasks against each release.

The result cuts against the model-blame instinct. Over 35 releases the resolve rate fluctuated between roughly 23% and 39% with no statistically significant improvement despite continuous development, while token consumption per task rose more than 70% and tool-call counts climbed with it. The team associates the swings with specific architectural components — context-management expansion with worse token efficiency, and the provider and extensibility layers with better efficiency — and links individual pull requests to measured quality shifts, all drawn as correlations across the 35-release series rather than controlled ablations. Their framing of the field’s failure is direct: practitioners routinely report that an agent’s quality deteriorated after an update and pin it on the model rather than the scaffolding layer. Release velocity is part of why the misattribution sticks — the paper observes some of these projects shipping more than two releases a day, so the harness a team ran last week is rarely the harness they run today, and the one variable that didn’t change in that window is often the model.

The practitioner takeaway the authors push: treat scaffolding as quality-critical software that needs regression evaluation beyond functional correctness, and — for anyone benchmarking agents — report and control for the scaffolding version, not just the LLM. A resolve-rate number attached to “GPT-5.4” or “Opus 4.6” without a harness version is close to meaningless.

A production postmortem in exactly the same shape

The sharpest real-world instance is Anthropic’s April 23 postmortem on Claude Code quality. The six weeks of user reports that the tool had degraded resolved to three overlapping changes, none of them a model change: a default reasoning-effort downgrade from high to medium (March 4, reverted April 7); a prompt-caching optimization whose bug cleared the model’s accumulated thinking on every turn instead of once per idle session (shipped March 26, fixed April 10); and a system-prompt change adding verbosity limits that testing measured as a 3% quality drop on Opus 4.6 and 4.7. The postmortem states the API and inference layer were unaffected — none of the three regressions was a model change.

Two details make this the canonical attribution case. First, the diagnosis was genuinely hard because the causes were in the harness: each change hit a different slice of traffic on a different schedule, so the aggregate looked like “broad, inconsistent degradation” rather than a clean step-change a single model swap would produce. Second, Anthropic’s own internal evals and dogfooding missed all three — staff ran different builds, the caching bug only surfaced in stale sessions, and the eval suite was too narrow to detect a 3% drop from a prompt edit. If the vendor that ships the model can burn six weeks before localizing a harness regression to the harness, the assumption that “it feels worse” implies “the model got worse” deserves far more skepticism than it usually gets.

Resolve rate is the wrong instrument for a regression

The reason attribution is hard is structural: the metric everyone watches is too coarse to localize anything. Shu and colleagues put it plainly — coding agents are ranked almost entirely by whether the final patch passes the target tests, yet “a single pass/fail label says nothing about why a run failed or why an accepted run spent extra steps, time, or tokens”. Two runs can reach the same outcome through completely different processes. A parallel argument runs through AgentAtlas, which makes the case against outcome-only leaderboards: the same final answer can be acceptable or not depending on whether the agent used the right tool, respected the user’s constraints, and recovered after a failed observation — so it separates outcome success from control-decision quality and trajectory quality, distinctions a pass/fail score collapses.

The operational consequence is that a harness regression can show up in efficiency before it shows up in resolve rate. The Qwen Code data is the tell — token and tool-call inflation climbed steadily while pass rate stayed flat. An agent that has started thrashing (re-reading the same files, looping on searches, skipping validation) can still limp to a passing patch on easy tasks while costing more and degrading on hard ones. Watch only the pass/fail bit and you see nothing until the regression is bad enough to flip outcomes; watch the trajectory and you see it while it’s still only bleeding tokens.

Instrument, tier one: trajectory taxonomies that make structure comparable

The problem with “watch the trajectory” is that raw traces are heterogeneous — every harness logs differently, and no two runs line up. TraceProbe (Shu et al.) is the most developed answer: it normalizes each raw run into a canonical nine-type action taxonomy — file read, file write, search, command, plan, navigate, fetch, agent spawn, and reason — and tags each action with a deterministic effect label (a write that survived to the final state versus one that was reverted; a read that was justified by task relevance versus off-anchor; an action that failed). Two rule-based modules sit on top. Insight flags single-trajectory anti-patterns drawn from debugging practice — search loops, re-read churn, tool oscillation, missing tail validation, an unsupported “done” claim with no successful validation behind it. Converge aligns a pair of runs and classifies where their behavior diverges under a controlled reference — which is exactly the operation you want for a regression: diff a known-good trajectory against a suspected-regressed one and see the structural break.

Applied to 2,500 trajectories across five real harness-and-model settings (Claude Code with Opus 4.6, Codex with GPT-5.4, and OpenCode paired with Opus 4.6, GPT-5.4, and GLM-5), TraceProbe reports that file-level choice is too coarse to separate success from failure, while function-level selection and completion behavior localize it; that the search-loop signal transfers most stably across benchmarks — its tie to run outcomes stays consistent on both; and that even resolved runs differ measurably in how fast they reach relevant code and how much failed work they accumulate. For a practitioner, the value isn’t the specific findings — it’s the shared vocabulary and deterministic labels that turn “the agent feels worse” into a comparable, auditable structure you can bisect. Two caveats travel with it: the off-anchor read label leans on a benchmark’s gold-patch file set to define task-relevance, so production traces have to substitute proxies — the authors suggest surviving writes and test/import references — and the effect labels are descriptive, which the paper is explicit about: they “should not be read as standalone explanations for why a specific run passed or failed.”

Instrument, tier two: automated diagnosis that names the decisive step and the layer

Taxonomies give you comparable structure; the next tier tries to point at the fault automatically. Three 2026 frameworks converge on the same move — collapse a long, noisy trajectory to the step where the run actually went wrong.

TrajAudit (Wang, Xie, Huo) frames this as failure localization for repository-level coding agents. An investigator agent uses semantic saliency folding to strip failure-irrelevant noise and test-failure reports as prior knowledge to steer attention, then pinpoints the earliest decisive error step. On RootSE, a benchmark of 102 real repo-level instances each annotated with the decisive step, it reports localization-accuracy gains of 10.8% and 21.6% over the strongest baselines (with and without a reference solution).
AgentRx (Barke et al., Microsoft Research) is domain-agnostic: it synthesizes constraints from a trajectory, checks them step by step, and uses an LLM judge to localize the failure, emitting an auditable log of constraint violations with evidence. It is evaluated on 115 annotated failed trajectories across three task domains.
HarnessFix (“From Failed Trajectories to Reliable LLM Agents”) is the one aimed squarely at attribution. It separates harness-caused failures from model limitations, sorting flaws into seven layers — execution environment, tool interface, context/memory, lifecycle/orchestration, observability, verification, and governance — and mapping each failure to the responsible step and layer via a harness-aware trace representation. Its thesis restates the attribution problem exactly: reliability depends not only on the base model but on the agent harness. The authors report step-level localization accuracy around 85%, against roughly 45–52% for prior techniques, across GAIA, SWE-bench, AppWorld, and Terminal-Bench.

These sit at different confidence levels — HarnessFix’s clean harness-vs-model split is the most ambitious claim and the one to test hardest against your own incidents — but the direction is consistent across a crowding field of 2026 attribution work: move from a pass/fail bit to a located, layer-attributed failure.

First-party proof: same tools, worse reviews, diagnosed by reading traces

The research pattern has a clean industrial mirror. GitHub migrated Copilot code review onto the shared Unix-style tools (grep, glob, view) its CLI already used — and review cost went up while fewer issues were caught. The instructive part is the diagnosis. Offline benchmarks caught the regression as an aggregate signal — the review agent came out less efficient and less effective, with average cost up and useful comments down — but the number alone couldn’t localize the fault; the traces could. The team could see the agent was “browsing a repository instead of investigating a pull request”, running a general-assistant exploration workflow where a reviewer’s evidence-gathering workflow belonged, and every extra tool result it pulled inflated the working context and blurred the review. The tools were not the problem; the instructions wrapped around them were. Reshaping the guidance to a reviewer’s rhythm — start from the diff, narrow with search before opening files, batch cheap discovery — cut roughly 20% off average review cost while holding quality. Same tools, different scaffolding, materially different quality — and while the aggregate metrics said something was wrong, only the trajectory said what.

Practical Implications

Version and log the harness like source code. Record the scaffolding version alongside the model on every eval run and, ideally, every production run. A regression you can’t bisect to a release is a regression you can’t fix — and with some agents shipping multiple releases a day, “which harness version” is a real question, not a formality.
Instrument trajectories, not just outcomes. Capture the action sequence — searches, reads, edits, tool calls, validations — with enough structure to compare runs. Resolve rate is a lagging, coarse signal; token count, tool-call count, and steps-to-first-relevant-file move earlier and localize better.
When quality drops, diff a good run against a bad one before you touch the model. The Converge-style operation — align a known-good trajectory against a regressed one and look for the structural divergence (a new search loop, re-read churn, a dropped validation step) — is the fastest path to the responsible layer. Swapping the model is the expensive default that’s often the wrong lever.
Ask “what changed in the scaffolding?” first. Context-management logic, system prompts, tool schemas, reasoning-effort defaults, and caching are the usual culprits in the documented cases, and all of them change far more often than the weights. Anthropic’s episode was three of these at once; none was the model.
Build harness regression tests, and widen the eval enough to see small drops. The recurring recommendation — call it agentic quality assurance — is automated regression testing on non-functional quality (efficiency and trajectory structure), not only pass/fail. Anthropic’s broader ablation found a 3% quality drop its narrower suite had missed — the case for widening per-model eval coverage and correlating user reports with configuration history. Don’t over-read it, though: a verbosity-limit prompt can degrade quality without moving token counts at all, so efficiency metrics are not guaranteed to catch that particular class — the reliable fix is a wider eval, not a single leading indicator.
Treat automated failure-diagnosis tools as accelerators, not oracles. The TrajAudit / AgentRx / HarnessFix family can point at a decisive step and even a harness layer, but they grade against human-annotated benchmarks. Use them to triage and prioritize which runs a human inspects, not to close the ticket unread.

Open Questions

Do trajectory diagnostics transfer off-benchmark? TraceProbe’s taxonomy and anti-patterns are validated on SWE-bench-style trajectories from five settings. Whether the same effect labels and detectors hold on messy production repositories, long-horizon tasks, and MCP-heavy tool surfaces is unestablished — and is exactly the gap a team should close with its own traces.
How well does automated attribution hold on unlabeled incidents? The localization-accuracy figures are measured against human-annotated “decisive step” labels. Whether that accuracy survives on real production failures with no ground-truth annotation is open.
Is model-vs-harness attribution ever fully clean? HarnessFix separates the two in principle, but the Anthropic case shows regressions that are interaction effects — a verbosity prompt that hurts one model family more than another. A prompt-times-model interaction has no single owner, and clean attribution may be the exception rather than the rule.
Is efficiency drift a reliable leading indicator? The Qwen Code data shows token and tool-call inflation without a resolve-rate change. Whether rising per-task cost is a dependable early warning of a coming quality regression, or just churn, needs more longitudinal data than any single study yet offers.