Scout: The Harness Is a Cost Lever: Evaluating and Choosing Agent Harnesses

Summary

For two years the dominant question in agentic coding was which model. A cluster of mid-2026 benchmarks moved the question to which harness — the agent loop, tool definitions, context management, system prompt, and interface that wrap the model. GitHub published a first-party evaluation arguing its Copilot harness reaches task-resolution parity with vendor harnesses (Claude Code, Codex CLI) at lower token consumption across most configurations; a Peking University benchmark found a 23.8-point spread between the best and worst harness on a fixed model and backend pool; and the Artificial Analysis Coding Agent Index started ranking model-plus-harness pairs rather than models in isolation. The catch is that harnesses are co-designed with their models through post-training — a vendor harness is tuned to the exact tool grammar and prompt structure its model was trained on — so both vendor parity claims and “model-agnostic” third-party claims deserve scrutiny. This briefing builds a decision framework: what to benchmark, how to measure token efficiency without fooling yourself, where first-party numbers should be discounted, and when owning or swapping the harness actually pays.

Key Findings

The harness is now a first-class performance variable, not a wrapper

The clearest statement of the shift comes from the survey literature. “From Question Answering to Task Completion: A Survey on Agent System and Harness Design” (Guo, Hao, Wang, et al.) decomposes the execution harness into six coupled runtime responsibilities: “observation, context, control, action, state, and verification”, and argues that agent quality — success, efficiency, safety, generalization — “emerges from the interaction between model capability, runtime infrastructure, task structure, and evaluation design.” In other words, the model is one of four inputs, and three of the four live in the harness and the eval.

A practitioner-facing definition that travels well: the harness is everything except the model — “the agent loop, the tool definitions, the context management, the system prompt, the interface”. That same comparison notes the design space spans from a minimal harness (a ~200-token system prompt with four tools) to a 10,000-token orchestration layer with sub-agents, hooks, and MCP plugins — a range wide enough that “the harness” is not a detail.

How much does the harness actually move the numbers?

Three data points of differing evidentiary weight — one controlled benchmark, then two practitioner/leaderboard observations (both drawn from Bustamante’s essay, so weigh them as practitioner evidence, not independent experiments):

Harness-Bench (Yao, Tan, Liu, et al., Peking University and Qiyuan Tech) fixes the task and varies the harness across six configurable harnesses and eight model backends over 106 sandboxed tasks. Its headline: NanoBot scored 76.2% aggregate while OpenClaw scored 52.4% — a 23.8-point gap “under the same task set and model-backend pool”. The paper also reports that weaker models show larger variance across harnesses, “suggesting that their performance is more sensitive to the surrounding execution substrate” — which suggests harness selection may matter most precisely when teams reach for a weaker or lower-cost model. (Even here, the authors frame the result as configuration-level variation across model–harness pairings, not a pure component ablation.)
Cursor’s own team reported moving their coding agent from roughly rank 30 to rank 5 on Terminal-Bench 2.0 “by only changing the harness,” holding the model constant.
On the same leaderboard, Claude Opus 4.6 paired with the ForgeCode harness reportedly scored 79.8% versus 75.3% with Capy — a 4.5-point spread between two harness configurations on the same named model. Read it as a strong configuration effect, not a clean harness-only ablation: public leaderboard entries don’t hold inference settings, budgets, and submission method constant.

Treat the specific leaderboard numbers as point-in-time snapshots (late April 2026) on one benchmark, not durable law — but the direction is consistent across the evidence: at a fixed model, the harness (and its surrounding configuration) moves the score by a margin comparable to, sometimes larger than, a model upgrade.

The GitHub parity claim: read the qualifiers, then discount accordingly

GitHub’s evaluation is the proximate cause of the “meta-harness” framing, and it is worth quoting precisely because the precise version is more careful than the headline version. GitHub reports that “task resolution rates for the GitHub Copilot agentic harness across these benchmarks are on-par with model-vendor harnesses when used with a fixed model and benchmark task,” and that the harness “achieves task completion rates on par with other model-vendor harnesses, while showing lower token consumption across most configurations.” The parity is explicitly hedged: “These results reflect effective parity, since the differences in either direction are within the variance due to the stochastic nature of the models.”

The evaluation spans SWE-bench Verified, SWE-bench Pro, SkillsBench (a public skills-use benchmark, not a GitHub-internal one), TerminalBench, and an internal Windows-container benchmark (Win-Hill), comparing Claude Sonnet 4.6 and Claude Opus 4.7 against Claude Code, and GPT-5.4 and GPT-5.5 against Codex CLI. Metrics are presented as pass@1 — but read that with point 4 below: on the smaller benchmark sets, the reported figure is the best of five runs, not their mean.

Where to discount, and why:

First-party builder running its own benchmark. GitHub built the harness, chose the benchmark suite, and reports the result. That is not disqualifying — the suite is public and the hedging is honest — but it is the textbook setup for a parity claim, and “lower token consumption across most configurations” is doing real work. Some configurations lost. The aggregate hides benchmark-specific differences (GitHub’s own charts show the Copilot harness trailing on some SWE-bench Verified runs).
“Parity within variance” is not “wins.” Effective parity at lower token cost is a genuine and useful result, but it is a tie on quality with a token edge — not a claim that the harness is better at the task. Read it as “you can plausibly run the same model through a different harness without losing resolution rate, and possibly spend fewer tokens,” not “switch and win.”
Second-hand multipliers are not in the source. Secondary aggregators circulated a tidy “35% token savings” figure; that number does not appear in GitHub’s post, which reports a comparative direction, not a headline percentage. If you cite a savings figure, cite the one you measured on your own workload.
Best-of-five on the small benchmarks. GitHub’s own methodology, verbatim: “To reduce the impact of run-to-run variability on smaller benchmarks (<100 instances), five independent runs were conducted, and the best scored run is reported.” That trims unlucky runs, but it is result-selection on exactly the stochastic variance the “within variance” parity hedge leans on — so on those sets the reported pass@1 reads better than a single production run would. This belongs higher in the discount stack than “some configurations lost”: a best-of-five number is not an expected-case number.

Why “model-agnostic” cuts both ways: harness and model appear to be co-adapted

The sharpest reason to distrust any harness-portability claim — vendor or third-party — is the growing evidence that vendor harnesses are co-adapted with their models. Two claims sit here, and they have different evidentiary standing. The well-supported, observable one: models perform differently with different tool schemas and prompt structures — Codex-family models are tuned to emit patches via a custom grammar, Claude to old_string/new_string replacement, and giving a model a tool format it wasn’t trained on can “cost extra reasoning tokens and produce more mistakes”. The stronger, inferred one — that vendors deliberately post-trained each model against its exact production harness — is Nicolas Bustamante’s argument (“Models are post trained against the harness, not just the API”), a well-reasoned reverse-engineering of tool shapes and routing, not a vendor-disclosed training fact; weigh it as a strong inference. Either way, Bustamante documents the runtime tell in Copilot’s own CLI: it routes tools per model family — including an apply_patch tool loaded only for Codex-family models and a deferred tool-loading (ToolSearch) pattern for Claude models — which shows even a multi-model harness relies on model-specific routing rather than one identical tool surface.

The practitioner consequence Bustamante draws is that “to swap a model cleanly, you have to swap the harness with it”. Read that as the model-facing parts of the harness — tool surface, schema shapes, system-prompt structure, memory rituals — not necessarily the whole stack: GitHub’s CLI keeps one overarching harness while routing per-model tool surfaces, so a clean swap can be a per-model adapter rather than a wholesale replacement. A third-party harness that claims to wrap any model is either re-tuning those model-facing parts under the hood (good, but then it’s N tool surfaces, not one) or accepting a portability tax it may not be reporting.

Honest token-efficiency measurement is harder than a token counter

Token efficiency is the headline metric of the meta-harness moment, and it is the easiest one to measure dishonestly. Several traps recur:

Tokens are not cost. Providers price input, cached input, cache-write, reasoning, and output tokens differently. The Artificial Analysis methodology reports cost as “average pay per token API cost per task, based on provider token pricing rather than consumer plans,” and treats cached input separately from uncached input where pricing supports it, including cache-write charges. A harness that aggressively warms the prompt cache can post a high raw token count and a low bill — or vice versa.
Subscription pricing hides the real comparison. Many developers and teams reach harnesses through subscription plans, not per-token billing, so the API-cost-per-task number that benchmarks report is not the number on your invoice. Benchmark on per-token cost to compare harnesses fairly; budget on your actual plan.
Run-to-run variance is large. Per-task token consumption swings widely across repeated runs of the same task — one secondary report of an OpenReview token-consumption study (not in this briefing’s bibliography; its full text sat behind a verification wall, so treat this as a directional, unverified pointer) describes some runs using a large multiple of the tokens of others, with command/tool output the highest-variance contributor. A single-run comparison is noise. Average across repeats — as some Artificial Analysis evaluations do (its Terminal-Bench v2.1 implementation averages three per task).
A lot of the token bill is removable waste. Analysis of agentic SWE-bench runs has found that a large share of tokens come from tool results, and a substantial fraction of those are removable with no measured performance loss — which means token efficiency is partly a context-pruning problem the harness either solves or doesn’t. (These figures come from secondary reporting of the OpenReview token-consumption study, whose full text was behind a verification wall during this research; treat the exact percentages as indicative rather than precise.)

The takeaway: a credible token-efficiency comparison holds the model and task fixed, runs multiple repeats, reports cost (not raw tokens) under a stated pricing model, and discloses caching assumptions. Anything less is exploratory evidence, not a decision-grade comparison.

Meta-harnesses and automated harness optimization

Two distinct things travel under “meta-harness,” and conflating them muddies the decision.

The orchestration meta-harness — a layer above individual agent harnesses that lets you plug in, schedule, govern, and collaborate across multiple agents through one interface. Several appeared during June 2026 (Databricks Omnigent, Vercel’s agent tooling, Cloudflare’s offering, Conductor, and others), alongside protocol-standardization efforts like Zed’s Agent Client Protocol, which is explicitly modeled on the Language Server Protocol — LSP decoupled language intelligence from the editor; ACP wants to decouple the agent from the editor. The appeal is owning the orchestration and memory layer rather than outsourcing it to a single vendor.
Automated harness optimization — “Meta-Harness” (Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, Chelsea Finn) is an outer-loop system that “searches over harness code for LLM applications” using an agentic proposer that reads the source, scores, and execution traces of prior candidates. It reports a 7.7-point improvement on a text-classification task “while using 4x fewer context tokens” over a strong baseline, gains on math reasoning, and discovered coding harnesses that surpass hand-engineered baselines on TerminalBench-2. The paper’s framing is the thesis in one sentence: LLM-system performance “depends not only on model weights, but also on their harness.”

Note the Latent Space “meta-harness” essay sits partly behind a paywall; the orchestration-layer framing and ACP-as-LSP analogy above come from its visible recap section, not the gated body.

Practical Implications

A decision framework for evaluating and choosing a harness independent of the model:

Benchmark the pair, on your repos, not the model alone. The unit of evaluation is model-plus-harness. Take two or three candidate harnesses, hold the model fixed, and run them against a set of your own representative tasks (a handful of real tickets beats a public leaderboard). Public indices that rank pairs are a starting filter, not the answer for your codebase.
Measure four axes, not one. Resolution rate (does it complete the task), cost per task (not raw tokens, under your real pricing), turn count / latency, and a safety/blast-radius axis (does the harness gate consequential actions). Harness-Bench’s dimensions — completion, tool use, consistency, robustness, security, token usage — are a reasonable checklist to adapt.
Make token measurement honest. Multiple repeats per task; report cost under a stated pricing model with caching assumptions disclosed; separate cached from uncached input. Reject any single-run token comparison, your own included.
Discount first-party parity claims by default — including the model vendor’s. Treat a builder benchmarking its own harness as a hypothesis to replicate, not a result to adopt. The qualifiers (“most configurations,” “within variance,” “on par”) are where the truth lives. This applies to the model vendor’s harness too: “our model works best in our harness” is also a first-party claim, and it is partly true by construction because of post-training coupling.
Know your switching cost before you chase token savings. Because vendor harnesses are tuned to their models via post-training, swapping the harness can quietly impose a tool-format tax — extra reasoning tokens, more mistakes — that eats the savings that motivated the swap. Measure end-to-end cost after the swap, not the advertised per-token rate.
Pick the harness by what currently hurts. One workable decision rule from the practitioner discourse (a heuristic, not a validated taxonomy): choose based on whether your binding constraint is control (data locality, model swappability, no vendor lock-in → third-party / BYOK harness), power (best-in-class capability on your model → likely the matched vendor harness), or cost/constraints (rate limits, runaway spend → whichever harness measures cheapest on your workload). Owning the harness buys flexibility and pays in configuration complexity; the vendor harness buys a co-designed model fit and pays in lock-in.

Open Questions

Does GitHub’s parity hold off-benchmark? The result is on public suites at pass@1 with normalized settings. Whether “effective parity at lower token cost” survives on messy production repositories, with real MCP servers and long-horizon tasks, is unestablished — and is exactly the gap a team should close with its own measurement.
How durable is any harness ranking? When a model upgrade can retire an entire context-management subsystem that a prior harness was built around (the “context anxiety” machinery built for one Claude generation and discarded at the next is one reported instance), harness leaderboards may have a short half-life. How often you should re-evaluate is unclear.
Do orchestration meta-harnesses actually reduce total cost? Owning the orchestration and memory layer is appealing strategically, but whether a meta-harness lowers aggregate spend versus a tuned vendor harness — after accounting for the engineering cost of running it — is not yet shown in public data.
Will automated harness optimization generalize? Meta-Harness’s gains are on bounded benchmarks. Whether outer-loop harness search transfers to large, idiosyncratic production codebases without overfitting to the eval is open.