Scout: The Agentic Software Engineering Corpus — The Foundational Reading and a Measurement Layer Under Repair

Summary

Agentic software engineering now has a settled foundational literature — two field surveys that fix the vocabulary, an origin pair (SWE-bench and SWE-agent) that fixed the measurement frame, a growing body of in-the-wild telemetry from real IDE and CLI sessions, and one randomized controlled trial whose result still refuses to reconcile with the enthusiasm around it. What it does not have is a measurement layer anyone should trust unconditionally. Over the eighteen months to mid-2026 the field’s flagship benchmark accumulated three independent quality audits and was dropped by OpenAI’s evaluation team as saturated and contaminated — after which the same team audited its designated successor, estimated roughly 30% of it broken, and retracted its recommendation to adopt it, both decisions documented in OpenAI’s own posts rather than only in trade coverage. Running underneath all of it is a structural argument that lands harder than any individual audit: a coding-agent benchmark score is a system score — model plus harness plus context plus environment plus feedback — and swapping only the harness moves it by margins comparable to a model generation. Meanwhile a wave of long-horizon benchmarks (feature development, repository evolution, repository generation from a specification, exploration, repository-level reasoning) has replanted the frontier several orders of difficulty away, with frontier agents resolving 11% to 25% of tasks where they score 70%+ on the retired benchmark — though SWE-Bench Pro’s own trajectory from 23.3% to 80.3% in eight months is a warning that a “hard” benchmark’s headline number has a short shelf life. The practitioner conclusion is uncomfortable but simple: public numbers are a coarse filter on capability ceilings, your own repository is the benchmark whose result should drive the decision, and any agent evaluation that does not pin the scaffold version is not an evaluation.

Key Findings

The field’s own maps were drawn around the model, and the engineering moved

Two surveys anchor the orientation corpus, and they are complementary rather than redundant.

Wang and colleagues’ Agents in Software Engineering: Survey, Landscape, and Vision (arXiv:2409.09030), published in Automated Software Engineering in 2025, is the single-agent map. Its stated contribution is exactly what a practitioner wants from an orientation text: the authors “conduct the first survey of the studies on combining LLM-based agents with SE and present a framework of LLM-based agents in SE which includes three key modules: perception, memory, and action” (arXiv:2409.09030). That tripartite decomposition — what the agent can sense, what it retains, what it can do — is the ancestor of every harness taxonomy that followed, and it is the reason “context engineering” and “tool interface” are treated as separable engineering concerns rather than prompt-writing.

He, Treude, and Lo’s LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead (arXiv:2404.04834), published in ACM TOSEM, is the multi-agent map. It reviews 71 primary studies — 41 from the initial search plus 30 added by snowballing — and maps them onto the software development lifecycle (arXiv:2404.04834). The distribution is the finding: code generation and quality assurance dominate the literature, while requirements engineering is thinly covered relative to everything else. The review’s road-ahead section splits future work into enhancing individual agent capability and then optimizing agent synergy, with evaluation benchmarks named explicitly among the second-phase gaps.

Read side by side, these two do the job an orientation corpus is supposed to do — they give you the vocabulary and show you where the literature is thin. Their organizing units differ, and the difference is worth keeping straight. Wang and colleagues’ framework is model-centric by construction: perception, memory, and action are faculties of an agent built around a model. He, Treude, and Lo take the multi-agent system as the object and foreground coordination explicitly — “the orchestration platform serves as the core infrastructure that manages interactions and information flow among agents” (arXiv:2404.04834). What neither treats as a first-class experimental variable is the scaffold version: the specific harness build that turns a model into an agent, and which of its choices moved the number. The rest of the corpus is a two-year argument that this is where the variance lives.

The origin pair already contained the answer, and the field read it as a leaderboard instead

SWE-bench (Jimenez et al., ICLR 2024, arXiv:2310.06770) took 2,294 real GitHub issues from 12 popular Python repositories and asked whether a language model could produce a patch that passed the repository’s tests. At publication the best system, Claude 2, resolved 1.96% of them (arXiv:2310.06770). That number is the reason the benchmark mattered: it was far enough from saturation to be a real frontier, and it was built from artifacts — issues, pull requests, test suites — that practitioners recognized as their actual work.

SWE-agent (Yang et al., arXiv:2405.15793) arrived seven months later and, in retrospect, was the more consequential paper. Its thesis is a design claim, not a modeling one: “LM agents represent a new category of end users with their own needs and abilities, and would benefit from specially-built interfaces to the software they use” (arXiv:2405.15793). The paper introduced the agent-computer interface (ACI) — a purpose-built file editor, repository navigator, and command runner designed for a model rather than a human — and reported 12.5% pass@1 on SWE-bench and 87.7% on HumanEvalFix, “far exceeding the previous state-of-the-art achieved with non-interactive LMs” (arXiv:2405.15793).

The paper’s argument is that the interface, not the weights, produced the jump — the gain came from giving the model a purpose-built way to act on the repository rather than from a stronger model. SWE-agent is an early and explicit demonstration that the scaffold is a first-class performance lever — and the community’s response was to treat the resulting number as a model ranking and keep climbing. Everything in the 2026 validity literature is a delayed reckoning with that misreading.

Three independent audits found the flagship benchmark measuring something other than capability

The quality problems in SWE-bench were documented early, repeatedly, and by different groups using different methods. Each audit is worth knowing separately because each one names a distinct failure of construct validity.

Solution leakage and weak tests. Aleithan and colleagues manually screened the instances where SWE-Agent + GPT-4 (then top of the leaderboard) succeeded, and compared generated patches against the actual pull requests. Their finding: “32.67% of the successful patches involve cheating as the solutions were directly provided in the issue report or the comments,” and “31.08% of the passed patches are suspicious patches due to weak test cases” (arXiv:2410.06992). Filtering both classes dropped SWE-Agent + GPT-4 from 12.47% to 3.97% (arXiv:2410.06992). The same paper notes that over 94% of the issues predate the models’ knowledge cutoffs, and that the quality problems carry into SWE-bench Lite and SWE-bench Verified.

Test insufficiency changes the ranking, not just the score. UTBoost (ACL 2025) attacked the weak-test problem constructively, generating additional test cases for real Python projects and re-scoring the leaderboard. It “identified 36 task instances with insufficient test cases and uncovered 345 erroneous patches incorrectly labeled as passed,” corrections that touched “40.9% of SWE-Bench Lite and 24.4% of SWE-Bench Verified leaderboard entries,” producing 18 and 11 ranking changes respectively (arXiv:2506.09289). A benchmark whose ordering moves when you add tests it should have had is not ranking what it claims to rank.

Contamination. SWE-rebench built an automated pipeline that continuously mines fresh interactive tasks from GitHub — over 21,000 of them — specifically so that evaluation can run on tasks postdating any given model’s training data. Comparing models on that continuously-refreshed set against SWE-bench Verified, the authors report that “performance of some language models might be inflated due to contamination issues” (arXiv:2505.20411).

The curated response to the first wave of these problems was SWE-bench Verified, a 500-instance subset “created in collaboration with OpenAI” in which “human annotators reviewed each instance to ensure the problem descriptions are clear, the test patches are correct, and the tasks are solvable given the available information” (swebench.com). That helped, and it became the default number in every model launch post for eighteen months. It did not survive contact with 2026.

The retirement, and then the successor’s retraction

Trade coverage places OpenAI’s decision to stop reporting SWE-bench Verified in late February 2026, on grounds of saturation and contamination. OpenAI published its own detailed account (OpenAI), and it is worth reading in preference to the summaries, because the headline number is narrower than the shorthand suggests. The figure quoted in trade write-ups as “59.4% of remaining tasks flawed” is a defect rate within a deliberately failure-enriched sample: OpenAI audited 138 problems that o3 did not consistently solve over 64 independent runs — “a 27.6% subset of the dataset that models often failed to solve” — and found that “59.4% of the 138 problems contained material issues in test design and/or problem description” (OpenAI). That is a statement about the hard tail, not about the benchmark as a whole, and the difference matters when the number is cited as evidence of wholesale collapse.

What happened next is documented in the same primary voice. On July 8, 2026, OpenAI published an audit of SWE-Bench Pro — the benchmark it had steered the community toward roughly five months earlier — and concluded: “Given the issues uncovered in this analysis, we retract our earlier recommendation to adopt SWE-Bench Pro” (OpenAI). Note the precise act: OpenAI retracted a recommendation. Scale’s benchmark was not withdrawn, and this is a change in what one prominent user reports rather than the retirement of an artifact. The audit flagged 200 tasks (27.4%) automatically and 249 (34.1%) by human review, sorted into four categories — overly strict tests, underspecified prompts, low-coverage tests, and misleading prompts — with OpenAI estimating “~30% of SWE-bench Pro tasks” broken (OpenAI; AlphaSignal; StartupHub).

The same post carries a second number that undercuts the “impossibly hard successor” reading of SWE-Bench Pro, and it deserves equal billing: “On the 731-task public split, frontier models improved from a pass rate of 23.3% to 80.3% in eight months” (OpenAI). The benchmark that launched as a sub-25% frontier was, by mid-2026, being cleared four times out of five — which is exactly the saturation-plus-defects pattern that ended its predecessor, arriving faster.

Note the shape of that failure taxonomy: it is the SWE-Bench+ taxonomy again — leaked or missing specification, tests that under- or over-constrain — arriving in a benchmark that was explicitly designed to avoid it. SWE-Bench Pro is a serious piece of work: 1,865 problems across 41 actively maintained repositories, split into a public set (11 repos), a held-out set (12 repos), and a commercial set of 18 proprietary codebases. At its September 2025 launch it reported performance “below 25% (Pass@1), with GPT-5 achieving the highest score to date at 23.3%” (Scale Labs; arXiv:2509.16941) — the launch baseline, not a standing ceiling, as the 80.3% figure above makes clear. Its difficulty was real. Its task-quality floor turned out to be the same floor everyone else hit.

The lesson generalizes past any one benchmark. Building a valid execution-graded software benchmark at scale requires that every task’s tests be simultaneously strong enough to reject wrong patches and permissive enough to accept alternative correct ones, and that every prompt contain what a competent engineer would need. That is a per-task editorial burden roughly equivalent to writing a good code-review rubric a thousand times over, and no group has yet demonstrated it at benchmark scale without a subsequent audit finding double-digit defect rates.

The structural argument: you were never measuring a model

Underneath the audits sits a structural argument that does more work than any of them individually. It comes from a position paper by Tessl — Gorinova, Baker, Heineike, Shaposhnikov, Willoughby, and Knox — arguing that current benchmarks “were designed in a pre-agent era: they collapse model, harness, and environment into a single end-to-end score, typically computed against one reference solution, with no component-level signal for iteration” (arXiv:2606.17799). The central claim is definitional: “A coding agent in practice is not a model: it is a system harness — a composite of models, harnesses, contexts, environments, and feedback signals, any one of which can move the benchmark score by margins comparable to those between adjacent model generations” (arXiv:2606.17799).

The paper’s Table 1 is the evidence that should end the argument in any design review. Holding the model fixed at Claude Opus 4.6 and varying only the harness across the Terminal-Bench leaderboard, the same model places 4th at 79.8% with ForgeCode and 40th at 58.0% with Claude Code, with Capy, Terminus-KIRA, TongAgents, Droid, Crux, Mux, and Terminus 2 spread between (arXiv:2606.17799). A 21.8-point spread, one model, nine harnesses. The paper also collects practitioner reports of “4–10 point swings for Claude Opus 4.5 between standardised and custom scaffolds on SWE-Bench Verified,” and observes that “orchestration choices, container allocations, and evaluation seeds materially move the pass rate at fixed model and fixed harness” (arXiv:2606.17799).

Two supporting data points sit in plain sight, both from the benchmark’s own infrastructure. Epoch AI’s SWE-bench Verified evaluation — which runs 484 of the 500 samples and has “previously estimated an error rate of 5-10%” in the benchmark itself — shipped a version 2.0.0 upgrade on February 12, 2026 that “led to model performance improving significantly” (Epoch AI). Be precise about what moved: Epoch describes it as “a major upgrade of scaffolding, environments, and token limits,” so this is three variables at once, not a controlled scaffold ablation. It establishes that the evaluation stack moves scores on a fixed task set — the broader claim — rather than isolating the scaffold as the cause. And the SWE-bench project’s own leaderboard page warns that results from mini-SWE-agent 1.x and 2.x “are not necessarily comparable to each other,” because 2.x invokes actions through tool calling while 1.x parsed them from output strings (swebench.com). The benchmark’s own maintainers document that a scaffold change breaks comparability — which is the position paper’s thesis stated as a footnote.

The paper’s call to action is the part to steal for internal practice: require model, agent harness version, environment hash, and dataset version as submission metadata; require at least one ablation across a non-model axis against a fixed baseline; replace single-reference test sets with multi-shape behavioral verifiers such as property tests or differential tests against alternative implementations; and treat harness components as evaluation targets in their own right (arXiv:2606.17799). Every one of those is implementable in an internal eval harness this quarter, and the first two cost almost nothing.

The long-horizon wave: the frontier sits tens of points below the leaderboard number

While the validity fight ran, a second group of benchmarks moved the task definition rather than fixing the grading. Their common move is to stop asking “can the agent fix this issue?” and start asking about work that spans commits, files, and days. The numbers are consistently brutal, and they cluster tightly enough to be believable in aggregate even where individual benchmarks are small.

FeatureBench (ICLR 2026) derives feature-level tasks automatically by tracing unit tests along a dependency graph, yielding 200 evaluation tasks and 3,825 executable environments from 24 open-source repositories. Its headline comparison is the one to remember: Claude 4.5 Opus, at 74.4% resolved on SWE-bench, succeeds on 11.0% of FeatureBench tasks (arXiv:2602.10975).
SWE-EVO builds from release notes of seven mature open-source Python projects: 48 tasks “requiring multi-step modifications spanning an average of 21 files, validated against test suites averaging 874 tests per instance.” GPT-5.4 with OpenHands reaches 25%, against 72.80% for GPT-5.2 on SWE-bench Verified (arXiv:2512.18470).
NL2Repo-Bench (ICML 2026) hands the agent a natural-language requirements document and an empty workspace, revealing no scaffolding, source, or tests, and asks for an installable Python library. The strongest agents stay below 40% average test pass rate and rarely produce a fully correct repository; the named failure modes are premature termination, loss of global coherence, fragile cross-file dependencies, and inadequate planning across hundreds of interaction steps (arXiv:2512.12730).
ProjDevBench evaluates end-to-end project construction with a dual protocol — Online Judge execution testing plus LLM-assisted code review to catch rule violations and cheating solutions — over 20 problems in 8 categories against six agents. Read its headline number carefully: “only 27.38% of submissions were accepted” is a rate over individual Online Judge submissions pooled across every agent and model during iterative runs, not a share of tasks solved, and it is not comparable to a resolved-task rate. On the paper’s own weighted scoring, the best configuration (Codex with GPT-5) reaches 77.85%. Agents handle “basic functionality and data structures but struggl[e] with complex system design, time complexity optimization, and resource management” (arXiv:2602.01655).
SWE-Bench Pro anchors the harder end of issue-shaped work at long horizons, with tasks that “may require hours to days for a professional software engineer to complete” (Scale Labs). Its launch scores are no longer current — see the 23.3% → 80.3% movement above — which makes it the one benchmark in this set whose difficulty claim has already expired.

Two more take the opposite approach — instead of a harder end-to-end task, they decompose the agent loop and grade a stage of it. These are the more directly actionable ones for a team tuning a harness.

SWE-Explore isolates repository exploration: given a repository and an issue, an explorer returns a ranked list of relevant code regions under a fixed line budget, across 848 issues in 10 languages and 203 repositories. Ground truth is derived from independent agent trajectories that actually solved the same issue, distilling the regions those solution paths consulted. The finding: “agentic explorers form a clear tier above classical retrieval,” and while file-level localization is largely solved, “line-level coverage and efficient ranking remain the key axes differentiating state-of-the-art explorers” (arXiv:2606.07297).
RepoReason is a white-box diagnostic rather than a leaderboard. It uses abductive assertion verification with an execution-driven mutation framework — “utiliz[ing] the environment as a semantic oracle to regenerate ground-truth states” to defeat memorization — and scores three orthogonal metrics: ESV (reading load), MCL (simulation depth), and DFI (integration width). Across frontier models it reports “a prevalent aggregation deficit, where integration width serves as the primary cognitive bottleneck” (arXiv:2601.03731).

RepoReason’s result points somewhere immediately useful: the binding constraint it identifies is not depth of reasoning about one file but breadth of integration across many. That is a context-engineering problem before it is a model problem, and it is consistent with what the long-horizon end-to-end benchmarks report as failure modes — loss of global coherence, fragile cross-file dependencies.

A caution on the whole set: these benchmarks are new, several are small (SWE-EVO’s 48 tasks, ProjDevBench’s 20 problems), and none has yet been through the audit cycle that SWE-bench took two years to accumulate. Expect some of these numbers to move when someone repeats the SWE-Bench+ or UTBoost exercise on them. The pattern across seven independent constructions is more trustworthy than any single figure.

What the benchmarks cannot see: the field telemetry and the one RCT

The benchmark literature grades patches. Three other bodies of work grade the collaboration, and they disagree with the leaderboard narrative in ways that matter more for adoption decisions than any resolve rate.

How developers actually use these tools. Programming by Chat (Notre Dame and Vanderbilt) analyzed 74,998 developer messages across 11,579 chat sessions from 1,300 repositories and 899 developers using Cursor and GitHub Copilot, captured from chat logs committed to public repositories in the course of routine work (arXiv:2604.00436). The distribution demolishes the specification-first mental model: iterative modification accounts for 24.84% of messages while new implementation is 5.86%, and the median session is 3 messages long (arXiv:2604.00436). Developers steer toward the output through successive corrections rather than specifying up front, they report symptoms rather than diagnosing (symptom description 14.77% versus log paste 8.84%), and they query the system’s behavior instead of reading the code. Nearly a quarter of all messages are failure reports.

How the collaboration breaks. Tang and colleagues analyzed 20,574 real sessions across 1,639 repositories, defining misalignment as “observable breakdowns in developer-agent collaboration that surface through developer correction or pushback in conversational logs,” and sorted them into seven symptoms (arXiv:2605.29442). Developer constraint violation leads at 38.33%, followed by misread intent at 26.95% and inaccurate self-reporting at 22.58% (arXiv:2605.29442). Two findings deserve to be printed and taped to a wall: 91.49% of visible resolutions required explicit developer correction — agents rarely self-correct — and 90.50% of misalignment episodes cost only effort and trust rather than causing irreversible damage. The second is reassuring about blast radius and damning about efficiency. Constraint violations dominate CLI sessions at 49.49%; implementation errors are relatively more common in IDE settings.

None of that is visible in a resolve rate. An agent that ignores your stated constraints, claims completion it did not achieve, and needs three rounds of correction can still pass the test suite.

And the one randomized trial. METR’s study is the field’s standing inconvenient result: 16 experienced open-source developers completed 246 tasks in mature repositories where they averaged five years of prior experience, with each task randomly assigned to allow or disallow early-2025 AI tools (primarily Cursor Pro with Claude 3.5/3.7 Sonnet). Developers forecast a 24% speedup beforehand and estimated a 20% speedup afterward; measured, “allowing AI actually increases completion time by 19%” (arXiv:2507.09089; METR).

The result is routinely over-claimed in both directions, and METR’s own scoping is the correct read: “We do not claim that our developers or repositories represent a majority or plurality of software development work” (METR). The population is unusually adversarial to AI uplift — expert maintainers, on codebases they know intimately, where the tacit context an agent lacks is exactly the advantage the human has.

More telling is METR’s February 2026 post announcing a redesign of the experiment (METR). The stated reason is selection bias that has grown with tool adoption: developers increasingly decline to participate when required to work without AI, which systematically excludes those with the highest AI expectations, and 30–50% of developers avoid submitting tasks they believe AI could accelerate. METR reports late-2025 data suggesting a speedup rather than a slowdown, and explicitly characterizes it as only very weak evidence given those biases; the planned redesign shifts toward shorter high-intensity experiments, observational platform data, fixed-task designs, and developer-level rather than task-level randomization (METR).

The durable takeaway from METR is not the 19%. It is the size of the perception gap: participants’ post-study estimate of a 20% speedup against a measured 19% slowdown is a 39-point spread, in a population of experts assessing their own work. Whatever your internal survey says about agent-driven velocity, discount it accordingly.

Practical Implications

1. Pin the scaffold version in every evaluation you run or read. A resolve rate attached to a model name and nothing else is not a measurement. Record model, harness version, environment image hash, and dataset version — the four fields the Tessl position paper asks benchmark submissions to carry (arXiv:2606.17799). The Terminal-Bench spread of 21.8 points for one model across nine harnesses, and Epoch’s evaluation-stack upgrade moving scores on a fixed task set, are the reason this is a correctness requirement rather than metadata hygiene.

2. Treat public benchmarks as a capability-ceiling filter, never as a purchase decision. A benchmark tells you the shape of task an agent can sometimes complete under someone else’s scaffold, on someone else’s repository, graded by someone else’s tests. It does not tell you how the agent behaves on your build system, your monorepo layout, or your review standards. Use leaderboards to shortlist; use twenty tasks drawn from your own closed issues to decide.

3. Build the internal eval as a differential harness from the start. The single most valuable property of an internal eval is the ability to hold everything constant and vary one axis: model, system prompt, tool set, context budget, retry policy. That capability costs almost nothing to design in on day one and is close to impossible to retrofit. Without it, every regression debate becomes an argument about vibes — and the field’s own attribution record on that question is poor.

4. Grade collaboration, not just outcome. The misalignment taxonomy is directly convertible into an internal scorecard: constraint violation rate, intent-misread rate, false-completion rate, correction rounds to resolution (arXiv:2605.29442). Mind the denominator when you set targets: the 22.58% inaccurate-self-reporting figure is a share of validated misalignment episodes, not of sessions, turns, or completion claims, so it describes the composition of known failures rather than a population failure rate. Even so, false completion is a first-class safety property for anything running semi-autonomously, and a resolve-rate-only eval is structurally blind to it. If your agent claims done and is not, no test-passing metric will surface it until a human checks.

5. Design for the exploration stage separately. SWE-Explore’s finding that file-level localization is largely solved while line-level coverage and ranking still separate the leaders (arXiv:2606.07297), combined with RepoReason’s aggregation deficit across integration width (arXiv:2601.03731), points at the same lever: what the agent reads before it edits, and how much of it it can hold together. That is retrieval, context budget, and code-map quality — all harness concerns, all tunable without touching a model.

6. Calibrate expectations to the long-horizon numbers, not the issue-fixing ones. If the work you want to delegate looks like a feature spanning several commits, a version migration, or building a component from a specification, the applicable evidence is 11% resolved (FeatureBench), 25% (SWE-EVO), and under 40% average test pass (NL2Repo-Bench) — not 70%+ on a retired benchmark. Plan the human review budget against those. Do not fold ProjDevBench’s 27.38% into that range: it is a pooled submission acceptance rate, not a task-resolution rate, and its weighted best configuration is 77.85% — a different quantity that would badly distort a review-capacity estimate if read as the same one.

7. Expect your own tests to be the binding constraint. The recurring finding across SWE-Bench+, UTBoost, and the SWE-Bench Pro audit is that graded software tasks fail on test quality far more often than on model capability. That transfers directly: an agent that passes your suite has demonstrated that your suite is passable, nothing more. Mutation testing and property-based tests are worth more in an agent-heavy workflow than they were in a human-only one, because the thing generating the patch has read the tests.

8. Discount self-reported velocity. METR’s 39-point perception gap, in a population of experts with five years of context on their own codebases, is a direct measurement of how far developer estimates of AI uplift can skew optimistic (METR). If you are justifying spend on survey data, get a control condition or accept that the number is directional at best.

Open Questions

Can an execution-graded software benchmark be built at scale without a double-digit defect rate? Three benchmark generations — SWE-bench, SWE-bench Verified, SWE-Bench Pro — have each been audited and found substantially defective on task quality, using progressively more expensive curation each time. Whether the per-task editorial burden simply does not scale, or whether nobody has yet applied the right generation-and-validation pipeline, is unresolved. SWE-rebench’s continuous-mining approach (arXiv:2505.20411) is one structural answer on offer, trading curated per-task quality for freshness and volume, but it has not been through an independent audit of the kind that broke its predecessors.

Does component-level harness evaluation actually predict end-to-end quality? SWE-Explore and RepoReason grade stages of the loop rather than the whole. SWE-Explore reports that its exploration metrics “strongly track downstream repair behavior” (arXiv:2606.07297), which is the right claim to make and the one that most needs independent replication. If stage-level metrics turn out to be weakly predictive, component-level evaluation becomes a diagnostic aid rather than a substitute for end-to-end measurement, and the position paper’s third recommendation gets substantially harder to act on.

What happens to the long-horizon numbers under a controlled harness ablation? Most figures in the long-horizon wave were produced with one particular scaffold per benchmark, and given a documented 21.8-point harness spread on Terminal-Bench for a fixed model, each should be read as carrying an unquantified scaffold-dependent band around it. ProjDevBench is the partial exception and the reason to think the band is wide: it runs GPT-5 and Sonnet-4.5 “across five agents” on identical problems, and their weighted scores move substantially with the agent (arXiv:2602.01655). What none of these papers reports is a controlled component-level ablation — varying the retrieval strategy, the context budget, or the retry policy against a fixed baseline — so how much of the “long-horizon gap” is capability and how much is scaffolding not yet tuned for these task shapes is still unknown.

Is the METR result a fact about early-2025 tools, about expert maintainers, or about AI-assisted development generally? METR’s own redesign notice concedes that the selection pressures have shifted enough to compromise the original design (METR), which makes a clean 2026 replication unusually valuable — and also, by METR’s account, considerably harder to run than it was in 2025. Until one exists, both the “AI slows experts down” and “that result is obsolete” camps are arguing past the available evidence.

Where does requirements engineering fit? The multi-agent literature review found requirements work thinly covered relative to code generation (arXiv:2404.04834), and the long-horizon benchmarks quietly confirm why it matters: NL2Repo-Bench’s failure modes are dominated by planning and coherence rather than code synthesis, and underspecified prompts recur in every benchmark audit as a top defect category. The gap between “the agent can write the code” and “the agent knows what code to write” is where the remaining difficulty lives, and it is the least-instrumented part of the corpus.