Scout: What Transfers Into the Loop: Which Human Engineering Practices Survive Agent Adoption

In Brief

• In a small exploratory ThoughtWorks comparison — five batches, one generator model, an LLM judge — prescribing TDD inside the agent loop produced no discernible quality difference at 2.96–8.5× the cumulative session tokens, a multiplier the author says to treat as directional, and the judge sometimes ranked non-TDD output higher.
• The practices transferring most cleanly are the ones whose consumer was always a machine or a future reader — small single-purpose commits, deterministic CI gates, enforced PR templates — and one maintainer’s account says agents comply once the rules are enforced rather than requested.
• Code review is not transferring as ceremony: telemetry from 22,000 developers shows incidents per PR up 242.7% and no-review merges up 31.3%, while a within-reviewer study finds the same reviewers approving more and commenting less as agent-PR exposure grows.
• The emerging replacement moves human judgment upstream to spec and architecture review and pushes determinism into the loop; long-horizon benchmark evidence suggests agents do not maintain regression discipline or persistent plans on their own.
• Maintainability still has no fast oracle — nothing in the harness or the practice canon prices architectural debt on test-passing timescales — and that gap is, for now, the practical case for keeping humans reading code.

Why This Matters

Every team that adopted coding agents inherited a question it rarely asks out loud: how much of the human practice canon should be prescribed to the agent? The default answer has been all of it — TDD instructions in the system prompt, review ceremony on every PR, commit conventions in the context file — on the assumption that what disciplined humans will discipline agents. That assumption now has its first price tags and its first counter-evidence, and the pattern in the data is consistent: practices carry value into the loop when their mechanism transfers, and burn tokens when only their ritual does. Sorting your own canon by that test is cheaper than discovering it from your incident graph.

Key Findings

TDD is the measured case, and the ritual bought nothing detectable

Birgitta Böckeler’s ThoughtWorks experiment is the first controlled attempt to price a prescribed practice inside the loop, and it deserves its influence. She ran TDD and non-TDD workflows across three task sizes — greenfield business logic, an 80% coverage requirement, Sonnet 4.6 generating — and had Opus 4.8 judge the results blind to which workflow produced them (Böckeler).

The verdict: “no clearly discernable difference” in quality between workflows, with the judge more than once ranking non-TDD solutions slightly higher on design and test quality. Mutation scores showed no meaningful separation either. The token multipliers ran 8.5× on small tasks, 2.96× on medium, 4.89× on large — cumulative session totals that sum input, output, and cache traffic across every turn, which the piece cautions likely overstate true dollar cost: “treat the multipliers as directional: TDD reliably cost several times more, how many times exactly is variable.” Her personal conclusion is blunt: “I personally have stopped telling my coding agents to write tests first, let alone do TDD (which I never did, to be honest), until I see evals or other strong arguments that convince me otherwise.”

The mechanism she offers for the null is the interesting part. TDD’s value in human hands includes psychological safety — in her words, “This is very much about managing a human’s fear and giving a human permission to relax” — and that does not transfer to an agent. On TDD’s fast-feedback benefit she is more hedged: the setup did not measure whether agents got stuck debugging more without TDD, but “in my general experience, agents are usually reasonably good at figuring out why a test is red, even without having taken small, deliberate steps to get there.”

The judge’s analysis even found a design cost: non-TDD runs architected up front, while TDD runs “tended to land on whatever shape the first test happened to lock in.” Ivett Ördög’s hypothesis, quoted in the piece, is that step-by-step TDD transcripts are vanishingly rare in training data, so models internally map requirements to code directly rather than through a process.

The caveats are real and Böckeler states them: small sample, quality defined largely by one LLM judge, greenfield tasks only, and no run executed TDD perfectly. One negative result does not close the question. But the prescription side of the ledger is thinner still. The Tweag agentic-coding handbook asserts that “a test becomes a natural language spec that guides the AI toward exactly the behavior you expect” and offers no comparative data (Tweag). An April 2026 workshop paper formalizes TDD phase ordering and bounded repair cycles as machine-readable governance for multi-agent generation — also without quantitative results yet (arXiv 2604.26615). The quantitative results that do exist sit in a neighboring regime: TDFlow, an agentic workflow that frames engineering as resolving human-written tests, reports an 88.8% pass rate on SWE-Bench Lite, a 27.8-point absolute gain over its next-best baseline (Han et al., EACL 2026). That is evidence for handing agents a human-authored test oracle, not for the practice Böckeler measured — the agent authoring and sequencing its own tests first. For that prescription, the burden of proof has moved to its advocates.

Kent Beck’s counter-position is about a different loop — and it maps where pairing went

The obvious counter-position comes from TDD’s own originator. Kent Beck has called TDD a “superpower” when working with AI agents, and reports fighting agents that delete failing tests to make them “pass” (Pragmatic Engineer). His augmented-coding practice keeps TDD enforced at the prompt level, drives one failing test at a time from a plan file, and watches intermediate output for warning signs — undirected feature expansion, test manipulation (Beck).

Read closely, the two positions sample different regimes rather than contradicting each other. Böckeler measured TDD as a delegated ritual inside an autonomous loop, judged after the fact. Beck runs TDD as his own supervision instrument in an interactive session — the test sequence is the leash he holds, and agents deleting tests is precisely why he stays holding it. TDD-as-agent-ceremony has no measured quality case; TDD-as-human-monitoring-protocol is a supervision choice whose cost lands on the human’s attention, not the token bill. Teams should be clear which one they are prescribing.

That same distinction is where pairing landed. The driver role went to the agent; what survives of pairing is Beck’s navigator — a human continuously reading intermediate results and steering. The comparative evidence on whether AI fills the partner’s seat is thin and mixed: a study of knowledge transfer found “developers tend to accept GitHub Copilot’s suggestions with less scrutiny than those from human pair programming partners” (arXiv 2506.04785), while also finding a similar frequency of successful knowledge-transfer episodes across both settings. What weakens when the second pair of eyes is the machine’s own is specifically scrutiny of accepted suggestions, not knowledge transfer wholesale — which is an argument for keeping the human in the navigator seat, not for pretending the agent occupies both.

What transfers cleanly is the discipline whose consumer was always a machine

Commit discipline is the clearest survivor in practitioner accounts — no one has run the comparative experiment — and the case for it is mechanistic: recoverable, attributable, single-purpose history was never about the author’s psychology. Eleanor Berger’s argument for git as an agent checkpointing layer makes the case directly: agents should commit frequently in small logical units, so that “You gain not just the final result, but a complete audit trail of how it got there” (Berger). At agent generation speed, rollback and attribution matter more than they did for humans, not less. The practice transfers intact; only its beneficiary changed, from the future maintainer to the present operator recovering from a bad run.

The open-source boundary shows the same selection effect at project scale. The GitHub Blog’s account of how Nicholas Tindle, founding AI engineer at AutoGPT, handles AI-first contribution volume is a catalogue of ceremony converted into enforcement: PR templates whose violation means automatic closure, coverage thresholds that force agents to write the missing tests themselves, and a rule that resolving a review thread requires linking the fixing commit by full SHA (GitHub Blog). The reported result: “They almost never get pull requests that don’t work anymore.”

The pattern across both is worth stating as a test. A practice transfers into the loop when its feedback signal can be made deterministic and machine-checkable — commit granularity, template compliance, coverage floors, test execution. It fails to transfer when its signal was a human judgment the ritual merely scheduled. The first category should move out of prose instructions and into hooks, CI, and merge gates, where compliance is not a matter of model attention; the second category has to go somewhere else, which is the next finding.

Code review is not transferring — it is breaking, and the replacement moves the gate off the diff

Of the four practices, review carries the largest body of field telemetry, and that telemetry describes deterioration rather than transfer. Faros AI’s 2026 report — two years of telemetry across 22,000 developers and 4,000+ teams — finds incidents per PR up 242.7%, bugs per developer up 54%, median review time up 441.5%, and 31.3% more PRs merging with no review at all, human or agentic (Faros AI). These figures compare each organization’s lowest- and highest-adoption periods, so they are associations; reviewer overload is the report’s own “more likely explanation”: that “reviewers cannot keep pace with the volume of AI-generated code arriving for their attention.”

The within-reviewer evidence is worse, because it shows the gate loosening from the inside — the dynamics behind how review of agent code degrades under sustained volume. Tracking the same 400 reviewers across 11,429 reviews over seven months, approval of agent PRs rose from 30.1% to 36.8% while inline comments fell 22%, a pattern the authors read as “most consistent with reflexive habituation under growing workload rather than rational trust calibration alone” (arXiv 2606.22721).

A study coding 3,100 practitioner documents lands on why this matters more than any other practice’s fate: “review is the control point through which a coding agent’s effect on software is decided” (arXiv 2607.07980).

The replacement discipline visible in practitioner accounts does not abandon review; it re-spends the budget upstream. HumanLayer’s essay on why lights-off software factories fail — written from a failed attempt at running one — prescribes front-loaded review in phases: product review, system architecture, program design, then implementation in vertical slices read in 100–200-line chunks, on the claim that “30 minutes of planning saves hours of review” (HumanLayer). The artifact under human scrutiny becomes intent and structure, where habituation has less surface to erode.

The far end of that relocation now has a documented, priced case. Joel Abenhaim’s specification-first refactor dismantled a lifetime invariant across 189 files of a 717,725-line production TypeScript codebase with no human code review and no pre-existing test oracle: the agent authored a formal spec, refined it over 14 audit cycles, implemented, then verified code against the frozen spec over 17 more — 201 defects corrected before any human ran the program, in three days for $2,430, with no bugs reported across roughly thirty subsequent sessions (arXiv 2608.12440).

One self-reported case establishes feasibility and price, not safety — and convergence against a spec proves agreement with the spec, not correctness of the spec. But it is the exact opposite trade from ceremony-in-the-loop: all human judgment concentrated in the specification, with the checking delegated to fresh sessions of the same agent auditing code against that frozen referent — probabilistic review anchored to a fixed document, plus a conventional compile-and-test loop, not deterministic verification.

The loop does not supply its own discipline over long horizons

The redundancy argument — the harness already compiles, tests, and iterates, so human practice is surplus — holds inside a single task and visibly fails across many. LoopsBench, Microsoft’s long-horizon benchmark of 112 real development tasks structured as dependency DAGs, frames the shift as one where the challenge “moves from harness engineering alone to loop engineering over the harness” (arXiv 2608.00267). Its findings are a list of human disciplines agents did not spontaneously reproduce: test authoring was sparse relative to native suites in every evaluated loop, recorded plans recovered only partial prerequisite structure (Edge F1 of 0.58–0.71 across the closed-source loops, with Claude Code highest at 0.71), regressions on previously completed units persisted across all four loop profiles tested, and the best configuration resolved 25.00% of tasks.

That is the empirical shape of the transfer question. The harness genuinely replaces the ceremony-scale practices: the red-green cycle, the compile check, the local test run. It does not replace the horizon-scale ones: keeping a global plan current, treating completed work as a standing regression obligation, sequencing dependent changes. Those have to be engineered into the loop as machine-enforced structure — released tests, frozen checks, named stopping states, the verification ladder that decides how much autonomy a loop can safely carry — because prompting the agent to “maintain the plan” is the same category of request as prompting it to do TDD.

Stephanie Jarmak’s 314-page monograph is the systems statement of the same position: agents are evaluated as models but deployed as systems, and “many apparent model failures originate elsewhere in the system, while improvements at one layer often fail to propagate to end-to-end outcomes” (arXiv 2608.13867). Its catalogue — 206 reliability records spanning harness, state, permissions, and review interfaces, 193 of them gated practices — reads as the practice canon being rebuilt from scratch as an operations discipline. The replacement for the rituals that did not transfer is not fewer practices; it is different ones, attached to the system rather than requested of the model.

Maintainability has no oracle, and that bounds the whole program

The sharpest limit on practice transfer is the quality signal no one can automate. The HumanLayer essay’s core argument is that this is a model-training problem: reinforcement optimizes for passing tests, and “Tests give you feedback in seconds, but the cost function of bad architecture is measured in weeks, months, maybe even years” (HumanLayer). There is no fast oracle for maintainability, so no reward signal during training and no deterministic gate during operation. The essay’s account of its author’s own lights-off experiment ends with the codebase rewritten from scratch after months of accumulating agent-generated debt.

Faros’s churn figure (code churn up 861% under high AI adoption, per Faros AI) is at least consistent with debt accumulating faster than it is being priced, though churn is a proxy and the report does not decompose it. On the proposed fix, verifier models judging code quality inside benchmarks, the HumanLayer essay is skeptical in a way practitioners should sit with: “If a model could reliably tell good code from bad, it might have written the good version to begin with.”

This closes the transfer test from the other side. Practices whose signal is deterministic move into the harness. Practices whose signal is human judgment about intent move upstream into spec review. The signal for long-run structural quality currently fits neither bucket — which means human code reading, targeted at architecture rather than line-level correctness, is the one ceremony with no replacement on any shelf. The teams deleting it are not ahead; they are unpriced.

Practical Implications

1. Stop prescribing TDD to autonomous loops by default. The one controlled comparison found ceremony cost without measurable quality gain, at several times the session tokens — 2.96–8.5×, a directional figure (Böckeler). Handing the agent human-written tests to resolve is a different regime with real quantitative support. If TDD earns its place in your harness, it should do so through an eval you ran, not through inheritance.
2. Keep TDD where it is a supervision instrument. Beck’s use — interactive sessions, one failing test at a time, human watching for test manipulation (Beck) — is a different practice from loop ceremony, and its cost falls on attention you were spending anyway.
3. Measure quality with instruments, not rituals. Böckeler’s alternatives point at the artifact rather than the process: mutation testing, static analysis and modularity review, refactoring triggers tied to concrete metrics (Böckeler).
4. Move commit discipline into agent instructions and harness defaults now. Frequent single-purpose commits give you rollback, attribution, and an audit trail at agent speed (Berger); requiring fixing-commit SHAs in review-thread resolution makes “resolved” checkable (GitHub Blog).
5. Convert every review rule with a deterministic check into a gate. Templates that auto-close on violation and coverage thresholds that force test authorship changed agent behavior where prose requests did not (GitHub Blog).
6. Re-spend the human review budget upstream and shrink what remains downstream. Review intent and architecture before generation; keep line-level reading to vertical slices small enough that scrutiny stays real (HumanLayer). Habituation data says a reviewer facing unbounded agent diffs will approve more and inspect less (arXiv 2606.22721).
7. Engineer regression obligations into the loop explicitly. Long-horizon evidence shows agents let completed work break and plans go stale (arXiv 2608.00267); completed units should carry standing, automatically re-run checks the loop cannot argue with.
8. Budget human architecture reading as a permanent line item. Until a maintainability oracle exists, structural quality is the one signal neither the harness nor upstream spec review captures (HumanLayer).

Open Questions

• Does the TDD null replicate? One exploratory study, one generator model, greenfield tasks, an LLM judge defining quality. A replication on brownfield work, with mutation-score and defect-escape instruments rather than judge rankings, would settle whether the null is about TDD or about the tasks.
• Where is the crossover between spec-audit convergence and human review? Abenhaim’s case prices one point — $2,430 for a sharply specifiable invariant migration (arXiv 2608.12440). At what task ambiguity does frozen-spec verification stop substituting for human judgment, and what does the escape rate look like when the spec itself is wrong?
• Does upstream review habituate the same way diff review did? The habituation study measured the diff gate. Whether spec and architecture review hold their scrutiny under volume, or merely reset the erosion clock, is unmeasured.
• Can maintainability get a fast oracle? Benchmark builders are trying verifier models and mutation-based checks; the skeptic’s objection — a model that could reliably judge good code would write it — has not been answered empirically in either direction.
• What is the pairing evidence worth? The navigator role survives in practitioner accounts, but the comparative literature is thin and mostly Copilot-era. Whether continuous human monitoring of intermediate agent output measurably reduces escaped defects, versus reviewing at checkpoints, is an open experiment.

Sources

1. TDD inside the agent loop — theater or actual value? — Birgitta Böckeler, martinfowler.com, August 10, 2026
2. Test-Driven Development — Agentic Coding Handbook — Tweag
3. TDD Governance for Multi-Agent Code Generation via Prompt Engineering (arXiv 2604.26615) — Hasanli, Siddeeq, Khanal, Kotilainen, Mikkonen, Abrahamsson
4. TDFlow: Agentic Workflows for Test Driven Development (EACL 2026) — Han, Maddikayala, Knappe, Patel, Liao, Barati Farimani, March 2026
5. TDD, AI agents and coding with Kent Beck — The Pragmatic Engineer, 2025
6. Augmented Coding: Beyond the Vibes — Kent Beck, June 2025
7. From Developer Pairs to AI Copilots: A Comparative Study on Knowledge Transfer (arXiv 2506.04785) — Welter, Schneider, Dick, Weis, Tinnes, Wyrich, Apel
8. Use Git for Automated Checkpointing — Eleanor Berger, October 2025
9. Your contributors are AI-first now. Is your project? — GitHub Blog, on AutoGPT founding AI engineer Nicholas Tindle’s practices, August 12, 2026
10. The AI Engineering Report 2026: The AI Acceleration Whiplash — Ten Takeaways — Faros AI, April 2026
11. Habituation at the Gate: Rising Approval and Declining Scrutiny in Human Review of AI Agent Code (arXiv 2606.22721) — Yu, Liu, Jiang, Jia, Wang, Qian, Chen
12. 3100 Opinions on Code Review in an AI World: Building Causal Theory from Practitioner Discourse (arXiv 2607.07980) — Agarwal, Miller, Kästner, Vasilescu
13. Why Software Factories Fail (wsff.md) — HumanLayer
14. Specification-first convergence with an AI coding agent: a case study of dismantling a core architectural invariant across 189 files (arXiv 2608.12440) — Joel Abenhaim
15. LoopsBench: From Harness Engineering to Loop Engineering in Coding Agent Evaluation (arXiv 2608.00267) — Li, Fang, Feng, et al.
16. Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Model (arXiv 2608.13867) — Stephanie Jarmak