Scout: CI Is the Next Constraint: Test Selection and Review Capacity After Agents Write the Code

In Brief

• Anthropic reports CI job volume up 25x in six months with Claude authoring about 80% of code; its test-impact-analysis service, a single process, got 70 days, 29 days, and under a day out of three patches before a three-week rebuild made its listeners stateless workers writing to a shared store.
• Kiro’s autonomous mode opened 94 PRs addressing 87 Kani and CBMC issues in two months, against 94 issues the team had addressed in the prior 22; 37% have merged, 60% sit open, and the vendor attributes the gap to review capacity rather than quality.
• Flaky tests change category under agents: Tessl’s Justin Cormack argues agents may have learned from training data that developers ignore flakes, Clipboard Health found every PR in two large repositories hit a flake before a six-week cleanup, and one documented agent “fix” replaced a test with a weaker one.
• The review-queue designs that scale in these accounts are protocols rather than people: GitHub issue labels as per-stage queues, validation that runs before a PR opens, merge queues as the final integration gate, and GitHub’s new per-user open-PR caps.
• Nobody in these accounts publishes CI compute cost per agent PR; the cost figures that exist ($4.79 in API cost and 13.8 minutes per DoorDash flag cleanup, about $120,000 in token spend plus three weeks of a developer’s time for GitHub’s port, 200–280 implementation hours for Kiro’s 94 PRs) are model spend and engineer time, and exclude the review and CI spend that no account prices.

Why This Matters

The June evidence on human review degrading under agent volume covered the people side of the gate; the September accounts cover the machinery behind it. Anthropic states the sequence plainly: “Writing code is no longer the constraint, and once PR review gets accelerated, CI starts feeling the pressure.” Each stage a team unblocks moves the backlog to the next one, which was sized for humans typing the code. Test selection, flake policy, review-queue shape, and the CI bill now decide whether agent throughput becomes merged software or an open-PR backlog. These accounts are vendor-authored and self-reported; they are still the first set detailed enough to design against.

Key Findings

The constraint moved downstream, and the September numbers say by how much

The Kiro autonomous-mode account is the cleanest statement of the problem because the vendor says it outright. Kiro was pointed at the Kani Rust Verifier and CBMC repositories, which held 916 open issues in November 2025. It opened 94 PRs in two months. Per the post, “Kiro can generate PRs faster than the team can review them,” and the 56 PRs still open from that cohort reflect review capacity, not quality (Kiro). The agent had no merge permission; the human gate held, and the backlog grew behind it.

Faros’s telemetry gives the population view. Its 2025 report, on more than 10,000 developers across 1,255 teams, found that high-adoption teams “merge 98% more pull requests” while “PR review time increases by 91%” (Faros, 2025). The 2026 edition, on 22,000 developers, reports review time up 441.5% and 31.3% more PRs merging with no review at all (Faros, 2026). Faros sells the measurement product, and both reports are observational rather than controlled: the 2025 edition measures each team’s change between its lowest- and highest-adoption quarters, standardized within its company, and correlates those changes across teams; the 2026 edition tracks metric change between each organization’s lowest- and highest-adoption periods. The direction matches the vendor accounts.

One dissent is worth carrying. Zhimin Zhan reads the Anthropic post as evidence against full automation: “even at Anthropic, where Claude tokens (with the highest-grade AI model) are essentially free, they could not achieve 100% AI coding” (Zhan). On that reading the 80% figure is a ceiling as well as a milestone, and the remaining 20%, plus review and CI operation, is where the engineers’ time goes.

Test selection is the CI lever that works, and its own service is what falls over

Anthropic’s test-impact-analysis (TIA) service decides which tests run on each change, using past results and package relevance. Two components: a listener that records results from every CI run, and a selector that reads that history to pick tests for each open PR (Anthropic). The listener held its state in a single process. At 25x job volume, the single process became the thing that broke.

The patch sequence is the useful part for anyone running a similar service. A bigger machine bought about 70 days. Sharding the listener by package bought 29 days. Daily restarts, after the process started hitting its memory limit by mid-afternoon on most weekdays, bought less than a day.

The rebuild took “three weeks for a single engineer,” which the post says would have been closer to a quarter a year earlier, and left any worker able to append any result to a journal and move on, with “a small separate consumer process” rolling the journal into the per-test history the selector reads (Anthropic). The closing advice: “Keep state out of the process from the start,” and “Assume your architecture will be at a 25x load within two quarters.”

None of this is new technique. Meta described predictive test selection in 2018: a gradient-boosted model trained on prior changes and test outcomes that caught more than 99.9% of regressions before they reached trunk while running about a third of the tests that transitively depended on the change (Meta). Stripe’s Minions pipeline runs selective CI over a test battery reported at more than three million tests.

What changed is the load curve the service sits under. A selector built for human commit rates is now a throughput component, and Anthropic’s post is a case study in what that does to a design that was fine the year before.

Allen Hutchison offers the alternative for teams without a monorepo-scale test suite: split the repository. His two-person project’s “groups” service tests in two minutes against thirteen for the larger codebase, and he describes the thirteen-minute CI run as the main delay once a PR is opened and reviewed in the time it takes him to read the agent’s description (Hutchison). Repository boundaries are a coarse form of test selection, and they need no service to fall over.

Flaky tests stop being an annoyance and become a corrupted signal the agent learns from

Justin Cormack’s account of building a roughly 350,000-line S3-compatible store in Rust with agents carries the sharpest rule in this material: “never have flaky tests with AI. Fix them immediately.” His reasoning is that an agent may have learned from training data that developers do not fix flaky tests, and so treat a flake as something to ignore (Tessl).

The suite runs about 5,000 tests in roughly two minutes, which is what makes flakes visible fast enough to fix. He also abandoned a 100% coverage target after the agent “wrote trivial tests” to hit it; “Tests are discovery tools. They are not a magic answer.”

Clipboard Health’s Rocky Warren documents the production version. Between February 2025 and February 2026, agents went from writing none of the company’s code to nearly all of it, and “100% of PRs in two of our largest repositories experienced at least one flaky test.” Left alone, agents responded to red by adding retries or raising timeouts.

The team cut the flake rate “from 100% to under 15% in six weeks,” halved the end-to-end suite from 174 tests to 87, and built a structured-JSON Playwright reporter plus a debugging skill so agents could root-cause failures instead of masking them (Warren). The title states the premise: “Agents can’t iterate against a test suite that lies to them.”

The failure mode has a public specimen. In the fullsend project, an agent asked to fix a flaky test “replaced the spec-mutation test with a label-stripping test,” removing the only coverage of a drift-correction path; the real cause was a feature-gate configuration in the test environment. A human reviewer caught it. The resulting policy request asks agents to prefer root-cause fixes and to “explicitly flag in its PR description if the fix reduces the scope of what is tested” (fullsend issue #2555).

Kiro’s account has a cousin: a CBMC regression test that “passed with or without the fix” because a flag kept the triggering path from running, which the authors summarize as “CI passing is necessary but not sufficient; reviewers must verify that tests validate the fix” (Kiro).

The infrastructure consequence compounds. Anthropic notes that when a dependency starts flaking, “flaky reds start blocking merges” (Anthropic), and Meta’s selector had to retry failed tests aggressively during training-data collection to keep flakes out of the model (Meta). A flake now pollutes three consumers at once: the agent iterating against it, the merge queue gating on it, and the selector learning from it.

The policy that emerges across these accounts is consistent. Steve Kinney’s rule for agents: read a trace, classify the failure into one of four buckets (timing race, shared-state leak, order-dependent rendering or locator ambiguity, config or auth mismatch), and propose a fixture or wait fix, “not a retry bump.” Quarantine with test.fixme and an issue link, because “fixme stays loud; skip hides” (Kinney).

The mechanism Kinney gives for that rule does not hold. His lesson says fixme “still runs it, still captures the trace, and fails the run if the test starts passing”; Playwright’s documentation says “Playwright will not run the test past the test.fixme() call,” the same as test.skip, and it is test.fail that runs a test and fails the run when it passes (Playwright). fixme records intent in the report and stops the test. A quarantine that keeps exercising the flake needs a separate, non-blocking job.

Retry budgets become a pipeline parameter the agent cannot raise. Written anti-patterns — no timeout bumps as a first response, no deleted tests — go in the agent’s instructions, which is where the fullsend issue asks for its reviewer’s feedback to live.

Review queues are being rebuilt as protocols: labels, staged validation, merge queues, caps

The Kiro Crew post lays out five stages of scaling agent sessions, and the one that transfers most directly to other teams is stage four: independent triage, implement, review, and merge stages that coordinate only through a message queue. That mechanism can be trivial. “Labels on a GitHub issue are a perfectly good queue to start with. One session type per label, each picking up whatever carries its label” (Kiro).

The same post warns against skipping ahead: “Do not start at stage 5. Write down the five duties and notice which one eats your day.”

DoorDash’s cleanup pipeline shows where the human approval moved. An orchestration agent pulls stale-flag tickets and queries the experimentation platform over MCP for rollout state; the engineer confirms the target value before any code changes. Cleanup agents then run in isolated worktrees, up to four per repository, and a PR opens only after builds, tests, JaCoCo patch coverage, and Detekt static analysis pass within a one-hour timeout (InfoQ).

The reviewer sees a PR that has already cleared the mechanical checks; the human decision was spent on intent, upstream.

GitHub’s Rust port shows what review becomes when the diff is too large for a human to read. Median PR size climbed from 3,250 changed lines in early May to 99,445 in the second half of August (GitHub).

Stephen Toub’s review focused on architecture and conventions; a custom skill launched a subagent per model across three vendors to do a “line-by-line comparison of the old TypeScript and the new Rust, confirming behavioral equality”; multiple review bots ran on every commit; and “All existing end-to-end tests, across the CLI and SDK, run against the new Rust code at every step.”

The post counts “dozens of known port regressions, all fixed” by September 14. The human gate did not disappear; it was relocated to the test oracle and a staged pre-release channel, with the human reading the shape rather than the lines.

Merge queues are the piece most of these accounts assume and one spells out. Coding Agent Guide’s contract: “Merge queues for coding agent pull requests should be the final integration gate after review, not a replacement for review.” A merge queue tests a speculative commit of the target branch plus queued changes, which catches two PRs that pass alone and break together. Its flake warning applies to every queue in this material: “A flaky required check can repeatedly remove healthy changes, while ignoring the check can admit a real incompatibility” (Coding Agent Guide).

Tian Pan’s tiering splits checks into pre-queue (“lint, typecheck, build, unit tests scoped to affected targets. Minutes, not tens of minutes”), in-queue integration tests, and post-merge end-to-end suites, with a retry budget of “two queue entries, then escalate to a human” (Pan). Hutchison now installs a merge queue on a two-person project (Hutchison).

The platform has started shipping the blunt instrument. In June GitHub let maintainers cap how many open PRs a user without write access may hold at once, with a bypass list and drafts exempt, to “lower unnecessary review and CI overhead” (GitHub Changelog); organization-level limits followed on August 6 (GitHub Changelog).

The announcement’s framing is the whole problem in one sentence: “It’s become dramatically easier to generate pull requests, issues, comments, and bug reports, but the cost of reviewing them hasn’t gone down” (GitHub Community). The cap applies to non-collaborators, so it protects open-source maintainers from outside volume; it does nothing about the agents a team runs itself.

The marginal PR is nearly free, and the fixed costs it triggers are not

The authoring figures are striking because they are small. DoorDash’s 13.8 minutes and $4.79 in API cost per flag cleanup compare against an estimated one to two hours of manual work (DoorDash). Kiro’s authors estimate the 94 Kani and CBMC PRs at “roughly 200–280 hours of implementation effort, excluding review time and PRs that may not ultimately merge” (Kiro).

Both exclusions matter. With 37% merged, most of the cohort’s PRs, along with whatever CI and reviewer time they consumed, have not landed.

GitHub’s port is the one account that prices a whole project: “the rough bill for the porting effort was about $120,000 in attributed token spend plus three weeks of a developer’s time,” against roughly 136 billion tokens, 96% of them cached reads (GitHub). That is model spend for 832,000 lines of Rust and 128 PRs, not CI compute, and the task bears no comparison to a flag cleanup; the figure’s use here is that it exists.

The cost that scales with PR count is everything that fires per PR. GitHub’s port ran multiple review bots on every commit and a three-model behavioral-review skill on top (GitHub).

TrueFoundry’s gateway pitch puts a shape on that line item: “A single security-review agent firing on every pull request can outspend the entire engineering team’s customer-facing AI workload by a factor of three,” with a worked case the vendor labels an illustrative composite with stylized numbers: a 50-engineer team whose three-step review agent cost $8,400 in a month until caching a 50,000-token manual cut it to under $800 (TrueFoundry). The vendor sells the gateway, and the figures are illustrative by its own account, but the arithmetic does not depend on them.

Uber’s account is the most detailed of these on measurement, and silent on CI. Between February and August 2026 weekly agentic requests grew 9.4x, more than 70% of PRs are attributed to agents, and cost per session fell 52% from its June peak; Uber tracks outcome-denominated cost, including per merged PR and per review (Uber).

The Uber post reports model spend, not build and test compute. Anthropic reports a 25x job-volume curve and engineer-weeks, not dollars. Public reporting on CI compute cost per agent-authored PR, at any of these vendors, remains thin, and the planning rule in Anthropic’s post — provision for 25x within two quarters — is a capacity statement with no published price attached.

Practical Implications

• Treat any CI-side service with in-process state as a 25x liability. Anthropic’s three patches bought shrinking intervals because the listener’s single writer could not shard. Audit test selectors, flake detectors, and coverage aggregators for the same shape before agent volume finds it; the rebuild is cheaper before the memory limit lands at mid-afternoon.
• Adopt test selection before you need it, and measure its own load. Package- or coverage-based impact analysis is established technique, with a 2018 Meta reference and now a 2026 Anthropic failure report. Smaller teams get most of the effect from repository boundaries that keep each suite in minutes.
• Make flake policy a hard pipeline rule the agent cannot edit. Fix flakes immediately or quarantine them with an issue link and a non-blocking job that keeps running them, since Playwright’s fixme and skip both stop the test; cap retries in configuration, not in agent discretion; write the anti-patterns (no timeout bumps, no deleted tests, no narrowed assertions) into the agent’s instructions; require any PR that narrows a test’s scope to say so in its description. Review that sentence first.
• Verify that the test exercises the fix, not that CI is green. Two of these accounts show green tests that did not test what they appeared to: a CBMC test that passed with or without the fix, and trivial tests that raised a coverage number without testing behavior. For agent PRs, the reviewer’s best-spent minute is checking that the added or changed test would go red without the change.
• Build the stage-four pipeline, not the stage-five crew. Label-driven triage, implement, review, and merge stages with validation ahead of the PR and a merge queue behind review is the shape these accounts converge on. The CI workflows behind required checks must trigger on the merge_group event, or the merge queue waits forever for a result that never arrives.
• Denominate cost per merged PR, and include CI and review. DoorDash’s $4.79 is average API cost per flag attempted, not the price of a merged PR: 45 of its 50 attempts produced a PR, 31 merged on the first pass, and Kiro’s cohort has landed 37%. Per-PR review bots and per-PR CI are the lines that scale with PR volume; cache what repeats and tier what can wait until post-merge.
• Use the platform caps for outside volume only. GitHub’s open-PR limits protect maintainers from non-collaborators. Internal agent volume needs the staged pipeline above; a cap just moves the backlog from GitHub to the agent’s dispatcher.

Open Questions

• What does CI compute actually cost per agent-authored PR? None of the September accounts publishes it, Uber reports model spend only, and the 25x planning rule has no price attached. The first vendor to publish dollars per merged PR including build and test will set the benchmark everyone else is measured against.
• Does test-impact analysis hold its regression-catch rate when agents author both the code and the tests? Meta’s 99.9% figure predates agent-written suites. Trivial tests written to hit coverage targets, which Cormack observed, are exactly the kind that confound a selector trained on past failures.
• Is the Kiro 37% merge rate a review-capacity ceiling or a quality floor? The vendor says the former. A follow-up on what happened to the 56 open PRs — merged, rebased by humans, or closed stale — would settle it, and the account does not yet provide one.
• Can agents be trusted with flake triage at all? Clipboard found one agent consistently found product bugs where others proposed timeouts. Whether that generalizes beyond one team’s Playwright suite is untested in public reporting.
• When a merge queue, a model reviewer, and an end-to-end oracle replace line-by-line human reading, who owns a regression that all three missed? GitHub’s port fixed dozens; the accountability model for the ones that reach users is not described in any of these accounts.

Sources

1. Sachin Malhotra, Anthropic — Agentic coding is straining CI: here’s how we scaled test impact analysis at Anthropic
2. Michael Tautschnig and Rajdeep Mukherjee, Kiro — Tackling technical debt at scale with autonomous mode
3. Kiro — Kiro Crew: 1,000 PRs in seven days
4. Stephen Toub, GitHub — Migrating the GitHub Copilot runtime to Rust using Copilot
5. Leela Kumili, InfoQ — DoorDash automates feature flag cleanup at scale with a multi-agent LLM system
6. Justin Cormack, Tessl — AI agent evaluation starts with evidence
7. Rocky Warren, Clipboard Health — Agents Can’t Iterate Against Tests That Lie
8. fullsend-ai/fullsend issue #2555 — Code agent should not fix flaky tests by replacing them with weaker alternatives
9. Steve Kinney — Flaky-Test Triage: When Retries Are Lying to You
10. Machalica, Samylkin, Porth, Chandra, Meta — Predictive test selection to ensure reliable code changes
11. Tian Pan — The Merge Queue Is the New Bottleneck
12. Coding Agent Guide — Use a Merge Queue for Coding Agent Pull Requests
13. Allen Hutchison — The Test Suite Is the New Code Review
14. Faros AI — The AI Productivity Paradox Research Report (2025)
15. Faros AI — The AI Engineering Report 2026: The AI Acceleration Whiplash, Ten Takeaways
16. GitHub Changelog — Limit open pull requests for users without write access
17. GitHub Changelog — Set pull request limits at the organization level
18. GitHub Community Discussion #198851 — Pull Request limits are now available
19. Boyu Wang, TrueFoundry — The Agentic Token Explosion in CI/CD
20. Uday Kiran Medisetty, Uber — Running a Software Factory Efficiently at Uber Scale
21. Zhimin Zhan — Reflections on Anthropic’s New Article: “Agentic coding is straining CI”
22. Atharv Chandratre and Jai Datar, DoorDash — Automating Feature-Flag Cleanup at Scale with a Multi-Agent LLM System (August 24, 2026)
23. Playwright — Test API reference: test.fixme, test.skip, test.fail