Scout: Do Context Files Actually Help? AGENTS.md Under Empirical Scrutiny

In Brief

• A controlled study across four agent-model pairs finds that context files do not generally improve task success while raising inference cost by over 20% on average; the paper’s own tests find the cost increase significant (p < 0.001%) and the small resolution-rate deltas not.
• LLM-generated files, the kind an init-style command writes for you, show a small, non-significant reduction in success on top of that cost, while developer-written files fare better on CTXbench for all agents but Claude Code (a 2.4% average gain, p = 21%), at a premium the paper puts at “at most 19%”, a four-model average that hides a 42% rise for GPT-5.2.
• A separate efficiency study, run under one agent configuration, finds an AGENTS.md was associated with lower median runtime and output-token use but checked patch correctness only by a 50-output sanity sample, so the two results answer different questions rather than conflicting.
• Two mining studies find the installed base bloated: restating lint rules a linter already enforces is the commonest smell, and the files are maintained like configuration code through frequent small additions.
• Context files earn their keep when they carry non-inferable, project-specific facts (custom build commands, non-standard conventions, hard boundaries) and become dead weight when they restate what the agent can already discover.

Why This Matters

The convention was adopted on faith: drop an AGENTS.md or CLAUDE.md at the repo root and let the coding agent read it on every task. Its first wave of empirical scrutiny arrived in early-to-mid 2026. The evidence base is young and preprint-heavy, so read the numbers as early signal, not settled fact. They still change the default: a context file is a per-task cost that has to be paid for by facts the agent cannot infer, and the mining evidence says most files in the wild carry something else.

Key Findings

Average lift is roughly nil, at a real cost premium

The most direct test is “Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?” from ETH Zurich and LogicStar.ai (Gloaguen, Mündler, Müller, Raychev, Vechev) [1]. It evaluates four agent-model pairs: Claude Code with Sonnet-4.5, Codex with GPT-5.2, Codex with GPT-5.1 mini, and Qwen Code with Qwen3-30b-coder.

The pairs run in two settings. One is SWE-bench Lite (300 tasks from 11 popular Python repositories). The other is a purpose-built benchmark the authors call CTXbench (named AGENTbench in the paper’s first version; 138 tasks drawn from 5,694 PRs across 12 niche repositories that ship developer-written context files) [1].

The finding that drew attention: providing context files “does not generally improve task success rates” while “increasing inference cost by over 20% on average” [1]. The headline 20%/23% figures are the per-setting numbers for the LLM-generated files: §4.2 reports those files “leading to a significant (p-value < 0.001%) cost increase of 20% and 23% on average” on SWE-bench Lite and CTXbench respectively [1].

Developer-written files cost less to run, with the paper putting their overhead at “at most 19%”, but they still impose a premium on every task [1]. That 19% is the average across the four model pairs on CTXbench, not a ceiling: by the paper’s Table 2, GPT-5.2’s per-task cost rose from $0.38 to $0.54, about 42%, while the other three pairs rose less [1]. The paper tests all of this: the cost increase is significant (p < 0.001%), while the small resolution-rate deltas below are not (p = 87% and 37% for LLM-generated files, p = 21% for developer-written ones), so treat them as the direction the data leans, not as established effects.

The two file types diverge sharply. LLM-generated context files, the kind a “run /init and commit whatever it writes” workflow produces, have a small, non-significant negative effect: the average resolution rate is “reduced by 0.5% and 2% on average” on SWE-bench Lite and CTXbench respectively [1].

Developer-written files fare better, improving “performance for all agents but Claude Code” on CTXbench [1]. That is a modest gain (the paper puts it at 2.4% on average, p = 21%, so not statistically significant) that still has to be weighed against the ~20% cost increase the file imposes on every task.

The mechanism the authors hypothesize amounts to over-obedience (the term is not theirs; the phrasing that follows is): “instructions in context files are generally followed,” and “we hypothesize that these additional instructions make the task harder” [1]. The agent treats every line as a constraint to satisfy. Pile on instructions the task didn’t need and you have enlarged the problem, not clarified it. The paper’s prescription is blunt: context files “should only contain specific additional instructions beyond what is already available in the codebase” [1].

A second, less flattering result targets one of the most common things people put in these files: the repository tour. The authors conclude that “context files are not effective at providing a repository overview” [1]. Agents are already competent at exploring a codebase; a prose summary of the directory layout mostly burns tokens. The paper’s appendix records the exception: after removing a repository’s documentation (Markdown files, example code, and docs/) so the context file is the only documentation left, LLM-generated files “consistently improve performance by 2.7% on average” and beat developer-written ones across the three model pairs tested, which the authors suggest may explain anecdotal gains on thinly documented repos [1]. The setting is artificial, but it marks where the overview finding stops: a repository with nothing else to read.

Success and efficiency are different questions, and the two studies don’t conflict

The headline above is about task success at a cost. A separate study measures operational efficiency and reaches a different-looking conclusion. “On the Impact of AGENTS.md Files on the Efficiency of AI Coding Agents” (Lulla, Mohsenimofidi, Galster, Zhang, Baltes, Treude) examines 10 repositories and 124 small pull requests, each capped at ≤100 changed lines across ≤5 files, run with a single agent configuration: OpenAI Codex on gpt-5.2-codex [2].

The study finds the presence of an AGENTS.md was associated with a lower median runtime (Δ 28.64%) and reduced median output-token consumption (Δ 16.58%), while agents kept “comparable task completion behavior” [2].

Two caveats the study states plainly. The saving is in output tokens specifically; median total token consumption actually edged up ~1.3%. And it did not evaluate whether the patches were correct, since “a comprehensive evaluation of the output quality, e.g., the semantic correctness or the functional equivalence to the merged PR, is beyond the scope of this paper” [2]. A sanity check of 50 sampled outputs is the only correctness signal, so “comparable task completion behavior” means “didn’t obviously fail,” not “verified correct.”

These two results are not in conflict; they answer different questions. The efficiency study finds that when an agent is working a task, a context file can help it converge faster and emit fewer output tokens (though not necessarily fewer total tokens). The CTXbench study finds that whether the agent ultimately resolves the issue doesn’t improve, and that total inference cost (reasoning tokens included) tends to rise.

Carry both. A good context file can make the agent more direct, but a file padded with non-essential requirements adds reasoning overhead that swamps the saving. The sign of the effect appears to track file quality more than the mere presence of a file. The two studies also differ in scale and design (124 small single-agent PRs across 10 repos versus a controlled multi-benchmark sweep), so treat the efficiency deltas as a directional signal rather than a fixed number.

The installed base is bloated with lint rules and rarely states guardrails

Two mining studies characterize the installed base, and the picture explains why the average lift is low.

“Configuration Smells in AGENTS.md Files” (dos Santos, Costa, Montandon, Silva, Valente) mined the 100 most-starred GitHub repositories carrying one of these files, 39 AGENTS.md and 61 CLAUDE.md, so each count below is out of 100 [3]. It catalogues six recurring smells.

Read the numbers carefully. The four LLM-detected smells are reported as detector flags, with a manual false-positive check yielding lower confirmed counts; the gap matters most for Conflicting Instructions, where detector precision was only 57%. The other two smells are purely threshold classifications.

• Lint Leakage: rules already enforced by a linter or formatter, restated in prose. The most common smell: flagged in 62 files, 58 confirmed [3].
• Context Bloat: files padded with low-priority detail until the signal drowns. 42 files, but this is a threshold rule (≥200 lines), not a manually validated semantic finding [3].
• Skill Leakage: specialized, rarely-used instructions parked in the always-loaded file instead of a dedicated skill or doc the agent loads on demand. 35 flagged, 29 confirmed [3].
• Conflicting Instructions: contradictory directives the agent can’t satisfy at once. 28 flagged, but only 16 confirmed after manual review (57% precision) [3].
• Init Fossilization: a file generated once (often by an init command) and never reviewed or updated again. 24 files, also a threshold proxy (a single commit to the file), not a semantic judgment [3].
• Blind Reference: pointers to external docs with no explanatory context, leaving the agent to chase a link blind. 16 flagged, 14 confirmed [3].

Lint Leakage is the standout. The most common smell in these files is restating rules a deterministic tool already checks for free: likely redundant token cost whose incremental benefit hasn’t been demonstrated, and exactly the kind of additional instruction the CTXbench study hypothesizes makes tasks harder.

“Agent READMEs: An Empirical Study of Context Files for Agentic Coding” (Chatlatanagulchai, Li, Kashiwa, Reid et al.) widens the lens to 2,303 context files across 1,925 repositories [4]. Its content-category breakdown comes from a manually-labeled subset of 332 Claude Code files, not the full corpus.

Build/run commands appear in 62.3% of that subset, implementation details in 69.9%, and architecture information in 67.7%, but security and performance requirements each in only 14.5% [4]. The authors describe these files not as static documentation but as “complex, difficult-to-read artifacts that evolve like configuration code, maintained through frequent, small additions” [4].

Developers load the file with functional context while rarely stating guardrails for the security and performance characteristics of the generated code. Absence from the context file doesn’t prove those concerns go unenforced; they may live in linters, CI gates, or policy.

Files rot two ways: fossilized after init, or grown a rule per mistake

The “evolve like configuration code” framing points at the maintenance problem. Most files, about two-thirds in the Agent READMEs data, are actively grown through “frequent, small additions” [4]. Every time the agent makes a mistake, the reflex is to add a rule, which is the engine that drives Context Bloat. That reflex is vendor guidance as much as habit (the smell catalogue relays Anthropic’s advice to update the file when the agent “makes the same mistake a second time” [3]), and it is a mechanism rather than a measurement: neither mining study records why an addition was made. The opposite failure mode is Init Fossilization: never revisited after init-time generation. Roughly one-third of the Agent READMEs corpus is touched in only one commit [4]; the smell catalogue, which defines the smell more strictly, flags 24 of its 100 repos [3].

A separate team, sharing one co-author (Kashiwa) with the Agent READMEs study, has an in-progress effort applying a software-maintenance lens to how these files change over time, mining a comparable corpus of ACF-modifying commits [5]. Its results are not yet reported, so the durable maintenance guidance for now comes from the smell catalogue and the efficiency evidence rather than a settled longitudinal finding.

The practical anti-rot moves follow from the smells. Periodically prune lint-checkable rules; delete them, since the linter is the enforcement layer. Treat additions as additions to a budget, not a free append. And review the file on a cadence so it doesn’t fossilize at its init-time state. The agent reads it on every request, so a stale path or renamed module actively misleads on every task, not just once.

Better files are possible, but the recipe is narrower than tune-it-once

The evidence isn’t “context files are useless.” It’s closer to “most context files are written wrong.” Two threads gesture at what better might look like, each with caveats.

GitHub has published practical writing guidance based on an informal review of more than 2,500 agents.md files [6]. Two caveats apply. Its subject is the new custom-agent persona files under .github/agents/ (specialist @docs-agent/@security-agent roles invoked by name), not the always-loaded root context file the research above is about.

And it discloses no outcome metric, so its six “core areas” are framed as what “puts you in the top tier,” not a measured comparison of effective vs. ineffective files. With those caveats, the concrete habits transfer cleanly to root files: cover commands, testing, project structure, code style, git workflow, and boundaries; put runnable commands (with flags) early; prefer “one real code snippet” over “three paragraphs describing it”; name specific versions (“React 18 with TypeScript, Vite, and Tailwind CSS,” not “React project”); and state hard boundaries the agent must never cross [6].

On the research side, “Probe-and-Refine Tuning of Repository Guidance for Coding Agents” (Shepard, Albrecht) shows that how you produce the file can matter more than whether you have one, but with sharp limits. Using synthetic bug-fix probes to iteratively refine guidance (with Qwen3.5-35B-A3B), they report a 33.0% mean resolve rate on SWE-bench Verified versus 28.3% for baseline guidance and 25.5% unguided (p < 0.001) [7]. The numbers belong to the paper’s own scaffold, not a product configuration: a ReAct-style loop with a 200-step budget, a 16k-token effective context window imposed by truncation, and a single-shot fallback that writes a patch when the loop exhausts its budget, so the authors present the comparison across conditions, not the absolute rates, as the result [7].

The gain “stems from coverage rather than precision,” with refined guidance producing evaluable patches for 14.5 percentage points more instances while per-patch precision held statistically constant (~59%) [7]. The authors hypothesize the mechanism is localization, guidance helping the agent reach the right file, but explicitly did not run the file-structure ablation that would test it [7].

Three caveats cut against reading this as a general recipe. Refined guidance used ~56% more prompt tokens. On a second model (Nemotron) both refined and static guidance underperformed no-context (27.0% and 24.6% vs 28.4%). And transferring the Qwen-tuned guidance to Nemotron collapsed resolution to 13.2%, a result the paper itself calls “catastrophic,” concluding that “guidance encodes model-specific behavioral calibration, not transferable repository knowledge” [7].

The practical reading is narrower than “tune your file once”: good guidance may have to be calibrated for the specific model, harness, and repo it runs in. It does square with the CTXbench finding that overviews are useless (the agent can navigate) while pointed, non-inferable facts (“the build is non-standard; run make x not npm build”) earn their place.

Practical Implications

• Stop committing whatever init generates in a documented repository. Auto-generated context files showed a small observed reduction in success in the strongest study (−0.5% to −2% on average, not statistically significant at p = 87% and 37%) while measurably adding cost [1]. The stated exception is the repository with no documentation of its own, where the same study’s ablation found a 2.7% average gain. If you run an init-style command on a documented repo, treat its output as a draft to gut, not a file to commit; on an undocumented one, test it before deciding.
• Delete lint-checkable rules from the file. Lint Leakage is the most prevalent smell (flagged in 62 of 100 files, 58 confirmed) [3], and the CTXbench study’s hypothesized mechanism says every superfluous instruction is a constraint the agent must satisfy [1]. Your linter and formatter already enforce style deterministically and for free; restating them in the context file is likely redundant token cost.
• Cut the repository tour. Even developer-written overviews “are not effective” [1]. Agents explore codebases competently on their own; the paper’s one recorded exception is a repository stripped of all other documentation [1]. Spend the words on what they can’t infer.
• Keep only non-inferable, project-specific facts. Custom build/test commands with their flags, non-standard conventions, and hard boundaries (“never modify generated files,” “never commit secrets”) are what a file is for. GitHub’s six-area structure (commands, testing, structure, style, git workflow, boundaries) is a reasonable skeleton [6], and the controlled study’s explicit recommendation is that context files “should only contain specific additional instructions beyond what is already available in the codebase” [1].
• Budget for the cost, then justify it. A context file imposes a real inference-cost premium on every task that loads it; the paper measures 20%/23% on average for auto-generated files and “at most 19%” on average for developer-written ones, with one model pair (GPT-5.2) paying 42% [1]. A developer-written file buying a few points of success on a tricky repo may clear that bar; a bloated auto-generated one does not. Measure on your own repos before assuming the file pays for itself.
• Move rarely-used instructions out of the always-loaded file. Skill Leakage (35 flagged / 29 confirmed [3]) is the smell where one-off, specialized guidance lives in the file every task reads. Push it into on-demand skills or docs the agent loads only when relevant.
• Don’t forget guardrails. Security and performance requirements appear in only ~14.5% of the manually-labeled Claude-file subset [4]. If you care about the non-functional characteristics of agent-generated code, the context file is one of the few places to state them, and few developers do (though those concerns may also be enforced in CI, linters, or policy).
• Review on a cadence. Init Fossilization (~24% [3]) and the “renamed path poisons every request” failure mode mean a stale file is worse than a short one. Prune as you add.

Open Questions

• How much does the result generalize beyond the tested agents and repo classes? The controlled evidence is strongest on Python (SWE-bench Lite is Python; CTXbench is Python) and on four specific agent-model pairs [1]. Whether the rule that a file should carry only instructions the codebase lacks holds for other languages, larger monorepos, or nested per-directory context files is not settled by these studies.
• What’s the right length? Practitioner guidance relayed in the smell work targets under 200 lines, and the smell work itself argues for concision [3], but no study reports a clean dose-response curve mapping file length to success or cost. The defensible guidance is “shorter and more specific,” not a number.
• Does iterative refinement transfer across models and out of the benchmark? Probe-and-refine’s coverage gain is real on SWE-bench Verified with one model, but the paper’s own cross-model test is discouraging: guidance tuned for one model hurt another, and transferring it dropped resolution to 13.2%, which the authors call “catastrophic” [7]. Whether hand-refinement by developers captures the “reach the right file” benefit, and whether any of it survives a model swap, is the open, and newly cautionary, question.
• What does longitudinal maintenance actually do to efficacy? The maintenance-focused work characterizing how these files change over time has a dataset and a plan but not yet reported outcomes [5]; the relationship between how a file evolves and the code-quality outcomes it produces is still open.

Since publication

• 2026-09-02 — The Agent READMEs study cited above as [4] was revised after the dateline. Version 2, posted August 9, 2026, reports different content-category figures from the v1 numbers quoted above: its abstract now lists “test procedures (75.9%), implementation details (70.8%), and architecture (68.1%)” as the most common categories, with “security (14.8%) and performance (14.5%)” still the rarest. The ordering the analysis rests on holds; the specific percentages no longer match the current version.

Sources

1. Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? (arXiv:2602.11988) — Gloaguen, Mündler, Müller, Raychev, Vechev (citing v2, revised 23 June 2026; the purpose-built benchmark was renamed CTXbench from v1’s AGENTbench, and v2 added the significance tests v1 lacked.)
2. On the Impact of AGENTS.md Files on the Efficiency of AI Coding Agents (arXiv:2601.20404) — Lulla, Mohsenimofidi, Galster, Zhang, Baltes, Treude
3. Configuration Smells in AGENTS.md Files (arXiv:2606.15828) — dos Santos, Costa, Montandon, Silva, Valente (full HTML: https://arxiv.org/html/2606.15828v1)
4. Agent READMEs: An Empirical Study of Context Files for Agentic Coding (arXiv:2511.12884) — Chatlatanagulchai, Li, Kashiwa, Reid, et al. (citing v1, November 2025)
5. How Do Developers Maintain and Evolve Their Agents’ Instructions? An Empirical Study (arXiv:2606.25257) — Voria, Cannavale, De Lucia, Kashiwa, Catolino, Palomba — study protocol: describes the planned classification, dataset, and analysis; no empirical results reported yet.
6. How to write a great agents.md: Lessons from over 2,500 repositories — The GitHub Blog (writing guidance drawn from custom-agent persona files under .github/agents/; informal pattern-observation, no disclosed outcome metric)
7. Probe-and-Refine Tuning of Repository Guidance for Coding Agents (arXiv:2606.20512) — Shepard, Albrecht
8. AGENTS.md — a simple, open format for guiding coding agents — the convention’s reference site
9. The research is in: your AGENTS.md is probably too long — Upsun Developer (practitioner reception)
10. New Research Reassesses the Value of AGENTS.md Files for AI Coding — InfoQ (coverage of the evaluation study)