Artificer Digital The Artificer's Grimoire

Scout: Do Context Files Actually Help? AGENTS.md Under Empirical Scrutiny

Summary

The convention everyone adopted on faith — drop an AGENTS.md or CLAUDE.md at the repo root and let the coding agent read it on every task — got its first wave of empirical scrutiny in early-to-mid 2026, though the evidence base is young and preprint-heavy, so read the numbers as early signal, not settled fact. The most comprehensive controlled study measuring issue resolution — comparing agent runs with and without context files across multiple benchmarks — finds that context files do not generally improve task success rates while raising inference cost by over 20% on average; auto-generated files show a small observed reduction in success (reported without significance testing) on top of that cost. A separate efficiency study, measuring different metrics, finds the files can cut runtime and output-token use — though it never checked whether the resulting patches were correct — so “does it help” depends entirely on what you measure. Two mining studies show the files in the wild are bloated, full of lint rules a linter already enforces, and maintained like config code through constant small patches. The actionable through-line: context files earn their keep when they carry non-inferable, project-specific facts (custom build commands, non-standard conventions, hard boundaries) and become dead weight when they restate what the agent can already discover.

Key Findings

The headline result: average lift is roughly nil, at a real cost premium

The most direct test is “Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?” from ETH Zurich and LogicStar.ai (Gloaguen, Mündler, Müller, Raychev, Vechev). It evaluates four agent-model pairs — Claude Code with Sonnet-4.5, Codex with GPT-5.2, Codex with GPT-5.1 mini, and Qwen Code with Qwen3-30b-coder — across two settings: SWE-bench Lite (300 tasks from 11 popular Python repositories) and a purpose-built benchmark the authors call CTXBench (named AGENTbench in the paper’s first version; 138 tasks drawn from 5,694 PRs across 12 niche repositories that ship developer-written context files) [1].

The finding that drew attention: providing context files “does not generally improve task success rates” while “increasing inference cost by over 20% on average” [1]. The headline 20%/23% figures are the per-setting numbers for the LLM-generated files: §4.2 reports those files “leads to a cost increase of 20% and 23% on average” on SWE-bench Lite and CTXBench respectively [1]. Developer-written files cost less to run — the paper puts their overhead at “at most 19%” — but still impose a premium on every task [1]. (The paper reports these as observed averages; it runs no significance tests, so treat the small resolution-rate deltas below as directional, not as measured effects.)

The two file types diverge sharply. LLM-generated context files — the kind a “run /init and commit whatever it writes” workflow produces — have a small observed negative effect: the average resolution rate is “reduced by 0.5% and 2% on average” on SWE-bench Lite and CTXBench respectively [1]. Developer-written files fare better, improving “the performance compared to no context files for all agents but Claude Code” by a few points on CTXBench [1] — a modest gain (the paper puts it at ~4% on average) that still has to be weighed against the ~20% cost increase the file imposes on every task.

The mechanism the authors identify is what amounts to over-obedience (their phrasing, not that term): “agents generally follow instructions present in the context files,” so “unnecessary requirements from context files make tasks harder” [1]. The agent treats every line as a constraint to satisfy. Pile on instructions the task didn’t need and you’ve enlarged the problem, not clarified it. The paper’s prescription is blunt: “human-written context files should describe only minimal requirements” [1].

A second, less-flattering result targets the most common thing people put in these files — the repository tour. The authors conclude that “context files, even developer-provided ones, are not effective at providing a repository overview” [1]. Agents are already competent at exploring a codebase; a prose summary of the directory layout mostly burns tokens.

”Does it help” depends on what you measure — success vs. efficiency

The headline above is about task success at a cost. A separate study measures operational efficiency and reaches a different-looking conclusion. “On the Impact of AGENTS.md Files on the Efficiency of AI Coding Agents” (Lulla, Mohsenimofidi, Galster, Zhang, Baltes, Treude) examines 10 repositories and 124 small pull requests (each capped at ≤100 changed lines across ≤5 files), run with a single agent configuration — OpenAI Codex on gpt-5.2-codex — and reports that the presence of an AGENTS.md was associated with a lower median runtime (Δ 28.64%) and reduced median output-token consumption (Δ 16.58%), while agents kept “comparable task completion behavior” [2]. Two caveats the study states plainly: the saving is in output tokens specifically — median total token consumption actually edged up ~1.3% — and it did not evaluate whether the patches were correct, since “a comprehensive evaluation of the output quality, e.g., the semantic correctness or the functional equivalence to the merged PR, is beyond the scope of this paper” [2]. A sanity check of 50 sampled outputs is the only correctness signal, so “comparable task completion behavior” means “didn’t obviously fail,” not “verified correct.”

These two results are not in conflict; they answer different questions. The efficiency study finds that when an agent is working a task, a context file can help it converge faster and emit fewer output tokens (though not necessarily fewer total tokens). The CTXBench study finds that whether the agent ultimately resolves the issue doesn’t improve — and that total inference cost (reasoning tokens included) tends to rise. The reconciliation a practitioner should carry: a good context file can make the agent more direct, but a file padded with non-essential requirements adds reasoning overhead that swamps the saving. The sign of the effect appears to track file quality more than the mere presence of a file. Note the two studies also differ in scale and design — 124 small single-agent PRs across 10 repos versus a controlled multi-benchmark sweep — so treat the efficiency deltas as a directional signal rather than a fixed number.

What’s actually in these files: bloat, lint rules, and gaps

Two mining studies characterize the installed base, and the picture explains why the average lift is low.

“Configuration Smells in AGENTS.md Files” (dos Santos, Costa, Montandon, Silva, Valente) mined the 100 most-starred GitHub repositories carrying one of these files — 39 AGENTS.md and 61 CLAUDE.md, so each count below is out of 100 — and catalogues six recurring smells [3]. Read the numbers carefully: the four LLM-detected smells are reported as detector flags, with a manual false-positive check yielding lower confirmed counts (the gap matters most for Conflicting Instructions, where detector precision was only 57%), while two smells are purely threshold classifications:

  • Lint Leakage — rules already enforced by a linter or formatter, restated in prose. The most common smell: flagged in 62 files, 58 confirmed [3].
  • Context Bloat — files padded with low-priority detail until the signal drowns. 42 files — but this is a threshold rule (≥200 lines), not a manually validated semantic finding [3].
  • Skill Leakage — specialized, rarely-used instructions parked in the always-loaded file instead of a dedicated skill/doc the agent loads on demand. 35 flagged, 29 confirmed [3].
  • Conflicting Instructions — contradictory directives the agent can’t satisfy at once. 28 flagged, but only 16 confirmed after manual review (57% precision) [3].
  • Init Fossilization — a file generated once (often by an init command) and never reviewed or updated again. 24 files — also a threshold proxy (a single commit to the file), not a semantic judgment [3].
  • Blind Reference — pointers to external docs with no explanatory context, leaving the agent to chase a link blind. 16 flagged, 14 confirmed [3].

Lint Leakage is the standout: the single most common content in these files is rules a deterministic tool already checks for free — likely redundant token cost whose incremental benefit hasn’t been demonstrated, and exactly the kind of “unnecessary requirement” the CTXBench study warns makes tasks harder.

“Agent READMEs: An Empirical Study of Context Files for Agentic Coding” (Chatlatanagulchai, Li, Kashiwa, Reid, and colleagues) widens the lens to 2,303 context files across 1,925 repositories [4]. Its content-category breakdown comes from a manually-labeled subset of 332 Claude Code files (not the full corpus): build/run commands appear in 62.3% of that subset, implementation details in 69.9%, and architecture information in 67.7% — but security and performance requirements each in only 14.5% [4]. The authors describe these files not as static documentation but as “complex, difficult-to-read artifacts that evolve like configuration code, maintained through frequent, small additions” [4]. The gap worth flagging: developers load the file with functional context while rarely stating guardrails for the security and performance characteristics of the generated code — with the caveat that absence from the context file doesn’t prove those concerns go unenforced (they may live in linters, CI gates, or policy).

Keeping the file from rotting

The “evolve like configuration code” framing points at the maintenance problem. Most files — about two-thirds, in the Agent READMEs data — are actively grown through “frequent, small additions” [4]: every time the agent makes a mistake, the reflex is to add a rule, which is the engine that drives Context Bloat. The opposite failure mode is Init Fossilization — the roughly one-third never revisited after their init-time generation. A separate team, sharing one co-author (Kashiwa) with the Agent READMEs study, has an in-progress effort applying a software-maintenance lens to how these files change over time, mining a comparable corpus of ACF-modifying commits; its results are not yet reported, so the durable maintenance guidance for now comes from the smell catalogue and the efficiency evidence rather than a settled longitudinal finding [5].

The practical anti-rot moves follow from the smells. Periodically prune lint-checkable rules (delete them; the linter is the enforcement layer). Treat additions as additions to a budget, not a free append. And review the file on a cadence so it doesn’t fossilize at its init-time state — the agent reads it on every request, so a stale path or renamed module actively misleads on every task, not just once.

There’s a constructive direction, not just a teardown

The evidence isn’t “context files are useless.” It’s closer to “most context files are written wrong.” Two threads gesture at what better might look like — each with caveats.

GitHub has published practical writing guidance based on an informal review of more than 2,500 agents.md files [6]. Two things to keep in view about it: its subject is the new custom-agent persona files under .github/agents/ (specialist @docs-agent/@security-agent roles invoked by name), not the always-loaded root context file the research above is about; and it discloses no outcome metric, so its six “core areas” are framed as what “puts you in the top tier,” not a measured comparison of effective vs. ineffective files. With those caveats, the concrete habits transfer cleanly to root files: cover commands, testing, project structure, code style, git workflow, and boundaries; put runnable commands (with flags) early; prefer “one real code snippet” over “three paragraphs describing it”; name specific versions (“React 18 with TypeScript, Vite, and Tailwind CSS,” not “React project”); and state hard boundaries the agent must never cross [6].

On the research side, “Probe-and-Refine Tuning of Repository Guidance for Coding Agents” (Shepard, Albrecht) shows that how you produce the file can matter more than whether you have one — but with sharp limits. Using synthetic bug-fix probes to iteratively refine guidance (with Qwen3.5-35B-A3B), they report a 33.0% mean resolve rate on SWE-bench Verified versus 28.3% for baseline guidance and 25.5% unguided (p < 0.001) [7]. The gain “stems from coverage rather than precision,” with refined guidance producing evaluable patches for 14.5 percentage points more instances while per-patch precision held statistically constant (~59%) [7]. The authors hypothesize the mechanism is localization — guidance helping the agent reach the right file — but explicitly did not run the file-structure ablation that would test it [7]. Three caveats cut against reading this as a general recipe: refined guidance used ~56% more prompt tokens; on a second model (Nemotron) both refined and static guidance underperformed no-context (27.0% and 24.6% vs 28.4%); and transferring the Qwen-tuned guidance to Nemotron collapsed resolution to 13.2% — a result the paper itself calls “catastrophic,” concluding that “guidance encodes model-specific behavioral calibration, not transferable repository knowledge” [7]. The practical reading is narrower than “tune your file once”: good guidance may have to be calibrated for the specific model, harness, and repo it runs in. It does square with the CTXBench finding that overviews are useless (the agent can navigate) while pointed, non-inferable facts (“the build is non-standard; run make x not npm build”) earn their place.

Practical Implications

  • Stop committing whatever init generates. Auto-generated context files showed a small observed reduction in success in the strongest study (−0.5% to −2% on average, reported without significance testing) while measurably adding cost [1]. If you run an init-style command, treat its output as a draft to gut, not a file to commit.
  • Delete lint-checkable rules from the file. Lint Leakage is the most prevalent smell (flagged in 62 of 100 files, 58 confirmed) [3], and the CTXBench mechanism says every superfluous instruction is a constraint the agent must satisfy [1]. Your linter and formatter already enforce style deterministically and for free; restating them in the context file is likely redundant token cost.
  • Cut the repository tour. Even developer-written overviews “are not effective” [1]. Agents explore codebases competently on their own. Spend the words on what they can’t infer.
  • Keep only non-inferable, project-specific facts. Custom build/test commands with their flags, non-standard conventions, and hard boundaries (“never modify generated files,” “never commit secrets”) are what a file is for. GitHub’s six-area structure — commands, testing, structure, style, git workflow, boundaries — is a reasonable skeleton [6], and “minimal requirements” is the explicit recommendation from the controlled study [1].
  • Budget for the cost, then justify it. A context file imposes a real inference-cost premium on every task that loads it — the paper measures up to 20%/23% for auto-generated files and “at most 19%” for developer-written ones [1]. A developer-written file buying a few points of success on a tricky repo may clear that bar; a bloated auto-generated one does not. Measure on your own repos before assuming the file pays for itself.
  • Move rarely-used instructions out of the always-loaded file. Skill Leakage (35 flagged / 29 confirmed [3]) is the smell where one-off, specialized guidance lives in the file every task reads. Push it into on-demand skills or docs the agent loads only when relevant.
  • Don’t forget guardrails. Security and performance requirements appear in only ~14.5% of the manually-labeled Claude-file subset [4]. If you care about the non-functional characteristics of agent-generated code, the context file is one of the few places to state them — and few developers do (though those concerns may also be enforced in CI, linters, or policy).
  • Review on a cadence. Init Fossilization (~24% [3]) and the “renamed path poisons every request” failure mode mean a stale file is worse than a short one. Prune as you add.

Open Questions

  • How much does the result generalize beyond the tested agents and repo classes? The controlled evidence is strongest on Python (SWE-bench Lite is Python; CTXBench is Python-heavy) and on four specific agent-model pairs [1]. Whether the “minimal requirements” rule holds for other languages, larger monorepos, or nested per-directory context files is not settled by these studies.
  • What’s the right length? Practitioner guidance clusters around a few hundred lines, and the smell work argues for concision, but no study reports a clean dose-response curve mapping file length to success or cost. The defensible guidance is “shorter and more specific,” not a number.
  • Does iterative refinement transfer across models and out of the benchmark? Probe-and-refine’s coverage gain is real on SWE-bench Verified with one model, but the paper’s own cross-model test is discouraging: guidance tuned for one model hurt another, and transferring it dropped resolution to 13.2% — which the authors call “catastrophic” [7]. Whether hand-refinement by developers captures the “reach the right file” benefit — and whether any of it survives a model swap — is the open, and newly cautionary, question.
  • What does longitudinal maintenance actually do to efficacy? The maintenance-focused work characterizing how these files change over time has a dataset and a plan but not yet reported outcomes [5]; the relationship between how a file evolves and the code-quality outcomes it produces is still open.

Sources

  1. Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? (arXiv:2602.11988) — Gloaguen, Mündler, Müller, Raychev, Vechev (citing v2, revised June 2026; the purpose-built benchmark was renamed CTXBench from v1’s AGENTbench. The paper reports observed averages and runs no significance tests.)
  2. On the Impact of AGENTS.md Files on the Efficiency of AI Coding Agents (arXiv:2601.20404) — Lulla, Mohsenimofidi, Galster, Zhang, Baltes, Treude
  3. Configuration Smells in AGENTS.md Files (arXiv:2606.15828) — dos Santos, Costa, Montandon, Silva, Valente (full HTML: https://arxiv.org/html/2606.15828v1)
  4. Agent READMEs: An Empirical Study of Context Files for Agentic Coding (arXiv:2511.12884) — Chatlatanagulchai, Li, Kashiwa, Reid, et al.
  5. How Do Developers Maintain and Evolve Their Agents’ Instructions? An Empirical Study (arXiv:2606.25257) — Voria, Cannavale, De Lucia, Kashiwa, Catolino, Palomba — study protocol: describes the planned classification, dataset, and analysis; no empirical results reported yet.
  6. How to write a great agents.md: Lessons from over 2,500 repositories — The GitHub Blog (writing guidance drawn from custom-agent persona files under .github/agents/; informal pattern-observation, no disclosed outcome metric)
  7. Probe-and-Refine Tuning of Repository Guidance for Coding Agents (arXiv:2606.20512) — Shepard, Albrecht
  8. AGENTS.md — a simple, open format for guiding coding agents — the convention’s reference site
  9. The research is in: your AGENTS.md is probably too long — Upsun Developer (practitioner reception)
  10. New Research Reassesses the Value of AGENTS.md Files for AI Coding — InfoQ (coverage of the evaluation study)