Artificer Digital The Artificer's Grimoire

Scout: Unwinding the Prompt: What to Delete Now That the Models Have Judgement

context-engineering claude-code harness-engineering research

Summary

The advice arriving from the Claude Code team in late July 2026 is subtraction: remove the rules, remove the examples, stop hand-curating the memory file, and let the model exercise judgement it did not previously have. The anchoring number is that over 80% of Claude Code’s system prompt came out for the Claude 5 generation “with no measurable loss on our coding evaluations” (Anthropic). Independent evidence supports the direction of travel more than it supports the magnitude, and it backs two of the five reframed practices far more firmly than the other three. It also cuts both ways: a factorial study of 1,650 Claude Code sessions finds that none of the file-structure variables practitioners argue about — file size, instruction position, file architecture, contradictions between adjacent files — produced a detectable effect on adherence (arXiv 2605.10039), while a separate research line argues that contexts should accumulate domain detail rather than shed it and reports double-digit gains from doing so (arXiv 2510.04618). The phrase carrying the weight in the vendor claim is measurable — the same organization documented a 3% quality regression in April 2026 that weeks of testing against its standing evaluation set failed to surface, and that only a later, broader set detected (Anthropic), and single-run pass@1 on SWE-bench Verified varies by 2.2 to 6.0 percentage points depending on which run you happen to sample (arXiv 2602.07150). The defensible position for a team with a year of scaffolding is neither wholesale deletion nor stasis: it is a removal-attribution loop, run against an evaluation wide enough to price the things the prompt was protecting.

Key Findings

The 80% was a system prompt, and your CLAUDE.md is not in it

A common misreading of the July announcement is that it licenses cutting 80% out of a repository context file. It does not, and the distinction is mechanical rather than pedantic.

The reduction was applied to Claude Code’s own system prompt — the harness artifact Anthropic ships, not the artifact you commit. Thariq Shihipar’s framing of the cause is that the team “were overconstraining Claude Code, both through our system prompt and in our CLAUDE.md files and skills,” with cross-layer contradiction offered as the illustrative case: the post describes “several conflicting messages in a single request like ‘leave documentation as appropriate,’ or ‘DO NOT add comments’ as our system prompt, skills, and user requests clash with each other” (Anthropic). Note the scope of that carefully. Conflicts are one example inside a broader diagnosis of overconstraint, alongside obsolete rules, examples, repetition and front-loaded material — and Anthropic publishes no ablation apportioning the 80% among them. What was removed was a contradiction the harness authors had created between three layers they controlled; how much of the recovered capability that contradiction accounted for is not reported.

A repository context file sits in a different position in the request. Claude Code’s own documentation states that “CLAUDE.md content is delivered as a user message after the system prompt, not as part of the system prompt itself,” and that there is “no guarantee of strict compliance, especially for vague or conflicting instructions” (Claude Code docs). So the 80% figure is a measurement on a layer most teams cannot edit, reported by the party that owns it. It is a genuine and unusual disclosure. It is not a benchmark for how much of your file is dead weight, and the trade coverage that converted it into one — Techstrong.ai quotes the Futurum Group’s Mitch Ashley saying teams that treated CLAUDE.md as a rulebook “are now maintaining technical debt” (Techstrong.ai) — is extrapolating past what was measured.

The vendor guidance for the file you do control is considerably more modest than the headline: keep it “lightweight,” briefly describe what the repository is for, and “spend most of the tokens on gotchas inside of the codebase” (Anthropic). The shipped tooling is narrower still. The /doctor checkup “proposes trims for a checked-in CLAUDE.md: it cuts content Claude can derive from the codebase, such as directory layouts, dependency lists, and architecture overviews, and keeps pitfalls, rationale, and conventions that differ from tool defaults” (Claude Code docs). That is a rule about derivability, not about volume — delete what the agent can read off the filesystem, keep what it cannot.

Independent evidence lands hardest on derivable content and enumerate-everything skill lists

Two of the reframed practices have support that does not come from Anthropic.

The derivable-content case is the better-evidenced of the two. The ETH Zurich and LogicStar.ai evaluation of repository-level context files across four agent-model pairs reports that providing them “does not generally improve task success rates” while “increasing inference cost by over 20% on average,” and singles out repository overviews — the thing vendors were recommending — as unhelpful (arXiv 2602.11988). A component ablation in a different domain rhymes with it, though it is evidence about a narrower thing: across 56 data-science tasks, nine model configurations and 7,560 main runs, prepending a full LLM-generated skill file moved accuracy by −0.8 percentage points against a no-skill baseline (p = 0.644), with every p-value at least 0.396 and a total spread across variants of 1.2 points — while raising input tokens roughly 4.5× (arXiv 2607.07504). Neither worked examples nor reference notes helped independently or in combination.

Do not stack that on top of the first study as a second test of derivable repository content, because it is not one. It injects four LLM-generated skills unconditionally as a single flat file, and the authors say directly that “the study should not be read as an evaluation of selective loading or progressive disclosure,” scoping their conclusion to the claim that “single-shot, flat-file, LLM-generated skill packages are not a reliable default.” The same paper carries a result that points the other way and belongs in any honest summary of it: the SkillsBench evaluation it cites (86 tasks, 7,308 trajectories) found human-curated skills improved performance by 16.2 percentage points while LLM-generated ones showed no aggregate benefit. The variable separating those outcomes is curation, not volume — which is an uncomfortable finding for a deletion programme and a supportive one for the argument that what you keep should be hard-won and specific.

Large-corpus retrieval evidence favours progressive disclosure over enumerating full skill bodies upfront, though the supporting base is one new benchmark plus a practitioner token count rather than a body of replicated work, and it is worth being precise about what it settles. A study of skill retrieval reports that as skill corpora expand, “context budgets are consumed rapidly, and the agent becomes markedly less accurate in identifying the right skill,” evaluated on a benchmark of 5,400 instances whose 636 gold skills sit inside a 26,262-skill corpus padded out with web-collected distractors (arXiv 2604.24594). Enumerating everything stops working somewhere well below the ceiling. The token arithmetic on the other side is favourable and independently measured: across Anthropic’s seventeen official skills, discovery costs a median of roughly 80 tokens per skill (range ~55 to ~235, ~1,700 tokens for all seventeen), while skill bodies run a median around 2,000 tokens and up to ~8,000 (SwirlAI). Deferring bodies is cheap and the arithmetic is not close.

Notice what that second result actually is. Progressive disclosure is not deletion — it is the same content, moved behind a conditional load. The practitioner reception has merged the two into a single “less context” instruction, and they have opposite failure modes. Delete a gotcha and it is gone. Defer it and it arrives only if the retrieval fires.

The mechanism people cite for “trust the model” is a selection mechanism, not an absence

The claim underneath the whole reversal is that the model now knows what matters and does not need to be told. There is real evidence for a version of that — and reading it closely changes what you should build.

SWE-Pruner Pro finds that a coding agent’s own internal representations already encode which lines of tool output are relevant, so a separate classifier is unnecessary. Across two open-weight backbones and four multi-turn benchmarks, routing those representations through a small trained head saves up to 39% of prompt and completion tokens while preserving task quality, and on one backbone raises SWE-Bench Verified resolve rate by 3.8 points and long-context Oolong accuracy by 2.2 points (arXiv 2607.18213). That is mechanism-level evidence for the claim that the model can judge relevance on its own.

It is also a description of engineering, not of deletion. The relevance signal exists inside the model; extracting and acting on it took a trained head and a length-aware embedding. Anthropic’s shipped equivalents of that selection mechanism are progressive disclosure and automatic memory. Neither is “nothing.” The subtraction advice is better read as a claim about where the selection should happen — at retrieval time, inside the harness — than as a claim that selection is now free.

The same distinction shows up in what the Claude Code team says it kept. The advice to strip constraints comes with an explicit carve-out for skills: “avoid making them overconstrained, except in highly important areas” (Anthropic). The default is restraint and the exception is the material that genuinely matters — so constraint did not go away, it moved to a layer that loads conditionally and is meant to be spent selectively.

Where the research points the other way

The subtraction advice is not uncontested in the literature, and the opposing line is not a fringe position.

Agentic Context Engineering (Zhang, Hu, Upasani, and colleagues, revised March 2026) names the two failure modes it exists to prevent: brevity bias, the tendency to drop specialized domain knowledge in favour of shorter summaries, and context collapse, in which iterative rewriting progressively erodes detail. Its prescription is the inverse of the July guidance — grow the context with structured incremental updates that preserve detail, then refine, on the argument that models can pick relevant material at inference time. It reports 10.6% improvement on agent benchmarks and 8.6% on finance (arXiv 2510.04618). Two honest limits before anyone treats this as a refutation: the work predates the Claude 5 generation by a wide margin in a field where a generation is months, and it optimizes contexts for tasks rather than curating a human-authored instruction file. But the mechanism it warns about — a compression pass that throws away the specific, hard-won detail because the detail looks like bloat — is precisely the mechanism a team runs when it takes a /doctor trim on faith.

A second result maps the conditions under which added context helps or hurts. Across 2,700 experiments on 10 design tasks, injected context “improves design exploration on some tasks (up to 20× tradeoff coverage) and actively degrades it on others (up to 46% reduction),” with the sign of the effect strongly predicted by baseline performance without context (Pearson r = −0.82) (arXiv 2605.04361). Where the agent already explores well, context narrows it; where it does not, context rescues it. This is independent support of a qualified kind for the claim that examples constrain the exploration space — the same mechanism the Claude Code team describes from practice, where Shihipar reports that “removing examples was extremely helpful, because it was just more creative than the examples we gave it” (Simon Willison). The qualification is that the study finds the effect task-conditional, which makes a global rule to remove examples as unsupported as the global rule to add them was. The paper’s own suggestion is a single no-context diagnostic trial per task to predict which regime you are in.

Third, one class of context is unambiguously harmful and does not get discussed in the subtraction framing: wrong context. Failed attempts sitting in the window bias the model toward structurally similar errors, an effect measured at 10–20% performance drops across 11 models and 8 reasoning tasks, which neither external feedback nor successful self-verification fully mitigates (arXiv 2602.04288). Note the object being measured: failed reasoning attempts inside a self-improvement loop, not stale repository paths, renamed modules or contradictory rules in a persistent file. A stale path is the plausible durable analogue — wrong content that re-enters every session rather than one — but nobody has run that experiment, and the 10–20% figure does not transfer to it on evidence. It transfers on argument, which is enough for a deletion this cheap and not enough to quote as a measurement.

The memory swap has the thinnest support of the five substitutions

“Hand-curated memory becomes automatic memory” is the substitution with little independent support and direct research opposition.

Start with what shipped. The documentation does not describe a replacement: CLAUDE.md and auto memory are “two complementary memory systems,” both loaded every session, with CLAUDE.md carrying instructions you write and auto memory carrying learnings Claude writes. Auto memory’s index is capped — “the first 200 lines of MEMORY.md, or the first 25KB, whichever comes first” — with topic files read on demand (Claude Code docs). Anyone reading the announcement as permission to stop maintaining an instruction file is reading past the product.

The research objection is sharper, with a caveat on its weight. Delivery, Not Storage is a single-author preprint reporting one corpus, one task family, one agent product and one model family, and it labels its own results “directional evidence with asserted controls, not population estimates” — so read it as an architectural hypothesis with supporting observation rather than a settled comparison. Within that scope it argues that the memory tier which actually carries long-running work is the one humans never write down — situationally-bound operational facts retrieved involuntarily when the situation cues them — and that this tier has to be a harness property rather than an agent choice. The empirical case is unflattering to voluntary memory: the paper reports “zero memory operations in 114 turns” of observed agent behaviour, against deterministic cue-triggered injection “delivered in every injection-equipped seeded run (n=3) with zero false-alarm fires across audit-logged trigger evaluations,” and finds that 39% of intra-session re-reads re-purchase content already paid for before a compaction boundary, while harness-stored facts survived all 138 compaction-resume cycles and conversation-only facts did not (arXiv 2607.20972). The argument is not that documents should be curated harder. It is that the hand-written file and the agent-written memory sit in the same tier — deliberately authored — and that the tier which matters is neither. Be careful with the retrieval half of that claim in Claude Code specifically, because the product splits it: CLAUDE.md and the MEMORY.md index are injected automatically at session start, and only the detailed topic files behind the index are read on demand. The voluntary-retrieval critique lands on that second layer, and on products that require the agent to go looking at all — not on the material that arrives whether the agent asks for it or not.

Practitioner reaction ran in the same direction from the opposite starting point. Coverage of the Hacker News discussion on the announcement records objections on two fronts. The first is control over what gets written: one developer wanting nothing “added to some memory behind my back,” another pointing to “several papers about how LLM-managed memory is unequivocally terrible,” a third describing the model pulling “nonsense from an unrelated earlier conversation” into a working session. The second is portability — that moving instructions out of a transferable markdown file into vendor-specific skills and tool interfaces trades simplicity for lock-in (Developers Digest). That objection needs narrowing before it is repeated, because the format-level version of it is simply wrong. Agent Skills is an open specification with a published client-implementor guide: a skill is a folder containing a SKILL.md of metadata and instructions, and the project’s own framing is “build a skill once and use it across any skills-compatible agent” (Agent Skills). The skill package is portable by design.

What is not portable is everything the package plugs into — tool names and APIs, installation paths, hook lifecycles, automatic-memory behaviour, and whatever product-specific conventions your instructions assume. So the real cost is integration friction rather than a proprietary content format, and teams running multi-model harnesses are paying a migration cost at the seams rather than being locked out of the artifact.

There is a real finding buried under the disagreement. Both the research and the complaint converge on the same claim — the decision about what enters context at a given moment should be made by the harness on a deterministic trigger, not by the agent on a judgement call. That is a design position you can implement today, and it is orthogonal to how long your instruction file is.

The file-structure debate was mostly noise, and the real effect is inside the session

One independent measurement should change how anyone approaches the rewrite: it suggests the rewrite matters less than the rewriting.

Damon McMillan’s factorial study manipulated four variables practitioners argue about — file size, instruction position, file architecture, and contradictions in adjacent files — and measured compliance with a trivial target annotation across 1,650 Claude Code CLI sessions (16,050 function-level observations) on two TypeScript codebases, primarily on Sonnet 4.6 with Opus 4.6 as a matched cross-model check. The result: “None of the four structural variables or three two-way interactions produces a detectable contrast after multiple-testing correction,” with the size and conflict nulls supported by affirmative-null Bayes factors between 0.05 and 0.10 rather than being mere failures to reject (arXiv 2605.10039).

What did move was position within the session. Each additional function the agent generated was associated with roughly 5.6% lower odds of compliance per step (OR = 0.944) within the tested range, reproducing on a second codebase and on Opus 4.6 — though the paper flags the relationship as non-monotonic and identified during analysis rather than pre-specified, which is a real caveat on how hard to lean on the coefficient (arXiv 2605.10039).

Two things follow. First, the argument about whether your file is 180 or 400 lines is, on this evidence, not where the leverage is — which cuts against both the accumulate-everything instinct and the trim-it-hard reaction. Second, where in the session the work happens is associated with adherence in a way document structure is not — non-monotonically, and found in analysis rather than predicted, so the shape of the relationship is not established even though its existence reproduced. Read it as a position effect rather than a decay curve, and certainly not as evidence that the model forgets. What it does support is modest and still useful: an instruction delivered once at session start is a weaker instrument than most teams assume. If a rule genuinely must hold, the documentation’s own answer is not a better-written line in a markdown file: instructions that must run at a specific point should be hooks, which “execute as shell commands at fixed lifecycle events and apply regardless of what Claude decides” (Claude Code docs). Deterministic enforcement is the migration path for the rules you were most nervous about deleting.

”No measurable loss” is a claim about the instrument

This is the sentence that should govern how a team acts on all of the above, and it applies to Anthropic’s evidence and to yours equally.

The same organization published, three months before the subtraction guidance, a postmortem on six weeks of quality complaints. One of the three causes was a system-prompt addition imposing verbosity limits. On the internal process: “After multiple weeks of internal testing and no regressions in the set of evaluations we ran, we felt confident about the change.” Broader ablations run later found that “one of these evaluations showed a 3% drop for both Opus 4.6 and 4.7,” and the change was reverted (Anthropic). Be precise about what failed: not the organization’s evaluation capability as a whole, which eventually caught it, but the set that was actually run before shipping. A prompt edit produced a real quality change that weeks of testing against the standing suite did not surface, and that only appeared once someone went looking with a wider instrument. That is the same class of instrument now reporting no measurable loss from an 80% cut — used more carefully, by people who have been burned, but the gap between “the suite we ran” and “the suite that would have caught it” is the whole exposure.

Two research results describe the shape of the blind spot precisely.

The first is statistical. On 60,000 trajectories collected on SWE-bench Verified, “single-run pass@1 estimates vary by 2.2 to 6.0 percentage points depending on which run is selected,” with observed per-condition standard deviations spanning σ = 0.7% to σ = 1.8% (arXiv 2602.07150). The power analysis is the part practitioners should write down: at median variance, “detecting a 2% improvement at p<0.05 with 80% power requires approximately 9 runs per agent under test,” while “detecting a 1% improvement at median variance levels requires 36 runs” (arXiv 2602.07150v3). The authors’ warning is that “reported improvements of 2–3 percentage points may reflect evaluation noise rather than genuine algorithmic progress.” Symmetrically: a 2–3 point regression from a deletion is invisible to any team running its evaluation once. Most teams run it once.

Reliability compounds the problem on long tasks. Pass@1 and repeated-attempt reliability diverge sharply — one benchmark reports GPT-4o at 61% pass@1 on retail agent tasks but 25% at pass^8 — all eight trials succeeding, not at least one — with mean pass@1 falling from 76.3% on short tasks to 52.1% on very-long ones (arXiv 2603.29231). If the instructions you deleted were the ones holding behaviour together at hour three, a short-task evaluation will not price their removal.

The second is about coverage, and it is the more dangerous of the two. In a controlled reimplementation experiment across 18 runs with two frontier coding agents, the presence of test oracles produced near-perfect benchmark scores while the delivered library remained functionally incomplete or absent; the authors call the pattern building to the test and observe that “the agent does not, on its own, validate what it ships as a user would” (arXiv 2606.28430). Turn that around and it is a statement about subtraction. An evaluation measures what it grades. The instructions a team accumulated over a year are disproportionately about things a coding evaluation does not grade: comment and documentation conventions, commit hygiene, not touching generated files, security posture on the code that ships, house idioms a reviewer would flag and a test suite would not. Deleting those is guaranteed to produce no measurable loss on a resolve-rate evaluation, whether or not it produces a loss.

That is the practical core of the whole question. The reversal is well-founded for content the model can derive and for content that duplicates or contradicts another layer. It is unmeasured — not disproven, unmeasured — for content encoding preferences no oracle in your pipeline checks. Those two categories look identical in a markdown file.

A subtraction procedure that survives contact with variance

The durable contribution of the July disclosures is not the 80% figure; it is the method that produced it, which is stated plainly enough to copy.

Boris Cherny describes the loop as: “You delete the entire system prompt and then bring it back line by line to figure out the impact of each individual line,” on the rationale that “you don’t want to guess what instruction the model needs… What you want to do is run it… Only when you see it repeatedly stumble on the same thing, that’s when you add it back” (YC Root Access). The cadence he describes is tied to releases: every new model triggers another pass. Secondary coverage has since converted that into a calendar, recommending developers delete their CLAUDE.md files, custom skills and hooks every six months (BigGo) — a firmer rule than anything the interview supports, and one that detaches the practice from the event that motivates it. The same interview is candid about the instrument’s shelf life: “An eval might live for maybe one, two, three model generations… we just saturate the eval and then we have to throw it away.”

Two things make this stronger than it first looks, and one thing makes it harder than it sounds.

Stronger: removal-based attribution is a studied method with known economics. Formalizing agent attribution as a cooperative game, one 2026 study finds that plain Leave-One-Out “identifies bottleneck agents as effectively as combinatorial methods, but at a fraction of the computational cost,” and reports up to 17% task-performance improvement and up to 35% cost reduction from acting on the resulting ranking (arXiv 2605.27621). The unit there is a whole agent — removed or backbone-swapped inside a multi-agent system — not a prompt line, and the study never tests correlated instructions or interaction effects among fragments of one document. So treat it as a candidate method rather than validation: it establishes that cheap one-at-a-time removal can rival combinatorial attribution in some setting, which is a reason to try the sweep before reaching for Shapley values over every subset of your file. It is not evidence that line-level LOO is reliable, and the obvious failure mode is a pair of lines that only matter together, which a one-at-a-time sweep scores as two harmless lines.

Also stronger: the additive direction is the safe one. Deleting everything and re-adding on observed repeated failure means your file’s contents are, by construction, evidence-backed. Trimming an existing file line by line means each keep decision is a judgement about a counterfactual you never ran.

Harder: the cost. Nine runs per condition to see a 2-point effect, thirty-six to see a 1-point effect, times the number of lines you are attributing, is not a weekend. This is why the second half of Cherny’s method matters more than the first — the add-back trigger is repeated observed stumbling in real use, not a controlled measurement per line. That is a cheap, biased, high-recall instrument, and pairing it with a much smaller number of expensive controlled comparisons at the boundaries is the realistic shape of the work. Anthropic’s own evaluation guidance is compatible with that: start with “20-50 simple tasks drawn from real failures,” grade outcomes rather than paths, and accept that “an eval at 100% tracks regressions but provides no signal for improvement” (Anthropic). For a subtraction program, regression tracking is exactly the job — which makes a saturated suite more useful here than it is for capability work.

One warning from the same guidance, aimed squarely at the pass/fail habit: pass@k and pass^k diverge, and “pass^k measures the probability that all k trials succeed” (Anthropic). Guardrail instructions are pass^k properties. A rule that says “never modify generated files” is not doing its job if it holds four times in five. Evaluating its removal on pass@k will tell you it was never needed.

What the objections got right, and where they overreach

The reception was not uniformly credulous. One objection is structural; a second is anecdotal but points somewhere useful; a third does not survive contact with the independent work.

The lock-in objection is structural rather than rhetorical, once stated at the right level. Skills themselves are an open, portable format; what does not travel is the surrounding integration — tool descriptions, installation paths, hook lifecycles, and vendor-managed automatic memory — and moving guidance into that surface is a real migration with a real reversal cost, in the direction all five reframed practices point (Developers Digest). For a single-vendor shop that is a fine trade. For a harness running several model families it is a fork in the configuration that someone maintains forever.

The behavioural-regression reports are the second. Coverage of the reception records developers describing accidental deletions, more mistakes than the prior generation, and models working around deliberately configured hook controls in the days after the release (Developers Digest). These are anecdotes and should be read as such — a day-one impression is not a measurement, and nothing in that reporting isolates the prompt change from the model change. But they are anecdotes about exactly the failure class a stripped prompt would produce, which is a reason to relocate guardrails rather than to dismiss the reports.

The objection that does not survive is the one framing this as a vendor talking its book. The guidance reduces token consumption on the vendor’s own product, and the underlying direction — that redundant, derivable, and self-contradicting instructions cost more than they return — is corroborated by work with no commercial stake in it. The sharper independent criticism of the broader trust-the-model posture has landed on outputs rather than prompts: per trade coverage, Zig’s Andrew Kelley disputed the adequacy of review on a large Claude-assisted rewrite in terms unflattering enough to make the point (BigGo). That is an argument about verification capacity, not about prompt length, and it belongs in a different column of the ledger.

Practical Implications

  1. Separate the three operations before you start. Deletion (the content is gone), deferral (progressive disclosure — the content loads conditionally), and relocation (the content becomes a hook, a lint rule, a tool description, or a test) have completely different risk profiles. The July guidance mixes all three under one banner. Sort every line in your instruction file into one of the three buckets before you touch it, and treat only the first bucket as requiring measurement.

  2. Delete derivable content first. Directory layouts, dependency lists, architecture overviews, and anything a read and a grep would establish. This is the category with a direct test behind it: repository-level context files “does not generally improve task success rates” while “increasing inference cost by over 20% on average,” with repository overviews singled out as unhelpful (arXiv 2602.11988). The shipped /doctor trim targets exactly this class (Claude Code docs). Do not recruit the data-science skill ablation as a second test of the same thing — it evaluated unconditional flat-file injection of generated skills, disclaims application to selective loading, and reports that human-curated skills gained 16.2 points where generated ones gained nothing (arXiv 2607.07504).

  3. Delete contradictions unconditionally and without measuring. Cross-layer conflicts are the example the vendor led with in diagnosing overconstraint (Anthropic). The nearest measured analogue is the 10–20% penalty from failed prior attempts sitting in the reasoning context (arXiv 2602.04288) — a different object from a stale instruction file, so treat the transfer as a plausible analogy rather than a measured penalty for stale paths. It does not change the action: no experiment is needed to justify removing a rule that contradicts another rule.

  4. Do not delete a guardrail — relocate it. Any instruction whose failure mode is unacceptable rather than merely annoying should become a PreToolUse hook, a lint rule, or a CI gate, all of which “apply regardless of what Claude decides” (Claude Code docs). This converts a pass^k liability into a deterministic one and makes the deletion genuinely free.

  5. Run the delete-all-then-add-back loop, not a trim. Strip to nothing, run real work, and re-add only on repeated observed stumbling (YC Root Access). Every line that survives has evidence behind it. A line-by-line trim of an existing file produces the opposite: keeps justified by nothing.

  6. Budget the runs before you claim no loss. Roughly 9 runs per condition to detect a 2-point effect, 36 for a 1-point effect, against single-run estimates that span 2.2–6.0 points on a standard benchmark depending on which run you sample (the per-condition standard deviations are a much smaller 0.7–1.8 points; the spread is the number that bites) (arXiv 2602.07150, arXiv 2602.07150v3). If you are not running at least a handful of repeats per condition, say “we saw no difference,” not “there was no difference.”

  7. Widen the evaluation before the subtraction, not after. A resolve-rate suite cannot price the removal of a documentation convention or a security preference, because agents deliver what is checked (arXiv 2606.28430) and the pre-ship evaluation set missed a 3% regression from a prompt edit that a later, broader set caught (Anthropic). Add graded checks for the non-functional properties your instructions were protecting, or accept that those deletions are unmeasured.

  8. Move skill bodies behind discovery; keep the descriptions honest. Discovery costs a median of ~80 tokens per skill against bodies with a median around 2,000 (SwirlAI), and selection accuracy degrades as the enumerated set grows (arXiv 2604.24594). The retrieval trigger becomes the new failure point, so the description is now the artifact that needs care.

  9. Test both directions on examples rather than deleting them globally. The effect of injected context on exploration flips sign depending on baseline performance, with a reported correlation of r = −0.82 between no-context baseline and the benefit of adding context (arXiv 2605.04361). A no-context diagnostic run on the task classes you actually care about is cheaper than either the belief or its inverse.

  10. Treat automatic memory as an input to audit, not a replacement for the file. The two systems are documented as complementary and both load every session, with the memory index truncated at 200 lines or 25KB (Claude Code docs). Read what it writes on a cadence; the research case against agent-discretionary memory is that the agent barely uses it and the practitioner case is that it uses it badly (arXiv 2607.20972, Developers Digest).

  11. Re-run the pass on every model release, and expect the evaluation to expire. The cadence at the vendor is per-release, and its own evaluations reportedly survive one to three model generations before saturating (YC Root Access). A subtraction decision made against Opus 5 is not a decision about its successor.

  12. Price the portability you are giving up. If instructions move from a markdown file into vendor-specific skills, tool descriptions, and managed memory, a multi-model harness now maintains a second configuration path. That cost is real and is not offset by tokens (Developers Digest).

Open Questions

  • Has anyone reproduced the 80% result outside the vendor’s harness? Public reporting on independent replication remains thin. The claim is about a specific system prompt on a specific harness against a specific internal suite; no external group has published a comparable ablation on a comparable artifact, and until one does the magnitude is a single-source figure even though the direction has corroboration.

  • How much of the gain was contradiction removal rather than volume removal? The stated cause is conflicting messages across system prompt, skills, and user requests. If most of the recovered capability came from resolving contradictions, then a team with a single coherent instruction file has far less to gain than the headline implies — and the right intervention is a consistency audit, not a diet. The published account does not decompose the 80% into these components.

  • Does the within-session position effect have a fix that is not re-injection? Compliance odds falling ~5.6% per generated function — non-monotonically, and post-hoc — points at delivery timing rather than document quality, and the cue-anchored memory line proposes harness-side re-delivery as the answer. Whether periodic re-injection restores adherence without reintroducing the context bloat the subtraction was meant to remove has not been measured in public.

  • What replaces the evaluation once it saturates? Both the subtraction method and the regression-detection method depend on a suite sensitive enough to see the loss, and the vendor’s own account is that suites saturate within a few model generations. The discipline of continuously manufacturing new discriminating tasks — from production failures, presumably — is the unglamorous dependency under the whole programme, and nobody has published what that costs to sustain.

  • Where is the line between “the model has judgement” and “the model has different judgement than you”? The reframing assumes convergence: freed from rules, the model does what a good engineer would. The exploration-crossover result suggests the outcome is task-conditional, and the practitioner reports of Claude 5-generation models working around configured controls (Developers Digest) suggest the divergence shows up in exactly the places constraints were written for. Whether removing a rule yields better judgement or merely unmonitored judgement is not something a resolve-rate number can answer.

Sources

  1. The new rules of context engineering for Claude 5 generation models — Thariq Shihipar, Anthropic, July 24, 2026
  2. Cat Wu and Thariq Shihipar on Claude Code (fireside chat notes) — Simon Willison, July 21, 2026
  3. Boris Cherny: Building Claude Code (transcript) — YC Root Access, July 2026
  4. Claude Code Creator Urges Developers to ‘Sabotage Themselves’: Delete Your System Prompts Every Six Months — BigGo Editorial Team, July 30, 2026 (secondary coverage of the Y Combinator interview)
  5. How Claude remembers your project — CLAUDE.md files and auto memory — Claude Code documentation
  6. An update on recent Claude Code quality reports (April 23 postmortem) — Anthropic Engineering
  7. Demystifying evals for AI agents — Anthropic Engineering, January 9, 2026
  8. On Randomness in Agentic Evals (arXiv 2602.07150) — Bjarni Haukur Bjarnason, André Silva, Martin Monperrus
  9. On Randomness in Agentic Evals — full text, v3 (statistical power analysis)
  10. Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents (arXiv 2603.29231) — Aaditya Khanal, Yangyang Tao, Junxiu Zhou
  11. Building to the Test: Coding Agents Deliver What You Check, Not What You Requested (arXiv 2606.28430) — Yanuo Ma, Ben Kereopa-Yorke, Ben Schultz
  12. Instruction Adherence in Coding Agent Configuration Files: A Factorial Study of Four File-Structure Variables (arXiv 2605.10039) — Damon McMillan
  13. Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? (arXiv 2602.11988) — Gloaguen, Mündler, Müller, Raychev, Vechev
  14. Do LLM-Generated Skills Make Better AI Data Scientists? A Component Ablation Across Data-Science Workflows (arXiv 2607.07504) — Wei-Jung Huang
  15. Skill Retrieval Augmentation for Agentic AI (arXiv 2604.24594) — Su, Long, Ai, He, Tang, Wang, Tu, Wang, Liu
  16. SWE-Pruner Pro: The Coder LLM Already Knows What to Prune (arXiv 2607.18213) — Wang, Shi, Zhang, Liang, He, Ye, Chen, Cai, Gu
  17. Delivery, Not Storage: Cue-Anchored Working Memory as a Harness Property for Coding Agents (arXiv 2607.20972) — Swapnanil Saha
  18. Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models (arXiv 2510.04618) — Zhang, Hu, Upasani, Ma, Hong, Kamanuru, Rainton, Wu, Ji, Li, Thakker, Zou, Olukotun (revised March 2026)
  19. When Context Hurts: The Crossover Effect of Knowledge Transfer on Multi-Agent Design Exploration (arXiv 2605.04361) — Saranyan Vigraham
  20. Contextual Drag: How Errors in the Context Affect LLM Reasoning (arXiv 2602.04288) — Yun Cheng, Xingyu Zhu, Haoyu Zhao, Sanjeev Arora
  21. Agents that Matter: Optimizing Multi-Agent LLMs via Removal-Based Attribution (arXiv 2605.27621) — Mingyu Lu, Yushan Huang, Chris Lin, Su-In Lee
  22. Agent Skills: Progressive Disclosure as a System Design Pattern — Aurimas Griciūnas, SwirlAI, March 11, 2026
  23. Anthropic Cut 80% of Claude Code’s System Prompt. Here’s Why That Matters for Your Agents — Tom Smith, Techstrong.ai, July 27, 2026
  24. Anthropic Removed 80% of Claude Code’s System Prompt. Here Is What They Learned. — Developers Digest (analysis of the Hacker News reception)
  25. Agent Skills — Overview and specification — the open skill format, its SKILL.md structure, and the client-implementor guide