Scout: Context Compaction for Long-Horizon Agents — Techniques, Tradeoffs, and When Summaries Lie

Summary

Compaction — taking a trajectory that is about to overflow the context window, summarizing it, and resuming under a compressed context — has crossed from an ad-hoc coding-agent feature into a first-class technique with three distinct footings: a configurable API primitive in Anthropic’s tool runner, a production RL training objective inside the pipeline that trained the 750B-parameter GLM-5.2, and, as of this month, a measurable failure surface. The design space sorts cleanly by one axis — reversibility. Reversible reduction (drop what you can reconstruct, keep a pointer; Manus’s file-system approach) keeps a recovery path to what it drops; lossy summarization (Claude Code’s auto-compact, Cognition’s compression model) is where subtle context silently disappears; and learned compression — ACON’s distilled compressors, CompactionRL’s jointly-trained summaries — is the frontier’s attempt to make the model itself good at deciding what to keep. The sharpest new result sits just adjacent: a July belief-divergence study shows that lossy harness transforms — compressed repairs, evidence pruning — routinely preserve terminal task success while shifting the agent’s own read on risk, failure mode, and expected cost. It doesn’t isolate window-summarization, but a compaction pass is the same lossy move, and the drift it surfaces is exactly what a pass/fail benchmark cannot see. The practitioner takeaway is an ordering and an instrument: prefer raw over reversible compaction over lossy summarization, compact for quality rather than only for capacity, and measure the agent’s beliefs, not just whether it finished.

Key Findings

Compaction is now a primitive, a training objective, and a measured risk — not a hack

A year ago, “compact the context” meant a coding agent summarizing its own history when it ran low on room. That behavior has since hardened into infrastructure on three fronts, and the spread is what makes this worth a practitioner’s attention now.

As an API primitive, Anthropic’s tool runner exposes automatic compaction through a compaction_control block with an explicit context_token_threshold (default 100,000 tokens) and an optional dedicated summarization model, per the Claude compaction cookbook (November 2025). The mechanism is blunt and worth understanding exactly: when the threshold trips, the runner injects a summary request, Claude emits a summary, and then — in the cookbook’s own description — “the entire conversation history (including all tool results) is replaced with just the summary.” The retained/discarded ledger is explicit: ticket IDs, categories, and progress status survive; “full knowledge base article text, complete drafted response text, detailed classification reasoning” are discarded. That is not a lossless operation, and the docs don’t pretend otherwise.

As a training objective, compaction has moved inside the model. CompactionRL (Li, Hou, Jing, Tang, Dong) trains the agent to jointly optimize task execution and the summaries that let it continue, and the authors state plainly that “CompactionRL is thus deployed in the RL pipeline for training the open GLM-5.2 model (750B-A40B).” Compaction is no longer only something a harness does to a model; it is something a frontier model is now explicitly trained to do to itself.

As a measured risk, the harness-induced belief-divergence study (Yi, Song; July 2026) supplies instrumented evidence that lossy context-shaping can change what the agent believes, not just what it can recall — covered in detail below. Lilian Weng’s July 4 harness-engineering roundup frames the surrounding discipline’s baseline rule the same way: a harness “should not carry the entire workflow and all logs in context; instead, it should keep durable state in files.” Compaction is the operation that happens when you can’t, or didn’t.

The reason this matters earlier than window-overflow suggests: models degrade well before the window is full. Chroma’s Context Rot report (Hong, Troynikov, Huber; July 2025) evaluated 18 models spanning several generations (including state-of-the-art GPT-4.1, Claude 4, Gemini 2.5, and Qwen3) and found that “model performance degrades as input length increases, often in surprising and non-uniform ways” — not a smooth slope but uneven decline. The operational implication is that a strategy of “compact only at the context limit” is already compacting too late for quality; the useful window is smaller than the advertised one.

The design space has one real axis: reversibility

Strip away vendor vocabulary and the compaction techniques sort by a single question — can you get the dropped information back?

Reversible reduction (offloading). The safest family drops only what can be reconstructed and keeps a pointer to it. Manus’s context-engineering writeup (Yichao “Peak” Ji, July 2025) is the clearest statement of the principle: “compression strategies are always designed to be restorable. For instance, the content of a web page can be dropped from the context as long as the URL is preserved,” and, bluntly, “any irreversible compression carries risk.” Lance Martin’s writeup of a Manus webinar (October 2025) sharpens the operational form: every tool result carries a full format (raw content, parked in the sandbox filesystem) and a compact format (a file-path reference), so the agent can refetch on demand. The file system becomes external memory; the context holds addresses, not payloads.

Lossy summarization. In production coding agents, the default “compaction” is irreversible: summarize the trajectory, discard the originals, resume. Anthropic’s context-engineering guide (September 2025) defines it as “the practice of taking a conversation nearing the context window limit, summarizing its contents, and reinitiating a new context window with the summary,” and describes Claude Code’s version as preserving “architectural decisions, unresolved bugs, and implementation details while discarding redundant tool outputs or messages.” This is where the risk lives, and Anthropic names it directly (see the next section). LangChain’s context-engineering taxonomy (July 2025) files this under “compress” — one of four strategies alongside write, select, and isolate — and flags the catch: “summarization can be a challenge if specific events or decisions need to be captured.”

Learned compression. The frontier response to lossy summarization’s fragility is to stop hand-writing the summarizer and train it. Two approaches bracket the space. ACON (Kang et al., Microsoft/KAIST, October 2025) optimizes a compression prompt by analyzing failed trajectories in natural language, then distills the large-model compressor into a small one — reporting that “distilled compressors retain over 95% of the performance of the gpt-4.1 teacher” while cutting overhead; separately, ACON’s overall method reduces peak token usage by 26–54% against existing compression baselines. CompactionRL goes further and folds the summary into the RL objective itself, so the model learns summaries that maximize downstream task success rather than summary readability. Both start from the same admission ACON states outright: “token truncation or generic summarization are insufficient, as they easily discard these critical details essential for multi-step reasoning.”

Structured eviction. A minority position argues the whole summarize-and-discard move is the wrong primitive. “Beyond Compaction: Structured Context Eviction” (Semenov, Dorofeev; May 2026) annotates the trajectory as “typed, dependency-linked episodes” and evicts content by a deterministic, LLM-free policy — dropping the oldest-and-most-recoverable items by dependency rather than oldest-in-time. Its pitch is a list of the failures it claims to avoid relative to summarization: “unpredictable lossiness, destruction of causal structure, blocking model cost, and compression-induced hallucination,” alongside a striking single-session result — “89 sequential tasks across 80 million tokens with no measurable degradation.” It is one paper with its own benchmark and should be read as a promising direction rather than a settled win, but the critique it levels at summarization is the honest one.

When summaries lie: the failure isn’t lost facts, it’s rewritten beliefs

The teams running compaction in production keep hitting the same first-order failure: you cannot know in advance what will matter. Anthropic puts it crisply — “overly aggressive compaction can result in the loss of subtle but critical context whose importance only becomes apparent later.” Manus reaches the identical conclusion from production: “you can’t reliably predict which observation might become critical ten steps later.” This is why the reversible-reduction camp keeps pointers instead of trusting a summarizer’s judgment about relevance.

But the more consequential failure is subtler than dropped facts, and it is what the belief-divergence study measures. The authors elicit a structured K-step “belief rollout” from the agent — its own read on progress, risk, recoverability, constraints, failure mode, uncertainty, success probability, repair cost, and next action — and compare how that trajectory evolves under different harness treatments while holding the task, environment, and base model fixed. The tested treatments are harness transforms, not window-summarization per se — but two of them (compressed repairs, evidence pruning) are compaction by another name, which is why the central finding lands for anyone running it: “blocked actions, compressed repairs, selective verification, and cost-aware evidence pruning often preserve terminal success while changing the beliefs that drive later decisions.” When a repair-heavy harness compresses a failure-and-fix into a clean repaired transition, the agent loses the evidence it needed to predict failure modes — the study reports that expanding those compressed repair traces materially changes the agent’s beliefs even as the final outcome holds. The conclusion the authors draw is one the field has mostly ignored: “harness design is an experimental variable in agent evaluation, not an implementation detail.”

Put the two together and “when summaries lie” stops being merely rhetorical — as a transfer, not a proven equivalence. The study tested compressed repairs and evidence pruning rather than window-overflow summaries directly; but a compaction pass is the same kind of lossy transform, so the mechanism plausibly carries: it can leave the agent able to finish the task while quietly changing the beliefs it’s acting on — how risky the task is, what’s already gone wrong, what it should do next — where a pass/fail benchmark, by construction, cannot detect the difference. This is confidence flattening at the architecture level: the compressed transcript reads as settled fact, and the uncertainty that lived in the raw failure path is gone.

The summarizer is the whole ballgame — which is why teams are training dedicated ones

If the summary is where information and belief get rewritten, then the quality of the summarizer is the variable that decides everything downstream — and the evidence that it dominates is quantitative. CompactionRL reports that changing only the summary agent, holding the execution agent fixed, moves final task performance by 6.5 absolute points (49.0% to 55.5% in their Table 1). The summarizer is not a utility function bolted onto the agent; it is a first-class determinant of task success.

The production teams reached this conclusion before the RL papers did. Cognition’s Don’t Build Multi-Agents (Walden Yan, June 2025) describes introducing “a new LLM model whose key purpose is to compress a history of actions & conversation into key details, events, and decisions,” and is candid that “this is hard to get right” — hard enough that “you might even consider fine-tuning a smaller model (this is in fact something we’ve done at Cognition).” ACON’s distillation and CompactionRL’s joint training are the same instinct formalized: the compressor deserves its own optimization loop. For most teams the practical read is that off-the-shelf “summarize this conversation” prompting is one of the weakest points in a long-horizon agent, and the one most worth investing in — whether by a purpose-built prompt, a distilled model, or careful schema design.

There is a real cost tension here that the sources don’t resolve. A dedicated summarization pass is a blocking model call on the critical path (“Beyond Compaction” lists “blocking model cost” among summarization’s drawbacks), and a fine-tuned compaction model is infrastructure most teams won’t build. The reversible-reduction camp sidesteps both by not summarizing at all until it has to — which is the ordering the next section makes concrete.

Compaction alone doesn’t carry a multi-session agent

One more limit is worth stating because it is the failure teams hit right after they get single-window compaction working. Anthropic’s long-running-agents guide (November 2025) reports that even Opus 4.5 across multiple context windows falls short on compaction alone — “however, compaction isn’t sufficient” — because “compaction doesn’t always pass perfectly clear instructions to the next agent.” The two failure modes they observed are specific and recognizable: over-ambition (an agent takes on too much, runs out of context mid-implementation, and leaves the next instance undocumented work) and premature completion (a later instance sees existing progress and wrongly declares the project done). Their fix is not a better summarizer but external scaffolding — explicit feature lists, progress files, and git commits that survive independent of any summary. Compaction manages the window; durable artifacts manage the horizon. Teams that treat compaction as the whole context-management story tend to discover this boundary the hard way, one hallucinated “task complete” at a time.

Practical Implications

Order your reductions: raw > reversible compaction > lossy summarization. This ordering is about recoverability of information, not raw performance — Chroma’s context rot means more raw tokens can themselves cost quality, so “keep raw” holds only while the window is still serving you (see the quality trigger below). When raw stops paying its way, first drop only what you can reconstruct and replace it with a pointer — a file path, a URL, an artifact ID — so the agent can refetch on demand (the Manus pattern). Reach for irreversible summarization only when reversible reduction stops freeing enough space. Staging the reductions this way means the lossy step operates on the smallest, oldest slice of the trajectory rather than the whole thing.
Compact for quality, not just for capacity. Because degradation sets in well before the window is full (Chroma’s context-rot finding), the trigger threshold is a quality knob, not just an overflow guard. If long-horizon accuracy matters more than token cost, compact earlier and keep the active window comfortably under the advertised limit. Treat the context_token_threshold (or its equivalent) as a tunable you set against observed task degradation, not a value you leave at default and forget.
Never summarize the recent tail. Keep the last few steps in raw, full-detail form — CompactionRL retains the two most recent steps by default. The immediate next decision depends on precise recent state; compress the distant past, not the live edge.
Externalize memory; keep addresses in context, payloads on disk. One of the highest-leverage moves is to write durable state to files and hold references in the window (Anthropic’s note-taking pattern, Manus’s file-system-as-memory, Weng’s “keep durable state in files”). This turns most of the compaction problem into a retrieval problem, which is reversible by construction.
Treat the summarizer as a component you own. A generic “summarize the conversation” prompt is one of the weakest links in a long-horizon agent and moves task success by several points on its own. Invest proportionally: design a schema for what the summary must preserve (goal, completed actions, open errors, current state, next steps — the CompactionRL summary contract is a reasonable template), and if the domain justifies it, distill or fine-tune a dedicated compaction model rather than borrowing the main model’s off-the-shelf summarization.
Instrument beliefs, not just outcomes. A passing benchmark can hide a compaction pass that rewrote the agent’s model of risk and failure. Where reliability matters, periodically elicit the agent’s own read on progress, remaining risk, and known failures, and check that it survives compaction intact — the belief-rollout idea applied as a smoke test. If the agent’s stated confidence or failure-mode read jumps discontinuously across a compaction boundary, the summary dropped something that mattered.
Back compaction with durable artifacts for anything spanning sessions. For multi-window or multi-session work, don’t rely on the summary to carry intent to the next instance. Maintain an external, human-legible record of scope, done-vs-remaining, and decisions (feature list, progress file, commit history). Compaction keeps a single run inside its window; artifacts are what keep a long project from declaring itself finished early.

Open Questions

Is summarize-and-discard the right primitive at all, or a transitional one? Structured eviction argues the causal graph of the trajectory should govern what’s dropped, not a summarizer’s prose judgment, and reports no measurable degradation across 80 million tokens in one session. Whether that generalizes beyond its own benchmark — and whether dependency-annotation overhead is worth it versus a well-tuned reversible-reduction scheme — is unsettled.
What is the actual cost curve of good compaction? A dedicated summarization model is a blocking call on the critical path, and a fine-tuned compactor is infrastructure most teams won’t build. Public reporting on the latency and dollar cost of production compaction, and where the break-even sits against simply buying a larger context window, remains thin.
How much does training-time compaction (CompactionRL) transfer to harnesses it wasn’t trained against? A model trained to compact its own context inside one agent framework may or may not compact well under a different harness’s triggers, summary schema, and tail-retention policy. Whether learned compaction is a portable capability or couples the model to its training harness is unresolved in the public results.
Can belief drift be caught cheaply in production? The belief-rollout diagnostic is a research instrument; eliciting a nine-dimensional belief vector on every compaction is not free. Whether a lightweight proxy — a single confidence or failure-mode probe across the boundary — is enough to flag the dangerous cases is an open and practically important question.
Where does the security surface move when compaction becomes automatic? A summarizer is a model reading a full untrusted trajectory and deciding what the next instance sees. If attacker-influenced content can steer what a compaction pass preserves or drops, compaction becomes a place to plant or erase context across a trust boundary. The analysis of that surface hasn’t caught up to the technique’s adoption.