Before You Blame the Model, Check What Your Compactor Deleted

[Anuj Kapoor]

A lot of production AI incidents look like intelligence failures.

• The assistant asks for data the user already gave.
• It violates a constraint that was clearly stated earlier.
• It makes a recommendation that ignores a prior commitment.

The default diagnosis is: the model hallucinated.

Sometimes that is true. But often, the model is doing exactly what we asked with the context it was given. The real bug is that the context was compacted badly before the model saw it.

This article is about that failure mode, and how to fix it with a lightweight probe harness.

What you will get from this:

1. A clear mental model for why compression breaks decisions.
2. One concrete, end-to-end example from failure to fix.
3. A domain-agnostic pattern you can apply in support, healthcare, finance, legal ops, IT, or logistics.
4. A practical CI approach that does not require live model calls in unit tests.

The Core Idea in One Sentence​

Readable summaries can still be operationally wrong.

That sentence is the center of this problem.

Most teams review compaction output like editors:

• "Does this read well?"
• "Does this sound complete?"

Production systems need a different check:

• "Did the exact state needed for the next decision survive?"

When state drops, downstream behavior looks irrational even when the model is behaving consistently with the compacted input.

Why This Happens Across Domains​

This is not a customer-support-specific issue. It is a long-horizon workflow issue.

In every domain, there are fields that are low-frequency in text but high-impact in action:

• Healthcare: contraindications, dosage constraints, referral ownership.
• Finance: approval thresholds, reversal commitments, compliance flags.
• Legal ops: jurisdiction constraints, filing deadlines, required reviewers.
• Incident response: blast radius, rollback thresholds, escalation owners.
• Supply chain: cold-chain constraints, delivery exceptions, handoff ownership.

These fields are exactly what prose-style compaction tends to blur first.

Walkthrough: From Failure to Reliable Behavior​

Let us walk a single incident-assistant example through the whole pattern.

Step 0: The Failure​

An SRE asks an assistant to coordinate a production issue.

Initial facts:

• Scope: us-east-1 only
• Hard constraint: do not restart database
• Trigger: if error rate stays above 4% for 10 minutes, run rollback plan B
• Open owner task: platform engineer approves traffic shift

After several tool calls and runbook lookups, compaction runs. A few minutes later, the assistant proposes a global restart and misses the rollback threshold.

The model did not suddenly become dumb. The compacted state lost critical constraints.

Step 1: Measure What Is Filling the Window​

Before touching prompts, measure token composition.

In the reference implementation used for this article:

conversation:    445 tokens
tool results:   5400 tokens   (92% of the window)

This finding matters because it changes where you spend effort.

If tool payloads are 92% of the context, compressing dialogue alone is a minor optimization.

Step 2: Clear Spent Tool Payloads Before Compression​

If a tool result has already informed a decision, keep a re-fetch pointer and remove the heavy payload from active context.

def clear_tool_results(self, keep_pointers: bool = True) -> int:
    """Replace consumed payloads with re-fetch pointers and return tokens reclaimed."""
    reclaimed = 0
    for t in self.tools:
        if t.used:
            before = t.context_tokens()
            t.used = False
            t.cleared_note = (
                f"[{t.tool} result for {t.entity_id} cleared; re-call to refetch]"
                if keep_pointers else ""
            )
            reclaimed += before - t.context_tokens()
    return reclaimed

This single change usually improves both cost and quality stability, because it removes stale high-volume noise.

Step 3: Compact Into Typed State, Not Narrative​

Narrative summaries are good for readability and bad for regression detection.

Use a typed state target instead:

from typing import TypedDict


class TaskState(TypedDict):
    entities: list[str]           # IDs, service names, task anchors
    constraints: list[str]        # hard boundaries and guardrails
    commitments: list[str]        # promised actions with conditions/amounts
    decisions: list[str]          # completed decisions with rationale
    open_items: list[str]         # unresolved tasks with explicit owners
    risk_flags: list[str]         # safety/compliance/escalation markers

Why this works:

• Missing fields are visible.
• Output is machine-checkable.
• Review becomes objective instead of impressionistic.

Step 4: Add Probe Questions​

A probe is an executable question over compacted state.

from dataclasses import dataclass
from typing import Callable


@dataclass
class Probe:
    question: str
    answer: Callable[[TaskState], object]
    expected: object


def score(state: TaskState, probes: list[Probe]) -> tuple[int, list[str]]:
    passed, failures = 0, []
    for p in probes:
        if p.answer(state) == p.expected:
            passed += 1
        else:
            failures.append(p.question)
    return passed, failures

For the incident example, probes include:

• Is scope still us-east-1 only?
• Is do-not-restart-database still present?
• Is error_rate > 4% for 10m preserved?
• Is platform engineer still owner of traffic-shift approval?

If these fail, you caught the regression before a production decision.

Step 5: Separate CI Into Two Lanes​

This is where teams often get confused.

No, you do not need to run live model prompts in unit tests.

Use two lanes:

1. Deterministic lane in normal CI

• Tests probe predicates and scoring against fixed state fixtures.
• No network calls, no model dependency.

1. Compactor eval lane

• Runs fixed transcripts through the real compactor.
• May call a model.
• Runs nightly or as a release-quality gate.

This keeps developer feedback fast and still catches model/prompt drift.

What the Score Actually Tells You​

In the reference benchmark:

naive prose compactor: 3/6 pass (50%)
field-preserving compactor: 6/6 pass (100%)

The key point is not the headline percentage.

The key point is diagnostic clarity.

Instead of "the assistant is flaky," you can now say: "commitment preservation regressed on this release." That changes debugging speed and architecture decisions.

What You Can Achieve With This Pattern​

Once compaction is measurable, teams stop guessing and start engineering.

Here are practical moves this unlocks:

• Risk-tiered compaction strategies by workflow class.
• Policy probes as first-class tests.
• Context-budget tuning from measured loss curves.
• Domain-specific templates over a shared harness.
• Compaction quality as an SLO next to latency and cost.

This is the part most teams miss: compaction instrumentation is not only a bug fix. It becomes a capability multiplier.

A Practical One-Week Rollout​

If you want to try this without replatforming:

Day 1:

• Instrument token composition by source.
• Pick one long-running workflow where errors are costly.

Day 2-3:

• Define typed state schema.
• Write 8 to 15 probes from known incident patterns.

Day 4:

• Add deterministic probe tests to CI.

Day 5:

• Add compactor eval lane with fixed transcript set.
• Set baseline score and regression threshold.

After week 1:

• Turn every incident into a new probe.
• Keep the harness small and specific to business risk.

Where People Still Go Wrong​

Three anti-patterns keep showing up:

1. Treating readability as a quality proxy.
2. Running a giant eval set but never adding targeted probes from real failures.
3. Mixing deterministic tests and live-model tests into one unstable CI lane.

Avoid those, and this approach is straightforward to maintain.

Key Takeaways​

1. Context compression is a reliability surface, not just a token-optimization step.
2. Long-running failures are often state-loss failures disguised as reasoning failures.
3. Tool payload bloat is usually a bigger problem than dialogue length.
4. Typed state compaction makes correctness testable.
5. Probe scoring turns vague quality concerns into specific regressions.
6. Deterministic CI and model-eval CI should be separate lanes.
7. The pattern applies across domains wherever AI decisions depend on preserved context.

If you remember one thing, make it this: when an assistant appears forgetful, inspect compaction quality before you replace the model.

Sources​

1. Anthropic Applied AI Team, Effective Context Engineering for AI Agents (2025): https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents
2. Chroma, Context Rot: How Increasing Input Tokens Impacts LLM Performance (2025): https://research.trychroma.com/context-rot

Companion implementation repo