Context rot: why LLM answers get worse as input grows, and what to send

Context rot is the way language models answer worse as the input gets longer, even when everything they need is in the context. It matters to any system that feeds documents, chat history or tool output to a model, because “send it all, the model will find it” does not hold up.

Facts checked against Chroma’s Context Rot report (July 2025) and its repository, as of October 2026. The code example was tested on Python 3.12.

What the study measured

Chroma published Context Rot: How Increasing Input Tokens Impacts LLM Performance on 14 July 2025. It tested 18 models from four vendors: five from Anthropic (Claude Opus 4, Sonnet 4, Sonnet 3.7, Sonnet 3.5, Haiku 3.5), seven from OpenAI (o3, GPT-4.1 plus mini and nano, GPT-4o, GPT-4 Turbo, GPT-3.5 Turbo), three from Google (Gemini 2.5 Pro, 2.5 Flash, 2.0 Flash) and three sizes of Qwen3.

Unlike the classic “needle in a haystack” test, the needle did not share exact words with the question, which is closer to real use. The key design choice: task difficulty stayed fixed while only input length changed. When scores drop, length is the cause, not a harder task.

Five findings

Experiment	What changed	Result
Question-answer similarity	Question worded unlike the answer (low similarity)	Scores fell faster with length than for similar pairs
Distractors	0, 1 or 4 passages that look like the answer but are not	One distractor hurt; four hurt more; each hurt by a different amount
Haystack structure	Coherent text vs shuffled sentences	Shuffled did better, across all 18 models
LongMemEval	Focused prompt ~300 tokens vs full ~113,000 tokens	Every model did significantly better on the focused prompt
Repeated words	Copy a sequence of repeated words, 25 to 10,000 words long	Steady decline with length, in every model

The third result is counterintuitive and the authors say they cannot yet explain the mechanism, so do not read it as “shuffle your documents”. The lesson is simpler: models do not attend to their context uniformly.

Behaviour also differed by vendor. Under distractors, Claude models had the lowest hallucination rates and often declined to answer when unsure; GPT models had the highest, often answering wrongly with confidence. A system that needs accuracy has to handle both an “I don’t know” and a confident wrong answer.

The context window tells you how much fits, not how well it will be used.

Worked example: a 40-turn chat

Start from the LongMemEval numbers: full prompts averaged ~113,000 tokens, focused ones ~300.

• 113,000 ÷ 300 ≈ 377 times more input
• The relevant share of the full prompt is about 300 ÷ 113,000 ≈ 0.27%; the model has to wade through the rest

Now a hypothetical support assistant after 40 turns, about 2,500 tokens per turn including tool output:

1. Full history = 40 × 2,500 = 100,000 tokens
2. The latest question really needs two turns = 2 × 2,500 = 5,000 tokens
3. Relevant share = 5,000 ÷ 100,000 = 5%. The other 95% is noise, and some of it is a real distractor, such as the delivery address the customer has since changed

Send a 500-token summary plus the two relevant turns, 5,500 tokens in all, and input drops by 1 − 5,500 ÷ 100,000 = 94.5%: more accurate, cheaper and faster. (These figures are a worked hypothetical, not from the study.)

Rules for what goes into the context

1. Set a token budget per question and fill to the budget, not to the window. Anthropic’s context engineering guide puts it as finding the “smallest possible set of high-signal tokens”.

2. Remove distractors before removing noise. The dangerous material is what looks like the answer: an old version of a document, a customer record with a similar name, staging config next to production config. If your data has versions, send only the current one.

3. Filter, do not dump. A RAG system should apply a minimum relevance score, not always send its top-k.

4. Give long reading jobs their own room. Let a subagent read the big pile in its own context and return a summary. Anthropic says such summaries are often 1,000-2,000 tokens, even when the subagent used tens of thousands.

5. Start a fresh session when the task changes rather than dragging one chat along all day. Write what must carry over into a file, as in the handoff-files article.

6. Clear tool output you are done with. A long grep result or log from ten turns ago is rarely needed again, but it still takes space.

Code: fit retrieved chunks to a budget

Standard library only. It takes scored chunks from your search, drops low scores, drops near-duplicates (often an older version of the same document), and fills up to the budget:

from difflib import SequenceMatcher

def pick_context(chunks, count, budget=2000, min_score=0.5, dup_ratio=0.9):
    """chunks: (score, text) pairs from retrieval; count: your tokenizer's counter."""
    picked, used = [], 0
    for score, text in sorted(chunks, reverse=True):
        if score < min_score:
            break                                   # the rest are noise and distractors
        if any(SequenceMatcher(None, text, p).ratio() > dup_ratio for p in picked):
            continue                                # near-copy, e.g. an old version of the same doc
        cost = count(text)
        if used + cost > budget:
            continue                                # over budget: leave it out
        picked.append(text)
        used += cost
    return picked, used

chunks = [
    (0.91, "Refund policy v3 (2026): refunds within 14 days of delivery."),
    (0.89, "Refund policy v3 (2026): refunds within 14 days of delivery!"),
    (0.74, "Shipping takes 2-4 business days inside Thailand."),
    (0.41, "Company picnic is on Friday."),
]
words = lambda t: len(t.split())                    # stand-in counter for the demo
picked, used = pick_context(chunks, words, budget=50)
print(len(picked), used)   # 2 17

Two chunks survive: one copy of the refund policy (the near-duplicate is dropped) and the shipping note; the picnic falls below the score threshold. In production pass your model’s own token counter instead of words, since token counts per word vary a lot between languages.

difflib only catches text that looks almost identical. A distractor with similar meaning but different wording has to be stopped at the source: store metadata such as version, date and status, and filter on it before searching.

Common mistakes

• Trusting the context window number and loading a whole folder because “it fits”. Fitting is not the same as being used well
• Testing with exact-match lookups and concluding the model handles long context fine. Real questions rarely share words with their answers, and those cases degrade faster
• Keeping every version of a document in one index, so search returns the old and the new. That is manufacturing your own distractors
• One chat for the whole day across unrelated tasks, so every new question wades through all of it
• Evaluating at short length and deploying at long length. If production sends 50,000 tokens, test at 50,000. Chroma’s repo lets you rerun the experiments on your models

Before adding something to the context, ask whether it helps answer this question or just makes you feel you sent everything.

Sources

• Context Rot: How Increasing Input Tokens Impacts LLM Performance — Chroma
• chroma-core/context-rot (replication code) — GitHub
• Effective context engineering for AI agents — Anthropic