blog.rust.gay

Context windows and why bigger is not always better

2026-09-08 · ai llm

A larger context window lets an LLM see more tokens in one request, but it does not guarantee that the model will use all of those tokens reliably. Benchmarks and prompting guidance show a recurring pattern: as prompts get longer, retrieval quality, position sensitivity, cost, and latency become part of the system design problem. Bigger context is useful when the extra tokens are relevant and well placed; it is not a substitute for selecting, structuring, and evaluating the information you send.

Advertised context is not the same as effective context

Model vendors and model cards often advertise a maximum context length, but that number is only an upper bound on how many tokens the API or runtime will accept. It does not mean every task will work equally well near that limit.

The RULER benchmark, introduced by Hsieh et al. in April 2024, evaluated 17 long-context language models on 13 tasks that go beyond simple needle-in-a-haystack retrieval. The paper reports that despite near-perfect accuracy on the vanilla needle-in-a-haystack test, almost all models showed large performance drops as context length increased. It also states that while the evaluated models claimed context sizes of 32,000 tokens or greater, only half of them maintained satisfactory performance at 32,000 tokens.

This gap matters in production systems. A prompt that fits inside the window can still fail because the relevant fact is hard to locate, because the task requires multi-step reasoning over distant pieces of text, or because irrelevant material competes with the signal the model needs.

Position and distractors affect long-context use

"Lost in the Middle: How Language Models Use Long Contexts" (Liu et al., submitted July 2023, revised November 2023, accepted in Transactions of the Association for Computational Linguistics, 2023) studied how models use relevant information placed at different positions in long inputs. The authors analyzed multi-document question answering and key-value retrieval tasks. They found that performance could degrade significantly when the position of relevant information changed: models often performed best when relevant information appeared at the beginning or end of the input context, and performance dropped when models had to access relevant information in the middle of long contexts, including for explicitly long-context models.

Chroma's "Context Rot" technical report (published 14 July 2025) extended this line of evaluation across 18 LLMs, including models the authors list as state of the art at the time, such as GPT-4.1, Claude 4, Gemini 2.5, and Qwen3. The report says model performance varied as input length changed, even on simple tasks, and that lower needle-question similarity increased the rate of performance degradation. It also reports that distractors had non-uniform effects and that their impact became more prominent as input length increased.

The practical takeaway is not that long context is unusable. It is that "can fit in the prompt" and "will reliably use the right part of the prompt" are different claims. If an application depends on a specific fact inside a large context, the fact's position, wording, and surrounding distractors can change the outcome.

Costs and prompt structure still matter

Long prompts have direct costs beyond model quality. More input tokens increase request size, can increase latency, and make failures harder to debug because there is more text that might be influencing the answer. When a task only needs a small subset of a large corpus, retrieval, filtering, ranking, or summarization can reduce the amount of irrelevant context the model has to process.

Prompt structure also affects long-context behavior. Anthropic's prompting guidance for long contexts recommends placing long-form data near the top of the prompt, above the query and instructions, and structuring multiple documents with clear tags. It also notes that placing the query at the end can improve response quality by up to 30 percent in tests, especially with complex multi-document inputs. For long document tasks, it suggests asking the model to quote relevant parts before completing the task, which can help focus attention on the relevant material.

These techniques do not remove the limits identified by long-context evaluations, but they can make large prompts easier for a model to navigate. They are best treated as mitigations to measure, not guarantees.

Conclusion

Longer context windows are genuinely useful for tasks that need broad local evidence: reviewing a document set, carrying relevant conversation state, or analyzing a codebase slice. The mistake is to treat the maximum window as the default operating point. For long-context features, measure accuracy against the actual prompt shape you ship, test where relevant information appears, control distractors, and prefer sending less but better-selected context when reliability matters more than raw token count.

Sources

← back to posts