The Position of Your Context Matters for LLMs

The most reliable finding in long-context research is also the most uncomfortable. Where you put a piece of information inside a prompt changes the model's answer more than what the information is. The same documents, the same question, the same model. Move the relevant passage from the start of the context to the middle and accuracy can drop by twenty points. Move it back and the model is fluent again.

This is not a quirk of one architecture or one training run. It shows up in every frontier model that has been tested rigorously, across three years and a dozen benchmarks. It shows up at 4K tokens and at 1M tokens. It shows up before any weights are even trained. The shape is so consistent that recent theory has begun treating it as an architectural property of decoder-only transformers, not a bug to be patched.

This piece is what the research actually says. The papers, the numbers, the geometry, and what to do with it on Monday morning.

The Shape Everyone Found

The reference experiment is Liu et al., “Lost in the Middle” (Stanford, TACL 2024). Take a multi-document QA setup with 20 retrieved passages where exactly one contains the answer. Hold everything else fixed and slide the answer passage from position 1 to position 20. Plot accuracy against position.

The curve is a U. Models score highest when the answer is at the beginning of the context (the primacy bias) or at the very end (the recency bias). When the answer is buried in the middle of a 20-document context, performance collapses, often below what the same model scores with no documents at all. GPT-3.5-turbo with the answer mid-context dropped under its own closed-book baseline of 56.1%. The model performed worse with the answer in the prompt than without it.

What made the result hard to dismiss was that it persisted across model size, architecture, and explicit long-context training. The paper found the shape in GPT-3.5, Claude 1.3, Llama-2, and purpose-built long-context models. Closing the window did not help; opening it made the trough deeper.

A year later, Hsieh et al. (Google + UW, ACL Findings 2024) measured the same effect inside the attention mechanism itself. They quantified the average attention weight that LLMs assigned to each position, independent of content, and found a U-shape there too. The model attends more to the start and end of its input as a function of position alone. Their calibration method, which subtracts this intrinsic bias from attention scores, recovered up to 15 percentage points on tasks where the gold document was in the middle. The U was not just an output artifact. It was inside the math.

Why the U Exists

For two years the prevailing explanation was that the U was learned. Models saw most of their important content at the start or end of training documents (titles, summaries, conclusions), so they inherited a positional prior at training time. Recent theory says the cause runs deeper than that.

Wu, Wang, Jegelka, and Jadbabaie (MIT, ICML 2025) built a graph-theoretic framework for position bias in multi-layer attention. Modeling each attention mask as a directed graph, they proved that causal masking inherently biases attention toward earlier positions, because tokens in deeper layers attend to increasingly contextualized representations of those early tokens. The bias is not learned; it is the geometry of stacking causal attention.

Their attention-only analysis predicted pure primacy (full collapse onto the first token at infinite depth), which empirical models do not show. Two 2026 papers (Herasimchyk et al., A Residual-Aware Theory) and (Chowdhury, Lost in the Middle at Birth) resolved the discrepancy by adding residual connections to the theory. Once you do, the closed-form influence density splits into three regimes:

• The primacy tail. Causal masking forces a logarithmic divergence of gradient influence at the start of the prompt. Early tokens become anchors that every later token reaches back to.
• The recency delta. Residual connections create an isolated, full-strength anchor at the final token. The token generating the next output is one residual hop away from everything it just produced.
• The factorial dead zone. Between the two extremes, influence falls as 1/(H−1)! where H is network depth. The middle is a topological valley that training does not climb out of.

Chowdhury validates the theory empirically. Untrained Qwen2 and GPT-2 architectures exhibit the U at Step 0, with random weights, and the shape is identical with or without RoPE. Standard pretraining does not flatten it. The U is what the architecture looks like before anything has happened to it. Everything we do downstream is built on that prior.

Longer Means Deeper

The U is the part of the picture that survives at any length. The part that gets worse with length is the trough. Modarressi et al., NoLiMa (Adobe Research, ICML 2025) is the cleanest evidence. They built a needle-in-a-haystack benchmark in which the question and the needle share minimal lexical overlap, forcing the model to retrieve via semantic association rather than string match. Twelve frontier models that all claim 128K+ context windows scored near-perfect under 1K tokens. At 32K, ten of the twelve fell below 50% of their short-context baseline. Even GPT-4o, the leader, dropped from 99.3% to 69.7%.

Chroma's Context Rot study (Hong, Troynikov, Huber, July 2025) made the result domain-general. Across 18 frontier models including Claude Opus 4, GPT-4.1, Gemini 2.5 Pro, and the Qwen3 family, every single model degraded with length on five distinct experiments. The headline numbers from their report:

• All 18 models showed accuracy drops with longer input, well before the advertised window limit.
• Models did worse on logically coherent haystacks than on randomly shuffled ones, inverting the “tidy your context” instinct.
• A length-only floor of about 7.9 percent shows up before any distractors are added; pure length is a tax on accuracy regardless of what fills the tokens.

Read alongside RULER, NoLiMa, and Lost in the Middle, the picture is consistent. The U is real. The middle deepens with length. The 1M-token window is a marketing number, not a working number.

The Window Is Not the Window

To make this measurable, Hsieh et al. at NVIDIA built RULER (2024), a synthetic benchmark with 13 long-context tasks across retrieval, multi-hop tracing, and aggregation. They define effective context length as the longest input where a model averages above Llama-2-7B's 4K score (85.6%) on the suite. The threshold is deliberately low; it is what a small, short-context model already cleared in 2023.

On that bar, half the frontier of 2024 falls short of its own number plate. Gemini 1.5 Pro (claimed 1M) is effective at 128K. GPT-4 (claimed 128K) is effective at 64K. Claude 3.5 Sonnet (claimed 200K) is effective at 64K. Mistral-v0.2 (claimed 32K) is effective at 16K. DBRX (claimed 32K) is effective at 8K. LWM (claimed 1M) is effective below 4K. The gap is not a few percent; it is often an order of magnitude.

The 2025 generation of models pushed the bar higher. Qwen3-32B and GLM4 hold above the threshold past 128K on the same suite. But the structure of the gap persists: claimed length is the maximum a model will accept, effective length is the maximum where it answers reliably, and the two are different numbers. Use the wrong one to design a system and you have a bug, not a feature.

Length Itself Is a Tax

The next question is whether the degradation comes from distractors, from coherent-but-irrelevant context, or from raw length. Levy, Jacoby, and Goldberg (Bar-Ilan + AI2, ACL 2024 Outstanding Paper) settled it with FLenQA, a controlled True/False reasoning benchmark.

The setup is sterile by design. Each sample contains two short facts that together imply the answer. The reasoning never changes. What changes is the amount of irrelevant padding wrapped around them, with controls for padding type, location, and dispersion. They generated versions at roughly 250, 500, 1K, 2K, and 3K tokens, every model getting the same logic at every length.

Average accuracy drops from around 0.92 at the shortest input to 0.68 at 3,000 tokens, across GPT-4, GPT-3.5, Claude 2.1, Gemini-Pro, Mistral 70B, and Mixtral 8x7B. The trend is universal across models, padding types, and dispersion.

Three thousand tokens. Not three hundred thousand. Well below the advertised window of every model tested. The lift you would intuitively expect from “more context” reverses into a consistent loss the moment the relevant signal stops fitting in a short prompt. Length is not free; it is paid out of accuracy before any distractor is added.

Combine FLenQA with the 7.9% length-only floor that Chroma measured in 2025 and a clean rule emerges. Two budgets are at stake on every long prompt: the budget the model says it has, and the budget the model can actually reason over. Treat the second as a fraction of the first.

The Practical Playbook

You cannot retrain the attention mechanism, and the calibration mechanisms that work in the paper (Found-in-the-Middle, residual rebalancing) are not exposed in production APIs. What you can do is engineer around the geometry. The four moves below survive contact with production systems because they respect the U instead of fighting it.

1. Edge-load the prompt

Place the most important documents and the question at the very start or the very end. Anthropic's own long-context guidance recommends putting 20K+ token documents at the top and restating the actual query at the bottom; their internal tests show up to a 30 percent improvement on multi-document tasks from this layout alone. Treat the middle 60% as eviction-cache territory: bulk reference material lives there, but no critical instruction or load-bearing fact does.

2. Cap the budget aggressively

The window on the spec sheet is not the window the model uses. A working rule that survives both RULER and Context Rot data is 25 to 30 percent of the advertised maximum. Past that, the length-only tax compounds with the U-shape and eventual collapse becomes nonlinear. If you must go longer, instrument what you ship: needle-in-a-haystack regression tests at the lengths your prompts actually reach, not just at 4K.

3. Retrieve, do not stuff

BABILong showed that retrieval-augmented generation hits roughly 60% on single-fact QA independent of context length, while in-context reasoning collapses past 32K and frontier models effectively use only 10 to 20% of their advertised context. The implication for production: prefer compaction, summarization, and just-in-time tool calls over stuffing the whole corpus into the prompt. Keep the working set small and the relevant signal close to the model's edges.

4. Structure the surface

When you must place documents in long context, wrap them in delimiters (XML tags, named blocks), keep their order stable across calls, and instruct the model to quote relevant passages before answering. The quote-first pattern is a poor man's attention calibration: it forces the model to re-emit the load-bearing tokens at the recency anchor before generating an answer, which moves them from the dead zone to the edge of the prompt. Found in the Middle measures up to 15 percentage points of recovery from the formal version of this; the informal version is one extra sentence in your system prompt.

What Comes Next

The 2026 research is converging on three honest answers about what can fix this and what cannot.

Position bias is structural, not stylistic. The theoretical work from MIT and the residual-aware analyses from early 2026 establish that the U-shape is what causal decoder-only transformers do by default. Asking standard pretraining to remove it is asking gradient descent to undo geometry. The shape persists at random initialization; pretraining does not climb out of the topological valley.

Calibration helps, but at the model layer. Found-in-the-Middle, Wu's graph-theoretic interventions, and subsequent calibration work all show consistent recoveries (5 to 15 percentage points) by editing attention scores after softmax. None of this is exposed in production APIs today. The vendors who ship calibrated long-context behavior natively will have a real structural advantage; the rest will compete on prompt structure and effective context length.

The benchmark gap is closing, slowly. The Qwen3 family in 2025 and the latest frontier releases in 2026 have pushed effective context past 128K on RULER and held NoLiMa scores higher than the 2024 cohort. But every cleanly-controlled length-isolation study, including FLenQA, Context Rot, and the residual-aware theory work, finds the same general shape. The U is shallower in 2026 than in 2023. It is still a U.

The takeaway is not that long context is broken. It is that long context is a budget, and budgets need to be planned. The advertised window is a ceiling, not a target. The middle of that window is the most expensive real estate in your prompt. The edges, structure, and retrieval discipline are still the levers that actually move accuracy.

Position is a feature of your prompt. Treat it like one.