How does the context window work, and what happens when it becomes full?

The model's desk has a fixed size. Something must fall off.

Picture it first

Picture a desk where you can spread out only 10 pages. Your teacher keeps handing you new pages. When the desk is full, you must push the oldest pages off the edge to make room. If the important page about the homework was one of them — oops, you can't use it anymore. And even when everything fits, you tend to look at the top and bottom pages more than the ones in the middle.

Now the same idea, in simple QA words

The AI can only read a limited amount of text at one time. This limit is called the context window. Everything counts towards it: instructions, chat history, documents and even the answer it will write. When it's full, something must be removed — usually old messages or part of a big document. Also, even when everything fits, the AI pays most attention to the start and end, and may miss things in the middle.

How the story connects to the real thing

In the storyIn the AI system
Your desk with space for 10 pagesContext window (token limit)
New pages from the teacherNew messages and documents
Pages pushed off the edgeText dropped when the limit is reached
Looking more at top and bottom pagesAI misses details in the middle

Real-world example

A legal team uploads a 400-page contract and asks 'What is the termination notice period?'. The app silently truncates to the first 300 pages, and clause 41 on page 350 is never seen. The model confidently answers from a different clause. Nobody gets an error message — the test must catch it.

Step by step, in plain words

  1. The app collects instructions + history + documents + your question.
  2. It counts the tokens.
  3. If it fits — it is sent to the AI.
  4. If it doesn't fit — old parts are cut or summarised (or you get an error).
  5. The AI answers using only what made it onto the 'desk'.

How a tester tests it

  • Needle-in-a-haystack test: hide one unique fact at the start, middle and end of a long document and ask for it.
  • Push the input to 90%, 100%, 110% of the limit — check graceful errors, not crashes or silent truncation.
  • Check that the max-output reservation is respected (a full input leaves no room for the answer).
  • Verify what the user is told when content is cut — silent truncation is a defect.

Try it yourself

1) Paste a long article into a chatbot and ask about a specific detail from the middle. 2) Hide a sentence like 'The secret code is MANGO-42' in the middle of a long text and ask for it.

Interview angle

“If a model has a 1-million-token window, do we still need RAG?”

Usually yes. Big windows cost more per request, increase latency, and still suffer from lost-in-the-middle. RAG sends only the relevant pieces, which is cheaper, faster, easier to cite, and keeps permissions per document. Long context is great for one-off deep analysis; RAG is better for repeated questions over large, changing knowledge bases.