How does chunk size affect RAG accuracy?
Too small loses meaning. Too big drowns it.
Picture it first
If you cut a comic strip into single words, nobody understands the story. If you don't cut it at all and hand over the whole book, it's too much to read for one question. The best is cutting it into whole panels or whole scenes — each piece makes sense on its own.
Now the same idea, in simple QA words
Chunk size means how big each piece of a document is. If pieces are too small, a rule and its exception may be separated — the AI sees only half the rule. If pieces are too big, one piece mixes many topics, search becomes confused, and it costs more. The best size keeps one complete idea per piece. There is no magic number — QA should test a few sizes and pick the one that gives the best answers.
How the story connects to the real thing
Real-world example
A leave policy says: 'Employees get 12 casual leaves per year. However, employees in their probation period get only 6.' With 20-token chunks, the 'However...probation' sentence lands in a separate chunk. The bot retrieves only the first and tells a new joiner they get 12. With section-based chunking, both sentences stay together and the answer is correct.
Step by step, in plain words
- Pick a way to cut documents (by size, by paragraph or by heading).
- Pick a size and a small overlap.
- Store the documents.
- Ask your standard test questions.
- Compare results for different sizes.
- Keep the size that gives the best answers.
How a tester tests it
- Treat chunk size as a test parameter: run the golden set at 3–4 sizes and plot quality vs cost.
- Write 'split-trap' test cases: rules with exceptions, tables, numbered steps spanning paragraphs.
- Check that tables are not split mid-row.
- Re-test after any chunking change — it requires re-indexing everything.
Try it yourself
1) Take a one-page policy and split it into 3-sentence chunks, then into section chunks. 2) For 5 questions, decide which version gives each question a complete answer in ONE chunk.
Interview angle
“How would you choose the right chunk size?”
Empirically, not by guessing. I build a golden set with known source passages, index the corpus with several chunk sizes and strategies, and compare context recall, precision, answer faithfulness and token cost. I deliberately include rule-plus-exception cases, because that's where small chunks fail.