How does RAG work internally, step by step?

Look it up first, then answer — an open-book exam for AI.

Picture it first

Before a quiz, you cut your textbook into small flashcards and sort them into a magic box that groups cards by topic. During the quiz, you read the question, the box instantly hands you the 3 most matching cards, and you write the answer using only those cards. The cutting and sorting happens before; the looking-up happens during.

Now the same idea, in simple QA words

RAG means: first find the right information, then answer from it. It works in two parts. Before anyone asks anything, your documents are cut into small pieces and stored in a smart search box. When someone asks a question, the system finds the 3–5 most relevant pieces and gives them to the AI with the instruction 'answer only from these'. The AI writes the answer and can show where it came from.

How the story connects to the real thing

In the storyIn the AI system
Cutting the textbook into flashcardsSplitting documents into chunks
The magic box that sorts cards by topicThe vector database
The box handing you the 3 best cardsRetrieving the most relevant chunks
Writing your answer using only those cardsThe AI answering from the retrieved text

Real-world example

An HR bot at a large IT company: all HR policy PDFs are indexed every night. An employee asks 'How many days of paternity leave do I get?'. The system embeds the question, finds the 'Parental Leave Policy v4 – section 2' chunk, sends it to the LLM with the question, and answers: '15 working days (Parental Leave Policy v4, §2)'.

Step by step, in plain words

  1. Collect the documents.
  2. Cut them into small pieces.
  3. Store the pieces in a smart search database.
  4. User asks a question.
  5. Find the most relevant pieces.
  6. Give the pieces + question to the AI.
  7. AI answers and shows the source.

How a tester tests it

  • Test each stage separately: did loading lose tables? Did chunking cut a rule in half? Did retrieval return the right chunk?
  • Log the retrieved chunks for every test case — they are your evidence.
  • Test freshness: update a document, re-index, and ask again.
  • Test 'not in the documents' questions: the bot must say it doesn't know.

Try it yourself

1) Draw the 7 steps on paper for a bot over your team's test-strategy document. 2) For each step, write one thing that could go wrong.

Interview angle

“Walk me through what happens when a user asks a RAG chatbot a question.”

The question is embedded with the same model used at indexing time, the vector DB returns the top-K closest chunks, often combined with keyword search and reranked, then the app builds a prompt with system instructions, the chunks and the question, and the LLM generates a grounded answer with citations. As a tester I instrument every step so I can tell retrieval failures from generation failures.