How does AI convert documents into embeddings?

From messy PDF to clean numbers — the ingestion pipeline.

Picture it first

It's like making a smoothie for a robot. First you peel the fruit (extract the text), throw away the stickers and stems (headers, page numbers), cut it into small pieces (chunks), label each cup with what's inside (metadata), and then the blender (embedding model) turns each piece into a special number-juice the robot can taste and compare.

Now the same idea, in simple QA words

Before documents can be searched, they must be prepared. The system takes the text out of the files, cleans away junk (page numbers, headers), cuts it into pieces, adds labels (which file, which page, which date), and finally turns each piece into numbers (embeddings) and stores them. For QA: if text is read wrongly here — for example a scanned '₹5,00,000' becomes 'R5,0O,OOO' — the AI will later give wrong answers, and no prompt can fix it.

How the story connects to the real thing

In the storyIn the AI system
Peeling the fruitTaking text out of PDFs / scans
Throwing away stickers and stemsRemoving headers, footers, page numbers
Cutting fruit into piecesSplitting text into chunks
Labelling each cupAdding file name, page, date
The blender making number-juiceEmbedding model turning text into numbers

Real-world example

An insurance company indexes 20,000 policy PDFs. Many are scanned images, so OCR is needed. One OCR setting reads '₹5,00,000' as 'R5,0O,OOO'. The embedding happily stores the wrong text, and later the bot quotes the wrong sum insured. The bug was in ingestion — no amount of prompt tuning will fix it.

Step by step, in plain words

  1. Read the text out of each file (use OCR for scanned files).
  2. Clean it.
  3. Cut it into pieces.
  4. Label each piece (source, page, date).
  5. Turn each piece into numbers.
  6. Save everything in the database.

How a tester tests it

  • Spot-check extracted text against the source PDF — especially numbers, tables, and non-English text.
  • Assert metadata is present and correct for every chunk (page number, document version, access level).
  • Count: number of chunks in DB vs expected; detect silently skipped files.
  • Test re-ingestion: updated docs replace old vectors; deleted docs disappear from search.

Try it yourself

1) Run a free PDF-to-text tool on a scanned document and compare the output line by line with the original. 2) Note every number that was misread.

Interview angle

“A RAG bot gives wrong amounts, but retrieval looks correct. Where do you look?”

At the ingestion pipeline. If the right chunk is retrieved but its text is wrong, the defect happened at extraction or OCR. I compare the stored chunk text against the source document, especially numbers and tables, and add ingestion-level assertions to the test suite.