An open-book exam
A language model on its own answers like a student in a closed-book exam: from what it remembers, which may be out of date, and with a guess where memory runs out. RAG makes it an open-book exam. For every question:
- Step 1
Understand the question
Work out what's really being asked, including follow-ups like “and for dining sets?”.
- Step 2
Look it up
Search your documents for the passages most likely to hold the answer.
- Step 3
Pick the best
Keep the few passages that fit best and drop the rest.
- Step 4
Answer from them
Write the reply using those passages, and name the sources.
The idea comes from a 2020 research paper that paired a language model with a searchable index of text, so its answers could draw on facts it was never trained on. Lewis et al.: retrieval-augmented generation
Why not just train the AI on my data?
You could fine-tune a model on your documents, but for a business's facts RAG is usually the better fit. AWS sums up the advantages: it's cheaper than retraining, answers can be kept current, they can cite sources, and you control which sources are used and who may see them. AWS: what is RAG?
| RAG | Fine-tuning | |
|---|---|---|
| A price changes | Update the page; the next answer uses it | Retrain the model |
| Where an answer came from | It can name the page or file | It can't say |
| Fixing a wrong answer | Fix the passage it used | Add examples and retrain |
| Different readers | Search only what each person may see | Everyone gets the same model |
| Cost to start | Low | Higher |
The parts, in plain words
- Chunking: long pages are cut into short passages, so an answer can use just the part that matters.
- Embeddings: each passage is turned into a list of numbers that captures its meaning, so “refund” and “money back” end up close together.
- Vector search: the question is turned into numbers the same way, and the passages with the closest numbers are found. A vector database is where those numbers are kept.
- Ranking: a second look at the passages found, to put the ones that really answer the question first.
- The prompt: the chosen passages are handed to the language model with instructions such as “answer only from these”.
- Citations: the reply keeps track of which passages it used, so it can show its sources.
Where RAG goes wrong
| What happens | Why | The fix |
|---|---|---|
| It invents an answer | Nothing in your documents covers the question | Add the answer, and let it say it doesn't know |
| It quotes an old price | An out-of-date page was found | Remove or update the old page |
| It misses part of the answer | The facts are on two pages and only one was found | Put related facts together, or let it combine sources |
| It applies the rule, not the exception | The exception is far from the rule | Write the exception next to the rule |
| It answers the wrong question | A short follow-up was read on its own | Use a chatbot that rewrites follow-ups with the conversation |
Our guide to stopping a chatbot from making things up covers the fixes in more depth.
How HeyKiko does it
- It plans each question first, rewriting a follow-up with the conversation and noticing when the customer is shopping or wants to book.
- It searches several ways at once, ranks what it finds, and hands the model about a dozen short passages.
- It searches only the knowledge the person's role may see.
- Live facts, such as stock or an order's status, come from your database at the moment they're asked, not from the index.
- If the search is ever unavailable, the agent says it can't check facts right now rather than answering from memory; bookings and cases still work.
- Your content is sent to the AI provider only to index it and to answer; it isn't used to train models.
In the Playground you can open “Why this answer” to see how confident it was and the passages it used. Training on your website and files shows what it can read, and you can try it free for 14 days.
Sources
Checked on 5 November 2026. We make HeyKiko, which this guide uses as its example.
