AI · Oct 2026 · 5 min read
Five questions to answer before you build a RAG chatbot
Most chatbot projects fail on the questions nobody asked in week one. These are the ones I start with.
1. What is the source of truth?
A retrieval-based bot is only as good as the documents behind it. Decide which sources are authoritative, and which ones must never be quoted.
2. Who keeps it fresh?
Content changes. Someone has to own re-indexing, and the bot should show where an answer came from so people can check it.
3. How will you measure a good answer?
Write down twenty real questions and the answers you would accept. Run them after every change. Without this, every tweak is a guess.
4. What should it refuse to do?
A good assistant knows its limits. Define out-of-scope topics, and give the bot a clear way to hand over to a human.
5. What does one conversation cost?
Track tokens per answer from the first prototype. Cost is easier to design in than to bolt on later.
If you cannot say what a good answer looks like, the model cannot either.