RAG is not enough: engineering retrieval that survives production
Naive vector search demos well and fails quietly. Here's what separates a weekend RAG prototype from a system your business can rely on.
The demo trap
Anyone can build a RAG demo in an afternoon: embed the documents, search by similarity, stuff the context window. It works on the five questions you tried. Then real users arrive with acronyms, typos, multi-hop questions, and documents that contradict each other — and accuracy falls off a cliff.
What production retrieval actually requires
- Hybrid search: semantic similarity plus keyword matching, because names, codes, and jargon don't embed well.
- Chunking that respects structure: tables, headers, and cross-references carry meaning that naive splitting destroys.
- Reranking: a second-stage model that orders candidates by actual relevance, not cosine distance.
- Citations: every answer traceable to a source page, so users can verify instead of trust.
Measure it or it isn't real
The teams that succeed treat retrieval quality as an engineering metric. Build a golden dataset of real questions, score answers automatically, and gate every change on the eval — the same discipline you'd apply to any other production system.
Building something in this space?
This is the work we do every day. Tell us about your project and an engineer will reply within one business day.
Talk to an engineer