RAG for Company Documents: What Works in Production and What Only Works in Demos

Retrieval-augmented generation is the most demonstrated pattern in enterprise AI, and for good reason. Point a language model at your documents, ask a question, get an answer with citations. The demo takes an afternoon. The production system takes months, and the gap between the two is where most internal AI projects stall.

We have shipped assistants that answer questions over company knowledge for mobile and web users, including an AI knowledge assistant app and a voice-enabled advisory platform. This is what changed between the demo and the version people use every day.

What the demo gets away with

A demo runs on a few hundred clean documents, is asked questions by the person who built it, has no permissions to respect, and is judged on whether the answers look plausible. Production reverses all four conditions. The document set is large, messy and changing. The questions come from people who phrase things in ways nobody anticipated. Some documents must be invisible to some users. And the answers are judged on whether they are correct.

Ingestion decides the ceiling

Everything downstream depends on what goes into the index, and the index is only as good as the ingestion pipeline.

Chunk by structure, not by character count

Splitting documents into fixed-size pieces is the first thing every tutorial does and the first thing that fails. A policy clause split in the middle loses its condition. A table split across chunks loses its headers. We split along document structure (headings, sections, table boundaries) and attach the heading path to every chunk, so a retrieved paragraph still knows it came from "Leave policy, section 4, parental leave". That alone fixes a large share of wrong answers.

Metadata from the first version

Every chunk carries its source, last-modified date, owner, access group and document type. Without these you cannot filter by permission, prefer recent versions or tell a user where an answer came from. Adding metadata after launch means re-indexing everything, so it belongs in the first version.

The documents change

Documents get updated, superseded and deleted. The index has to follow, or the assistant confidently quotes last year's price list. Incremental re-indexing with version tracking is routine engineering and it is mandatory.

Retrieval is a search problem

Vector similarity on its own is not enough. Users search for product codes, acronyms and exact phrases that embeddings handle poorly. Our retrieval layer combines vector search with keyword search, merges the results, and re-ranks the merged set with a model built for that job. Hybrid retrieval with re-ranking is the largest single quality improvement we make on any RAG system, and it takes a few days.

Two more habits pay off. Rewrite the user's question before retrieval, because "what about last year" cannot be answered without the previous turn of the conversation. And retrieve more than you use, then let the re-ranker choose, because recall at the first stage is cheap and precision at the last stage is what the user sees.

Permissions

If the assistant can read the HR folder, every employee can ask it HR questions. Permission-aware retrieval filters chunks by the current user's access before the model sees anything, using the same groups that govern the source systems. This has to be enforced at retrieval time, not by asking the model to be discreet. A model cannot forget a document it has been shown, and prompt instructions are not an access control.

The model is not the quality gate

The language model writes the answer. It does not decide whether the answer can be trusted. Three mechanisms do that work.

• Grounding checks. If the retrieved chunks do not support an answer, the assistant says it could not find one and suggests where to look. Users come to trust a system faster when it admits gaps than when it guesses fluently.
• Citations the user can open. Every answer links to its sources. This is how a user verifies, and how you debug a wrong answer in thirty seconds.
• Scope limits. The assistant answers questions about the document set. It does not give medical, legal or financial advice beyond what the documents say, and it tells the user so.

Evaluation

You cannot improve what you do not measure, and "it seems better" is not a measurement. Before launch we build a test set of real questions with known good answers, drawn from support tickets, search logs and interviews with the people who will use the system. Every change to chunking, retrieval or prompts is scored against that set. After launch the set grows from logged questions the assistant handled badly, which turns complaints into regression tests.

The questions users ask are the most surprising input. They are shorter, vaguer and more dependent on context than anything the team wrote during development. Logging them, with permission, is the fastest route to a system that feels like it understands.

Cost and speed are architecture decisions

A naive RAG system sends every question to the most capable model with the largest possible context. That is slow and expensive. In production we route. A small model handles classification and query rewriting, retrieval narrows the context to what is relevant, and the capable model answers only with the chunks it needs. Caching repeated questions and streaming answers token by token make the experience feel fast even when the model is not. The cost per question should be a number you know, not a surprise on the invoice.

A production checklist

1. Structure-aware chunking with heading paths and full metadata.
2. Incremental re-indexing that tracks versions and deletions.
3. Hybrid retrieval with re-ranking and query rewriting.
4. Permission filtering at retrieval time, mirrored from the source systems.
5. Grounding checks, citations and explicit scope limits.
6. An evaluation set that grows from real usage.
7. Model routing, caching and streaming for cost and speed.
8. Logging that can answer "why did it say that" for any response.

None of these items is exotic. Together they are the difference between an impressive demo and a tool people trust with their work. Our AI software development and AI automation pages describe how we build these systems. If you have a document set and a list of questions it should answer, that is enough to start.