How to Build a RAG Knowledge Base for Your Business

To build a RAG knowledge base, you connect your documents, split them into chunks, index those chunks for search, retrieve the best few at question time, and have an LLM write an answer from only that retrieved text, with citations back to the source. That's the whole loop. Six steps: (1) connect and clean your sources, (2) chunk them, (3) index them with embeddings and keyword search, (4) retrieve and rerank, (5) generate a cited answer, (6) measure it. Most of the quality is decided in steps 1, 2 and 4, not in the choice of LLM.

Two founder-level rules before the detail. First, check whether you need RAG at all. Anthropic notes that if your knowledge base is smaller than 200,000 tokens (about 500 pages), you can put the whole thing in the prompt and skip retrieval entirely. Second, permissions are part of the build, not a later phase. A knowledge base that shows an intern the board minutes is worse than no knowledge base.

The rest of this post walks through each step, what typically goes wrong, and what to ask a vendor or agency before you sign anything.

What is a RAG knowledge base?

A RAG knowledge base is a search-and-answer system over your own documents. Retrieval-augmented generation (RAG) was introduced in a 2020 paper by Patrick Lewis and colleagues, who described models that "combine pre-trained parametric and non-parametric memory for language generation." In plain terms: the LLM supplies language skill, and your documents supply the facts.

The user experience is a chat box. An employee types "what's our refund policy for annual plans?" and gets a direct answer plus links to the two documents it came from. The difference from ordinary search is that the answer is written for the question, not a list of files to open.

If you're still deciding between this and training a model on your data, read RAG vs fine-tuning first. For knowledge that lives in documents, RAG is almost always the right starting point.

Do you actually need RAG, or can you paste the documents in?

You may not need it. Anthropic's guidance on retrieval says that for a knowledge base under 200,000 tokens (roughly 500 pages), you can include the entire thing in the prompt, with no separate retrieval system at all.

That makes the decision simple:

• Under about 500 pages, rarely changing: put it in the prompt, ideally with prompt caching. Cheaper to build, nothing to maintain.
• Over that, or changing weekly, or spread across Confluence, Drive and Notion: you need retrieval.
• Different people allowed to see different documents: you need retrieval, because you must filter before the model sees anything.

Be honest about which bucket you're in.

Step 1: How do you prepare your documents?

Connect the real sources and clean them before anything else. Garbage in, confident garbage out.

The work here is unglamorous: pulling documents from each system on a schedule, stripping navigation and boilerplate, handling PDFs and tables, and re-indexing when a page changes. A knowledge base built on a one-time export goes stale within weeks, and stale answers delivered confidently are the main reason internal AI tools lose trust.

Decide three things up front:

1. Which sources are authoritative. Three wikis with conflicting vacation policies will produce a conflicting answer. Pick one source of truth or tag documents with an owner and date.
2. What metadata you'll keep. Source system, URL, last-updated date, author, and the access group. You'll need every one of these later for citations and permissions.
3. What stays out. Drafts, archived pages, and anything legal says shouldn't be searchable.

Step 2: How should you chunk documents?

Split documents into pieces small enough to be specific and large enough to make sense on their own. Microsoft's RAG architecture guidance puts it this way: if you pass entire documents or oversized chunks, "it's expensive, might overwhelm the token limits of the model, and doesn't produce the best results." In the other direction, "chunks that are too small and don't contain sufficient context to address the query can result in poor outcomes."

There is no universally correct chunk size. Microsoft's guide gives no default number and tells you to test against your own documents. Anthropic describes chunks as "usually no more than a few hundred tokens" in its own examples. Treat that as a starting range, not a rule.

Practical guidance that holds up:

• Split on structure first. Headings, sections, and paragraphs beat arbitrary character counts. A policy clause shouldn't be cut mid-sentence.
• Use overlap so an idea that straddles a boundary isn't lost.
• Attach context to each chunk. A chunk reading "revenue grew 3% over the previous quarter" is useless without knowing which company and period. Anthropic's Contextual Retrieval technique prepends a short explanation of where the chunk sits in its document before indexing it.

Chunking is the step most often skipped over in vendor demos and most often responsible for "it gave a wrong answer even though the document is in there."

Step 3: How do you index and search them?

Index each chunk two ways: as an embedding for meaning, and as plain text for exact keywords. Then search both at once.

Embeddings find conceptually similar text even when no words match. Keyword search (BM25) finds exact matches. You want both, and Microsoft's documentation explains why: "Some scenarios, such as querying over product codes, highly specialized jargon, dates, and people's names, perform better with keyword search because it can identify exact matches." Ask an embeddings-only system for "invoice INV-20418" and it may return invoices that merely look similar.

This combination is called hybrid search. Microsoft describes it as "a single query request configured for both full-text and vector queries," with the two result lists merged by Reciprocal Rank Fusion into one ranking.

On vector database choice: for a first version, don't over-think it. Our own support agent retrieved from roughly 5,000 documents using embeddings in Redis with sub-100ms retrieval, because a dedicated vector database would have been overkill at that size. Bigger corpora, like the 200-person-company knowledge base below, justify a managed vector store.

Step 4: How do you make retrieval accurate?

Retrieve more than you need, then rerank, and keep the top handful. Anthropic's published experiments show how much this matters. Measuring how often the right chunk was missing from the top 20 results (a 5.7% failure rate at baseline), they reported:

Technique	Reduction in failed retrievals
Contextual embeddings	35%
Contextual embeddings + contextual BM25	49%
Above + reranking	67%

Source: Anthropic, Introducing Contextual Retrieval. These are Anthropic's results on their own test datasets. Your documents will behave differently, so treat them as evidence the techniques help, not as a promise of 67%.

The reranking setup Anthropic describes pulled the top 150 chunks, then had a reranker pick the best 20 to hand to the model. The pattern, a wide net followed by a precise filter, is what to ask for.

For founders, the takeaway is: a better LLM does not fix bad retrieval. If the right paragraph never reaches the model, no model can answer correctly. When a pilot disappoints, check retrieval before blaming the model.

Step 5: How do you generate answers people can trust?

Instruct the model to answer only from the retrieved text, cite its sources, and say "I don't know" when the text doesn't contain the answer. Three behaviors make or break trust:

1. Citations on every answer, linking to the original document so people can verify. This is also what makes the tool defensible when it's wrong. The 2020 RAG paper itself noted that "providing provenance for their decisions" was an open problem for language models. Citations are your answer to it.
2. Refusal over invention. A wrong, confident answer costs far more than "I couldn't find that."
3. Freshness. Show the document's last-updated date next to the citation.

If you later want the system to do things rather than just answer, such as filing a ticket or updating a record, you've moved from knowledge base to agent. We cover that line in chatbot vs AI agent and what AI agents actually do for business.

Step 6: How do you handle permissions and security?

Filter results by the asker's access rights at query time, before anything reaches the model. This is the most expensive thing to retrofit and the easiest to get wrong.

The standard pattern is called security trimming. Microsoft's Azure AI Search documentation describes it as a filter that "trims search results based on a string containing a group or user identity": each chunk is stored with the groups allowed to see it, and every query includes a filter for the groups the current user belongs to. The same documentation adds an important warning: the principal "is just a string, used in a filter expression," so the application itself must supply the user's identity from a trusted source. And "document-level authorization is enforced by applying the security filter to every query."

In our RAG-powered knowledge base build, permissions mirror the source systems' own access controls, so employees never see answers from documents they couldn't open in the original tool. That requirement shaped the whole design.

Questions to put to any vendor:

• Where do access rules get enforced, in the search query or only in the interface?
• What happens when someone's access is revoked in Confluence? How fast does the index reflect it?
• Where is our data stored, and is it used to train anything?

How do you know the knowledge base is working?

Build a test set of real questions with known good answers before launch, and measure retrieval and generation separately. Microsoft's evaluation guidance says you must "collect your test documents and queries" and lists the language-model metrics to track: groundedness (is the answer based fully on the retrieved context), completeness, utilization, relevance, and correctness. It also warns that model responses are nondeterministic, so use a target range rather than a single target number.

A workable minimum:

• 50 to 100 real employee questions, collected from Slack and support history.
• For each, the document that should answer it.
• A check of whether that document appears in the retrieved results (retrieval), and whether the final answer is faithful to it (generation).
• A weekly review of thumbs-down answers after launch.

Our case study for the knowledge base reports an 85% reduction in search time, 78% employee adoption in the first month, and 45% fewer questions asked in Slack. Those are the client results as recorded in our own case study, not independent measurements, and they depend on that company's document mess. Yours will differ. Decide your own success metric before you start.

What does it cost and how long does it take?

We're not going to quote a build price here, because it depends mostly on how many sources you have, how messy they are, and how strict the permissions need to be. What we can say precisely is the shape of the cost:

• Build cost is driven by connectors (each source system is its own project), permission mapping, and evaluation work. Not by the LLM.
• Running cost is per-question model usage plus search hosting. Published provider pricing makes this calculable; we walk through the arithmetic in what it actually costs to run an AI agent.
• Maintenance is the hidden line: re-indexing, source changes, and reviewing bad answers.

Anthropic estimates a one-time cost of $1.02 per million document tokens to generate contextualized chunks, under its stated assumptions (800-token chunks, 8k-token documents). Verify current pricing before budgeting, since it changes.

Frequently asked questions

How long does it take to build a RAG knowledge base? A working prototype on one or two clean sources can come together quickly. A production system with several sources, permission mapping, and an evaluation set takes materially longer, and the connectors are usually the long pole. We scope this per project; talk to us with your source list and we'll give you a straight estimate.

What's the difference between a RAG knowledge base and a chatbot? A RAG knowledge base is a chatbot whose answers are grounded in your documents and cited. A generic chatbot answers from its training data. See chatbot vs AI agent for where each sits.

Do I need a vector database? Not always. For a few thousand documents, embeddings stored in a regular database or in Redis can be enough. A managed vector database makes sense as the corpus grows. Whatever you choose, pair it with keyword search.

Will it hallucinate? It can, but RAG reduces it, and you can reduce it further by requiring citations, refusing when the sources are silent, and measuring groundedness. It doesn't eliminate it, so keep a feedback button on every answer.

Can it respect who is allowed to see what? Yes, via query-time permission filtering, but only if it's designed in from the start. Ask any vendor exactly where the filter runs.

Should I use my own LLM or a provider's API? For most startups, a provider API. Retrieval quality drives results far more than model choice, so spend your effort on steps 1 to 4 before optimizing the model.

Ready to build yours?

We build retrieval systems with citations, permission-aware search, and evaluation sets, the unglamorous parts that decide whether people actually use the thing. See our AI integration work, the RAG knowledge base case study, and our intelligent document processor. If you've got a pile of documents and a team that can't find anything in them, tell us what you're working with. We'll tell you honestly, including when the answer is "just paste it in the prompt."

---

Sources

• Anthropic, Introducing Contextual Retrieval
• Microsoft Learn, RAG chunking phase
• Microsoft Learn, Hybrid search overview
• Microsoft Learn, Security trimming in Azure AI Search
• Microsoft Learn, RAG LLM evaluation phase
• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020)