Building Production-Grade RAG: Chunking Strategies, Reranking, and Evaluation

RAG systems are significantly harder to build correctly than they appear in tutorials. The basic architecture — chunk documents, embed them, retrieve by similarity, pass to LLM — works well in demonstrations with clean data and forgiving evaluation criteria. Production systems face a different reality: large, heterogeneous document corpora, strict latency requirements, and users who will immediately notice when the system retrieves the wrong passage or hallucinates a citation.

Chunking Strategy Is Where Most Systems Go Wrong

The default approach — fixed-size chunking at 512 or 1024 tokens — is a reasonable starting point but breaks down for most enterprise document types. Legal contracts have clause-level semantics that span multiple paragraphs. Technical documentation has hierarchical structure that matters for retrieval. Financial reports have tables that parse incorrectly when split mid-row. We use document-type-aware chunking strategies: semantic chunking for prose, structure-aware extraction for tables and forms, and parent-child chunking for hierarchical documents.

Hybrid Search Consistently Outperforms Pure Vector Search

Pure semantic vector search is excellent at conceptual similarity but poor at exact keyword matches, product codes, names, and identifiers. Pure keyword (BM25) search misses paraphrased or semantically similar content. In production, we always run both and combine scores using Reciprocal Rank Fusion. For most enterprise corpora, hybrid search improves precision@10 by 15-30% over pure vector search — a difference users notice immediately.

Reranking Changes Everything

First-stage retrieval (vector + keyword) optimises for recall — getting the right documents into the candidate set. Reranking optimises for precision — putting the most relevant documents at the top. A cross-encoder reranker reads the query and each candidate document together, producing a relevance score that is dramatically more accurate than embedding similarity. Adding a reranking step typically increases answer quality significantly without changing the underlying retrieval infrastructure.

You Cannot Improve What You Don't Measure

The most important engineering decision in a production RAG system is how you evaluate it. You need three things: a representative ground-truth Q&A dataset from real users, metrics that measure retrieval quality independently of generation quality (NDCG, precision@k), and generation quality metrics (RAGAS, faithfulness, answer relevance). Without evaluation infrastructure, every change to chunking, retrieval, or prompt is a guess. We treat evaluation as a prerequisite, not a phase that comes after deployment.