Building a Production RAG: Architecture, Chunking, and the Subtle Science of Context

Every generation of engineers rediscovers a simple truth: data pipelines age like fruit. They look fine at first, then quietly decay until something starts to smell wrong. Retrieval-Augmented Generation (RAG) systems are no exception. A prototype might look elegant on a whiteboard — a user query, a vector search, a friendly LLM — but production reality is messier. Documents are inconsistent. Metadata vanishes. Context windows overflow. Latency creeps up like debt interest.

If you want a RAG system that survives in the wild, you need to think like an architect, not a prompt engineer. Let's break down what that architecture looks like, and how to optimize its most misunderstood layer: the document-processing pipeline.

The Architecture in Motion

Imagine your RAG system as a conversation between two specialists. One is the librarian, fluent in embeddings and vector distances; the other is the storyteller, the LLM that weaves those retrieved facts into meaning.

For that duet to work, someone has to manage the choreography — the ingestion, indexing, and retrieval loop that ensures the storyteller always hears from the right sources.

A production-ready RAG usually includes these stages:

1. Ingestion & normalization – bringing raw text from PDFs, HTML, APIs, or databases into a consistent structure.
2. Chunking & embedding – segmenting text into semantically meaningful units and converting them into vector representations.
3. Vector database & retrieval – storing and searching embeddings efficiently, typically using approximate nearest-neighbor methods.
4. Reranking & prompt assembly – refining search results and packaging them for the LLM within token limits.
5. Monitoring, caching, and updates – keeping latency, cost, and data freshness in check.

A RAG architecture is less a pipeline and more an ecosystem. Each stage leaks assumptions into the next: chunking affects retrieval, retrieval affects prompting, prompting affects model cost. Optimization happens at the boundaries.

The Document Processing Pipeline

The document-processing pipeline is where your "knowledge" becomes machine-readable. It's also where most production failures begin.

Preprocessing Requirements

Metadata extraction

Metadata is context. Capture timestamps, authors, section titles, and source identifiers. They let you filter, debug, and rank results later. Systems that skip metadata inevitably end up with semantic collisions — the same fact twice, from two different times, with no way to tell which one wins.

Format normalization

Every document format is a dialect of chaos. PDFs come with phantom line breaks, HTML hides text behind CSS, and Word exports produce text like a ransom note. Normalize early: extract text, flatten structure, but preserve hierarchy. You'll need those heading boundaries later for hierarchical chunking.

Language detection & splitting

Multilingual corpora require language awareness. Mixing embeddings from different languages confuses similarity space — it's like sorting by sound instead of meaning. Detect language, split by section, and tag accordingly.

Deduplication and structure preservation

Duplicates poison retrieval accuracy and inflate costs. Keep only canonical versions. Preserve structure where possible — page numbers, sections, tables. They're not noise; they're semantic hints.

Chunking: Cutting Text at Its Natural Joints

Chunking determines how your documents breathe. It's how you decide what the model can remember in one glance. Too coarse, and retrieval returns blobs of unrelated context. Too fine, and you lose coherence.

Modern RAG systems treat chunking as both art and engineering. Here are the dominant strategies, and when to use each.

Semantic Chunking

Split where meaning shifts, not where a token counter says so. Compute embeddings over small windows, measure cosine similarity between adjacent sections, and cut when it drops below a threshold. It's computationally heavier but yields contextually pure chunks — ideal for technical manuals and research papers.

Sliding Window (with Overlap)

The pragmatic workhorse. Slice text into fixed-length segments with 20–30% overlap. The overlap ensures continuity across chunk boundaries — critical when ideas span multiple paragraphs. You pay a small storage and embedding penalty, but avoid losing context at the seams.

Hierarchical Chunking

Documents have structure, and that structure carries meaning. Hierarchical chunking captures it by creating parent–child relationships: section summaries at the top, detailed paragraphs below. Retrieval can then start broad and zoom in, mimicking how humans search long documents.

Token-Aware Chunking

All of this only works if your chunks fit the LLM's memory. For a model with an 8K context window, you might allocate 1K per chunk, allowing space for instructions, the query, and multiple retrieved contexts. Exceeding the window leads to silent truncation — the most expensive way to lose accuracy.

Optimizing for Reality

The elegance of your pipeline means nothing if it can't meet SLAs. Production RAGs demand balance: relevance, latency, and cost, all pulling in opposite directions.

Chunk size vs retrieval quality — Larger chunks dilute topical focus; smaller ones fragment meaning. The sweet spot depends on your corpus: around 500–800 tokens for FAQs, 1–1.5K for dense technical documents.

Retrieval speed — Use approximate nearest-neighbor indexes (HNSW, IVFPQ). Retrieval latency above 200ms feels sluggish in a live chat application. Parallelize embedding generation and cache frequent queries.

Cost control — Re-embed only when content changes. Cache both embeddings and generation results. Use smaller models for reranking where possible.

Observability — Track retrieval precision and recall, not just uptime. Log which chunks contribute to correct answers. In RAG systems, "it runs" isn't the same as "it retrieves well."

The Pragmatic Stack

For solo developers or small teams, the production setup can stay lean: TypeScript + Node.js for the runtime, PostgreSQL + pgvector for local experiments, Qdrant or Pinecone for scale, OpenAI text-embedding-3-large or Cohere embed-v3-multilingual for embeddings, and Vercel + GitHub Actions for automated ingestion.

This architecture keeps costs predictable, deployments simple, and latency low enough for real-time apps.

Closing Thoughts: Precision Over Poetry

Production RAG is not a prompt-engineering problem; it's a systems-design problem disguised as one. The difference between a demo and a dependable system lives in the preprocessing scripts, the chunking logic, and the quiet discipline of metadata hygiene.

Think of each chunk as a paragraph in your company's collective memory. If you slice, tag, and store them well, your LLM can answer truthfully without pretending to know more than it does.

Good RAGs don't hallucinate less because the models are smarter — they hallucinate less because the data is well-behaved.