How to Evaluate a RAG Pipeline Before Production

1. Why Naive RAG Fails in Production

A naive RAG pipeline splits documents into arbitrary 500-token chunks, embeds them with an off-the-shelf embedding model, runs a top-5 cosine similarity search, and injects the retrieved chunks into an LLM prompt. In production, this architecture breaks down due to three fundamental flaws:

• Chunk Boundary Loss: Critical context (such as conditional clauses or pricing tables) gets split in half across chunk borders, making it impossible for the model to extract accurate facts.
• Keyword vs. Semantic Mismatch: Dense vector embeddings excel at broad conceptual similarity but struggle with exact alphanumeric IDs, SKU codes, legal section numbers, or acronyms.
• Context Window Distraction: Injecting irrelevant chunks degrades LLM reasoning. As documented in foundational long-context research (Liu et al., "Lost in the Middle", 2023), LLM reasoning degrades significantly when key facts are placed in the middle of lengthy context prompts rather than near the boundaries.

2. The 4 Quantitative Evaluation Pillars

To diagnose and optimize a RAG system, decouple Retrieval Evaluation from Generation Evaluation:

For enterprise teams looking to build robust pipelines, see our AI Agent & LLM Engineering Services.

3. Hybrid Search: Combining Dense Vectors and BM25 Sparse Search

Dense semantic embeddings (e.g. OpenAI text-embedding-3-large, Cohere embed-english-v3.0) capture conceptual context ("how to terminate agreement early" matches "cancellation clause"). However, they frequently fail on specific queries containing alphanumeric codes ("Error code 0x8004", "Clause 14.2(b)").

Production RAG architectures implement Hybrid Search, pairing dense vector search with sparse BM25 full-text indexing, combined using Reciprocal Rank Fusion (RRF):

In PostgreSQL, hybrid search can be executed natively in a single query by joining pgvector cosine distance with PostgreSQL's built-in tsvector text search.

4. Cross-Encoder Reranking: Filtering Noise & Improving Precision

Bi-encoder embedding models compute query and document representations independently in order to enable sub-10ms approximate nearest neighbor search. This computational efficiency sacrifices cross-attention between specific words in the query and candidate documents.

A Cross-Encoder Reranker (e.g. Cohere Rerank v3, BAAI/bge-reranker-large) takes the query and document together as a unified pair, allowing full cross-attention across all tokens. In empirical evaluations and published benchmarks (such as BEIR benchmark evaluations on models like Cohere Rerank and BGE-Reranker), a second-stage cross-encoder consistently demonstrates substantial precision gains over raw bi-encoder vector similarity alone. For example, retrieving an initial candidate set of 20–30 chunks via hybrid dense-sparse search and reranking them down to the top 3–5 items filters out semantic false positives before context reaches the LLM generation stage.

5. Automated Evaluation Frameworks: Ragas & TruLens in CI/CD

Manual spot-checking ("vibe testing") does not scale. When prompt templates, chunk sizes, or embedding models change, automated regression tests are essential.

We recommend integrating Ragas or DeepEval into your GitHub Actions test suite with a golden benchmark dataset of 50–100 curated query-context-answer triplets. A PR that drops Faithfulness below 0.95 or Context Recall below 0.90 is automatically blocked from merge.

Want dedicated AI talent to set up this pipeline? Explore our Hire AI Engineers service.

6. Deterministic Guardrails & Structured Output Validation

Even with high retrieval precision, generative LLMs can occasionally formulate answers with invalid syntax or unverifiable claims. Production architectures wrap generation in a deterministic schema validation layer:

If the response fails schema validation or fails to cite valid chunk UUIDs, the system routes the request to a deterministic fallback or triggers automated re-generation with a higher temperature penalty.