HomeArticles

Chapter 1 of RAG and retrieval

What Is a RAG Pipeline? A Practitioner's Guide

What is a RAG pipeline? Why RAG suits private data better than fine-tuning, the ingestion, embedding, retrieval and generation stages, and how it fails.

The four stages of a RAG pipeline: ingestion, embedding, hybrid retrieval and reranked generation.

What is a rag pipeline? It's the architecture that connects a retrieval system to a language model so it can answer questions using your data — not just its training data. You send a query, retrieve relevant documents from your vector database, stuff them into a prompt, and let the LLM generate an answer grounded in those documents.

Simple concept. Brutal in practice.

Most people think RAG is just "search + ChatGPT wrapper." They're wrong because they ignore what happens between the retrieval and generation stages. That middle ground — the pipeline orchestration — is where systems live or die.


Why RAG Exists (And Why Fine-Tuning Won't Save You)

Here's the cold truth: fine-tuning teaches a model facts it already mostly knows. It doesn't teach it new information reliably.

RAG solves a different problem. It says: don't bake knowledge into weights. Keep it in a database. Retrieve it at inference time.

This matters because:

  • Your data changes daily. Retraining weekly costs thousands.
  • You need citations. RAG can show exactly which document generated each answer.

But RAG introduces its own pain.


Core Components: The Four Stages

What is a rag pipeline? Four stages. Each stage can break your system.

Stage 1: Ingestion — Where Garbage Collects

You have PDFs, Confluence pages, Slack exports, maybe some Notion docs. You need to chunk them into pieces the model can digest.

Most teams chunk by character count. 512 tokens. 1024 tokens. Dead simple.

Dead wrong.

Fixed-size chunks split text wherever the count runs out. In a legal document, a single clause can span two chunks, and the model never sees the complete definition.

What works better: semantic chunking. Split on natural boundaries — headers, paragraphs, sentence completions. LangChain's RecursiveCharacterTextSplitter with overlap helps, but it's not enough.

A hybrid approach works better:

  • Split on markdown headers first
  • Then paragraph breaks
  • Then sentence boundaries
  • Never split mid-sentence

Here's an example ingestion pipeline:

def semantic_chunker(documents, max_tokens=512, overlap_tokens=64):
    chunks = []
    for doc in documents:
        # Split by headers first
        header_sections = re.split(r'(#+ .+)', doc.text)
        current_chunk = ""
        for section in header_sections:
            if len(tokenizer.encode(section)) > max_tokens:
                # Split further by paragraphs
                paragraphs = section.split('

')
                for para in paragraphs:
                    if len(tokenizer.encode(para)) > max_tokens:
                        # Split by sentences as last resort
                        sentences = sent_tokenize(para)
                        for sent in sentences:
                            if len(tokenizer.encode(current_chunk + sent)) > max_tokens:
                                chunks.append(current_chunk)
                                current_chunk = sent
                            else:
                                current_chunk += " " + sent
                    else:
                        current_chunk += "

" + para
            else:
                current_chunk += "

" + section
        if current_chunk:
            chunks.append(current_chunk)
    return chunks

This isn't perfect. Nothing is. But it keeps related text together in one chunk.

Stage 2: Embedding — The Vibe Check

You take each chunk, run it through an embedding model, get a vector. Store it in Pinecone or Weaviate or Qdrant.

The embedding model choice matters more than anything else.

In early 2023, everyone used OpenAI's text-embedding-ada-002. It's fine. It's generic. For specialized domains — legal, medical, code — it fails.

You want specialized embeddings for your data type. Code? Use code-search-ada-code-002 or starcoder-embed. Medical? Use PubMedBERT. Financial filings? Use fine-tuned gte-large.

Here's the embedding pipeline pattern:

from sentence_transformers import SentenceTransformer
import numpy as np

class EmbeddingService:
    def __init__(self, model_name="BAAI/bge-large-en-v1.5"):
        self.model = SentenceTransformer(model_name)
        # bge models need this prefix for query-doc separation
        self.query_prefix = "Represent this query for searching: "
        self.doc_prefix = "Represent this document: "

    def embed_documents(self, chunks):
        texts = [self.doc_prefix + c for c in chunks]
        return self.model.encode(texts, normalize_embeddings=True)

    def embed_query(self, query):
        return self.model.encode([self.query_prefix + query],
                                 normalize_embeddings=True)[0]

That prefix matters: bge models were trained with an instruction prefix for queries, so use it the way the model card describes.

Stage 3: Retrieval — It's Not Just Cosine Similarity

Here's where most people stop. Query comes in. Embed it. Find nearest neighbors. Done.

That's barely a search engine. It's not a RAG pipeline.

What is a rag pipeline if not retrieval that understands context? You need hybrid search. Keyword + vector. BM25 + cosine similarity. You need filters. You need re-ranking.

A typical failure: a vector-only "RAG" system can't answer "what's the policy on PTO for contractors in Germany?" because the vector search returned semantically similar chunks about "vacation time" — but not the exact policy document.

The missing piece: a keyword component. BM25 catches exact matches that semantic search misses.

Here's the retrieval pattern that works at production scale:

import numpy as np
from rank_bm25 import BM25Okapi
from typing import List, Tuple

class HybridRetriever:
    def __init__(self, documents, alpha=0.3):
        self.documents = documents
        self.alpha = alpha  # weight for vector search
        self.bm25 = BM25Okapi([d.split() for d in documents])
        self.vector_store = None  # Initialize with your vector DB

    def retrieve(self, query: str, k: int = 10) -> List[Tuple[str, float]]:
        # Vector scores
        query_embedding = self.embedding_service.embed_query(query)
        vector_results = self.vector_store.similarity_search(query_embedding, k=k)
        vector_scores = {doc_id: score for doc_id, score in vector_results}

        # Keyword scores
        bm25_scores = self.bm25.get_scores(query.split())
        top_bm25 = np.argsort(bm25_scores)[-k:][::-1]

        # Normalize and combine
        combined = {}
        for i, doc_id in enumerate(vector_results.keys()):
            combined[doc_id] = self.alpha * (1.0 - vector_scores.get(doc_id, 0))
        for i, idx in enumerate(top_bm25):
            doc_id = self.documents[idx]
            combined[doc_id] = (combined.get(doc_id, 0) +
                               (1 - self.alpha) * (bm25_scores[idx] / max(bm25_scores)))

        # Sort by combined score
        sorted_docs = sorted(combined.items(), key=lambda x: x[1], reverse=True)
        return sorted_docs[:k]

Alpha at 0.3 means 30% vector, 70% keyword. Tune this for your domain. Legal documents? Lean keyword (alpha=0.2). Conversational queries? Lean vector (alpha=0.6).

Stage 4: Generation — The Prompt Trap

You retrieved 10 chunks. Now what?

Most people concatenate them into a prompt with "Answer the question based on these documents." This works in demos. In production, it produces rambling nonsense.

The problem: you're giving the model too much irrelevant context. Ten chunks might be 4,000 tokens. Only two chunks are actually helpful. The rest distract.

Solution: chunk re-ranking. Use a cross-encoder model to score each retrieved chunk against the query. Keep only the top 2-3. This is expensive (cross-encoders are slow) but necessary.

Here's the generation pattern:

from sentence_transformers import CrossEncoder
import openai

class RAGGenerator:
    def __init__(self):
        self.reranker = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')
        self.max_context_tokens = 2000  # Keep it tight

    def generate(self, query: str, documents: List[str]) -> str:
        # Rerank documents
        pairs = [(query, doc) for doc in documents]
        scores = self.reranker.predict(pairs)

        # Sort by relevance score, keep top 2
        ranked = sorted(zip(documents, scores), key=lambda x: x[1], reverse=True)
        best_docs = [doc for doc, score in ranked[:2]]

        # Build concise context
        context = "
---
".join(best_docs)

        # Trim to token limit
        context_tokens = len(context.split())
        if context_tokens > self.max_context_tokens:
            context = " ".join(context.split()[:self.max_context_tokens])

        prompt = f"""You are a technical support assistant. Use ONLY the following documents to answer the question. If the documents don't contain the answer, say "I don't have that information."

Documents:
{context}

Question: {query}

Answer (with citations to document numbers 1-2):"""

        response = openai.ChatCompletion.create(
            model="gpt-4",
            messages=[{"role": "user", "content": prompt}],
            temperature=0.1,
            max_tokens=500
        )
        return response.choices[0].message.content

Temperature at 0.1. Not 0.0 (which makes the model repetitive) and not 0.7 (which invites hallucination).


The Pipeline Orchestration Problem

RAG orchestration checklist: idempotent stages, dead letter queues, embedding quality checks, p95 latency, stage logs.

Individual stages are table stakes. The hard part is orchestration.

What is a rag pipeline at its core? A state machine. Each stage depends on the previous one succeeding. When something fails — and it will — you need observability.

A typical silent failure: a PDF library update breaks text extraction for certain files, and the vector store fills with empty embeddings. No alert fires, and users see garbage answers.

Build your pipeline with:

  • Idempotent stages — re-running ingestion produces identical results
  • Dead letter queues — failed chunks go somewhere you can inspect
  • Embedding quality checks — alert if the average embedding norm drops below a threshold
  • Generation latency tracking — if p95 exceeds 5 seconds, something's wrong

Here's a minimal monitoring pattern:

import time
import logging

class PipelineMonitor:
    def __init__(self):
        self.stage_times = {}
        self.failures = []

    def track_stage(self, stage_name: str):
        def decorator(func):
            def wrapper(*args, **kwargs):
                start = time.time()
                try:
                    result = func(*args, **kwargs)
                    elapsed = time.time() - start
                    logging.info(f"Stage {stage_name}: {elapsed:.2f}s")
                    if elapsed > 5.0:
                        logging.warning(f"Slow stage: {stage_name}")
                    return result
                except Exception as e:
                    self.failures.append((stage_name, str(e)))
                    logging.error(f"Failed stage {stage_name}: {e}")
                    raise
            return wrapper
        return decorator

When RAG Fails (And It Will)

Three common failure modes:

The "Lost in the Middle" problem — Models pay attention to the first and last chunks, ignore the middle. The research from Liu et al. (2023) showed this clearly Lost in the Middle paper. Solution: put the most relevant chunk first, least relevant last. Or use the reranking approach above.

Context window overflow — You stuffed in 10 chunks. The LLM runs out of context. It starts ignoring parts. Hallucinations spike. Every model has a context window. Treat it as 70% of the stated size — the model works well within that, but degrades at the edges.

Stale retrieval — You indexed documents last week. Someone updated the price list yesterday. Your RAG system still returns old prices. Solution: timestamp-aware retrieval. Filter chunks by last_modified >= query_date. Or re-index daily.


What Is a RAG Pipeline? The Real Answer

It's a feedback loop. Not a one-way pipe.

You deploy it. Users ask questions. Some answers are wrong. You trace the error back: was the chunk missing? Was the embedding off? Was the prompt too vague? Each failure teaches you something about your data, your model, your users.

Good RAG systems treat the pipeline as malleable. They tweak chunk sizes weekly. They swap embedding models quarterly. They A/B test prompts.

If you're building one today, start simple. Three chunks, no reranker, GPT-4. Get it working. Then iterate. Add hybrid search when you hit recall limits. Add reranking when generation quality plateaus. Add monitoring when something breaks at 2 AM.

That's the honest answer to "what is a rag pipeline?" It's a living system. It will break. And then you'll fix it.


FAQ

RAG FAQ quick answers: minimum stack, pgvector to at least 1M vectors, document updates, testing, images and audio.

Q: What is a rag pipeline in simple terms?
A: You search your documents for relevant info, hand that info to an LLM, and the LLM answers based on that info instead of guessing.

Q: What's the minimum viable RAG pipeline?
A: A vector database (Pinecone free tier works), an embedding model (BGE-small for speed), and a chat completion API (GPT-4-mini costs pennies). You can build this in a weekend. It won't be production-grade, but it'll work.

Q: Do I need a vector database, or can I use Postgres?
A: Postgres with pgvector works fine for under 1 million vectors. Beyond that, you'll hit query latency issues. Pinecone or Qdrant scale better.

Q: How do I handle updates to my documents?
A: Timestamp your chunks. On retrieval, filter by updated_at <= now(). Re-index on a schedule or trigger. Don't delete old embeddings immediately — keep them for audit trails.

Q: What's the biggest mistake teams make?
A: Not testing with real user queries. They test with "What is our company policy on X?" — but users ask "How do I do Y?" and the system fails. Build a test set from actual support tickets.

Q: Can I use RAG with images or audio?
A: Yes, but it's harder. Use multimodal embeddings (CLIP, ImageBind) for images. For audio, transcribe to text first, then index the transcript.

Q: How much does a RAG pipeline cost at scale?
A: The cost splits across embedding, the vector database and LLM inference. Embedding is mostly a one-time cost per document; LLM tokens are the variable cost that grows with query volume.

Want to talk this through?

Sivaro AI →