RAG — retrieval-augmented generation

RAG combines information retrieval with LLM generation. It retrieves relevant documents from a knowledge base and includes them in the prompt context, allowing the LLM to generate answers grounded in specific data. RAG reduces hallucination, enables access to private/current data without fine-tuning, and is more cost-effective than retraining models.

A RAG pipeline consists of: (1) document ingestion and chunking, (2) embedding generation for chunks, (3) vector storage in a database, (4) query embedding and similarity search at inference time, (5) context assembly with retrieved chunks, and (6) LLM generation using the augmented prompt. Each component can be optimised independently.

A vector database stores and indexes high-dimensional vector embeddings for efficient similarity search. Unlike traditional databases that use exact matching on structured fields, vector databases use approximate nearest neighbour (ANN) algorithms like HNSW or IVF for fast similarity search. Popular options include Pinecone, Weaviate, Qdrant, ChromaDB, and pgvector.

Hallucination occurs when LLMs generate plausible-sounding but factually incorrect information. It happens because LLMs are trained to predict probable next tokens, not verified facts. Mitigation strategies include RAG for grounding in verified sources, chain-of-thought prompting, constrained decoding, and implementing fact-checking pipelines.

Strategies include chunking the document and processing each chunk separately, using map-reduce patterns (summarise chunks then synthesise), implementing rolling summaries, using RAG to retrieve only relevant sections, and leveraging models with longer context windows. Choose based on whether the task needs global context (use map-reduce) or local information (use RAG).

Instruct the model to only use provided context and say 'I don't know' when uncertain. Use structured extraction from grounding documents rather than open generation. Ask for citations or evidence. Implement confidence scoring in the output format. Use chain-of-thought to make reasoning transparent. Combine with RAG for factual grounding.

Fixed-size chunking splits by token/character count with overlap. Semantic chunking splits at natural boundaries (paragraphs, sections). Recursive chunking tries multiple separators hierarchically. Document-structure-based chunking uses headings and sections. Choose based on document type: structured documents benefit from semantic chunking, while unstructured text works with fixed-size plus overlap.

Evaluate retrieval quality using recall@k, precision@k, and MRR (Mean Reciprocal Rank). Evaluate generation quality using faithfulness (is the answer grounded in retrieved context?), relevancy (does it answer the question?), and correctness. Use frameworks like RAGAS for automated evaluation. Also measure end-to-end latency and user satisfaction.

Hybrid search combines dense vector search (semantic similarity) with sparse keyword search (BM25/TF-IDF). Dense search captures semantic meaning but can miss exact terms. Sparse search is excellent for specific keywords, names, and codes. Combining both with reciprocal rank fusion (RRF) or weighted scoring captures both semantic and lexical matches.

Reranking applies a more expensive cross-encoder model to rescore initially retrieved documents based on their relevance to the query. The initial retrieval uses fast bi-encoder embeddings to get candidate documents, then the reranker (like Cohere Rerank or a cross-encoder) scores each query-document pair jointly. This two-stage approach balances speed and accuracy.

Naive RAG follows a simple retrieve-then-generate pipeline. Advanced RAG adds pre-retrieval optimisation (query rewriting, expansion), retrieval optimisation (hybrid search, reranking, iterative retrieval), and post-retrieval processing (context compression, lost-in-the-middle mitigation). Advanced RAG also includes feedback loops and self-evaluation for continuous improvement.

Query decomposition breaks a complex question into simpler sub-questions, retrieves information for each, and synthesises the results. Use it when questions require information from multiple domains or documents. For example, 'Compare the AI policies of India and the US' decomposes into separate retrievals for each country's policy, then a comparison synthesis step.

Implement incremental indexing that detects changed documents via checksums or timestamps. Use metadata to track document versions and update timestamps. For critical applications, maintain version history and allow querying against specific versions. Implement a document deletion pipeline that removes stale vectors. Schedule periodic full re-indexing for consistency.

LLMs tend to pay more attention to information at the beginning and end of the context, potentially ignoring relevant information in the middle. Mitigate by placing the most relevant chunks first and last, limiting the number of retrieved chunks, using reranking to prioritise relevance, or restructuring retrieved content into a focused summary before generation.

Consider the MTEB leaderboard for benchmark performance. Evaluate on your specific domain data using retrieval metrics. Compare dimensionality (trade-off between quality and storage), latency, and cost. Test multilingual capability if needed. Consider whether a domain-specific fine-tuned embedding outperforms general-purpose ones. Popular choices include OpenAI ada-002, Cohere embed, and open-source models like BGE and E5.

Dense retrieval uses learned vector embeddings from neural models to represent queries and documents as continuous vectors, capturing semantic similarity. Sparse retrieval uses traditional methods like BM25 with term frequency and inverse document frequency. Dense is better for semantic matching; sparse excels at exact term matching. Combining both often yields the best results.

Include source metadata (date, authority, reliability) with retrieved chunks. Instruct the LLM to identify and flag contradictions. Prioritise more recent or authoritative sources. Present multiple viewpoints when appropriate. Implement a confidence scoring mechanism. For critical applications, surface contradictions to users rather than silently choosing one version.

Monitor retrieval latency, embedding computation time, and end-to-end response time. Track retrieval hit rate and relevance scores. Measure answer faithfulness and hallucination rate. Monitor index size growth and query volume. Track user feedback (thumbs up/down, corrections). Alert on unusual query patterns, error rates, or degradation in response quality.

Use table detection models or specialised PDF parsers (like Unstructured, Camelot, or Tabula) to extract tables. Convert tables to markdown or structured text preserving column relationships. Store tables as separate chunks with table-specific metadata. Consider embedding tables alongside their captions and surrounding context. For complex tables, create text descriptions.

There is no universal optimal size; it depends on document type, query patterns, and embedding model. Typical ranges are 256-1024 tokens. Smaller chunks give more precise retrieval but may lack context. Larger chunks provide more context but may include irrelevant information. Experiment with different sizes and evaluate retrieval quality. Use overlap (10-20%) to maintain context.

Parent-child chunking stores documents at two granularity levels. Small child chunks are embedded and used for retrieval (precise matching). When a child is retrieved, its larger parent chunk is returned to the LLM (richer context). This combines the precision of small-chunk retrieval with the context richness of larger passages, improving both retrieval accuracy and generation quality.

Include source metadata (document title, page, URL) with each chunk. Instruct the LLM to reference specific sources using identifiers. Parse citations from the output and validate them against retrieved chunks. Display inline citations with links to source documents. For accuracy, use structured output to separate the answer from citations and validate each reference.

Metadata filtering narrows the search space before or after vector search, improving relevance and performance. Design metadata fields based on common query patterns: date ranges for recency, categories for topic scoping, access levels for permissions, and source types for authority. Keep metadata normalised and indexed. Implement pre-filtering for mandatory constraints and post-filtering for preferences.

Use RAG when you need access to frequently updated data, require citations, or have a large knowledge base. Use fine-tuning when you need to change the model's behaviour, style, or reasoning patterns, or when knowledge is stable and fits in training data. RAG is cheaper and more flexible; fine-tuning gives better performance for specific formats. Many production systems combine both approaches.

Create an MCP server that exposes search and retrieval tools for the knowledge base. Implement tools like search_documents(query), get_document(id), and list_categories(). Use resources to expose document content. Add authentication via the transport layer. Handle pagination for large result sets. Include proper error handling and rate limiting. Test with multiple MCP clients.

Implement short-term memory as the conversation context window. Use long-term memory with vector databases for past interactions and learned information. Implement working memory for the current task state. Summarise older context to stay within token limits. Use structured memory stores (key-value) for facts and episodic memory for past experiences. Implement memory retrieval based on relevance.

For retrieval: precision@k, recall@k, MRR, and NDCG to measure document relevance. For generation: faithfulness (is the answer grounded in context?), answer relevance (does it address the question?), and completeness. End-to-end: accuracy against ground truth, hallucination rate, latency, and cost per query. Use frameworks like RAGAS or custom evaluation scripts.

Evaluate: query performance (latency at scale), scalability (data size and QPS), index types supported (HNSW, IVF), filtering capabilities (metadata filters), managed vs self-hosted, cost model, backup and replication, cloud region availability (Indian regions), API compatibility with your stack, and community/support quality. Popular choices include Pinecone, Weaviate, Qdrant, Milvus, and pgvector.

Choose a model with strong multilingual capabilities (like GPT-4, Claude, or IndicBERT for embedding). Implement language detection for routing. Use RAG with a knowledge base containing Indian language content. Fine-tune on Indian language conversation data if quality is insufficient. Handle transliteration (Hindi in Roman script) and code-mixing (Hinglish) explicitly.

Ingest PDFs of policy documents with structure-aware parsing for Indian government formatting. Handle Hindi and English content with multilingual embeddings. Implement metadata filtering by ministry, date, and policy type. Use hybrid search for exact policy numbers and semantic queries. Include citation generation with specific section references. Handle amendments and superseded policies.

For tables, extract and store them as structured text or markdown. For images, generate text descriptions using vision models and index those descriptions. For charts, extract underlying data points. Use multi-modal embedding models that can embed both text and images into the same space. Store modality metadata for filtering and presentation.

Contextual retrieval enriches each chunk with additional context about its position and meaning within the broader document before embedding. For example, prepending a chunk with a brief summary of the document or section it came from. This reduces the ambiguity of isolated chunks and improves retrieval accuracy, especially for chunks that reference external context.

Store access control metadata (tenant, department, role) with each chunk. Apply metadata filters at query time to enforce permissions. Integrate with the enterprise's existing IAM (like Active Directory). Ensure vector database supports metadata filtering efficiently. Implement audit logging for compliance. Handle Indian data localisation requirements by keeping sensitive data in India-based storage.

Cache frequent queries and their results. Use approximate nearest neighbour indices for faster search. Implement query embedding caching. Batch vector lookups. Use streaming for LLM generation. Pre-compute and store embeddings for static content. Optimise chunk sizes to reduce the number of retrievals. Consider using smaller, faster models for retrieval stages.

Graph RAG combines knowledge graphs with vector-based retrieval. It constructs a graph of entities and relationships from documents, enabling multi-hop reasoning across connected information. When a query arrives, it retrieves both vector-similar chunks and graph-connected entities, providing richer context. It excels at questions requiring reasoning across multiple related documents.

HyDE generates a hypothetical answer to the query using an LLM, then embeds that hypothetical answer to search the vector store instead of embedding the query directly. Since the hypothetical answer is closer in semantic space to actual document chunks than the question is, it often retrieves more relevant results, especially for complex queries.

Agentic RAG adds an agent layer that can iteratively decide whether to retrieve more information, refine queries, switch data sources, or synthesise across multiple retrievals. Unlike standard RAG with a fixed retrieve-then-generate flow, agentic RAG adapts its retrieval strategy based on initial results, enabling complex multi-step information gathering.

Collect domain-specific query-document pairs from search logs or manually curated examples. Create training triplets (query, positive document, negative document) using hard negative mining. Fine-tune a pre-trained embedding model (like BGE or E5) using contrastive loss. Evaluate on a held-out domain-specific test set using retrieval metrics. This typically yields significant improvement over general embeddings.

Equip the agent with tools for order lookup, return processing, refund status, and FAQ search. Support multilingual interactions (Hindi, English, and regional languages). Implement escalation to human agents for complex issues. Include tools for checking shipping status with Indian logistics providers (Delhivery, BlueDart). Handle Indian-specific payment issues (UPI failures, COD returns). Add sentiment detection for priority routing.

Collect domain-specific query-passage pairs from search logs or expert annotation. Create training triplets with hard negatives (similar but irrelevant passages). Fine-tune using contrastive learning (InfoNCE or multiple negatives ranking loss). Evaluate with retrieval metrics on a held-out test set. Use a pre-trained model like BGE or E5 as the base. Typically 5K-50K pairs suffice.