Retrieval-Augmented Generation (RAG)

1. Ingestion pipeline crawls enterprise sources (wikis, PDFs, ticketing systems, databases), then chunks each document into semantically coherent passages of a few hundred tokens with overlap to preserve context.
2. Embedding model converts each chunk into a dense vector, capturing semantic meaning rather than keywords, and attaches metadata (source, ACL tags, timestamp, version).
3. Vector database stores these embeddings in an indexed structure (HNSW or IVF) enabling millisecond approximate nearest-neighbor search across millions of chunks.
4. User submits a natural-language question through an application or chat interface.
5. Retrieval service embeds the query with the same model, runs a similarity search filtered by the user's access entitlements, and returns the top-k most relevant chunks.
6. Orchestrator assembles a grounded prompt: system instructions + retrieved context + the user question, enforcing a token budget and citation requirements.
7. LLM generates an answer constrained to the supplied context, returning inline citations that link back to source documents for verification.

• Grounded, current answers vs the operational cost of maintaining an embedding pipeline and re-indexing on document change.
• No model retraining vs added retrieval latency on every request.
• Source citations and auditability vs sensitivity to chunking strategy and retrieval quality.
• Access-controlled context vs the complexity of propagating document-level entitlements into vector search.

A global insurance carrier deploys a RAG assistant so claims adjusters can ask, "What is our coverage limit for water damage under the 2024 homeowner policy in Texas?" The ingestion pipeline embeds thousands of versioned policy PDFs and state-specific endorsements nightly. When an adjuster queries, the retrieval service filters vectors by the adjuster's regional entitlements, pulls the three most relevant clauses, and the LLM composes a plain-language answer with citations to exact policy sections. Because responses are grounded in retrieved clauses rather than model memory, the carrier reduces incorrect coverage guidance and gives auditors a verifiable evidence trail for every recommendation.

• Embedding version coupling: Queries and stored chunks must be embedded by the identical model version; changing embedding models invalidates the entire index and forces a full re-embed, so pin the version and plan backfills as migrations.
• Stale and orphaned vectors: When source documents are deleted or superseded, their vectors must be purged or tombstoned, or retrieval will surface withdrawn policies; drive index mutations from document versions and reconcile deletions on each ingestion run.
• Chunk boundary loss: Fixed-size chunking can split tables, clauses, or code mid-context; tune chunk size and overlap per document type and store parent-document references so the orchestrator can expand context when a chunk is truncated.
• Entitlement enforcement: ACL tags in metadata must be applied as pre-filters in the ANN query, not post-filtering, otherwise top-k is computed over documents the user cannot see, degrading recall and risking leakage.
• Retrieval observability: Instrument recall@k, retrieval latency, chunks-retrieved-but-uncited, and empty-result rates; log the exact chunks injected per response for audit and to diagnose hallucinations traced to poor retrieval versus generation.
• Cost drivers: Re-embedding volume, index memory footprint (HNSW graphs are RAM-resident), and prompt token count dominate spend; cap top-k and context budget, and schedule index compaction to control fragmentation from frequent upserts.