What does RAG stand for in AI?

RAG stands for retrieval-augmented generation. The name comes from a 2020 research paper led by Patrick Lewis, who later told NVIDIA the team would have put more thought into the name had they known how widely it would spread.

What is RAG in AI in simple terms?

It is an open-book exam for a chatbot. Before answering, the system searches a set of documents for the passages most related to your question, hands them to the model, and the model writes its answer from them.

What is the difference between RAG and fine-tuning?

Fine-tuning changes the model itself by training it further, which suits style and repeated formats. RAG leaves the usual model alone and feeds it facts at question time, which suits information that changes, because you only update the documents.

What is the difference between RAG and MCP?

RAG is a technique for finding information and adding it to a prompt. MCP, the Model Context Protocol, is an open standard for connecting AI apps to outside data and tools. An app could use an MCP connection to reach the search system that does the retrieving.

Does RAG stop AI hallucinations?

No. It reduces them when the right passage is found, but the search can miss it and the model can still misread it. Stanford researchers found legal tools built on RAG gave wrong information in 17% to 34% of test queries, so citations still need checking.

What is RAG in AI? How retrieval-augmented generation works, explained

The technique behind chatbots that answer from your own files: what it does, where you already use it, and why it still gets things wrong.

NextDiff EditorialEditorial team · Published Nov 7, 2026 05:11 UTC · Updated Nov 7, 2026 12:49 UTC

What is RAG in AI? How retrieval-augmented generation works, explained

+ In short

RAG, short for retrieval-augmented generation, is a technique that makes an AI model search a chosen set of documents or data first, then write its answer from what it found. It lets chatbots use current or private information they were never trained on, though it reduces wrong answers rather than eliminating them.

What is RAG in AI? RAG, short for retrieval-augmented generation, is a technique that makes an AI model search a chosen set of documents or data first, then write its answer from what it found. It lets chatbots answer from current or private sources they were never trained on.

Our 27-second version:

Watch on YouTube

A large language model on its own answers from patterns learned in training. It knows nothing about your company handbook or anything published after its training ended, and it may still produce a confident answer. RAG changes where the facts come from. AWS defines it as making a model reference "an authoritative knowledge base outside of its training data sources before generating a response".

The idea got its name in a May 2020 paper led by Patrick Lewis, accepted at NeurIPS 2020, with coauthors from what is now Meta AI, University College London and New York University. The original system searched an index of Wikipedia. The name was never meant to stick. "We always planned to have a nicer sounding name," Lewis told NVIDIA, "but when it came time to write the paper, no one had a better idea."

What is RAG in AI, and how does it work?

Think of an open-book exam. A closed-book student answers from memory and sometimes bluffs. An open-book student finds the right page first, then writes the answer with that page in front of them. RAG gives the model the book and a fast way to find the page.

Diagram of how RAG works: documents become chunks, embeddings and a search index; each question is embedded, retrieves chunks, augments the prompt and the LLM generates an answer Image: NextDiff

The real mechanics come in two stages.

First, build the index, once and then whenever documents change:

Split. Documents are cut into chunks, usually no more than a few hundred tokens each, so the system can fetch one relevant passage rather than a whole manual.
Embed. An embedding model turns each chunk into a list of numbers. OpenAI describes an embedding as a vector whose distance from another vector measures how related the two are. Its text-embedding-3-small model produces 1,536 numbers per chunk by default, and text-embedding-3-large produces 3,072.
Store. The vectors go into a search index, typically a vector database.

Then, on every question:

Retrieve. The question is embedded the same way and the closest chunks are pulled out. That matches meaning, so "time off" can find a passage about "annual leave". Microsoft recommends hybrid queries that combine keyword and vector search for maximum recall, and a reranking step that re-scores results by meaning.
Augment. Those chunks are pasted into the prompt along with the question.
Generate. The model writes its answer from them and, if the product supports it, cites them.

Keeping answers current then means updating the index, not the model. The 2020 paper already showed that the retrieved memory can be replaced to update what the system knows as the world changes.

Examples of RAG you may already use

NotebookLM. Google said at launch in 2023 that the model "only has access to the source material" you upload, with citations showing the original quotes. Google's help pages now call it Gemini Notebook and list limits from 50 sources per notebook on the standard plan to 600 on the top Ultra plan.
Claude Projects. Anthropic says RAG activates automatically when project files approach the context window limit, using a project knowledge search tool, and expands capacity "by up to 10x".
OpenAI file search. A built-in API tool that searches uploaded files through semantic and keyword search.
Legal research. Lexis+ AI and Westlaw AI-Assisted Research apply the approach to case law, with the mixed results covered below.

Google NotebookLM interface with a sources panel, a chat answer and a Studio notes panel, showing 7 sources in the question box Image: Google (NotebookLM interface, December 2024)

RAG vs fine-tuning vs a plain chatbot

	Plain chatbot	Paste it all in	RAG	Fine-tuning
Where facts come from	Training data	The prompt	Search results added to the prompt	Further training
Changes the model?	No	No	Usually no	Yes
Update facts by	Waiting for a new model	Pasting new text	Updating the index	Training again
Can point to a source?	Not reliably	Yes	Yes	Not reliably
Best for	General knowledge	Small document sets	Large or changing document sets	Tone and repeated formats

The "paste it all in" column matters more than most guides admit. Anthropic wrote in 2024 that a knowledge base under 200,000 tokens, about 500 pages, can simply be included in the prompt, without retrieval.

Why RAG matters

Most useful company knowledge is not in any model's training data: contracts, support tickets, product docs, this morning's numbers. RAG is how a general model gets to work on that material without being retrained, and how an AI agent looks things up mid-task. Anthropic counts retrieval among the basic building blocks of an agent.

It also means most answer quality depends on an old problem: search. If the right passage is not retrieved, the model never sees it. Anthropic measured this in its own tests by adding context to each chunk before indexing it. The share of searches that missed the right passage in the top 20 chunks fell from 5.7% to 1.9% once it added contextual keyword search and reranking.

Bar chart of Anthropic's retrieval failure rates: 5.7% baseline, 3.7% with contextual embeddings, 2.9% with contextual BM25, 1.9% with reranking Image: NextDiff, data from Anthropic. A vendor's own test, and it measures retrieval, not final answers.

Common misconceptions about RAG

"RAG stops hallucinations." It reduces them. Stanford researchers tested legal tools built on RAG with over 200 queries in 2024. Lexis+ AI and Ask Practical Law AI gave incorrect information more than 17% of the time, and Westlaw AI-Assisted Research more than 34%. That beat GPT-4 used on its own, but vendors had advertised "hallucination-free" citations. LexisNexis and Thomson Reuters disputed parts of the findings.

Stanford chart of hallucination and incomplete answer rates for Lexis, Westlaw, Practical Law and GPT-4 Image: Magesh et al., Stanford, via Stanford HAI

"RAG retrains the model." Usually not. The 2020 paper did fine-tune its retriever and generator together, but today's common setup leaves the model's weights alone and only changes what goes into the prompt.

"You need a vector database." Retrieval can also be keyword search, and Anthropic's best results combined both. For a small collection you may not need retrieval at all.

"A citation proves the answer is right." It proves a passage was retrieved. The model can still misread it, so open the source when it matters.

"Your documents are only ever data." Retrieved text goes straight into the prompt. OWASP lists indirect prompt injection, where instructions hidden in files or web pages steer the model, among the top risks for LLM apps.

Latest news

What is an LLM? The model that RAG feeds, and why it hallucinates.
What are AI agents? Retrieval is one of the tools agents use.
Gemini vs ChatGPT How the two biggest assistants compare, including reading your files.
Mistral Large 4 open weights A downloadable model that a company could run on its own hardware behind its own search.

Questions readers ask

What does RAG stand for in AI?: RAG stands for retrieval-augmented generation. The name comes from a 2020 research paper led by Patrick Lewis, who later told NVIDIA the team would have put more thought into the name had they known how widely it would spread.
What is RAG in AI in simple terms?: It is an open-book exam for a chatbot. Before answering, the system searches a set of documents for the passages most related to your question, hands them to the model, and the model writes its answer from them.
What is the difference between RAG and fine-tuning?: Fine-tuning changes the model itself by training it further, which suits style and repeated formats. RAG leaves the usual model alone and feeds it facts at question time, which suits information that changes, because you only update the documents.
What is the difference between RAG and MCP?: RAG is a technique for finding information and adding it to a prompt. MCP, the Model Context Protocol, is an open standard for connecting AI apps to outside data and tools. An app could use an MCP connection to reach the search system that does the retrieving.
Does RAG stop AI hallucinations?: No. It reduces them when the right passage is found, but the search can miss it and the model can still misread it. Stanford researchers found legal tools built on RAG gave wrong information in 17% to 34% of test queries, so citations still need checking.