When RAG is the wrong tool for your feature
Decide whether to put documents in the prompt, query a database, or build retrieval by checking the answers, access rules, and cost of each option.
On this page
You do not need a retrieval pipeline merely because an AI feature answers questions about documents. Start by asking which information each request needs, how fresh it must be, and who may access it.
Retrieval-augmented generation (RAG) searches a source collection and gives selected material to the model as context for its answer. A small, current document set may work directly in the prompt. A precise account fact may call for a structured database query. Retrieval becomes useful when selecting relevant material from a larger or changing collection improves the result enough to justify operating the pipeline.
TL;DR Try direct context for a small document set, a database query for a current account fact, and retrieval for a larger collection. Check access, answer quality, and total cost on every path.
Choose the source path by the question
| Request | Starting point |
|---|---|
| Summarize a supplied document | Direct context |
| Find the current balance for this account | Authorized structured query |
| Answer across a large document collection | Retrieval with source and permission filters |
| Follow a stable output format | Clear instructions and structured output; evaluate before tuning |
| Combine current records and supporting policy | Structured query plus selected evidence |
These paths can coexist. A support feature might query the order system and retrieve the relevant returns policy. The model still needs to connect them correctly.
Fine-tuning can change a model's learned behavior, but it does not reliably fetch a current balance or apply changing account permissions. Those facts still need a current source and access checks.
Test direct context before building an index
Use the intended model's tokenizer to count how many input units, called tokens, the material contains. Page count is an unreliable proxy because layouts, tables, and languages vary.
Include instructions, conversation history, tool definitions, and the expected output budget. Fitting within the model's advertised window is only a capacity check. Evaluate whether the model uses the relevant material accurately.
Lost in the Middle reports sensitivity to where information appears in the evaluated contexts. Chroma's context-rot research provides further evidence that more input does not automatically improve performance. Neither study supplies a universal context-length cutoff for your application.
Use retrieval when selection earns its cost
A retrieval system must ingest sources, preserve useful structure, track versions, and return evidence for the question. It also needs an update and deletion process.
Evaluate whether the correct passage was retrieved before scoring the answer. If the evidence never arrived, changing the generation prompt may not solve the failure. If the correct passage arrived and was misread, another index may not help either.
Chunk boundaries can separate a rule from its exception. Search can favor a familiar but outdated document. Keep source identity and version information so those failures can be diagnosed.
Enforce access before the model sees data
Retrieval does not automatically provide access control. Check which customer organization, user, and documents the request may access during source selection. A direct-context design needs the same filtering before assembling its prompt.
Check caches and conversation history too. A previously authorized passage should not be reused after access is revoked unless the application's retention and authorization design permits it. Do not ask the model to conceal information it should never have received.
The OWASP authorization guidance supports checking permission on each relevant request. Model output is not a substitute for those controls.
Compare complete costs
For direct context, include the input, generation, and applicable caching charges. For retrieval, include importing documents, creating embeddings (numeric representations used for search), index storage, search, reranking (sorting the matches by relevance) if used, retrieved input, and generation. Both paths also require evaluation and maintenance.
Do not compare a full long-context model call with only the vector-search charge. That omits the rest of the retrieval workflow. Use a shared set of questions and the same completed-task definition.
A large reusable prefix may benefit from caching, depending on the provider's rules. A changing document collection may require frequent index updates. Those details can reverse an otherwise simple cost comparison.
Decide from a small comparison
Build the simplest viable baseline and test it on representative questions, permission boundaries, and missing evidence. Add retrieval only where the comparison shows a useful improvement or a requirement demands it.
If the source material is contradictory or nobody owns updates, fix that first. Both architectures can produce confidently wrong answers from poor evidence.
For the next design review, bring the questions, authorized source set, observed failures, and full operating costs. Choose the path that meets those requirements with an operational burden your team can maintain.
Sources
- Liu et al.: Lost in the Middle
- Chroma: Context rot
- OWASP: Authorization cheat sheet
Common questions
When can we skip a retrieval system?
When can we skip a retrieval system?
When a small, authorized, current set of material can be supplied directly and the result meets quality, latency, and cost requirements. Measure the actual prompt with the target tokenizer.
Does only RAG support access control?
Does only RAG support access control?
No. Both retrieval and direct-context systems must filter sources to what the user may access before the model receives them. Apply authorization again to any resulting action.
Is long context always more expensive than RAG?
Is long context always more expensive than RAG?
No. Compare complete request costs, including generation, caching, indexing, storage, retrieval, and maintenance. The result depends on workload and reuse.
Can fine-tuning replace current source data?
Can fine-tuning replace current source data?
It can change learned behavior and knowledge, but it is not a dependable way to retrieve current, account-specific facts or apply changing permissions. Use an authorized source for those facts.
