Retrieval for agents
In a RAG pipeline, retrieval runs once on the user's words. In an agent, retrieval is a tool the agent calls with a query it wrote, possibly several times, and that difference changes every part of the design.
How it works
- Expose retrieval as a tool with a clear description of what corpus it covers.
- Let the agent write the query, and log both the agent's query and the user's original words.
- Return sources with the text, always, so the answer can cite and the verifier can check.
- Cap how many times it may search per run, because search is where loops happen.
- Measure retrieval quality on the agent's queries, not on the user's questions.
Optional: load a real model
See it work
When it pays, and when it does not
| Use it when | Skip it when |
|---|---|
| The corpus is larger than the context window | Everything fits in the prompt, where retrieval adds a failure mode for nothing |
| Queries vary widely | There are ten possible questions, which you can answer with ten cached answers |
| Freshness matters | The content is static and small |
| You need citations | Nobody will ever check where the answer came from, which is rarely true in practice |
How it fails
Bad agent queries
The agent searches for the whole user sentence and gets nothing. Log queries and fix the tool description first.
Retrieval loops
Search, unsatisfied, search again, forever. Cap searches per run and require a different query each time.
Unsourced answers
If the agent can answer without citing the retrieved text, it will answer from the model's memory instead.
What it costs
| Tokens | Retrieved chunks are expensive in a loop, because they persist in the transcript for every later step. |
|---|---|
| Latency | One extra round trip per search, and agents search more than pipelines do. |
| Quality ceiling | Your retrieval quality is the ceiling on your answer quality, whatever model you use. |
Log the query the agent wrote. Most bad agentic retrieval is a bad query, not a bad index, and you cannot see it without that log.
Related: Tool calling · Memory · Verification · all patterns · agent jobs
Related: Tool calling · Memory · Verification · all patterns · agent jobs
Free from AI School - no signup, everything runs in your browser.