$ explain rag --agents
How AI Season teaches RAG
Retrieval-augmented generation (RAG) lets an AI agent answer from your own documents and data instead of its memory: find the right passages, hand them to the model, and make it answer from them. Here is how RAG works step by step, why it fails, how to measure and fix it, and how it plugs into AI agents.
By Abdul Rahman Azam, founder of AI Season · Updated
Key facts
| What RAG does | Retrieves relevant passages from your data and gives them to the model as context for its answer |
|---|---|
| Why it matters | Models don't know your private or recent information and can hallucinate; RAG grounds answers in sources |
| Pipeline | Load → chunk → embed → store in a vector database → retrieve → generate an answer with citations |
| Hardest part | Retrieval quality — getting the right chunks back, not the prompt |
| In agents | Retrieval becomes a tool the agent can decide to call, possibly several times |
| RAG vs fine-tuning | RAG adds knowledge at question time; fine-tuning changes the model's behaviour or style |
Why AI agents need RAG
A language model only knows what was in its training data, up to its knowledge cutoff. It has never seen your university's rules, your company's HR policy or yesterday's price list — and when asked, it may confidently make something up. RAG fixes that by looking up the relevant information at question time and putting it in front of the model, so the answer comes from your sources. It is the most common pattern in real-world AI applications.
How RAG works, step by step
Indexing (done ahead of time)
- Load your documents — PDFs, Word files, web pages, database rows.
- Chunk them into passages of a few hundred words, usually with some overlap so ideas aren't cut in half.
- Embed each chunk: an embedding model turns the text into a vector of numbers that captures its meaning.
- Store the vectors and the original text (plus metadata such as source and page) in a vector database.
Answering (every question)
- Embed the question with the same embedding model.
- Retrieve the chunks whose vectors are closest in meaning — often combined with keyword search (hybrid search).
- Optionally rerank the candidates with a more precise model and keep the best few.
- Generate: send the question and the retrieved chunks to the LLM with instructions to answer only from them and cite the source.
- Refuse honestly when the retrieved text doesn't contain the answer, instead of guessing.
Why RAG gives bad answers — and the fixes
| Symptom | Likely cause | Fix |
|---|---|---|
| Answer misses information that is in the documents | The right chunk was never retrieved | Better chunking, hybrid search, more candidates plus reranking |
| Answer mixes up two similar documents | Chunks lack context about where they came from | Add metadata (title, section, date) and filter on it |
| Answer is vague | Chunks too large or too many, burying the key sentence | Smaller chunks, fewer but better results |
| Answer contradicts the source | The model ignored the context | Stricter instructions, quoting and citations, answer checks |
| Confident answer to a question the documents don't cover | No refusal path | Tell the model to say "not in the documents" and test that it does |
| Scanned PDFs return nothing | No text layer | Run OCR before chunking |
Always test retrieval on its own before blaming the model: for a set of questions, check whether the chunk containing the answer appears in the top results.
Measuring RAG quality
Build a small test set — 20 to 50 real questions with the document passage that answers each — and track two things every time you change the pipeline:
- Retrieval quality — how often the right passage is in the top results (recall at k).
- Answer quality — whether the final answer is correct, faithful to the retrieved text, and cites the right source.
Rerun the test set after every change to chunk size, embedding model, search method or prompt. One good demo proves nothing; a test set tells you whether you improved.
Agentic RAG: retrieval as a tool
In a classic RAG app, every question triggers exactly one search. In an agent, retrieval becomes a tool the model decides when to use: it can search, read the result, notice something is missing, search again with a better query, combine sources, or call a different tool entirely — then answer. That makes agents much better at multi-part questions, at the cost of more calls and the need for step limits. See what AI agents are and how to build one.
RAG vs fine-tuning vs long context
| Approach | Use it when | Limits |
|---|---|---|
| RAG | Knowledge is private, large or changes often; you need citations | Quality depends on retrieval |
| Fine-tuning | You need a consistent style, format or narrow skill | Doesn't reliably add facts; retraining for updates |
| Long context (paste it all) | The documents are small enough to fit in the context window | Slower and costlier per question; weaker on very long inputs |
How AI Season teaches RAG
Session 3 of the AI Season curriculum builds a document Q&A agent over your own PDFs: loading, chunking, embeddings, vector search, retrieval methods and hybrid search, answer generation, and deliberately breaking the pipeline to see how it fails. The Cohort 01 code for this session is open source. Later sessions turn retrieval into an agent tool and add honest refusal and verification.
Frequently asked questions
What does RAG stand for?
Retrieval-augmented generation: retrieving relevant information and adding it to the model's input so the generated answer is grounded in it.
Do I need a vector database for RAG?
For anything beyond a handful of documents, yes — it stores embeddings and finds similar passages quickly. For learning, a local store such as Chroma or FAISS is enough.
Is RAG better than fine-tuning?
For adding knowledge — especially private or frequently changing information — RAG is usually the better first choice. Fine-tuning is better for changing a model's style or teaching a narrow format.
Can RAG work with Urdu or Hindi documents?
Yes, with a multilingual embedding model and a model that handles the language well. Test retrieval carefully, because quality varies more across languages.
What is agentic RAG?
RAG where an agent decides when and how to search — possibly several times with refined queries — instead of running exactly one retrieval per question.