RAG Tutorial in Python — Retrieval-Augmented Generation with Citations

RAG = find the relevant passages in your documents, put them in the prompt, and tell the model to answer only from them. It's the most widely deployed LLM pattern in companies — support bots, policy assistants, internal search.

Case study: the official college help-desk bot

Students ask the same questions every semester: attendance, fees, hall tickets, re-evaluation. The college wants a bot that answers from the official handbook only, shows the section it used, and sends anything else to the office.

1. Chunking

Long documents are split into chunks (a few hundred words each), small enough to be specific, big enough to keep context. A little overlap stops answers being cut in half:

def chunk_text(text: str, size: int = 40, overlap: int = 10) -> list[str]:
    words = text.split()
    chunks, start = [], 0
    while start < len(words):
        chunks.append(" ".join(words[start:start + size]))
        if start + size >= len(words):
            break
        start += size - overlap                      # step forward, keeping some overlap
    return chunks

page = " ".join(f"w{i}" for i in range(100))          # a 100-word page
for c in chunk_text(page):
    print(c.split()[0], "...", c.split()[-1], f"({len(c.split())} words)")

w0 ... w39 (40 words)
w30 ... w69 (40 words)
w60 ... w99 (40 words)

Real systems split on headings and paragraphs first (a section is a natural unit) and keep metadata: document name, section number, page.

2. Retrieve, then generate with sources

import anthropic
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

SECTIONS = {
    "4.1 Attendance": "Students must have at least 75% attendance in each subject to appear in the end-semester exam.",
    "4.2 Medical leave": "Absence with a medical certificate submitted within 7 days can be condoned up to 10%.",
    "6.1 Fees": "The semester fee must be paid by the 10th of the first month; a late fee of Rs 500 applies after that.",
    "7.3 Hall tickets": "Hall tickets are issued two weeks before exams to students who meet the attendance rule.",
}
titles, texts = list(SECTIONS), list(SECTIONS.values())
vectorizer = TfidfVectorizer(stop_words="english")
index = vectorizer.fit_transform(texts)

def retrieve(question: str, k: int = 2, min_score: float = 0.1):
    scores = cosine_similarity(vectorizer.transform([question]), index)[0]
    ranked = scores.argsort()[::-1][:k]
    return [(titles[i], texts[i]) for i in ranked if scores[i] >= min_score]

def answer(question: str) -> str:
    sources = retrieve(question)
    if not sources:                                     # nothing relevant: don't let the model guess
        return "I can't find this in the handbook. Please contact the academic office."
    context = "\n".join(f"[{n}] ({title}) {text}" for n, (title, text) in enumerate(sources, 1))
    prompt = f"""Answer the student's question using ONLY the handbook sections below.
Cite the section numbers you used like [1]. If the sections don't contain the answer, say you don't know.

<context>
{context}
</context>

Question: {question}"""
    response = anthropic.Anthropic().messages.create(
        model="claude-opus-5-5", max_tokens=300, messages=[{"role": "user", "content": prompt}])
    used = ", ".join(t for t, _ in sources)
    return response.content[0].text + f"\n(sources: {used})"

print(answer("What attendance do I need to write the end-semester exam?"))
print(answer("Who won the college cricket tournament?"))

Output:

You need at least 75% attendance in each subject to sit the end-semester exam [1].
(sources: 4.1 Attendance, 7.3 Hall tickets)
I can't find this in the handbook. Please contact the academic office.

(The first answer's wording comes from our local stand-in API; the retrieval, the sources and the out-of-scope refusal are computed by the code above.)

The second question is the important one: nothing relevant was retrieved, so the bot refuses before calling the model — no chance to invent an answer.

3. Measuring retrieval

If retrieval brings the wrong passages, even the best model fails. Build a small test set of real questions with the section that should be found, and measure hit rate @ k:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

SECTIONS = {
    "4.1 Attendance": "Students must have at least 75% attendance in each subject to appear in the end-semester exam.",
    "4.2 Medical leave": "Absence with a medical certificate submitted within 7 days can be condoned up to 10%.",
    "6.1 Fees": "The semester fee must be paid by the 10th of the first month; a late fee of Rs 500 applies after that.",
    "7.3 Hall tickets": "Hall tickets are issued two weeks before exams to students who meet the attendance rule.",
}
titles = list(SECTIONS)
vectorizer = TfidfVectorizer(stop_words="english")
index = vectorizer.fit_transform(SECTIONS.values())

tests = [
    ("What attendance is needed for exams?", "4.1 Attendance"),
    ("When will I get my hall ticket?", "7.3 Hall tickets"),
    ("What is the late fee?", "6.1 Fees"),
    ("I was ill for a week, will my absence count?", "4.2 Medical leave"),
    ("Can I bunk classes and still sit the exam?", "4.1 Attendance"),
]
for k in (1, 2):
    hits = 0
    for question, expected in tests:
        scores = cosine_similarity(vectorizer.transform([question]), index)[0]
        top = [titles[i] for i in scores.argsort()[::-1][:k]]
        hits += expected in top
    print(f"hit rate @ {k}: {hits}/{len(tests)}")

hit rate @ 1: 4/5
hit rate @ 2: 5/5

Misses show you what to improve: better chunking, adding synonyms, real embeddings, or hybrid search. Fix retrieval before blaming the model.

4. Making it production-ready

Citations everywhere — users (and you) can check answers.
Freshness — re-index when the handbook changes; store the version and date.
Access control — only retrieve documents the user may see (staff vs students).
Limits — top-k of 3–8 chunks keeps prompts small and focused.
Feedback — a 👍/👎 button gives you new test questions.

Common mistakes

Giant chunks (whole chapters) or tiny ones (single sentences without context).
No "not in the documents" path — the model fills gaps with guesses.
Evaluating only the final answers, never the retrieval.
Leaking confidential documents through retrieval without permission checks.

Quick recap

RAG: chunk → index → retrieve top-k → prompt with numbered sources → answer with citations.
Refuse when nothing relevant is found; instruct the model to answer only from the context.
Measure retrieval with a test set (hit rate @ k) and fix it first.

RAG — Retrieval-Augmented Generation with Chunking, Citations and Evaluation