Embeddings and Semantic Search — Finding Text by Meaning, Vector Databases

Before an LLM can answer from your documents (next lesson), you must find the right passages. Searching by exact keywords fails when students use different words: "how many classes can I bunk?" vs the handbook's "minimum attendance requirement".

1. Text as vectors

Search systems turn each piece of text into a vector — a list of numbers — so that similar texts get nearby vectors. Then search means: turn the question into a vector and find the nearest passages.

Closeness is measured with cosine similarity: 1 = same direction (very similar), 0 = unrelated.

2. A runnable search over a handbook

Embedding models (neural networks trained for this) produce the best vectors, but they need an API or a model download. To see the mechanics with nothing to install beyond scikit-learn, we use TF-IDF vectors — word-based, but the search code is exactly the same as with real embeddings:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

handbook = [
    "Attendance: students must have at least 75% attendance in each subject to appear in the end-semester exam.",
    "Medical leave: absence with a medical certificate submitted within 7 days can be condoned up to 10%.",
    "Fees: the semester fee must be paid by the 10th of the first month; a late fee of Rs 500 applies after that.",
    "Hostel: mess charges are billed monthly and must be paid at the hostel office.",
    "Exams: hall tickets are issued two weeks before exams to students who meet the attendance rule.",
    "Library: books can be borrowed for 14 days; a fine of Rs 5 per day applies to late returns.",
]

vectorizer = TfidfVectorizer(stop_words="english")
doc_vectors = vectorizer.fit_transform(handbook)        # one vector per passage

def search(question: str, k: int = 2):
    q = vectorizer.transform([question])
    scores = cosine_similarity(q, doc_vectors)[0]
    best = scores.argsort()[::-1][:k]
    return [(round(float(scores[i]), 2), handbook[i][:60] + "…") for i in best]

for question in ["What attendance do I need for the exam?",
                 "Is there a fine if I pay the fee late?",
                 "how many classes can I bunk"]:
    print(question)
    for score, text in search(question):
        print("  ", score, text)

What attendance do I need for the exam?
   0.6 Attendance: students must have at least 75% attendance in ea…
   0.15 Exams: hall tickets are issued two weeks before exams to stu…
Is there a fine if I pay the fee late?
   0.48 Fees: the semester fee must be paid by the 10th of the first…
   0.33 Library: books can be borrowed for 14 days; a fine of Rs 5 p…
how many classes can I bunk
   0.0 Library: books can be borrowed for 14 days; a fine of Rs 5 p…
   0.0 Exams: hall tickets are issued two weeks before exams to stu…

Look at the last question: "how many classes can I bunk" shares no important words with the attendance rule, so word-based search fails (scores near 0). That is exactly why embedding models exist — they learn that "bunk classes" and "attendance requirement" mean related things.

3. Real embeddings

In production you call an embedding model instead of TF-IDF:

• Hosted APIs: e.g. Voyage AI (recommended by Anthropic, as Claude itself doesn't produce embeddings), OpenAI, Google, Cohere.
• Open-source: sentence-transformers models you run yourself (free; needs a bit of compute), including multilingual ones that handle Hindi and English.

The rest of the code is unchanged: embed the documents once, embed each question, compare with cosine similarity.

4. Vector databases

Comparing a question against a million passages one by one is slow. Vector databases index vectors for fast nearest-neighbour search:

They store the vector plus the original text and metadata (source, page, date) so you can filter ("only the 2026 handbook") and cite.

5. Hybrid search

Keywords still matter — course codes (CS301), names, exact error messages. Strong systems combine keyword search (BM25) and semantic search, then optionally re-rank the top results with a more precise model.

Common mistakes

• Using only semantic search for codes and IDs (keyword search is better there).
• Embedding whole 50-page documents as one vector — split them into passages first (next lesson).
• Mixing vectors from different embedding models in one index.
• Forgetting to re-embed after the documents change.

Quick recap

• Text → vectors; similar meaning → nearby vectors; cosine similarity measures closeness.
• TF-IDF shows the mechanics but only matches words; embedding models capture meaning.
• Vector databases (pgvector, Chroma…) make nearest-neighbour search fast and keep metadata.
• Hybrid (keyword + semantic) search is the robust default.