# Embeddings and Semantic Search — Finding Text by Meaning, Vector Databases

Source: https://skills101.in/learn/ai-engineering/embeddings-semantic-search — Skills101 free tutorial (AI Engineering — Building Apps with Large Language Models). Updated 2026-11-09.

Before an LLM can answer from your documents (next lesson), you must **find** the right passages. Searching by exact keywords fails when students use different words: *"how many classes can I bunk?"* vs the handbook's *"minimum attendance requirement"*.

## 1. Text as vectors

Search systems turn each piece of text into a **vector** — a list of numbers — so that similar texts get nearby vectors. Then search means: turn the question into a vector and find the **nearest** passages.

![Texts about attendance, minimum presence and bunking cluster together; hostel fees and mess charges cluster elsewhere; the query about missing classes lands next to the attendance cluster](https://skills101.in/learn-img/ai-engineering/vectors.svg)

Closeness is measured with **cosine similarity**: 1 = same direction (very similar), 0 = unrelated.

## 2. A runnable search over a handbook

**Embedding models** (neural networks trained for this) produce the best vectors, but they need an API or a model download. To see the mechanics with nothing to install beyond scikit-learn, we use **TF-IDF** vectors — word-based, but the search code is exactly the same as with real embeddings:

```python
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

handbook = [
    "Attendance: students must have at least 75% attendance in each subject to appear in the end-semester exam.",
    "Medical leave: absence with a medical certificate submitted within 7 days can be condoned up to 10%.",
    "Fees: the semester fee must be paid by the 10th of the first month; a late fee of Rs 500 applies after that.",
    "Hostel: mess charges are billed monthly and must be paid at the hostel office.",
    "Exams: hall tickets are issued two weeks before exams to students who meet the attendance rule.",
    "Library: books can be borrowed for 14 days; a fine of Rs 5 per day applies to late returns.",
]

vectorizer = TfidfVectorizer(stop_words="english")
doc_vectors = vectorizer.fit_transform(handbook)        # one vector per passage

def search(question: str, k: int = 2):
    q = vectorizer.transform([question])
    scores = cosine_similarity(q, doc_vectors)[0]
    best = scores.argsort()[::-1][:k]
    return [(round(float(scores[i]), 2), handbook[i][:60] + "…") for i in best]

for question in ["What attendance do I need for the exam?",
                 "Is there a fine if I pay the fee late?",
                 "how many classes can I bunk"]:
    print(question)
    for score, text in search(question):
        print("  ", score, text)
```

```text
What attendance do I need for the exam?
   0.6 Attendance: students must have at least 75% attendance in ea…
   0.15 Exams: hall tickets are issued two weeks before exams to stu…
Is there a fine if I pay the fee late?
   0.48 Fees: the semester fee must be paid by the 10th of the first…
   0.33 Library: books can be borrowed for 14 days; a fine of Rs 5 p…
how many classes can I bunk
   0.0 Library: books can be borrowed for 14 days; a fine of Rs 5 p…
   0.0 Exams: hall tickets are issued two weeks before exams to stu…
```

Look at the last question: *"how many classes can I bunk"* shares **no** important words with the attendance rule, so word-based search fails (scores near 0). That is exactly why **embedding models** exist — they learn that "bunk classes" and "attendance requirement" mean related things.

## 3. Real embeddings

In production you call an embedding model instead of TF-IDF:

- **Hosted APIs**: e.g. Voyage AI (recommended by Anthropic, as Claude itself doesn't produce embeddings), OpenAI, Google, Cohere.
- **Open-source**: `sentence-transformers` models you run yourself (free; needs a bit of compute), including multilingual ones that handle Hindi and English.

The rest of the code is unchanged: embed the documents once, embed each question, compare with cosine similarity.

## 4. Vector databases

Comparing a question against a million passages one by one is slow. **Vector databases** index vectors for fast nearest-neighbour search:

| Option | Good for |
|---|---|
| **pgvector** (PostgreSQL extension) | you already use Postgres — simplest choice for most apps |
| **Chroma**, **Qdrant**, **Weaviate**, **Milvus** | dedicated vector search, filters, scale |
| **FAISS** | a library for very fast in-memory search |

They store the vector **plus** the original text and metadata (source, page, date) so you can filter ("only the 2026 handbook") and cite.

## 5. Hybrid search

Keywords still matter — course codes (*CS301*), names, exact error messages. Strong systems combine **keyword search** (BM25) and **semantic search**, then optionally **re-rank** the top results with a more precise model.

## Common mistakes

- Using only semantic search for codes and IDs (keyword search is better there).
- Embedding whole 50-page documents as one vector — split them into passages first (next lesson).
- Mixing vectors from different embedding models in one index.
- Forgetting to re-embed after the documents change.

## Quick recap

- Text → vectors; similar meaning → nearby vectors; cosine similarity measures closeness.
- TF-IDF shows the mechanics but only matches words; embedding models capture meaning.
- Vector databases (pgvector, Chroma…) make nearest-neighbour search fast and keep metadata.
- Hybrid (keyword + semantic) search is the robust default.


Practice quiz and free certificate: https://skills101.in/learn/ai-engineering/embeddings-semantic-search