# Hybrid Search — BM25 Keyword Ranking, Reciprocal Rank Fusion and Re-ranking

Source: https://skills101.in/learn/ai-engineering/ai-engineering-hybrid-search-bm25 — Skills101 free tutorial (AI Engineering — Building Apps with Large Language Models). Updated 2026-11-21.

The embeddings lesson ended with a warning: semantic search is good at **meaning** but weak at **exact tokens** such as course codes (*CS301*), roll numbers, people's names and error messages (*ORA-00942*). Real RAG systems therefore run two searches, keyword and semantic, and merge the results. This is called **hybrid search**. In this lesson you will write the standard keyword ranking, **BM25**, in plain Python, then merge two result lists with **reciprocal rank fusion**.

## 1. What a keyword ranker must get right

Imagine the question *"CS301 lab fee"* against a college handbook. A good keyword ranker should follow three common-sense rules:

1. **Rare words matter more.** "CS301" appears in one passage; "fee" appears in many. A match on "CS301" should count for much more. This weight is the **inverse document frequency (IDF)**.
2. **Repeating a word helps, but with diminishing returns.** A passage that says "fee" five times is not five times as relevant as one that says it once. This is **term-frequency saturation**.
3. **Long passages should not win just by being long.** A 500-word passage contains more words by chance. Scores are adjusted for **passage length**.

**BM25** (Best Matching 25) is a formula that does all three. It has been the default ranking in search engines such as Elasticsearch, OpenSearch and Lucene for years, and it is still a strong baseline.

## 2. BM25 in plain Python

For each query word that appears in a passage:

score += IDF(word) × tf × (k1 + 1) / (tf + k1 × (1 − b + b × length / average_length))

- `tf` = how many times the word appears in the passage.
- `k1` (usually 1.2 to 2.0) controls how fast repeats stop helping.
- `b` (usually 0.75) controls how strongly long passages are penalised.

```python
import math
import re
from collections import Counter

passages = [
    "CS301 Database Systems: the lab fee is Rs 1500 per semester, paid with the tuition fee.",
    "Fee policy: the semester fee is due by the 10th. Late fee Rs 500. Fee receipts are on the portal.",
    "Library: a fine of Rs 5 per day applies to late returns.",
    "CS302 Operating Systems lab: bring your own laptop; no lab fee.",
    "Hostel fee covers rent and mess. Hostel fee is paid every semester at the hostel office.",
]

def tokens(text):
    return re.findall(r"[a-z0-9]+", text.lower())

docs = [tokens(p) for p in passages]
N = len(docs)
avg_len = sum(len(d) for d in docs) / N
doc_freq = Counter(word for d in docs for word in set(d))

def idf(word):
    n = doc_freq[word]
    return math.log((N - n + 0.5) / (n + 0.5) + 1)

def bm25(query, doc, k1=1.5, b=0.75):
    tf = Counter(doc)
    score = 0.0
    for word in tokens(query):
        if tf[word]:
            score += idf(word) * tf[word] * (k1 + 1) / (tf[word] + k1 * (1 - b + b * len(doc) / avg_len))
    return score

print("IDF  cs301 =", round(idf("cs301"), 2), "  lab =", round(idf("lab"), 2), "  fee =", round(idf("fee"), 2))
query = "CS301 lab fee"
ranked = sorted(range(N), key=lambda i: bm25(query, docs[i]), reverse=True)
for i in ranked:
    print(f"{bm25(query, docs[i]):5.2f}  {passages[i][:55]}")
```

```text
IDF  cs301 = 1.39   lab = 0.88   fee = 0.29
 2.60  CS301 Database Systems: the lab fee is Rs 1500 per seme
 1.69  CS302 Operating Systems lab: bring your own laptop; no 
 0.49  Fee policy: the semester fee is due by the 10th. Late f
 0.40  Hostel fee covers rent and mess. Hostel fee is paid eve
 0.00  Library: a fine of Rs 5 per day applies to late returns
```

Read the output:

- "cs301" has the highest IDF because it appears in only one passage; "fee" appears in four, so it is worth little.
- The CS301 passage wins clearly. The fee-policy passage says "fee" four times, but saturation and the low IDF of "fee" keep it from winning on repetition alone.
- The library passage scores 0: it shares no query words.

## 3. Diminishing returns, shown with numbers

```python
def tf_part(tf, k1=1.5):
    return tf * (k1 + 1) / (tf + k1)        # length part left out (a passage of average length)

for tf in (1, 2, 3, 5, 10, 50):
    print(f"word appears {tf:2d} times -> contribution x{tf_part(tf):.2f}")
```

```text
word appears  1 times -> contribution x1.00
word appears  2 times -> contribution x1.43
word appears  3 times -> contribution x1.67
word appears  5 times -> contribution x1.92
word appears 10 times -> contribution x2.17
word appears 50 times -> contribution x2.43
```

![BM25 term-frequency curve: the contribution rises quickly for the first few repeats of a word and then flattens out below k1 plus 1, while a simple count keeps rising in a straight line](https://skills101.in/learn-img/ai-engineering/bm25-saturation.svg)

The contribution can never exceed `k1 + 1` (2.5 here). That is why stuffing a passage with a keyword does not trick BM25, which is one reason it beat plain word counting.

## 4. Merging keyword and semantic results: reciprocal rank fusion

Now we have two ranked lists: one from BM25 and one from embedding search. Their scores are on **different scales** (BM25 might give 2.8, cosine similarity 0.61), so adding the scores directly is unreliable. **Reciprocal rank fusion (RRF)** ignores the scores and uses only the **ranks**:

RRF score of a passage = Σ over lists of 1 / (k + rank in that list), with k usually 60.

A passage near the top of **both** lists gets the highest combined score. In the example below the BM25 ranking is the one computed above. For the semantic ranking, we assume the order an embedding search might return for this question; in a real app it comes from your vector database (lesson 6).

```python
bm25_ranking = ["cs301", "cs302", "fee_policy", "hostel"]          # from BM25 above
semantic_ranking = ["fee_policy", "cs301", "hostel", "library"]    # assumed vector-search order

def rrf(rankings, k=60):
    scores = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
    return sorted(scores.items(), key=lambda item: item[1], reverse=True)

for doc_id, score in rrf([bm25_ranking, semantic_ranking]):
    print(f"{doc_id:11s} {score:.4f}")
```

```text
cs301       0.0325
fee_policy  0.0323
hostel      0.0315
cs302       0.0161
library     0.0156
```

![Reciprocal rank fusion: the BM25 list and the semantic list are combined by rank, so the CS301 passage, ranked first and second, comes out on top, while the library passage found by only one list ends last](https://skills101.in/learn-img/ai-engineering/rrf-fusion.svg)

The CS301 passage is 1st in one list and 2nd in the other, so it comes out on top. The library passage appears in only one list, at rank 4, so it ends last. RRF needs no tuning of score scales, which is why many vector databases and search engines offer it as their built-in hybrid mode.

## 5. Re-ranking: a slower, sharper second look

BM25 and embedding search are fast because they score the query and each passage **separately**. A **re-ranker** (also called a **cross-encoder**) reads the query and a passage **together** and gives a more accurate relevance score, but it is far slower. So it is used in two stages:

| Stage | Method | Passages handled | Speed |
|---|---|---|---|
| 1. Retrieve | BM25 + embeddings, merged with RRF | whole collection → top 30–100 | fast |
| 2. Re-rank | cross-encoder model | top 30–100 → best 3–10 | slower |
| 3. Generate | LLM answers from the best passages, with citations | 3–10 | per answer |

Re-rankers are available as hosted APIs (for example from Cohere and Voyage AI) and as open-source models you can run yourself (for example cross-encoder models through the `sentence-transformers` library). Whether re-ranking is worth the extra time and cost depends on your data, so measure it with the retrieval tests from the RAG lesson (recall@k) before and after.

## Common mistakes

- Using only semantic search for queries full of codes, names or IDs.
- Adding BM25 scores and cosine similarities directly, even though they are on different scales.
- Tokenising the documents and the queries differently (for example, lower-casing one but not the other), so exact matches are missed.
- Re-ranking the entire collection instead of only the top candidates.
- Adding hybrid search or a re-ranker without measuring whether retrieval actually improved.

## Quick recap

- Hybrid search = keyword search (BM25) + semantic search, merged.
- BM25 rewards rare words (IDF), gives diminishing returns for repeats (k1) and adjusts for passage length (b).
- Reciprocal rank fusion merges ranked lists using ranks only: Σ 1 / (k + rank).
- A re-ranker reads query and passage together; use it on the top candidates only.
- Measure retrieval (recall@k) before and after every change.


Practice quiz and free certificate: https://skills101.in/learn/ai-engineering/ai-engineering-hybrid-search-bm25