Filtered and Hybrid Retrieval at Scale
A filter that keeps 1% of the corpus turns approximate search into a lottery. Partition, pre-filter, or over-fetch — with numbers.
The first version of our search filtered after the vector search: ask the index for the top 100, keep the ones belonging to the tenant. It worked in staging, where the test tenant owned 40% of the corpus. In production the median tenant owned 0.4%, and the feature looked broken.
This is the most common way a vector search product ships with good latency and terrible relevance.
The arithmetic of post-filtering
With a filter that keeps a fraction s of the corpus, the probability that a true neighbour survives a post-filter over k results is roughly s per result. For the top-10:
Selectivity s | Expected kept from top-100 | Effective recall |
|---|---|---|
| 50% | 50 | ~0.98 |
| 10% | 10 | ~0.65 |
| 1% | 1 | ~0.10 |
The index did its job: it returned the global top-100. Your tenant was not in it.
Over-fetching helps until it does not. At s = 1%, to keep 10 results you need to fetch about 1,000 candidates, which costs latency and still gives you an approximate set. At s = 0.1%, it is hopeless.
The three strategies that work
1. Partition the index. One index (or one partition) per tenant or category. The filter becomes routing, and the search inside a partition is unfiltered. This is the only option that keeps both recall and latency at low selectivity.
2. Pre-filter with a bitmap, then search inside it. Engines that support filtered HNSW (Qdrant's filterable HNSW, Weaviate's filtered vector search) traverse the graph while respecting the filter. When the filter is very selective, this degrades to a brute-force scan over the allowed set; when it is loose, it is close to unfiltered HNSW.
3. Brute force on the filtered subset. If the allowed set is small enough — under ~100k vectors — a flat scan is predictable and exact. On 500k × 768-dim, a single node takes about 120ms, which may be acceptable for background jobs and unacceptable for a search box.
What it looks like on our data
10M vectors, 768 dimensions, HNSW ef=64, k=10, one 16-core node:
| Strategy | Selectivity | Recall@10 | p99 |
|---|---|---|---|
| No filter | 100% | 0.96 | 4.2ms |
| Post-filter, fetch 100 | 5% | 0.31 | 4.3ms |
| Post-filter, fetch 1000 | 5% | 0.72 | 17ms |
| Pre-filter bitmap | 5% | 0.95 | 6.8ms |
| Partition by tenant | 5% avg | 0.96 | 4.5ms |
| Brute force filtered | 5% | 1.00 | 120ms |
The partition row is the reason we rebuilt the index: routing by tenant is a hash lookup, and the recall cost disappears.
Hybrid retrieval, briefly
Vector similarity is bad at exact identifiers, rare tokens, and negation; BM25 is bad at paraphrase. Combining them is usually worth the 2ms:
score = w_bm25 / (60 + rank_bm25) + w_vec / (60 + rank_vec)On our judgment set (400 queries with graded relevance), nDCG@10:
| Method | nDCG@10 | p99 |
|---|---|---|
| BM25 only | 0.50 | 9ms |
| Vector only | 0.54 | 5ms |
| Hybrid RRF, equal weights | 0.61 | 11ms |
| Hybrid, vector weight 1.5 | 0.62 | 11ms |
The gain is not from a clever fusion formula. It is from the two retrievers failing on different queries, and rank fusion being robust without score calibration.
Evaluation details that matter
- Build the judgment set from real queries, including the ones you think are silly. The silly ones are where BM25 wins.
- Evaluate per filter slice, not just globally. A 0.9 average can hide 0.3 on your largest tenant.
- Measure recall of the retrieval stage separately from the ranking model. Otherwise you will tune the ranker to compensate for missing documents.
What I would do first
- Measure selectivity distribution; if the median filter keeps under 10%, partitioning is not optional.
- Add hybrid retrieval before adding a cross-encoder; it is cheaper and helps more queries.
- Keep a brute-force path for audits and for measuring recall.
A filter is not a WHERE clause bolted onto a similarity search. It changes which index structure can answer the question, and at low selectivity it changes the answer entirely.