Moving nDCG
As a surprise to no one, every company has different data. Because of that we’ve been spending the last several months building a search datastore around a learning loop. The idea is we can record how users interact with your search and improve from it. Add a feature, test the changes, verify there are no regressions, and ship. It seemed like a simple idea when we started.
It’s time to start putting it to the test and showing the results. We tested some of our relevance features on four public datasets. Two are product catalogs: WANDS (Wayfair, 480 judged queries) and a 1,000-query subset of Amazon’s ESCI . Two are prose: SciFact and NFCorpus from BEIR , which are science and medical abstracts. We score with nDCG@10, which asks how close the top ten results come to the best order the human labels allow. 1.0 is perfect.
Here’s the scoreboard, losers included.
Each change was measured against the setup it would ship into, so the gains don’t add up across rows.
Hybrid search won everywhere
Hybrid search runs a keyword query and a semantic query, scales both scores to the same range, and blends them. It lifted all four datasets:
That’s the change I’d make first on almost any index. But even the winner didn’t have one right setting. The best weight on the vector side was 0.5 on SciFact, 0.6 on WANDS, and 0.7 to 0.8 on ESCI and NFCorpus. A single global value leaves points on the table for most indexes.
Future work here will use a dynamic weighting similar to what Daniel Wrigley showcased back at Haystack in 2025
A learned reranker won three of four
Next we trained a LambdaMART model for each dataset that reorders the top 30 hybrid results. It uses 20 features, like which fields matched and how many query words each result covers.
| Dataset | Hybrid | Reranked | Change | p |
|---|---|---|---|---|
| WANDS | 0.7575 | 0.7702 | +0.0127 | 0.002 |
| ESCI | 0.5549 | 0.5706 | +0.0157 | < 0.001 |
| SciFact | 0.7295 | 0.7575 | +0.0279 | 0.004 |
| NFCorpus | 0.3856 | 0.3777 | -0.0079 | 0.056 |
p is the p-value of a paired bootstrap test on per-query nDCG@10. The Hybrid column comes from the reranker’s own run, which used a weight of 0.6 on every dataset. The chart above uses each dataset’s best weight, so the two differ slightly.
On NFCorpus it made things slightly worse, so that index just doesn’t get one. The cost was one extra query on the first page: 8 to 17 ms at the median and 14 to 30 ms at p95 on a single local node.
Two things surprised us. A shared model trained on three datasets and tested on the fourth gained nothing on three of them, so each index needs its own model. And the amount of labels mattered more than we expected. 302 judged ESCI queries gave +0.001, while 1,000 gave +0.016.
Catalogs and prose want different knobs
Boosting the name field x2 is the clearest split. It added 0.035 on ESCI and 0.010 on WANDS, where people type product names. It cost 0.015 on SciFact and 0.012 on NFCorpus, where a title is just one sentence of the paper. So when a new Relevan index reaches 1,000 documents, it gets that boost only if its documents look like a catalog.
Searching facet text, like a product type such as “shower doors”, added 0.019 on WANDS because shoppers type those types. The same idea on ESCI’s color field added 0.001. People don’t search for colors there.
Tuning BM25’s k1 and b (how fast repeated words stop adding score, and how much document length counts) was the knob I expected the most from (thanks Jo Bergum). The best grid cell gained under 0.01 on every dataset, and we picked that cell with the labels. A rule that runs without labels would get only part of it.
Zero results are their own metric
nDCG can’t tell a correct empty result from a failed one. A query with no matching document should return nothing, and the score doesn’t know which empty pages are which. Oberdex, a Pokémon card catalog, had 27 distinct zero-result queries in 250 logged searches. Many were partial names: Bulb didn’t find Bulbasaur.
We shipped prefix and phonetic matching for their name field. Replaying the same queries against that config left 7 empty. Four more changes (fuzzy prefixes, a prefix on the last word, longer phonetic codes, and card-number matching) took it to 5. Those five should stay empty: ETB, eTB, Project, binder, and zzzznotreal have no matching document. We rejected a vector fallback for the same reason. It always returns something, which shows wrong cards and hides the signal.
What does this mean for you?
Test test test test test. We’ve learned from lots of trials that some features work great for one customer and actively hurt search for another. Just because someone you trust wrote a blog about it doesn’t mean it will work for you.
Or you can just send us the blog and we’ll build it for you. Try our free tier, or book a call.