How well does product search handle Hinglish — code-mixed
Hindi–English queries like lal running shoes and
gym ke liye shoes? A reproducible benchmark on Amazon ESCI comparing
BM25, multilingual dense retrieval, and reranking.
On code-mixed queries, lexical BM25 with reranking beats multilingual dense retrieval (nDCG@10 0.30 vs 0.16) — the opposite of the naive expectation. Dense is the best method on clean English but the worst on Hinglish.
nDCG@10 for each method, split by query language. Every method drops sharply from English to Hinglish — and dense retrieval collapses hardest.
Colour intensity encodes nDCG@10 — vivid = strong, faded = near-zero.
Dense craters on roman_hindi, voice_search, and
pure_hindi while staying strong on English-shaped categories.
| Category | n | BM25 | BM25+CE | Dense | Dense+CE |
|---|
pure_hindi) scores 0.00 on BM25 — zero character overlap with the
Latin-script catalog.348 hand-authored Hinglish queries (10 domains, 13 categories) are mapped — via exact + fuzzy string matching, then a multilingual-embedding (E5) semantic fallback — onto the US subset of Amazon ESCI (1.8M graded judgments), inheriting Exact/Substitute/Complement/Irrelevant labels as graded relevance. All 348 queries map to an answer key, each with a recorded confidence tier. Three systems are compared: BM25 (from scratch), a multilingual E5 bi-encoder with FAISS, and a cross-encoder / LLM reranker. Metrics (nDCG, MRR) are implemented from scratch. Everything is reproducible from a fixed seed, pinned dependencies, and numbered scripts, guarded by a unit-test suite.
Honest limitations: answer keys are product-level (fine colour/size/price constraints aren't separately scored); ~24% of mapped keys are fuzzy; Indian-vernacular items are under-represented in the US catalog; the dense model is CPU-sized (e5-small). See the technical report for full detail.