Query Understanding

Users type 'iphone 15 pric', 'best headfones under 100', and 'where to buy nike shos cheap'. A retrieval system that takes queries literally returns zero results or completely irrelevant documents for all three. Query understanding is the layer that transforms messy human language into structured retrieval signals - it is the difference between a search engine and a text-matching system.

• **Google** processes 15% of daily queries it has never seen before (2023). Query understanding - including spell correction, synonym expansion, and entity disambiguation - handles this by mapping novel surface forms to known concepts. Without it, every new product release would have near-zero results until the index was updated.
• **Amazon** applies intent classification at 200+ granularities per query. 'Batteries' classified as 'replenishment' (AA/AAA batteries for remotes) returns different results than 'batteries' classified as 'accessory' (phone battery packs). This routing alone contributes $1B+ in annual GMV according to Amazon's 2022 re:Invent talk.
• **LinkedIn Search** uses entity linking to normalize company names: 'Google', 'Alphabet', 'GOOGL', and 'Google Inc.' all resolve to the same company entity, ensuring that a recruiter searching 'Google engineers' finds profiles mentioning any form of the name across 930 million member profiles.

Spell Correction and Normalization

Search queries contain 10-15% misspelled terms (Google's internal data). Spell correction operates in two stages: candidate generation and candidate selection. Candidates are generated by edit distance (Levenshtein <= 2) or phonetic matching (Soundex, Metaphone). Selection ranks candidates by a noisy-channel model: P(correction | query) ∝ P(query | correction) * P(correction), where P(correction) is the language model unigram/bigram probability from query logs and P(query | correction) is the confusion matrix probability of the observed typo given the intended word.

**Context-sensitive correction** (used by Google since 2011) uses the surrounding words to disambiguate. 'teh' → 'the' in isolation, but 'I was shocked by teh results' correctly corrects to 'the' because 'results' and 'shocked' constrain the language model. Modern systems use BERT-based models (SpellBERT, MLM masking) that attend to full context, achieving <1% error rate on real query logs.

Query Expansion

Query expansion adds or substitutes terms to improve recall for queries that are too specific or use non-standard vocabulary. Approaches: (1) thesaurus-based - add synonyms from WordNet or a domain lexicon; (2) pseudo-relevance feedback (PRF) - retrieve top-k documents, extract frequent terms from them, add to query; (3) embedding-based - find lexically different terms close in embedding space. Bing's query expansion (2019 disclosure) expanded 40% of queries, improving click-through rate by 6% on long-tail queries.

**Query drift** occurs when expansion terms change the query meaning. 'Apple' expanded with 'fruit, orchard, cider' would be catastrophic for device searches. Production systems use selective expansion: only expand low-frequency queries (head queries already return good results), gate expansions by query-term co-occurrence probability in click logs, and apply strict topical coherence filters. Google's Multitask Unified Model (MUM) uses semantic understanding to avoid drift.

Intent Classification

Intent classification determines what action the user wants to take with a query. The classic taxonomy (Broder 2002): navigational (go to a specific site: 'youtube'), informational (learn something: 'how does HNSW work'), and transactional (do something: 'buy iPhone 15'). Modern systems use fine-grained multi-label intent (40+ categories at Google, 200+ at Amazon). Intent routes the query to different ranking pipelines: navigational → direct URL redirect; transactional → product carousel + ads; informational → web results + Knowledge Panel.

Named Entity Recognition and Linking

Entity linking identifies named entities in a query (people, places, products, companies) and maps them to entries in a knowledge base (Wikipedia, Wikidata, Google KG). The pipeline: (1) NER detects entity spans ('Apple' in 'Apple quarterly earnings'); (2) candidate generation retrieves KB entries for the surface form; (3) entity disambiguation selects the correct entry using context (Apple Inc. vs apple the fruit). Google processes 2 trillion queries per year with entity linking active on ~30% (Orr et al., 2021), enabling direct Knowledge Panel answers.

**Entity linking enables structured answers**: once 'Apple' is linked to Q312 (Apple Inc.) in Wikidata, the search engine can directly answer 'Apple CEO' by traversing the CEO relation in the knowledge graph rather than retrieving web documents. This powers Google's 'One Box' features - 40% of searches return a direct answer without a click (2023 data).

Key Ideas

• **Spell correction** uses the noisy-channel model (P(correction) * P(query | correction)) to select the most probable intended word. Context-sensitive BERT-based correctors outperform edit-distance heuristics by using surrounding query terms.
• **Query expansion** adds related terms to improve recall, but risks query drift. Production systems apply expansion selectively: only on low-frequency queries, gated by co-occurrence probability in click logs.
• **Intent classification and entity linking** route queries to the right pipeline and enable structured answers. They are evaluated end-to-end in the full serving system because errors in one module propagate and amplify in downstream components.

Related Topics

Query understanding is the entry point to the retrieval pipeline:

• Vector Databases — After query understanding, the processed query is embedded into a vector and sent to the ANN index for dense retrieval. Entity linking can produce multiple query vectors (one per detected entity) for a multi-vector search strategy.
• Learning to Rank — Intent classification determines which ranking model is applied. A transactional query uses a revenue-optimized ranker; an informational query uses a quality-and-freshness ranker. The same candidate set is ranked differently based on intent signals.

Вопросы для размышления

• A user searches 'python' on a programming platform. Design a full query understanding pipeline: which intents are possible, how would spell correction interact with the programming language name, and how would entity linking disambiguate from python-the-snake? What signals would be most useful?
• Pseudo-relevance feedback can cause a negative feedback loop: if the initial retrieval is poor, the extracted expansion terms are irrelevant, further degrading recall. How would a production system detect and break this loop?
• Google's Multitask Unified Model (MUM) processes queries in 75 languages simultaneously. How does this affect the query expansion design - specifically, can expansion terms from one language's knowledge base be used to improve results in another language's index?

Связанные уроки

• ml-01-intro