Tl;dr: Search ads systems make ranking mistakes because they lack world knowledge about queries, items, and user history. This post builds the knowledge in as four discrete, reusable signals: query type, ESCI class, item-semantic embedding, and user embedding. All four come from a language model’s world knowledge, computed once offline and cached, and none require behavioral data to produce. Build them in stages: ship query type, ESCI class, and item embeddings first, since that alone is a real improvement with no new model architecture required. Add an untrained user embedding next, and measure how much of the gap it already closes before training anything on clicks. Only if a real gap remains do you build a teacher model, then distill it into cached two-tower vectors for production. Understudy Labs runs the entire ladder locally: frontier labeling, open-model screening, distillation, and held-out evaluation, with no raw data ever leaving the platform.
The first post in this series laid out why search ads systems make systematic ranking mistakes. They lack world knowledge about queries, items, and what a user’s history implies. The fix is to build this knowledge into the pipeline explicitly, as four discrete signals. This post covers what these signals are, how they connect to each other, and how to build them in stages so that value ships early and each stage tells you whether the next one is worth doing.
The four signals are query type, ESCI class per query-item pair, item-semantic embedding, and user embedding. The first two feed retrieval and eligibility decisions as well as ranking. The last two are inputs to the ranking models. None of the four requires behavioral data from your platform to produce. All four come from language model world knowledge, applied once offline and cached.
What each signal is and why it matters
Query type classifies each search as branded or non-branded, and narrow or broad. A search for a specific product by name and a broad category browse are structurally different situations, and every downstream decision in the pipeline should know which one it’s dealing with. Without this, retrieval returns the same candidate set for both. Eligibility applies the same rules regardless. Ranking models average over both contexts. All three are wrong in different ways.
ESCI class classifies each query-item pair as Exact match, Substitute, Complement, Irrelevant, or Offensive. This is the signal that makes eligibility rules explicit and position-aware. A rule that says “position one on a branded narrow query requires an Exact match” is precise and auditable. A threshold on a continuous relevance score is an approximation that drifts and is never quite right. The ESCI class also removes the relevance-model tradeoff: the same label that gates eligibility feeds the ranking model as a feature, without a separate model doing a separate job with different training data.
Item-semantic embedding is a vector representation of what an item actually is: its name, brand, category, and price, run through a language model that understands what these words mean. Two running shoes from different brands end up with similar vectors because they serve the same purpose, even if no user on your platform has ever bought both. New listings get a correct embedding the first time they are added to the catalog, before any click data exists.
User embedding is a compact representation of what a user’s history implies about them, built from the item-semantic embeddings of what they have viewed, clicked, and bought. A language model reads the sequence and produces a summary grounded in world knowledge. A user whose history includes a treadmill, running shoes, and a protein shaker is, to the model, a fitness household. This inference doesn’t require seeing what similar users bought. It requires understanding what the items are.
How they connect to each other
Query type and ESCI class are foundational. Retrieval needs query type to know which categories of items to look for. Eligibility needs both signals to decide which items can appear in which positions, and the ranking models need them for the same reason: to stop averaging over situations that are structurally different.
Item-semantic embedding is the input that makes ESCI classification efficient at scale. A language model that understands what items are can classify a query-item pair without needing to observe what users did with it. The same embedding also feeds the ranking model directly, giving it a representation of the item grounded in meaning rather than in click patterns.
User embedding is built on top of item-semantic embeddings. Because each item in a user’s history already has a meaning-based representation, the user’s history can be summarized in terms of what the items actually are rather than in terms of which item IDs co-occurred. A user who bought three things in different categories can still be understood as expressing a coherent interest if the model knows what they are.
The four signals are also independent of each other: you don’t need one to compute another. Query type is a fact about the query, and ESCI class is a fact about the pair. Item embedding describes the item; user embedding summarizes history. None of them is a prediction about behavior. All four are reusable across retrieval, eligibility, pCTR, pCVR, organic ranking, and recommendations. You build them once and every component in the pipeline can consume them.
How to build them in stages
The stages are ordered so that the first one is already a real, shippable improvement, and each subsequent stage is gated on the previous one showing a gap worth closing.
Stage 1: Query type, ESCI class, item-semantic embedding, added to the existing ranking model.
This is the first thing to build and the first thing to ship. No new model architecture is required. A frontier language model panel establishes ground truth labels on a sample, checking each other’s outputs and treating agreement as the verified standard. A compact open-weight model, fine-tuned to reproduce the standard, runs the classification at full catalog and query scale at a fraction of the cost, inside the platform’s own infrastructure. The labels are cached and served as lookups.
Add query type, ESCI class, and item-semantic embedding to the current production pCTR model as discrete features alongside whatever embeddings it uses today. Run the experiment. Compare overall ranking accuracy, and specifically the accuracy on Exact match versus Substitute, and the bias by relevance class, against the model without these features. These two cuts are the ones that matter, because they measure whether the averaging problem is actually fixed.
The cost of this stage is low. The frontier panel runs once on a representative sample. The distilled model handles the rest. The experiment tells you how much of the ranking improvement was already available without any new model training at all.
Stage 1.5: Untrained user embedding, added to Stage 1.
Before building anything that requires training on behavioral data, add the user embedding in its simplest form. Aggregate each user’s interaction history using the Stage 1 item-semantic embeddings, feed the sequence to a language model, and produce a user vector from world knowledge alone. Add it to the Stage 1 pCTR model and run the same experiment.
This stage exists to answer a specific question before committing to more expensive work: how much of what CTR training would add is already captured by semantic understanding of history? If Stage 1.5 closes most of the gap, the stages below are never needed, and the entire shipped pipeline has no component trained on clicks or conversions, which means it is fully reusable across pCTR, pCVR, organic, and recommendations without any adaptation.
Stage 2: Teacher model, offline only.
If Stage 1.5 leaves a real gap, the next step is to measure how large the gap is before deciding whether to close it. Fine-tune a compact single-tower model on real click outcome data, using the Stage 1 query type, ESCI class, and item-semantic embedding as inputs. This is the teacher. It’s accurate but slow, produces no cacheable representation, and never gets served.
The teacher’s job is to run once over historical logs and compare its ranking accuracy against Stage 1.5. The gap between them is the direct answer to how much signal CTR training adds on top of semantic understanding. If the gap is small, stop here. If the gap is large, Stage 3 is justified. Either answer is useful. Either answer is cheaper to get from a one-time teacher run than from building production infrastructure that turns out not to be needed.
Break the comparison down by ESCI class, especially Substitute, Complement, and Irrelevant. Click data on these classes is structurally sparse at any RMN or RMP, and the reason is the same problem described in the first post. Substitutes and complements rarely enter the auction under current organic retrieval and eligibility rules, so the click log has almost no coverage of what a well-calibrated outcome looks like on them. Irrelevant items have the opposite problem: when they do get through on thin-coverage queries, they appear in positions where click rate reflects position more than relevance, and that signal actively misleads the model. The teacher’s quality on minority classes is therefore bounded by how little data exists on them, not by model capacity. A teacher that performs well on Exact match but poorly on Substitute and Complement isn’t telling you the model needs to be bigger. It’s telling you they haven’t yet accumulated enough clean signal to train on. The fix is shipping the ESCI-based eligibility layer first. That gets substitutes and complements appearing in the right positions, and the click data to train on them starts to accumulate. Stage 3 should be scoped with that in mind.
Stage 3: Student towers, distilled from the teacher.
If Stage 2 shows a gap worth closing, the teacher’s predictions become training labels for a two-tower student model. An item tower takes the ESCI class and item-semantic embedding and produces a cached item vector. A user tower takes the user’s interaction history, represented through the item tower’s outputs, and produces a cached user vector. Both towers are trained so that their dot product matches the teacher’s predictions on a large sample of historical pairs. This recovers more of the teacher’s accuracy than training the two towers directly on click labels would, because the teacher’s predictions are a smoother signal than raw binary outcomes.
At serving time, the item and user vectors are read from a cache and combined with live session signals in the existing online scorer. No language model runs inside a live ad request. The expensive work happened offline. Instacart’s cascaded storefront architecture is a production instance of this same pattern: a closed-weight frontier teacher, a LoRA-fine-tuned Llama-3.2-3B student trained to near-parity, and a curated taxonomy constraint that prevents hallucination at retrieval. The mechanics are the same; the surface is personalized placement rather than search ads.
There is an open design decision in the user tower: pool the item embeddings as an unordered set, which is cheaper and faster to build, or run a sequential model over the ordered history, which captures recency and order effects but adds real complexity. This is a genuine tradeoff, not yet settled, and the right answer depends on how much the sequential signal actually improves ranking accuracy relative to pooling.
Stage 4: Production integration.
Wire the cached vectors from Stage 3 into the existing online model and run a full A/B test against current production. Define rollback criteria before launch, and slice the results by relevance class as well as overall averages, because a model that improves on Exact match while regressing on Substitute is not an unconditional win.
Stage 5: Extend to other objectives and markets.
Once the shared layer exists, adding pCVR is cheap: fine-tune a new teacher on conversion outcomes rather than click outcomes, reusing the same query type, ESCI class, and item-semantic embedding. The item and user towers can either be shared across objectives or kept separate, a multi-task versus single-task tradeoff that depends on how often CTR and CVR point in different directions on the platform.
Each additional market needs its own tagging and embeddings, plus a freshly fine-tuned teacher. Nothing transfers directly. But the pipeline is already built, so each new market is a re-run of existing infrastructure rather than a new build.
What Understudy does in this pipeline
The frontier panel, the open-model screening, the distilled model deployment, and the held-out evaluation at each stage are all jobs the Understudy Labs local agent harness runs. In a validation study on a real marketplace query classifier, Understudy ran the full ladder in under a day. It established frontier consensus on a 30,000-row sample, screened open-weight candidates across quality and cost, and ran prompt optimization via GEPA with no weight updates. The result was a compact open-weight model that moved balanced accuracy from 76.7 percent to 82.7 percent and macro-F1 from 72.4 percent to 81.2 percent on held-out data, before any fine-tuning. The cost reduction was roughly 85 percent compared to running Sonnet on the full table.
Everything runs locally. Raw queries, item data, and click logs never leave the platform’s infrastructure. The frontier panel calls go out to establish ground truth on a sample, but the sample is controlled and the results come back as labels, not as training data for the frontier models. The compact model that does the production work runs entirely on owned compute.
Where it can break
Negative sampling is the most common failure mode in two-tower student training. The dot product between item vector and user vector is only meaningful if the negatives in training reflect real competition. If negatives are random items rather than items that actually compete for the same auction slot, the model learns a representation that looks good offline and fails online. Getting this right requires sampling negatives not from the full item catalog, but from the actual candidate set the system considers.
Embedding staleness is the other structural risk. Cached item and user vectors drift out of sync with the online model as behavior and catalog shift between refresh cycles. Item vectors need to be refreshed at catalog update cadence. User vectors need to be refreshed incrementally as users take new actions. The gap between refresh cycles determines how much staleness the system tolerates, and this tolerance depends on how fast the platform’s catalog and user behavior actually move.
The distillation gap itself is a known limitation. The student won’t fully match the teacher, especially on unusual user-item combinations where compressing everything into two fixed vectors loses information a joint model would have kept. Some accuracy loss relative to the teacher is expected and acceptable. The question is how much, and Stage 3 is the place to measure it before committing to the serving infrastructure.
The attribution question
Once all four signals are in place, it is worth running a proper ablation: hold out each signal in turn and measure the resulting drop in ranking accuracy. A single before-and-after number doesn’t tell you which of the four did the work. That matters because it determines what to invest in maintaining and refreshing. If most of the gain came from query type and ESCI class, the item and user embeddings are cheap to maintain but not the priority to refresh at high cadence. If the user embedding is doing significant work, its refresh cadence becomes important in a way it wasn’t before.