Teacher/student CTR pipeline with LLM-derived features
Click any box for details. This ships in phases, not as one build. Phase 1 (query type, ESCI tag, item-semantic embedding) ships first, first real experiment. Phase 1.5 (untrained user embedding) is the cheapest next step, no CTR training, tried before spending anything on a teacher. Phase 2 (teacher) is the third experiment, offline only, and compares against Phase 1.5 to decide whether Phase 3 is worth it. Phase 3 (student towers) only starts if that gap is large. See the implementation roadmap for the full phased plan and timing.
Click a box above
Phase 5 isn't shown as new boxes here. Once this pipeline exists for pCTR, the same query type, ESCI tag, and item embedding feed a second teacher and student pair fine-tuned on conversions instead of clicks, for pCVR, and the whole pipeline repeats per market, each needing its own run. This is a second pass through this same diagram with a different objective, not a new step.
Frontier LLM consensus, ground truth, not production. A frontier LLM, or an ensemble with consensus labeling, establishes ground truth for two things: the ESCI classification of query-item pairs, and the structure the item-semantic embedding should preserve (near-duplicates cluster together, substitutes sit closer than irrelevant items). Expensive per call, runs periodically, never touches the serving path.
Query type, distilled, per query. Part of the first experiment. Branded or non-branded, narrow or broad. Classifying a query-item pair as Exact, Substitute, Complement, or Irrelevant doesn't require or produce this as a byproduct, they're related but separate outputs, and this one has to be built and cached explicitly on its own. It also isn't a pCTR-only signal: retrieval and auction eligibility (the rules cube) need it independently of any CTR model, so it gets built once here and reused by all three consumers.
ESCI tag, distilled, per query-item pair. First experiment. A compact distilled model, fine-tuned on the frontier's consensus labels, produces Exact, Substitute, Complement, or Irrelevant for every head and torso query-item pair, cached. Same-day refresh for new pairs, tied to the existing Secchi tagging trigger. Once this exists, the first real test runs immediately: add it to current pCTR, compare against today's production model. This doesn't wait for anything else in this diagram.
Item-semantic embedding, distilled, per item. Part of the first experiment. A separate, simpler step than the ESCI tag: no fine-tuning needed here, just a plain LLM run once over the item's title, brand, category, and price, producing a vector that captures what the item is. Frontier LLM validates the structure on a sample first, then a compact distilled model generates it at catalog scale. Cheapest piece in this whole diagram to build, and it ships in the same first experiment as the ESCI tag, added to current pCTR alongside it.
User embedding, untrained. Phase 1.5, try this before the teacher. Aggregate a user's raw interaction history, fed as an ordered sequence, using the item-semantic embedding above, no CTR training, no teacher, LLM world knowledge only. Order and recency come from the LLM's own reading of the sequence, no separate sequential model needed here. Cached and usable immediately, feeds the online model directly. If adding this to Phase 1's pCTR already closes most of the gap, the teacher and student below are never built. In that outcome the whole shipped pipeline, query type through this embedding, stays fully shared across pCTR, pCVR, organic, and retrieval, since nothing in it was ever trained on CTR or CVR.
Teacher: CTR fine-tuned, never served. Phase 2, third experiment. The same compact model class validated in production, not a frontier model; fine-tuning and running a frontier model over historical logs at this volume would be prohibitively expensive. Query, item, and user history go in together; query type, the ESCI tag, and the item-semantic embedding are inputs, computed and cached before the teacher runs, not inferred by it. The teacher uses the user's raw history directly, not the Phase 1.5 embedding. Fine-tuned on real CTR outcomes. Accurate, but produces no reusable cacheable representation and can't run at auction scale. This is the go/no-go experiment: compare its AUC against Phase 1.5's result. The teacher is discarded either way, its job is answering this question, not serving. If the gap is small, stop here, the student below is never built, and the shipped pipeline has no CTR-specific component anywhere. If it's large, that's the case for Phase 3.
Item tower, student, distilled. Phase 3, gated on Phase 2. Takes the ESCI tag and item-semantic embedding as input, produces a single item vector. Trained, together with the user tower, so their dot product matches the teacher's CTR predictions on a large sample of historical pairs. This is distillation, not training from scratch on raw labels, and it recovers more of the teacher's accuracy than contrastive training alone would.
User tower, student, distilled. Phase 3, gated on Phase 2. Takes the sequence of a user's historical items, each represented through the same item tower, and aggregates them into a single user vector. Trained jointly with the item tower against the teacher's predictions. Open design decision, not yet settled: pool the item embeddings as an unordered set, cheaper to build, or run a proper sequential model over the ordered history, a transformer-style architecture, more accurate, captures recency and order effects directly, more complex to validate. This is different from the session signals shown separately below: those cover live behavior in the current session, this tower covers longer-run historical patterns across sessions. Refresh is incremental, not a full daily recompute: only users with new activity get their cached vector updated, given DAU scale.
Feature store. A low-latency key-value store the online path reads from: item vector, query type, ESCI tag, and a user vector, either the Phase 1.5 untrained embedding or, if Phase 3 was built, the student's distilled output replacing it. Wiring this store into the online model is Phase 4. Nothing upstream of the store ever runs inside the request path.
Long-tail queries, outside the head/torso cache, roughly 20% of traffic. Two fallbacks, tried in order. First, semantic query mapping: find the nearest cached head or torso query and apply its labels directly, gated by a confidence threshold, if the match isn't confident enough, don't trust it. Second, for queries with no confident mapping, a compact relevance model trained on the labeled head/torso set produces a continuous score used with a conservative threshold, admitting only the clearest matches. Organic retrieval handles whatever's left. This bounds the risk in low-coverage situations instead of leaving it open-ended.
Cached lookups. The online path never runs the teacher, the student, or any LLM. It reads the item vector, user vector, query type, and ESCI tag out of the feature store, or the tail fallback value if nothing was cached.
Session signals. Shelf position, fed to PAL, and near-real-time in-session behavior: what the user just viewed, dwelt on, or added to cart. These can't be precomputed and have to be captured within the request.
Light model. The existing fast online scorer. Computes a dot product between the cached item and user vectors as its core signal, combined with PAL for position and cross-terms for interactions with the live session signals. This is what actually runs within the latency budget.
pCTR / pCVR score. Final output, combining offline LLM-derived signal, distilled down to two cacheable vectors, with request-time freshness, produced within the single-digit millisecond budget.