Block 3 · The Prediction Stack, 2021-2026: What pCTR and pCVR Have Become

← Day index · Block 3 of 10 · ← Previous · Next →

Time budget

~90 minutes. Read the calibration section and its worked example first - it is the part that costs money. Then read SIM and HSTU for the two directions the ranker is being pulled in, and MTGR for the caution. Skim the delayed-feedback lineage once, and keep the multi-task, LLM-relevance and uplift sections as reference.

Why the prediction stack is the auction §

Everything the bidding layer does begins from a value estimate, and the value of an impression is

$$v=m\cdot \text{pCTR}\cdot \text{pCVR},$$

margin times the two probabilities. Under a relevance-weighted score the bid a rational advertiser submits is that value divided by the platform’s quality multiplier $q$, so $b=m\cdot \text{pCTR}\cdot \text{pCVR}/q$. Two of the four terms are model outputs, and the fourth - $q$ - is a model output the platform owns. The prediction stack is not a support function for the auction. It is the auction’s pricing engine, and every property of a model that matters elsewhere - discrimination, calibration, freshness, bias - shows up here as a dollar amount.

Amazon Ads, Deeply covers the first two eras: sparse logistic regression with FTRL and hashed features, then the interaction-aware deep networks from Wide & Deep through DIN and DIEN. This block picks up in 2020 and asks what changed. Three things did. Behavior sequences got three orders of magnitude longer. Ranking models started following scaling laws, and a generative framing appeared that threatens to dissolve the retrieve-then-rank pipeline. And the unglamorous layers - multi-task conversion structure, delayed-feedback correction, calibration - became subfields, because the systems were good enough that their residual errors were now the dominant source of mispricing.

A caveat runs through the block. Most of this literature is from Alibaba, Kuaishou, Meta, ByteDance, Meituan, Google and Walmart, and the auction relevance is an engineering inference, not evidence that Amazon uses a particular architecture. Where Amazon’s own science org has published - semantic search, ad-sequence pre-training, calibration of click-attributed purchases, position-aware click models, LLM relevance distillation - the block says so.

Section takeaway. The bid is $m\cdot \text{pCTR}\cdot \text{pCVR}/q$; three of the four terms are learned, and the properties of the learners are the properties of the prices.

(a) Long sequences: from attention over fifty items to attention over fifty thousand §

DIN’s insight was target attention: represent the user relative to the candidate ad. The limit is that attention is linear in history length, and a shopper’s history on a large marketplace is not fifty items. Alibaba’s SIM (2020) split the problem in two ^[1]: a General Search Unit retrieves a candidate-relevant sub-sequence from the full history, cheaply, and an Exact Search Unit runs target attention on what survives. Production sequences reach 54,000 events; the paper reports a 7.1% CTR and 4.4% RPM lift on Alibaba display ads ^[1]. ETA (2021) made the retrieval stage hashable: SimHash-style fingerprints, top-$K$ by Hamming distance, then attention over the $K$, taking complexity from $O(L\cdot d)$ to $O(L)+O(K\cdot d)$; the reported deployment serves about 120,000 queries per second with 1.8% CTR and 3.1% GMV lifts ^[2]. Kuaishou’s TWIN (KDD 2023) fixed the inconsistency between the two stages - the cheap retriever and the exact attender disagreed about relevance - with a consistency-preserved GSU whose item-side computation is cached ^[3]; TWIN-V2 (CIKM 2024) then compressed the lifecycle history with offline hierarchical clustering and cluster-aware target attention, deployed to hundreds of millions of daily users ^[4]. Pinterest’s TransAct (KDD 2023) and TransAct V2 (CIKM 2025) are the hybrid: a real-time transformer over the last minutes of actions fused with a batch long-term embedding, so session intent is fresh and the fallback is cheap ^[5][6].

On Amazon-shaped data the case is direct. “Mascara” is ambiguous; the session disambiguates it; the long history prices the conversion. Retrieve-then-attend is how all three reach a model that scores hundreds of candidates in the milliseconds an auction allows.

Section takeaway. Long-sequence CTR is a retrieval problem in disguise: retrieve or compress the history relative to the candidate, then attend exactly over what survives.

(b) Scaling laws and generative rankers: the pipeline itself is in question §

Recommendation models were long thought not to scale. Ardalani and coauthors’ 2022 study of DLRM-style models found parameter scaling running “out of steam” while data scaling still helped ^[7], and that framed the question the 2024 papers answered. Zhang and coauthors showed clean scaling laws for large sequential recommenders (RecSys 2024) ^[8]; Wang and coauthors showed them for online ad retrieval specifically ^[9]. Meta’s HSTU - the Hierarchical Sequential Transduction Unit behind “Actions Speak Louder than Words” (ICML 2024) - reframes recommendation as sequential transduction: the action stream is one sequence, items and actions interleaved, and the model predicts the next token. It reports up to 65.8% NDCG improvement, 5.3x to 15.2x speedups over FlashAttention2 Transformers at length 8192, and a 12.4% online metric gain for 1.5-trillion-parameter generative recommenders ^[10]. Wukong (ICML 2024) is the non-Transformer counterpoint: stacked factorization machines that hold a quality advantage across two orders of magnitude of compute ^[11]. Amazon’s own entry is Chitlangia, Kesari and Agarwal’s generative pre-training over user ad activity sequences (AdKDD 2023), which finds a power-law loss curve and shows the representation transfers to conversion prediction ^[12].

The 2025 systems push toward one model. Kuaishou’s OneRec replaces the retrieve-then-rank cascade with a single generative model plus preference alignment; the technical report puts its operating expense at 10.6% of the cascade’s while serving roughly 25% of Kuaishou’s QPS, alongside a 1.6% watch-time gain ^[13][14]. ByteDance’s RankMixer (CIKM 2025) is hardware-aware scaling that takes model FLOPs utilization from 4.5% to 45% ^[15]; LONGER (RecSys 2025) scales long-sequence transformers ^[16]. Meta’s GEM ads foundation model exists only as a November 2025 engineering blog - no paper - and describes Wukong as the non-sequence tower, distillation into vertical models, and reported gains of about 5% on Instagram conversions and 3% on Facebook Feed ^[17].

The caution comes from Meituan. MTGR (CIKM 2025) reports that dropping the hand-built cross features when moving to a generative architecture hurts so badly that “scaling cannot compensate” ^[18]. That is a specific finding about ads-shaped data - a (query, product, placement, advertiser) cross carries information that no amount of sequence modeling recovers - and it should temper any reading of the HSTU numbers as a death notice for feature engineering.

Here the auction diverges from the feed. A feed can afford a ranker that is stochastic, retrained end to end, whose only output is an order. An auction cannot, for three reasons. Determinism and auditability: advertisers are charged based on the score, and Amazon’s guidance and its 2026 FTC response both describe a ranking that weighs relevance and bid, with the dispute being exactly whether the pricing layer was what advertisers were told ^[19][20]; an unreproducible score is a legal liability before it is a modeling choice. Calibration, not ordering: a generative ranker is trained to order, but pCTR feeds $q$ and pCVR feeds value, and a model that improves NDCG 65% while shifting its probability scale 10% has repriced every auction by 10%. Constraints live outside the model: eligibility, reserves, pacing and placement rules are applied after the score, and a unified generator must somehow be told that some candidates it would emit are not for sale.

Section takeaway. Ranking models now scale and generative transduction may absorb the cascade - but ads crosses still matter (MTGR), and an auction needs calibrated, reproducible, constraint-aware probabilities that a pure ranker does not promise.

(c) Conversion is a multi-task problem, and the tasks are sequential §

ESMM fixed sample-selection bias by estimating pCTR and pCTCVR over the entire impression space and reading pCVR off as the ratio. The next years elaborated the funnel, the sharing structure, and then proved ESMM itself is biased.

ESM2 (SIGIR 2020) inserts post-click behaviors - add-to-cart, wishlist - between click and purchase, so the sparse purchase label gets supervision from dense intermediate labels ^[21]; for an hourly repricer that is what makes tail keywords estimable. MMoE (KDD 2018) gives each task its own gate over shared experts ^[22]; PLE (RecSys 2020) names the seesaw - one task improves as another degrades - and separates shared from task-specific experts progressively ^[23]; AITM (KDD 2021) transfers information from the earlier task to the later one with an attention gate, respecting funnel order ^[24]. ESCM² (SIGIR 2022) proves what practitioners suspected: the entire-space estimator carries an inherent estimation bias and over-estimates post-click CVR, and inverse-propensity and doubly robust weighting recover it ^[25]. ECAD (CIKM 2023) extends the cascade to effective conversions net of refunds, with cascaded delayed feedback ^[26] - and refunds are a margin term in $m$ that most bidders ignore.

Two lines are explicitly about sponsored-product auctions. Google’s online multi-task framework for Feed Ads auction models (KDD 2022) jointly models several auction-facing tasks, trains continuously, handles label delay with multi-stage training, uses MMoE, and learns task weights automatically, reporting quality gains and resource savings over per-task models ^[27]. Walmart’s PACC (SIGIR 2023) jointly models click and conversion for sponsored products with position-bias mitigation built in ^[28], and Xue and coauthors’ practical lessons on optimizing Walmart sponsored products (2023) is the closest public account of a retail-search ads stack end to end ^[29].

Section takeaway. pCVR is a funnel of sequential tasks with learned sharing and counterfactual correction; the platform-side analogue is a continuously trained multi-task auction model, and Walmart’s papers are the nearest public retail example.

(d) Delayed feedback, 2014-2026: the label you have is not the label you want §

Amazon attributes Sponsored Products sales up to 14 days after the click; the repricer runs hourly. Every update trains on clicks whose conversions are mostly not in yet, and a model that reads “no conversion so far” as “no conversion” learns a pCVR biased low, worst on the freshest data. Chapelle (KDD 2014) modeled the delay distribution jointly with the conversion ^[30]. Ktena and coauthors (RecSys 2019) kept a continuously trained neural model unbiased with importance-weighted duplicated samples and reported a 55% RPMq online gain - the headline number of the lineage ^[31]. FSIW (WWW 2020) corrected the feedback shift with importance weights ^[32]. ES-DFM (Yang, Li, Han, Zhuang and coauthors, AAAI 2021) derived instance-level weights from elapsed time since click ^[33]; DEFER (KDD 2021) argued that fake negatives distort the feature distribution and ingested duplicated real negatives instead, with more than 6% CVR gains in Alibaba display deployments ^[34]; DEFUSE (WWW 2022) gave the asymptotically unbiased label-correction version ^[35]; ULC (KDD 2023) unified the label-correction family ^[36]. GDFM (NeurIPS 2022) treats post-click behaviors and early conversions as one class of stochastic post-click information ^[37]; MISS (AAAI 2024) screens across multiple intervals ^[38]; IF-DFM (2025) uses influence functions to approximate retraining in 14.8 seconds, 1.1% of a full retrain, with venue unverified ^[39]; Luo and coauthors (WWW 2026) extend cascaded delay modeling to net CVR after refunds ^[40].

The bidding consequence in one sentence: if pCVR for a keyword is biased low by 20% for 48 hours after a bid change, the repricer cuts the bid it just raised - a control loop oscillating on its own measurement lag. Delay correction is what keeps the hourly loop stable.

Section takeaway. A CVR trained on immature labels is biased low on exactly the data an hourly repricer weights most; elapsed-time weighting, real-negative ingestion, label correction and post-click signals are how that bias is removed.

(e) Calibration is money: a worked example §

Ranking metrics ask whether the model orders correctly. The auction multiplies. Bergemann, Dütting, Paes Leme and Zuo’s “Calibrated Click-Through Auctions” (WWW 2022) gives the mechanism-design reason: an auction that prices from predicted click probabilities is only well-defined - only truthful in the sense Block 1 cares about - if those predictions are calibrated ^[41]. Yan, Qin, Wang, Bendersky and Najork (KDD 2022, on Google sponsored-search data) explain why deep rankers fail that test: learning-to-rank losses are translation-invariant, so the scores they produce carry no probability scale at all ^[42]. Sheng and coauthors’ JRC (KDD 2023) shows the obvious fix is worse than it looks - adding a ranking loss to log-loss destroys the meaning of the probabilities - and proposes joint ranking-and-calibration training ^[43]; SBCR (KDD 2024) is the self-boosted follow-up ^[44]. Fan, Si and Zhang’s “Calibration Matters” (ICLR 2023) identifies maximization bias: the auction selects the highest predicted score, so the selected prediction is an overestimate even when the model is unbiased on average, and a variance-adjusting debias corrects it ^[45].

The remaining papers are about where calibration fails. Field-aware calibration (WWW 2020) shows a globally calibrated model can be badly miscalibrated within a field - placement, device, advertiser - which is exactly the field along which the auction slices ^[46]; MBCT (WWW 2022), DESC (2024) and ConfCalib (2024, venue unverified) are the multi-field, shape-aware, confidence-aware successors ^[47][48][49]. Deng, Wang, Tan, Xu and Gai (ECML-PKDD 2020) calibrate and delay-correct jointly, since a calibrator fit on immature labels is itself biased ^[50]; Kweon, Kang and Yu (AAAI 2022) show calibrators must also be debiased for exposure ^[51]. Amazon’s own work is older and specific: Chaudhuri, Bagherjeiran and Liu ranked and calibrated click-attributed purchases in display (KDD 2017) ^[52], and Karra, Zhao, Murray and Pellegrini nudged neural click models to attend to position (CIKM 2023) so the calibration is not absorbing a position artifact ^[53].

Here is why a within-field error is a pricing error. In a relevance-weighted GSP with $s_i=b_i\cdot q_i$ and $q_i={\text{pCTR}}_i$, the winner pays

$$p_1=\frac{b_2{q}_2}{q_1}.$$

Suppose the platform’s model overestimates pCTR by 10% in one placement field, so $\hat{q}=1.1q$ there, and the runner-up sits in a calibrated field. The winner’s rank is inflated and its price deflated:

$${\hat{p}}_1=\frac{b_2{q}_2}{1.1q_1}\approx 0.91p_1.$$

The platform undercharges that field about 9% per click on ads it wrongly promotes. If the advertiser’s own model shares the miscalibration, its value-based bid $b=m\cdot \widehat{\text{pCTR}}\cdot \text{pCVR}/q$ is 10% above true willingness to pay. Flip the sign and the advertiser bids 10% low, loses auctions it wanted, and the platform overcharges winners in that field by $1/0.9-1\approx 11\%$. The error is not a few points of log-loss. It is a per-field tax or subsidy of roughly the miscalibration percentage on every auction in the field - and because the auction selects the maximum, maximization bias compounds it ^[45]. This is why the operating metrics are log-loss and field-level calibration error, why a post-hoc calibrator follows every ranker change, and why a 65% NDCG gain means nothing for pricing until someone shows per-field calibration.

Section takeaway. A 10% within-field miscalibration is a ~10% within-field mispricing on both bid and charge; calibration is a mechanism-design requirement, not modeling hygiene.

(f) Position bias: the labels are poisoned by the slot §

Top-of-search gets clicked because it is top-of-search. Joachims, Swaminathan and Schnabel (WSDM 2017) treat clicks as biased feedback under an examination model and learn with inverse-propensity weights ^[54]; the WSDM 2018 personal-search paper compares the randomization schemes - RandTopN, RandPair - that produce the propensities ^[55]; Amazon’s bandit learning-to-rank in the position-based model (CIKM 2020) is covered in the deep note ^[56], and its counterfactual-evaluation line is in Block 7. For the bidder the point is that $q$ is estimated from the same poisoned clicks; a platform that does not de-bias $q$ rewards whoever bought the top slot yesterday with a higher multiplier today.

Section takeaway. Clicks encode position as much as relevance; propensity-weighted learning is the price of a $q$ that means what it claims.

(g) LLMs for relevance: teachers and labelers, not yet the live ranker §

Amazon’s semantic product search (KDD 2019) brought dense retrieval to the catalog ^[57]. CTR-BERT (NeurIPS 2021 ENLSP workshop) is the ads-side template: a 70-million-parameter twin-encoder student distilled from a 1.5-billion teacher, serving under 5 milliseconds with a 2% online CTR gain ^[58]. The PAKDD 2023 web-scale paper distills a 75-million-parameter model that improves relevance up to 23% over a same-latency DSSM while giving up 3% against a teacher 20 times larger ^[59]. Agrawal, Ahemad and Sembium’s rationale-guided distillation (COLING 2025, organic search team) gets a 110-million cross-encoder within 1% of 7-billion-parameter LLMs at 50 times lower latency ^[60]. The 2025 relevance-labeling paper moves LLMs upstream to generate query-product labels for evaluation ^[61], and Saha, Zaman, Zou, Chen, Shu, Vase and Bagherjeiran’s keyword expansion (Amazon Ads, 2025) grows the advertiser’s targeting set with cluster-adaptive expansion and relevance tuning ^[62]. Outside Amazon, ClickPrompt (WWW 2024) and CTRL (venue unverified) couple LLMs to CTR models ^[63][64], Lin and coauthors survey the field ^[65], and Walmart’s INSPIRE (SIGIR eCom 2026) is the retail-search counterpart ^[66].

The pattern is uniform: the LLM is the teacher or the labeler, and the thing in the serving path is a distilled model two orders of magnitude smaller. No public paper shows an LLM as the live Sponsored Products ranker. The reasons are section (b)‘s three plus one: a labeler’s biases are as real as a ranker’s, and position leakage into a prompt is still position leakage.

Section takeaway. LLMs are in the ads relevance stack as teachers and labelers; an LLM in the auction-time scoring path is an open engineering problem.

(h) Incrementality as a third factor in the bid §

pCVR estimates attributed conversion. The advertiser wants caused conversion. Uplift models estimate the treatment-control difference directly - Moraes and coauthors’ Booking tutorial and Zhang, Li and Liu’s CATE/uplift survey are the entry points ^[67][68] - and Ke and coauthors (ICDM 2021) show uplift models for ads inherit the exposure bias of section (f) ^[69]. The gap between attributed and incremental is large: Gordon, Moakler and Zettelmeyer’s “Close Enough?” (Marketing Science 2023) finds RCT lifts of 29%, 18% and 5% where double machine learning on observational data reports 83%, 58% and 24% ^[70]. Lewis and Wong’s “Incrementality Bidding and Attribution” (2022) draws the conclusion for the bidder: the bid should be $b=m\cdot \text{pCTR}\cdot \text{pCVR}\cdot \iota /q$, where $\iota$ is an incrementality multiplier, and pricing from attribution alone overvalues bottom-funnel brand terms and undervalues discovery ^[71]. Lifetime value is the other direction of the same correction ^[72]. Both inherit every problem in this block - delay is worse for LTV, and $\iota$ needs the experiments in Block 7.

Section takeaway. The bid has a third learned factor, incrementality, and it is the hardest one to measure.

Assembling the stack §

Read as a system: retrieve-then-attend (SIM, ETA, TWIN) brings the history into scope; a scaling-law ranker (HSTU, Wukong, RankMixer) produces scores, with crosses kept (MTGR); a counterfactual multi-task funnel (ESM2, PLE, AITM, ESCM²) yields pCTR and pCVR over the impression space; delay correction (Ktena, DEFER, DEFUSE) keeps fresh estimates unbiased; position de-biasing keeps $q$ honest; joint ranking-and-calibration with field-aware post-hoc correction (JRC, MBCT) turns everything into probabilities the auction can multiply; an incrementality multiplier reweights toward caused value. Then $b=m\cdot \text{pCTR}\cdot \text{pCVR}\cdot \iota /q$. The Bayesian conversion widget in The Ad Auction, From the Inside is the smallest version of the middle of that pipeline; Block 4 is what happens once these estimates become a policy.

Reading list for this block §

1. Fan, Si, Zhang - Calibration Matters - ICLR 2023 - 15 min - maximization bias is the single most important idea for anyone who prices from predictions.
2. Yan et al. - Scale Calibration of Deep Ranking Models - KDD 2022 - 10 min - why LTR scores carry no probability scale; then skim JRC for the joint fix.
3. Bergemann et al. - Calibrated Click-Through Auctions - WWW 2022 - 10 min - the mechanism-design reason calibration is not optional.
4. Qi et al. - SIM - 2020 - 10 min - the retrieval-then-attend template.
5. Zhai et al. - HSTU - ICML 2024 - 15 min - the reformulation and scaling section; then MTGR’s caution, 5 min.
6. Wang et al. - ESCM² - SIGIR 2022 - 10 min - the proof that ESMM over-estimates.
7. Ktena et al. - Addressing Delayed Feedback - RecSys 2019 - 10 min - the 55% RPMq result and the duplicated-sample scheme; skim DEFER.
8. Gordon, Moakler, Zettelmeyer - Close Enough? - Marketing Science 2023 - 10 min - the attributed-versus-incremental gap in numbers.
9. Reference only: everything else in the references.

Questions to carry forward §

• If the platform’s $q$ and the advertiser’s pCTR come from different models with different miscalibrations, who captures the surplus, and can either side detect it from logs?
• Generative rankers optimize ordering; auctions consume probabilities. Is a post-hoc calibrator enough, or does auction safety require joint ranking-and-calibration training in the base model?
• MTGR says crosses cannot be scaled away. Which (query, product, placement, advertiser) crosses are the irreducible ones for a sponsored-search ranker?
• Every delay correction is a model. How would you detect that the delay model has drifted after a Prime Day before the bids do?
• Incrementality-priced and attribution-priced bids disagree most on discovery keywords. Which side of that disagreement does the mechanism reward?

References §

1. Qi et al. (Alibaba). Search-based User Interest Modeling with Lifelong Sequential Behavior Data for CTR Prediction. 2020. arXiv:2006.05639
2. Chen et al. (Alibaba). Efficient Long Sequential User Data Modeling for CTR Prediction (ETA). 2021. arXiv:2108.04468
3. Chang et al. (Kuaishou). TWIN: Two-stage Interest Network for Lifelong User Behavior Modeling. KDD 2023. arXiv:2302.02352
4. Si et al. (Kuaishou). TWIN V2. CIKM 2024. arXiv:2407.16357
5. Xia et al. (Pinterest). TransAct. KDD 2023. doi.org/10.1145/3580305.3599918
6. Xia et al. (Pinterest). TransAct V2. CIKM 2025. arXiv:2506.02267
7. Ardalani et al. (Meta). Understanding Scaling Laws for Recommendation Models. 2022. arXiv:2208.08489
8. Zhang et al. Scaling Law of Large Sequential Recommendation Models. RecSys 2024. arXiv:2311.11351
9. Wang et al. Scaling Laws for Online Advertisement Retrieval. 2024/25. arXiv:2411.13322
10. Zhai et al. (Meta). Actions Speak Louder than Words (HSTU). ICML 2024. arXiv:2402.17152
11. Zhang et al. (Meta). Wukong: Towards a Scaling Law for Large-Scale Recommendation. ICML 2024. proceedings.mlr.press
12. Chitlangia, Kesari, Agarwal (Amazon). Scaling Generative Pre-training for User Ad Activity Sequences. AdKDD 2023. amazon.science
13. Deng et al. (Kuaishou). OneRec: Unifying Retrieve and Rank with Generative Recommender and Preference Alignment. 2025. arXiv:2502.18965
14. Kuaishou. OneRec Technical Report. 2025. arXiv:2506.13695
15. Zhu et al. (ByteDance). RankMixer: Scaling Up Ranking Models in Industrial Recommenders. CIKM 2025. arXiv:2507.15551
16. ByteDance. LONGER. RecSys 2025. arXiv:2505.04421
17. Meta Engineering. Meta’s Generative Ads Model (GEM). Blog, Nov 2025 (no paper). engineering.fb.com
18. Meituan. MTGR. CIKM 2025. arXiv:2505.18654
19. Amazon Ads. Sponsored Products across retailers (auction guidance). advertising.amazon.com
20. Amazon. Response to the FTC’s Lawsuit Regarding Sponsored Ads. Aug 2026. aboutamazon.com
21. Wen et al. (Alibaba). ESM2: Entire Space Multi-Task Modeling via Post-Click Behavior Decomposition. SIGIR 2020. arXiv:1910.07099
22. Ma et al. (Google). MMoE. KDD 2018. doi.org/10.1145/3219819.3220007
23. Tang et al. (Tencent). PLE. RecSys 2020. doi.org/10.1145/3383313.3412236
24. Xi et al. AITM. KDD 2021. arXiv:2105.08489
25. Wang et al. (Alibaba). ESCM²: Entire Space Counterfactual Multi-Task Model. SIGIR 2022. arXiv:2204.05125
26. ECAD: Effective Conversion Rate with Cascaded Delayed Feedback. CIKM 2023. arXiv:2308.04768
27. Ma et al. (Google). An Online Multi-task Learning Framework for Google Feed Ads Auction Models. KDD 2022. doi.org/10.1145/3534678.3539055
28. Wang et al. (Walmart). PACC: Joint Click-Conversion Modeling for Sponsored Products. SIGIR 2023. arXiv:2307.16060
29. Xue et al. (Walmart). Practical Lessons on Optimizing Sponsored Products. 2023. arXiv:2304.09107
30. Chapelle (Criteo). Modeling Delayed Feedback in Display Advertising. KDD 2014. doi.org/10.1145/2623330.2623634
31. Ktena et al. (Twitter). Addressing Delayed Feedback for Continuous Training with Neural Networks in CTR Prediction. RecSys 2019. arXiv:1907.06558
32. Yasui et al. A Feedback Shift Correction in Predicting Conversion Rates under Delayed Feedback (FSIW). WWW 2020. arXiv:2002.02068
33. Yang, Li, Han, Zhuang et al. ES-DFM. AAAI 2021. arXiv:2012.03245
34. Gu et al. (Alibaba). Real Negatives Matter (DEFER). KDD 2021. arXiv:2104.14121
35. Chen et al. Asymptotically Unbiased Estimation for Delayed Feedback Modeling via Label Correction (DEFUSE). WWW 2022. arXiv:2202.06472
36. Wang et al. ULC: Unified Label Correction for Delayed Feedback. KDD 2023. arXiv:2307.12756
37. Yang, Zhan. Generalized Delayed Feedback Model with Post-Click Information (GDFM). NeurIPS 2022. arXiv:2206.00407
38. Liu et al. MISS: Multi-Interval Screening and Synthesizing under Delayed Feedback. AAAI 2024. doi.org/10.1609/aaai.v38i8.28726
39. Ding et al. IF-DFM: Delayed Feedback via Influence Functions. 2025 (venue unverified). arXiv:2502.01669
40. Luo et al. Cascaded Delayed Feedback Modeling for Net Conversion Rate. WWW 2026. arXiv:2601.19965
41. Bergemann, Dütting, Paes Leme, Zuo. Calibrated Click-Through Auctions. WWW 2022. arXiv:2105.09375
42. Yan, Qin, Wang, Bendersky, Najork (Google). Scale Calibration of Deep Ranking Models. KDD 2022. doi.org/10.1145/3534678.3539072
43. Sheng et al. (Alibaba). Joint Optimization of Ranking and Calibration (JRC). KDD 2023. arXiv:2208.06164
44. Self-Boosted Calibrated Ranking (SBCR). KDD 2024. arXiv:2406.08010
45. Fan, Si, Zhang. Calibration Matters: Tackling Maximization Bias in Large-scale Advertising Recommendation Systems. ICLR 2023. arXiv:2205.09809
46. Pan et al. Field-aware Calibration. WWW 2020. arXiv:1905.10713
47. Huang et al. MBCT: Tree-Based Feature-Aware Binning for Calibration. WWW 2022. arXiv:2202.04348
48. Yang et al. Deep Ensemble Shape Calibration (DESC). 2024. arXiv:2401.09507
49. Zhao et al. Confidence-Aware Multi-Field Model Calibration (ConfCalib). 2024 (venue unverified). arXiv:2402.17655
50. Deng, Wang, Tan, Xu, Gai (Alibaba). Calibrating User Response Predictions in Online Advertising. ECML-PKDD 2020. doi.org/10.1007/978-3-030-67667-4_13
51. Kweon, Kang, Yu. Obtaining Calibrated Probabilities with Personalized Ranking Models. AAAI 2022. arXiv:2112.07428
52. Chaudhuri, Bagherjeiran, Liu (Amazon). Ranking and Calibrating Click-Attributed Purchases in Performance Display Advertising. KDD 2017. amazon.science
53. Karra, Zhao, Murray, Pellegrini (Amazon). Nudging Neural Click Prediction Models to Pay Attention to Position. CIKM 2023. amazon.science
54. Joachims, Swaminathan, Schnabel. Unbiased Learning-to-Rank with Biased Feedback. WSDM 2017. doi.org/10.1145/3018661.3018699
55. Wang et al. Position Bias Estimation for Unbiased Learning to Rank in Personal Search. WSDM 2018. doi.org/10.1145/3159652.3159732
56. Amazon. Learning to Rank in the Position-Based Model with Bandit Feedback. CIKM 2020. arXiv:2004.13106
57. Nigam et al. (Amazon). Semantic Product Search. KDD 2019. amazon.science
58. Muhamed et al. (Amazon). CTR-BERT: Cost-effective Knowledge Distillation for Billion-parameter Teacher Models. NeurIPS 2021 ENLSP workshop. neurips2021-nlp.github.io
59. Muhamed et al. (Amazon). Web-scale Semantic Product Search with Large Language Models. PAKDD 2023. amazon.science
60. Agrawal, Ahemad, Sembium (Amazon). Rationale-Guided Distillation for E-Commerce Relevance Classification. COLING 2025. amazon.science
61. Amazon. Automated Query-Product Relevance Labeling using LLMs for E-commerce Search. 2025. arXiv:2502.15990
62. Saha, Zaman, Zou, Chen, Shu, Vase, Bagherjeiran (Amazon Ads). Improving Ad Matching via Cluster-Adaptive Keyword Expansion and Relevance Tuning. 2025. arXiv:2505.18897
63. ClickPrompt. WWW 2024. arXiv:2310.09234
64. CTRL: Connect Collaborative and Language Model for CTR Prediction. 2023 (venue unverified). arXiv:2306.02841
65. Lin et al. How Can Recommender Systems Benefit from Large Language Models: A Survey. TOIS. arXiv:2306.05817
66. Walmart. INSPIRE. SIGIR eCom 2026. arXiv:2606.23889
67. Moraes et al. (Booking). Uplift Modeling: From Causal Inference to Personalization. 2023. arXiv:2308.09066
68. Zhang, Li, Liu. A Unified Survey of Treatment Effect Heterogeneity Modelling and Uplift Modelling. ACM CSUR. arXiv:2007.12769
69. Ke et al. Addressing Exposure Bias in Uplift Modeling for Large-scale Online Advertising. ICDM 2021. doi.org/10.1109/icdm51629.2021.00138
70. Gordon, Moakler, Zettelmeyer. Close Enough? A Large-Scale Exploration of Non-Experimental Approaches to Advertising Measurement. Marketing Science 2023. arXiv:2201.07055
71. Lewis, Wong. Incrementality Bidding and Attribution. 2022. arXiv:2208.12809
72. Wu, Jia, Dong, Tang. Customer Lifetime Value Prediction: Towards the Paradigm Shift of Recommender System Objectives. RecSys 2023. doi.org/10.1145/3604915.3609499

---

← Day index · ← Block 2 · Block 4 → · Parent note: Amazon Ads, Deeply