Block 4 · Reinforcement Learning for Bidding: From MDPs to Generative Policies

← Day index · Block 4 of 10 · ← Previous · Next →

Time budget

~90 minutes. Read Cai et al. (2017), Wu et al. (2018), and the AIGB paper (2024) in full - together they are the spine of the field. Skim USCB, SORL, and the AuctionNet paper for their formulations and headline numbers. Keep the 2025-2026 generative papers and the LLM-agent section as reference; they are moving too fast to read closely in one sitting.

Why bidding became a reinforcement learning problem §

The previous block ended with a value: $v=\text{pCTR}\cdot \text{pCVR}\cdot m$ per impression. If every auction were independent and the budget were infinite, the bid would be a formula and this block would not exist. Neither is true. A campaign has a daily budget, so a dollar spent at 9 a.m. is a dollar not available at 9 p.m. Competitors adjust to your bids. The platform’s own dynamic bidding rewrites the number you submit before the auction sees it (see the controls in Block 8). And the logs never show what would have happened at a bid you did not submit. Sequential decisions, a scarce resource, a reactive environment, and bandit feedback: that is the definition of a reinforcement learning problem, and the literature reached that conclusion around 2017.

This block traces four eras. The action started as a bid, became a multiplier on a formula, then a policy trained offline on logs, and most recently a whole trajectory generated by a diffusion model or transformer. Each move answered a failure of the previous one, and the failures are more instructive than the architectures.

The MDP framing (2014-2018) §

The pre-RL baseline is Zhang, Yuan, and Wang’s KDD 2014 paper on optimal real-time bidding, which derived closed-form bidding functions from a bid-landscape model and a budget constraint ^[1]. The bid was a static function of predicted value, and the optimality was over a single campaign horizon with a known landscape. Every later paper asks what breaks when the landscape is unknown, nonstationary, or shaped by your own actions.

Cai et al. (WSDM 2017) made the sequential structure explicit ^[2]. They modeled a campaign as a Markov decision process

$$\mathcal{M}=⟨\mathcal{S},\mathcal{A},P,r,\gamma ⟩,s_t=(t,B_t,x_t),$$

where $t$ is the number of auctions remaining in the episode, $B_t$ the remaining budget, $x_t$ the feature vector of the current impression, the action $a_t$ is the bid, and the reward is the predicted value of the impression if won and zero otherwise. The transition is driven by whether the bid clears the (unknown) market price. With $x_t$ marginalized out the state space is small, so they solved it model-based: estimate the market-price distribution from logs, then run dynamic programming over $(t,B_t)$; the value function’s gradient in budget is a shadow price. The optimal bid emerges as $a^{\ast }(s)=\arg {\max }_a\mathbb{E}[r\mid a]-\text{(marginal value of the budget the win consumes)}$, which is the marginal-equalization result from the previous block, rederived as a Bellman equation.

Wu et al. (CIKM 2018), from Alibaba, kept the MDP but changed the action ^[3]. Instead of choosing a bid per impression, their DRLB agent chooses a multiplier $\lambda$ that scales a formula bid, $b=v/\lambda$, and adjusts $\lambda$ at fixed intervals. Two additions matter. RewardNet learns a reward shaping that compensates for sparse, delayed conversions, and an adaptive $ϵ$-greedy schedule controls exploration in a live system. They report gains of 100.92%, 18.33%, and 16.80% over three baselines on one evaluation and a 4.3% click improvement on another ^[3]. They also report the result most useful to a practitioner: DRLB does worse on data with low AUC or low CTR, which is to say that a policy is only as good as the value model underneath it ^[3].

Zhao et al. (KDD 2018) is the one paper in this era that is literally about sponsored search ^[4]. Working on Alibaba’s e-commerce search ads, they argued that per-impression policies are fragile under the nonstationarity of a search marketplace, and instead used a robust MDP in which the agent selects, hourly, a parameterized bidding model - a “control-by-model” design where the action is a setting of the formula, not a price. Reported deployment results were a 35.04% improvement in a purchase-related cost metric, a 23.7% conversion-rate increase, and a 21.38% ROI increase over the comparison method ^[4]. Amazon’s own hourly cadence, and every third-party tool that copies it, is structurally this paper.

Jin et al. (CIKM 2018) added the missing piece: other bidders ^[5]. They clustered thousands of advertisers into groups, trained a MADDPG-style multi-agent policy per cluster, and showed that cooperative and competitive dynamics between clusters change the optimal bid (the MADDPG attribution comes from secondary indexing and should be checked against the proceedings). From any single bidder’s point of view, other learning agents make the environment nonstationary, and this paper is the first to take that seriously rather than treating competitors as a fixed price distribution.

Wen et al.’s MAAB (WSDM 2022, Alibaba) is the multi-agent paper to read after Jin ^[38]. It frames auto-bidding as cooperative-competitive: advertisers share the platform’s interest in a healthy market but compete for slots, and left alone, learning agents can coordinate on collusive underbidding. MAAB adds “bar agents” that set personalized floors to prevent that, and scales to millions of advertisers with mean-field grouping. It is the bidder-side view of the soft floors Amazon studies from the platform side, and Block 6 takes up the collusion question directly.

Section takeaway. By 2018 the field had settled the state (time and budget remaining, plus features), was still arguing about the action (a bid, a multiplier, or a model), and had discovered that the value model and the competitor population determine whether any of it works.

Constrained bidding (2019-2021) §

Real campaigns have more than a budget. They have a target cost per acquisition, a ROAS floor, a CPC cap, and sometimes a minimum spend. Yang et al. (KDD 2019) treated the problem as multivariable control, with one controller per constraint and a coordination layer ^[6]. He et al.’s USCB (KDD 2021), again from Alibaba, unified the variants into a single Lagrangian ^[7]. For a campaign choosing bids $b_i$ over impressions $i$ with values $v_i$, costs $c_i(b_i)$, budget $B$, and a constraint set $\{g_k(\cdot )\le 0\}$, the relaxed objective is

$$\underset{b}{\max }\sum _i{v}_i{x}_i(b_i)-\lambda (\sum _i{c}_i(b_i)x_i(b_i)-B)-\sum _k{\mu }_k{g}_k(b),$$

where $x_i(b_i)$ is the win indicator. The optimal bid has the form $b_i^{\ast }=f(v_i;\lambda ,\mu )$ for a small set of multipliers, so the policy’s job collapses to choosing the multipliers over time. USCB learns that choice with RL and shows one bidding function serving budget, ROI, and CPC constraints simultaneously ^[7]. The insight that survives into everything after: the action space should be the dual variables, not the primal bids. It is lower-dimensional, it has an economic meaning, and it makes a safe fallback trivial (freeze the multipliers).

The RL papers have a theoretical twin that rarely shares a bibliography: single-bidder no-regret learning. Feng, Podimata, and Syrgkanis (EC 2018) studied a bidder in a sponsored-search model who does not know its own value and learns from outcome feedback ^[39]. Han, Zhou, and Weissman (OR 2025) gave optimal no-regret rates for repeated first-price auctions with censored feedback ^[40]; Aggarwal, Fikioris, and Zhao (2024) proved an $\Omega (T^{2/3})$ lower bound under bandit feedback in first-price auctions with budget and ROI constraints ^[41]; Deng, Li, Tang, and Zhang (NeurIPS 2025) extend the constrained setting ^[42]; and Feng, Padmanabhan, and Wang (WWW 2023) gave a primal-dual controller for return-on-spend constraints that is, structurally, USCB’s multiplier update with a regret bound attached ^[43]. The RL work asks how well a policy does on one platform’s logs; this work asks how well any policy can do. Block 6 covers what happens when every bidder runs one.

Section takeaway. The constrained-bidding literature turned “bid optimization” into “multiplier control,” and that reparameterization is why the later generative models can generate whole trajectories in a space of a few dimensions rather than millions of bids.

Offline and conservative RL (2022-2025) §

Online RL on live spend is expensive and slow. The obvious alternative is to train on logs, and the obvious problem is that logs only cover the bids the old policy submitted. Value estimates for out-of-distribution actions are unconstrained, and a Q-learner will happily exploit them.

Mou et al.’s SORL (NeurIPS 2022) named the gap directly: the “virtual advertising system” (a simulator or offline replay) and the “real advertising system” differ enough that direct deployment is unsafe ^[8]. Their answer has two parts. V-CQL, a variance-suppressed conservative Q-learning, penalizes Q-values for actions unsupported by the data while damping the instability plain CQL exhibits on bidding data. A safe-exploration rule then bounds how far the online policy may deviate from the offline one. The reported real-world A/B test lists 139,599 purchases, ROI of 3.15, and CPA of 40.05 against multi-agent and other baselines ^[8]. Code is proprietary.

Korenkevych et al. (arXiv 2023; ACM record 2024, exact venue label flagged in the research record) took a different route to safety ^[9]. Rather than replace the production bidder, they kept it as a base policy and let a neural policy tune a handful of its parameters. The evaluation covered roughly 200,000 week-long campaigns and about 1.2 billion steps, with statistically significant online gains, and the authors are explicit that the method is bounded by the quality of the base policy and by distribution shift ^[9]. For a Sponsored Products account this is the most realistic migration path: a trusted formula with learned knobs.

Three 2024-2025 papers address what offline data cannot show. PE-MORL (Mou et al., 2025 preprint) learns an environment model over the states and actions of all competitors and applies pessimistic penalties where the model is uncertain; against a GSP simulator it reports MAE down 69.6% and MSE down 88.2% versus baselines, and it compares against V-CQL and USCB ^[10]. Lin, Zheng, and Wu (KDD 2024) treat censorship and distribution shift as the primary threats and optimize a worst-case surplus rather than an average ^[11]. Jiang et al.’s RRLB (arXiv 2022; ACM record flagged) builds a risk-aware value, mean response minus a tunable multiple of predicted standard deviation, so the bidder decides how much uncertainty it is willing to buy ^[12].

Terminology

“IQL-style bidding” is sometimes used loosely. No verified auto-bidding paper centers on Implicit Q-Learning; the closest primary sources are V-CQL and the hybrid base-policy work. Treat IQL as a method family to test, not a citation.

Section takeaway. Offline RL for bidding is a set of techniques for being pessimistic about actions the logs never tried: conservative Q-values, a base policy you never leave far, a pessimistic world model, a worst-case objective, or an explicit variance penalty. All of them are ways of saying “do not trust the simulator where it has no data.”

Generative bidding (2024-2026) §

The most recent turn changes what the model outputs. Instead of a policy $\pi (a\mid s)$, a generative bidder produces an entire trajectory of states and actions conditioned on what you want the campaign to achieve.

AIGB (Guo et al., KDD 2024) introduced the paradigm with DiffBid ^[13]. A conditional diffusion model learns the distribution of trajectories $\tau =(s_0,a_0,\dots ,s_T,a_T)$ in logged campaigns, conditioned on the realized return $R(\tau )$ and constraint satisfaction $C(\tau )$. At deployment you condition on the return you want:

$${\tau }^{\ast }\sim p_{\theta }\left(\tau |R(\tau )=R_{\text{target}},C(\tau )=1,s_0\right),a_t=f_{\varphi }(s_t,s_{t+1}),$$

with an inverse-dynamics head $f_{\varphi }$ recovering the action (the multiplier) from consecutive generated states. Because the model conditions on the whole trajectory, it is not Markovian and can plan pacing across the day rather than reacting hour by hour. Alibaba’s online A/B reported GMV +2.81%, ROI +3.36%, and purchases +2.09%, with fewer failures than the RL comparison ^[13].

The 2025-2026 follow-ups each address a weakness of unconditioned generation:

• GAS (Li et al., WWW 2025; arXiv 2024) adds post-training search: generate candidate plans, then search or rerank them against the current objective, so the generator does not need retraining when constraints change ^[14].
• GAVE (Gao et al., 2025; SIGIR record flagged) uses a decision-transformer backbone with value-guided exploration, steering samples toward high-value regions while preserving coverage; final online metrics are not reliably exposed in the research record ^[15].
• Return-to-go guided Decision Transformer (Jiang et al., 2025; WWW record flagged) turns the desired outcome into a control input, which maps cleanly onto a target ACOS or spend ^[16].
• AIGB-Pearl (Mou et al., 2025 preprint) adds a learned trajectory evaluator and a KL-Lipschitz-constrained score maximization for safe exploration beyond the dataset. Sources disagree on the reported GMV lift - roughly 3% in the OpenReview record versus 5.1% elsewhere - and the discrepancy is unresolved ^[17].
• Expert-Guided Diffusion Planner (Peng et al., 2025 preprint; CIKM 2025 per ACM listing) supplies a behavioral prior from expert trajectories so plausible-but-inferior plans are suppressed ^[18].
• GRAD (Lei et al., KDD 2026, Meituan) combines an Action-MoE generator with a causal-transformer value estimator; reported online results are GMV +2.18%, ROI +10.68%, CTR +3.93%, CPC_CR +5.64% ^[19].
• JD-BP (Meng et al., 2026 preprint, JD.com) generates bids and prices jointly, on the argument that the two co-evolve; online A/B reports ad revenue +4.70% and target cost +6.48% ^[20].
• Constrained Auto-Bidding via Generative Response Modeling (Yang, Zuo, Kim, KDD 2026) inverts the recipe: learn a generative response model of the auction, then solve for feasible multipliers, which makes constraints auditable. On AuctionNet it reports 33.88 against 31.43 for the EBaReT baseline, a 7.8% gain ^[21].
• GRB (Wu et al., KDD 2026, Tencent) extends generative RL bidding to five channels with online A/B testing; author list and full metrics are incomplete in the record ^[22].

Read the list as design axes, not a leaderboard: coverage (diffusion, transformer), guidance (value, expert, return-to-go), feasibility layer (search, response model), and the object generated (multipliers, bid plus price, channels). The numbers are not comparable across platforms with different objectives and attribution windows.

Section takeaway. Generative bidding moved the unit of decision from an action to a plan. Its risk is the mirror image of its strength: an expressive model can produce a fluent, infeasible plan, so every serious paper since AIGB has bolted on an evaluator, a search step, or an explicit response model.

Benchmarks: AuctionGym and AuctionNet §

Two open environments anchor evaluation. AuctionGym (Jeunen, Murphy, Allison; Amazon Science 2022, KDD 2023) is a simulator for bandit learning in advertising auctions ^[23][24]. Its lasting contribution is a taxonomy: most “learning to bid” methods are value-based (estimate the value, bid a function of it), and the paper adds policy-based and doubly robust formulations that reduce dependence on modeling every auction outcome. It exists because the counterfactual for an unsubmitted bid is unobservable and competitors adapt, so offline replay of logs is not enough (see Block 7).

AuctionNet (Su et al., NeurIPS 2024 Datasets and Benchmarks; author list incomplete in the record) is the large-scale complement ^[25]. It packages an environment with a customizable GSP module, 10 million ad opportunities generated by deep generative networks, 48 diverse auto-bidding agents, and more than 500 million auction records, plus baseline evaluations. The associated NeurIPS 2024 competition drew more than 1,500 teams; online linear programming was the strongest baseline, and the authors note residual biases between generated and real data ^[25]. BAT (Khirianova et al., 2025) is a newer auto-bidding benchmark with a dataset and baselines; the available record was too thin to summarize its mechanism ^[26].

Section takeaway. A benchmark score measures performance under one bidder population and one mechanism. It is evidence of comparative quality, not of transfer to Amazon’s auction, and every paper in the previous section should be read with that caveat attached.

Pacing and landscape forecasting §

An RL bidder sits on two supporting models. The first is pacing: how to spread a budget over a day. Balseiro et al.’s field guide (ICML 2024) catalogs the algorithms, with multiplicative feedback controllers on the dual variable dominant in practice ^[27]. Mystique (WWW 2024 companion; authors not recovered) is a production pacing system built on soft throttling against a daily target curve, and the record does not establish that it uses RL ^[28]. Hajiaghayi et al. (2022) analyze a learning-based pacer in isolation, which is useful for separating “bid quality failed” from “delivery rate failed” ^[29]. Amazon’s own contribution is control-theoretic: Karlsson (IEEE CDC 2025) designs cascaded feedback controllers for bid multipliers under a budget and a CPC cap, proving stability with the circle criterion, following a 2023 hierarchical-control companion ^[44]. Pacing is the advertiser’s job on Amazon, not the platform’s: Sponsored Products daily budgets are averaged over a calendar month and are not paced within the day (see Block 8), which is why every third-party tool ships its own pacer. The pacing equilibria that arise when everyone paces are covered in Block 2.

The second is the bid landscape: the distribution of the market price you must beat. Ren et al. (KDD 2019) modeled it with recurrent networks plus survival analysis, because the observations are censored - you see the price only when you win ^[30]. Wu, Yeh, and Chen (KDD 2018) had earlier treated the censored winning price with deep models ^[45], and Ou et al. (KDD 2023) extended forecasting to correlated multi-slot pages, predicting position-specific winning-price distributions ^[31]. A search results page with several sponsored slots is exactly that setting. For evaluating a policy against such a model rather than against live spend, the root reference is Bottou et al. (JMLR 2013) on counterfactual reasoning for ad systems ^[46]; Ou et al.’s TKDD 2024 survey of bid optimization in RTB is the field’s own map ^[47].

Section takeaway. Landscape forecasting is the response model, pacing is the spend controller, and the RL or generative policy is the layer that chooses how aggressively to trade one against the other. Keep the three separable in code and in diagnosis.

LLM agents as bidders §

A parallel literature puts language models in auctions. AucArena (Chen et al., NeurIPS 2024 Open-World Agents workshop, after withdrawal from ICLR 2024) evaluates LLM agents’ strategic planning in ascending auctions ^[32]. RTBAgent (Cai et al., 2025 preprint) gives an LLM memories, retrieval, and daily reflection for RTB, on a small 9-campaign benchmark with revenue disclosure deferred ^[33]. HARBOR and InfoBid (both 2025) are testbeds for strategic competition and information disclosure ^[34][35]; a 2026 ICML-listed paper models LLM-generated creatives and bidding as a Stackelberg game (status flagged) ^[36].

A different use is more promising: LLMs as simulated advertisers. Shah et al. (2025) ran language models as auction participants and found they behave like risk-averse humans, at a cost of over 1,000 auctions for under $400 ^[48] - a cheap way to populate a simulator like AuctionGym with heterogeneous, human-like bidders.

None of this replaces the auction-time bidder. The action loop for Sponsored Products is hourly at the coarsest and per-impression at the finest, the decisions are numerical and constrained, and the cost of a fluent mistake is real money. LLM agents plausibly belong one level up - campaign structure, anomaly diagnosis, keyword hypotheses - with the numerical bidder behind deterministic guardrails. Cicero’s architecture in Block 5 is the right mental model: a language model supplies context and communication, a separate planner chooses actions.

Section takeaway. LLM auction agents are a strategic-reasoning appendix to this literature, not a replacement for a calibrated, constraint-safe numerical policy.

What transfers to Sponsored Products §

The field has four control mechanisms, and they are complements rather than rivals.

Dimension	Model-free / constrained RL	Offline conservative RL	Generative bidding	Robust / risk-aware control
Action	Multiplier or bid policy	Policy constrained to logged support	Trajectory or return-conditioned plan	Feasible action under uncertainty
Strength	Direct optimization and adaptation	Safe use of historical data	Long-horizon plans, many candidates	Explicit downside protection
Failure mode	Unsafe exploration, nonstationarity	Extrapolation, dataset bias	Plausible but infeasible plans	Conservatism, lower upside
Best evidence here	Wu, Zhao, USCB	SORL, Korenkevych, PE-MORL	AIGB, GAS, GRAD, response modeling	RRLB, Lin-Zheng-Wu
Sponsored Products use	Multipliers and pacing	Cold start, sparse logs	Campaign-level bid trajectories	Budget, ACOS, tail-keyword guardrails

A defensible reading-to-design sequence: (1) define the MDP and action timescale; (2) build a calibrated landscape and response model; (3) add pacing and hard budget controls; (4) train a conservative baseline on logs; (5) compare a generative trajectory model; (6) add value, expert, or search guidance; (7) stress-test shift, sparse products, competitor changes, and unsafe exploration. Skipping to step 5 is the most common failure in practice.

One bridge deserves its own paragraph, because the next block is built on it. Conservative offline RL and Noam Brown’s human-regularized search are the same idea in different clothes. V-CQL penalizes Q-values for actions the logged policy never took; piKL penalizes a search policy’s KL divergence from an imitation-learned human policy. Both say: the data-generating behavior is a prior, and the optimizer may deviate from it only as far as its evidence justifies. AIGB-Pearl’s KL-Lipschitz constraint on score maximization is piKL with a diffusion model in place of a Diplomacy policy network. The Korenkevych hybrid, a trusted base policy with learned adjustments, is safe subgame solving: improve locally, never fall below the blueprint. Block 5 takes that correspondence seriously and asks what else transfers.

Section takeaway. The most transferable architecture for Sponsored Products is a constrained, conservative controller with a calibrated response model and an optional generative planner. The least defensible shortcut is treating a benchmark or LLM auction score as evidence of safe deployment.

Reading list for this block §

1. Cai et al., Real-Time Bidding by Reinforcement Learning in Display Advertising, WSDM 2017 - 20 min. The MDP formulation; read for the state definition and the shadow-price interpretation.
2. Wu et al., Budget Constrained Bidding by Model-free Reinforcement Learning, CIKM 2018 - 15 min. The multiplier action and the low-AUC warning.
3. Zhao et al., Deep Reinforcement Learning for Sponsored Search Real-time Bidding, KDD 2018 - 15 min. The hourly control-by-model design that production systems copy.
4. He et al., A Unified Solution to Constrained Bidding, KDD 2021 - 10 min. Skim for the Lagrangian and the dual-variable action space.
5. Mou et al., Sustainable Online RL for Auto-bidding, NeurIPS 2022 - 10 min. Skim for the virtual-vs-real gap and V-CQL.
6. Guo et al., AIGB: Generative Auto-bidding via Diffusion Modeling, KDD 2024 - 20 min. Read the conditioning setup and the A/B section in full.
7. Su et al., AuctionNet, NeurIPS 2024 D&B - 10 min. Read the environment description and the competition results.
8. Aggarwal et al., Auto-bidding and Auctions in Online Advertising: A Survey, 2024 - reference. Use it to place the 2025-2026 papers.

Questions to carry forward §

• If the action is a multiplier on $v/\lambda$, how much of the reported RL gain is really the value model, and how would you tell?
• Which of the offline-RL pessimism devices (conservative Q, base policy, pessimistic world model, worst-case objective, variance penalty) fails most gracefully when the competitor population shifts overnight?
• A generated trajectory can violate a budget in hour 23. Where does the feasibility check belong: in the generator’s conditioning, in a post-training search, or in the pacer?
• The AuctionNet competition’s strongest baseline was online linear programming. What does that say about how much RL is buying over a well-tuned dual controller?
• Is piKL’s KL penalty toward human play a better or worse prior than CQL’s penalty toward logged play, when the “humans” are yesterday’s bidding algorithm?

References §

1. Zhang, Yuan, Wang. Optimal Real-Time Bidding for Display Advertising. KDD 2014. github.com/wnzhang/rtb-papers
2. Cai, Ren, Zhang, Malialis, Wang, Yu, Guo. Real-Time Bidding by Reinforcement Learning in Display Advertising. WSDM 2017. doi.org · arXiv:1701.02490
3. Wu, Chen, Yang, Wang, Tan, Zhang, Xu, Gai. Budget Constrained Bidding by Model-free Reinforcement Learning in Display Advertising. CIKM 2018. doi.org · arXiv:1802.08365
4. Zhao, Qiu, Guan, Zhao, He. Deep Reinforcement Learning for Sponsored Search Real-time Bidding. KDD 2018. doi.org · arXiv:1803.00259
5. Jin, Song, Li, Gai, Wang, Zhang. Real-Time Bidding with Multi-Agent Reinforcement Learning in Display Advertising. CIKM 2018. doi.org · arXiv:1802.09756
6. Yang et al. Bid Optimization by Multivariable Control in Display Advertising. KDD 2019. (Cited via the Aggarwal et al. survey; verify DOI before formal citation.)
7. He, Chen, Wu, Pan, Tan, Yu, Xu, Zhu. A Unified Solution to Constrained Bidding in Online Display Advertising. KDD 2021. doi.org
8. Mou, Huo, Bai, Xie, Yu, Xu, Zheng. Sustainable Online Reinforcement Learning for Auto-bidding. NeurIPS 2022. arXiv:2210.07006
9. Korenkevych, Cheng, Balakir, Nikulkov, Gao, Cen, Xu, Zhu. Offline Reinforcement Learning for Advertising. arXiv 2023; ACM 2024 (venue label flagged). arXiv:2310.09426 · doi.org
10. Mou, Xu, Chen, Bai, Yu, Xu. PE-MORL: Pessimistic Environment Model-based Offline RL for Auto-bidding. arXiv 2025. arXiv:2506.17919
11. Lin, Zheng, Wu. Robust Auto-bidding for Censored and Distribution-shifted Environments. KDD 2024. doi.org
12. Jiang, Zhou, Zhang, Chen, Hu, Choi. Risk-aware Reinforcement Learning for Real-time Bidding. arXiv 2022; ACM record flagged. arXiv:2212.12533
13. Guo, Huo, Zhang, Wang, Yu, Xu, Zhang, Zheng. AIGB: Generative Auto-bidding via Diffusion Modeling. KDD 2024. arXiv:2405.16141 · doi.org
14. Li, Mao, Gao, Jiang, Xu, Cai, Pan, Peng, Zheng et al. GAS: Generative Auto-bidding with Post-training Search. WWW 2025 (arXiv 2024). arXiv:2412.17018 · code
15. Gao, Li, Mao, Jiang, Jiang, Wang, Cai, Pan, Jiang, Gai, An, Zhao. GAVE: Generative Auto-bidding with Value-guided Explorations. 2025 (venue flagged). arXiv:2504.14587
16. Jiang, Tang, Zeng et al. Optimal Return-to-Go Guided Decision Transformer for Auto-Bidding in Advertisement. 2025 (WWW record flagged). arXiv:2506.21956
17. Mou et al. Enhancing Generative Auto-bidding with Offline Reward Evaluation and Policy Search (AIGB-Pearl). arXiv 2025. arXiv:2509.15927 · OpenReview
18. Peng et al. Expert-Guided Diffusion Planner for Auto-bidding. 2025. arXiv:2508.08687 · doi.org
19. Lei, Zhao, Zhao, Zhang, Cai, Xie, Wang. GRAD: Generative Reward-driven Ad Bidding. KDD 2026. arXiv:2508.02002
20. Meng et al. JD-BP: A Joint-Decision Generative Framework for Auto-Bidding and Pricing. arXiv 2026. arXiv:2604.05845
21. Yang, Zuo, Kim. Constrained Auto-Bidding via Generative Response Modeling. KDD 2026. arXiv:2605.27811
22. Wu et al. GRB: A Generative Reinforcement Bidding Framework for Multi-Channel Online Advertising. KDD 2026 (authors flagged). doi.org
23. Jeunen, Murphy, Allison. Learning to Bid with AuctionGym. Amazon Science 2022. amazon.science · code
24. Jeunen, Murphy, Allison. Off-Policy Learning-to-Bid with AuctionGym. KDD 2023. doi.org
25. Su, Huo, Zhang, Dou, Yu et al. AuctionNet: A Novel Benchmark for Decision-Making in Large-Scale Games. NeurIPS 2024 Datasets & Benchmarks. arXiv:2412.10798
26. Khirianova et al. BAT: Benchmark for Auto-bidding Task. arXiv 2025 (author list partial). arXiv:2505.08485
27. Balseiro et al. A Field Guide for Pacing Budget and ROS Constraints. ICML 2024. proceedings.mlr.press
28. Mystique: A Budget Pacing System for Performance Optimization in Online Advertising. WWW 2024 Companion (authors not recovered). doi.org
29. Hajiaghayi et al. Analysis of a Learning Based Algorithm for Budget Pacing. arXiv 2022. arXiv:2205.13330
30. Ren, Qin, Zheng, Yang, Zhang, Yu. Deep Landscape Forecasting for Real-time Bidding Advertising. KDD 2019. arXiv:1905.03028 · doi.org
31. Ou, Chen, Yang, Dai, Liu, Zhang, Tang, Yu, Xia, Xue. Deep Landscape Forecasting in Multi-Slot Real-Time Bidding. KDD 2023. doi.org
32. Chen, Yuan, Ye, Majumder, Richardson. AucArena: Evaluating Strategic Planning and Execution of LLM Agents in an Auction Arena. NeurIPS 2024. arXiv:2310.05746
33. Cai, He, Li, Liang, Lin, Quan, Zeng, Xu. RTBAgent: A Large Language Model-based Agent for Real-Time Bidding. arXiv 2025. arXiv:2502.00792
34. Jiang, Xiong, Liu. HARBOR: A Testbed for Large Language Model Agents in Auctions. arXiv 2025. arXiv:2502.12149
35. Yin et al. InfoBid: A Simulation Framework for Studying Information Disclosure in Auctions with LLM-based Agents. arXiv 2025. arXiv:2503.22726
36. Wang et al. Autobidding Auctions with LLM-Powered Creatives. ICML 2026 (status flagged). OpenReview
37. Aggarwal et al. Auto-bidding and Auctions in Online Advertising: A Survey. 2024. arXiv:2408.07685
38. Wen et al. A Cooperative-Competitive Multi-Agent Framework for Auto-bidding in Online Advertising (MAAB). WSDM 2022. arXiv:2106.06224
39. Feng, Podimata, Syrgkanis. Learning to Bid Without Knowing your Value. EC 2018. arXiv:1711.01333
40. Han, Zhou, Weissman. Optimal No-regret Learning in Repeated First-price Auctions. Operations Research 2025. arXiv:2003.09795
41. Aggarwal, Fikioris, Zhao. No-Regret Algorithms in Non-Truthful Auctions with Budget and ROI Constraints. 2024. arXiv:2404.09832
42. Deng, Li, Tang, Zhang. Learning to bid under budget and ROI constraints. NeurIPS 2025. arXiv:2510.16869
43. Feng, Padmanabhan, Wang. Online Bidding Algorithms for Return-on-Spend Constrained Advertisers. WWW 2023. arXiv:2208.13713
44. Karlsson. Multivariable Feedback Control for Multi-Constraint Optimization in Online Advertising. IEEE CDC 2025. amazon.science
45. Wu, Yeh, Chen. Deep Censored Learning of the Winning Price in the Real Time Bidding. KDD 2018. doi.org
46. Bottou et al. Counterfactual Reasoning and Learning Systems: The Example of Computational Advertising. JMLR 2013. arXiv:1209.2355
47. Ou et al. A Survey on Bid Optimization in Real-Time Bidding Display Advertising. ACM TKDD 2024. doi.org
48. Shah et al. Learning from Synthetic Labs: Language Models as Auction Participants. 2025. arXiv:2507.09083

---

Part of A Day on Amazon Ads. The map this block expands is Section 3 of Amazon Ads, Deeply; the hourly repricer in The Ad Auction, From the Inside is a playable version of the control-by-model design.