Block 5 · Noam Brown's Program, and What It Says About Bidding Under Imperfect Information

← Day index · Block 5 of 10 · ← Previous: RL for bidding · Next: Learning agents in auctions →

Time budget

~75 minutes. Read the Libratus and ReBeL abstracts and the piKL paper’s Section 3 first (35 min); skim Safe and Nested Subgame Solving and Depth-Limited Solving for the definitions (20 min); keep the bridges table at the end as the thing you carry into Block 6. Nothing here is an established result about Amazon Ads. It is a set of transfers, each labeled with what would have to be true for it to work.

Why a poker researcher belongs in a day on ad auctions §

A sponsored-search auction has the exact information structure that made poker hard and Go easy. In Go, both players see the board; a position has a value, and search can back values up from leaves. In poker, you never see your opponent’s cards, so a “position” has no value independent of what each player believes about the other, and a naive depth-limited search from a mid-hand state is provably wrong - the value of the subgame depends on parts of the game that were never reached ^[3]. A keyword bid on Amazon is poker, not Go. You never see rival bids. You observe a win or a loss, a price, aggregated hourly reports, and a conversion that may arrive two weeks later. The platform’s reserve logic is partly hidden ^[20]. The set of competitors changes by the hour. There are hundreds of thousands of players, and the game is not zero-sum, because a shopper and a platform sit at the table too.

Noam Brown spent a decade making machines play exactly this kind of game at superhuman level: heads-up no-limit poker (Libratus, 2017-2018) ^[4], six-player poker (Pluribus, 2019) ^[8], and no-press and full-press Diplomacy (Cicero, 2022) ^[13], before moving to test-time reasoning in language models at OpenAI ^[16]. The research program has a single organizing thesis, stated most plainly in the ReBeL paper: combine a learned policy with search at decision time, and do it in a way that is sound under imperfect information ^[9]. The claim of this block is that the ideas in that program - regret minimization, safe subgame re-solving, depth-limited lookahead with a portfolio of opponent continuations, public belief states, human-regularized planning - are the right vocabulary for a bidding agent, and that the RL-for-bidding literature you just read has been independently reinventing several of them under other names.

This is a bridge, not a proof. Every transfer below comes with the assumption it depends on.

The program in one table §

Year	Paper	The move	Result
2015-2017	Regret-Based Pruning; Reduced Space via Pruning; Dynamic Thresholding [1][2]	Prune actions with persistently negative regret; keep convergence guarantees	7x less space in the cited setting [1]
2017	Safe and Nested Subgame Solving [3]	Re-solve a subgame in real time without becoming exploitable relative to the blueprint	NeurIPS best paper; the engine of Libratus
2018	Libratus [4]	Blueprint via MCCFR abstraction + nested real-time solving + self-improvement overnight	Beat four top professionals over 120,000 hands
2018	Depth-Limited Solving [5]	At the depth limit, let the opponent choose among several continuation strategies rather than one value	Master-level poker on a laptop
2019	Deep CFR [6]; Discounted CFR [7]	Replace tabular regrets with a network; discount early iterations’ regrets	Deep CFR removes hand-built abstraction; DCFR variants beat CFR+ in every game tested [7]
2019	Pluribus [8]	Six players; abandon the Nash guarantee; blueprint + depth-limited search with four continuation policies	Superhuman in six-player no-limit hold’em
2020	ReBeL [9]	Public belief states make imperfect-information games look like perfect-information ones; RL + search converges to Nash in two-player zero-sum	General algorithm, approximate equilibria in poker and Liar’s Dice
2021	No-Press Diplomacy from Scratch (DORA) [10]; Off-Belief Learning [11]; Scalable Online Planning [12]	Huge action spaces via policy proposals and double-oracle; robustness to conventions; replace tree search with online RL fine-tuning	Superhuman in a two-player variant; new tools for many-agent play
2022	piKL [14]; Cicero [13]	KL-regularize search toward an imitation policy; language model + strategic planner	Human-like and strong; top-10% in a human Diplomacy league
2023	Diplodocus (DiL-piKL) [15]	Distribute the regularization strength; train with RL against regularized planners	First in a 200-game, 62-human tournament
2023-2024	Abstracting imperfect information away; update-equivalence planning [41]	Regularized equilibria computable as perfect-information problems; planning that mimics last-iterate updates instead of solving public subgames	~100x less search time than public-belief search in Hanabi [41]
2024-	o1 and reasoning models [16]	Scale compute at inference time, not just training	The program’s thesis, applied to language

Read down the “move” column and a shape appears. First, make equilibrium computation tractable (regret minimization, pruning, abstraction). Second, make it local - compute the fine strategy only where you are, and make the local computation safe. Third, replace tables with learning (Deep CFR, ReBeL’s value nets). Fourth, give up guarantees you cannot have (many players, general-sum) and replace them with regularization toward observed behavior. Fifth, generalize the whole idea to “spend compute at decision time.” Every one of those five steps has a bidding counterpart.

Section takeaway. Brown’s program is not “poker AI.” It is a method for acting well when you cannot see the other players’ state, cannot search the whole game, and cannot trust a policy computed offline to be right for the situation you are in now. That is a description of an ad auction.

Idea 1: Regret minimization is already the bidder’s algorithm §

Counterfactual regret minimization (CFR) ^[6][7] is a self-play procedure: at every decision point, track for each action how much better you would have done had you always taken it, and play actions in proportion to positive regret. The average strategy converges to a Nash equilibrium in two-player zero-sum games; in general games it converges to a coarse correlated equilibrium. Brown’s contributions sharpen the procedure. Discounted CFR ^[7] weights early iterations less - a fix for the observation that the first iterations of self-play are junk and should not anchor the average - and Linear CFR / DCFR variants outperform CFR+ across the board. Deep CFR ^[6] trains a network to predict the counterfactual advantages (regrets) from sampled traversals so that no hand-built card abstraction is needed.

The bidding literature independently arrived at regret. Han, Zhou, and Weissman ^[17] study a bidder in repeated first-price auctions who sees only the winning bid at the end of each round, and give the minimax-optimal no-regret learner for that censored-feedback setting. Balseiro and Gur ^[18] give adaptive pacing rules that are no-regret against arbitrary competitors and converge to an approximate Nash equilibrium when everyone uses them. Kolumbus and Nisan ^[19] ask what happens when every participant in an auction is a regret-minimizing agent and show the dynamics change what the auction “is” for the humans who set the agents’ inputs. Closest of all is Feng, Podimata, and Syrgkanis ^[40], which is explicitly a sponsored-search model: the bidder learns its own value only when it wins, a bid determines a slot, and utility is observed only for the slot obtained. Their “outcome-based feedback” algorithm gets regret logarithmic in the number of bid levels, which is the feedback structure of a keyword auction stated as a theorem rather than an analogy. These are the topics of Block 6; the point here is the lineage. Regret matching is the primitive on both sides of the analogy.

Transfer. A per-keyword bidder can maintain regrets over a discretized bid grid, weighted by recency (DCFR’s discounting is exactly the right response to a bid landscape that drifts), and play the regret-matching mixture. This is a bandit algorithm in disguise - which is the AuctionGym paper’s point about most “learning to bid” methods ^[21]. Assumption it depends on: the feedback you get (win/loss, price) is informative enough to compute counterfactual regret for bids you did not submit. Under second-price rules it is: the price you paid tells you the threshold you needed to beat, so every alternative bid’s outcome is known. Under soft reserves and relevance weighting it is only partly so. Under first-price rules with censored losses it is not, and you are in Han-Zhou-Weissman territory.

Idea 2: Safe subgame solving is the template for hourly re-solving §

Libratus computed a coarse “blueprint” strategy for the whole game offline, then, once a hand reached the later betting rounds, re-solved that subgame in real time with a much finer abstraction ^[4]. The subtlety Brown and Sandholm proved in 2017 is that you cannot just optimize the subgame in isolation: the opponent’s strategy in this subgame depends on what they would have done in subgames that did not happen, and a naive re-solve can be exploited. “Safe” subgame solving constrains the re-solved strategy so the opponent cannot gain, in any of their possible states, relative to the blueprint ^[3]. Nested re-solving then repeats this every time a new off-tree action appears.

The production bidding loop described in Amazon Ads, Deeply - ingest the hour’s logs, update posteriors, re-solve the budget allocation, write new bids - is subgame re-solving without the safety constraint. The hour is the subgame. The daily plan is the blueprint. Amazon’s own OPTIMUS ^[22] is the offline blueprint solver: forecast landscapes, equalize marginal ROAS across targeting clauses, output bids. What none of the industrial papers do is bound the damage the hourly re-solve can do relative to the daily plan. Brown’s construction is the right formalization: the re-solved hourly policy should be admissible only if, for every plausible competitor state, it does no worse than the blueprint’s continuation. In bidding terms, an hourly re-solve that spends the day’s remaining budget by noon because the last hour’s conversion posterior spiked is an unsafe subgame solution.

Transfer. A two-level bidder: daily Lagrangian blueprint, hourly local re-solve, with an explicit safety margin against the blueprint’s guaranteed spend and value. Assumption: you can enumerate or sample the competitor states that matter. In poker they are the opponent’s hands; in an auction they are the rivals’ bid distributions at this hour, which is what a bid-landscape model estimates ^[23].

Idea 3: Depth-limited search with a portfolio of opponent continuations §

You cannot search a poker hand to the end in real time, and you cannot simulate a campaign to the end of the month before writing each bid. Brown, Sandholm, and Amos ^[5] showed how to stop a search early in an imperfect-information game without assuming a single value for the leaf: at the depth limit, the opponent gets to choose among a small set of continuation strategies (in Pluribus, four: the blueprint, and three biased toward folding, calling, or raising ^[8]), and the searcher must be robust to whichever they pick. This is what made Pluribus run on two CPUs rather than a supercomputer.

For a bidder, the depth limit is the end of the hour or the day, and the opponent’s continuations are the competitor population’s plausible reactions: they hold bids, they follow you up, they get outbid and drop out, a new entrant arrives. Amazon’s own auction-realism paper ^[24] models rivals as adversarial bandit agents with partial feedback and shows that conclusions about mechanism parameters (soft floors) flip depending on how rivals adapt. That is a portfolio of continuations. Evaluating a candidate bid trajectory against, say, four competitor-response scenarios and choosing the one with the best worst case is Pluribus’s search, transplanted.

Transfer. Small action set (a few candidate bid changes), shallow horizon (the next hour or two), a handful of competitor-response models, minimax or soft-minimax over the portfolio. The generative bidders in Block 4 - AIGB’s trajectory generation ^[25] and GAS’s post-training search ^[26] - are already generating candidate trajectories and then selecting; what they lack is the adversarial portfolio at the leaf. Assumption: the response models are not so wrong that search exploits their errors. This is the failure mode Brown’s papers warn about most: search amplifies model misspecification, which is why ReBeL trains its value net on the states search actually visits ^[9].

Idea 4: Public belief states, and what “state” means when you cannot see the other bids §

ReBeL’s central trick ^[9] is to redefine the state. Instead of the (unobservable) joint hands, the state is the public belief state: everything publicly observed so far, plus each player’s probability distribution over private information given that public history. In that representation the game is a perfect-information game over beliefs, and the whole AlphaZero-style machinery - a value network over states, search, self-play - becomes applicable. The value network is trained on public belief states reached during self-play search, so it is accurate where it is used.

The bidding analogue is precise. The public history is the sequence of your bids, wins, prices, positions, and aggregate reports. The private information is rivals’ values and budgets - and, on Amazon, the platform’s reserve parameters, which the FTC complaint alleges were set per-auction by an undisclosed model ^[20]. A public belief state for a bidder is its posterior over the bid landscape and the reserve, given what it has observed. The RL-for-bidding papers that use a raw state like (time left, budget left, win rate so far) ^[27] are using a lossy summary of that belief; PE-MORL’s opponent-aware environment model ^[28] is a step toward the belief itself. Brown and coauthors’ 2022 work on inference-time fine-tuning of belief models ^[29] goes further: specialize the belief model to the local observations at decision time.

Transfer. Make the bidder’s state a posterior, not a scalar summary: a distribution over the clearing-price landscape per (keyword, placement, hour), plus a posterior over the reserve layer. Train the value model on the beliefs the hourly loop actually reaches. Assumption: the posterior is well-calibrated. The delayed-feedback and calibration literature in Block 3 is what makes this assumption defensible or not.

Idea 5: Many players, general-sum - give up Nash and regularize toward behavior §

Everything above has a clean guarantee in two-player zero-sum games and none in six-player poker or seven-player Diplomacy. Pluribus ^[8] simply proceeded without a guarantee, and it worked; the paper’s explanation is that the same machinery produces strategies that are empirically hard to exploit even when Nash is not the right concept. Diplomacy pushed harder. Self-play from scratch produced agents that were strong against each other and inept with humans, because they converged to conventions humans do not use ^[10]. Off-Belief Learning ^[11] attacks the convention problem directly. piKL ^[14] is the general fix: run regret-minimizing search, but regularize toward an imitation-learned policy ${\pi }_h$ with a KL penalty,

$${\pi }^{\ast }(a)\propto {\pi }_h(a)\exp (Q(a)/\lambda ),$$

so that $\lambda \to \infty$ recovers pure imitation and $\lambda \to 0$ recovers pure optimization. The result is an agent that is both stronger than the humans it imitates and predictive of what they do ^[14]. DiL-piKL ^[15] puts a distribution over $\lambda$ to model that different humans deviate by different amounts, and Diplodocus is trained with RL against such regularized planners.

Now look back at the offline-RL section of Block 4. SORL’s V-CQL ^[30] penalizes Q-values on actions unsupported by the logged data. Korenkevych et al. ^[31] keep a trusted base policy and let a network adjust only its parameters. The expert-guided diffusion planner ^[32] pulls generated trajectories toward expert ones. These are all ${\pi }_h\cdot \exp (Q/\lambda )$ with a different name for ${\pi }_h$: the logging policy, the base bidder, the expert demonstrations. piKL is the principled statement of what conservative offline RL for bidding is doing, and it comes with something the offline-RL papers lack - a semantic for $\lambda$. In Diplomacy, $\lambda$ is how much you trust the human prior over the search. In bidding, $\lambda$ is how much you trust last month’s bids over this hour’s model. DiL-piKL’s distribution over $\lambda$ is a way to hedge across regimes: a stable keyword gets a large $\lambda$, a keyword whose landscape just shifted gets a small one.

Transfer. Anchor the bid policy to a behavior prior (the account’s own history, or a production bidder), optimize with a KL leash, and set the leash per keyword by how much the environment has moved. Assumption: the behavior prior is not itself bad. Regularizing toward a bidder that systematically overpays freezes the overpayment in. Sweep $\lambda$ and measure, as the Diplomacy papers do.

Idea 6: Separate the language model from the planner §

Cicero ^[13] is two systems. A strategic reasoning module (piKL planning over a learned policy and value) decides what to do; a language model, conditioned on the planned intents, decides what to say, and a set of filters keeps the two consistent. The language model never chooses the moves.

The 2024-2026 wave of LLM bidding agents - AucArena ^[33], RTBAgent ^[34], HARBOR ^[35] - mostly puts a language model in the bidder’s seat, with memory and reflection. The results are interesting as strategic-reasoning benchmarks and weak as bidding systems; RTBAgent’s own authors defer revenue results to future work ^[34]. Cicero’s architecture is the corrective. The language model belongs where language is: reading a brand’s catalog and margins, writing the campaign structure, explaining an anomaly, negotiating a goal with a human. The bids belong to a calibrated numeric planner. The interface between them is intents - “protect these ten keywords, grow this niche at a 30% ACOS ceiling” - exactly as Cicero conditions dialogue on planned moves.

Transfer. LLM as campaign planner and explainer; regret-minimizing numeric controller as bidder; a typed intent interface between them; filters that reject bids inconsistent with stated intent. Assumption: intents can be expressed as constraints the controller can honor, which is what the constrained-bidding literature (USCB ^[36], the response-model formulation in ^[37]) gives you.

Idea 7: Spend compute where the stakes are §

Libratus did not re-solve the first betting round; it played the blueprint there and saved real-time computation for the rounds where pots were large and the abstraction was coarsest ^[4]. The reasoning-model work Brown moved to at OpenAI is the same idea generalized: some problems deserve more inference-time computation than others, and the return on that computation can be large ^[16].

Sponsored Products has a brutal stakes distribution. A handful of head keywords at top-of-search carry most of the spend; the tail of keywords with three impressions a month carries most of the count. A bidder that spends the same compute per keyword is doing Libratus backward. The head deserves the depth-limited portfolio search of Idea 3 every hour. The tail deserves a Beta-Binomial posterior and Thompson sampling, which is what the interactive lab’s second widget does, and nothing more until it earns attention.

Transfer. Tier keywords by stakes; allocate re-solve depth, competitor-portfolio size, and model complexity by tier. Assumption: stakes are known, which requires the margin-aware value model from the bidding-brain section of the parent note.

What does not transfer §

Honesty about the analogy is the point of this block, so the mismatches deserve their own list.

• Zero-sum. ReBeL’s convergence proof and CFR’s Nash guarantee hold for two-player zero-sum games ^[9]. An ad auction is general-sum with a platform, many advertisers, and shoppers. Pluribus shows the machinery can still work; nothing shows it must.
• Known rules. Poker’s rules are public. The FTC complaint and Amazon’s response ^[20][38] disagree about the pricing rule of the auction you are bidding in. A game whose payoff function is disputed in court is not a game you can solve; it is one you must first infer.
• Clean feedback. Poker reveals the showdown. Amazon reveals hourly aggregates, a 14-day attribution window, and no losing bids. The measurement block is entirely about this gap.
• Stationarity of the opponent population. Libratus faced the same four humans for 20 days. A keyword faces a different set of rivals every hour, some of them other learning agents. That is Block 6’s subject.
• Scale. Pluribus searched one hand at a time. A bidder writes millions of bids an hour. The compute-by-stakes principle is a necessity here, not a refinement.

The report that compiled this block’s sources found no verified paper that is authored by Brown, applies CFR or his search architecture directly, and targets a sponsored-search mechanism ^[39]. That gap is a research opportunity, and Block 10 states it as one.

Bridges, collected §

Brown’s idea	Bidding analogue	Where Block 4 already does it	What must hold
Regret matching / DCFR	Per-keyword no-regret bidding over a bid grid, recency-weighted	AuctionGym’s value-based bandits [21]	Feedback rich enough to compute counterfactual regret
Safe subgame solving	Hourly re-solve bounded against the daily blueprint	Hourly loops; OPTIMUS as blueprint [22]	Competitor states enumerable via landscape model
Depth-limited search, opponent portfolio	Evaluate candidate bid paths against several rival-response models	AIGB / GAS candidate trajectories [25][26]	Response models not exploitably wrong
Public belief states	State = posterior over landscape and reserve	PE-MORL’s opponent model [28]	Posterior calibrated
piKL / DiL-piKL	KL-leashed policy anchored to behavior prior, per-keyword $\lambda$	V-CQL, hybrid base policies, expert-guided diffusion [30][31][32]	Prior not systematically wrong
Cicero’s split	LLM plans intents; numeric controller bids	RTBAgent, AucArena as the cautionary tale [33][34]	Intents expressible as constraints
Compute by stakes	Deep search on head keywords, posteriors on the tail	Not yet in the literature	Stakes known from a margin model
Abstraction (card bucketing)	Keyword clustering, shared priors	Bayesian shrinkage in the tail	Clusters share the response structure

Reading list for this block §

1. Brown & Sandholm. Safe and Nested Subgame Solving for Imperfect-Information Games. NeurIPS 2017. 20 min. The definitions of safety you will reuse for hourly re-solving.
2. Brown & Sandholm. Superhuman AI for heads-up no-limit poker: Libratus. Science 2018. 15 min. The blueprint-plus-real-time architecture and where compute was spent.
3. Brown, Sandholm & Amos. Depth-Limited Solving for Imperfect-Information Games. NeurIPS 2018. 15 min. The opponent-portfolio trick.
4. Brown, Bakhtin, Lerer & Gong. Combining Deep RL and Search for Imperfect-Information Games (ReBeL). NeurIPS 2020. 20 min. Public belief states.
5. Jacob et al. Modeling Strong and Human-Like Gameplay with KL-Regularized Search (piKL). ICML 2022. 20 min. Read Section 3; then reread SORL from Block 4.
6. Bakhtin et al. Mastering No-Press Diplomacy via Human-Regularized RL and Planning. ICLR 2023. 10 min. DiL-piKL’s distribution over $\lambda$.
7. Meta FAIR. Human-level play in Diplomacy by combining language models with strategic reasoning (Cicero). Science 2022. 10 min. The planner/LM split.
8. Brown & Sandholm. Solving Imperfect-Information Games via Discounted Regret Minimization. AAAI 2019. Skim. Why to discount early regrets in a drifting environment.

Questions to carry forward §

• If the pricing rule is disputed (Block 8), what is the “game” a bidder should compute regret against - the rule Amazon describes, the rule the FTC describes, or a posterior over both?
• What is the right safety constraint for an hourly re-solve on a daily budget - guaranteed spend, guaranteed value, or a bound on the Lagrange multiplier’s movement?
• Can a bid-landscape model play the role of Pluribus’s four continuation strategies, and how many scenarios are enough?
• Is there a natural $\lambda$ schedule for piKL-style bidding driven by posterior width, so that stable keywords imitate and shifting keywords optimize?

References §

1. Brown & Sandholm. Reduced Space and Faster Convergence in Imperfect-Information Games via Pruning. ICML 2017. proceedings.mlr.press. Prerequisite: Regret-Based Pruning in Extensive-Form Games. NeurIPS 2015. papers.nips.cc
2. Brown, Kroer & Sandholm. Dynamic Thresholding and Pruning for Regret Minimization. AAAI 2017. ojs.aaai.org
3. Brown & Sandholm. Safe and Nested Subgame Solving for Imperfect-Information Games. NeurIPS 2017 (best paper). arXiv:1705.02955 · proceedings.neurips.cc
4. Brown & Sandholm. Superhuman AI for heads-up no-limit poker: Libratus beats top professionals. Science 2018. doi.org/10.1126/science.aao1733
5. Brown, Sandholm & Amos. Depth-Limited Solving for Imperfect-Information Games. NeurIPS 2018. arXiv:1805.08195
6. Brown, Lerer, Gross & Sandholm. Deep Counterfactual Regret Minimization. ICML 2019. arXiv:1811.00164 · proceedings.mlr.press
7. Brown & Sandholm. Solving Imperfect-Information Games via Discounted Regret Minimization. AAAI 2019. doi.org/10.1609/aaai.v33i01.33011829 · arXiv:1809.04040
8. Brown & Sandholm. Superhuman AI for multiplayer poker. Science 2019. doi.org/10.1126/science.aay2400
9. Brown, Bakhtin, Lerer & Gong. Combining Deep Reinforcement Learning and Search for Imperfect-Information Games. NeurIPS 2020. arXiv:2007.13544
10. Bakhtin, Wu, Lerer & Brown. No-Press Diplomacy from Scratch. NeurIPS 2021. arXiv:2110.02924 · proceedings.neurips.cc. Predecessor: Gray, Lerer, Bakhtin & Brown. Human-Level Performance in No-Press Diplomacy via Equilibrium Search. ICLR 2021. arXiv:2010.02923
11. Hu, Lerer, Cui, Pineda, Brown & Foerster. Off-Belief Learning. ICML 2021. proceedings.mlr.press
12. Fickinger, Hu, Amos, Russell & Brown. Scalable Online Planning via Reinforcement Learning Fine-Tuning. NeurIPS 2021. arXiv:2109.15316
13. Meta Fundamental AI Research Diplomacy Team (FAIR), incl. Brown. Human-level play in the game of Diplomacy by combining language models with strategic reasoning. Science 2022. doi.org/10.1126/science.ade9097
14. Jacob, Wu, Farina, Lerer, Hu, Bakhtin, Andreas & Brown. Modeling Strong and Human-Like Gameplay with KL-Regularized Search. ICML 2022. arXiv:2112.07544 · proceedings.mlr.press
15. Bakhtin, Wu, Lerer, Gray, Jacob, Farina, Miller & Brown. Mastering the Game of No-Press Diplomacy via Human-Regularized Reinforcement Learning and Planning. ICLR 2023 (best paper honorable mention). arXiv:2210.05492 · openreview.net
16. OpenAI. Learning to Reason with LLMs (Sept 2024) and OpenAI o1 System Card (Dec 2024; Brown is in the author list). openai.com · arXiv:2412.16720. Brown’s role is described on noambrown.com; these are public technical documents, not peer-reviewed auction papers. An arXiv author search during this sweep found no 2025-2026 Brown research paper beyond the o1 and GPT-5 system cards.
17. Han, Zhou & Weissman. Optimal No-regret Learning in Repeated First-price Auctions. Operations Research 2025 (arXiv 2020). arXiv:2003.09795 · doi.org/10.1287/opre.2020.0282
18. Balseiro & Gur. Learning in Repeated Auctions with Budgets: Regret Minimization and Equilibrium. Management Science 2019. doi.org/10.1287/mnsc.2018.3174
19. Kolumbus & Nisan. Auctions between Regret-Minimizing Agents. WWW 2022. arXiv:2110.11855
20. Federal Trade Commission. Complaint, FTC et al. v. Amazon (Sponsored Products advertising). Aug 31, 2026. ftc.gov · press release ftc.gov. Allegations, not adjudicated facts.
21. Jeunen, Murphy & Allison. Learning to Bid with AuctionGym (2022) / Off-Policy Learning-to-Bid with AuctionGym. KDD 2023. amazon.science · doi.org/10.1145/3580305.3599877
22. Mondal, Kandregula, Agrawal & Sembium. OPTIMUS: Optimal Offline Bidding Strategy for Manual Targeting Advertising Campaigns. ECML-PKDD 2026 (per Amazon Science listing). amazon.science
23. Ren, Qin, Zheng, Yang, Zhang & Yu. Deep Landscape Forecasting for Real-time Bidding Advertising. KDD 2019. arXiv:1905.03028
24. Chen, Nabi & Siniscalchi. Advancing Ad Auction Realism: Practical Insights & Modeling Implications. AdKDD 2023 / Amazon Science. amazon.science · arXiv:2307.11732
25. Guo et al. AIGB: Generative Auto-bidding via Diffusion Modeling. KDD 2024. arXiv:2405.16141
26. Li et al. GAS: Generative Auto-bidding with Post-training Search. WWW 2025 (per code repository). arXiv:2412.17018
27. Cai et al. Real-Time Bidding by Reinforcement Learning in Display Advertising. WSDM 2017. arXiv:1701.02490
28. Mou et al. PE-MORL: Pessimistic Environment Model-based Offline RL for Auto-bidding. 2025 preprint. arXiv:2506.17919
29. Sokota, Hu, Wu, Kolter, Foerster & Brown. A Fine-Tuning Approach to Belief State Modeling. ICLR 2022. iclr.cc
30. Mou et al. Sustainable Online Reinforcement Learning for Auto-bidding. NeurIPS 2022. arXiv:2210.07006
31. Korenkevych et al. Offline Reinforcement Learning for Advertising. 2023 preprint / ACM 2024. arXiv:2310.09426
32. Peng et al. Expert-Guided Diffusion Planner for Auto-bidding. 2025 preprint (venue unverified). arXiv:2508.08687
33. Chen, Yuan, Ye, Majumder & Richardson. Put Your Money Where Your Mouth Is: Evaluating Strategic Planning and Execution of LLM Agents in an Auction Arena (AucArena). NeurIPS 2024 Open-World Agents workshop (withdrawn from ICLR 2024). arXiv:2310.05746
34. Cai et al. RTBAgent: A LLM-based Agent System for Real-Time Bidding. 2025 preprint. arXiv:2502.00792
35. Jiang, Xiong & Liu. HARBOR: A Testbed for LLM Agents in Auctions. 2025 preprint. arXiv:2502.12149
36. He et al. A Unified Solution to Constrained Bidding in Online Display Advertising. KDD 2021. doi.org/10.1145/3447548.3467199
37. Yang, Zuo & Kim. Constrained Auto-Bidding via Generative Response Modeling. KDD 2026. arXiv:2605.27811
38. Amazon. Amazon’s response to the FTC’s lawsuit regarding Sponsored Ads. Aug 2026. aboutamazon.com
39. Noam Brown, curriculum vitae and publication list. noambrown.com. The 2015-2023 entries above were cross-checked against it during the sweep that built this section; see also the PhD thesis, Equilibrium Finding for Large Adversarial Imperfect-Information Games (CMU 2020, AAAI/ACM SIGAI Doctoral Dissertation Award). noambrown.com/thesis.pdf
40. Feng, Podimata & Syrgkanis. Learning to Bid Without Knowing your Value. EC 2018. arXiv:1711.01333
41. Sokota, D’Orazio, Ling, Wu, Kolter & Brown. Abstracting Imperfect Information Away from Two-Player Zero-Sum Games. ICML 2023. arXiv:2301.09159; Sokota, Farina, Wu, Hu, Wang, Kolter & Brown. The Update-Equivalence Framework for Decision-Time Planning. ICLR 2024. arXiv:2304.13138

---

Part of A Day on Amazon Ads. The parent note is Amazon Ads, Deeply; the interactive companion is The Ad Auction, From the Inside.