RLHF can be read as a game. Five candidate answers compete in pairwise comparisons, and a policy is a mixed strategy over them. Train it the reward-model way (fit Bradley–Terry scores, then take a KL-regularized softmax), or the game way (run a no-regret learner in self-play until its average reaches a Nash equilibrium). Edit the preferences, blend in an AI judge, and compare what each method ships.
Nash mix of the current game
“Explain to a 10-year-old why the sky is blue.”
Cell (row, column) is P(row ≻ column). Edit the upper triangle; the lower triangle is 1 − p.
Both methods train on (1 − λ)·human + λ·judge. The comparison table below also scores every policy against the human preferences, so you can see what a biased judge costs.
Fit a score s(y) so that P(y ≻ y′) ≈ σ(s(y) − s(y′)), then solve max E[s] − β·KL(π‖πref). The closed form is one entropy-regularized FTRL step, i.e. MWU with η = 1/β. Def. L5.7 · Alg. L5.3
Bar: πβ. Tick: πref (the base model). Right column: πβ and the fitted score s.
The policy plays against a copy of itself. Each round it observes how every response would fare against its current mix (canonical setup, Def. L4.8). Because M is antisymmetric, the saddle-point gap of the average equals 2·Reg(T)/T exactly Thm. L4.9. Any sublinear-regret learner therefore drives the average to the von Neumann winner.
MWU only. Regularized mirror descent (as in NLHF). Anchoring settles the last iterate but moves the target away from the exact Nash.
“Best reply wins” is the win rate of the strongest single response against the policy: maxy P(y ≻ π). A Nash policy scores exactly 50%, meaning nothing beats it. Higher means the policy can be exploited.
Load the cyclic human preset. No score vector can reproduce A ≻ B ≻ C ≻ A, so the BT fit error jumps and a best reply beats πβ more than half the time. Lower β and πβ concentrates further, getting more exploitable. The Nash mix stays balanced across the cycle.
Select FTL. Its last iterate jumps from one pure response to the next as the leader changes, which is fictitious play (Remark L5.1). Its average still converges in this zero-sum game, just without FTL having a worst-case guarantee.
With MWU and the cyclic preset, the last iterate spirals around the equilibrium while the average settles. Increase the KL anchor and the last iterate settles too, at a regularized point pulled toward πref.
Push the RLAIF mix toward 100%. The length-biased judge rewards the padded answer D. Watch the “vs human” column as both methods drift toward D. Self-play only optimizes the labels it is given.
| Lecture | In the lab | In LLM alignment |
|---|---|---|
| Remark L5.1 · FTL | Best-respond to the cumulative payoff. The last iterate jumps between pure responses. | Repeatedly optimizing against the latest reward model or opponent. Unstable when preferences cycle. |
| Alg. L5.1–L5.2 · RM, RM+ | No hyperparameters. RM+ clips regrets at zero and usually converges fastest here. | Hyperparameter-free baselines for small preference games and for evaluation. |
| Alg. L5.3 · MWU | Softmax of cumulative regret. Bound 2√(T log|A|) scales with log|A|. | The KL-regularized RLHF optimum π ∝ πref·exp(r/β). SPPO-style updates are MWU on the preference game. log|A| is what makes huge response spaces tractable. |
| Def. L5.8 · OMD | MWU with an extra KL pull toward πref (the anchor slider). | Iterative / online RLHF and PPO with a KL penalty to the previous policy. NLHF uses regularized mirror descent. |
| Def. L5.15 · OGD | Euclidean projection onto the simplex. Bound √(2T|A|). | The Euclidean baseline. It hits the simplex boundary and zeroes out responses abruptly. |
| Thm. L4.9 (S04) | Gap of x̄ = 2·Reg(T)/T, shown live. | Why self-play preference optimization targets a Nash / minimax winner, not a single reward maximizer. |
Further reading: Munos et al., Nash Learning from Human Feedback (2023); Swamy et al., A Minimaximalist Approach to RLHF (2024); Wu et al., Self-Play Preference Optimization (2024); Rosset et al., Direct Nash Optimization (2024).