A preference pair is deceptively simple: given a prompt, one answer was chosen and another was rejected. DPO asks the model to make the former more likely than the latter, while staying anchored to a reference model. The result is a loss that looks more like ordinary supervised learning than a separate reward-model-plus-RL pipeline.

THE OLD PIPELINE

In the classic RLHF recipe, pairwise preferences first train a reward model. A policy is then optimized against that learned reward, usually with a penalty for drifting too far from a reference policy. This can be powerful, but it introduces another model, an online optimization loop, and another set of failure modes to monitor.

DPO starts from the same kind of pairwise data but uses a useful observation: for the KL-regularized objective, the optimal policy and reward are linked by a log-probability ratio. Substituting that relationship into a preference model produces a direct objective over the language model itself.

THE LOSS, IN PLAIN ENGLISH

L_DPO = -E[log σ(β · ((log πθ(y+|x) − log πθ(y−|x)) − (log πref(y+|x) − log πref(y−|x))))]

The notation is less intimidating than it looks. For each prompt x, the model compares its likelihood of the chosen answer y+ with its likelihood of the rejected answer y−. It makes that gap larger than the equivalent gap under the reference model. The scalar β controls how aggressively that preference is enforced.

Put differently: DPO does not say “make this answer probable.” It says “make this answer win this matchup by more than it used to.” That relative framing is the whole point.

WHY IT IS PRACTICAL

  • One training loop. Preference pairs can be trained in batches without fitting an explicit reward model first.
  • A built-in anchor. The reference-policy term makes the update about improvement relative to the starting behavior, rather than unconstrained imitation.
  • Clear debugging surface. If a pair is wrong, ambiguous, or contaminated, it is directly part of the signal—there is no second model to obscure where that mistake went.

WHAT DPO DOES NOT FIX

DPO is only as good as the comparison it receives. It can learn a strong preference for a verbose answer, a stylistic convention, or a benchmark artifact if those are what the labels reward. The objective is not a truth detector, a process verifier, or a substitute for evaluations.

This is especially important for reasoning and agentic systems. Outcome preferences can improve the final answer while leaving a brittle trajectory underneath. In those settings, I treat DPO as a reliable baseline for converting high-quality preferences into a policy update—and pair it with environments and evaluations that can test the behavior we actually care about.

START HERE

Direct Preference Optimization: Your Language Model is Secretly a Reward Model ↗

Rafailov et al., 2023.