1 Introduction

Reinforcement Learning from Human Feedback (RLHF) is introduced as a way to incorporate human preferences into AI systems, especially for tasks where goals are hard to specify. It rose from early control settings to mainstream prominence with conversational language models and now sits as a central component of post-training—the multi-stage process that turns a pretrained base model into a helpful assistant. At a high level, the RLHF pipeline consists of training an instruction-following model, fitting a reward model to human preference data, and optimizing the policy with reinforcement learning; together these stages align behavior to what people actually want and frame the book’s practical decisions and implementation recipes.

Beyond enabling reliable question-answering, RLHF primarily shapes a model’s style and higher-level behavior—making answers concise, warm, reliable, safer, and better formatted—while also improving generalization relative to instruction fine-tuning alone. Conceptually, instruction tuning adjusts token-by-token predictions toward desired features, whereas RLHF evaluates whole responses and learns from comparisons of better versus worse outputs (a contrastive signal), including negative feedback. This flexibility brings challenges: training robust reward models from noisy data, avoiding over-optimization of proxy rewards, managing biases such as length, and budgeting the added compute and data costs. Consequently, RLHF works best alongside other post-training stages and depends on a strong base model.

The chapter places RLHF in the broader trajectory of open and closed model development: early instruction-only recipes captured “vibes” but plateaued; direct preference optimization simplified preference learning and spread widely; and newer reinforcement-learning approaches with verifiable rewards are pushing reasoning and agentic capabilities while consuming growing compute. Framed by an elicitation view, post-training extracts latent ability from base models—much like refining a car around a fixed chassis—often yielding large gains without changing pretraining. Looking ahead, RLHF remains the core of preference fine-tuning and a bridge to broader RL methods for reasoning, with the field converging on multi-stage post-training as the route to state-of-the-art systems.

A rendition of the early, three stage RLHF process: first training via supervised fine-tuning (SFT, chapter 4), building a reward model (RM, chapter 5), and then optimizing with reinforcement learning (RL, chapter 6).

• RLHF incorporates human preferences into AI systems to solve problems that are hard-to-specify programmatically, and became widely known through ChatGPT’s breakout, which made the capabilities of language models more approachable.
• The basic RLHF pipeline has three steps: instruction fine-tuning to teach the model to follow the question-answering format, training a reward model on human preferences, and optimizing the model with RL against that reward.
• RLHF is known to primarily change the style, tone, and format of model responses – making them more helpful, warm, and engaging. But it’s not “just style transfer”: RLHF also improves benchmark performance, though over-optimization (e.g., excessive length or chattiness) can harm capabilities in other domains.
• The elicitation theory of post-training suggests that base models contain latent potential, and post-training’s job is to extract and cultivate that intelligence into useful behaviors.
• RLHF is one component of modern post-training, alongside instruction fine-tuning (IFT/SFT) and reinforcement learning with verifiable rewards (RLVR), used together in an intertwined manner to craft particular training recipes.

FAQ