1 Introduction

Reinforcement Learning from Human Feedback (RLHF) integrates human preferences into AI systems to solve objectives that are hard to specify directly, especially in interactive settings. It rose to prominence with chat assistants and is now a core part of post-training—the suite of methods applied after large-scale pretraining to make models useful, safe, and engaging. Within post-training, instruction/supervised fine-tuning teaches format and basic behaviors, preference fine-tuning (where RLHF dominates) aligns models to subtle human values and style, and reinforcement learning with verifiable rewards targets domains with objective signals. Framed this way, post-training is about eliciting and shaping the latent capabilities of a pretrained base model into reliable, user-centered performance.

The chapter outlines the standard RLHF pipeline—train an instruction-following model, collect preference data to build a reward model, then optimize the policy against that reward (or learn directly from preferences via “direct alignment” methods). RLHF operates at the response level with contrastive signals, guiding models toward better answers and away from worse ones. This shifts outputs from unfocused next-token continuations to concise, helpful, well-formatted, and often warmer responses, and it tends to generalize more broadly than instruction tuning alone. The approach is powerful but delicate: reward models are proxies, optimization can overfit (e.g., to length), and success depends on strong starting models, high-quality data, and careful regularization—making RLHF more complex and costly than simple fine-tuning.

Historically, open efforts first leaned on instruction tuning before skepticism about RLHF gave way to widespread adoption, including simplified preference-optimization methods such as Direct Preference Optimization. Meanwhile, large closed labs advanced multi-stage post-training pipelines at scale, highlighting that effective systems combine instruction tuning, RLHF, prompt and format design, and increasingly, verifiable-reward RL and reasoning-focused training. The chapter situates RLHF as both a mature pillar of preference fine-tuning and a bridge to newer reinforcement-learning approaches, while emphasizing practical goals: clarify trade-offs, provide starting points for implementation, and equip readers with the intuition and tools to contribute to modern post-training in research and industry.

A rendition of the early, three stage RLHF process: first training via supervised fine-tuning (SFT, chapter 4), building a reward model (RM, chapter 5), and then optimizing with reinforcement learning (RL, chapter 6).

• RLHF incorporates human preferences into AI systems to solve problems that are hard-to-specify programmatically, and became widely known through ChatGPT’s breakout, which made the capabilities of language models more approachable.
• The basic RLHF pipeline has three steps: instruction fine-tuning to teach the model to follow the question-answering format, training a reward model on human preferences, and optimizing the model with RL against that reward.
• RLHF is known to primarily change the style, tone, and format of model responses – making them more helpful, warm, and engaging. But it’s not “just style transfer”: RLHF also improves benchmark performance, though over-optimization (e.g., excessive length or chattiness) can harm capabilities in other domains.
• The elicitation theory of post-training suggests that base models contain latent potential, and post-training’s job is to extract and cultivate that intelligence into useful behaviors.
• RLHF is one component of modern post-training, alongside instruction fine-tuning (IFT/SFT) and reinforcement learning with verifiable rewards (RLVR), used together in an intertwined manner to craft particular training recipes.

FAQ