1 Introduction

RLHF emerged as a practical way to bring human preferences into AI systems, especially when the desired behavior is hard to specify directly. It gained major prominence with the rise of ChatGPT and large language models, but its roots extend back to earlier reinforcement learning applications. The chapter presents RLHF as part of a broader post-training toolkit for turning pretrained models into useful assistants, emphasizing that it is now one component of a larger family of methods rather than the whole story.

The core RLHF workflow is presented as a three-stage process: first, instruction fine-tuning teaches a model to answer in a question-and-answer format; second, a reward model is trained on human preference data to estimate which responses are better; and third, reinforcement learning uses that reward signal to improve the model’s outputs. The key intuition is that instruction tuning builds the basic structure of responses, while RLHF refines the model’s behavior at the response level, steering it toward answers that are more helpful, reliable, warm, and engaging. This makes RLHF especially valuable for shaping style and subtle preferences that are difficult to capture with simple supervised learning alone.

The chapter also argues that post-training can unlock much of a base model’s latent capability, much like refining a car chassis into a high-performance race car. While pretraining provides the raw potential, post-training makes that potential accessible in a form suited to interaction with users. The text acknowledges that RLHF is costly, complex, and vulnerable to issues like proxy rewards and over-optimization, but maintains that it remains central to modern model development. The book’s purpose is to explain these methods clearly, show how they fit together, and help readers understand both the mechanics and the broader significance of RLHF in the evolution of AI.