1 Understanding reasoning models
Welcome to the next phase of large language models: systems that do more than produce fluent text and instead spend tokens on intermediate steps to solve harder problems. This chapter introduces the book’s hands-on approach to reasoning models, showing how they can improve performance on tasks like logic, math, coding, and agentic workflows that require planning, tool use, and recovery from mistakes. The central aim is to move from a general pre-trained model toward a smaller, more capable reasoning model by implementing the key methods from scratch.
In this context, reasoning means generating intermediate steps before the final answer, whether those steps are visible to the user or hidden behind special tags. This is often called chain-of-thought, but the book uses the term in an engineering sense rather than claiming human-like thought. The chapter contrasts this with conventional LLM behavior: ordinary models mostly rely on statistical pattern matching, while reasoning models are encouraged or trained to work through multi-step problems. Even so, LLM reasoning remains probabilistic and unlike deterministic rule-based systems, which follow explicit logic and can guarantee consistency.
The chapter also reviews the standard LLM pipeline of pre-training and post-training, explaining how next-token prediction on massive text corpora builds general language capability, and how instruction tuning and preference tuning refine that base model for helpful responses. It then introduces three broad ways to improve reasoning: inference-time compute scaling, reinforcement learning, and distillation. The chapter closes by motivating why building these methods from scratch matters: reasoning is powerful but costly, not always necessary, and best understood through careful implementation, evaluation, and trade-off analysis before applying it to real products or research.
A simplified illustration of how a conventional, non-reasoning LLM might respond to a question with a short answer.
A simplified illustration of how a reasoning LLM might tackle a multi-step reasoning task using a chain-of-thought. Rather than just recalling a fact, the model combines several intermediate reasoning steps to arrive at the correct conclusion. The intermediate reasoning steps may or may not be shown to the user, depending on the implementation.
Overview of a typical LLM training pipeline. The process begins with an initial model initialized with random weights, followed by pre-training on large-scale text data to learn language patterns by predicting the next token. Post-training then refines the model through instruction fine-tuning and preference fine-tuning, which enables the LLM to follow human instructions better and align with human preferences.
Example responses from a language model at different training stages. The prompt asks for a summary of the relationship between sleep and health. The pre-trained LLM produces a relevant but unfocused answer without directly following the instructions. The instruction-tuned LLM generates a concise and accurate summary aligned with the prompt. The preference-tuned LLM further improves the response by using a friendly tone and engaging language, which makes the answer more relatable and user-centered.
Three approaches commonly used to improve reasoning capabilities in LLMs. These methods (inference-compute scaling, reinforcement learning, and distillation) are typically applied after the conventional training stages (initial model training, pre-training, and post-training with instruction and preference tuning), but reasoning techniques can also be applied to the pre-trained base model.
Contradictory premises lead to a logical inconsistency. From "All birds can fly" and "A penguin is a bird," we infer "Penguin can fly." This conclusion conflicts with the established fact "Penguin cannot fly," which results in a contradiction.
An illustrative example of how a language model (GPT-4o in ChatGPT) appears to "reason" about a contradictory premise.
Token-by-token generation in an LLM. At each step, the LLM takes the full sequence generated so far and predicts the next token, which may represent a word, subword, or punctuation mark depending on the tokenizer. The newly generated token is appended to the sequence and used as input for the next step. This iterative decoding process is used in both standard language models and reasoning-focused models.
A high-level roadmap of what we build in this book. We start with a conventional LLM, add evaluation methods so that we can measure progress, and then explore two broad families of reasoning improvements, namely, inference techniques and training techniques.
A detailed roadmap of the chapter-level substeps. After loading the base model, we cover benchmark-based and judgment-based evaluation, then inference-time methods such as advanced text generation and voting plus self-refinement, and finally training-time methods based on reinforcement learning and distillation.
Summary
- Conventional LLM training occurs in several stages:
- Pre-training, where the model learns language patterns from vast amounts of text.
- Instruction fine-tuning, which improves the model's responses to user prompts.
- Preference tuning, which aligns model outputs with human preferences.
- Reasoning methods are applied on top of a conventional LLM.
- Reasoning in LLMs refers to improving a model so that it explicitly generates intermediate steps (chain-of-thought) before producing a final answer, which often increases accuracy on multi-step tasks.
- Reasoning in LLMs is different from rule-based reasoning and it also likely works differently than human reasoning; currently, the common consensus is that reasoning in LLMs relies on statistical pattern matching.
- Pattern matching in LLMs relies purely on statistical associations learned from data, which enables fluent text generation but lacks explicit logical inference.
- Improving reasoning in LLMs can be achieved through:
- Inference-time compute scaling, enhancing reasoning without retraining (e.g., chain-of-thought prompting).
- Reinforcement learning, training models explicitly with reward signals.
- Supervised fine-tuning and distillation, using examples from stronger reasoning models.
- Building reasoning models from scratch provides practical insights into LLM capabilities, limitations, and computational trade-offs.
FAQ
What does “reasoning” mean for an LLM in this chapter?
In this book, reasoning means that the model generates intermediate steps before giving its final answer. These steps may be shown to the user or hidden inside special tags like <think>...</think>.
How is a “reasoning model” different from a conventional LLM?
A reasoning model is an LLM that has been improved through prompting or training techniques so it can produce intermediate steps and handle complex tasks more accurately, such as coding, logic puzzles, and math problems.
What is chain-of-thought (CoT) reasoning?
Chain-of-thought is the style of generating intermediate reasoning steps before the final answer. It makes the model’s problem-solving process more explicit and easier to follow.
How do pre-training and post-training fit into the standard LLM pipeline?
Pre-training teaches the model general language patterns using next-token prediction on massive text data. Post-training then refines the model with instruction fine-tuning and preference tuning so it follows user requests better and aligns more closely with human preferences.
What are the main ways to improve reasoning in LLMs?
The chapter introduces three broad approaches: inference-time compute scaling, reinforcement learning, and distillation. These methods can help a model reason better without changing the basic idea of next-token generation.
How is pattern matching different from logical reasoning in LLMs?
Pattern matching means the model produces likely continuations based on statistical associations in training data. Logical reasoning means drawing conclusions systematically from premises and rules, often using intermediate steps and checking for contradictions.
Why doesn’t an LLM use the same kind of deterministic reasoning as a symbolic logic engine?
Unlike a symbolic logic engine, an LLM generates text autoregressively, one token at a time, based on learned probabilities. That means its reasoning-like steps are not guaranteed to be logically sound or consistent.
Why can a conventional LLM sometimes appear to reason correctly?
A conventional LLM can sometimes simulate reasoning well because it has seen many similar patterns in training data. It may produce the right answer without explicitly applying rule-based logic.
Why are reasoning models often more expensive to use?
They tend to produce longer outputs because they include intermediate steps, and they may require multiple model calls for sampling, tool use, or verification. Both factors increase compute cost and latency.
Why is this book focused on building reasoning models from scratch?
Implementing the methods from scratch is one of the best ways to understand how they work. The book aims to help readers design, prototype, evaluate, and improve reasoning methods, especially for practical tasks and agent workflows.
Build a Reasoning Model (From Scratch) ebook for free
