1 Peeking inside the black box

Generative AI applications feel familiar when we use tools like ChatGPT, but building them reveals a very different kind of programming. Unlike traditional software, where code defines a deterministic sequence of steps, GenAI systems must handle natural-language inputs and non-deterministic outputs from a language model. The chapter explains that successful GenAI applications are not just “LLM in, answer out” systems; they require orchestration around the model so the application can preserve context, inject relevant knowledge, trigger actions, and shape behavior for real-world use.

The chapter’s main idea is that an LLM is a powerful but limited black box: it is probabilistic, stateless, not inherently explanatory, and trained on fixed knowledge. To make an application seem conversational or informed, the developer must supply conversation history, relevant documents, or other external context each time the model is called. This is how systems like ChatGPT and Copilot appear to remember users or know company-specific information—by combining the LLM with memory, retrieval, tools, and other components that extend what the model can do on its own.

The chapter then peeks under the hood of an LLM by describing it as a text-completion engine built from machine learning stages. In a simplified view, text is tokenized, converted into embeddings, transformed into context, and used to predict the most likely next token; in a real model like GPT-3, these steps are implemented with massive neural networks, transformers, and extensive pre-training on huge datasets. The chapter closes by emphasizing that model knowledge is static unless refreshed through techniques such as prompting, retrieval, or fine-tuning, and that the rest of the book will teach how to combine these pieces into practical, context-rich GenAI applications.