1 Small Language Models

This chapter cuts through the hype around large language models and provides a grounded overview of how today’s language technologies work and where small language models fit. It traces the roots of modern capabilities to the Transformer architecture and the shift to self-supervised training, which enabled broad skills such as understanding, generation, summarization, question answering, and basic reasoning. It explains key evolutions—encoder-only and decoder-only variants (BERT and GPT families) and the use of human feedback–driven reinforcement learning to shape conversational behavior—while emphasizing that SLMs and LLMs share the same foundations and differ primarily in scale and operational focus.

Small Language Models are defined as models with far fewer parameters (typically up to around ten billion), optimized for speed, memory, and energy efficiency so they can run on CPUs, consumer GPUs, edge devices, and on-prem servers with data kept local. Their deployment flexibility enables offline, embedded, and near–real time applications where privacy and latency matter. Crucially, SLMs can be specialized cost-effectively for domain tasks using transfer learning and parameter-efficient fine-tuning on public and private corpora—an approach well suited to regulated industries like healthcare, finance, manufacturing, chemistry, and biotech. Recent work also argues that agentic systems often benefit from SLMs for economy and responsiveness, sometimes alongside larger models in heterogeneous pipelines. A vibrant open-source ecosystem further lowers barriers and costs versus building from scratch, while improving sustainability due to smaller training and inference footprints.

The chapter also details the risks of closed, generalist LLMs—external deployment and potential data exposure, security and leakage concerns, opacity around models and training data, hallucinations, and code-related misuse—making them ill-suited for sensitive or highly regulated settings. Domain-specific models, privately deployed, can deliver higher accuracy, better intent alignment, and stronger governance. The chapter closes by outlining decision factors (task specificity, data sensitivity, compute budget) and previews the book’s practical focus: optimizing and quantizing SLMs for efficient inference, serving and deployment across diverse hardware, and integration with retrieval and agentic workflows—supported by hands-on code and minimal prerequisites.

Some examples of diverse content an LLM can generate.

The timeline of LLMs since 2019 (image taken from paper [3])

Order of magnitude of costs for each phase of LLM implementation from scratch.

Order of magnitude of costs for each phase of LLM implementation when starting from a pretrained model.

Ratios of data source types used to train some popular existing LLMs.

Generic model specialization to a given domain.

A LLM trained for tasks on molecule structures (generation and captioning).

• The definition of SLMs.
• Transformers use self-attention mechanisms to process entire text sequences at once instead of word by word.
• Self-supervised learning creates training labels automatically from text data without human annotation.
• BERT models use only the encoder part of Transformers for classification and prediction tasks.
• GPT models use only the decoder part of Transformers for text generation tasks.
• Word embeddings convert words into numerical vectors that capture semantic relationships.
• RLHF uses reinforcement learning to improve LLM responses based on human feedback.
• LLMs can generate any symbolic content including code, math expressions, and structured data.
• Open source LLMs reduce development costs by providing pre-trained models as starting points.
• Transfer learning adapts pre-trained models to specific domains using domain-specific data.
• Generalist LLMs risk data leakage when deployed outside organizational networks.
• Closed source models lack transparency about training data and model architecture.
• Domain-specific LLMs provide better accuracy for specialized tasks than generalist models.
• Smaller specialized models require less computational power than large generalist models.
• Fine-tuning costs significantly less than training models from scratch.
• Regulatory compliance often requires domain-specific models with known training data.

FAQ