1 Introduction

This chapter introduces the book’s practical mission: to help you build low‑latency applications with clear techniques, tools, and mental models. It defines latency as the time delay between a cause and its observed effect, sets expectations for how to think about and measure it across different contexts, and contrasts it with related concepts like throughput and bandwidth. At a high level, the chapter explains why latency matters for user experience, real‑time requirements, and efficiency, and previews the trade‑offs you’ll encounter—including those between latency and throughput, and between latency and energy.

Grounding the definition, the chapter walks through concrete examples that reveal how latency compounds and varies: flipping a (smart) light switch, serving an HTTP request from DNS lookup to page rendering, and moving a network packet from the NIC through the kernel to user space. These scenarios show that end‑to‑end delay is the sum of many small delays across layers, that variance matters as much as averages, and that context determines what you measure. The chapter also frames latency in units that span nanoseconds to milliseconds across the memory, storage, and networking hierarchy, noting that physical limits (including the speed of light) bound what’s achievable and that intuition at microsecond and nanosecond scales often needs deliberate calibration.

The motivations for low latency are threefold. For user experience, people perceive sub‑100 ms interactions as instant, tolerate about a second with minimal friction, and need feedback mechanisms for multi‑second operations—latency reductions correlate with higher engagement and conversion. For real‑time systems, hard deadlines must never be missed, while soft real‑time systems can occasionally miss them without catastrophic failure. For efficiency, the end of free speedups from frequency scaling shifts focus to smarter software (echoing observations about multicore trends and software bloat). Finally, the chapter clarifies latency versus bandwidth and throughput, illustrates the classic latency–throughput trade‑off via pipelining, and highlights latency–energy tensions (for example, busy polling versus sleep–wake), emphasizing that the right choices depend on workload patterns and goals.

60 ms Length of a nanosecond. Source: https://americanhistory.si.edu/collections/search/object/nmah_692464

Processing without pipelining. We first perform step W (washing) fully and only then perform step D (drying). As the time to complete W is 30 minutes and the time to complete D is 60 minutes, each step takes 90 minutes in total. Therefore, we say that the latency to wash and dry clothes is 90 minutes and the throughput is 1/90 loads of laundry washed per minute.

Processing with pipelining. We perform step W (washing) in full, but as soon as it completes, we start another step W. In parallel, we perform step D (drying) for the previous step W. If we ignore the initial step where there is no completed step W, the time to complete a load of laundry is 120 minutes because W and D run in parallel, but we’re bottlenecked by D, making latency worse than without pipelining. However, due to pipelining, we have now increased throughput to 1/60 loads of laundry per minute, which means that we can complete four loads of laundry in the same time as non-pipelined does three.

• Latency is the time delay between a cause and its observed effect.
• Latency is measured in units of time.
• You need to understand the latency constants of your system when designing for low latency.
• Latency matters because people expect a real-time experience.
• When optimizing for latency, there are sometimes throughput and energy efficiency trade-offs.

FAQ