1 Introducing event streams

Businesses can be viewed as generators and responders to a continuous flow of events. While people think in terms of tasks, systems, and teams, computers thrive on modeling the world as event streams. Framing operations this way unlocks fresher insights, a single version of the truth, faster reactions, and simpler architectures by replacing ad hoc integrations with well-modeled streams.

An event is a discrete occurrence at a specific point in time; it’s not an ongoing state, a generic recurring phenomenon, a collection of events, or a duration. A continuous event stream is an unterminated, time-ordered succession of such events: it may begin before we start observing and extends into the future. Many familiar tools already work this way—application logs produce time-stamped events, web analytics tags emit interaction events, and publish/subscribe messaging distributes event messages by topic—though their schemas and uses are often siloed and inconsistent.

The chapter traces an evolution across three eras. In the classic era, on-prem transactional systems fed a batch-loaded data warehouse: reliable, but plagued by high latency, point-to-point “spaghetti,” and rigid modeling assumptions. The hybrid era added SaaS platforms, Hadoop “log everything” stacks, and selective low-latency pipelines, but created new issues: no single source of truth, fragmented decision loops, proliferating brittle integrations, and a tradeoff between latency and coverage.

The unified era centers on a unified log—an append-only event log read at low latency by multiple consumers at their own pace, retaining a rolling window with historical archives elsewhere. All systems write their event streams to this log and, unless strict transactional guarantees are needed, communicate through it rather than via bespoke connections. This yields a practical single version of the truth upstream of warehouses, dramatically fewer point-to-point links, and “unbundled” local loops where applications collaborate through shared streams.

• Fresher insights: stream processing reduces time to analysis and action from hours to seconds.
• Single truth: the unified log plus archive align operational and analytical views on the same data.
• Simpler architectures: write-once, read-many streams replace bespoke integrations.
• Use cases: real-time customer feedback loops, holistic systems monitoring across all signals, and hot-swapping data application versions for zero-downtime upgrades and A/B comparisons.

Overall, adopting event streams and a unified log turns fragmented, latency-prone data flows into a coherent, real-time fabric that supports consistent analytics and rapid, intelligent reactions across the business.

1.5   Summary

FAQ