Event Streaming

Updated September 30, 20263 min read

In short

Event streaming is the practice of recording events as a continuous, ordered and durable log that many applications can read, replay and process in real time.

What is event streaming?

Event streaming treats everything that happens in a system, such as a click, a payment, a sensor reading, or a database change, as an event, and records those events in order as they occur. The events are kept in a durable, append-only log, and any number of applications can read that log, either the moment events arrive or later, from any point in its history.

Platforms such as Apache Kafka and Apache Pulsar organize events into topics, which are split into partitions so they can be spread across servers and processed in parallel. Order is guaranteed within a partition, so events with the same key, such as one customer's ID, stay in sequence. Each consumer tracks its own position in the log, called an offset, which means a new service can start from the beginning and replay months of history, and a buggy consumer can rewind and reprocess. Events are kept for a configured retention period, from hours to forever, rather than being deleted once they are read.

Event streaming is like a security camera recording rather than a doorbell: a doorbell only alerts whoever is home right now, while a recording can be watched live or rewound later by anyone who needs it. It is used for real-time analytics, fraud detection, activity tracking, copying data between databases and data warehouses through change data capture, and connecting microservices in an event-driven architecture. Stream processing tools such as Kafka Streams and Apache Flink read streams continuously to filter, join, and aggregate events as they flow.

Event streaming is often confused with message queues and event sourcing. A traditional message queue deletes a message once a consumer acknowledges it, and each message goes to one worker, while an event stream keeps events so many consumers can read them independently and replay them. Event sourcing is an architectural pattern in which an application stores its state as the sequence of events that produced it; it often uses an event log, but event streaming is the broader infrastructure for moving and processing events across a whole organization.

Key takeaways

Events are recorded in order in a durable, append-only log.
Many consumers read the same stream independently, each tracking its own offset.
Streams can be replayed from any point within the retention period.
Partitions allow parallel processing while keeping order for each key.
Unlike in a queue, reading an event doesn't remove it.

Example

Writing and replaying a stream with Kafka's command-line toolsbash

# Create a topic with 3 partitions
kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic page-views --partitions 3

# Write an event to the stream
echo '{"user": 42, "page": "/pricing"}' | \
  kafka-console-producer.sh --bootstrap-server localhost:9092 --topic page-views

# Read the stream from the very first event (replay)
kafka-console-consumer.sh --bootstrap-server localhost:9092 \
  --topic page-views --from-beginning

Readers ask

What is the difference between event streaming and a message queue?

A message queue hands each message to one consumer and deletes it once processed. An event streaming platform keeps events in an ordered log for a retention period, so many consumers can read the same events independently and replay them later.

Is Kafka a database?

Not in the usual sense. Kafka stores events durably and can keep them indefinitely, but it is designed for appending and reading streams in order, not for ad hoc queries or updating individual records, so it usually runs alongside databases.

What is stream processing?

Stream processing is continuous computation over events as they arrive, such as counting page views per minute or flagging suspicious payments within seconds. It contrasts with batch processing, which runs periodically over data that has already been stored.