Big Data

Kafka: Architecture and Patterns

LinkedIn published Kafka in 2011 with a clear mission: move the activity of 175 million users through dozens of services without losses or duplicates. Today Kafka processes more than 7 trillion messages per day at LinkedIn alone. At Uber, Netflix, and Airbnb, Kafka became the nervous system of the entire real-time infrastructure. Behind that scale sit four straightforward concepts: topics, partitions, consumer groups, and delivery semantics.

  • **LinkedIn** runs Kafka for the activity stream: 7+ trillion messages per day, a single event bus connecting 1,000+ microservices
  • **Uber** builds its real-time demand map via Kafka: GPS events partitioned by geohash and processed with sub-minute latency
  • **Confluent Cloud** manages millions of topics for enterprise customers: Schema Registry plus Kafka Connect turns Kafka into a central data integration hub

Topics and the Commit Log

In 2010, LinkedIn faced a specific problem: 300 services needed to receive user-activity events. Direct HTTP calls between services created n-squared connections, and the system collapsed under any spike. The solution was a **distributed event log**. Producers write to one end; consumers read independently at their own pace. That design became Kafka.

A **topic** is a named stream of messages - analogous to a table in a database. Messages inside a topic are stored as an append-only log: new ones are added at the tail, old ones are not deleted immediately (retention is by time or size). Each message carries an **offset** - a monotonically increasing position number within the log.

**Retention vs deletion:** Kafka does not delete messages on consumption (unlike RabbitMQ). Messages are removed only when the retention policy triggers (time or size). This allows multiple independent consumers to read one topic and enables historical replay - critical for debugging and event sourcing.

0

1

Sign In

What happens to a Kafka message after a consumer reads it?

Partitions: Horizontal Scaling

A single log means a single write stream. At one million messages per second, a single broker cannot keep up. **Partitions** are the solution: a topic is split into N independent logs, each hosted on a separate broker. Producers write to different partitions in parallel; consumers read in parallel.

Routing to a partition is determined by the **message key**: `partition = hash(key) % num_partitions`. This guarantees that all messages sharing a key (for example, the same user_id) land in the same partition - preserving order for that key. Messages without a key are distributed round-robin.

**Replication and leader election:** each partition has one leader broker and N-1 followers. All reads and writes go through the leader. If the leader fails, Kafka automatically elects a new leader from the ISR (In-Sync Replicas). With replication_factor=3, the cluster tolerates the loss of 2 out of 3 brokers without data loss.

Kafka guarantees message ordering:

Consumer Groups: Parallel Consumption

A topic with 12 partitions carries events for analytics: one service builds a real-time dashboard, another writes to a Data Lake, a third sends notifications. Each service must receive every event. A **Consumer Group** enables multiple services to read one topic independently, and allows each service to scale horizontally.

Within a Consumer Group, each partition is assigned to exactly one consumer - this enables parallel processing without duplication. Different Consumer Groups read the same topic independently and each receives every event.

**Rebalance:** when a consumer in a group crashes or a new one joins, Kafka runs a rebalance - redistributing partitions among live consumers. During a rebalance, processing stops (stop-the-world). Kafka 2.4+ supports Incremental Cooperative Rebalancing, where only affected partitions are redistributed while the rest continue processing.

A topic has 6 partitions. A Consumer Group contains 8 consumers. How many consumers will actively read data?

Exactly-Once Semantics

A payment service processes a transaction via Kafka. The message is received, the database is updated, but when committing the offset Kafka is temporarily unavailable. The consumer restarts and processes the message again - the transaction is doubled. This illustrates the **delivery semantics** problem: at-most-once (losses), at-least-once (duplicates), exactly-once (ideal, but complex).

**Kafka Streams** provides exactly-once semantics automatically with `processing.guarantee=exactly_once_v2`. Internally: a transactional producer + atomic offset commit + result written to an output topic. Without Kafka Streams, exactly-once requires either manual transaction management or idempotent business logic (deduplication by event_id checked in an external database).

Enabling enable_idempotence=True on the producer is sufficient for full exactly-once

An idempotent producer eliminates duplicates only on the write side (producer retries). Full exactly-once requires a transaction that atomically combines reading, processing, and writing - otherwise duplicates still occur on the consumer side.

Producer idempotence and exactly-once delivery are different levels of guarantee. A producer can reliably write a message exactly once, yet the consumer can still process it twice on restart without a transactional offset commit.

At-least-once semantics means that:

Kafka: Architecture and Patterns

  • Topic is an append-only event log; messages are not deleted on consumption, stored by retention policy
  • Partition is the unit of scaling and parallelism; ordering is guaranteed only within a partition
  • Consumer Group: N consumers share partitions among themselves; different groups read the topic independently
  • At-least-once (commit after processing) is the standard; exactly-once requires transactions or idempotent logic
  • Idempotent producer eliminates write-side duplicates; full exactly-once must be achieved with a transaction

Related topics

Kafka sits at the center of the real-time Big Data infrastructure:

  • Stream Processing: Flink and Spark Streaming — Kafka as the data source for stream processing engines
  • Data Lake architecture — Kafka to Delta Lake/Iceberg: the standard event ingestion pattern
  • Delta Lake, Iceberg, Hudi — Kafka Sink Connector for streaming writes into open table formats

Вопросы для размышления

  • Under what condition does increasing the number of partitions fail to speed up processing - and how is this diagnosed?
  • Why does exactly-once in Kafka not protect against duplicates when the consumer writes results to an external database without additional guarantees?
  • How should the partitioning key be chosen - by user_id or by event_type - and what determines the right answer?

Связанные уроки

  • ds-01-intro
Kafka: Architecture and Patterns