Kafka: Architecture

2010. LinkedIn processes 1 billion events per day. Oracle and ActiveMQ are buckling. Every service connects directly to every other - a spaghetti of 12-way integrations, each with its own schema and retry logic. Jay Kreps, Neha Narkhede, and Jun Rao build Kafka internally. Named after Franz Kafka - 'a system optimized for writing.' Open-sourced in 2011. By 2020, LinkedIn's Kafka cluster handles 7 trillion messages per day. Uber, Netflix, Airbnb, Stripe - all running on the same append-only log idea.

• **Uber**: billions of GPS events per day from drivers → Kafka → real-time surge pricing computed in under 1 second
• **Netflix**: play/pause/seek activity events → Kafka → recommendation engine updates every few minutes
• **Banks**: every transaction → Kafka → fraud detection pipeline with sub-100ms decisions
• **LinkedIn**: change data capture (CDC) from PostgreSQL → Kafka → all downstream services notified without polling

The log as the universal abstraction

In 2011 Jay Kreps published 'The Log: What every software engineer should know about real-time data's unifying abstraction' - a blog post that reframed distributed systems thinking. The insight: an append-only log is the universal primitive. Databases use logs for durability (WAL). Kafka makes the log the product. Messages write only to the end, never modified. This enables horizontal scaling via partitioning and replication while preserving ordering guarantees per partition. In 2014 Kreps, Narkhede, and others founded Confluent to commercialize Kafka. Confluent reached a $4.5B valuation in 2019. Kafka changed enterprise architecture the way MapReduce changed batch processing - by naming and exporting a primitive that was previously buried inside systems.

Topics and Partitions

**Topic** is a named stream of messages. **Partition** is the physical storage unit - an append-only, immutable log. Every message in a partition gets a unique **offset**: its absolute position in that log. Offsets never repeat, never decrease.

**Choosing partition count:** More partitions means more parallelism and higher throughput. The cost: more file descriptors per broker, longer leader election during failover, more metadata overhead. Rule of thumb: start at throughput / 100 MB/s, multiply by a safety factor. Most topics: 12-48 partitions.

Replication and ISR

Each partition is replicated across multiple brokers. The **Leader** handles all reads and writes. **Followers** replicate data from the Leader. **ISR** (In-Sync Replicas) is the subset of replicas that aren't lagging behind the Leader - only ISR members are eligible to become the new Leader after a failover.

**Leader election during failover:** When a Leader crashes, Kafka elects a new one from the ISR. With `unclean.leader.election.enable=false`, only in-sync replicas qualify - guaranteeing the new Leader has all acknowledged messages. Failover takes seconds, not minutes.

Consumer Groups and Offset Management

A **Consumer Group** is a set of consumers that collectively process partitions from a topic. Each partition is assigned to exactly one consumer in the group. Different groups read the same topic independently - each group tracks its own offsets.

Log Retention and Performance

Kafka stores messages on disk in segment files. Messages are not deleted after consumption - only by retention policy. This is the fundamental architectural difference from traditional message queues.

**Log Compaction vs Delete:** Delete retention removes old segments entirely. Compaction (for keyed topics) retains only the latest value per key - like a materialized view. Used for event sourcing, CDC, and configuration topics where only the current state matters.

Key Ideas

• **Topic + Partition**: a topic splits into partitions - the units of parallelism and scalability. More partitions = higher throughput.
• **Offset**: a message's immutable position within a partition. The basis for replay, consumer tracking, and exactly-once semantics.
• **Replication + ISR**: each partition has a Leader and Followers. In-Sync Replicas are eligible for leader election. `acks=all` + `min.insync.replicas=2` gives CP guarantees.
• **Consumer Group**: multiple consumers share partitions of a topic. Maximum parallelism = partition count. Different groups read independently.
• **Log Retention**: messages persist by policy, not by consumption. Replay from any offset - impossible in RabbitMQ, fundamental in Kafka.

What's next

Architecture understood. Now process the streams.

• Kafka Streams — Stateful stream processing built on partitions, offsets, and consumer groups
• Exactly-Once Semantics — Kafka transactions and idempotent producers for guaranteed delivery

Вопросы для размышления

• Kafka guarantees ordering within a partition but not across partitions. How does that constraint shape partition key design for bank transactions?
• ISR with acks=all guarantees durability but adds latency. In which scenarios is acks=1 an acceptable trade-off?
• Consumer Groups enable horizontal scale, but max parallelism equals partition count. Why can't there be more active consumers than partitions?

Связанные уроки

• stream-03 — Streaming fundamentals - producers, consumers, and message delivery semantics
• stream-05 — Kafka Streams and KSQL operate on top of the partition/offset/consumer-group model
• ds-02-cap-theorem — Kafka's ISR and acks settings are explicit CAP trade-off levers
• db-34-lsm — Both Kafka segments and LSM-Tree SSTables exploit sequential writes for maximum disk throughput
• stream-06 — Exactly-once semantics and transactions build on the partition/offset foundations covered here
• dist-11-replication