Event-Driven Architecture

Jay Kreps, LinkedIn Kafka, and the birth of modern EDA

2010: Greg Young formalizes CQRS and Event Sourcing in a series of articles and conference talks. The pattern existed in finance but received its first rigorous definition for software development. 2011: LinkedIn open-sources Kafka - Jay Kreps, Neha Narkhede, Jun Rao. The paper describes a log as a universal abstraction for all data in a company. 2013: Jay Kreps writes 'The Log: What every software engineer should know about real-time data's unifying abstraction' - an essay that changed how the industry thinks about event streams. Kafka became the infrastructure for event sourcing at scale: LinkedIn, Uber, Netflix, Airbnb.

A bank account shows a balance of USD 1500, but the customer insists it was USD 1800 yesterday. In a traditional database UPDATE overwrote the old value without a trace. What if the storage held not the sum but every transaction? Then any point in time becomes replayable - reproduce a bug, pass an audit, prove compliance.

• **LinkedIn (2011)** - Kafka as company-wide event bus: 700 billion events per day today
• **Axon Framework** - CQRS + Event Sourcing framework for Java: used at ABN AMRO, ING, Rabobank
• **EventStoreDB** - purpose-built database for event sourcing: fintech, insurance, healthcare
• **Confluent Platform** - commercial Kafka: CQRS projections via Kafka Streams and ksqlDB
• **Git** - event sourcing for code: every commit is an event, the current file = replay of all commits
• **Stripe** - event-driven API: every action generates a webhook event for integrations

Предварительные знания

• Batch vs Stream Processing

Events - facts of what happened

LinkedIn in 2011 could not handle the load. Services called each other in chains - ProfileService -> RecommendationService -> NotificationService. One slow service stalled the whole chain. Jay Kreps, Neha Narkhede, and Jun Rao built Kafka: services publish **events** to a log, other services read from it independently. No direct coupling. That is how the modern event-driven approach was born.

**Event** - an immutable fact of something that happened. OrderCreated, PaymentProcessed, ItemShipped. An event is named in the past tense - it already occurred and cannot be 'rejected'. Components subscribe to events and react autonomously without knowing about each other.

**Loose coupling** is the primary win. The publisher knows nothing about subscribers. A new analytics service connects to the Kafka topic with zero changes to OrderService. NotificationService going down does not affect InventoryService. Each consumer scales independently. Kafka at Netflix processes 700 billion events per day - exactly because of this decoupling.

Commands - requests for action

**Command** - a request to perform an action. CreateOrder, CancelSubscription, UpdateProfile. Unlike an event, a command **can be rejected**: insufficient funds, item out of stock, no permission. A command is addressed to a specific handler, not broadcast to all. The semantics are fundamental: command is an intention, event is the result.

**Event vs Command:** Event - what happened (fact, immutable, broadcast). Command - what is wanted (request, can be rejected, addressed to a specific handler). Flow: Command -> processing and validation -> Event (if successful).

In a well-designed system, command handlers are a thin layer of validation and coordination. All reactive logic - notifications, analytics, synchronization - lives in event handlers. A new subscriber requires zero changes to the command handler. This is what gives EDA elasticity: Uber adds a new trip analytics service and TripService needs zero modifications.

Event Sourcing

A traditional database stores **current state**: balance = 1500. **Event Sourcing** stores **all events**: Deposited(1000), Deposited(800), Withdrawn(300). Current state = replay of all events. Audit log is free - the event store is the source of truth. Greg Young formalized this pattern in 2010, inspired by... double-entry bookkeeping. The same principle has been used in accounting since the 14th century.

**Event Sourcing advantages:** 1. Full history - restore state to any point in time. 2. Audit trail for free. 3. Add new projections without touching historical data. 4. Debugging: reproduce a bug by replaying events. 5. Temporal queries: 'what was the balance yesterday at 14:23?'

Replaying billions of events can be slow. Solution: **snapshots** - periodically persist the current state. On restore: load the latest snapshot and replay only events after it. EventStoreDB does this automatically every N events.

CQRS

**CQRS** (Command Query Responsibility Segregation) - Greg Young and Udi Dahan, 2010. Separating read and write models. Commands (writes) go to the event store. Queries (reads) go to optimized read models (projections). The same events build different projections - each in a format suited for specific queries. Netflix builds hundreds of projections from a single Kafka event stream.

**Eventual consistency in CQRS:** Read models update asynchronously after an event is published. There is a delay between writing an event and its reflection in the projection - typically milliseconds, sometimes seconds under high load. For most use cases this is acceptable. For critical operations (account balance, inventory) a read-after-write pattern or versioning is needed.

CQRS and Event Sourcing are often used together but they are **separate patterns**. CQRS can be applied without ES: split read/write models in plain PostgreSQL. ES can be used without CQRS: a single unified model. Together they offer maximum flexibility - and maximum complexity. Axon Framework, EventStoreDB, and Confluent Platform build their ecosystems around this combination.

Key Ideas

• **Events** - immutable facts (OrderCreated): broadcast to all subscribers, cannot be rejected
• **Commands** - requests for action (CreateOrder): addressed to a specific handler, can be rejected
• **Event Sourcing** - storing all events instead of current state: full history, audit trail, temporal queries, snapshots for performance
• **CQRS** - separate models for write (event store) and read (projections): independent scaling, eventual consistency
• ES + CQRS combination is a proven pattern for finance and audit-heavy systems; overkill for simple CRUD
• Core ES contradiction: immutable events vs GDPR right to erasure - solved via crypto-shredding

Related Topics

EDA is an architectural pattern implemented through specific technologies:

• Message Brokers: Kafka vs RabbitMQ vs NATS — Brokers are the transport layer for events and commands
• Batch vs Stream Processing — Events are the foundation of stream processing; batch handles events in chunks
• Event-Driven Patterns — Saga, Outbox, DLQ - production patterns on top of EDA

Вопросы для размышления

• Git stores history as a chain of commits. Which patterns from this lesson can be seen in Git?
• How would one solve the GDPR right-to-erasure problem in an event sourcing system?
• In what scenarios is CQRS eventual consistency unacceptable? How would one compensate for it?

Связанные уроки

• stream-01 — Batch vs stream - foundation for understanding EDA
• stream-03 — Kafka, RabbitMQ, NATS - the transport layer for events and commands
• bt-17-event-driven — EDA patterns in practice: Saga, Outbox, DLQ
• bt-18-saga — Saga pattern for distributed transactions via events
• ds-02-cap-theorem — Eventual consistency in CQRS is a direct consequence of CAP
• isd-10-message-queues
• dist-07-transactions