Redis Streams and Pub/Sub

In 2018, Antirez added Streams to Redis and wrote: 'I wanted to create something like Kafka, but something that could live inside Redis.' The result was not Kafka - but for 80% of cases where Kafka is overkill, Redis Streams became the right choice.

• BullMQ - the most popular job queue for Node.js - uses Redis to store jobs, and newer versions support Streams as a backend
• Instagram uses Redis for activity feed fan-out: a post publication is an event in a Stream, workers distribute it to feeds of 300+ million followers
• Cloudflare uses Redis Streams for edge events: rate limiting, firewall events, analytics pipeline

Redis Pub/Sub: ephemeral messaging

Redis Pub/Sub has existed since 2010. It is the simplest fire-and-forget: PUBLISH sends a message to all active channel subscribers. No persistence, no buffering, no ACK. If a subscriber is offline the message is lost permanently.

Pub/Sub in Redis works through a dedicated connection - a subscribed client cannot run other commands. Redis keeps the subscriber list in memory. One PUBLISH triggers O(N) sends, where N is the number of subscribers. At 10,000 subscribers on a single channel, this becomes a bottleneck.

When Pub/Sub is sufficient: real-time notifications to browsers via WebSocket (BullMQ uses it internally), chat messages when some loss is acceptable, cache invalidation in multi-node setups. When Pub/Sub is not enough - use Streams.

Redis Streams: append-only log in memory

Redis Streams arrived in 2018 (Redis 5.0). Antirez (Salvatore Sanfilippo) said openly: the goal was something like Kafka, but living inside Redis. Streams are an append-only log with ID-based addressing and built-in consumer group support.

Each message in a Stream has a unique ID in the format `milliseconds-sequence`. XADD with ID `*` auto-generates a monotonically increasing ID. This guarantees ordering - an important property for an event log.

A Stream is stored in memory in a radix tree - an efficient structure for range queries by ID. `XRANGE orders from to` runs in O(log N + M). Data can be persisted to disk via RDB/AOF like any other Redis data. `MAXLEN` with tilde (~) is an approximate trim - faster than an exact one.

Redis Streams are not a Kafka throughput replacement. Redis is single-threaded per command (though I/O is multi-threaded since Redis 6). Kafka is partitioned and horizontally scalable. Redis Streams are for moderate loads where Redis is already in the stack.

Consumer Groups: distributed processing

Consumer Groups in Redis Streams are the direct analog of Kafka Consumer Groups. A group of consumers splits the Stream: each message is delivered to exactly one consumer in the group. Different groups read independently - each sees all messages.

Pending Entries List (PEL) is the key concept. Redis stores all delivered but unacknowledged messages in the PEL. If a consumer crashes - its pending messages can be reclaimed via XCLAIM. This is the at-least-once guarantee in Redis Streams.

BullMQ - the most popular job queue library for Node.js - is built on Redis. It uses a combination of List (for queuing) and Hash (for job data). BullMQ v4+ adds Streams support as an alternative backend.

Redis Streams vs Kafka: a pragmatic choice

Redis is already in 80% of production stacks - for caching, sessions, rate limiting. Redis Streams means messaging without an additional tool. Kafka is a separate infrastructure with JVM, ZooKeeper or KRaft, and operational overhead.

Redis Streams loses to Kafka when: long-term retention is needed (event sourcing), horizontal throughput scaling is required, Kafka Connect integration matters (CDC from PostgreSQL), or stream processing with Kafka Streams is needed. Streams win when: Redis is already in the stack, load is under 100K msg/s, operational simplicity matters, or sub-millisecond latency is critical.

Redis Streams usage patterns

Activity feed is the classic use case. Instagram, Twitter, TikTok build feeds through fan-out. Redis Streams fits as the intermediate layer: an event is published to the Stream, workers fan it out into Redis Sorted Sets (each follower's feed).

Audit log is another strong fit. User actions are written to a Stream with MAXLEN to retain the last N events. Compliance reads through a separate Consumer Group. A real-time dashboard uses a third group. Three consumers, one Stream, no additional infrastructure.

Rate limiting plus Streams: a popular pattern. XADD with time-based MAXLEN creates a sliding window of events. XLEN then gives the event count for the period. Combining Redis Streams with Sorted Sets enables sophisticated rate limiters without Kafka.

Related Topics

Redis Streams in the context of messaging and data storage:

• Kafka — Analog with partitioning for high throughput
• Redis as a Datastore — Basic Redis data structures
• NATS — Alternative lightweight messaging
• Event-Driven Architecture — Streams as an EDA backbone

Key Ideas

• Redis Pub/Sub is ephemeral with no persistence. If a subscriber is offline, the message is lost
• Redis Streams are an append-only log with ID addressing, Consumer Groups, and PEL for at-least-once delivery
• XADD/XREADGROUP/XACK is the triad for a producer-consumer pattern with delivery guarantees
• Redis Streams vs Kafka: choose Streams when Redis is already in the stack and load is moderate

Вопросы для размышления

• When is Redis Pub/Sub preferable to Streams despite lacking delivery guarantees?
• How should MAXLEN for a Stream be chosen: by message count or by time?
• Can Redis Streams replace a task queue like BullMQ, or are these different levels of abstraction?

Связанные уроки

• bt-13-kafka — Kafka is the reference for comparing with Streams
• bt-15-nats — NATS - analogous lightweight messaging
• db-19-redis — Redis as a datastore - foundation for understanding Streams
• bt-11-messaging-intro — Messaging fundamentals before Redis Streams
• bt-17-event-driven — Redis Streams as backbone for event-driven systems
• net-55-message-queues