Redis Streams

Pub/Sub in Redis works like radio: if no one is listening, the signal is gone. Redis Streams is a tape recorder with rewind: every message is on tape, you can re-read, and if the handler crashed, the data is safe.

• **Payment service**: orders are added to the `orders` stream, three workers read via Consumer Group. Each payment is processed exactly once, even if a worker restarts.
• **User action audit log**: every event is written to a stream. An analytics service reads them independently of the main handler. Both get the full picture without duplicating logic.
• **IoT telemetry**: thousands of sensors push readings, MAXLEN trims the stream to the last 24 hours, several consumers in parallel aggregate data by different metrics.

Redis Streams

Redis Streams is a persistent message log built right into Redis. Each message gets a unique ID of the form `1699000000000-0` (timestamp-sequence), and the log does not disappear on read, unlike regular Pub/Sub.

`XADD` adds a record. The `*` field means auto-generated ID. `XREAD` reads from a position: from the beginning, from a specific ID, or only new records (`>`). This lets several services read the same stream independently.

Redis Streams vs Kafka: Streams live in memory (optionally persisted via RDB/AOF), need no separate cluster, and are managed with one command. Kafka is built for terabytes of logs and thousands of partitions. Streams win at up to ~50M messages per day, and when Redis is already in the stack.

• `XADD key [MAXLEN ~ N] * field value ...`: add a record
• `XREAD COUNT N BLOCK ms STREAMS key id`: read records starting from id
• `XRANGE key start end`: range of records (- is the beginning, + the end)
• `XLEN key`: record count in the stream
• `XDEL key id`: delete a specific record

Consumer Groups

A Consumer Group is a named group of consumers that splits one stream among themselves. Each message goes to exactly one consumer of the group, which enables horizontal scale-out of processing. Multiple groups can read the same stream independently, each gets every message.

A typical pattern: several workers in one group. For example, three payment-service instances run `XREADGROUP GROUP payment-processors consumer-{hostname}`. Redis automatically distributes messages among them. No Zookeeper or coordinator needed.

Difference from Kafka partitions: in Kafka you cannot have more consumers than partitions. In Redis Streams you can have as many consumers as you like, and Redis balances them itself. But Streams does not guarantee order between different consumers of one group (unlike Kafka, where order is guaranteed within a partition).

Pending Entries

When a consumer receives a message via XREADGROUP, Redis does not treat it as processed. The message lands in the PEL (Pending Entry List). It stays there until explicit `XACK`. If the worker crashes before acking, the message is not lost. It is recorded as pending against that consumer.

The `delivery-count` field in XPENDING shows how many times the message was delivered. If count > 3 it is a poison message (the handler keeps crashing on it). Such messages must be moved to a dead-letter stream and alerted, not retried forever.

Stream Claim

XCLAIM and XAUTOCLAIM are commands for 'intercepting' stuck messages. XCLAIM reassigns a specific ID to another consumer. XAUTOCLAIM (Redis 6.2+) scans the entire PEL and grabs every message older than a threshold in one call. Easier to automate.

A full production cycle looks like this: main workers read via `XREADGROUP ... >`, process, call `XACK`. A separate 'janitor' process runs `XAUTOCLAIM` on a schedule with a multi-minute timeout, claims stuck messages, and either reprocesses them or moves them to a dead-letter stream if `delivery-count` exceeded the limit.

MAXLEN trimming is mandatory in production. Without a cap, the stream grows forever and eats Redis memory. `XADD key MAXLEN ~ 100000 *`: the tilde means approximate trim (faster than exact). Alternative: `XTRIM key MAXLEN ~ 100000` on a schedule, or keep data only for the last N hours via `XRANGE` + `XDEL`.

Summary

• **Redis Streams = persistent log**: XADD adds, XREAD reads. Messages don't vanish on read, unlike Pub/Sub.
• **Consumer Groups = horizontal scaling**: each message goes to exactly one consumer of the group. Multiple groups receive every message independently.
• **PEL + XACK = at-least-once guarantee**: a message stays pending until explicit XACK. XAUTOCLAIM returns stuck messages to work when a worker dies.
• **MAXLEN is mandatory in production**: without it, the stream grows forever. `XADD key MAXLEN ~ N` keeps only the last N records with no perceptible perf cost.

Related topics

Redis Streams complement other queue and log patterns:

• Pub/Sub — No-persistence alternative. Messages are lost if there is no subscriber. Streams fix this
• BullMQ / Bull — Popular Node.js queue on top of Redis Lists. Streams give more control over groups and delivery-count
• Kafka — Same log + consumer groups pattern, but for terabyte scale. Streams are the lightweight option when Redis is already in the stack
• Dead-letter Queue — Pattern for handling poison messages from PEL. Messages with delivery-count > N move to a separate stream for manual review

Вопросы для размышления

• A service reads a stream via XREADGROUP and periodically dies mid-batch. Which mechanism guarantees no message is lost, and what is needed for automatic recovery?
• Two different services should process the same events from stream `events`. One builds analytics, the other sends notifications. How to set up Consumer Groups so each service independently sees every event?
• The stream is growing and after a month occupies 8 GB of memory. What size-limit strategies exist, and how do you choose between MAXLEN by record count and deleting by time?

Связанные уроки

• db-19-redis