Windowing and Time

Google Analytics reports: 'Daily active users: 1.2M.' That number is computed from an infinite stream of page view events - by windowing. A credit card fraud system flags: 'Card 4111 has 15 transactions in 5 minutes.' That is computed from a sliding window. Spotify computes: 'User session length: 47 minutes.' That is a session window. Every real-time analytics number you see is the output of a windowing function applied to a stream. The choice of window type - tumbling, sliding, or session - and the watermark strategy that drives it, determines whether the number is correct, fresh, and complete.

• **Google Ads billing**: uses 1-hour tumbling windows to compute per-advertiser spend for rate limiting; event-time processing with 5-minute late-data tolerance ensures clicks arriving from slow mobile networks are attributed to the correct hourly budget window
• **Stripe fraud detection**: sliding windows (10-minute size, 1-minute slide) over card transactions detect velocity anomalies in real-time; Stripe processes 250+ million API requests per day, with sub-100ms fraud scoring latency including window aggregation
• **Spotify session analytics**: session windows with 30-minute inactivity gap compute listening session duration, skip patterns, and playlist completion rates; session data feeds the recommendation model trained on Spark offline

Tumbling Windows

An infinite stream of events cannot be aggregated in total - there is no 'end'. Windowing divides the stream into finite chunks for processing. **Tumbling windows** are the simplest: non-overlapping, fixed-size buckets. Every event belongs to exactly one window. A 1-minute tumbling window groups events from 12:00:00-12:00:59, then 12:01:00-12:01:59, and so on. When the window closes (watermark advances past its end), the aggregate is computed and emitted.

**Tumbling window properties:** - Non-overlapping: each event counted once - Aligned: boundaries at fixed intervals (every minute, every hour) - Low memory: only one window's data in memory at a time per key - Use cases: hourly revenue, daily active users, minute-level error rates - Limitation: events at the boundary have no context from the adjacent window (boundary artifact)

**Processing time vs event time tumbling windows**: processing-time windows are simpler (no watermarks needed) but produce non-reproducible results - replaying the same data produces different windows. Event-time windows are deterministic (same data always produces same windows) but require watermark management. Use event time for financial and compliance reporting; processing time for real-time dashboards where reproducibility matters less.

Sliding Windows

A fraud detection model should flag a credit card with 10 transactions in any 5-minute period - not just aligned 5-minute blocks. A tumbling window misses the pattern if 9 transactions happen in one window and 1 in the next. **Sliding windows** overlap: a 5-minute window that slides every 1 minute produces a new count every minute covering the last 5 minutes. Every event is counted in multiple windows (5 windows if slide = window/5). This provides continuous monitoring at the cost of higher memory.

**Sliding window parameters:** - **Size**: how long each window spans (e.g., 5 minutes) - **Slide (step)**: how frequently a new window starts (e.g., 1 minute) - **Overlap**: size/slide windows overlap (5-min window, 1-min slide: 5x overlap) - **Memory**: each event lives in (size/slide) windows simultaneously - Use cases: rolling averages, anomaly detection, moving rate metrics

**Sliding windows with non-incremental aggregations** (like percentiles, median, top-K) are expensive: every event must be stored because the aggregate cannot be updated incrementally. A 5-minute/1-minute sliding window with raw event storage uses 5x more memory than a tumbling window for the same time period. For latency percentiles at scale, use approximate algorithms (T-Digest, HDR Histogram) that can be merged incrementally.

Session Windows

User sessions do not follow a clock - a user browses for 2 minutes, disappears for 20 minutes, then returns. Tumbling and sliding windows split this session artificially. **Session windows** are defined by activity gaps: a session is a sequence of events with no gap exceeding a timeout. When the gap exceeds 30 minutes (common for web sessions), the window closes and the session aggregate is emitted. Session windows have variable duration and do not align to clock boundaries.

**Session window mechanics:** - **Gap**: inactivity timeout (30 minutes for web, 5 minutes for mobile, 1 hour for IoT) - **Dynamic size**: each user's session has its own start/end based on their event pattern - **Merging**: when a new event arrives, if it falls within gap of the last event, the session window extends; adjacent windows may be merged - **Memory**: one window per active session per key; sessions with long gaps expire naturally - Use cases: session duration analytics, user journey analysis, abandoned cart detection

Watermark Strategies in Practice

Watermarks are easy to define but hard to tune. Too aggressive (tight bound): events arrive late, windows close before all data is present, results are incomplete. Too conservative (wide bound): windows wait unnecessarily, latency increases. The art is in measuring actual late arrival distributions from production data and setting bounds based on the 99th or 99.9th percentile of observed latency.

**Idle partitions block watermarks**: in a Kafka topic with 16 partitions, if one partition receives no events (e.g., a geographic region with no activity at 3 AM), its watermark never advances. Flink's minimum watermark across all partitions stays stuck. Fix: `.withIdleness(Duration.ofMinutes(1))` marks a partition idle after 1 minute of silence and excludes it from watermark computation. Without this, one silent partition can halt all window triggering.

Key Takeaways

• **Tumbling windows**: non-overlapping fixed-size buckets; each event in exactly one window; lowest memory; boundary artifacts can split related events
• **Sliding windows**: overlapping buckets with size and slide parameters; each event in (size/slide) windows; best for continuous anomaly detection; higher memory unless using incremental aggregation
• **Session windows**: variable-size windows defined by inactivity gaps; natural for user behavior analysis; dynamic merging; one active window per key per session
• **Watermark strategy**: must be tuned from measured production late-arrival distributions; idle partition silence blocks all watermarks without withIdleness(); allowedLateness and side outputs handle the completeness-latency tradeoff

Related Topics

Windows are the aggregation layer between raw streams and analytics:

• Apache Flink Internals — Flink's checkpointing and state backends store window state; watermarks from this lesson determine when Flink triggers window computations
• Time Series Databases — Time series databases (InfluxDB, TimescaleDB) implement similar windowing concepts for query-time aggregation; stream windows are the real-time equivalent
• Event Sourcing and CQRS — Event sourcing systems store the raw event log; windowing projections are one pattern for materializing read models from event streams

Вопросы для размышления

• An e-commerce site wants to show 'trending products in the last hour, updated every minute.' Which window type fits - and how would the result differ if the site used processing time vs event time for the window clock?
• A fraud detection sliding window (10-min size, 1-min slide) must store raw events because the risk model requires the exact sequence. With 2M active cards and 100 events/card/hour: estimate the memory footprint. What design changes would reduce it by 10x?
• A Kafka topic has 32 partitions. During a regional network partition, 8 partitions receive no events for 15 minutes. A Flink job processes this topic with 10-minute tumbling windows. What happens to watermark progress and window triggering - and what is the correct withIdleness() configuration?

Связанные уроки

• alg-27-sliding-window