Flink and stream processing

Some answers cannot wait for a nightly batch job. "Clicks per ad in the last minute", "fraud signals in the last five transactions", "average delivery time per region". Stream processing computes these continuously as events arrive.

Crux: a stream processor keeps state per key while it reads an endless stream, groups events into time windows by when they happened, and checkpoints that state so a crash does not lose or double-count.

What it is and why it exists

Apache Flink is a framework for stateful computation over streams. You write a job that reads from a source (usually Kafka), transforms, aggregates and joins events, and writes to a sink (a database, a search index, another topic).

Why not just write a Kafka consumer? For simple work you should. A stream processor earns its place when you need:

• State that outlives one event (counts, sessions, running averages).
• Windows over time.
• Correct results with late and out-of-order events.
• Fault tolerance of that state, with no hand-written recovery.
• Joins between streams.

Spark Structured Streaming is the main alternative. Flink is generally the one named for low-latency, true event-at-a-time streaming.

How it works, at the level you must explain

Keyed streams and state

You partition the stream by a key (key_by(ad_id)). Events with the same key go to the same task, and that task holds keyed state for it, in memory or in an embedded store on local disk. Parallelism is across keys, as in Kafka partitions.

Event time, processing time and watermarks

• Processing time is when the machine sees the event.
• Event time is when it actually happened, stamped by the source.

Phones go offline and networks reorder, so events arrive late and out of order. Computing by processing time gives wrong counts. Computing by event time needs to know when to stop waiting for stragglers. A watermark is a marker flowing through the stream that says "no event older than time T is expected any more". When the watermark passes the end of a window, the window can close and emit. Events that arrive after that are late: drop them, send them to a side output, or allow a bounded lateness that updates the result.

Windows

• Tumbling: fixed size, no overlap. "Clicks per minute".
• Sliding: fixed size, moving by a smaller step. "Last 5 minutes, updated every minute".
• Session: closes after a quiet gap per key. "A user's visit".

Checkpoints and delivery

Flink periodically takes a consistent checkpoint of all state plus the source offsets, and stores it durably. After a failure it restores the last checkpoint and replays from those offsets. This gives exactly-once state updates inside Flink. To get end-to-end exactly-once, the sink must support transactions or be idempotent (write by a unique key, so a repeat does the same thing). Otherwise you have at-least-once.

A worked shape

Count clicks per ad per minute: read the clicks topic, key by ad_id, assign event time and a watermark that allows a few seconds of lateness, apply a one-minute tumbling window, count, and write (ad_id, window_start, count) to a database with that triple as the key. A replay overwrites the same row, so it is idempotent.

Do you need real time at all?

Streaming is costly to build and to run, and it is easy to get subtly wrong. Before choosing it, ask whether anyone will notice if the answer is an hour old. A scheduled batch job (or a query on a table updated every few minutes) is simpler, cheaper and easier to repair. Pick streaming when the result must react in seconds: fraud blocks, live dashboards, alerts.

What a hand-written consumer runs into

A consumer that reads, transforms and writes each event is easy. Add state (clicks per user in the last five minutes) and three problems appear.

• Crash: the counts in memory are gone. Replaying the whole topic from the start takes hours.
• Rescale: adding or removing workers means moving the state for each key to a new owner, safely.
• Late and out-of-order events: a phone reconnects and sends events from ten minutes ago, which distort windows already closed.

A stream processor is that bookkeeping done once, correctly.

How the cluster is laid out

• A JobManager coordinates: it accepts a job, plans it as a graph of operators (source, transformations, sink), schedules tasks, and triggers checkpoints.
• TaskManagers are the workers. Each offers a number of slots, and each slot runs one parallel slice of the job.
• Parallelism is how many slices run per operator. For a keyed operator, keys are grouped into key groups, and each slot owns a range of groups. Rescaling moves whole groups, not single keys.
• State backend: state lives in memory on the heap for small state, or in an embedded on-disk store (RocksDB) when it is larger than memory. Checkpoints copy it to durable storage.
• A savepoint is a checkpoint you take by hand, to upgrade a job, change parallelism or move a cluster, and then resume from it.
• Backpressure: when a sink is slow, the slowness travels upstream so the source reads less, rather than memory filling up.

A second sample shape

Fraud signal: read card transactions from a topic, key by card number, hold the last few transactions in keyed state, and emit an alert if the total in a ten-minute sliding window exceeds a limit or if two countries appear within an hour. The state per card is small, the key spreads load well, and the alert goes to a topic that the case-handling service reads. A sink that writes alerts by a unique id makes replays harmless.

When to use it, trade-offs, numbers

• What: a stateful stream-processing engine with event-time windows and checkpointing.
• Use when: real-time aggregations, windowed metrics, sessionisation, fraud and anomaly signals, joining streams, and continuous ETL with low latency.
• Know: keyed state, event time versus processing time, watermarks, window types, checkpointing, and idempotent sinks.
• Careful: it is another cluster to run, state can grow large, and many designs do not need it. If a periodic query on a database does the job, use that.

Flink or something else?

• A plain consumer service: pick it for per-event work with little or no state.
• Spark Structured Streaming: pick it when the team already runs Spark and can accept slightly higher latency.
• A batch job on a schedule: pick it when minutes or hours of delay is fine. Often the honest answer.
• A database with a pre-aggregated table or materialized view: fine at modest volume.
• Managed options (for example Kinesis-based processing) exist. Name a category if you do not know them.

Numbers: window size, allowed lateness and checkpoint interval are choices that trade latency against accuracy and overhead. State them as assumptions with a reason, such as "one-minute windows, 10 seconds of allowed lateness, because the dashboard can lag by a minute".

Common mistakes

• Using processing time and getting wrong counts after a delay.
• Forgetting late events, or setting no policy for them.
• Writing to a sink that double-counts on replay.
• Choosing Flink for work a simple consumer or a SQL query can do.
• Ignoring state size, which grows with the number of keys and window length.

• Choosing streaming when a batch job meets the freshness need.
• Ignoring the cost of rescaling and state size, until the job cannot be resized.
• Skipping savepoints before an upgrade.

Check yourself

• When is a plain Kafka consumer enough, and when do you need Flink?
• What is the difference between event time and processing time?
• What does a watermark tell the engine?
• Name the three window types and give an example of each.
• How do checkpoints let a job recover without losing or double-counting?
• What must a sink do to make results exactly-once end to end?
• What three problems appear when you add state to a plain consumer?
• What does a JobManager do, and what does a TaskManager do?
• When is a batch job the better answer than Flink?