Spark Structured Streaming on the Data Engineering interview

Why interviewers love this topic

If you're prepping for a Data Engineering loop at Databricks, Stripe, DoorDash, Uber, or Snowflake, expect a round on streaming. Most production pipelines are "Kafka in, Delta or Iceberg out, Spark job in the middle running every minute". Interviewers want to know whether you understand what happens between those arrows — specifically watermarks, output modes, and exactly-once semantics — or whether you'll cargo-cult the syntax and blow up the cluster the first time late events arrive.

Spark Structured Streaming is the default streaming engine you'll meet on the job, not because it's the lowest-latency option (Flink wins there) but because it shares the DataFrame API with batch Spark. That single design choice is why a team running batch Spark can stand up streaming in a sprint without learning a second framework. Questions are rarely about syntax — they're about the mental model: when does state grow unbounded, when does append mode silently drop rows, what does processingTime='1 minute' actually guarantee.

Architecture: micro-batching vs continuous

The default execution model is micro-batching: Spark collects events for a small interval, runs them as a regular batch DataFrame job, commits results, starts the next batch. Latency is 1 second to 1 minute depending on the trigger. This is 99% of production and the model your interviewer wants to hear about first.

Continuous processing landed experimental in Spark 2.3 and promises sub-millisecond latency via long-lived tasks. It supports only a thin slice of operations (map-like, no aggregations, limited sources), so almost nobody runs it. Mention it exists, say "experimental", pivot back to micro-batch.

Triggers and how they shape latency

The trigger setting controls when the next micro-batch starts. Four variants matter on the interview, and the difference between them often comes up as a follow-up question after you've drawn the architecture diagram.

from pyspark.sql import functions as F

df = (spark.readStream
        .format("kafka")
        .option("kafka.bootstrap.servers", "broker:9092")
        .option("subscribe", "events")
        .load())

query = (df.writeStream
           .format("delta")
           .trigger(processingTime='1 minute')
           .outputMode("append")
           .option("checkpointLocation", "s3://bucket/checkpoints/events/")
           .start("s3://bucket/silver/events/"))

Use processingTime='1 minute' for steady near-real-time pipelines. Use availableNow=True for incremental batch backfills — it processes everything available and exits, replacing the deprecated once=True. Use continuous='1 second' only if you've opted into the experimental engine. No trigger means "process as fast as possible", which sounds great until your cluster autoscales into a six-figure monthly bill.

Windowing and watermarks

Windowing is how you reduce an infinite stream into something countable. Three window shapes show up on every interview, and you should be able to write them from memory.

# Tumbling: non-overlapping, 10-minute buckets
tumbling = df.groupBy(
    F.window("event_time", "10 minutes"),
    "user_id"
).count()

# Sliding: 10-minute window, advancing every 5 minutes (overlap)
sliding = df.groupBy(
    F.window("event_time", "10 minutes", "5 minutes"),
    "user_id"
).count()

# Session: dynamic window closed by inactivity gap (Spark 3.2+)
session = df.groupBy(
    F.session_window("event_time", "5 minutes"),
    "user_id"
).count()

The watermark is the part most candidates muddle. It's the engine's promise about how late an event is allowed to be before being dropped. Concretely, watermark = max(event_time seen so far) - threshold. Set it like this:

df.withWatermark("event_time", "10 minutes") \
  .groupBy(F.window("event_time", "5 minutes"), "user_id") \
  .count()

The watermark is what lets Spark eventually close a window and free the state. No watermark means no closing, which means no garbage collection of state, which means an OOM whose root cause is invisible in the logs until you check the state store metrics.

Late data handling

A "late" event is one whose event_time is older than the current watermark. What Spark does with it depends on the output mode and whether the relevant window is still open in state.

In append mode, late events that arrive after their window has been closed are silently dropped. There is no error, no warning at info level — just a missing row in the silver table. The mitigation is to widen the watermark threshold (e.g. 30 minutes instead of 5), accept the extra state cost, and monitor late-event metrics from the StreamingQuery listener.

In update mode, a late event can still update an existing window's count as long as the window's state is still alive in memory. This is more forgiving but produces multiple emits per window, which downstream consumers must handle (Delta merge or upsert into the sink).

Output modes

There are three output modes and the rules about which combinations are legal change between Spark versions. Get them straight before the interview.

Append writes only new rows that are now stable — meaning their window has closed past the watermark. It's the default and the one to use for event-level passthrough into a Delta or Iceberg sink. For aggregations, append is allowed only with a watermark.

Update writes any row whose value changed in this micro-batch. Aggregation results are written every time their count moves. This is the right mode for live dashboards backed by a kv store like Redis or DynamoDB, where you want the latest count fast and don't care about emit duplicates.

Complete writes the entire result table on every micro-batch. Allowed only for aggregations and only for relatively small result sets. Avoid this in production unless your aggregation is bounded by hour-of-day or some other small dimension. For an unbounded grouping the cost is brutal.

Exactly-once guarantees

Spark Structured Streaming gives you end-to-end exactly-once semantics when three conditions hold together. Missing any one of them downgrades you to at-least-once, which usually means duplicates in the sink after a restart.

First, the source must be replayable: Kafka, Kinesis, file source, or Delta source all qualify. A socket source or a custom REST puller without offset bookkeeping does not. Second, the sink must be either idempotent (Delta Lake, Iceberg, Postgres with INSERT ... ON CONFLICT) or transactional (foreachBatch with explicit txn control). Third, you must configure a checkpointLocation on durable storage — S3, GCS, ADLS, or HDFS. The checkpoint stores source offsets, state-store snapshots, and the commit log that lets Spark resume exactly where it left off.

.option("checkpointLocation", "s3://bucket/checkpoints/orders-v1/")

Stateful operations and state stores

Anything beyond a stateless map needs state: aggregations, deduplication, stream-stream joins, mapGroupsWithState, flatMapGroupsWithState. State lives in a state store keyed by the grouping columns, snapshotted to the checkpoint every micro-batch.

The default backend is HDFS-style state on local disk. For tens of GB per executor switch to the RocksDB state store via spark.sql.streaming.stateStore.providerClass. RocksDB spills to disk and is the Databricks Runtime default for streaming since DBR 13.

Stream-stream joins are the spiciest stateful op and a frequent interview drill. Both sides need a watermark, and the join condition must include an event-time range (e.g. "events within 1 hour"). Without the range constraint, state grows quadratically and the cluster melts.

Common pitfalls

When a streaming job goes wrong in production, the failure mode is rarely a stack trace — it's quietly wrong results or a cluster that gets gradually more expensive. These are the traps that come up in postmortems and, conveniently, in interview "tell me about a time" questions.

Running without a checkpoint location. Looks fine in a demo. The first time you restart the cluster, Spark has no idea what offsets it processed, so it either replays from the earliest Kafka offset (millions of duplicates) or skips to the latest (silent data loss). Always set checkpointLocation on durable storage.

Aggregating without a watermark. The state store grows by one entry per unique grouping key, forever. On a groupBy(user_id) over a consumer app with 10M monthly actives, you'll OOM around week three. The fix is to add withWatermark before the groupBy and accept that very late events get dropped.

Append mode on a window aggregation without a watermark. Spark refuses to start the query because it can't know when a window is "done" and safe to emit. The error message is unhelpful enough that this question is asked just to see if you've actually deployed one of these jobs.

Non-idempotent sinks under retries. Writing to a plain JDBC sink with INSERT (no ON CONFLICT) or to S3 as flat Parquet without transactions means every executor retry produces duplicate rows. Pair Spark with Delta Lake or Iceberg for the sink, or use foreachBatch with an explicit transaction.

Joining a stream to a slowly-changing dimension as a batch table. If you read the dimension once at query start, you'll never see updates. The fix is spark.readStream over a Delta dimension table, or a periodic refresh inside foreachBatch. Static reads of mutable dimensions are the single most common "why is the data stale" ticket.

Ignoring StreamingQueryListener metrics. processedRowsPerSecond, inputRowsPerSecond, and the state-store size are the three numbers that tell you whether the job is keeping up. Wire them into Prometheus or Datadog from day one.

Related reading

• Kafka on the Data Engineering interview
• Kafka Streams on the Data Engineering interview
• Spark RDD vs DataFrame vs Dataset
• CDC and Debezium on the Data Engineering interview
• Apache Flink on the Data Engineering interview
• Spark shuffle and skew

If you want to drill streaming questions like this on a daily timer, NAILDD ships Data Engineering interview problems — Spark, Kafka, watermarks, exactly-once — in a format that matches what the loops at Databricks and Stripe actually ask.

FAQ

Spark Structured Streaming vs Flink — which one do interviewers prefer?

Neither, honestly — they want to see that you understand the trade-off. Flink is true event-at-a-time streaming with sub-10ms latency, native event-time windowing, and a more mature state backend. Spark Structured Streaming is micro-batch with 100ms to 1-minute typical latency but shares the DataFrame API with batch Spark, which means one engine, one tool chain, one set of skills for your team. Pick Flink when you genuinely need sub-100ms response time (fraud detection, real-time bidding). Pick Spark when your team already runs batch Spark and you want streaming without doubling the platform footprint.

What's the difference between processing time and event time?

Processing time is the wall clock on the executor when the record is processed. Event time is the timestamp embedded in the record itself (when the event actually happened at the source). All windowing, watermarks, and late-data logic operate on event time, which is why your records must carry a timestamp column you can extract. If you use processing-time windows, you're effectively doing micro-batch counting with no late-data semantics — usually a sign you've avoided the hard problem rather than solved it.

How big should my watermark threshold be?

It depends on the p99 of your event-arrival lag, not a magic number. If most events arrive within 2 minutes but a tail straggles in at 8, set the watermark to 10 minutes to capture 99%+. Measure with a side query first. Wider watermark = more state in memory; narrower = silently dropped late rows.

When should I use foreachBatch instead of a structured sink?

foreachBatch gives you the micro-batch as a regular DataFrame and lets you do whatever you want with it: merge into Delta with custom logic, write to multiple sinks, call an external API per batch, or wrap the write in a manual transaction. Use it when the built-in sink doesn't fit (multiple writes, custom upsert logic, external system) or when you need exactly-once into a non-idempotent sink and can implement the transaction yourself.

What happens if my checkpoint gets corrupted?

A corrupted checkpoint usually means the safe recovery is to point the query at a new path and decide what to do about the gap. If the source is Kafka with enough retention you can replay from a known offset, but you'll likely get duplicates unless the sink is idempotent. Back up the checkpoint directory for any business-critical pipeline and alert on checkpoint write failures.

Is this article official?

No. It's a practitioner summary based on Apache Spark 3.x and Databricks Runtime docs. Experimental features (continuous mode, RocksDB state config) can change between minor versions — cross-check against the docs for your exact Spark version before relying on a specific guarantee.