Apache Spark fundamentals — Data Engineer interview

What interviewers actually probe

Spark on a Data Engineer loop is rarely about syntax. Anyone can chain df.groupBy().count() after one week of tutorials. The interviewer at Databricks, Stripe, or DoorDash wants to know whether you understand what happens under the hood when that one-line query runs against 1 TB of clickstream and takes 38 minutes instead of 4. That gap — between "it works on the sample" and "it scales on prod" — is the entire signal.

A senior DE screen will cycle through five themes: the execution model (lazy evaluation, DAG, transformations vs actions), shuffle mechanics (why groupBy is slow and filter is not), the optimizer (Catalyst, predicate pushdown, AQE), join strategies (broadcast vs sort-merge), and debugging (reading the Spark UI, spotting skew, fixing driver OOMs). Hold a 25-minute conversation across those five and you're at mid-to-senior signal regardless of years on resume.

None of this is memorizable trivia — it only sticks after you've actually killed an executor in anger.

RDD vs DataFrame vs Dataset

This is the opening question on roughly 70% of Spark loops — partly because it filters out candidates who only know the modern DataFrame API and have no model of what Spark actually distributes.

RDD is the original abstraction: a distributed collection of arbitrary objects, lazy, no schema, manual control over serialization and operators. DataFrame sits on top but adds a schema and routes every operation through Catalyst, so predicate pushdown, projection pruning, and join reordering happen for free. Dataset is the typed Scala/Java sibling that combines DataFrame's optimizer with compile-time type safety.

When the interviewer asks "when would you reach for RDD today?", the honest answer is almost never — custom transforms that don't fit DataFrame + pyspark.sql.functions, fine-grained serialization, and legacy code. In modern Spark 3.5+ shops on Databricks or EMR, RDD shows up in interview answers more than in production.

Shuffle, partitions, skew

A shuffle is the redistribution of data across workers whenever Spark colocates records by key — groupBy, join, repartition, distinct, window functions without a matching partition. It's the most expensive operation in the engine: write intermediate files to local disk on every executor, transfer them over the network, read them back on the other side. That round-trip is why a filter over a TB takes seconds and a groupBy over the same TB takes minutes.

Partitions are the unit of parallelism. One partition equals one task equals one CPU core for the duration of that stage. The default after a shuffle is 200 partitions (spark.sql.shuffle.partitions), which is wrong for almost every workload — too few for 1 TB inputs, too many for 10 GB inputs. The rule of thumb most senior engineers use: aim for 100 MB to 1 GB per partition after the shuffle, then tune from the Spark UI.

Skew is when one partition holds dramatically more data than the others — usually a long-tail join key (user_id = null, country = US, one whale with 40% of events). One task runs for 25 minutes while the other 199 finished in 30 seconds. Three Spark UI signals: a single straggler in the task duration histogram, uneven shuffle read sizes, and memory pressure on exactly one executor.

The fix toolkit has three tiers. Salting the join key adds a random suffix to fan heavy keys across partitions, then aggregates the salt away. AQE skew join handling (Spark 3+) detects oversized partitions at runtime and splits them — this covers most real cases with zero code. Broadcasting the smaller side eliminates the shuffle when one table fits in executor memory.

from pyspark.sql.functions import concat, lit, rand, floor

# Salt the skewed key with 10 buckets to spread the load
df_salted = df.withColumn(
    "user_salted",
    concat(df.user_id, lit("_"), floor(rand() * 10).cast("int"))
)

Catalyst and AQE

Catalyst is Spark SQL's query optimizer. Every DataFrame operation you write — Python, Scala, or SQL — gets compiled into a tree and rewritten through four phases before execution.

The natural follow-up: "how do you actually inspect the plan?" Answer: df.explain(True) or df.explain(mode="extended"). Pay attention to PushedFilters: [...] (predicate pushdown worked), BroadcastExchange (join went broadcast), and Exchange hashpartitioning (this is where the shuffle happens).

Adaptive Query Execution (AQE) is the runtime layer added in Spark 3.0 and on-by-default in 3.2+. Catalyst plans the query before it sees real data; AQE rewrites the plan between stages based on actual statistics from completed shuffles.

spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true")
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")

Three wins: AQE coalesces tiny post-shuffle partitions to whatever fits actual data; demotes SortMergeJoin to BroadcastHashJoin if a side turns out smaller at runtime; splits skewed partitions into balanced sub-tasks. On a real workload, flipping AQE on can cut a 40-minute job to 8 minutes with zero code change — a great answer to "lowest-effort Spark optimization you've shipped?"

Joins: broadcast vs shuffle

Join strategy questions are guaranteed on any Spark loop because the wrong strategy is the most common cause of out-of-memory failures in production.

BroadcastHashJoin works because if one table is 100 MB and the other is 1 TB, replicating the small one to all 50 executors costs 5 GB of network total — cheap compared to shuffling a TB. The threshold is controlled by spark.sql.autoBroadcastJoinThreshold (default 10 MB, often raised to 50–100 MB in production).

from pyspark.sql.functions import broadcast

result = (
    big_events
    .join(broadcast(small_users), on="user_id", how="left")
)

Classic question: "1 TB of events joined to 100 MB of users, what join does Spark pick?" Answer: BroadcastHashJoin if you've raised the threshold above 100 MB or hint it explicitly with broadcast(). Default 10 MB threshold falls back to SortMergeJoin — 10x slower for this shape. Knowing both the answer and the config knob separates mid from senior.

For the deeper version with skew-aware broadcasting and bucketed joins, see Spark broadcast joins.

The optimization checklist

When a job is slow, work through the same five checks in order. This is the checklist most senior DEs will recite verbatim, and the interviewer wants to hear it sound rehearsed.

1. Partition sizing. Too many tiny partitions burns scheduling overhead on the driver. Too few starves parallelism. Target 100 MB to 1 GB per partition after every shuffle. Use repartition(N) to increase or coalesce(N) to decrease without a full shuffle. With AQE on, coalescing happens automatically post-shuffle.

2. Predicate pushdown. Filters should be physically pushed into the file scan, not applied after read. With Parquet, ORC, or Iceberg, partition filters work at the file-listing level. Verify by running df.explain(True) and grepping for PushedFilters: [IsNotNull(country), EqualTo(country,US)]. If your filter is missing from that line, refactor.

3. Caching reused DataFrames. If a DataFrame is referenced more than once in your job, Spark recomputes the entire DAG every time unless you call df.cache() or df.persist(StorageLevel.MEMORY_AND_DISK). The classic trap: building a lookup DataFrame, then joining it to three downstream tables — without cache, the build runs three times.

4. UDF performance. Python UDFs are slow — every row serializes JVM→Python→JVM. Prefer built-ins from pyspark.sql.functions (zero serialization), then pandas UDFs (10–100x faster), then native Scala only when no built-in fits.

5. Skew handling. Already covered above — but in the checklist context, the question is whether you remembered to check at all. Open the Spark UI, sort tasks by duration, and look for a single straggler.

Common pitfalls

The most damaging mistake is calling collect() on a large DataFrame. collect() pulls every row to the driver as a single Python list, which works fine on a 10k-row sample and crashes the driver instantly on 1 TB of events. The fix is to either take(N) for sampling or write to S3/HDFS and inspect from there. Every senior DE has killed exactly one cluster this way and they don't forget.

A subtle trap is ignoring cache() for re-used DataFrames. Lazy evaluation means Spark recomputes the full lineage on every action, so a DataFrame referenced in three downstream joins gets built three times unless cached. The diagnostic: if a downstream count() is suspiciously slow and the source involves a heavy transform, you're recomputing.

Another common failure is confusing transformations with actions. Transformations like select, filter, join build the DAG but execute nothing; only actions like count, collect, write, show trigger computation. Candidates panic when count() takes 20 minutes — but it's the first time the full DAG ran, not a bug. This one trips even people with two years of PySpark on their resume.

Reaching for large Python UDFs when a built-in exists is the most common performance own-goal. pyspark.sql.functions has 200+ built-ins; check there first, then pandas UDFs, then Scala. A single replaced Python UDF can cut a 4-hour job to 25 minutes.

Finally, running Spark on small data. For inputs under 10 GB, plain pandas (or DuckDB, or Polars) is usually faster end-to-end because Spark's startup, JVM warm-up, and serialization overhead dominate. Spark earns its keep starting at tens of millions of rows or when the dataset doesn't fit on a single machine. Recommending Spark for a 2 GB CSV in an interview is a small red flag.

Related reading

• Spark broadcast joins — Data Engineer interview
• Spark Catalyst and AQE deep dive
• Databricks for Data Engineers
• Window functions — Data Engineer interview
• Airflow — Data Engineer interview

If you want to drill Spark, SQL, and system-design questions every day, NAILDD ships a daily DE interview question that builds the muscle this article describes.

FAQ

PySpark or Scala Spark — which should I learn first?

PySpark is the default on US data teams at this point — Databricks, Stripe, Airbnb, and most FAANG-adjacent shops run PySpark in production. Scala still wins for performance-critical UDFs, library development, and the small set of teams who started on Spark before 2.0. For an interview loop targeting analytics or platform DE roles, PySpark is what you'll be tested on. Learn Scala later if you join a team that uses it; don't gate your job search on it.

When does Spark beat Trino, Presto, or DuckDB?

Spark wins for batch ETL and large transformations — the workload pattern where you read terabytes, transform with multiple joins and window functions, and write a derived table. Trino and Presto win for interactive SQL over the same warehouse — sub-second to single-digit-second queries on prepared tables. DuckDB wins for single-node analytics under 100 GB. They're not competitors so much as different tools for different points on the latency-vs-volume curve. A good interview answer names the workload, not the engine.

How much hands-on Spark do I need before applying?

The non-negotiable minimum is two or three projects where you've actually moved real data — not toy CSVs, but at least tens of millions of rows with a non-trivial join. You should have personally debugged at least one slow job using the Spark UI, identified the cause (skew, bad partitioning, missing broadcast, runaway UDF), and shipped the fix. Interviewers can tell within five minutes whether you've ever opened the Spark UI in anger or just read about it.

Do I need to memorize all the config knobs?

No — and trying to is a tell that you've crammed rather than used Spark. Memorize the ones tied to mental models: spark.sql.shuffle.partitions (because it ties to partition sizing), spark.sql.autoBroadcastJoinThreshold (because it ties to join strategy), spark.sql.adaptive.enabled (because AQE is the modern default). For everything else, the right answer is "I'd check the docs or the Spark UI" — interviewers prefer that to a recited list.

What's the single highest-leverage Spark optimization I can ship this week?

Turn on AQE and raise the broadcast threshold to 50–100 MB. Two config changes, zero code, and on a typical messy DE pipeline you'll cut wall-clock by 30–60% with no risk. If you can describe a concrete before/after — "job dropped from 47 minutes to 18 minutes after enabling AQE and bumping the broadcast threshold to 50 MB" — that's exactly the kind of impact statement that lands in the senior column of the rubric.