Spark RDD vs DataFrame vs Dataset on the DE interview

Why interviewers ask this

Spark is still the default distributed engine for batch DE work at Netflix, Stripe, Airbnb, Databricks customers, and most data platforms that touch more than a few terabytes a day. On a Data Engineering loop the question "what's the difference between RDD, DataFrame, and Dataset?" is almost guaranteed — it tells the interviewer in 90 seconds whether you've actually shipped Spark jobs or just read a blog post.

The load-bearing concept they're checking is whether you understand that DataFrame is not just RDD with a nicer API. It is a fundamentally different execution path: a schema, the Catalyst query optimizer, and the Tungsten runtime that generates JVM bytecode. Senior candidates are expected to talk about codegen, predicate pushdown, and when typed Datasets actually buy you something over a DataFrame.

The painful real-world consequence of getting this wrong: a DE writes RDD code for a DataFrame-shaped problem and loses 5–10x of throughput that Catalyst would have given for free. On a 2 TB nightly job that's the difference between a 40-minute run and a 6-hour run that wakes someone up.

RDD: the original abstraction

RDD (Resilient Distributed Dataset) is the lowest-level Spark API. It models data as a distributed collection of opaque objects with map / filter / reduce operations, and it has been in Spark since version 0.x.

The defining properties are: immutable (every transformation returns a new RDD), distributed across N executors split into partitions, resilient through a lineage graph that lets Spark recompute any lost partition, schema-less (Spark sees a collection of Any), and crucially unoptimized — what you write is exactly what runs.

rdd = sc.parallelize([1, 2, 3, 4, 5])
result = rdd.filter(lambda x: x > 2).map(lambda x: x * x).collect()
# [9, 16, 25]

Cases where RDD is still the right tool are narrow: graph processing through GraphX, custom partitioning that you cannot express in DataFrame DSL, weird binary formats with no schema, or legacy code on Spark < 2.0. For 95% of modern DE pipelines, DataFrame wins.

If you're reaching for RDD on a join or a groupBy in 2026, stop and check whether you've shipped a footgun.

DataFrame: schema plus Catalyst

A DataFrame is a table with a schema — named columns with types. Under the hood it's still RDD[Row], but the schema metadata unlocks the entire query optimization stack.

The wins over RDD are concrete and measurable:

• Catalyst optimizer rewrites your query: predicate pushdown into the data source, projection pruning, join reordering, constant folding.
• Tungsten does off-heap memory management and whole-stage codegen — instead of interpreting your plan, Spark generates Java bytecode for the whole pipeline.
• A unified API across SQL and the DataFrame DSL — you can mix them freely.
• Typical speedup of 5–10x over naive RDD code on aggregations, joins, and filters.

from pyspark.sql import functions as F

df = spark.read.parquet("s3://bucket/orders/")
result = (df.filter(F.col("amount") > 100)
            .groupBy("user_id")
            .agg(F.sum("amount").alias("total"))
            .orderBy(F.desc("total")))
result.show()

The SQL equivalent runs through the exact same optimizer:

df.createOrReplaceTempView("orders")
spark.sql("""
  SELECT user_id, SUM(amount) AS total
  FROM orders
  WHERE amount > 100
  GROUP BY user_id
  ORDER BY total DESC
""")

Catalyst parses your query, runs rule-based and cost-based optimization, picks a physical plan, then Tungsten codegens it to JVM bytecode. This is the fastest API in Spark, period.

Dataset: typed DataFrame for the JVM

Dataset is DataFrame plus static typing. It exists only in Scala and Java — in PySpark, Dataset == DataFrame because Python doesn't have a JVM type system to plug into.

case class Order(userId: Long, amount: Double)
val ds: Dataset[Order] = spark.read.parquet("...").as[Order]
ds.filter(_.amount > 100).show()

The wins are compile-time type safety (a typo in a column name is a compiler error, not a 3 a.m. pager alert) and the ability to combine Catalyst optimization with strongly-typed business logic. The cost is Encoder serialization overhead for complex types and the fact that lambdas inside .filter(_.amount > 100) are opaque to Catalyst — the optimizer sees a black box and can't push the filter into Parquet.

In Python you skip this debate entirely. Type-safety happens at the edges with mypy, pandera for schema validation at runtime, and DataFrames with explicit StructType schemas for the rest.

Lazy evaluation, transformations vs actions

Every operation on RDD or DataFrame is lazy — it builds a logical plan and returns immediately. Real execution only starts when you call an action.

# zero compute happens here
df1 = df.filter(F.col("amount") > 100)
df2 = df1.groupBy("user_id").sum("amount")

# this line triggers the whole plan
df2.show()

Catalyst takes your plan through logical plan → optimized logical plan → physical plan → codegen. The single most useful debugging command is df2.explain(mode="extended") — it prints every stage.

df2.cache()
df2.show()      # computes, then caches
df2.count()     # served from cache

When to pick which

On a modern Spark 3.x+ stack with Parquet files and a Delta or Iceberg table format, roughly 95% of DE work is DataFrame plus SQL. RDD shows up in maybe 5% of jobs and almost always in legacy code you're about to rewrite.

The honest interview answer is: "DataFrame by default, RDD only when I can name the specific feature I need that DataFrame doesn't expose."

Common pitfalls

Reaching for RDD out of habit. Engineers who learned Spark in 2016 still write sc.textFile(...).map(...).reduceByKey(...) for problems that are obviously DataFrame-shaped. Catalyst can't optimize what it never sees, so the same join that DataFrame finishes in 3 minutes runs 30 minutes on RDD. The fix is to start every new job with spark.read.format(...) and only drop to .rdd if you have a written-down reason.

collect() on large data. df.collect() pulls every row to the driver JVM. On a 100 GB DataFrame this is a guaranteed OutOfMemoryError, and the worst part is that it sometimes works in dev (small sample) and only blows up in prod. Use df.take(n) to peek, df.show() for human inspection, and df.write... for real output.

Python UDFs instead of built-ins. A Python UDF serializes each row from JVM to a Python worker, runs the function, then serializes back. The overhead is roughly 10–100x slower than the equivalent F.* expression. If you genuinely need custom logic, use Pandas UDFs (vectorized, Arrow-based) — they keep most of the speed. Plain udf(lambda x: ...) is a bug pattern.

Forgetting cache() for multi-action workloads. Computing the same DataFrame three times reads the source three times. df.cache() before the first action, then df.unpersist() when you're done. Caching everything is also wrong — it costs memory and can spill to disk; cache only the DataFrames you actually reuse.

Writing without partitionBy. df.write.parquet("s3://...") with no partitioning produces a single huge directory, and downstream readers have to scan it all. Adding partitionBy("event_date") on a clear filter column lets future queries skip 99% of files. On wide tables this is the single biggest read-side win you can ship.

Treating Spark DataFrame like pandas DataFrame. They share a name and almost nothing else. Spark is distributed, lazy, immutable; pandas is single-node, eager, mutable. Code that iterates rows, modifies in place, or calls .iloc[] is pandas thinking applied to Spark and it will either error or be catastrophically slow.

Over-partitioning after repartition(N). Every partition carries scheduling overhead. The rule of thumb is 100–200 MB per partition for shuffle stages. Bumping spark.sql.shuffle.partitions from 200 to 20,000 because "more is faster" usually makes things slower.

Related reading

• Spark fundamentals — Data Engineering interview
• Spark Catalyst and AQE — Data Engineering interview
• Spark broadcast joins — Data Engineering interview
• Spark on Kubernetes — Data Engineering interview
• Spark for data analysts

If you want to drill DE interview questions like this every day on real Spark, SQL, and system-design problems, NAILDD is launching with 500+ problems calibrated to FAANG and unicorn DE loops.

FAQ

What is pandas-on-Spark?

The pandas API on Spark — formerly the open-source Koalas project — exposes a pandas-shaped surface that's backed by a DataFrame under the hood. It became part of Spark in 3.2. Useful for migrating existing pandas notebooks to scale, but you should still understand what's being lazy-evaluated underneath; otherwise you write pandas patterns that quietly trigger full shuffles.

What does df.rdd give me?

df.rdd returns the underlying RDD[Row]. You sometimes need it for legacy libraries that only accept RDD input, but you lose Catalyst optimization the moment you drop down. Treat it as an escape hatch, not a workflow. If you find yourself calling .rdd regularly, your DataFrame code probably has a smell.

What's the difference between cache() and persist()?

cache() is exactly persist(StorageLevel.MEMORY_ONLY). persist() accepts a storage level — MEMORY_AND_DISK (the safe default for big DataFrames), DISK_ONLY if you're memory-constrained, MEMORY_ONLY_SER for serialized in-memory. Use MEMORY_AND_DISK when in doubt; pure MEMORY_ONLY can silently evict and trigger recomputation.

How many shuffle partitions should I configure?

The default spark.sql.shuffle.partitions = 200 is tuned for tiny clusters. Target 100–200 MB of data per partition after shuffle. On a 500 GB shuffle that means roughly 2,500–5,000 partitions. Adaptive Query Execution (AQE) in Spark 3.x can coalesce post-shuffle partitions automatically — turn it on with spark.sql.adaptive.enabled = true.

Does Catalyst optimize Python UDFs?

No. A Python UDF is opaque to the optimizer — Catalyst can push filters and projections up to the UDF boundary but cannot see inside it. The optimizer will at least try to apply pushdown before the UDF runs, but a chain of three UDFs in the middle of a query effectively disables most optimization. Prefer built-in expressions or Pandas UDFs for anything performance-sensitive.

Is the typed Dataset API faster than DataFrame in Scala?

Usually no, and often slightly slower. The Encoder for case classes adds serialization overhead, and lambda-based operations like .filter(_.amount > 100) are opaque to Catalyst, so you lose predicate pushdown. The win is compile-time safety, not runtime speed. If you want both, write Dataset code that calls column expressions: ds.filter($"amount" > 100) instead of the lambda form.