Spark on Kubernetes for Data Engineering interviews

Why Spark on Kubernetes matters

If you are interviewing for a Data Engineer role in 2026 at a company that runs anything resembling a modern platform — Databricks-adjacent stacks, EKS-hosted lakehouses, GKE-based ML platforms, or Snowflake-supplementing batch jobs at Uber, DoorDash, Stripe — there is a very high chance the interviewer will ask you to walk through running Apache Spark on Kubernetes. The question used to be a YARN question. It is not anymore. Hiring managers want to hear that you understand the move from Hadoop-era resource managers to cloud-native orchestration, and that you can reason about the pieces: pods, scheduling, dynamic allocation, and the operator pattern.

The trap candidates fall into is treating "Spark on k8s" as a deployment trivia question. It is not. It is a systems-design question disguised as ops. The interviewer wants to see that you understand why the driver and executors map naturally to pods, what breaks when you turn on dynamic allocation without a shuffle service, and how the Spark Operator changes the job-submission story. If you can speak to those three points clearly, you will land in the top 20 percent of candidates for the question.

This post is a structured walk-through of the answer you should be able to give on a whiteboard in roughly six minutes. It covers the architecture, a working spark-submit invocation, the dynamic-allocation story since Spark 3.0, the operator pattern, and the side-by-side with YARN that every interviewer eventually asks for.

Architecture: driver and executor pods

The Spark-on-k8s model is brutally simple once you internalize the pod abstraction. When you call spark-submit against a Kubernetes API endpoint, the client talks to the k8s API server, which schedules a driver pod somewhere in the cluster. The driver then talks back to the API server to request executor pods, which are scheduled wherever capacity exists. Communication between driver and executors happens over standard k8s networking — services and cluster DNS — rather than the YARN ApplicationMaster RPC dance.

spark-submit ──▶ kube-apiserver ──▶ schedules driver pod
                                          │
                                          └──▶ driver requests executor pods
                                                    │
                                                    └──▶ kube-apiserver schedules executors

Since Spark 3.1, this mode is officially marked production-ready. Before that, it was experimental — and you will occasionally still meet teams running Spark 2.4 on k8s, which is why interviewers sometimes probe whether you know the version where stability landed. If you are asked when to use it in production, the safe answer is "Spark 3.1 or newer, anything else is a research project."

A few details that interviewers like to hear:

• The driver pod runs your application code (main method, DAG construction, scheduler).
• Executor pods run task code (transformations, shuffles, output writes).
• Each executor pod is a single JVM, sized by spark.executor.memory and spark.executor.cores.
• Resource limits are translated into k8s resource requests and limits, so the Kubernetes scheduler can pack pods onto nodes alongside other workloads.

That last point — multi-tenant scheduling — is the architectural advantage of k8s over YARN, and it is worth saying out loud during the interview.

spark-submit in cluster mode

The canonical invocation looks like this. You should be able to write it from memory.

spark-submit \
  --master k8s://https://kubernetes.default.svc:443 \
  --deploy-mode cluster \
  --name spark-pipeline \
  --class com.example.MainJob \
  --conf spark.executor.instances=10 \
  --conf spark.executor.memory=4g \
  --conf spark.executor.cores=2 \
  --conf spark.kubernetes.container.image=spark:3.5 \
  --conf spark.kubernetes.namespace=data-platform \
  --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark-sa \
  local:///opt/jobs/app.jar

Three things are non-obvious and worth calling out. First, the master URL uses the k8s:// scheme and points at the API server, not at any Spark master. There is no Spark master process — Kubernetes itself is the cluster manager. Second, --deploy-mode cluster is almost always what you want; client mode runs the driver on your laptop and is fine for debugging but is unsuitable for production. Third, the container image must contain the Spark distribution that matches your client version. A version skew between the local spark-submit binary and the image is the single most common source of cryptic failures, and a good interviewer will probe it.

Dynamic allocation on k8s

Dynamic allocation lets Spark scale the executor count up and down based on pending task pressure. Before Spark 3.0, this required an external shuffle service running as a sidecar on every node — a real operational headache on k8s, where the daemonset model conflicts with how shuffle files are stored.

Since Spark 3.0, dynamic allocation on k8s works without an external shuffle service by using a feature called executor decommissioning. When an executor is about to be killed (because it is idle or because the cluster autoscaler reclaimed the node), Spark proactively migrates its shuffle blocks to surviving executors. This is the modern path and the one you should describe in an interview.

The minimum config is short:

spark.dynamicAllocation.enabled=true
spark.dynamicAllocation.shuffleTracking.enabled=true
spark.dynamicAllocation.minExecutors=2
spark.dynamicAllocation.maxExecutors=50
spark.dynamicAllocation.executorIdleTimeout=60s

The two settings that matter most: shuffleTracking.enabled=true is what unlocks decommissioning, and maxExecutors is what protects your cluster from a runaway query consuming every node. If you skip maxExecutors, you will eventually wake up to a Sunday-night incident where one bad SQL query has scheduled 800 pods across your platform.

Spark Operator and declarative jobs

spark-submit is imperative — you fire it, it runs. The Spark Operator (from kubeflow-adjacent projects, originally maintained by Google) flips this to a declarative model using a Kubernetes Custom Resource Definition (CRD) called SparkApplication. You write a YAML manifest, apply it, and the operator reconciles the desired state by creating driver and executor pods on your behalf.

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: nightly-aggregations
  namespace: data-platform
spec:
  type: Scala
  mode: cluster
  image: spark:3.5
  mainClass: com.example.NightlyJob
  mainApplicationFile: local:///opt/jobs/app.jar
  driver:
    cores: 1
    memory: 2g
    serviceAccount: spark-sa
  executor:
    cores: 2
    instances: 10
    memory: 4g

The benefits in production are real. The operator integrates with the k8s ecosystem of tooling — Prometheus metrics scraping, structured logging into a sidecar, GitOps-style management via ArgoCD or Flux, RBAC-controlled namespaces per team. You stop treating Spark jobs as one-off shell scripts and start treating them as versioned platform objects.

The trade-off is operational surface area. The operator itself is a controller you have to deploy, upgrade, and monitor. For small teams running fewer than a dozen jobs, plain spark-submit inside an Airflow DAG is often still the right call. Mention this trade-off if asked — it shows you have run this in production rather than only read about it.

k8s vs YARN — the comparison interviewers want

Every interview that touches Spark on k8s eventually arrives here. Be ready with a side-by-side, not a vague "k8s is more modern" hand-wave.

In 2026, new clusters at Databricks customers, batch shops next to Snowflake, and ML platforms at Anthropic-style labs are built on Kubernetes. YARN is what you maintain for an existing Hadoop estate nobody has budget to migrate.

Common pitfalls

The first pitfall is version skew between the spark-submit client and the container image. If your local CLI is Spark 3.5.1 but the image you point at is 3.4.0, you will get an opaque serialization error or a missing-class exception during driver startup. The fix is to pin both sides to the same patch version in CI and to fail your build if they drift apart.

The second pitfall is forgetting to set maxExecutors on dynamic allocation. A single bad query — usually a missing join predicate that explodes into a cross product — can request hundreds of executors in seconds. Without a cap, Spark will happily ask Kubernetes for every pod the cluster autoscaler will give it, and you will spend the next morning explaining a five-figure cloud bill. Always set spark.dynamicAllocation.maxExecutors and treat it as a hard limit, not a hint.

The third pitfall is assuming external shuffle service is required. On YARN it was. On Kubernetes since Spark 3.0, you should use shuffle tracking with decommissioning instead. Candidates who quote outdated advice signal they have not run this in production. The interviewer wants to hear shuffleTracking.enabled=true, not "we deploy a shuffle daemonset."

The fourth pitfall is misunderstanding the service account model. The driver pod needs a service account with RBAC permissions to create, list, and watch pods in its namespace. New teams almost always forget this and waste hours debugging a driver that starts but never spawns executors. The fix is a one-time RBAC ClusterRoleBinding that you ship with your platform helm chart and reference from every job.

The fifth pitfall is using client deploy mode in production. It pins the driver to whichever machine ran spark-submit, so if your CI runner dies, your job dies. Use cluster mode for anything unattended.

Related reading

• Spark fundamentals — Data Engineering interview
• Spark broadcast joins — Data Engineering interview
• Spark Catalyst and AQE — Data Engineering interview
• Spark Iceberg integration — Data Engineering interview
• Airflow deployment — Data Engineering interview

If you want to drill Data Engineering questions like this every day, NAILDD is launching with 1,500+ problems across Spark, Kafka, SQL, and system design.

FAQ

Is Spark on Kubernetes production-ready?

Yes, as of Spark 3.1 and onward. Earlier versions (2.4, 3.0) supported it but carried the experimental label and had real rough edges, especially around dynamic allocation and shuffle. From 3.1 forward, large-scale users at Databricks, Lyft, Pinterest, and Apple have published case studies running Spark on k8s for petabyte-scale batch workloads. The honest answer for an interview: production-ready on Spark 3.1+, and the default for any new cluster being built today.

Do I still need HDFS if I run Spark on k8s?

No. The whole point of running Spark on Kubernetes is that you decouple from the Hadoop ecosystem. Storage typically lives in object storage — S3, GCS, or Azure Blob — and is accessed through table formats like Apache Iceberg, Delta Lake, or Hudi. HDFS is still a valid backing store if you happen to have one, but no new platform in 2026 stands up HDFS to feed Spark on k8s.

When should I prefer YARN over Kubernetes for Spark?

Two scenarios. First, you already operate a large Hadoop cluster with a mature YARN setup and shared HDFS — migrating without a forcing function is expensive. Second, your team has zero k8s expertise and Spark is the only workload pushing you onto it. In every other case, prefer Kubernetes.

What is the difference between Spark Operator and bare spark-submit?

spark-submit is imperative — you run it, pods get created, the job runs. The Spark Operator is a controller that watches SparkApplication custom resources and reconciles them into the same pods, giving you declarative, GitOps-friendly definitions and lifecycle features like retries and TTL cleanup. For dozens of recurring jobs, the operator is worth the extra moving part. For a handful, spark-submit from an Airflow task is simpler.

How do I size driver and executor pods on Kubernetes?

For a typical ETL job, executors with 4 GB memory and 2 cores are a sane default. The driver usually needs 1–2 GB and 1 core unless you collect large results back to it. Always set memory requests equal to limits to avoid pod evictions, and reserve roughly 10 percent overhead above your Spark memory for the JVM.

What is executor decommissioning and why does it matter?

It is the mechanism Spark 3.0 introduced to drain shuffle data off an executor before killing it. When dynamic allocation decides an executor is idle, the executor migrates its shuffle blocks to surviving executors and only then shuts down. This is what lets Spark on Kubernetes run dynamic allocation without an external shuffle service — the simplification that finally made the platform viable at scale.