Airflow deployment in a Data Engineering interview

Why this question shows up

When a Data Engineering interviewer asks "walk me through how you would deploy Airflow", they are checking whether you have actually run a scheduler that processes 10,000+ task instances per day without going down at 3 AM. The question filters out candidates who only know Airflow from a laptop with LocalExecutor and a SQLite metadata DB.

The signal the panel wants is concrete: which executor, how many schedulers, and what the failure mode looks like when the metadata Postgres goes read-only for two minutes during a Multi-AZ failover. If you cannot answer those three crisply, the loop will assume you only run toy pipelines. This guide assumes Airflow 2.7+ — older versions are irrelevant for interviews at Stripe, Airbnb, DoorDash, Snowflake, or Databricks in 2026.

Components you must name

Before you touch executors, you have to enumerate the moving parts. An interviewer who hears "Airflow has a scheduler and workers" knows you have never deployed it. The full component list is six pieces.

The webserver serves the UI and REST API, sitting behind a load balancer with at least two replicas. The scheduler parses DAG files, evaluates schedule intervals, and queues task instances — since 2.0 you can run multiple active schedulers, coordinating through row-level locks on the metadata DB. The worker executes tasks; its shape depends on the executor. The triggerer is a separate async process for deferrable operators — long sensors that should not pin a worker slot. The metadata DB stores DAG state, task instances, XCom payloads, and connection secrets — Postgres 13+ is standard. The message broker (Redis or RabbitMQ) is only present with a Celery-flavored executor.

Executors and when to pick each

The executor choice is the first real decision and the one interviewers probe hardest. Memorize the trade-offs in a table you can sketch on a whiteboard in 30 seconds.

LocalExecutor is fine until you outgrow one machine. The day you need a second host, you move to Celery — not because it is elegant, but because the operational story is well understood and every cloud has a runbook. Workers register with the broker, pull tasks, and report state back to the metadata DB. Scaling is horizontal: add workers, raise worker_concurrency, watch broker queue depth.

KubernetesExecutor is the right call when DAGs span very different dependency stacks — a Spark submitter, a dbt runner, a Pandas notebook, and a CUDA inference job in one Airflow. Each task spawns its own pod with its own image. The cost is 10-60 seconds of pod startup overhead per task, a poor fit for thousands of tiny tasks. That is when CeleryKubernetesExecutor earns its keep: small tasks ride the Celery pool, heavy or weird tasks declare executor_config and get their own pod.

The lazy answer is "we use Celery". The interview answer is "Celery for the bulk, Kubernetes for the long tail, selected per-task."

High availability that actually holds

HA in Airflow has three independent failure domains and you must address all of them. New candidates address one and hope the rest is implicit.

Schedulers — run at least two. They coordinate through SELECT ... FOR UPDATE SKIP LOCKED on the metadata DB, so no leader election is needed. Two also reduce scheduler latency — the gap between ready and queued — which on a single scheduler can creep above 10 seconds under load. Three is the practical cap; beyond that DB contention costs more than the latency win.

Metadata DB — the real single point of failure. Use managed Postgres with synchronous replication: RDS Multi-AZ, Cloud SQL HA, or Azure Flexible Server zone-redundant. Plan for a 30-90 second failover window during which schedulers throw and running tasks cannot heartbeat. Set core.killed_task_cleanup_time high enough that cleanup does not murder mid-failover tasks.

Broker (Celery only) — Redis cluster or RabbitMQ in mirrored-queue mode. Workers behind an autoscaling group, with worker_concurrency tuned so one pod uses 60-80% CPU at steady state. Lower wastes money; higher pushes into OOM on bursts. The webserver is the easy one — stateless, behind an ALB, two replicas, health check on /health.

Kubernetes executor in practice

When you describe Kubernetes deployment, the interviewer wants to hear pod_override come out of your mouth. The pattern that earns nods looks like this:

from kubernetes.client import models as k8s

heavy_task = PythonOperator(
    task_id="train_ranker",
    python_callable=train,
    executor_config={
        "pod_override": k8s.V1Pod(
            spec=k8s.V1PodSpec(
                containers=[
                    k8s.V1Container(
                        name="base",
                        image="registry.example.com/ml-train:2026.05",
                        resources=k8s.V1ResourceRequirements(
                            requests={"cpu": "4", "memory": "16Gi"},
                            limits={"cpu": "8", "memory": "32Gi"},
                        ),
                    )
                ],
                node_selector={"workload": "ml-gpu"},
                tolerations=[
                    k8s.V1Toleration(key="gpu", operator="Exists", effect="NoSchedule")
                ],
            )
        )
    },
)

Call out three things unprompted. First, per-task images — the trainer runs CUDA while the rest of the DAG runs generic Airflow. Second, per-task resources — 16 GiB guaranteed, 32 GiB ceiling, without inflating the worker default. Third, node affinity and tolerations — the pod lands on a GPU node group ordinary tasks cannot touch.

Kubernetes also cleans up the secrets story: mount k8s secrets per pod, rotate out of band, use RBAC to limit which DAGs read what. On Celery, every worker has every secret — fine for a small team, not fine when ten squads share one Airflow. The cost is real: pod startup latency, k8s API pressure on fan-out, and engineers who understand kubelet, not just kubectl apply.

# values.yaml fragment for the official Helm chart
executor: "CeleryKubernetesExecutor"
workers:
  replicas: 6
  resources:
    requests: { cpu: "2", memory: "4Gi" }
    limits:   { cpu: "4", memory: "8Gi" }
postgresql:
  enabled: false  # use managed RDS / Cloud SQL instead
redis:
  enabled: true
  replicas: 3

Managed Airflow services

The interviewer will ask "would you self-host or use managed?" and the only wrong answer is "always self-host because it is cheaper". It is not, once you price engineer time honestly. Here is the comparison to draw.

A defensible heuristic: under 2,000 DAG runs per day with a small team, pay for managed and move on. Above 20,000 DAG runs per day with a platform team, self-host because the unit economics flip. The middle zone is ambiguous and is where the interviewer wants reasoning, not a rule.

Name alternatives honestly: Dagster for asset-aware orchestration, Prefect for Python-native simplicity, Argo Workflows when you are deep in Kubernetes and do not need a Python DAG abstraction, Apache Beam for unified batch and streaming. You are not obligated to defend Airflow; you are obligated to know when not to use it.

Common pitfalls

The first common pitfall is treating the metadata DB as a regular application database. Engineers point BI tools at it, dump it nightly into the warehouse, or run ad-hoc queries against it. Every one of those workloads competes with the scheduler for row locks and tanks scheduling latency. The fix is a strict rule: nothing reads the metadata DB except Airflow itself; for analytics, ship data out via a DagRun listener into a separate table.

A second trap is over-provisioning workers and under-provisioning the scheduler. Teams add workers reflexively when tasks queue, but the bottleneck is often scheduler parsing on large DAG repos. If dag_processor_manager cannot finish a parse loop in under 30 seconds, effective task latency degrades regardless of worker count. The fix is profiling slow DAG files, splitting the repo into directories with different parse intervals, and using the standalone DAG processor in Airflow 2.7+.

A third pitfall is mixing dev and prod secrets in connections. Airflow connections are global, so a careless dev DAG can read a prod warehouse credential if both live in one metadata DB. The fix is genuinely separate deployments per environment, each with its own metadata DB and its own Secrets Backend (AWS Secrets Manager, GCP Secret Manager, HashiCorp Vault) so credentials never round-trip through the Airflow DB at all.

A fourth pitfall is deploying the Helm chart with default Postgres. The bundled chart ships a single-replica Postgres meant for dev; teams promote it to prod and discover during the first node drain that the orchestration layer is on a pod with no persistent storage. Set postgresql.enabled: false from day one and point at managed Postgres with backups, PITR, and a tested restore runbook.

A fifth pitfall, specific to Kubernetes executor, is forgetting that every task pod must read the DAG files. Without git-sync or a baked DAG image, tasks fail on cold pods because the DAGs are not where the scheduler thinks they are. Pick one strategy — git-sync sidecar, baked image, or persistent volume — and use it consistently across executor and scheduler.

Related reading

• Apache Airflow at the DE interview
• Airflow backfill in the DE interview
• What is Apache Airflow
• SQL for the Data Engineer interview

If you want to drill Data Engineering questions like this every day, NAILDD is launching with 1,500+ interview problems across exactly this pattern.

FAQ

How many workers do I actually need?

Start with the formula workers x worker_concurrency >= peak parallel tasks, then add 20% headroom for autoscaling lag. For most teams the answer is 4-8 workers at 4-8 concurrency each, sized so that one worker uses 60-80% CPU at peak. If you cannot articulate peak parallel tasks, instrument the scheduler's running_tasks gauge for two weeks before sizing.

Can I run Airflow without a message broker?

Yes — LocalExecutor and KubernetesExecutor do not need one. Celery is the only flavor that requires Redis or RabbitMQ. If you are a small team on a single beefy host, LocalExecutor is genuinely fine and removes an entire failure domain. The cost is no horizontal scaling and no task isolation beyond OS processes.

What is the right Postgres size for the metadata DB?

For up to 50,000 DAG runs per day, a 4 vCPU / 16 GiB managed instance is plenty. Past that, the bottleneck is usually IOPS on the task instance table rather than CPU — provision SSD with at least 3,000 IOPS and watch the slowest scheduler queries with pg_stat_statements. Vacuum tuning matters: autovacuum_vacuum_scale_factor of 0.05 on task_instance and xcom keeps bloat under control.

Is CeleryExecutor deprecated in favor of Kubernetes?

No. Celery remains the default for the official Helm chart in 2026 because it is operationally simpler than Kubernetes for homogeneous workloads. The two are complementary — CeleryKubernetesExecutor lets you use both. Pick Kubernetes-only when isolation and per-task images are core requirements; pick Celery-only when you want the simplest path to production.

How do I handle DAG deployment without restarting workers?

Use git-sync as a sidecar on the scheduler and worker pods, pointed at a branch that your CI updates after tests pass. The DAG processor will pick up new files within the next parse interval — typically 30 seconds — without any pod restart. Worker code changes (custom operators, plugins) still require an image rebuild, which is one reason to keep operator logic thin and DAG logic thick.

What is the right Airflow version to claim experience with?

In 2026, claim 2.7 or 2.8 unless you have hands-on Airflow 3 alpha experience. Calling out the multi-scheduler behavior, the deferrable operator pattern, and the standalone DAG processor is what signals real production exposure. Naming a version older than 2.5 in an interview answer signals stale experience.