Airflow Sensors for Data Engineering Interviews

Why interviewers grill you on sensors

Almost every production Airflow DAG eventually waits for something — a file landing in S3, an upstream DAG finishing, a row appearing in a Postgres table. That waiting is exactly where naive pipelines burn worker slots, miss SLAs, and surprise on-call engineers at 3 a.m. So when a hiring manager at Snowflake, Stripe, or Databricks pulls up a whiteboard and says "design the ingestion for our partner feed," the very first follow-up is usually: how do you wait for the file to arrive without melting the cluster?

The honest answer involves three load-bearing concepts — sensor mode (poke vs reschedule), deferrable operators backed by the Triggerer, and timeouts that prevent zombie tasks. Get those three right and you sound like someone who has paged themselves at 3 a.m. before. Get them wrong and the interviewer immediately knows you copy-pasted DAGs from a tutorial. This guide walks through what to say, what code to write on the whiteboard, and what trap questions to expect.

If you have only ever used PythonOperator and BashOperator, sensors are the next layer of Airflow fluency that separates "uses Airflow" from "owns Airflow in production."

What an Airflow sensor actually is

A sensor is a special Operator whose only job is to wait until a condition becomes true, and then mark itself successful so downstream tasks can run. It is still a task — it occupies the same scheduling and logging plumbing as every other task — but its run loop is a polling loop rather than a single shot of work.

from airflow.sensors.filesystem import FileSensor

wait_for_file = FileSensor(
    task_id="wait_for_partner_drop",
    filepath="/data/incoming/partner_feed.csv",
    timeout=3600,        # fail after one hour
    poke_interval=60,    # check once a minute
    mode="reschedule",
)

Two parameters carry almost all the weight. timeout is the wall-clock budget after which the sensor gives up and fails the task — never omit it, or a stuck upstream will pile up infinite waiting tasks. poke_interval is how often the sensor checks the condition; 60 seconds is the safe default, and anything below 10 seconds is almost always a mistake.

Poke vs reschedule mode

In poke mode (the default), the sensor's Python process stays alive for the entire wait. Every poke_interval seconds it runs the check; in between checks it sleeps but still holds a worker slot. A four-hour wait at poke_interval=60 ties up one slot for four hours even though it does maybe 240 seconds of real work.

In reschedule mode, each failed check releases the slot and reschedules the sensor as a brand-new task instance poke_interval seconds later. The worker pool is free between checks, and the scheduler simply picks the task back up when it is due. The trade-off is metadata DB load — every reschedule writes rows to the task instance table — so very tight intervals like 5 seconds turn the DB into the bottleneck.

The interviewer's favorite trap: someone enables mode="reschedule" on a sensor with poke_interval=5. You just turned a worker problem into a metadata DB problem. Push poke_interval to 60 seconds or more when running reschedule.

Sensor types you must know

The Airflow ecosystem ships dozens of sensors. You will not memorize all of them, but you must know the five that come up on whiteboards. Below is the cheat sheet I would draw if asked.

A few patterns worth seeing in code. ExternalTaskSensor is the one most candidates get wrong:

from datetime import timedelta
from airflow.sensors.external_task import ExternalTaskSensor

wait_for_upstream = ExternalTaskSensor(
    task_id="wait_for_upstream",
    external_dag_id="ingest_orders",
    external_task_id="load_final",
    execution_delta=timedelta(minutes=30),
    mode="reschedule",
    timeout=60 * 60 * 2,
    poke_interval=120,
)

The execution_delta is the difference between this DAG's logical date and the upstream DAG's logical date. If both DAGs run on the same schedule, leave it at zero — but if downstream runs at 09:00 and upstream at 08:30, you owe Airflow that 30-minute delta. Forget it and the sensor will wait for a task instance that does not exist and never will.

SqlSensor is the one that bites teams who treat it like a free lunch:

from airflow.providers.common.sql.sensors.sql import SqlSensor

new_rows_landed = SqlSensor(
    task_id="check_landed",
    conn_id="warehouse",
    sql="SELECT 1 FROM raw.events WHERE event_date = '{{ ds }}' LIMIT 1",
    mode="reschedule",
    poke_interval=300,
    timeout=60 * 60 * 6,
)

Every poke runs that SQL. On a 50-billion-row table, even a LIMIT 1 with no partition pruning can scan a lot of data. Always add a partition filter, and consider an indexed sentinel table instead of polling the raw fact table.

FileSensor and S3KeySensor look superficially identical, but their cost models differ — local FS checks are essentially free, while an S3 head_object call is a paid API request. At scale, S3KeySensor in poke mode at 5-second intervals can rack up real money on AWS bills. Use reschedule or deferrable.

Deferrable operators — the modern path

The deferrable model introduces a new component called the Triggerer: an async Python process that uses asyncio to watch thousands of conditions concurrently. When a deferrable sensor runs, it does the cheap setup work synchronously, then yields control to the Triggerer with a Trigger object describing what it is waiting for. The worker slot is released immediately. The Triggerer polls (or, for true event-driven sources, subscribes), and when the condition fires it pushes the task back into the queue for a regular worker to complete in milliseconds.

from airflow.providers.amazon.aws.sensors.s3 import S3KeySensor

wait_for_drop = S3KeySensor(
    task_id="wait_for_partner_drop",
    bucket_key="s3://partner-bucket/2026/05/feed.csv",
    deferrable=True,
    poke_interval=60,
    timeout=60 * 60 * 4,
)

Why this matters in interview answers: a single Triggerer can hold thousands of waiting tasks in roughly the memory footprint of one worker. That is a 1000x reduction in worker capacity needed to handle the same fan-out. Companies running large Airflow fleets — Airbnb, Uber, DoorDash — moved sensitive pipelines to deferrable specifically to stop overprovisioning workers.

The catch is that not every provider exposes a deferrable variant, and the ones that do sometimes hide the option behind a different class name like S3KeySensorAsync in older releases. Always check the provider docs for your Airflow version before promising the interviewer "everything is deferrable now."

Common pitfalls

The trap I see most often in interview answers is leaving a long sensor in poke mode. A junior engineer copies a FileSensor snippet from the docs, sets poke_interval=30, and ships it. Six months later the team is paying for double the worker capacity because half the slots are sleeping on file-arrival waits. The fix is to flip to mode="reschedule" for waits longer than a minute or two, and to deferrable for anything longer than an hour — the migration is usually a one-line code change.

A second trap is omitting timeout. Without a timeout, a sensor will happily wait forever, and a stuck upstream system turns into a slow-motion outage as task instances accumulate. Every sensor in production must have an explicit, documented timeout, and the timeout must be shorter than the SLA of the DAG it lives in. If your DAG SLA is six hours, no single sensor inside it should have an eight-hour timeout — the math gives the on-call engineer no warning.

The third trap is polling too aggressively. poke_interval=5 feels responsive but it generates 12 checks per minute, 720 per hour, and in reschedule mode that is 720 metadata DB writes per sensor per hour. Multiply by a few hundred running sensors and the Airflow scheduler grinds. Sixty seconds is the floor for most production sensors; thirty seconds is acceptable only if you have measured the DB and the cost is fine.

A fourth, sneakier trap is the misconfigured ExternalTaskSensor. If the two DAGs run on different schedules and you forget execution_delta (or its more flexible cousin execution_date_fn), the sensor silently waits for a task instance that never existed and you only discover it when the SLA fires. Always pair ExternalTaskSensor with an explicit delta and, ideally, an integration test that verifies both DAGs' schedules.

The fifth trap is expensive SqlSensor queries. The sensor will run that query on every poke. A SELECT COUNT(*) over an unpartitioned 50-billion-row table is a way to discover, the expensive way, that your warehouse has per-query costs. Push the predicate down, point at a partition, or maintain a tiny landing-marker table that the sensor can probe cheaply.

The sixth trap is ignoring deferrable when it is available. If you are on Airflow 2.7+ and the provider supports deferrable, opting out is leaving free efficiency on the table. The interviewer will ask why you used reschedule, and "it's what I'm used to" is not a great answer at a Databricks loop.

Related reading

• Airflow data engineering interview
• Airflow backfill data engineering interview
• Airflow deployment data engineering interview
• What is Apache Airflow

If you want to drill Airflow and data engineering whiteboards at this depth every day, NAILDD is launching with hundreds of DE-focused problems across exactly these patterns.

FAQ

Should I use reschedule mode or deferrable operators?

If you are on Airflow 2.7+ and the provider exposes a deferrable variant, prefer deferrable. It releases the worker slot and pushes the wait into the Triggerer, which can handle thousands of concurrent conditions in a single async process. Reschedule remains useful when the provider has not been ported, or when you are on an older Airflow version without a Triggerer running. Avoid plain poke mode for any wait longer than a couple of minutes.

How short can poke_interval safely be?

For poke mode, anything down to a few seconds is technically fine because the check is in-process. For reschedule mode the floor is closer to 30-60 seconds, because every check writes to the metadata database. For deferrable sensors backed by the Triggerer, the cost is mostly a coroutine wake-up, so 30 seconds is realistic; some event-driven triggers respond in real time without polling at all.

What happens if a sensor is restarted mid-wait?

Sensors must be idempotent: scheduler restarts, manual retries, and catchup runs can all cause a sensor to re-run from scratch. In practice this is fine because the condition either holds or it does not — the FileSensor will re-check the same path, the ExternalTaskSensor will re-check the same task state. Trouble appears only when you put non-idempotent side effects inside a sensor, which you should not be doing in the first place.

Can I write my own sensor?

Yes — subclass BaseSensorOperator and implement poke(self, context) returning True when the condition holds. For deferrable behavior, subclass BaseSensorOperator and implement the execute method to yield a Trigger. Custom sensors are common when waiting on an internal service that no provider covers, but always set sensible defaults for timeout, poke_interval, and mode.

Is there a hard cap on the number of concurrent sensors?

In poke mode, you are capped by your worker pool — every wait consumes a slot. In reschedule mode you are effectively unlimited from the worker angle but bounded by metadata DB write throughput. With a Triggerer and deferrable sensors, the practical limit is the Triggerer process's memory, which comfortably holds thousands of concurrent waits per process; large shops run multiple Triggerers behind a queue for redundancy.

Are sensors the right tool for event-driven pipelines?

Sometimes. If you genuinely have an event bus — Kafka, EventBridge, Pub/Sub — it is often cleaner to have that system push into a DAG via the Airflow REST API or a dataset-aware schedule than to poll for the event with a sensor. Sensors shine when the upstream system has no push capability and you must observe a side effect like a file landing or a row appearing.