OpenLineage for Data Engineering interviews

What OpenLineage actually is

OpenLineage is an open standard for emitting and collecting lineage events from data pipelines. The whole point is to decouple the producer (Airflow, dbt, Spark, Flink) from the consumer (Marquez, DataHub, OpenMetadata, Atlan, custom warehouse). The producer doesn't care who's listening; the consumer doesn't care who's emitting. That separation is the single most important idea to lead with on an interview panel.

If the interviewer asks "why not just use DataHub directly?" the answer is that DataHub is a catalog and OpenLineage is a protocol. Before the standard existed, every catalog shipped its own ingestion plugin for every orchestrator — quadratic complexity. OpenLineage collapses that to N producers + M consumers wired through one event shape. Same story as OpenTelemetry vs vendor APM agents.

The mental model is a chain: operator runs → emit event → HTTP POST → backend ingests → UI graph. Every box in that chain is swappable.

Airflow task → OpenLineage event (JSON) → Marquez / DataHub → lineage graph in UI

The event spec, line by line

Every event is a JSON document with a fixed envelope. The pieces hiring panels expect you to name:

A real COMPLETE event looks like this:

{
  "eventType": "COMPLETE",
  "eventTime": "2026-05-25T08:14:33.221Z",
  "producer": "https://github.com/OpenLineage/openlineage-airflow/1.18.0",
  "schemaURL": "https://openlineage.io/spec/2-0-2/OpenLineage.json",
  "run": {
    "runId": "9c2d8e0a-3b1f-4a2e-9c6d-1f5a7b9c0d1e",
    "facets": {
      "nominalTime": {
        "nominalStartTime": "2026-05-25T08:00:00Z",
        "nominalEndTime": "2026-05-25T09:00:00Z"
      }
    }
  },
  "job": {
    "namespace": "etl.prod",
    "name": "process_orders.silver_orders"
  },
  "inputs": [
    {"namespace": "warehouse.raw", "name": "orders_raw"}
  ],
  "outputs": [
    {"namespace": "warehouse.silver", "name": "silver_orders",
     "facets": {
       "schema": {"fields": [
         {"name": "order_id", "type": "STRING"},
         {"name": "amount_usd", "type": "DECIMAL"}
       ]}
     }}
  ]
}

Notice the facets. Facets are the extension mechanism — pluggable, versioned blobs of metadata attached to a run, job, or dataset. The spec defines a standard set (schema, dataQuality, columnLineage, sql, errorMessage, ownership, documentation) and lets vendors add their own under a custom namespace. This is where most of the real value lives — bare events without facets give you DAG nodes but no column-level lineage and no data-quality signal.

The START event for the same run carries the same runId but eventType: "START" and no outputs. The backend stitches them by runId. If FAIL arrives, the backend marks the run as failed and surfaces the errorMessage facet.

Marquez as a reference backend

Marquez is the reference implementation maintained by the OpenLineage project. It's a small Java service backed by Postgres with a React UI. Architecture is deliberately boring: HTTP endpoint accepts events, writes to a normalized schema (jobs, runs, datasets, dataset_versions, lineage_events), the UI queries it.

When to reach for Marquez:

• Pilots and demos. Spin it up in Docker, point Airflow at it, get a lineage graph in an afternoon.
• Small platform teams that don't need the catalog features (governance, glossary, popularity) of DataHub or OpenMetadata.
• CI assertions — run Marquez in a test harness to verify your pipeline emits the events you expect.

When to skip Marquez:

• You already have DataHub, OpenMetadata, Atlan, or Collibra. They all ingest OpenLineage natively. Adding Marquez creates two sources of truth.
• You need column-level lineage at scale across thousands of datasets — Marquez's UI starts to crawl past a few thousand jobs.
• You need governance workflows (data contracts, stewardship, access requests). Marquez is a viewer, not a catalog.

Integrations with Airflow, dbt, Spark

The producer side is where 90% of interview questions land. Here's the matrix:

Airflow. The provider auto-wires a Listener plugin that fires on task_instance.running, task_instance.success, and task_instance.failed. For each task it tries to extract input/output datasets — PostgresOperator parses the SQL, S3CopyObjectOperator reads bucket paths, BashOperator gets nothing unless you add a manual extractor. Set AIRFLOW__OPENLINEAGE__TRANSPORT to your collector URL and you're done.

dbt. Two paths. The wrapper approach (dbt-ol run from openlineage-dbt) shells out to dbt, parses target/run_results.json + target/manifest.json, and emits one event per model. The newer path is dbt's own openlineage adapter, where dbt emits events as it goes. The wrapper is more reliable today; the adapter is where the project is heading.

Spark. Drop the JAR on the classpath, set spark.extraListeners=io.openlineage.spark.agent.OpenLineageSparkListener, and configure spark.openlineage.transport.url. The listener hooks into the QueryExecutionListener and pulls datasets out of the logical plan. This is the only integration that gives you column-level lineage from arbitrary user code without any SQL parsing of your own — it's worth knowing why.

Custom. When the platform team has homegrown orchestration, the SDK is the escape hatch. You call client.emit(RunEvent(...)) at the start and end of each job. Tedious but explicit.

Common pitfalls

The first trap is forgetting the START event. Some teams instrument only the success path and emit one COMPLETE event per run. The backend then can't compute duration, can't show runs in progress, and misses any run that started but never finished. Always emit START first with the same runId, even if you have to refactor the wrapper to do it. The lineage graph is built from completes, but the run timeline needs both.

A second trap is dataset namespace inconsistency. One pipeline emits outputs: [{"namespace": "snowflake://acme.region", "name": "raw.orders"}] and another emits {"namespace": "snowflake", "name": "acme.region.raw.orders"}. The backend treats them as two distinct datasets, lineage breaks across boundaries, and the column-level graph turns into disconnected islands. Pick a namespace convention up front — usually <engine>://<account> for the namespace and <db>.<schema>.<table> for the name — and enforce it in code review.

The third trap shows up at scale: event volume blowing up the collector. A Spark job with thousands of micro-tasks can emit tens of thousands of events per minute. Marquez's Postgres backend chokes around 10k events/min without tuning. The fix is either to filter events at the producer (only emit on the driver, not per-stage), buffer through Kafka with a consumer that batches inserts, or run a sampled emitter for high-cardinality jobs. Mention Kafka in the interview — it's the answer the platform team wants to hear.

A fourth common mistake is trusting auto-extracted lineage blindly. The Airflow operator may not understand your SQL dialect, the Spark listener may not detect a writer that wraps df.write in custom code, and dbt's manifest can miss inputs hidden inside Jinja macros. Treat the auto-emitted graph as 80% correct on day one and budget for manual facets to fill the gaps. Sticking column lineage on top of a partially-wrong dataset graph compounds the error.

Finally, schema evolution silently breaks downstream consumers. The schema facet captures the columns at emit time, but if your producer is on spec 1.0.5 and your backend expects 2.0.x, fields get dropped during ingestion with no error. Pin schemaURL in the producer, monitor spec versions on both sides, and treat OpenLineage upgrades as a coordinated platform release — not a quiet pip install --upgrade.

Operational tips

For production rollouts, transport choice matters more than backend choice. The default HTTP transport is fine for low-volume Airflow installs (a few thousand events per day) but synchronous POSTs add latency to every task. For high volume, switch to the Kafka transport — events drop on a topic, a consumer fans them out to backends, and producer pipelines decouple from backend availability. If Marquez is down, your Airflow DAGs don't slow down.

Set OPENLINEAGE_DISABLED=true as a kill switch. Every SDK respects it, every integration checks for it. The day Marquez wedges and starts returning 500s, you want a one-environment-variable rollback, not a code change to every DAG.

For column-level lineage, the cheap win is enabling the columnLineage facet on dbt (set dbt-ol to parse compiled SQL) and Spark (it's on by default in recent versions). Airflow's column lineage is weaker — it depends on the SQL parser in the provider, which lags behind dialect features. Don't promise column lineage for hand-written SQL in PostgresOperator without testing first.

Run a smoke test in CI: spin up Marquez in Docker, run a tiny DAG, query the Marquez API for the expected datasets, fail the build if the lineage graph doesn't match the golden file. This catches namespace drift before it hits prod.

Related reading

• Data lineage for Data Engineering interviews
• Airflow for Data Engineering interviews
• Airflow XCom for Data Engineering interviews
• dbt fundamentals for Data Engineering interviews
• Spark RDD vs DataFrame vs Dataset

If you want to drill DE interview questions like this every day, NAILDD has a growing bank of platform and lineage problems modeled on real hiring panels.

FAQ

Is OpenLineage the same as DataHub or OpenMetadata?

No, and conflating them is the fastest way to lose credibility on a panel. OpenLineage is a protocol — a JSON event spec plus client SDKs for emitting those events. DataHub and OpenMetadata are catalogs — they ingest OpenLineage events (among other sources) and store them with governance metadata, search, and a UI. You usually run both: OpenLineage on the producer side, a catalog on the consumer side.

Do I need Marquez to use OpenLineage?

No. Marquez is the reference implementation, useful for pilots and small teams, but OpenLineage events are just HTTP POSTs of JSON. You can write them to a flat file, drop them on Kafka, or send them to DataHub, OpenMetadata, Atlan, Collibra — anything that speaks the spec. The interview-grade answer is "I'd use Marquez for a proof of concept and route events to the org's existing catalog in production."

What does column-level lineage actually mean here?

The columnLineage facet on an output dataset maps each output column to the input columns it depends on. So for SELECT a.user_id, b.amount FROM users a JOIN orders b ON a.id = b.user_id, the facet says user_id depends on users.id, and amount depends on orders.amount. dbt and Spark emit this for free; Airflow's coverage depends on the SQL operator. It's what unlocks impact analysis like "if I drop users.email, what breaks?"

How does OpenLineage compare to OpenTelemetry?

Same conceptual playbook — a vendor-neutral protocol for instrumentation — but different domain. OpenTelemetry is for traces, metrics, and logs; OpenLineage is for data lineage events. They overlap zero in terms of payload but rhyme in terms of architecture: SDK on the producer, OTLP/HTTP transport, backend consumes. Some platform teams run both — OTel for service observability, OpenLineage for pipeline observability.

What's the relationship between OpenLineage and dbt's manifest?

dbt's manifest.json already contains the model graph and ref relationships — that's structural lineage at parse time. OpenLineage adds runtime lineage: which models actually ran, which sources they read, what columns they produced, and whether they succeeded. The wrapper dbt-ol reads the manifest plus run_results.json and emits OpenLineage events that capture both. If you only need structural lineage, the manifest alone is enough; if you need run history and SLA tracking, OpenLineage is the bridge.

Can I emit OpenLineage events from a custom Python pipeline?

Yes — pip install openlineage-python, instantiate OpenLineageClient, and emit RunEvent objects at start and complete. It's the canonical escape hatch for in-house orchestrators, ML training jobs, or any place auto-instrumentation doesn't reach. The trade-off is that you have to construct datasets, facets, and run IDs by hand, so it's verbose. Most teams write a small wrapper around the SDK that hardcodes their namespace conventions.