Spark and Iceberg on the data engineering interview

Why Spark + Iceberg shows up on the loop

If you are interviewing for a data engineering role at any company that runs a modern lakehouse — Netflix, Apple, Stripe, Airbnb, Snowflake, Databricks — expect at least one round on how Apache Iceberg plugs into Spark. Iceberg is the table format Netflix open-sourced to fix the partition evolution, schema evolution, and concurrent writer problems that Hive tables on S3 quietly created at scale.

The interview question is rarely "what is Iceberg" — that gets a one-line answer. The real question is operational: how do you configure the catalog, what happens when two writers append to the same partition, how do you migrate a 50 TB Hive table without downtime, and how do you keep small files from killing query latency.

Catalog setup

The catalog is where Iceberg stores the pointer to the current metadata file for every table. Without a catalog, Iceberg is just a pile of Parquet plus JSON on object storage. Spark needs three things to talk to an Iceberg catalog: the catalog implementation class, its type, and where the warehouse data lives.

spark.conf.set("spark.sql.catalog.iceberg", "org.apache.iceberg.spark.SparkCatalog")
spark.conf.set("spark.sql.catalog.iceberg.type", "hive")
spark.conf.set("spark.sql.catalog.iceberg.uri", "thrift://hive-metastore:9083")
spark.conf.set("spark.sql.catalog.iceberg.warehouse", "s3://my-warehouse/")
spark.conf.set("spark.sql.extensions",
  "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")

The Iceberg runtime jar — currently iceberg-spark-runtime-3.5_2.12 for Spark 3.5 — must be on the classpath. The spark.sql.extensions line is what unlocks CALL procedures like rewrite_data_files; forget it and the procedures look like syntax errors at runtime.

Once the catalog is wired up, you use SQL identifiers in three parts: catalog.namespace.table. This is the part candidates fumble most.

USE iceberg.warehouse;

CREATE TABLE events (
  id BIGINT,
  ts TIMESTAMP,
  user_id BIGINT,
  event_name STRING
) USING iceberg
PARTITIONED BY (days(ts))
TBLPROPERTIES (
  'write.format.default' = 'parquet',
  'write.target-file-size-bytes' = '536870912'
);

Note the days(ts) transform. Iceberg partition transforms are first-class: days, hours, bucket(N, col), truncate(N, col). Hive forced you to materialize a separate dt column; Iceberg derives it from ts and stores the mapping in metadata. This is what makes partition evolution safe.

Reads and writes with ACID

Reading an Iceberg table from Spark looks like any other DataFrame source. The interesting properties show up under load.

df = spark.read.format("iceberg").load("iceberg.warehouse.events")

# Same thing in SQL
df = spark.sql("""
  SELECT user_id, COUNT(*) AS event_count
  FROM iceberg.warehouse.events
  WHERE ts >= TIMESTAMP '2026-05-01'
    AND event_name = 'purchase'
  GROUP BY user_id
""")

# Append from a new DataFrame
new_df.writeTo("iceberg.warehouse.events").append()

# Idempotent overwrite of a single partition
filtered_df.writeTo("iceberg.warehouse.events") \
  .overwritePartitions()

Two writers can append to the same table at the same time without overlapping logical data, and both will succeed. Each writer produces its own data files and proposes a new metadata file; the catalog uses an optimistic concurrency check to commit. If two writers touch the same rows, one commit wins and the other retries or fails.

The read side gets a consistent snapshot. Once Spark plans a query, it pins to a single snapshot ID; concurrent writes do not affect the running query. This enables time travel:

-- Read the state of the table as of a specific snapshot
SELECT * FROM iceberg.warehouse.events
VERSION AS OF 2348923489234;

-- Or as of a wall-clock time
SELECT * FROM iceberg.warehouse.events
TIMESTAMP AS OF '2026-05-15 00:00:00';

Practical use: reproducible reports, debugging late-arriving data, and recovery from a bad pipeline by rolling back with CALL iceberg.system.rollback_to_snapshot(...).

Schema evolution without rewrites

Schema evolution is where Iceberg embarrasses Hive. In Hive, renaming a column is a metadata bomb — the new name does not match the Parquet files, so queries silently return NULL. Iceberg avoids this with column IDs: every column gets a stable numeric ID at creation, and Parquet files store data under those IDs, not under names.

-- Add a column at the end (default: nullable)
ALTER TABLE iceberg.warehouse.events
  ADD COLUMN device_type STRING;

-- Rename without rewriting any data file
ALTER TABLE iceberg.warehouse.events
  RENAME COLUMN ts TO event_ts;

-- Reorder
ALTER TABLE iceberg.warehouse.events
  ALTER COLUMN device_type FIRST;

-- Widen a type (BIGINT → DECIMAL is allowed; reverse is not)
ALTER TABLE iceberg.warehouse.events
  ALTER COLUMN user_id TYPE BIGINT;

-- Drop a column (data files keep the bytes, reads ignore them)
ALTER TABLE iceberg.warehouse.events
  DROP COLUMN device_type;

All of these are O(1) metadata operations. A 100 TB table can be renamed-and-reordered in under a second.

Iceberg also supports partition evolution — switch from daily to hourly without rewriting history:

ALTER TABLE iceberg.warehouse.events
  REPLACE PARTITION FIELD days(event_ts) WITH hours(event_ts);

Old data stays partitioned by day; new data lands partitioned by hour. Metadata tracks both, and query planning prunes correctly across the boundary.

Compaction and snapshot expiry

The biggest performance trap with Iceberg on Spark is small files. Every streaming micro-batch, every late backfill, every MERGE INTO produces new files. After weeks of streaming writes you can have hundreds of thousands of files under 1 MB per partition, and metadata overhead alone tanks query latency.

The fix is regularly scheduled compaction:

CALL iceberg.system.rewrite_data_files(
  TABLE => 'warehouse.events',
  strategy => 'binpack',
  options => map(
    'target-file-size-bytes', '536870912',
    'min-input-files', '10'
  )
);

This rewrites small files into larger ones — target size 512 MB is the common default for analytics workloads. For sorted queries (range scans on event time), use strategy => 'sort' with a sort order to physically cluster the data.

Compaction creates new snapshots but does not delete old ones, so disk usage grows monotonically until you expire snapshots:

CALL iceberg.system.expire_snapshots(
  TABLE => 'warehouse.events',
  older_than => TIMESTAMP '2026-04-01',
  retain_last => 5
);

The retain_last parameter is the safety net — even if older_than would delete everything, keep the most recent 5 snapshots. This protects time-travel reads and rollback capability. Run snapshot expiry on a slower cadence than compaction (weekly vs daily is typical).

Catalog options compared

The catalog is the part of the stack interviewers probe to see if you have actually shipped Iceberg or just read the docs. Different catalogs have very different operational properties:

In an interview, "we use Glue because we are already on AWS" is a perfectly good answer. "We use the file-system catalog in production" is a red flag — that catalog has no atomicity guarantees across writers and is documented as not-for-production.

Common pitfalls

The most common pitfall is treating Iceberg as a drop-in replacement for Hive without rewiring the writers. Engineers add the Iceberg jar, create a table, and then keep writing via the old Hive partition path. The result is files on disk that the Iceberg catalog never learns about — they are invisible to reads and survive snapshot expiry as orphans. The fix is to write through the Iceberg DataFrame writer (format("iceberg") or writeTo), never directly to the partition path on S3.

A second trap is skipping compaction in the first month of production, then trying to compact a year of small files in one job. That job runs for hours, holds a metadata commit lock long enough to fail readers, and may not finish before the cluster times out. Schedule compaction from day one with min-input-files set high enough that it no-ops on already-clean partitions.

Another pitfall is forgetting that Iceberg snapshots retain deleted data until expiry. Engineers run DELETE FROM to comply with a data deletion request, then are surprised the data is still readable via time travel three months later. To actually purge data you need DELETE and expire_snapshots newer than the delete, and remove_orphan_files. Compliance teams care about this distinction.

Finally, candidates routinely conflate Iceberg with Delta Lake or Hudi under pressure. Iceberg is a table format spec; Delta Lake is Databricks' format with tighter coupling to their runtime; Hudi optimizes for upsert-heavy workloads. Knowing which one solves which problem matters more than reciting feature lists.

Related reading

• Apache Iceberg deep dive — DE interview
• Iceberg time travel — DE interview
• Lakehouse: Iceberg vs Delta — DE interview
• Spark Catalyst and AQE — DE interview
• Spark broadcast joins — DE interview

If you want to drill data engineering questions like this every day, NAILDD is launching with hundreds of lakehouse, Spark, and SQL problems built from real interview loops.

FAQ

Should I use Iceberg or Delta Lake for a new lakehouse in 2026?

Pick based on engine portability. Iceberg has the broadest multi-engine support — Spark, Trino, Flink, Snowflake, BigQuery, DuckDB all read it natively. Delta Lake is the better choice if you are committed to Databricks and want the deepest engine integration available today. For a new build at a non-Databricks shop, Iceberg is the safer long-term bet because vendor lock-in is the failure mode that actually hurts five years later.

How do I migrate a Hive table to Iceberg without downtime?

Use CALL iceberg.system.migrate('namespace.hive_table'). This wraps the existing Parquet files in an Iceberg metadata layer in-place — no data movement. Once migrated, the table can no longer be read by pure-Hive engines. For zero-downtime cutover, run dual writes for one full backfill window, validate row counts and a few checksums, then flip readers.

What is the right cadence for compaction and snapshot expiry?

For high-write tables, run rewrite_data_files daily during a low-traffic window. For low-write tables, weekly is plenty. Snapshot expiry should lag compaction by at least one cycle — daily compaction, weekly expiry, with a retention window of 7 to 30 days. If no one has ever queried VERSION AS OF in your shop, you are paying storage for nothing.

Does Iceberg replace the need for a data warehouse like Snowflake or BigQuery?

It depends on your latency SLA. Iceberg on S3 with Spark, Trino, or Athena gives you cheap storage any engine can read — great for batch analytics and ML feature stores. Sub-second BI dashboards still want a closer-to-compute layer. A practical pattern: Iceberg as the source of truth, Snowflake or BigQuery as the serving layer for dashboards.

What is merge-on-read vs copy-on-write?

Copy-on-write rewrites the entire data file when any row is deleted or updated — high read performance, expensive writes. Merge-on-read appends a delete file recording which rows are tombstoned — cheap writes, more work at read time. Choose copy-on-write for read-heavy analytics with rare updates, merge-on-read for CDC-style pipelines where updates dominate.