Hive Metastore for Data Engineering interviews

Why HMS shows up on every DE loop

You walk into a virtual onsite at a data platform team — a Snowflake/Databricks shop migrating off legacy EMR. The hiring manager asks you to whiteboard the read path for a query that lands in Trino and scans Parquet in S3. The first arrow you draw is almost always to a Hive Metastore (HMS). If you can't explain why that arrow exists, what lives at the other end, and what would replace it in 2026, you've already failed the systems half of the interview.

HMS is the sleeper question on DE loops at companies like Stripe, Airbnb, DoorDash, and Databricks. It's not glamorous like streaming, but it's the connective tissue of every lakehouse. Expect questions like "why does Iceberg need a catalog at all", "Glue vs HMS — when would you self-host", and "what breaks if HMS goes down for ten minutes". Candidates either over-rotate on Hive (the engine, mostly dead) or skip the catalog layer entirely.

What Hive Metastore actually is

HMS is a thin service that owns metadata for tables stored as files in a data lake — typically Parquet, ORC, or Avro in S3, GCS, or HDFS. The service is a Java daemon backed by a relational database (Postgres or MySQL). It exposes a Thrift API that engines like Spark, Trino, Presto, and Hive itself call over the network.

A useful mental model: the lake is the warehouse of physical pallets (files). HMS is the inventory clerk holding a clipboard with "shelf C, pallet 17 contains events partitioned by event_date". Without the clerk, every forklift driver walks the warehouse.

When SELECT * FROM events WHERE event_date = '2026-05-01' hits Trino, the engine asks HMS three things: what columns does events have, where do its files live, and which partitions match my filter. HMS replies with paths and stats; Trino reads the data straight from S3. HMS never touches the data — it only manages the metadata.

What HMS stores

The metadata model is small but loaded with semantics. A complete HMS row covers five categories of state:

Most candidates skip the stats column — and it's the one Trino and Spark Catalyst rely on for cost-based optimization. Stale or missing stats force the planner onto broadcast-vs-shuffle heuristics that are often wrong by orders of magnitude. That's the actual reason your "trivial" join takes 40 minutes.

All this state lives in the backing RDBMS (usually Postgres). The HMS daemon is essentially a typed cache plus a Thrift wire format over that database.

Integration with Spark and Trino

Spark. When you start a Spark session with spark.sql.catalogImplementation=hive, Spark uses HMS as its catalog. A query like:

spark.sql(
    "SELECT user_id, COUNT(*) "
    "FROM analytics.events "
    "WHERE event_date = '2026-05-01' "
    "GROUP BY user_id"
)

…triggers a Thrift call to HMS for schema and partition locations, then Spark reads matching Parquet files from S3. HMS is not in the data path — only the metadata path. Bandwidth through HMS is tiny (kilobytes per query), but latency matters because every job pays a round trip.

Trino. Trino models HMS as a connector called hive:

SELECT *
FROM hive.analytics.events
WHERE event_date = DATE '2026-05-01'
LIMIT 100;

Trino's killer feature is federated queries — you can join hive.analytics.events against postgres.public.users and kafka.streams.clickstream in a single SQL statement. Each connector has its own catalog; HMS is the catalog for lake tables.

dbt. dbt-spark and dbt-trino both write models that ultimately register through HMS. Your dbt run is, under the hood, a CREATE TABLE ... USING parquet LOCATION ... against HMS plus an Insert into S3.

AWS Glue Catalog and alternatives

Most teams in 2026 do not run HMS themselves. AWS Glue Catalog is a managed, HMS-API-compatible service that removes the operational pain of HA Postgres plus a Java daemon. The typical pipeline looks like:

Source data in S3 → Glue Crawler → Glue Catalog → Athena / Redshift Spectrum / EMR / Trino

A Crawler scans an S3 prefix, infers schema, and registers tables in Glue. Athena queries hit Glue for metadata and S3 for data. Pricing is per-object plus per-request; for most workloads it disappears into the AWS bill.

Beyond Glue, the modern catalog landscape splits along two axes — open vs proprietary and table-format-aware vs format-agnostic:

If an interviewer asks "would you self-host HMS in 2026?", the right answer is almost always no, unless you're locked into on-prem Hive or have specific compliance needs. The cost-benefit favors Glue, Unity, or Polaris.

Iceberg and Delta — do you still need HMS

Here's the most modern question on the loop, because it cuts to whether you understand the difference between a table format and a catalog.

Apache Iceberg carries its own metadata in S3 — manifest files, snapshots, and a metadata.json pointer per table. The catalog's only job is to know which metadata.json is current. So HMS becomes optional:

• HMS — works for legacy compatibility, but doesn't unlock Iceberg's branch/snapshot features fully.
• AWS Glue — most common production choice on AWS.
• Project Nessie — git-style branches and tags over Iceberg tables.
• Polaris — Snowflake's open Iceberg-native catalog.
• REST catalog — the new vendor-neutral spec; both Snowflake and Databricks support it.
• File-based — hadoop catalog, fine for laptops, never use in production (no atomic commits across writers).

Delta Lake is even more self-contained: its _delta_log directory holds the entire transaction log next to the Parquet files. HMS is purely for discovery — SHOW TABLES will list Delta tables, but the actual table identity lives in _delta_log. In Unity Catalog setups, even the discovery role moves up to Unity.

The directional trend is unmistakable. Iceberg + Glue or Nessie or Polaris is now the default greenfield stack at companies like Apple, Netflix, and Stripe. HMS hangs around in legacy Hive/Spark setups and in vendor-neutral on-prem deployments. Know the trend, but don't bury HMS too early — half the postings you'll interview at are mid-migration.

Common pitfalls

The classic scar is treating HMS as fire-and-forget infrastructure. The single-point-of-failure problem bites first. HMS sits in the metadata path of every query; when its Postgres backend hiccups, every Spark job, Trino query, and Athena scan blocks. The fix is two HMS instances behind a load balancer with a HA Postgres backend — RDS Multi-AZ or a managed cluster. In production it's the difference between a four-minute blip and a four-hour incident.

A subtler pitfall is untrusted statistics. Trino and Spark Catalyst lean hard on column-level NDV and row counts. If you write to a table but never run ANALYZE TABLE events COMPUTE STATISTICS, the planner falls back to assumptions like "every join is 1:1" and picks the wrong shuffle strategy. Bake ANALYZE into your dbt post-hook or Airflow DAG. Without this, a query that should run in 30 seconds takes 45 minutes.

Schema sprawl is the third trap. Hive table identifiers are bounded — historically 128 characters for fully-qualified names, and many backends silently truncate longer ones. Teams generate names like analytics_growth_funnel_step_three_completed_events_v2_backup and hit collisions. Use short names with descriptive comments in HMS properties.

Mass partition mutations are the fourth scar. ALTER TABLE ... ADD COLUMN against a table with 100,000+ partitions rewrites metadata for every partition. On self-hosted HMS this locks the backing database for hours. Either partition less aggressively (daily, not hourly) or use Iceberg, where schema evolution is metadata-only.

Finally, multi-region latency — HMS in us-east-1 while data lives in eu-west-1. Every round-trip pays the cross-region tax (60-90 ms per call), and queries with thousands of partition lookups feel underwater. Co-locate HMS with the storage it describes, or use a regional Glue Catalog on AWS.

Optimization tips

A few practical levers for system-design rounds. Tune the HMS Postgres connection pool — a default of 10 collapses under concurrent Trino workers; 50-100 is realistic. Enable direct SQL mode (hive.metastore.try.direct.sql=true) to bypass DataNucleus ORM for partition listing — a free 5-10x speedup. Add an index on PARTITIONS(SD_ID) to the backing schema; it becomes a bottleneck above ~1M partitions per table.

On Glue, the usual slowdown is partition pruning at the catalog level. Use partition projection for time-series tables — Athena computes partition values from the predicate instead of asking Glue. For Iceberg-on-Glue, enable hidden partitioning so catalog metadata stays small.

Related reading

• Apache Iceberg deep dive
• Lakehouse: Iceberg vs Delta
• Spark Catalog API
• Schema evolution interview guide
• Spark Iceberg integration

If you want to drill DE systems questions like this every day with structured feedback, NAILDD is launching with 500+ interview problems across Spark, lakehouse architecture, and SQL.

FAQ

What is the difference between HMS and HiveServer2?

HMS is the metadata catalog — a Thrift service backed by Postgres that answers "what tables exist and where do their files live". HiveServer2 is a separate query execution service that parses HiveQL, plans it, and runs MapReduce or Tez jobs. They're often deployed together in classic Hive clusters but solve different problems. In a Spark-plus-Trino lakehouse, you typically run HMS without HiveServer2 at all — the modern engines do their own planning and execution.

Can I use HMS without running Hive itself?

Yes, and that's the standard pattern in 2026. Spark, Trino, Presto, dbt-spark, and dbt-trino all integrate with HMS over Thrift without needing a HiveServer2 process anywhere in the stack. You install the HMS package (or use a Docker image), point it at a Postgres database, and configure your engines' catalog settings to find it. The "Hive" in the name is historical baggage.

Should I migrate from HMS to Glue Catalog?

If your data is already on AWS, almost always yes. Glue removes the operational burden of running HA Postgres and a Java daemon, integrates with IAM for table-level permissions, and connects natively to Athena, Redshift Spectrum, EMR, and Glue ETL. The migration itself is well-trodden — AWS provides a CLI tool that copies HMS metadata into Glue, and you can run both in parallel during cutover. The main reason not to migrate is multi-cloud or on-prem requirements where Glue would lock you to AWS.

How does Unity Catalog compare to HMS?

Unity Catalog is Databricks's catalog layer, recently open-sourced. It supersedes HMS for Databricks-first teams and adds fine-grained access control (column-level masking, row filters), data lineage captured automatically, and unified governance across Delta, Iceberg, views, and ML models. Unity speaks the Iceberg REST catalog spec, so it integrates with Trino, Spark, and Snowflake outside Databricks. Think of HMS as a 2012-era catalog and Unity as a 2024-era one — Unity is what you'd build if you started today.

Does Iceberg or Delta require HMS?

Neither requires HMS as a hard dependency. Iceberg stores its metadata in S3 manifest files and only needs a catalog to track the current snapshot pointer — that catalog can be HMS, Glue, Nessie, Polaris, REST, or even a file in S3 (for tests only). Delta Lake keeps its full transaction log in the table's _delta_log directory next to the Parquet files; a catalog is purely for discovery. The choice of catalog matters more for governance and concurrency (atomic multi-table commits, branching) than for table integrity.

How do I make HMS highly available in production?

Run at least two HMS instances behind a TCP load balancer (HMS uses Thrift over TCP, not HTTP). Point both at a HA Postgres backend — RDS Multi-AZ, Cloud SQL HA, or a self-managed Patroni cluster. Tune the pool to 50-100 connections per instance, enable direct SQL mode, and monitor Postgres connection saturation as your top SLI. Bake ANALYZE TABLE into your loading pipelines so stats never drift. With these in place, HMS becomes boring infrastructure — which is what you want.