Kafka consumer groups on the data engineering interview

Why interviewers love this topic

Kafka is the streaming backbone at almost every place that hires data engineers, so consumer groups sit at the intersection of distributed systems intuition, operational scar tissue, and the discipline to reason about exactly-once without hand-waving. Fluency on offset commits, rebalance behavior, and partition assignment carries most DE loops.

Questions at Stripe, Airbnb, and Snowflake follow the same shape. Mid-level: how does a group work, what happens on restart. Senior: walk through a rebalance storm, when to pick CooperativeStickyAssignor, why static membership matters. Staff: design Kafka-to-Postgres with effectively exactly-once, justify every config.

The cost of getting it wrong is concrete. A fintech DE set enable.auto.commit=true with an 8-second processing loop, the pod was OOM-killed mid-batch, and on restart 1.2 million events re-emitted into a non-idempotent downstream. Cleanup took two days.

What a consumer group actually is

A consumer group is a set of clients that share the same group.id and split partitions between them. The contract is simple: each partition is assigned to exactly one consumer in the group at a time, and offsets are tracked per group. That gives queue semantics inside a group and pub-sub across groups — one topic can feed an ETL pipeline, a fraud model, and a search indexer without any of them blocking the others.

Topic "events" — partitions [0, 1, 2, 3]

Consumer Group "etl-warehouse"
├─ consumer-a   → partitions [0, 1]
└─ consumer-b   → partitions [2, 3]

Consumer Group "fraud-scoring"
├─ consumer-c   → partitions [0, 1, 2, 3]   (single instance, owns all four)

Two facts trip people up. If you spin up more consumers than partitions, the extras sit idle — parallelism is hard-capped by partition count, which is why partition planning is design-time. And offsets live in the compacted __consumer_offsets topic on the brokers, so a restarted consumer with the same group.id resumes from the last committed position. The group coordinator — a broker chosen by hashing the group.id — tracks membership, triggers rebalances, and accepts commits.

Offsets and commits

An offset is the consumer's position in a partition. The commit makes it durable. Get commit semantics wrong and you lose data or double-process it — usually discovered months later when an analyst spots duplicates in a fact table.

Four commit modes matter. Auto-commit (enable.auto.commit=true) commits whatever poll() last returned on a 5-second timer — the most common source of "we processed this twice" incidents, because the commit can fire before processing finishes. Manual sync commit (commit_sync() after each batch) gives at-least-once and is the right default. Per-partition manual commit lets you advance partitions independently when one is jammed by a poison pill. Transactional commit, paired with an idempotent producer, is the only way to get true exactly-once within Kafka.

# at-least-once with manual commit — the boring correct answer
from kafka import KafkaConsumer

consumer = KafkaConsumer(
    "events",
    group_id="etl-warehouse",
    bootstrap_servers=["broker-1:9092", "broker-2:9092"],
    enable_auto_commit=False,
    auto_offset_reset="earliest",
    max_poll_records=500,
    session_timeout_ms=30_000,
    max_poll_interval_ms=300_000,
)

for batch in consumer:
    try:
        process(batch)              # idempotent sink, please
        consumer.commit()           # commit AFTER successful processing
    except Exception:
        # do not commit — next poll will redeliver, downstream must be idempotent
        raise

Rebalance and the stop-the-world problem

A rebalance triggers whenever group membership changes: a consumer joins, leaves cleanly, misses session.timeout.ms of heartbeats, or the topic gains partitions. In the eager protocol the whole group stops, the coordinator reassigns, every consumer revokes everything, then resumes. On a 50-instance group reading 1,000 partitions, this stop-the-world pause runs 30 to 90 seconds — long enough to look like an outage.

Two operational levers matter: static membership and incremental cooperative rebalancing, both since Kafka 2.4. Static membership ties a consumer to a group.instance.id so a pod restart inside session.timeout.ms does not trigger a rebalance. Cooperative rebalancing turns the eager "revoke everything" flow into a two-phase protocol where only the partitions that must move are revoked, and the rest of the group keeps processing.

Pair static membership with cooperative-sticky and a rolling deploy of a 100-pod consumer fleet finishes without anyone losing partitions, and the dashboards never blink.

The single config change with the biggest reliability win on most streaming pipelines.

Partition assignment strategies

The partition.assignment.strategy config decides how the coordinator hands partitions out. Four strategies matter — differences hit fairness and rebalance cost.

RangeAssignor (default before 2.4) sorts partitions and consumers, hands out contiguous ranges. Fine for one topic, skews badly across multiple topics in one group. RoundRobinAssignor distributes one partition at a time for the most even spread, but does not preserve assignments across rebalances. StickyAssignor keeps as many existing assignments as possible on the next rebalance. CooperativeStickyAssignor combines sticky-ness with cooperative rebalancing — the modern default and in 2026 the right pick for nearly every new pipeline on Kafka 3.x and 4.x.

One detail: CooperativeStickyAssignor requires every consumer in the group to support it. A rolling upgrade with mixed strategies falls back to eager until the migration finishes, which is why production migrations are two passes — add the new strategy to the list ([CooperativeStickyAssignor, RangeAssignor]), finish the rolling deploy, then drop the old one.

Delivery semantics

Three semantics, one decision tree. At-most-once commits before processing; if the pod dies after commit, the message is gone. At-least-once commits after processing; if the pod dies between processing and commit, the next poll redelivers and downstream sees the message twice. This is the default and the only honest answer for most pipelines.

Exactly-once (EOS) is what interviewers probe hardest, and the honest answer is "it depends on where the data is going." Within Kafka — read, transform, write — you get true exactly-once with transactional.id on the producer, isolation.level=read_committed on the consumer, and sendOffsetsToTransaction that atomically commits the offset with the produced records. GA since 0.11, foundation of Kafka Streams EOS.

Outside Kafka — Postgres, S3, BigQuery — exactly-once needs cooperation from the sink. Two patterns: two-phase commit (rare, painful) or at-least-once plus an idempotent receiver, which is what every serious shop does. INSERT ... ON CONFLICT (event_id) DO NOTHING, an S3 sink keyed by offset, a Snowflake MERGE keyed by event id — all turn at-least-once into effectively exactly-once without distributed transactions.

-- the boring exactly-once pattern that actually works in production
INSERT INTO fact_events (event_id, user_id, occurred_at, payload)
VALUES ($1, $2, $3, $4)
ON CONFLICT (event_id) DO NOTHING;

Two producer configs are non-negotiable: enable.idempotence=true (default in 3.x) prevents broker-side duplicates from producer retries, and acks=all ensures every in-sync replica acks before the write is considered committed. Without idempotence, a retry after a network blip writes the same record twice and your "exactly-once" pipeline is broken at the source.

Lag and monitoring

Consumer lag is latest_offset - committed_offset for a partition — the single most important health metric for a streaming pipeline. Sum across partitions for a group total, watch per-partition values for skew, and alert on rate of change as well as the absolute number. A flat 2-million-message lag is fine for batch; a lag growing at 10k/sec in real-time is an incident.

Tooling: kafka-consumer-groups.sh --describe --group <id> is the built-in answer for interviews. In production, shops run Burrow for evaluation-based lag, Kafka Lag Exporter for Prometheus, Cruise Control for cluster rebalancing, or Confluent Control Center.

When lag is rising, the order of operations is: confirm the bottleneck is the consumer and not the downstream sink (usually it isn't); raise max.poll.records and fetch.max.bytes to pull more per poll; parallelize processing inside the consumer with a worker pool while keeping commits on the main thread; and only as a last resort, increase partitions and add consumers. Adding partitions rehashes the key-to-partition mapping for new messages and breaks per-key ordering, which is why it's last, not first.

Common pitfalls

Auto-commit with long-running processing is the most common production failure mode. The auto-commit timer fires every five seconds by default, so if your processing loop takes twelve seconds and the pod crashes at second nine, the offset is already committed for records you never finished. The fix is enable.auto.commit=false plus commit_sync() only after the batch is durably handled downstream.

Setting max.poll.interval.ms too low turns a slow consumer into a rebalance storm. If processing genuinely takes four minutes and the interval is the default five, a single GC pause pushes you over the limit, the broker assumes the consumer is dead, triggers a rebalance, processing gets slower, another rebalance fires. Size the interval against observed p99 processing time, not the average, and monitor time-between-poll-avg as an early warning.

A single partition on a high-throughput topic is a one-lane bridge. Consumer parallelism is hard-capped by the partition count, so a single-partition topic can never have more than one active consumer in a group. Plan partition counts at design time using the rule that one partition handles 10 to 50 MB/sec, then round up 2-3x for headroom.

Forgetting group.id puts the consumer in an anonymous mode where every restart picks a new identity, no offsets persist, and the consumer either replays from earliest or jumps to latest depending on auto.offset.reset. The fix is trivial — always set group.id to a meaningful, environment-prefixed name like etl-warehouse-prod.

Reusing the same group.id across unrelated pipelines is the silent killer. Two services with the same group.id compete for partitions in the same group, each sees a random subset of messages, and neither works correctly. Use one group.id per logical consumer, prefixed by environment and service, and a CI check that fails the build on reuse.

Related reading

• Apache Kafka on the systems analyst interview
• Apache Iceberg deep dive for data engineering interviews
• ClickHouse MergeTree on the data engineering interview
• Airflow on the data engineering interview
• MERGE and UPSERT on the data engineering interview

If you want to drill data engineering questions like this every day, NAILDD is launching with 500+ interview problems covering Kafka, Spark, Airflow, and the rest of the streaming stack.

FAQ

How many partitions should a topic have?

The starting heuristic is one partition per 10-50 MB/sec of sustained throughput, so a topic doing 500 MB/sec wants 10 to 50 partitions. Round up 2-3x for growth, because adding partitions later rehashes the key-to-partition mapping and breaks per-key ordering. Brokers typically tolerate a few thousand partitions before metadata pressure bites.

Can two consumers in the same group share a partition?

No. Exactly one consumer per partition per group at any moment, because Kafka's ordering guarantee is per-partition. If you need more parallelism than your partition count allows, add partitions, or parallelize inside a single consumer with a worker pool while keeping the poll loop single-threaded.

What happens if a consumer dies mid-commit?

Commits to __consumer_offsets are atomic broker writes — committed or not, no half-state. If the consumer crashes after the broker accepted the commit, the next consumer reads the new offset and moves forward. If the consumer crashes before the broker accepted, the next consumer redelivers the batch — which is why downstream must be idempotent for at-least-once.

Is there an acks=all for consumers?

No, acks is producer-side and controls how many in-sync replicas must acknowledge a write. Consumers have parallel concepts on the fetch side — fetch.min.bytes, fetch.max.wait.ms, isolation.level — but no symmetric acks. The closest analog is isolation.level=read_committed, which skips records from aborted producer transactions.

How do I read a topic from the beginning?

auto.offset.reset=earliest only takes effect when the group has no committed offset — useful for first-time consumers, useless for resetting an existing pipeline. To force an existing group back to the beginning, call seek_to_beginning() after subscribing, or run kafka-consumer-groups.sh --reset-offsets --to-earliest --execute while the group is stopped.

Is this official Apache Kafka documentation?

No, this guide is a synthesis based on public Apache Kafka 3.x and 4.x docs, Confluent platform docs, and production practice. Consult upstream docs for config reference, and verify behavior against your Kafka version — rebalance protocols evolved meaningfully between 2.x and 4.x.