Kafka share groups swap partition ownership for a 2,000-record lock budget

A share-partition does not track a committed offset. It keeps a window of records, each carrying its own state, and no partition rebalance happens when a member joins or leaves ^[6]. The broker locks a record for 30 seconds by default while a consumer works on it ^[10].

The budget is 2,000 unacknowledged records per share-partition, set by `group.share.partition.max.record.locks` and raised from 200 in Kafka 4.0 ^[8]. Divide that budget by the average time a record spends delivered and you have the group's throughput ceiling ^[9]. At 10 milliseconds of processing, one share-partition tops out near 200,000 records a second ^[1]. If the handler stalls until the lock expires, the same budget yields about 67 ^[2]. One default produces a factor of roughly 3,000 between the two answers. The only input that transfers from someone else's cluster is your own in-flight time ^[5]. A record that expires on every attempt holds its slot for up to 150 seconds across five deliveries ^[4].

The dev.to study lists what the application has to bring: an idempotent destination, per-record error handling, lock sizing, new metrics and new autoscaling rules, and it reports extra cost on the brokers as well ^[12]. A group takes 200 members by default and can be configured up to 1,000 ^[11]. The member cap is the ceiling on decoupling consumer count from partition count, and on a four-partition topic under a consumer group the fifth and sixth consumers receive nothing at all ^[19].

The study works from KIP-932 and the Apache Kafka 4.0-to-4.2 source, so it holds for any distribution that implements the protocol. For metrics, limits and availability it uses Amazon MSK as its managed reference ^[17]. Consumer groups are unchanged by any of this; a share group is a second model, chosen per group ^[18]. Where per-partition order is required, the study keeps the consumer group and its committed offset ^[7][21].