Using Same Group ID with Multiple Consumer Groups Causes Problems

[ugrkm]

Description

I use ConsumerGroups to consume kafka messages. I have two instances that run on kubernetes. I give the same group ID to both instances. This is to have fault tolerance (one of the pod goes down, the other should be able to keep consuming). What happens is, two instances run side by side. One of them consumes messages alright. But, when I kill one of the pods, the other one can't keep consuming. My guess is maybe broker only accepts one of the pod's ConsumerGroup and just ignores the other one? Could someone please clarify the behavior?

Versions

Sarama	Kafka	Go
v1.43.2	3.5.1	1.22.2

Configuration

config.Consumer.Offsets.AutoCommit.Enable = false
config.Consumer.Offsets.Initial = sarama.OffsetOldest
config.Consumer.Return.Errors = true

[lbydev]

Does your current topic have only one partition? If so, only one consumer will be able to consume messages. Could you check the number of partitions in your topic?

[puellanivis]

Does your current topic have only one partition? If so, only one consumer will be able to consume messages. Could you check the number of partitions in your topic?

I mean, even if this were the case, it should redistribute partitions to the consumers if one of them fails, right?

[lbydev]

Thank you for your reply. I apologize for the misunderstanding. You are correct that Kafka should redistribute partitions among the remaining consumers in the group when one of them fails.