Whats the right sizing of producers & consumers on a partitioned topic with all subscription types ?

[nareshv]

Other messaging systems that support competing consumer pattern allows multiple consumers on a queue with same subscription name.

Kafka supports one consumer group to have multiple consumers that have 1:1 mapping with the partitions in a topic

With Pulsar, due to multiple types of consumption, it isnt' very clear on right sizing the (#producers , topic, #partitions, subscription, subscriptionType, #consumers) to ensure that there is steady state.

| Producers/Partition | Partitions in Topic | Subscription | Subscription Type | Consumer Instances |
|---------------------|---------------------|--------------|-------------------|--------------------|
| 1                   | 10                  | 1            | Exclusive         | ??                 |
| 1                   | 10                  | 1            | Failover          | ??                 |
| 1                   | 10                  | 1            | Shared            | ??                 |
| 1                   | 10                  | 1            | Key_Shared        | ??                 |

[lhotari]

There's probably not a definite answer to the right sizing, but I'll try to provide some answers to other questions.

The Shared and Key_Shared subscription types implement the competing consumer pattern in Pulsar. Multiple consumers use the same subscription name.

In the case of partitioned topics, each partition is independent in the Pulsar broker and the aggregation is handled on the client side.
You can use the same subscription types also in that case.

Client side consumers can consumer from all partitions using the name of the partitioned topics, and there's also the possibility to consume from individual partitions. The name of the partition for a topic is the topic name appended with "-partition-N" where N is 0-based index for the partition.

While each partition is independent on the broker side, there are some small details where there are dependencies between the partitions on the client side. For producing there is a dependency on the client side such that if the send queue for a single partition is full, it will block sending also for other partitions. It's reasonable that this happens, but it might become as a surprise when doing failure tests for Pulsar.
Producers can also be created to individual partitions independently and in some use cases this might be useful. The Java client also supports providing a MessageRouter implementation by the application for choosing the partition for individual messages on a partitioned topic producer.

[nareshv]

Actually, my question is a very simple one. Trying to understand if messages/partitions left behind in any of the 4 Subscription Modes

When we create a pulsar producer on a partitioned topic, pulsar will write to all partitions.

Similarly, when we create a Pulsar Consumer , want to understand if it subscribes to all partitions or not, w.r.t 4 subscription modes.

Thanks for explaining the per-partition-topic based produce/consume. Right now not looking at that flexibility.

[lhotari]

Similarly, when we create a Pulsar Consumer , want to understand if it subscribes to all partitions or not, w.r.t 4 subscription modes.

When using Shared or Key_Shared, each consumer instance will subscribe to all partitions of a partitioned topic.

The only other usable option is Failover consumer where one consumer will be active for a single partition at a time. There's a solution for balancing the consumers of individual partitions across the instances so that a single instance doesn't consume all partitions when there are multiple instances available. I found the documentation in the javadoc of ConsumerBuilder.priorityLevel.
ConsumerBuilder.priorityLevel applies also to Shared subscription type, but it has a different purpose, it's for assigning the actual priorities or weights.

I guess it's rare to have a use case for exclusive consumers for multi-partitioned topics although it would work technically.
The use case for an exclusive consumer is for active-passive configurations and usually there needs to be partition level control so that all partitions are handled on a single instance instead of splitting consumers across multiple instances. Perhaps there are good usecases also for exclusive multi-partitioned consumers. Someone else could elaborate on this.

[yuweisung]

Please share if the subscriptions, is not clear to you.
Thanks.