You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It would be tremendously useful to be able to subscribe to multiple topics via a wildcard. The config file would then look something like this:
[INPUT]
Name kafka
Brokers 192.168.1.3:9092
Topics .*channel-topic.*
poll_ms 100
[OUTPUT]
Name stdout
In a world where you may have X number of "channel-topics", you could elegantly subscribe to all of them via a wildcard. The current approach, where you have to list a comma-separated list of topics, is not practical for a large amount of topics.
Also, architecturally speaking, having a wildcard is extremely useful when more topics get created in the future, because they'll never be missed. Example: say you introduce 5 new "channel-topics" in the future; with the current setup, unless you remember to go and add them to the comma-separated list, they will not be consumed by fluentbit. But if you use wildcard chars, they would automatically be monitored. If you follow good naming conventions in your project, your topics will have well-defined names, so this would be a no-brainer.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
It would be tremendously useful to be able to subscribe to multiple topics via a wildcard. The config file would then look something like this:
In a world where you may have X number of "channel-topics", you could elegantly subscribe to all of them via a wildcard. The current approach, where you have to list a comma-separated list of topics, is not practical for a large amount of topics.
Also, architecturally speaking, having a wildcard is extremely useful when more topics get created in the future, because they'll never be missed. Example: say you introduce 5 new "channel-topics" in the future; with the current setup, unless you remember to go and add them to the comma-separated list, they will not be consumed by fluentbit. But if you use wildcard chars, they would automatically be monitored. If you follow good naming conventions in your project, your topics will have well-defined names, so this would be a no-brainer.
Lastly, wildcard subscription should be supported by Kafka libraries:
https://github.com/dpkp/kafka-python/blob/68c8fa4ad01f8fef38708f257cb1c261cfac01ab/kafka/consumer/group.py#L717
Beta Was this translation helpful? Give feedback.
All reactions