Is it possible to use multiple threads to consume message, rather than process?

[xiaoronglv]

we have defined 4 consumers in our project, when starting the project, I have to start 4 process to consume the messages in kafka.

bundle exec racecar FooConsumer
bundle exec racecar BarConsumer
bundle exec racecar AppleConsumer
bundle exec racecar BananaConsumer

Is it possible to run all consumers in one process? it will significantly reduce the overhead.

e.g.

bundle exec racecar *

[dasch]

No, and it's unlikely we will support that. Separating each consumer into its own process isolates the workloads from each other, and allows scaling each independently. I don't think the memory overhead of running separate processes is a big enough problem on modern computing platforms to warrant introducing a shared memory model.

[xiaoronglv]

I don't think the memory overhead of running separate processes is a big enough problem on modern computing platforms to warrant introducing a shared memory model.

Thanks for your reply. I would like to propose again.

In the beginning, many web and job process frameworks are process-based—for example, the unicorn and Resque.

If people would like to serve more traffic, they have to spawn more processes. Each process shares nothing with others, which requires the developer to upgrade the server to a larger CPU and Memory.

Then, people implemented the thread-based solution to save cost, for example, puma and sidekiq. The new framework uses a thread to respond rather than a process.

Zendesk is a successful corp, I believe the throughput of the message is huge, that's the reason you recommend isolating each consumer to a different process.

But for small businesses, the throughput is very low, we would like to use one process to handle several consumers to save cost.

[xiaoronglv]

It can help small businesses save lots of money if racecar could support this kind of feature.

bundle exec racecar FooConsumer BarConsumer AppleConsumer BananaConsumer

or

bundle exec racecar  *

[mensfeld]

No, and it's unlikely we will support that.

Since you do not plan to support this, I hope you don't mind me pointing out ppl that needs this to karafka as it supports this instead of separate processes (as racecar): https://github.com/karafka/karafka/tree/2.0

[dasch]

Sure.

I would also say that you can use simple process managers such as Foreman to spin up multiple processes in with a shared parent; I don't think the overhead is anything that a small business should be worried about.