Consumers/Executors hanging

[kaveh-hariri]

Hi,

I'm using the distributed approach, 1 queue, 10 consumers/containers. After some time, a consumer will "hang" in the sense that the streaming window closes and moves on, but the container from the previous window never finishes. I've noticed that this causes a big drag in performance.

In the picture below there should be 10 consumers/containers, but RabbitMQ shows 12. The prefetch count is at 100, and you see the messages ready and unacked messages at 200. This is the new floor as 2 of the consumers/containers are hung up. If I kill the job completely and restart, then these 2 hung consumers go away. Any idea what may be causing this? I've also pasted my configurations below.

[nelsou]

+1 got it too. I made a quick fix with a Thread that force to close "unused connection".

[compae]

Hi,

@nelsou could you make one PR with the bugfixing?

[yenki]

+1 I'm having this problem also