NATS consumers not pulling messages from Jetstream. Restarting NATS server solved the problem

[funkye]

Observed behavior

There are messages for consumers in the stream (confirmed by analyzing the stream file and publisher logging), but the messages do not reach the consumers. NATS CLI also shows that there are no messages for these consumers.

Restarting the consumers does not change anything. Restarting the server led to the messages being sent to consumers and the NATS CLI began to show other information:

And after consuming the message:

The problem occurred for 13 out of 100 consumers of one stream.
There is nothing suspicious in the server logs for this period (around 15.10.2024 20:30:00)
nats_logs.zip

Expected behavior

Messages must be sent to consumers consistently.

Server and client version

NATS server version: 2.10.20, not a cluster
.NET client version: 2.2.3

Host environment

OS: Ubuntu 22.04.1

Steps to reproduce

No stable reproduce.

[neilalexander]

Can you please post the nats stream info for the stream too?

It looks like before the restart, the stream sequence moves back down, which makes me think that this might be something to do with #6014. Can you please test v2.10.22 and let me know if that improves things or if the same thing happens?