-
-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Huge performance hit on stream with 3M unique subjects [v2.10.18] #5803
Comments
Each call to if (msg.info.pending === 0) {
break
} Could you check with that change? |
That's a good point, figured it out reading examples earlier and tried already.
|
I made a POC with rudimentary code replacing KV and JS with mongodb collections but with the same code base and some indexes. I know the driver is optimized and enqueue changes but I added write concern to 1, so I have an ack from the MongoDB instance, effectively throttling to prevent the server from overwhelming. Tell me if I can help in any way. |
Observed behavior
We have built a pipeline that, from various information sources, allows us to produce an entity whose data is composed on the fly.
The messages are stored in a stream with
Per Subject Messages Limit 1
and incoming messages have this subject templateproduct.{sku}.{source}
This allows us to maintain in NATS the latest information from a source for a given product.
The pipeline subscribes to
product.>
to trigger the composition of a product sheet, which it stores in a KV bucket, a concept similar to an event store. During this phase, the pipeline must collect all the messages concerning a SKUproduct.{sku}.>
To do this, an ephemeral consumer is created to collect these messages and then deleted :
This is called each time for each message in the stream.
With a base of 50,000 messages, this phase takes an average of 7ms, which is acceptable at this stage.
However, during testing 3,500,000 messages this phase takes an average of 420ms, which represents a significant performance hit.
It is important to note that no publication occurs in parallel. The messages were loaded in advance before the test was triggered.
Disk I/O is pretty low
Expected behavior
We except a smaller performance hit.
Server and client version
The server is standalone 2.10.18
Javascript client 2.28.2
Host environment
Docker version 26.0.2
Volume driver : local
Steps to reproduce
No response
The text was updated successfully, but these errors were encountered: