The memory usage of jetstream pod is large [v2.10.20]

[AllenZMC]

Observed behavior

The memory usage of jetstream pod is large and remains at 2Gi. At this time, the jetstream stream does not have any memory data. Why is this?

In addition, when the pod is restarted, the memory usage will be very small, about tens of MB.

What is the reason for this?

Expected behavior

When the jetstream stream does not have any memory data, it will theoretically reduce memory usage.

Server and client version

nats-server: v2.10.20

Host environment

No response

Steps to reproduce

No response

[wallyqs]

if the system is running in k8s, need to make sure that you are setting GOMEMLIMIT so that that GC does not work based on the host memory instead.

[neilalexander]

Can you please provide memory profiles so we can see what's going on?

Either nats server request profile allocs using the system account, or https://docs.nats.io/running-a-nats-service/nats_admin/profiling.

[AllenZMC]

Can you please provide memory profiles so we can see what's going on?

Either nats server request profile allocs using the system account, or https://docs.nats.io/running-a-nats-service/nats_admin/profiling.

@neilalexander
How to configure prof_port = 65432 in jetstream？
The jetstream pod fails to run with an error message: "This parameter is not recognized"

[neilalexander]

It's a configuration option on the NATS Server config file, if you are doing it under Kubernetes you may need to do something like this:

config:
  merge:
     prof_port: 65432

[neilalexander]

Also note that you don't need prof_port enabled to use the NATS CLI, you just need access to the system account.

[AllenZMC]

mem.prof.zip
@neilalexander Here is the memory profile.

[derekcollison]

Thanks @AllenZMC! @neilalexander will take a look.

[neilalexander]

Thanks for the profile, was this taken when the memory usage was inflated? It is only showing 30MB heap allocations. Are you using GOMEMLIMIT?

[AllenZMC]

Thanks for the profile, was this taken when the memory usage was inflated? It is only showing 30MB heap allocations. Are you using GOMEMLIMIT?

It is indeed obtained when the memory is very large. I also feel confused.

And what is GOMEMLIMIT? NATS runs in a container. Is this configuration in the Go program or somewhere else?

[derekcollison]

How are you measuring memory?

[AllenZMC]

How are you measuring memory?

container_memory_working_set_bytes{pod='a', namespace='b', container='c'}

[derekcollison]

Is that RSS or VSS?

[AllenZMC]

Is that RSS or VSS?

container_memory_working_set_bytes is a metric of the k8s container platform, and it is the amount of physical memory actually used by the container, excluding memory that can be reclaimed, not RSS or VSS.

[derekcollison]

That means its RSS, Resident Set Size.

Do you set GOMEMLIMIT as an ENV for the pod running the server?

This is required for Go applications when the host has more memory then is being allocated to the pod running the nats-server.

[AllenZMC]

That means its RSS, Resident Set Size.

Do you set GOMEMLIMIT as an ENV for the pod running the server?

This is required for Go applications when the host has more memory then is being allocated to the pod running the nats-server.

No set.

[derekcollison]

What are you restricting the pod's memory to? What does the underlying host memory look like?

[AllenZMC]

What are you restricting the pod's memory to? What does the underlying host memory look like?

I configured resource limits in the container, for example

apiVersion: v1
kind: Pod
metadata:
  name: frontend
spec:
  containers:
  - name: app
    resources:
      requests:
        memory: "64Mi"
        cpu: "250m"
      limits:
        memory: "10Gi"
        cpu: "500m"

[derekcollison]

That is a big spread. You could try GOMEMLIMIT=6Gi

[AllenZMC]

In addition, I want to point out that I am using a memory stream.

When the stream data is automatically cleared, the memory still does not decrease much. In theory, if there is no data in the stream, NATS server should occupy very little memory.

[derekcollison]

What is the maximum size for the memory stream that you expect to have? What does stream info report for that stream when you see the general memory usage issue?

[AllenZMC]

What is the maximum size for the memory stream that you expect to have? What does stream info report for that stream when you see the general memory usage issue?

The maximum memory flow is 4G, no problem is seen, and when there is no message in the mem stream, the NATS pod service still occupies a large amount of memory. 😮‍💨

[neilalexander]

Can you please run nats server ls against the system account and report the result? Can you also get the result of running free -m within each of the NATS pods when the memory usage is high?