[kube-prometheus-stack] Allow Overriding or Disabling Default Grafana Dashboards

[vladmalynych]

Is your feature request related to a problem ?

Problem:

The Grafana dashboard defined in charts/kube-prometheus-stack/templates/grafana/dashboards-1.14/k8s-resources-pod.yaml includes a graph for memory consumption per pod. The memory consumption query is currently sourced from a different repository (https://github.com/prometheus-operator/kube-prometheus):

https://github.com/prometheus-operator/kube-prometheus/blob/main/manifests/grafana-dashboardDefinitions.yaml#L8300

                  "targets": [
                      {
                          "datasource": {
                              "type": "prometheus",
                              "uid": "${datasource}"
                          },
                          "expr": "sum(container_memory_working_set_bytes{job=\"kubelet\", metrics_path=\"/metrics/cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", container!=\"\", image!=\"\"}) by (container)",
                          "legendFormat": "__auto"
                      },
                      {
                          "datasource": {
                              "type": "prometheus",
                              "uid": "${datasource}"
                          },
                          "expr": "sum(\n    kube_pod_container_resource_requests{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", resource=\"memory\"}\n)\n",
                          "legendFormat": "requests"
                      },
                      {
                          "datasource": {
                              "type": "prometheus",
                              "uid": "${datasource}"
                          },
                          "expr": "sum(\n    kube_pod_container_resource_limits{job=\"kube-state-metrics\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", resource=\"memory\"}\n)\n",
                          "legendFormat": "limits"
                      }
                  ],
                  "title": "Memory Usage (WSS)",
                  "type": "timeseries"
              },

When a pod is restarted, the current query combines memory usage data from both the old and new containers, leading to temporary spikes in displayed memory consumption. This can cause the dashboard to show memory usage exceeding the container's memory limit, even though actual usage remains within limits.

Steps to Reproduce:

• Trigger a pod restart (e.g OOM kill, or Evict).
• Compare graphs with expression grouped by just container field with graph that has expression that groups by container and id:

"expr": "sum(container_memory_working_set_bytes{job=\"kubelet\", metrics_path=\"/metrics/cadvisor\", cluster=\"$cluster\", namespace=\"$namespace\", pod=\"$pod\", container!=\"\", image!=\"\"}) by (container, id)"

Describe the solution you'd like.

Possible solution:

Would it be possible to add functionality in Helm to allow default charts to be overwritten or patched?

Describe alternatives you've considered.

Alternative solutions:

Would it be possible to add functionality in Helm to disable specific dashboards ?

Additional context.

This issue is also rised in prometheus-operator/kube-prometheus#2522