why the rule "cluster_verb_scope_le:apiserver_request_sli_duration_seconds_bucket:increase1h" does not add {job="apiserver"}

[haima95]

What happened?

Did you expect to see some different?

How to reproduce it (as minimally and precisely as possible):

Environment

• Prometheus Operator version:

  quay.io/prometheus-operator/prometheus-operator:v0.72.0

• Kubernetes version information:

  kubectl version
  Client Version: v1.28.2
  Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
  Server Version: v1.28.11

• Prometheus Logs:

why the rule "cluster_verb_scope_le:apiserver_request_sli_duration_seconds_bucket:increase1h" does not add {job="apiserver"} ???
`

• expr: |
  sum by (cluster, verb, scope) (increase(apiserver_request_sli_duration_seconds_count{job="apiserver"}[1h]))
  record: cluster_verb_scope:apiserver_request_sli_duration_seconds_count:increase1h
  • expr: |
    sum by (cluster, verb, scope) (avg_over_time(cluster_verb_scope:apiserver_request_sli_duration_seconds_count:increase1h[30d]) * 24 * 30)
    record: cluster_verb_scope:apiserver_request_sli_duration_seconds_count:increase30d
  • expr: |
    sum by (cluster, verb, scope, le) (increase(apiserver_request_sli_duration_seconds_bucket[1h]))
    record: cluster_verb_scope_le:apiserver_request_sli_duration_seconds_bucket:increase1h
  • expr: |
    sum by (cluster, verb, scope, le) (avg_over_time(cluster_verb_scope_le:apiserver_request_sli_duration_seconds_bucket:increase1h[30d]) * 24 * 30)
    record: cluster_verb_scope_le:apiserver_request_sli_duration_seconds_bucket:increase30d
    `

then that will case the function in apiserver_request:availability30d get wrong value, the value is less than zero

`

write too slow

        sum by (cluster) (cluster_verb_scope:apiserver_request_sli_duration_seconds_count:increase30d{verb=~"POST|PUT|PATCH|DELETE"})
        -
        sum by (cluster) (cluster_verb_scope_le:apiserver_request_sli_duration_seconds_bucket:increase30d{verb=~"POST|PUT|PATCH|DELETE",le="1"})

`

Anything else we need to know?:

[haima95]