server,tests: add additional lease metrics and test #18711
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: vivekpatani The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
Hi @vivekpatani. Thanks for your PR. I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
Codecov ReportAttention: Patch coverage is
❗ Your organization needs to install the Codecov GitHub app to enable full functionality. Additional details and impacted files
... and 19 files with indirect coverage changes @@ Coverage Diff @@
## main #18711 +/- ##
==========================================
+ Coverage 68.79% 68.80% +0.01%
==========================================
Files 420 420
Lines 35523 35538 +15
==========================================
+ Hits 24437 24453 +16
+ Misses 9658 9655 -3
- Partials 1428 1430 +2 Continue to review full report in Codecov by Sentry.
|
| leaseAttached = prometheus.NewCounter(prometheus.CounterOpts{ | ||
| Namespace: "etcd_debugging", | ||
| Subsystem: "lease", | ||
| Name: "attach_total", | ||
| Help: "The number of leases that are attached to a lease item.", | ||
| }) | ||
|
|
||
| leaseDetached = prometheus.NewCounter(prometheus.CounterOpts{ | ||
| Namespace: "etcd_debugging", | ||
| Subsystem: "lease", | ||
| Name: "detach_total", | ||
| Help: "The number of leases that are detached from a lease item.", | ||
| }) |
There was a problem hiding this comment.
I feel like we can consolidate the two metrics (leaseAttached and leaseDetached) into one something like below,
leaseTotalAttachedKeys = prometheus.NewGaugeVec(prometheus.GaugeOpts{
Namespace: "etcd_debugging",
Subsystem: "lease",
Name: "attached_keys_total",
Help: "Total number of attached key for each lease",
},
[]string{"lease_id"},
)
There was a problem hiding this comment.
I don't have enough background about cardinality but my understanding is that, adding a key multiplies the cardinalities. But if not, then the idea sounds good. @ahrtr thoughts?
There was a problem hiding this comment.
adding a key multiplies the cardinalities
Yes, it's true. Indeed, It isn't a good idea to add label "lease_id". The prometheus official document clearly clarifies it https://prometheus.io/docs/practices/naming/#labels
There was a problem hiding this comment.
So
- either keep it as it's (two metrics, leaseAttached and leaseDetached), but shouldn't them counter instead of gauge?
- or consolidate them into one as mentioned above, but without the label "lease_id". In this case, it should be Gauge for sure.
There was a problem hiding this comment.
I have background in metrics cardinality. label_id is a big no-no. Please note that not everything can be expressed in metrics, they are aggregations that allow you to observe state of program. Some things, especially for debugging purposes are better expressed as events. So write logs.
There was a problem hiding this comment.
or consolidate them into one as mentioned above, but without the label "lease_id". In this case, it should be Gauge for sure.
This seems like a good idea. Thoughts? @serathius
Some things, especially for debugging purposes are better expressed as events. So write logs
Logs are great but sometimes it's difficult to miss a trend, when there's log. Makes it easier to setup a trigger based on logs, than the trend of metrics, if that makes sense.
| @@ -49,11 +49,88 @@ var ( | |||
| // 1 second -> 3 months | |||
| Buckets: prometheus.ExponentialBuckets(1, 2, 24), | |||
| }) | |||
|
|
|||
| leaseAttached = prometheus.NewCounter(prometheus.CounterOpts{ | |||
There was a problem hiding this comment.
What's the purpose of metrics about lease attachements? What you are trying to measure? number of keys per lease?
There was a problem hiding this comment.
number of keys per lease?
Precisely, yes.
- metrics to capture leases attached and detached - metric to capture initial lease count at startup Signed-off-by: vivekpatani <[email protected]>
Help
LeaseLeasesresponse.