Query on 'request durations'

[mr-onion-2]

Hi there

Great tool, thank you :)

Do you have any explanation of what some of the metrics mean please? E.g. blocky_request_duration_ms_bucket.
Googling blocky_request_duration_ms_bucket returns only one result (the code in this repo)

I noticed a large proportion of upstream requests with a duration between 2 seconds and +Inf. I am curious whether this is indicating that something can be improved or whether it is normal.

I have tailed the application logs and the majority of requests have duration_ms less than 50. I don't see any with anything close to 2 seconds

Many thanks

[0xERR0R]

blocky_request_duration_ms_bucket is a historgram with request duration distribution. Blocky measures the request time from start (request was received) to the end (request was resolved and is about to be sent to the client). Prometheus collects request durations into predefined buckets (5-10 ms, 10 - 20 ms ... 2s - inf).

Predefined Grafana dashboard shows only requsts with "RESOLVED" type - without cached and internally resolved (aka custom dns, conditional etc). So basically you have only request duration of external resolved requests.

Grafana accumulates the values: for example you have a requst with > 2 s duration at 13:00 and one at 14:00, Grafana will show 0 in this bucket til 13:00, 1 from 13:00 - 14:00 and 2 after 14:00.

Here is my current stat:

I hope, you can see, that I have 1 requst with > 2s at ca 13:50. And here is my query log sorted by duration (opened in OpenOffice as CSV):

I had a request at 13:48 with 2033 ms duration. Such "long" time for a particular requests could be the Garbage collection of golang runtime or for example a parallel list refresh in the background.

From my point of view, it works as expected, maybe you can provide more data to analyze.

Did you check your query log (CSV file) or standard out?

[mr-onion-2]

Thanks for the quick and informative response 👍

I wasn't suggesting there was an issue with Blocky by the way. It was more whether this value was something to worry about or could potentially be reduced with some tinkering. By the definition and comparing to yours, it looks like something on my network isn't quite optimal.

Ah I'd overlooked the query log option, that makes it easier! I was just scanning through the standard out so I probably just missed the slow ones. I've enabled it now so will see.

If the slow ones are anything of interest I'll come back with an update.

Thanks again