feat: Add histogram support and TTFT histogram metric

[yinggeh]

What does the PR do?

1. Adds histogram support to core metrics.
2. Adding a new triton version TTFT (time-to-first-token) metric nv_inference_first_response_histogram_ms. The new histogram metric is specific to decoupled models only.
3. Add --metrics-config histogram_latencies=<bool> to disable histograms.

Example a request sent to TRT-LLM backend ensemble model.

# HELP nv_inference_first_response_histogram_ms Duration from request to first response in milliseconds
# TYPE nv_inference_first_response_histogram_ms histogram
nv_inference_first_response_histogram_ms_count{model="ensemble",version="1"} 1
nv_inference_first_response_histogram_ms_sum{model="ensemble",version="1"} 1159
nv_inference_first_response_histogram_ms_bucket{model="ensemble",version="1",le="100"} 0
nv_inference_first_response_histogram_ms_bucket{model="ensemble",version="1",le="500"} 0
nv_inference_first_response_histogram_ms_bucket{model="ensemble",version="1",le="2000"} 1
nv_inference_first_response_histogram_ms_bucket{model="ensemble",version="1",le="5000"} 1
nv_inference_first_response_histogram_ms_bucket{model="ensemble",version="1",le="+Inf"} 1
nv_inference_first_response_histogram_ms_count{model="tensorrt_llm",version="1"} 1
nv_inference_first_response_histogram_ms_sum{model="tensorrt_llm",version="1"} 1137
nv_inference_first_response_histogram_ms_bucket{model="tensorrt_llm",version="1",le="100"} 0
nv_inference_first_response_histogram_ms_bucket{model="tensorrt_llm",version="1",le="500"} 0
nv_inference_first_response_histogram_ms_bucket{model="tensorrt_llm",version="1",le="2000"} 1
nv_inference_first_response_histogram_ms_bucket{model="tensorrt_llm",version="1",le="5000"} 1
nv_inference_first_response_histogram_ms_bucket{model="tensorrt_llm",version="1",le="+Inf"} 1
nv_inference_first_response_histogram_ms_count{model="tensorrt_llm_bls",version="1"} 0
nv_inference_first_response_histogram_ms_sum{model="tensorrt_llm_bls",version="1"} 0
nv_inference_first_response_histogram_ms_bucket{model="tensorrt_llm_bls",version="1",le="100"} 0
nv_inference_first_response_histogram_ms_bucket{model="tensorrt_llm_bls",version="1",le="500"} 0
nv_inference_first_response_histogram_ms_bucket{model="tensorrt_llm_bls",version="1",le="2000"} 0
nv_inference_first_response_histogram_ms_bucket{model="tensorrt_llm_bls",version="1",le="5000"} 0
nv_inference_first_response_histogram_ms_bucket{model="tensorrt_llm_bls",version="1",le="+Inf"} 0

Example a request sent to TRT-LLM backend tensorrt_llm_bls model.

# HELP nv_inference_first_response_histogram_ms Duration from request to first response in milliseconds
# TYPE nv_inference_first_response_histogram_ms histogram
nv_inference_first_response_histogram_ms_count{model="ensemble",version="1"} 0
nv_inference_first_response_histogram_ms_sum{model="ensemble",version="1"} 0
nv_inference_first_response_histogram_ms_bucket{model="ensemble",version="1",le="100"} 0
nv_inference_first_response_histogram_ms_bucket{model="ensemble",version="1",le="500"} 0
nv_inference_first_response_histogram_ms_bucket{model="ensemble",version="1",le="2000"} 0
nv_inference_first_response_histogram_ms_bucket{model="ensemble",version="1",le="5000"} 0
nv_inference_first_response_histogram_ms_bucket{model="ensemble",version="1",le="+Inf"} 0
nv_inference_first_response_histogram_ms_count{model="tensorrt_llm_bls",version="1"} 1
nv_inference_first_response_histogram_ms_sum{model="tensorrt_llm_bls",version="1"} 1125
nv_inference_first_response_histogram_ms_bucket{model="tensorrt_llm_bls",version="1",le="100"} 0
nv_inference_first_response_histogram_ms_bucket{model="tensorrt_llm_bls",version="1",le="500"} 0
nv_inference_first_response_histogram_ms_bucket{model="tensorrt_llm_bls",version="1",le="2000"} 1
nv_inference_first_response_histogram_ms_bucket{model="tensorrt_llm_bls",version="1",le="5000"} 1
nv_inference_first_response_histogram_ms_bucket{model="tensorrt_llm_bls",version="1",le="+Inf"} 1
nv_inference_first_response_histogram_ms_count{model="tensorrt_llm",version="1"} 1
nv_inference_first_response_histogram_ms_sum{model="tensorrt_llm",version="1"} 1113
nv_inference_first_response_histogram_ms_bucket{model="tensorrt_llm",version="1",le="100"} 0
nv_inference_first_response_histogram_ms_bucket{model="tensorrt_llm",version="1",le="500"} 0
nv_inference_first_response_histogram_ms_bucket{model="tensorrt_llm",version="1",le="2000"} 1
nv_inference_first_response_histogram_ms_bucket{model="tensorrt_llm",version="1",le="5000"} 1
nv_inference_first_response_histogram_ms_bucket{model="tensorrt_llm",version="1",le="+Inf"} 1

Checklist

• PR title reflects the change and is of format <commit_type>: <Title>
• Changes are described in the pull request.
• Related issues are referenced.
• Populated github labels field
• Added test plan and verified test passes.
• Verified that the PR passes existing CI.
• Verified copyright is correct on all changed files.
• Added succinct git squash message before merging ref.
• All template sections are filled out.
• Optional: Additional screenshots for behavior/output changes with before/after.

Commit Type:

• feat

Related PRs:

triton-inference-server/server#7694

Where should the reviewer start?

n/a

Test plan:

L0_metrics--base
L0_response_cache--base

• CI Pipeline ID:
  19614087

Caveats:

1. Neelay suggested adding backend_name to metric label. During the implementation, I found it wasn't easy to only add the backend label to new metric without changing all existing ones. @nnshah1
2. Customer asked for TTFT in millisecond. However, it is not consistent with microsecond we use in other metrics. Please advise.

Background

Standardizing Large Model Server Metrics in Kubernetes

[kthui]

LGTM! @rmccorm4 should also take a look before merging.

[rmccorm4]

For an initial merge with a default of histograms being disabled - I think this is fine to go ahead with if we need to cherry-pick. However, please take note of the following:

I think this is relatively on the hot path, possibly impacting latency, compared to our other inference metrics (TRITONBACKEND_ReportStatistics) which are generally reported after response sending in backends (impacting throughput but not response latency).

You can find some perf numbers of each prometheus-cpp metric type at the bottom of the README here: https://github.com/jupp0r/prometheus-cpp

One individual observation for a single metric and a small number of buckets may not be impactful for one request, but as we scale up high concurrency, more metrics, more buckets, etc - this could present a noticeable latency impact.

It would be good to do some light validation of overall latency before/after the feature via genai-perf. Especially for high concurrency and streaming many responses/tokens - as there can be some synchronization in interaction with the prometheus registry with many concurrent responses as well.

It would probably be advantageous to do the actual prometheus registry interaction after sending the response if possible, such as by only doing the bare minimum of determining if we should report metrics (check if first response and record latency), then using that information to report the metric after initiating response send.

[rmccorm4]

Deferring to Jacky as I'll be OOO, but let's make sure the test cases give us confidence in the feature and that we're addressing the spirit of the use case that motivates the feature

[kthui]

LGTM! The test cases are giving us confidence in the feature and the feature is addressing the use case.

Please make sure the CI passes with the all final changes before merging.