You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is there is any endpoint within the API server where we are able to pull the metics like Running Requests
Waiting Requests, Swapped Requests, GPU Cache Usage, CPU Cache Usage, Latency, Prompt Tokens, Generation Tokens?
This is to pull the metrics for the LMDeploy hosted inference engine in to prometheus.
Related resources
No response
Additional context
No response
The text was updated successfully, but these errors were encountered:
Motivation
Is there is any endpoint within the API server where we are able to pull the metics like Running Requests
Waiting Requests, Swapped Requests, GPU Cache Usage, CPU Cache Usage, Latency, Prompt Tokens, Generation Tokens?
This is to pull the metrics for the LMDeploy hosted inference engine in to prometheus.
Related resources
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: