-
Notifications
You must be signed in to change notification settings - Fork 101
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Metrics API enhanced for Failure counters #377
Conversation
@@ -558,7 +559,8 @@ TritonModelInstance::PrepareRequestsOrRespond( | |||
// If any errors occurred, respond with error for each request. | |||
if (!status.IsOk()) { | |||
for (auto& r : requests) { | |||
InferenceRequest::RespondIfError(r, status, true /* release_requests */); | |||
InferenceRequest::RespondIfError( | |||
r, status, true /* release_requests */, FailureReason::OTHER); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you think of any test cases to test some of the reason=OTHER
scenarios (not counting ensembles)? Can be a follow-up, don't need to be in these 24.07 PRs, but would be good to have a test case for these sooner than later if we have any in mind.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
High level code LGTM since we're aiming for 24.07 cherry-pick, left some small comments and follow-ups.
Enhances existing metrics API to incorporate failure and failure counts in inference requests --------- Co-authored-by: Kevin Wang <[email protected]> Co-authored-by: krishung5 <[email protected]>
Enhances existing metrics API to incorporate failure and failure counts in inference requests --------- Co-authored-by: Kevin Wang <[email protected]> Co-authored-by: krishung5 <[email protected]>
What does the PR do?
Currently We don't have any mechanism to count any failures that happen in triton core before the request gets to the backend.
This PR enhances existing metrics API to incorporate failure and failure counts in inference requests, along with categories for the types of failures as a "reason" label with initial support for: BACKEND, CANCELED, REJECTED, and OTHER.
NOTE: Ensemble failure metrics will reflect the failure counts of their composing models,
but currently do not capture the same granularity for the "reason" label and will default to the "OTHER" reason.
Checklist
<commit_type>: <Title>
Commit Type:
Check the conventional commit type
box here and add the label to the github PR.
Related PRs:
triton-inference-server/server#7423
Where should the reviewer start?
RespondIfError function call in core
Test plan:
See test plan in triton-inference-server/server#7423
Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)