feat: Metrics API enhanced for Failure counters #377

indrajit96 · 2024-07-08T17:09:13Z

What does the PR do?

Currently We don't have any mechanism to count any failures that happen in triton core before the request gets to the backend.

This PR enhances existing metrics API to incorporate failure and failure counts in inference requests, along with categories for the types of failures as a "reason" label with initial support for: BACKEND, CANCELED, REJECTED, and OTHER.

NOTE: Ensemble failure metrics will reflect the failure counts of their composing models,
but currently do not capture the same granularity for the "reason" label and will default to the "OTHER" reason.

Checklist

PR title reflects the change and is of format <commit_type>: <Title>
Changes are described in the pull request.
Related issues are referenced.
Populated github labels field
Added test plan and verified test passes.
Verified that the PR passes existing CI.
Verified copyright is correct on all changed files.
Added succinct git squash message before merging ref.
All template sections are filled out.

Commit Type:

Check the conventional commit type
box here and add the label to the github PR.

Related PRs:

triton-inference-server/server#7423

Where should the reviewer start?

RespondIfError function call in core

Test plan:

See test plan in triton-inference-server/server#7423

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

closes Jira issue: DLIS-5530

…bhosale-metrics

src/backend_model_instance.cc

src/metric_model_reporter.cc

src/sequence_batch_scheduler/sequence_batch_scheduler.cc

src/dynamic_batch_scheduler.cc

src/infer_request.cc

src/infer_request.h

src/infer_stats.cc

src/ensemble_scheduler/ensemble_scheduler.cc

src/infer_request.cc

src/backend_model_instance.cc

src/infer_stats.h

rmccorm4 · 2024-07-11T04:54:08Z

src/backend_model_instance.cc

@@ -558,7 +559,8 @@ TritonModelInstance::PrepareRequestsOrRespond(
  // If any errors occurred, respond with error for each request.
  if (!status.IsOk()) {
    for (auto& r : requests) {
-      InferenceRequest::RespondIfError(r, status, true /* release_requests */);
+      InferenceRequest::RespondIfError(
+          r, status, true /* release_requests */, FailureReason::OTHER);


Can you think of any test cases to test some of the reason=OTHER scenarios (not counting ensembles)? Can be a follow-up, don't need to be in these 24.07 PRs, but would be good to have a test case for these sooner than later if we have any in mind.

src/ensemble_scheduler/ensemble_scheduler.cc

src/infer_request.cc

src/infer_request.h

rmccorm4

High level code LGTM since we're aiming for 24.07 cherry-pick, left some small comments and follow-ups.

Enhances existing metrics API to incorporate failure and failure counts in inference requests --------- Co-authored-by: Kevin Wang <[email protected]> Co-authored-by: krishung5 <[email protected]>

Kevin Wang and others added 7 commits June 11, 2024 12:42

Add metrics support for rejected requests

cd7a82e

Additional changes for metrics support for rejected requests

b214ca5

Feedback changes on metrics support for rejected requests

f7bf12a

Testing

37471fc

Merge branch 'main' of github.com:triton-inference-server/core into i…

67738d9

…bhosale-metrics

Add logs

98d5c5a

PR Cleanup Crash Fixed

78d283f

indrajit96 requested review from Tabrizian, rmccorm4 and krishung5 July 8, 2024 17:09

indrajit96 mentioned this pull request Jul 8, 2024

test: Tests for Metrics API enhancement to include error counters triton-inference-server/server#7423

Merged

19 tasks

indrajit96 added 2 commits July 8, 2024 10:32

Merge remote-tracking branch 'origin/main' into ibhosale-metrics

7e7bbc5

Remove Logging code

688a327

rmccorm4 changed the title ~~feat:Metrics API enhanced for Failure counters~~ feat: Metrics API enhanced for Failure counters Jul 8, 2024