Rewrite tests #346

RAMitchell · 2023-03-22T15:46:26Z

This PR moves the model generation and loading into pytest.

This achieves a few things:

Generation of configurations and models on the fly by pytest
Testing prediction results directly against original libraries instead of gtil
The ability to test shapley values
Less code
Fixes [BUG] Predict proba for cuml classifier with CPU backend broken #348

The tests run faster than before, are easy to extend and we can generate more coverage.

Todo:

cuml tests
sklearn tests
update docs

… into test

qa/L0_e2e/generate_example_model.py

qa/run_tests.sh

src/tl_model.h

qa/L0_e2e/test_model.py

… into test

RAMitchell · 2023-04-03T14:40:18Z

I have updated this so that models are generated using the original code but cached automatically using joblib.memory. Here below is an example of the test matrix (68 items), taking 130s. I think it will be particularly useful to test the experimental_optimization flag for all as the backend is updated with the latest cuml changes.

../../qa/L0_e2e/test_model.py::TestClassifiers::test_xgb[use_json:True-num_class:2-instance:KIND_CPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_xgb[use_json:True-num_class:2-instance:KIND_CPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_xgb[use_json:True-num_class:2-instance:KIND_GPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_xgb[use_json:True-num_class:2-instance:KIND_GPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_xgb[use_json:True-num_class:10-instance:KIND_CPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_xgb[use_json:True-num_class:10-instance:KIND_CPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_xgb[use_json:True-num_class:10-instance:KIND_GPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_xgb[use_json:True-num_class:10-instance:KIND_GPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_xgb[use_json:False-num_class:2-instance:KIND_CPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_xgb[use_json:False-num_class:2-instance:KIND_CPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_xgb[use_json:False-num_class:2-instance:KIND_GPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_xgb[use_json:False-num_class:2-instance:KIND_GPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_xgb[use_json:False-num_class:10-instance:KIND_CPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_xgb[use_json:False-num_class:10-instance:KIND_CPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_xgb[use_json:False-num_class:10-instance:KIND_GPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_xgb[use_json:False-num_class:10-instance:KIND_GPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_lgbm_classification_model[num_class:2-instance:KIND_CPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_lgbm_classification_model[num_class:2-instance:KIND_CPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_lgbm_classification_model[num_class:2-instance:KIND_GPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_lgbm_classification_model[num_class:2-instance:KIND_GPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_lgbm_classification_model[num_class:10-instance:KIND_CPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_lgbm_classification_model[num_class:10-instance:KIND_CPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_lgbm_classification_model[num_class:10-instance:KIND_GPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_lgbm_classification_model[num_class:10-instance:KIND_GPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_sklearn[num_class:2-instance:KIND_CPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_sklearn[num_class:2-instance:KIND_CPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_sklearn[num_class:2-instance:KIND_GPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_sklearn[num_class:2-instance:KIND_GPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_sklearn[num_class:10-instance:KIND_CPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_sklearn[num_class:10-instance:KIND_CPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_sklearn[num_class:10-instance:KIND_GPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_sklearn[num_class:10-instance:KIND_GPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_cuml[num_class:2-instance:KIND_CPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_cuml[num_class:2-instance:KIND_CPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_cuml[num_class:2-instance:KIND_GPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_cuml[num_class:2-instance:KIND_GPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_cuml[num_class:10-instance:KIND_CPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_cuml[num_class:10-instance:KIND_CPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_cuml[num_class:10-instance:KIND_GPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_cuml[num_class:10-instance:KIND_GPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_sklearn_gbm[num_class:2-instance:KIND_CPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_sklearn_gbm[num_class:2-instance:KIND_CPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_sklearn_gbm[num_class:2-instance:KIND_GPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_sklearn_gbm[num_class:2-instance:KIND_GPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_sklearn_gbm[num_class:10-instance:KIND_CPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_sklearn_gbm[num_class:10-instance:KIND_CPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_sklearn_gbm[num_class:10-instance:KIND_GPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestClassifiers::test_sklearn_gbm[num_class:10-instance:KIND_GPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_xgb[instance:KIND_CPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_xgb[instance:KIND_CPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_xgb[instance:KIND_GPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_xgb[instance:KIND_GPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_lgbm[instance:KIND_CPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_lgbm[instance:KIND_CPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_lgbm[instance:KIND_GPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_lgbm[instance:KIND_GPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_sklearn[instance:KIND_CPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_sklearn[instance:KIND_CPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_sklearn[instance:KIND_GPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_sklearn[instance:KIND_GPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_cuml[instance:KIND_CPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_cuml[instance:KIND_CPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_cuml[instance:KIND_GPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_cuml[instance:KIND_GPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_sklearn_gbm[instance:KIND_CPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_sklearn_gbm[instance:KIND_CPU-exper_optim:False] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_sklearn_gbm[instance:KIND_GPU-exper_optim:True] PASSED
../../qa/L0_e2e/test_model.py::TestRegressors::test_sklearn_gbm[instance:KIND_GPU-exper_optim:False] PASSED
```

RAMitchell added 11 commits March 17, 2023 07:59

Test via pytest

9bc160d

Test experimental optimizations flag.

368b0ce

Merge branch 'main' of github.com:triton-inference-server/fil_backend…

e537afe

… into test

Allow different GPUs for testing.

dbc28f8

Test xgb/lgbm

488a9fe

Test lightgbm regression.

1e398f3

Refactor test model.

a3944f2

cuml regression tests.

4c60a8b

Fix triton-inference-server#348

c2a7a87

Update docs

c88d3b8

Add sklearn GBM

8787ad3

wphicks requested changes Mar 24, 2023

View reviewed changes

RAMitchell added 3 commits March 29, 2023 11:38

threshold for cuml rf binary classifiers.

29d8047

Cache with joblib.memory

273764c

Merge branch 'main' of github.com:triton-inference-server/fil_backend…

b4e843f

… into test

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rewrite tests #346

Rewrite tests #346

RAMitchell commented Mar 22, 2023 •

edited

Loading

RAMitchell commented Apr 3, 2023

Rewrite tests #346

Are you sure you want to change the base?

Rewrite tests #346

Conversation

RAMitchell commented Mar 22, 2023 • edited Loading

RAMitchell commented Apr 3, 2023

RAMitchell commented Mar 22, 2023 •

edited

Loading