Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Documentation for single operator E2E tests #558

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Documentation for single operator E2E tests #558

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f0f64e6
Single operator E2E tests documentation
vbrkicTT Oct 29, 2024
302cc32
Filter test plan by INPUT_SHAPES and KWARGS
vbrkicTT Nov 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 69 additions & 0 deletions docs/src/test.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,3 +45,72 @@ pytest -svv forge/test/mlir/test_ops.py::test_add
```

> - The `-svv` flag is optional and used to display more information about the test run.

## Single operator E2E tests

Single operator E2E tests consists of pre configured collections of in-depth tests for each operator according to test plan.
Tests include small models consisting of single operator with or without combination with few other operators.
More details about test plan available on [Test template page](https://github.com/tenstorrent/tt-forge-fe/blob/main/forge/test/operators/test_plan_template.md)

To run all tests from test plan use `test_plan` method:

```sh
pytest -svv forge/test/operators/pytorch/test_all.py::test_plan
```

Full list of supported query criteria

| Parameter | Description |
| ------------------ | ------------------------------------------------------------------------------------------ |
| OPERATORS | List of operators |

To run single test based on a test id use `test_single` method with `TEST_ID` parameter:

```sh
TEST_ID='ge-FROM_HOST-None-(1, 2, 3, 4)-Float16_b-HiFi4' pytest -svv forge/test/operators/pytorch/test_all.py::test_single
```

| Parameter | Description |
| ------------------ | ------------------------------------------------------------------------------------------ |
| TEST_ID | Id of a test containing test parameters |

To see examples of all available tests use `test_unique` method:

```sh
pytest -svv forge/test/operators/pytorch/test_all.py::test_unique --collect-only
```

Full list of supported query criteria

| Parameter | Description |
| ------------------ | ------------------------------------------------------------------------------------------ |
| OPERATORS | List of operators |

To query subset of test plan based on a query criteria use `test_query` method:

```sh
OPERATORS=add,div FILTERS=HAS_DATA_FORMAT,QUICK DEV_DATA_FORMATS=Float16_b,Int8 MATH_FIDELITIES=HiFi4,HiFi3 RANGE=5 pytest -svv forge/test/operators/pytorch/test_all.py::test_query --collect-only
```

```sh
OPERATORS=add,div FAILING_REASONS=DATA_MISMATCH,UNSUPPORTED_DATA_FORMAT SKIP_REASONS=FATAL_ERROR RANGE=5 pytest -svv forge/test/operators/pytorch/test_all.py::test_query --collect-only
```

```sh
FAILING_REASONS=NOT_IMPLEMENTED INPUT_SOURCES=FROM_HOST pytest -svv forge/test/operators/pytorch/test_all.py::test_query --collect-only
```

Full list of supported query criteria

| Parameter | Description |
| --------------------- | --------------------------------------------------------------------------------------------- |
| OPERATORS | List of operators |
| FILTERS | List of labmda filters. See `test/operators/pytorch/test_all.py` for all values |
| INPUT_SOURCES | Supported values: FROM_ANOTHER_OP, FROM_HOST, FROM_DRAM_QUEUE, CONST_EVAL_PASS |
| INPUT_SHAPES | List of input shapes to filter |
| DEV_DATA_FORMATS | List of dev data formats. Full list available in `test/operators/utils/test_data.py` |
| MATH_FIDELITIES | List of math fidelities. Full list available in `test/operators/utils/test_data.py` |
| KWARGS | List of kwargs dictionaries to filter |
| FAILING_REASONS | List of failing reasons. Full list available in `test/operators/utils/failing_reasons.py` |
| SKIP_REASONS | List of skip reasons. Full list available in `test/operators/utils/failing_reasons.py` |
| RANGE | Limit number of results |
21 changes: 13 additions & 8 deletions forge/test/operators/pytorch/test_all.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,6 @@


# Examples
# pytest -svv forge/test/operators/pytorch/test_all.py::test_unique --collect-only
# TEST_ID='no_device-ge-FROM_HOST-None-(1, 2, 3, 4)-Float16_b-HiFi4' pytest -svv forge/test/operators/pytorch/test_all.py::test_single
# OPERATORS=add,div FILTERS=HAS_DATA_FORMAT,QUICK DEV_DATA_FORMATS=Float16_b,Int8 MATH_FIDELITIES=HiFi4,HiFi3 RANGE=5 pytest -svv forge/test/operators/pytorch/test_all.py::test_query --collect-only
# OPERATORS=add,div FAILING_REASONS=DATA_MISMATCH,UNSUPPORTED_DATA_FORMAT SKIP_REASONS=FATAL_ERROR RANGE=5 pytest -svv forge/test/operators/pytorch/test_all.py::test_query --collect-only
# FAILING_REASONS=NOT_IMPLEMENTED INPUT_SOURCES=FROM_HOST pytest -svv forge/test/operators/pytorch/test_all.py::test_query --collect-only

# pytest -svv forge/test/operators/pytorch/test_all.py::test_plan
# pytest -svv forge/test/operators/pytorch/test_all.py::test_failed
# pytest -svv forge/test/operators/pytorch/test_all.py::test_skipped
Expand All @@ -20,6 +14,9 @@
# pytest -svv forge/test/operators/pytorch/test_all.py::test_data_mismatch
# pytest -svv forge/test/operators/pytorch/test_all.py::test_unsupported_df
# pytest -svv forge/test/operators/pytorch/test_all.py::test_custom
# pytest -svv forge/test/operators/pytorch/test_all.py::test_query
# pytest -svv forge/test/operators/pytorch/test_all.py::test_unique
# pytest -svv forge/test/operators/pytorch/test_all.py::test_single


import os
Expand Down Expand Up @@ -74,8 +71,10 @@ def build_filtered_collection(cls) -> TestCollection:
Query criterias are defined by the following environment variables:
- OPERATORS: List of operators to filter
- INPUT_SOURCES: List of input sources to filter
- INPUT_SHAPES: List of input shapes to filter
- DEV_DATA_FORMATS: List of data formats to filter
- MATH_FIDELITIES: List of math fidelities to filter
- KWARGS: List of kwargs dictionaries to filter
"""
operators = os.getenv("OPERATORS", None)
if operators:
Expand All @@ -88,7 +87,9 @@ def build_filtered_collection(cls) -> TestCollection:
input_sources = input_sources.split(",")
input_sources = [getattr(InputSource, input_source) for input_source in input_sources]

# TODO INPUT_SHAPES
input_shapes = os.getenv("INPUT_SHAPES", None)
if input_shapes:
input_shapes = eval(input_shapes)

dev_data_formats = os.getenv("DEV_DATA_FORMATS", None)
if dev_data_formats:
Expand All @@ -100,13 +101,17 @@ def build_filtered_collection(cls) -> TestCollection:
math_fidelities = math_fidelities.split(",")
math_fidelities = [getattr(forge.MathFidelity, math_fidelity) for math_fidelity in math_fidelities]

# TODO KWARGS
kwargs = os.getenv("KWARGS", None)
if kwargs:
kwargs = eval(kwargs)

filtered_collection = TestCollection(
operators=operators,
input_sources=input_sources,
input_shapes=input_shapes,
dev_data_formats=dev_data_formats,
math_fidelities=math_fidelities,
kwargs=kwargs,
)

return filtered_collection
Expand Down
Loading