[UnitTests] Automatic parametrization over targets, with explicit opt-out #8010
Commits on Jun 23, 2021
-
[UnitTests] Explicitly list tests that were enabled by TVM_TEST_TARGE…
…TS but were skipped Previously, these were removed by a filter in tvm.testing._get_targets(), and weren't listed at all. With this change, they are instead removed by pytest.skipif, and show up as explicitly skipped tests in pytest's summary when using tvm.testing.parametrize_targets.
-
[UnitTests] Automatic parametrize_targets for tests that use (target,…
…dev) Should make it easier to convert tests from using tvm.testing.enabled_targets to use pytest's parametrized tests instead.
-
[UnitTests] Added ability to explicitly exclude a target from a parti…
…cular test Uses tvm_exclude_targets variable, which can be set (1) in the conftest.py to apply to a test directory, (2) in a test script to apply to that module, or (3) on an individual test function to apply to it. The @tvm.testing.exclude_targets decorator is provided for readability in case apache#3.
-
[UnitTests] Added tvm_known_failing_targets option for the unittests.
Intended to mark tests that fail for a particular target, and are intended to be fixed in the future. Typically, these would result either from implementing a new test, or from an in-progress implementation of a new target.
-
[UnitTests] Removed tvm_excluded_targets and tvm_known_failing_targets
These were implemented to exclude or mark as failing an entire file or directory of tests. In https://discuss.tvm.apache.org/t/rfc-parametrized-unit-tests/9946/4, it was pointed out that the global variables would be vulnerable to typos in the names, resulting in the option being silently ignored. The decorators `@tvm.testing.exclude_targets` and `@tvm.testing.known_failing_targets` do not have this failure mode, and are the preferred version.
-
[UnitTests] Added helper functions to tvm.testing.
- tvm.testing.parameter() defines a parameter that can be passed to tests. Tests that accept more than one parameter are run for all combinations of parameter values. - tvm.testing.parameters() defines multiple sets of parameter values. Tests that accept more than one parameter are run once for each set of parameter values. - tvm.testing.fixture() is a decorator that defines setup code. The `cache=True` argument can be passed to avoid repeating expensive setup across multiple tests.
-
[UnitTests] Bugfix for auto parametrizing of "target"
Previously, if the @parametrize_targets were present, but had other @pytest.mark.parametrize after it, "target" would get parametrized a second time. Now, it checks more than just the closest "parametrize" marker.
-
[UnitTests] Minor updates to parametrized test implementation.
As recommended by @tkonolige: - Avoid infinite loop if LLVM target isn't enabled - Update documentation for preferred use cases of tvm.testing.parametrize_targets, and recommended alternatives.
-
[UnitTests] Minor updates to parametrized test implementation
- Documentation, removed previous example usage of tvm.testing.parametrize_targets
-
[UnitTests] Changed accidental use of pytest fixtures to a NameError.
- Previously, a fixture function defined in a module was accessible through the global scope, and the function definition is accessible if a test function uses that name but fails to declare the fixture as a parameter. Now, it will result in a NameError instead.
-
[UnitTests] More careful removal of fixture functions from module glo…
…bal scope. - Initial implementation only checked hasattr(obj, "_pytestfixturefunction") before removing obj, which gave false positives for objects that implement __getattr__, such as caffe.layers. Now, also check that the value contained is a FixtureFunctionMarker.
-
[UnitTests] Copy cached values when using tvm.testing.fixture(cache_r…
…eturn_value=True) To avoid unit tests being able to influence each other through a shared cache, all cached fixtures are passed through copy.deepcopy prior to use.
-
