Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug][CI] integration tests not currently running in ci_gpu image #12529

Open
leandron opened this issue Aug 22, 2022 · 3 comments
Open

[Bug][CI] integration tests not currently running in ci_gpu image #12529

leandron opened this issue Aug 22, 2022 · 3 comments
Assignees
Labels
dev:ci dev:test-infra type: bug type:ci Relates to TVM CI infrastructure

Comments

@leandron
Copy link
Contributor

leandron commented Aug 22, 2022

When looking at recent CI test run (which I'm enabling torch/ONNX tests on AArch64), I noticed an issue with a missing transformers package while executing test_meta_schedule_integration_extract_from_bert_base , causing the integration tests to fail on AArch64.

Investigating the issue a bit deeper, it seems that integration tests are not running on ci_GPU image, which is the one that is supposed to contain torch and therefore would reproduce the problem.

I'm raising this issue just because I won't have the time in the next weeks to enable that package and test the GPU image and test it to solve the problem. So I create this PR #12528 to make the test to be properly skipped if transformers is not installed, but leaving this issue so that we can investigate two potential problems:

  1. Integration tests not running in the GPU environment
  2. Missing transformers installation, if that is what we want (pending to be confirmed, I didn't fully investigate)

cc @Mousius @areusch @driazati @gigiblender

@leandron leandron added type: bug type:ci Relates to TVM CI infrastructure labels Aug 22, 2022
@leandron leandron changed the title [Bug] Missing integration tests in ci_gpu image [Bug][CI] integration tests not currently running in ci_gpu image Aug 22, 2022
@leandron
Copy link
Contributor Author

leandron commented Aug 22, 2022

ONNX testing that runs as part of Integration tests is also not running anywhere. One is actually broken, #12532

@areusch
Copy link
Contributor

areusch commented Aug 25, 2022

@gigiblender is it possible to leverage #12436 to add another GH check to assert we aren't skipping a couple of critical integration tests in a PR?

@gigiblender
Copy link
Contributor

gigiblender commented Aug 26, 2022

@gigiblender is it possible to leverage #12436 to add another GH check to assert we aren't skipping a couple of critical integration tests in a PR?

Sure thing. We could change that script to pass a list of critical tests to ensure that some tests are not skipped and call it in a GH check. Let me know if that sounds right.

@areusch areusch added the needs-triage PRs or issues that need to be investigated by maintainers to find the right assignees to address it label Oct 19, 2022
@janetsc janetsc added dev:ci dev:test-infra and removed needs-triage PRs or issues that need to be investigated by maintainers to find the right assignees to address it labels Oct 19, 2022
driazati pushed a commit that referenced this issue Nov 13, 2022
In this PR, the skipped tests script will also check if tests in the `required_tests_to_run.json` have not been skipped. If there are skipped tests, they will be added to the returned comment. 

I am not entirely sure where it's best to place the `required_tests_to_run` file, so I left it in `tvm/ci/scripts/`. I am happy to take suggestions.

Aims to prevent situations such as #12529
xinetzone pushed a commit to daobook/tvm that referenced this issue Nov 25, 2022
In this PR, the skipped tests script will also check if tests in the `required_tests_to_run.json` have not been skipped. If there are skipped tests, they will be added to the returned comment. 

I am not entirely sure where it's best to place the `required_tests_to_run` file, so I left it in `tvm/ci/scripts/`. I am happy to take suggestions.

Aims to prevent situations such as apache#12529
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dev:ci dev:test-infra type: bug type:ci Relates to TVM CI infrastructure
Projects
None yet
Development

No branches or pull requests

4 participants