-
-
Notifications
You must be signed in to change notification settings - Fork 751
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Run st2-self-check in Github Actions #5609
Conversation
… with an return code != 0 if any of the tests fails
The idea with the https://hub.docker.com/r/stackstorm/packagingtest/tags containers is that it comes with an init system and ssh and so you basically use it the same as a VM for testing. You can try to use that Docker Image as an environment for installing/running st2. |
I found a solution for the ssh issue and these actions work now. Now I'll see if I find any hint why the two HTTP actions fail. |
…m/st2 into winem-self-check-in-gha-actions
…the GHA blacklist of tests in the self-check
f87d6d2
to
16c71b0
Compare
The execution of the self-checks works as expected now. I'll add the CHANGELOG-text soon and will also have a look at the failing integration test. |
As just agreed in the TSC I'll create issues to run the self-check on other Py versions as well. For now it's Python3.6 that's interesting but Python 3.9 or even 3.10 will become interesting "soon". There'll be another issue to run the self-check against different OS. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I like the idea of a generic skip variable, overall this looks good to me.
So I switched to a generic TESTS_TO_SKIP variable that can be used by anyone and added a note to the changelog. So, from my side, this PR is ready to be reviewed again. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It works 🎉 Thanks for all the efforts!
A couple of minor comments before merging
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great work @winem!
Thanks!
This PR enables us to run the st2-self-check as part of our CI run in Github Actions as well.
The most crucial part was to handle the environment correctly and to determine the context when running the self-check. A few modifications are needed to adapt to the GHA environment.
Another big change is that the st2-self-check exits with a return code != 0 if any of the tests fail. This is required to ensure that the CI run fails.
I do not see a use-case where this could break anything on the users end but I'll add a note to the changelog to highlight this change as well. I don't think it's a breaking change - but let me know if you disagree.
All tests but 4 (2 remote actions that try to connect to localhost via ssh and 2 HTTP actions) succeed.I'm open for any input on how to handle the remaining 4 tests.@armab suggested to have a look at https://github.com/StackStorm/st2packaging-dockerfiles/tree/master/packagingtest and see if we can re-use this here. I'll evaluate it and am open for further ideas.It's the same for the HTTP actions.I'll mark this PR as draft until we have a solution for the 4 failing tests but feel free to review the changes I already made and give your feedback!There may always be tests that may not succeed in GitHub Actions due to special infrastructure, environment or further external dependencies to be satisfied to make them work. So I added some kind of "skip list" or "black list" of tests. These tests will not be executed by GitHub Actions. This list is a space separated list of string configured as environment variable for the
self-check
job. So you don't have to handle this kind of exceptions in the test files itself and just add them in a single place.As agreed in a previous TSC meeting the latest Python3.8 release is used.
closes #5496