Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PR: Reduce test workflow for each PR/commit #1780

Merged
merged 7 commits into from
Aug 21, 2023
Merged

PR: Reduce test workflow for each PR/commit #1780

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
7e97ade
unit and integration matrix created dynamically, exclude works, no de…
bigtedde Aug 17, 2023
518fe3f
FIXED: faulty debug logic always runs tmate session on manual workflow
bigtedde Aug 19, 2023
0975b11
cleanup + exclude working as intended
bigtedde Aug 19, 2023
df33623
new exclude
bigtedde Aug 20, 2023
29a7727
made matrix script easier to read
bigtedde Aug 20, 2023
9655633
corrected json formatting
bigtedde Aug 21, 2023
8f94ad3
'short' integration test matrix os=linux only
bigtedde Aug 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .github/scripts/generate-matrix.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
event_name="$1"
branch_name="$2"

if [[ "$event_name" == "workflow_dispatch" ]] || [[ "$branch_name" == "master" ]]; then
echo '{"python-version": ["3.8", "3.9", "3.10", "3.11"], "os": ["ubuntu-latest", "macos-latest"], "borg-version": ["1.2.4"]}' > matrix-unit.json
echo '{"python-version": ["3.8", "3.9", "3.10", "3.11"], "os": ["ubuntu-latest", "macos-latest"], "borg-version": ["1.1.18", "1.2.2", "1.2.4", "2.0.0b5"]}' > matrix-integration.json
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you write the exclude like that when testing:

Suggested change
echo '{"python-version": ["3.8", "3.9", "3.10", "3.11"], "os": ["ubuntu-latest", "macos-latest"], "borg-version": ["1.1.18", "1.2.2", "1.2.4", "2.0.0b5"]}' > matrix-integration.json
echo '{"python-version": ["3.8", "3.9", "3.10", "3.11"], "os": ["ubuntu-latest", "macos-latest"], "borg-version": ["1.1.18", "1.2.2", "1.2.4", "2.0.0b5"], "exclude": [{"borg-version":"2.0.0b5, "python-version":"3.8"}]}' > matrix-integration.json

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@real-yfprojects This worked like a charm :)


elif [[ "$event_name" == "push" ]] || [[ "$event_name" == "pull_request" ]]; then
echo '{"python-version": ["3.8", "3.9", "3.10", "3.11"], "os": ["ubuntu-latest", "macos-latest"], "borg-version": ["1.2.4"]}' > matrix-unit.json
echo '{"python-version": ["3.10"], "os": ["ubuntu-latest", "macos-latest"], "borg-version": ["1.2.4"]}' > matrix-integration.json
fi
64 changes: 45 additions & 19 deletions .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,16 +26,36 @@ jobs:
shell: bash
run: make lint

prepare-matrix:
runs-on: ubuntu-latest
outputs:
matrix-unit: ${{ steps.set-matrix-unit.outputs.matrix }}
matrix-integration: ${{ steps.set-matrix-integration.outputs.matrix }}
steps:
- uses: actions/checkout@v3

- name: Give execute permission to script
run: chmod +x ./.github/scripts/generate-matrix.sh

- name: Generate matrices
run: |
./.github/scripts/generate-matrix.sh "${{ github.event_name }}" "${GITHUB_REF##refs/heads/}"

- name: Set matrix for unit tests
id: set-matrix-unit
run: echo "matrix=$(cat matrix-unit.json)" >> $GITHUB_OUTPUT

- name: Set matrix for integration tests
id: set-matrix-integration
run: echo "matrix=$(cat matrix-integration.json)" >> $GITHUB_OUTPUT

test-unit:
needs: prepare-matrix
timeout-minutes: 20
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false

matrix:
python-version: ["3.8", "3.9", "3.10", "3.11"]
os: [ubuntu-latest, macos-latest]
borg-version: ["1.2.4"]
matrix: ${{fromJson(needs.prepare-matrix.outputs.matrix-unit)}}

steps:
- uses: actions/checkout@v3
Expand All @@ -51,15 +71,15 @@ jobs:

- name: Setup tmate session
uses: mxschmitt/action-tmate@v3
if: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.debug_enabled }}
if: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.debug_enabled == 'true' }}
Copy link
Collaborator Author

@bigtedde bigtedde Aug 19, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This corrects a serious error I found while testing. Before, the conditional 'debug_enabled' would always evaluate to True since the call returns a string which is truthy. This was causing a manual workflow to always be ran in debugging mode, even with debugging disabled.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another thing I found when running in debug mode is that the tests would always timeout (execution time > 20 minutes). Should we consider raising the execution timeout if debug mode is enabled?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that's because of the remote shell created. It stops execution and allows you to connect to the box.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, guess the timeout is fine as is then.


- name: Run Unit Tests with pytest (Linux)
if: runner.os == 'Linux'
env:
BORG_VERSION: ${{ matrix.borg-version }}
run: |
xvfb-run --server-args="-screen 0 1024x768x24+32" \
-a dbus-run-session -- make test-unit
-a dbus-run-session -- make test-unit

- name: Run Unit Tests with pytest (macOS)
if: runner.os == 'macOS'
Expand All @@ -78,51 +98,57 @@ jobs:
env_vars: OS, python

test-integration:
needs: prepare-matrix
timeout-minutes: 20
runs-on: ${{ matrix.os }}
env:
EXCLUDE: false # used to exclude incompatible combinations of python and borg
strategy:
fail-fast: false

matrix:
python-version: ["3.8", "3.9", "3.10", "3.11"]
os: [ubuntu-latest, macos-latest]
borg-version: ["1.1.18", "1.2.2", "1.2.4", "2.0.0b5"]
exclude:
- borg-version: "2.0.0b5"
python-version: "3.8"
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Biggest hurdle I ran into with the integration tests is excluding the combination of Borg 2.0.0b5 with python 3.8. Skipped this for now.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't adding the exclude to the matrix work?

Copy link
Collaborator Author

@bigtedde bigtedde Aug 19, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately it does not work with the way I'm dynamically creating the matrix, this only works when the matrix is static I believe.

matrix: ${{fromJson(needs.prepare-matrix.outputs.matrix-integration)}}

steps:
- uses: actions/checkout@v3

- name: Validate version combination # excludes all tests involving incompatible python and borg versions
run: |
if [[ "${{ matrix.python-version }}" == "3.8" && "${{ matrix.borg-version }}" == "2.0.0b5" ]]; then
echo "Python version 3.8 and Borg version 2.0.0b5 detected. This combination is excluded. Exiting job."
echo "EXCLUDE=true" >> $GITHUB_ENV
fi

- name: Install system dependencies
if: env.EXCLUDE == 'false'
uses: ./.github/actions/install-dependencies

- name: Setup python, vorta and dev deps
if: env.EXCLUDE == 'false'
uses: ./.github/actions/setup
with:
python-version: ${{ matrix.python-version }}
install-nox: true

- name: Setup tmate session
uses: mxschmitt/action-tmate@v3
if: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.debug_enabled }}
if: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.debug_enabled == 'true' && env.EXCLUDE == 'false'}}

- name: Run Integration Tests with pytest (Linux)
if: runner.os == 'Linux'
if: runner.os == 'Linux' && env.EXCLUDE == 'false'
env:
BORG_VERSION: ${{ matrix.borg-version }}
run: |
xvfb-run --server-args="-screen 0 1024x768x24+32" \
-a dbus-run-session -- make test-integration
-a dbus-run-session -- make test-integration

- name: Run Integration Tests with pytest (macOS)
if: runner.os == 'macOS'
if: runner.os == 'macOS' && env.EXCLUDE == 'false'
env:
BORG_VERSION: ${{ matrix.borg-version }}
PKG_CONFIG_PATH: /usr/local/opt/openssl@3/lib/pkgconfig
run: echo $PKG_CONFIG_PATH && make test-integration

- name: Upload coverage to Codecov
if: env.EXCLUDE == 'false'
uses: codecov/codecov-action@v3
env:
OS: ${{ runner.os }}
Expand Down