Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set up basic integration tests #13

Merged
merged 1 commit into from
Apr 13, 2021
Merged

Set up basic integration tests #13

merged 1 commit into from
Apr 13, 2021

Conversation

per1234
Copy link
Collaborator

@per1234 per1234 commented Apr 13, 2021

Although the action's Python code itself is currently well tested via unit tests, this does not provide any checks for
how the action interacts with the GitHub Actions framework and GitHub API.

Even though it won't provide full verification of the report, simply running the action provides a
"smoke test" as well as facilitating manual verification.

On every pull request event, the action will run and source the sketches reports test data from a local path.

When the job is triggered by an event from a fork, it will fail due to the access token not having the write permissions
required to comment on the PR thread. The workflow is triggered to still pass under these conditions. Checking the log to
see whether the action failed in the expected manner may still provide a useful validation of the PR.

When the PR is from a branch of the repository, the deltas report will be commented on the PR thread.

The workflow is also configured to run on push, schedule, workflow_dispatch, and repository_dispatch events. In this case,
it sources the sketches report from a workflow artifact. A dummy PR (#14) will be left permanently open in the repository with
the sole purpose of serving as a target for these tests.


"Golden" sketches reports for integration tests were generated from the repository attached here: #8 (comment)


Expected size deltas report resulting from the integration test workflow:


Memory usage change @ 651f05f4d4aca30ac359e972c01568f873112d43

Board flash % RAM for global variables %
arduino:avr:leonardo 💚 -40 - 0 -0.14 - 0.0 💚 -16 - 0 -0.62 - 0.0
arduino:sam:arduino_due_x 0 - 0 0.0 - 0.0 N/A N/A
arduino:samd:mkrzero ❔ -28 - +84 -0.01 - +0.03 🔺 0 - +32 0.0 - +0.1
Click for full report table
Board examples/MIDIUSB_clock
flash
% examples/MIDIUSB_clock
RAM for global variables
% examples/MIDIUSB_loop
flash
% examples/MIDIUSB_loop
RAM for global variables
% examples/MIDIUSB_read
flash
% examples/MIDIUSB_read
RAM for global variables
% examples/MIDIUSB_write
flash
% examples/MIDIUSB_write
RAM for global variables
%
arduino:avr:leonardo 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 -40 -0.14 -16 -0.62
arduino:sam:arduino_due_x 0 0.0 N/A N/A 0 0.0 N/A N/A 0 0.0 N/A N/A 0 0.0 N/A N/A
arduino:samd:mkrzero 84 0.03 0 0.0 0 0.0 0 0.0 -28 -0.01 0 0.0 0 0.0 32 0.1
Click for full report CSV
Board,examples/MIDIUSB_clock<br>flash,%,examples/MIDIUSB_clock<br>RAM for global variables,%,examples/MIDIUSB_loop<br>flash,%,examples/MIDIUSB_loop<br>RAM for global variables,%,examples/MIDIUSB_read<br>flash,%,examples/MIDIUSB_read<br>RAM for global variables,%,examples/MIDIUSB_write<br>flash,%,examples/MIDIUSB_write<br>RAM for global variables,%
arduino:avr:leonardo,0,0.0,0,0.0,0,0.0,0,0.0,0,0.0,0,0.0,-40,-0.14,-16,-0.62
arduino:sam:arduino_due_x,0,0.0,N/A,N/A,0,0.0,N/A,N/A,0,0.0,N/A,N/A,0,0.0,N/A,N/A
arduino:samd:mkrzero,84,0.03,0,0.0,0,0.0,0,0.0,-28,-0.01,0,0.0,0,0.0,32,0.1

Although the action's Python code itself is currently well tested via unit tests, this does not provide any checks for
how the action interacts with the GitHub Actions framework and GitHub API.

Even though it won't provide full verification of the report, simply running the action provides a
"smoke test" as well as facilitating manual verification.

On every pull request event, the action will run and source the sketches reports test data from a local path.

When the job is triggered by an event from a fork, it will fail due to the access token not having the write permissions
required to comment on the PR thread. The workflow is triggered to still pass under these conditions. Checking the log to
see whether the action failed in the expected manner may still provide a useful validation of the PR.

When the PR is from a branch of the repository, the deltas report will be commented on the PR thread.

The workflow is also configured to run on push, schedule, workflow_dispatch, and repository_dispatch events. In this case
it sources the sketches report from a workflow artifact. A dummy PR will be left permanently open in the repository with
the sole purpose of serving as a target for these tests.
@per1234 per1234 added topic: infrastructure Related to project infrastructure type: enhancement Proposed improvement labels Apr 13, 2021
@per1234 per1234 merged commit ef5174b into arduino:main Apr 13, 2021
@per1234 per1234 deleted the integration-tests branch April 13, 2021 04:19
@per1234 per1234 self-assigned this Jun 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
topic: infrastructure Related to project infrastructure type: enhancement Proposed improvement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant