Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using my GitHub Action to observe the performance regression introduced by PR #4491 and fixed by PR #5463 #6

Open
wants to merge 2 commits into
base: before-4491
Choose a base branch
from

Conversation

Anirban166
Copy link
Owner

This pull request mirrors #4491 which introduces a performance regression that got fixed in #5463 and includes two atime test cases - one to test for this, and one to check for another unrelated regression test case.

This will trigger my GitHub Action, commenting a plot on this PR thread, along with the commit that generated it, the link to the artifact containing the test results, the elapsed time from the job's start to the completion of the installation steps, and the time it took to run all the tests. The comment will get updated on subsequent commits or pushes to this PR.

The plot is based on time and memory performance analysis using the atime package on different versions of data.table, and will have the two test cases each with seven different labels for points associated with commits:

  • HEAD (PR source or after-4491 here),
  • base (PR target or before-4491 here),
  • merge-base (common ancestor between base and head),
  • CRAN (latest version of data.table on the platform),
  • Regression (a commit that is affected by the performance regression),
  • Before (a commit from before the introduction of the performance regression),
  • Fixed (a commit where the performance regression has been fixed, should be the same as or better than 'Before' in terms of performance)

Since this replicates the PR that introduces the regression for 'Test regression fixed in #​5463', I would expect the plot to show the commits associated with the base, merge-base, CRAN, Before, and Fixed labels to be fast while HEAD and Regression are slow.

For the other test case or 'Test regression fixed in #​4440', I would expect all the commits to be fast except the one associated with 'Regression'.

To reproduce on another fork, one can use the git commands I used locally on my clone of this repository:

git checkout -b before-4491
git reset 3fa8b20
git push --set-upstream origin before-4491
git switch -f master
git checkout -b after-4491
git reset 85adf09
git push --set-upstream origin after-4491

Then place my workflow under .github/workflows for the before-4491 branch, and include the tests under inst/atime for the after-4491 branch like I did here.

Copy link

github-actions bot commented Apr 9, 2024

Comparison Plot

Generated via commit 242c3d9

Download link for the artifact containing the test results: ↓ atime-results.zip

Time taken to finish the standard R installation steps: 11 minutes and 53 seconds

Time taken to run atime::atime_pkg on the tests: 3 minutes and 17 seconds

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants