[bug][github actions] codecov base commit does not match the "merge commit" that the branch used to generate coverage #564

Ryang20718 · 2024-11-15T23:51:57Z

On codecov reports (github actions), we see the base commit used to compare the PR is rather stale. example PR is Ryang20718/cflag#28

i.e if my branch was branched off commit C whilst A is head, if I submit a pull request to run in github actions (for CI), we run on the merged commit.

A --> main head
|
B
|
C --> user/ryang/user_branch

Thus, the state of our branch is actually the following diagram.

A --> main head --> user/ryang/user_branch
|
B
|
C

This leads to a couple of unexpected bugs found in code coverage status checks. I.e because we run all of our tests on the merged commit, there may be a decent amount of files changed between commits C and A which we have inherited because we are running on the merged commit.

Thus, project level coverage may differ drastically even though the files we have touched have 100% patch coverage.

I have tried a couple of ways of bypassing this by overriding the commit to use the merged commit however, that doesn't work. If you manually rebase your PR and resubmit CI, then the base commit for the codecov report is accurate again. However, this is not a great developer experience as we'd like to make our codecov status checks blocking, but currently this is a bug

It seems like https://community.codecov.com/t/selected-base-commit-when-pr-is-based-on-an-older-commit-of-the-target-branch/4101 is a similar issue

I tried setting commit_parent in codecov-action similar to codecov/codecov-action#597, but to no avail
and also setting commit_parent to the HEAD of main, but to no avail

even though main has a more recent commits with coverage

Logs from attempt to use parent_commit

=> Running git config --global --add safe.directory /home/runner/work/cflag/cflag
/usr/bin/git config --global --add safe.directory /home/runner/work/cflag/cflag
==> Running command '/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov --codecov-yml-path .codecov.yml -v create-commit'
/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov --codecov-yml-path .codecov.yml -v create-commit --parent-sha de19c68b335e6e17e1e5b3204935d3c5e5d5b716 --git-service github -C 37d7897188310ffc193ea3340b682505f9ca2db8
info - 2024-11-16 00:[48](https://github.com/Ryang20718/cflag/actions/runs/11865673830/job/33071259742#step:4:49):48,182 -- ci service found: github-actions
debug - 2024-11-16 00:48:48,185 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
debug - 2024-11-16 00:48:48,185 -- Loading config from .codecov.yml
debug - 2024-11-16 00:48:48,188 -- Starting create commit process --- {"codecov_yml_path": ".codecov.yml", "verbose": true, "auto_load_params_from": null, "enterprise_url": null, "version": "cli-0.9.4", "command": "create-commit", "parent_sha": "de19c68b335e6e17e1e5b320[49](https://github.com/Ryang20718/cflag/actions/runs/11865673830/job/33071259742#step:4:50)35d3c5e5d5b716", "git_service": "github", "commit_sha": "37d7897188310ffc193ea3340b682505f9ca2db8", "pull_request_number": "28", "branch": "ryang/rebase", "fail_on_error": false, "slug": "Ryang20718/cflag"}
info - 2024-11-16 00:48:48,597 -- Process Commit creating complete
debug - 2024-11-16 00:48:48,597 -- Commit creating result --- {"result": "RequestResult(error=None, warnings=[], status_code=202, text='{\"status\":\"queued\"}\\n')"}
==> Running command '/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov --codecov-yml-path .codecov.yml -v create-report'
/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov --codecov-yml-path .codecov.yml -v create-report --git-service github -C 37d7897188310ffc193ea3340b682[50](https://github.com/Ryang20718/cflag/actions/runs/11865673830/job/33071259742#step:4:51)5f9ca2db8
info - 2024-11-16 00:48:49,317 -- ci service found: github-actions
debug - 2024-11-16 00:48:49,320 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
debug - 2024-11-16 00:48:49,320 -- Loading config from .codecov.yml
debug - 2024-11-16 00:48:49,324 -- Starting create report process --- {"codecov_yml_path": ".codecov.yml", "verbose": true, "auto_load_params_from": null, "enterprise_url": null, "version": "cli-0.9.4", "command": "create-report", "git_service": "github", "commit_sha": "37d7897188310ffc193ea3340b682505f9ca2db8", "code": "default", "pull_request_number": "28", "fail_on_error": false, "slug": "Ryang20718/cflag"}
info - 2024-11-16 00:48:49,806 -- Process Report creating complete
debug - 2024-11-16 00:48:49,806 -- Report creating result --- {"result": "RequestResult(error=None, warnings=[], status_code=202, text='{\"status\":\"queued\"}\\n')"}
info - 2024-11-16 00:48:49,806 -- Finished creating report successfully --- {"response": "{\"status\":\"queued\"}\n"}
==> Running command '/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov --codecov-yml-path .codecov.yml -v do-upload'
/home/runner/work/_actions/codecov/codecov-action/v4/dist/codecov --codecov-yml-path .codecov.yml -v do-upload -f torchscript_input_files_coverage.dat -F torchscript_input_files --git-service github -C 37d7897188310ffc193ea3340b682505f9ca2db8
info - 2024-11-16 00:48:50,523 -- ci service found: github-actions
debug - 2024-11-16 00:48:50,526 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
debug - 2024-11-16 00:48:50,526 -- Loading config from .codecov.yml
debug - 2024-11-16 00:48:50,530 -- Starting upload processing --- {"codecov_yml_path": ".codecov.yml", "verbose": true, "auto_load_params_from": null, "enterprise_url": null, "version": "cli-0.9.4", "command": "do-upload", "flags": ["torchscript_input_files"], "git_service": "github", "commit_sha": "37d7897188310ffc193ea3340b682505f9ca2db8", "report_code": "default", "network_root_folder": "/home/runner/work/cflag/cflag", "files_search_root_folder": "/home/runner/work/cflag/cflag", "files_search_exclude_folders": [], "disable_search": false, "disable_file_fixes": false, "build_code": "11865673830", "build_url": "https://github.com/Ryang20718/cflag/actions/runs/11865673830", "job_code": "ci", "name": "11865673830", "branch": "ryang/rebase", "pull_request_number": "28", "env_vars": {}, "plugin_names": ["xcode", "gcov", "pycoverage"], "dry_run": false, "use_legacy_uploader": false, "handle_no_reports_found": false, "report_type": "coverage", "network_filter": null, "network_prefix": null, "gcov_args": null, "gcov_ignore": null, "gcov_include": null, "gcov_executable": null, "swift_project": null, "fail_on_error": false, "slug": "Ryang20718/cflag"}
debug - 2024-11-16 00:48:50,530 -- Selected preparation plugins --- {"selected_plugins": ["<class 'codecov_cli.plugins.xcode.XcodePlugin'>", "<class 'codecov_cli.plugins.gcov.GcovPlugin'>", "<class 'codecov_cli.plugins.pycoverage.Pycoverage'>"], "cli_config": {}}
debug - 2024-11-16 00:48:50,530 -- Running preparation plugin: <class 'codecov_cli.plugins.xcode.XcodePlugin'>
debug - 2024-11-16 00:48:50,530 -- Running xcode plugin...
warning - 2024-11-16 00:48:50,531 -- xcrun is not installed or can't be found.
debug - 2024-11-16 00:48:50,531 -- Running preparation plugin: <class 'codecov_cli.plugins.gcov.GcovPlugin'>
debug - 2024-11-16 00:48:50,531 -- Running gcov plugin...
warning - 2024-11-16 00:48:50,532 -- No gcov data found.
debug - 2024-11-16 00:48:50,532 -- Running preparation plugin: <class 'codecov_cli.plugins.pycoverage.Pycoverage'>
warning - 2024-11-16 00:48:50,533 -- coverage.py is not installed or can't be found.
debug - 2024-11-16 00:48:50,533 -- Collecting relevant files
info - 2024-11-16 00:48:50,542 -- Found 2 coverage files to report
info - 2024-11-16 00:48:50,542 -- > /home/runner/work/cflag/cflag/onboard_av_binary_coverage.dat
info - 2024-11-16 00:48:50,542 -- > /home/runner/work/cflag/cflag/torchscript_input_files_coverage.dat
debug - 2024-11-16 00:48:50,545 -- Selected uploader to use: <class 'codecov_cli.services.upload.upload_sender.UploadSender'>
debug - 2024-11-16 00:48:50,546 -- Sending upload request to Codecov
info - 2024-11-16 00:48:50,983 -- Your upload is now processing. When finished, results will be available at: https://app.codecov.io/github/Ryang20718/cflag/commit/37d7897188310ffc193ea3340b682505f9ca2db8
debug - 2024-11-16 00:48:50,983 -- Upload request to Codecov complete. --- {"response": {"raw_upload_location": "https://storage.googleapis.com/codecov/shelter/github/Ryang20718%3A%3A%3A%3Acflag/37d7897188310ffc193ea3340b682505f9ca2db8/c39a133b-565c-46e3-b6bc-d590a640d796.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EJWQHUGIBILH4J7Q6ZUSCIFNEOLYSNDS7L3B4N5SIBQ2J4YLYE5CRFCD%2F20241116%2Fus%2Fs3%2Faws4_request&X-Amz-Date=20241116T004850Z&X-Amz-Expires=30&X-Amz-SignedHeaders=host&X-Amz-Signature=abf385f33cff4a5bfb2bf19cacdd6f640cf9e1d629645a9bd3f65b1dea1ce3d5", "url": "https://app.codecov.io/github/Ryang20718/cflag/commit/37d7897188310ffc193ea3340b682505f9ca2db8"}}
debug - 2024-11-16 00:48:50,983 -- Sending upload to storage
info - 2024-11-16 00:48:[51](https://github.com/Ryang20718/cflag/actions/runs/11865673830/job/33071259742#step:4:52),164 -- Process Upload complete
debug - 2024-11-16 00:48:51,164 -- Upload result --- {"result": "RequestResult(error=None, warnings=[], status_code=200, text='')"}

The text was updated successfully, but these errors were encountered:

Ryang20718 · 2024-11-15T23:53:20Z

cc @thomasrockhu-codecov

From a user's perspective this seems to be a bug. Wondering what your thoughts on this are and if there are any work arounds that I'm missing?

I'm really pushing hard to adopt codecov and am happy to help get this feature in if needed

thomasrockhu-codecov · 2024-11-19T20:44:37Z

@Ryang20718 sorry I had a stomach bug yesterday. I'm going to escalate this to our product team to take a look

Ryang20718 · 2024-11-21T17:11:40Z

@thomasrockhu-codecov let me know if there's any additional information I can help provide. We're currently in the middle of switching to codecov and I would love to get these edges ironed out!

thomasrockhu-codecov · 2024-11-21T18:08:46Z

@Ryang20718 sorry I forgot to respond. We made some changes here which should go back to the way we were pulling commits initially for merged commits. Do you mind checking with the vanilla action v5?

Ryang20718 · 2024-11-21T21:17:30Z

@thomasrockhu-codecov
Tried with codecov-action v5 in this run URL for this PR

Tried with merge commit sha

      - name: Checkout repository
        uses: actions/checkout@v4

      - uses: codecov/codecov-action@v5
        with:
          files: torchscript_input_files_coverage.dat
          flags: torchscript_input_files
          verbose: true
          commit_parent: ${{ github.event.pull_request.merge_commit_sha }} 
          token: ${{ secrets.CODECOV_TOKEN }}
          # codecov_yml_path: .codecov.yml

      - uses: codecov/codecov-action@v5
        with:
          files: onboard_av_binary_coverage.dat
          flags: onboard_av_binary
          verbose: true
          commit_parent: ${{ github.event.pull_request.merge_commit_sha }} 
          token: ${{ secrets.CODECOV_TOKEN }}

==> Running create-commit
      ./codecov --verbose create-commit -t <redacted> --git-service github --parent-sha 6cdbd778f354568e7915e152a7c43dd426d8b2e2 --sha bdf2b0479f7cf6a2b5f351eb01af4f6e35958e8d
info - 2024-11-21 20:55:57,653 -- ci service found: github-actions
debug - 2024-11-21 20:55:57,657 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
debug - 2024-11-21 20:55:57,659 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
debug - 2024-11-21 20:55:57,662 -- Loading config from /home/runner/work/cflag/cflag/.codecov.yml
debug - 2024-11-21 20:55:57,666 -- Starting create commit process --- {"verbose": true, "auto_load_params_from": null, "codecov_yml_path": null, "enterprise_url": null, "version": "cli-0.9.4", "command": "create-commit", "git_service": "github", "parent_sha": "6cdbd778f354568e7915e152a7c43dd426d8b2e2", "commit_sha": "bdf2b0479f7cf6a2b5f351eb01af4f6e35958e8d", "pull_request_number": "28", "branch": "ryang/rebase", "fail_on_error": false, "slug": "Ryang20718/cflag"}
info - 2024-11-21 20:55:58,016 -- Process Commit creating complete
debug - 2024-11-21 20:55:58,017 -- Commit creating result --- {"result": "RequestResult(error=None, warnings=[], status_code=202, text='{\"status\":\"queued\"}\\n')"}
 
==> Running create-report
      ./codecov --verbose create-report -t <redacted> --git-service github --sha bdf2b0479f7cf6a2b5f351eb01af4f6e35958e8d
info - 2024-11-21 20:55:58,767 -- ci service found: github-actions
debug - 2024-11-21 20:55:58,770 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
debug - 2024-11-21 20:55:58,773 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
debug - 2024-11-21 20:55:58,776 -- Loading config from /home/runner/work/cflag/cflag/.codecov.yml
debug - 2024-11-21 20:55:58,782 -- Starting create report process --- {"verbose": true, "auto_load_params_from": null, "codecov_yml_path": null, "enterprise_url": null, "version": "cli-0.9.4", "command": "create-report", "git_service": "github", "commit_sha": "bdf2b0479f7cf6a2b5f351eb01af4f6e35958e8d", "code": "default", "pull_request_number": "28", "fail_on_error": false, "slug": "Ryang20718/cflag"}
info - 2024-11-21 20:55:59,315 -- Process Report creating complete
debug - 2024-11-21 20:55:59,315 -- Report creating result --- {"result": "RequestResult(error=None, warnings=[], status_code=202, text='{\"status\":\"queued\"}\\n')"}
info - 2024-11-21 20:55:59,315 -- Finished creating report successfully --- {"response": "{\"status\":\"queued\"}\n"}
 
==> Running do-upload
      ./codecov --verbose do-upload -t <redacted> --file onboard_av_binary_coverage.dat --flag onboard_av_binary --gcov-executable gcov --git-service github --sha bdf2b0479f7cf6a2b5f351eb01af4f6e35958e8d
info - 2024-11-21 20:56:00,074 -- ci service found: github-actions
debug - 2024-11-21 20:56:00,077 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
debug - 2024-11-21 20:56:00,080 -- versioning system found: <class 'codecov_cli.helpers.versioning_systems.GitVersioningSystem'>
debug - 2024-11-21 20:56:00,082 -- Loading config from /home/runner/work/cflag/cflag/.codecov.yml
debug - 2024-11-21 20:56:00,087 -- Starting upload processing --- {"verbose": true, "auto_load_params_from": null, "codecov_yml_path": null, "enterprise_url": null, "version": "cli-0.9.4", "command": "do-upload", "flags": ["onboard_av_binary"], "gcov_executable": "gcov", "git_service": "github", "commit_sha": "bdf2b0479f7cf6a2b5f351eb01af4f6e35958e8d", "report_code": "default", "network_root_folder": "/home/runner/work/cflag/cflag", "files_search_root_folder": "/home/runner/work/cflag/cflag", "files_search_exclude_folders": [], "disable_search": false, "disable_file_fixes": false, "build_code": "1[196](https://github.com/Ryang20718/cflag/actions/runs/11961496503/job/33347848788#step:5:203)1496503", "build_url": "https://github.com/Ryang20718/cflag/actions/runs/11961496503", "job_code": "ci", "name": "11961496503", "branch": "ryang/rebase", "pull_request_number": "28", "env_vars": {}, "plugin_names": ["xcode", "gcov", "pycoverage"], "dry_run": false, "use_legacy_uploader": false, "handle_no_reports_found": false, "report_type": "coverage", "network_filter": null, "network_prefix": null, "gcov_args": null, "gcov_ignore": null, "gcov_include": null, "swift_project": null, "fail_on_error": false, "slug": "Ryang20718/cflag"}
debug - 2024-11-21 20:56:00,088 -- Selected preparation plugins --- {"selected_plugins": ["<class 'codecov_cli.plugins.xcode.XcodePlugin'>", "<class 'codecov_cli.plugins.gcov.GcovPlugin'>", "<class 'codecov_cli.plugins.pycoverage.Pycoverage'>"], "cli_config": {}}
debug - 2024-11-21 20:56:00,088 -- Running preparation plugin: <class 'codecov_cli.plugins.xcode.XcodePlugin'>
debug - 2024-11-21 20:56:00,088 -- Running xcode plugin...
warning - 2024-11-21 20:56:00,088 -- xcrun is not installed or can't be found.
debug - 2024-11-21 20:56:00,088 -- Running preparation plugin: <class 'codecov_cli.plugins.gcov.GcovPlugin'>
debug - 2024-11-21 20:56:00,088 -- Running gcov plugin...
warning - 2024-11-21 20:56:00,090 -- No gcov data found.
debug - 2024-11-21 20:56:00,090 -- Running preparation plugin: <class 'codecov_cli.plugins.pycoverage.Pycoverage'>
warning - 2024-11-21 20:56:00,090 -- coverage.py is not installed or can't be found.
debug - 2024-11-21 20:56:00,090 -- Collecting relevant files
info - 2024-11-21 20:56:00,101 -- Found 2 coverage files to report
info - 2024-11-21 20:56:00,101 -- > /home/runner/work/cflag/cflag/torchscript_input_files_coverage.dat
info - 2024-11-21 20:56:00,101 -- > /home/runner/work/cflag/cflag/onboard_av_binary_coverage.dat
debug - 2024-11-21 20:56:00,104 -- Selected uploader to use: <class 'codecov_cli.services.upload.upload_sender.UploadSender'>
debug - 2024-11-21 20:56:00,105 -- Sending upload request to Codecov
info - 2024-11-21 20:56:00,446 -- Your upload is now processing. When finished, results will be available at: https://app.codecov.io/github/Ryang20718/cflag/commit/bdf2b0479f7cf6a2b5f351eb01af4f6e35958e8d
debug - 2024-11-21 20:56:00,447 -- Upload request to Codecov complete. --- {"response": {"raw_upload_location": "https://storage.googleapis.com/codecov/shelter/github/Ryang20718%3A%3A%3A%3Acflag/bdf2b0479f7cf6a2b5f351eb01af4f6e35958e8d/655ef708-fcea-4876-98d2-0bdecc5ed896.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EJWQHUGIBILH4J7Q6ZUSCIFNEOLYSNDS7L3B4N5SIBQ2J4YLYE5CRFCD%2F20241121%2Fus%2Fs3%2Faws4_request&X-Amz-Date=20241121T205600Z&X-Amz-Expires=30&X-Amz-SignedHeaders=host&X-Amz-Signature=f4d90076c86462bbfe75d9e6a792db722dde96b105399302128e578a18ef7d1b", "url": "https://app.codecov.io/github/Ryang20718/cflag/commit/bdf2b0479f7cf6a2b5f351eb01af4f6e35958e8d"}}
debug - 2024-11-21 20:56:00,447 -- Sending upload to storage
info - 2024-11-21 20:56:00,689 -- Process Upload complete
debug - 2024-11-21 20:56:00,689 -- Upload result --- {"result": "RequestResult(error=None, warnings=[], status_code=[200](https://github.com/Ryang20718/cflag/actions/runs/11961496503/job/33347848788#step:5:207), text='')"}

but still 3 commits behind

Ryang20718 · 2024-11-21T21:18:56Z

also tried using a commit from main directly, but same result (report is still not using latest main to compare as base)

      - uses: codecov/codecov-action@v5
        with:
          files: onboard_av_binary_coverage.dat
          flags: onboard_av_binary
          verbose: true
          commit_parent: de19c68b335e6e17e1e5b3204935d3c5e5d5b716 #commit from main
          token: ${{ secrets.CODECOV_TOKEN }}

Ryang20718 · 2024-11-21T21:19:48Z

side note

codecov_yml_path is broken on v5 using the vanilla v5 action

thomasrockhu-codecov · 2024-11-21T21:21:16Z

@Ryang20718 yeah, ~~I'm about to push a fix for the codecov_yml_path~~ I'm going to push a fix after the US holiday, however note that that's not related to coverage.

thomasrockhu-codecov · 2024-11-21T21:25:42Z

@Ryang20718 perhaps I'm not understanding, is this what your commit structure is like

         --- C' --- D' (feature)
       /            ^ HEAD
A --- B   --- C --- D  (main)
                    ^HEAD

And on D' you are expecting coverage to be generated between D' and D or D' and B?

Ryang20718 · 2024-11-21T22:53:19Z

Assuming the current git structure looks like this

           --- C' --- D' (feature)
          /                  ^ HEAD
A --- B   --- C --- D  (main)
                    ^HEAD

When running coverage right now, base is getting set to B

            --- C' --- D' (feature)
          /                  ^ HEAD
A --- B   --- C --- D  (main)
          ^BASE          ^HEAD

However, since github actions pull requests merges main into your PR, when running CI, my actual feature branch would have a merged main into my feature branch to run CI, so the merge base commit is D, or you can call it the parent commit.

This is what I expect codecov to do when I specify the merge commit as the parent commit.

                              D' (feature)
                             /    ^ HEAD
A --- B   --- C --- D  (main)
                               ^HEAD
                               ^BASE

So I am expecting D' to be compared with D. Currently coverage is being comapared with D' and B.

I'm assuming parrent commit is supposed to allow you to specify the base commit? @thomasrockhu-codecov

are there any examples on how to do this with github actions or is there a bug here?

thomasrockhu-codecov · 2024-11-22T03:11:48Z

Ok, setting the base to B makes sense in that first git structure. What I'm not understanding is the GHAction pull request merging main into PR

Parent commit should be the commit that is the direct parent to the commit you are uploading. I can look for an example but I don't have one for you.

Let me pull back to this PR old.

HEAD
2127e0f right now
BASE - expected
de19c68 because that is the HEAD of main right now and your PR is trying to merge to main.
BASE - actual
4abb84a Codecov comment is showing BASE to be this which is of course not the same.

Looking at the commit history in GitHub

I see that all 4 of the first commits on main have Codecov status checks, which means that we have some upload from those commits.

What I do notice is that the FIRST commit on this PR on Codecov c177be5 is comparing against 4abb84a.

Did I get all this right? I will mention that I'm going to be offline for the next week, and going to see if someone on the team can take a look.

drazisil-codecov · 2024-11-22T15:14:59Z

Hi @Ryang20718

Tom handed this issue off to me, let me read though it and see what the issue is.

Ryang20718 · 2024-11-22T15:35:44Z

Did I get all this right?

correct !

What I'm not understanding is the GHAction pull request merging main into PR

the checkout action on a pull request uses the merge commit of a PR (HEAD of pr merged into target branch) https://github.com/actions/checkout#Checkout-pull-request-HEAD-commit-instead-of-merge-commit

Ryang20718 · 2024-11-22T15:36:37Z

This is the crux of the issue

HEAD
2127e0f right now
BASE - expected
de19c68 because that is the HEAD of main right now and your PR is trying to merge to main.
BASE - actual
4abb84a Codecov comment is showing BASE to be this which is of course not the same.

drazisil-codecov · 2024-11-25T14:14:07Z

@rohan-at-sentry I'd love to huddle with you on this one if you have time. I think I understand the issue here, would like to see if it makes any sense.

Ryang20718 · 2024-11-28T02:16:07Z

let me know if there's anything I can do to help investigate/verify! I'm hoping to make code coverage blocking at dayjob once this is resolved!

Ryang20718 · 2024-12-02T17:01:35Z

@drazisil-codecov Wondering if you were able to root cause the issue?

Ryang20718 · 2024-12-04T17:33:33Z

Sorry, for the ping again. Wondering if @drazisil-codecov is still investigating or @thomasrockhu-codecov ?

drazisil-codecov · 2024-12-04T19:13:09Z

Hi @Ryang20718 ,

I've been super busy, sorry. I believe that Jesse is escalating with engineering as part of your ticket.

Ryang20718 · 2024-12-07T00:14:26Z

thomasrockhu-codecov · 2024-12-13T15:50:03Z

@Ryang20718 I think I understand your question a little better, let me try to make sure I get it and perhaps answer your question.

            --- C' --- D' (feature)
          /                  ^ HEAD
A --- B   --- C --- D  (main)
          ^BASE          ^HEAD

Let's assume that the structure is like this before you run CI

GitHub Actions will do something like

            --- C' --- D' 
          /              > ---- E (merge commit)
A --- B   --- C --- D           ^HEAD

So we actually receive E

However, E doesn't actually exist as a commit on the git tree. If we were to take coverage for that commit, it would never show up on Codecov nor on GitHub. Thus, we take D' instead as that is the actual commit SHA that exists in the git tree.

If you want the merge of the two, then you would need to merge/rebase the feature branch onto main in order to get C and D. Does this make sense or did I miss something crucial?

Ryang20718 · 2024-12-13T17:42:47Z

I was able to narrow down the bug and work around this issue via the CLI. @thomasrockhu-codecov codecov-action has a bug

        shared_flags="--auto-load-params-from=GithubActions --codecov-yml-path=${CODECOV_YML} --verbose"
        sha="${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}"
        file=()
        IFS=',' read -r -a files <<< "$FILES"
        for f in "${files[@]}"; do
          file+=("--file=${f}")
        done
        echo "${file[@]}"
        flag=()
        for f in ${FLAG}; do
          flag+=("--flag=${f}")
        done
        ./codecov \
          ${shared_flags} \
          create-commit \
          --slug="${{ github.repository }}" \
          --sha="${sha}" \
          --git-service github \
          -t ${CODECOV_TOKEN}
        ./codecov \
          ${shared_flags} \
          create-report \
          --slug="${{ github.repository }}" \
          --sha="${sha}" \
          --git-service github \
          -t ${CODECOV_TOKEN}

        ./codecov \
          ${shared_flags} \
          pr-base-picking \
          --slug="${{ github.repository }}" \
          --base-sha="${{ github.event.pull_request.base.sha }}" \
          --service github \
          -t ${CODECOV_TOKEN}
        ./codecov \
          ${shared_flags} \
          do-upload \
          --slug="${{ github.repository }}" \
          --sha="${sha}" \
          --disable-search \
          ${file[@]+"${file[@]}"} \
          ${flag[@]+"${flag[@]}"} \
          --git-service github \
          -t ${CODECOV_TOKEN}

thomasrockhu-codecov self-assigned this Nov 19, 2024

thomasrockhu-codecov added the bug Something isn't working label Nov 19, 2024

thomasrockhu-codecov removed their assignment Nov 19, 2024

thomasrockhu-codecov added Medium Medium Priority Issues (to be fixed or re-evaluated in 3 months Area: Report Processing Issues with report processing labels Nov 19, 2024

thomasrockhu-codecov self-assigned this Nov 21, 2024

thomasrockhu-codecov removed the bug Something isn't working label Nov 22, 2024

rohan-at-sentry added the bug Something isn't working label Nov 22, 2024

rohan-at-sentry added Investigation Needs investigation from engineering high High Priority Issues (to be fixed within 2 sprints) and removed Medium Medium Priority Issues (to be fixed or re-evaluated in 3 months labels Nov 25, 2024

This was referenced Dec 12, 2024

Codecov on PR not using upstream commit_parent codecov/codecov-action#1704

Open

(Internal) Coverage changes based on how the commit is pushed codecov/codecov-action#691

Closed

Ryang20718 closed this as completed Dec 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[bug][github actions] codecov base commit does not match the "merge commit" that the branch used to generate coverage #564

[bug][github actions] codecov base commit does not match the "merge commit" that the branch used to generate coverage #564

Ryang20718 commented Nov 15, 2024 •

edited

Loading

Ryang20718 commented Nov 15, 2024 •

edited

Loading

thomasrockhu-codecov commented Nov 19, 2024

Ryang20718 commented Nov 21, 2024

thomasrockhu-codecov commented Nov 21, 2024

Ryang20718 commented Nov 21, 2024

Ryang20718 commented Nov 21, 2024

Ryang20718 commented Nov 21, 2024

thomasrockhu-codecov commented Nov 21, 2024 •

edited

Loading

thomasrockhu-codecov commented Nov 21, 2024 •

edited

Loading

Ryang20718 commented Nov 21, 2024

thomasrockhu-codecov commented Nov 22, 2024 •

edited

Loading

drazisil-codecov commented Nov 22, 2024

Ryang20718 commented Nov 22, 2024

Ryang20718 commented Nov 22, 2024

drazisil-codecov commented Nov 25, 2024

Ryang20718 commented Nov 28, 2024

Ryang20718 commented Dec 2, 2024

Ryang20718 commented Dec 4, 2024

drazisil-codecov commented Dec 4, 2024

Ryang20718 commented Dec 7, 2024

thomasrockhu-codecov commented Dec 13, 2024

Ryang20718 commented Dec 13, 2024

[bug][github actions] codecov base commit does not match the "merge commit" that the branch used to generate coverage #564

[bug][github actions] codecov base commit does not match the "merge commit" that the branch used to generate coverage #564

Comments

Ryang20718 commented Nov 15, 2024 • edited Loading

Ryang20718 commented Nov 15, 2024 • edited Loading

thomasrockhu-codecov commented Nov 19, 2024

Ryang20718 commented Nov 21, 2024

thomasrockhu-codecov commented Nov 21, 2024

Ryang20718 commented Nov 21, 2024

Ryang20718 commented Nov 21, 2024

Ryang20718 commented Nov 21, 2024

thomasrockhu-codecov commented Nov 21, 2024 • edited Loading

thomasrockhu-codecov commented Nov 21, 2024 • edited Loading

Ryang20718 commented Nov 21, 2024

thomasrockhu-codecov commented Nov 22, 2024 • edited Loading

drazisil-codecov commented Nov 22, 2024

Ryang20718 commented Nov 22, 2024

Ryang20718 commented Nov 22, 2024

drazisil-codecov commented Nov 25, 2024

Ryang20718 commented Nov 28, 2024

Ryang20718 commented Dec 2, 2024

Ryang20718 commented Dec 4, 2024

drazisil-codecov commented Dec 4, 2024

Ryang20718 commented Dec 7, 2024

thomasrockhu-codecov commented Dec 13, 2024

Ryang20718 commented Dec 13, 2024

Ryang20718 commented Nov 15, 2024 •

edited

Loading

Ryang20718 commented Nov 15, 2024 •

edited

Loading

thomasrockhu-codecov commented Nov 21, 2024 •

edited

Loading

thomasrockhu-codecov commented Nov 21, 2024 •

edited

Loading

thomasrockhu-codecov commented Nov 22, 2024 •

edited

Loading