Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ci][tvm-bot] Re-running an in-progress github action workflow fails #12847

Closed
driazati opened this issue Sep 20, 2022 · 0 comments
Closed

[ci][tvm-bot] Re-running an in-progress github action workflow fails #12847

driazati opened this issue Sep 20, 2022 · 0 comments
Labels
dev:ci needs-triage PRs or issues that need to be investigated by maintainers to find the right assignees to address it

Comments

@driazati
Copy link
Member

driazati commented Sep 20, 2022

See #12785 (comment) on #12785:

This happened while one job was failed (MacOS) and 2 were still running (Android and Windows). The rerun request went to workflow 3091384727 which was the stopped MacOS build, but it might have errored out since the MacOS is a failed job within a workflow that contains other running jobs (Windows / MacOS). The fix here might be to break up main.yml in .github/workflows to macos.yml / windows.yml / android.yml (though the rerun-failed-jobs endpoint sounds like it should work fine with this case). We could also try cancelling the whole workflow, waiting a bit, then issuing the rerun.

Traceback (most recent call last):
  File "/home/runner/work/tvm/tvm/ci/scripts/git_utils.py", line 121, in _request
    with request.urlopen(req, data) as response:
  File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "ci/scripts/github_tvmbot.py", line 574, in comment_failure
    raise item
  File "ci/scripts/github_tvmbot.py", line 685, in run
    pr.rerun_github_actions()
  File "ci/scripts/github_tvmbot.py", line 563, in rerun_github_actions
    raise e
  File "ci/scripts/github_tvmbot.py", line 555, in rerun_github_actions
    actions_github.post(f"actions/runs/{workflow_id}/rerun-failed-jobs", data={})
  File "/home/runner/work/tvm/tvm/ci/scripts/git_utils.py", line 143, in post
    return self._request(self.base + url, data, method="POST")
  File "/home/runner/work/tvm/tvm/ci/scripts/git_utils.py", line 126, in _request
    raise RuntimeError(f"Error response: {msg}\n{error_data}")
RuntimeError: Error response: HTTP Error 403: Forbidden
{"message":"This workflow is already running","documentation_url":"https://docs.github.com/rest/reference/actions#re-run-workflow-failed-jobs"}

cc @Mousius @areusch @gigiblender

@driazati driazati added the type:ci Relates to TVM CI infrastructure label Sep 20, 2022
@areusch areusch added dev:ci needs-triage PRs or issues that need to be investigated by maintainers to find the right assignees to address it labels Oct 17, 2022
@hpanda-naut hpanda-naut removed the type:ci Relates to TVM CI infrastructure label Nov 29, 2022
@tqchen tqchen closed this as completed Sep 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dev:ci needs-triage PRs or issues that need to be investigated by maintainers to find the right assignees to address it
Projects
None yet
Development

No branches or pull requests

4 participants