Bug: Workflow stuck at the Annotating resources step

[lucafaggianelli]

What happened?

The k8s-deploy action get stuck at the Annotating resources step:

Everything goes fine and the pods are correctly deployed, but the Github Actions job get stuck and I need it to cancel it manually or wait for it to timeout, resulting in a failed run.

I tried also to set the option annotate-namespace to false but nothing changes

Version

• I am using the latest version

Runner

self-hosted

Relevant log output

Debug log of the annotating resources step:

Annotating resources
  ##[debug]Kubectl run with command: /opt/hostedtoolcache/kubectl/1.25.4/x64/kubectl get,pods,-o,json,--namespace,engie-dash
  ##[debug]Getting workflows for repo: gempy/engie-dash
  ##[debug]Re-evaluate condition on job cancellation for step: 'Deploy to Kubernetes'.
  Error: The operation was canceled.
  ##[debug]System.OperationCanceledException: The operation was canceled.
  ##[debug]   at System.Threading.CancellationToken.ThrowOperationCanceledException()
  ##[debug]   at GitHub.Runner.Sdk.ProcessInvoker.ExecuteAsync(String workingDirectory, String fileName, String arguments, IDictionary`2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, Channel`1 redirectStandardIn, Boolean inheritConsoleHandler, Boolean keepStandardInOpen, Boolean highPriorityProcess, CancellationToken cancellationToken)
  ##[debug]   at GitHub.Runner.Common.ProcessInvokerWrapper.ExecuteAsync(String workingDirectory, String fileName, String arguments, IDictionary`2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, Channel`1 redirectStandardIn, Boolean inheritConsoleHandler, Boolean keepStandardInOpen, Boolean highPriorityProcess, CancellationToken cancellationToken)
  ##[debug]   at GitHub.Runner.Worker.Handlers.DefaultStepHost.ExecuteAsync(IExecutionContext context, String workingDirectory, String fileName, String arguments, IDictionary`2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, Boolean inheritConsoleHandler, String standardInInput, CancellationToken cancellationToken)
  ##[debug]   at GitHub.Runner.Worker.Handlers.NodeScriptActionHandler.RunAsync(ActionRunStage stage)
  ##[debug]   at GitHub.Runner.Worker.ActionRunner.RunAsync()
  ##[debug]   at GitHub.Runner.Worker.StepsRunner.RunStepAsync(IStep step, CancellationToken jobCancellationToken)
  ##[debug]Finishing: Deploy to Kubernetes

[jaiveerk]

Hey @lucafaggianelli,
Thanks for raising this issue! Sorry that this has been causing you trouble.

We believe this may be related to #264 which we've already created a fix for, and it will be part of our next release.

[lucafaggianelli]

cool, I'll try the next release thanks!

[jaiveerk]

Working on getting one out now! Will update as soon as it's out

[jaiveerk]

Release should be out with 4.8!

[lucafaggianelli]

thanks for the release but sadly I have a new error (I'm on GH Enterprise):

[lucafaggianelli]

@jaiveerk should I open another issue?

[jaiveerk]

Hm, that's interesting - not sure how something on our end could cause an error there. I wonder if an intermediate step is failing, would you mind running with debug logs?

Correct me if I'm wrong or misunderstanding, but the timeout message you posted above looks like the action timed out because our step was taking too long, but it looks like here the runner is failing to pull information from your workflows?

[lucafaggianelli]

Yes because the timeout seems to belong to the Annotating resources group, the rest of the workflow is OK and I don't send any HTTP API request to read the workflow.

And this is the workflow code:

name: Deploy to Kubernetes
uses: Azure/k8s-deploy@v4.8
with:
   action: deploy
   namespace: ${{ env.K8S_NAMESPACE }}
   manifests: |
     .deployment/k8s-manifests/deploy.yml
   images: |
     ${{ env.DOCKER_REGISTRY }}/riskpnl/${{ env.NAME }}:${{ github.sha }}
   annotate-namespace: 'false'

[jaiveerk]

Has this action worked for you in the past? I wonder if the issue is stemming from restricted repo access.

[lucafaggianelli]

as far as I tried this action, it never worked for me, v4.7 hanged forever at the Annotating resources step (this issue) and now this error. BTW if I remove the k8s-deploy action, the workflow passes

[jaiveerk]

We believe this may be due to an error with the runner's ability to access your repo due to the fact that you appear to be using a custom-hosted Github repo.

To resolve the issue for now, we've added an extra catch so that your action doesn't fail as a result of one single failed annotation attempt.

[lucafaggianelli]

Ok thanks a lot! And yes I use GitHub enterprise self hosted and the specific repo is marked as internal, if that may help I’ll test it and let you know. Il giorno gio 8 dic 2022 alle 18:11 Jaiveer Katariya < ***@***.***> ha scritto:

…

We believe this may be due to an error with the runner's ability to access your repo due to the fact that you appear to be using a custom-hosted Github repo. To resolve the issue for now, we've added an extra catch so that your action doesn't fail as a result of one single failed annotation attempt. — Reply to this email directly, view it on GitHub <#267 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCDSA6WOYGMMHSIIH2MJ2LWMIJC3ANCNFSM6AAAAAASVLNGOM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jaiveerk]

Sounds good! I'm waiting on PR approval from my team and we need to cut a new release, but I'll let you know when that's done!

[OliverMKing]

Reopening to track this until we release.