Webhook Not Triggering Application Refresh After Updating ArgoCD from v2.6.7 to v2.6.8

[yassinepoly]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

After updating from version 2.6.7 to 2.6.8, the webhook is no longer triggering a refresh of our applications. This issue is impacting our workflow and preventing automatic updates of our deployed applications.

We have verified that the webhook configuration in our Git repository is correctly set up and that the URL is pointing to the correct ArgoCD API endpoint. The events that should trigger the refresh are properly configured as well.

We have also checked the delivery settings of the webhook in the Git repository to ensure that the requests are reaching the ArgoCD API endpoint without any restrictions or blocks.

Upon reviewing the ArgoCD server logs, we have found this warning message:

level=warning msg="Failed to refresh app 'infra-manifest' for controller reprocessing: error patching annotations in application \"infra-manifest\": the server could not find the requested resource (patch applications.argoproj.io infra-manifest)"

To Reproduce

Update ArgoCD from v2.6.7 to v2.6.8 and test webhook with a push event.

Expected behavior

Webhook triggers Applications refresh after a push event.

Version

v2.6.8+674cf8d

Logs
ArgoCD server logs warning message:

level=warning msg="Failed to refresh app 'infra-manifest' for controller reprocessing: error patching annotations in application \"infra-manifest\": the server could not find the requested resource (patch applications.argoproj.io infra-manifest)"

[lusu007]

Today, I discovered that I'm also encountering an ongoing issue with my Webhook. It has been malfunctioning for several months now, and despite being on the latest version (v2.7.7+4650bb2.dirty), the problem persists. The last time I observed it functioning properly was around version v2.6.7.

Currently, I am receiving the following error in my argocd-server pod for every application in ArgoCD:

time="2023-07-18T15:11:06Z" level=warning msg="Failed to refresh app 'redacted' for controller reprocessing: error patching annotations in application \"redacted\": the server could not find the requested resource (patch applications.argoproj.io redacted)"
time="2023-07-18T15:11:39Z" level=info msg="finished streaming call with code OK" grpc.code=OK grpc.method=Watch grpc.service=application.ApplicationService grpc.start_time="2023-07-18T15:10:35Z" grpc.time_ms=64185.07 span.kind=server system=grpc
time="2023-07-18T15:11:48Z" level=info msg="Notifying 1 settings subscribers: [0xc001100000]"

[crenshaw-dev]

Probably related to this then. https://github.com/argoproj/argo-cd/pull/12386/files

I bet it's somehow targeting the wrong ns for the patch.

[mikebryant]

I think this is a duplicate of #13158. There's a fix awaiting review here: #13976

[crenshaw-dev]

@mikebryant thank you so much for tracking that down, it completely escaped me.

[lusu007]

This issue is fixed in the latest release v2.7.9. Thank you! 😊

[crenshaw-dev]

Awesome, thanks for confirming, @lusu007!