Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

401 during CD deployment #4382

Closed
jglick opened this issue Nov 7, 2024 · 12 comments
Closed

401 during CD deployment #4382

jglick opened this issue Nov 7, 2024 · 12 comments
Assignees
@dduportal
Labels
artifactory
Milestone
infra-team-sync-2...

Comments

@jglick
Copy link

jglick commented Nov 7, 2024

Service(s)

Artifactory

Summary

I am getting 401 errors trying to deploy CD releases of workflow-job. Most recently: https://github.com/jenkinsci/workflow-job-plugin/actions/runs/11731857921/job/32682817656

Acc. to https://github.com/jenkinsci/workflow-job-plugin/settings/secrets/actions the MAVEN_TOKEN is 9h old. Should it have been refreshed more recently?

Reproduction steps

No response
@jglick jglick added the triage Incoming issues that need review label Nov 7, 2024
@jglick jglick mentioned this issue Nov 7, 2024
Merged
@jglick
Copy link
Author

jglick commented Nov 7, 2024

I think it might be all plugins for which CD is broken: https://github.com/jenkinsci/jenkins-infra-test-plugin/actions/runs/11732201191/job/32683942682

@dduportal dduportal added this to the infra-team-sync-2024-11-12 milestone Nov 8, 2024
@dduportal dduportal removed the triage Incoming issues that need review label Nov 8, 2024
@dduportal dduportal self-assigned this Nov 8, 2024
@dduportal
Copy link
Contributor

First quick check: the RPU job on trusted.ci is running with success every 3 hours

Capture d’écran 2024-11-08 à 07 48 44

=> it is not a job failure.

Now checking the current state (as the issue is 8h old with two RPU runs) of tokens, and what the build is doing exactly (build log analysis)

@dduportal
Copy link
Contributor

Confirmed what @jglick says: token and username seems to have stopped being updated on GitHub side

Capture d’écran 2024-11-08 à 07 50 33

@dduportal
Copy link
Contributor

Audit trail:

  • Last successful build (known to have updated token ran is 38737:
[2024-11-07T13:01:29.007Z] 2024-11-07 13:01:29.003+0000 [id=1]	INFO	i.j.i.r.GitHubImpl#createOrUpdateRepositorySecret: Create/update the secret MAVEN_USERNAME for jenkinsci/workflow-step-api-plugin encrypted with key <redacted>
[2024-11-07T13:01:29.307Z] 2024-11-07 13:01:29.210+0000 [id=1]	INFO	i.j.i.r.GitHubImpl#createOrUpdateRepositorySecret: Create/update the secret MAVEN_TOKEN for jenkinsci/workflow-step-api-plugin encrypted with key <redacted>
  • Next build start to show stacktrace marked as WARNING when generating tokens on Artifactory:
2024-11-07T15:34:04.975Z] 2024-11-07 15:34:04.960+0000 [id=1]	WARNING	o.c.g.r.c.PlainObjectMetaMethodSite#doInvoke: Failed to generate token for jenkinsci/zos-connector-plugin
[2024-11-07T15:34:04.975Z] java.io.IOException: Server returned HTTP response code: 401 for URL: https://repo.jenkins-ci.org/access/api/v1/tokens
[2024-11-07T15:34:04.975Z] 	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:2013)
[2024-11-07T15:34:04.975Z] 	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1614)
[2024-11-07T15:34:04.975Z] 	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:223)
[2024-11-07T15:34:04.975Z] 	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
[2024-11-07T15:34:04.975Z] 	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
[2024-11-07T15:34:04.975Z] 	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
[2024-11-07T15:34:04.975Z] 	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
[2024-11-07T15:34:04.975Z] 	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:351)
[2024-11-07T15:34:04.975Z] 	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:64)
[2024-11-07T15:34:04.975Z] 	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:160)
[2024-11-07T15:34:04.975Z] 	at io.jenkins.infra.repository_permissions_updater.ArtifactoryAPI$ArtifactoryImpl$_generateTokenForGroup_closure2.doCall(ArtifactoryAPI.groovy:226)
[2024-11-07T15:34:04.975Z] 	at io.jenkins.infra.repository_permissions_updater.ArtifactoryAPI$ArtifactoryImpl$_generateTokenForGroup_closure2.doCall(ArtifactoryAPI.groovy)
[2024-11-07T15:34:04.975Z] 	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
[2024-11-07T15:34:04.975Z] 	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
[2024-11-07T15:34:04.975Z] 	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
[2024-11-07T15:34:04.975Z] 	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
[2024-11-07T15:34:04.975Z] 	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263)
[2024-11-07T15:34:04.975Z] 	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
[2024-11-07T15:34:04.975Z] 	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:37)
[2024-11-07T15:34:04.975Z] 	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:119)
[2024-11-07T15:34:04.975Z] 	at io.jenkins.infra.repository_permissions_updater.ArtifactoryAPI$ArtifactoryImpl$__clinit__closure1.doCall(ArtifactoryAPI.groovy:289)
[2024-11-07T15:34:04.975Z] 	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
[2024-11-07T15:34:04.975Z] 	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
[2024-11-07T15:34:04.975Z] 	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
[2024-11-07T15:34:04.975Z] 	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
[2024-11-07T15:34:04.975Z] 	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263)
[2024-11-07T15:34:04.975Z] 	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
[2024-11-07T15:34:04.975Z] 	at org.codehaus.groovy.runtime.InvokerHelper.invokePogoMethod(InvokerHelper.java:1020)
[2024-11-07T15:34:04.975Z] 	at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:1003)
[2024-11-07T15:34:04.975Z] 	at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodN(ScriptBytecodeAdapter.java:180)
[2024-11-07T15:34:04.975Z] 	at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeClosure(ScriptBytecodeAdapter.java:586)
[2024-11-07T15:34:04.975Z] 	at io.jenkins.infra.repository_permissions_updater.ArtifactoryAPI$ArtifactoryImpl.generateTokenForGroup(ArtifactoryAPI.groovy:214)
[2024-11-07T15:34:04.975Z] 	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
[2024-11-07T15:34:04.975Z] 	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
[2024-11-07T15:34:04.975Z] 	at org.codehaus.groovy.runtime.callsite.PlainObjectMetaMethodSite.doInvoke(PlainObjectMetaMethodSite.java:43)
[2024-11-07T15:34:04.975Z] 	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:190)
[2024-11-07T15:34:04.975Z] 	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:70)
[2024-11-07T15:34:04.975Z] 	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:143)
[2024-11-07T15:34:04.975Z] 	at io.jenkins.infra.repository_permissions_updater.ArtifactoryPermissionsUpdater$_generateTokens_closure9.doCall(ArtifactoryPermissionsUpdater.groovy:463)
[2024-11-07T15:34:04.975Z] 	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
[2024-11-07T15:34:04.975Z] 	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
[2024-11-07T15:34:04.975Z] 	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
[2024-11-07T15:34:04.975Z] 	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:323)
[2024-11-07T15:34:04.975Z] 	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:263)
[2024-11-07T15:34:04.975Z] 	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1041)
[2024-11-07T15:34:04.975Z] 	at groovy.lang.Closure.call(Closure.java:405)
[2024-11-07T15:34:04.975Z] 	at groovy.lang.Closure.call(Closure.java:421)
[2024-11-07T15:34:04.975Z] 	at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2330)
[2024-11-07T15:34:04.975Z] 	at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2315)
[2024-11-07T15:34:04.976Z] 	at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:2356)
[2024-11-07T15:34:04.976Z] 	at org.codehaus.groovy.runtime.dgm$186.invoke(Unknown Source)
[2024-11-07T15:34:04.976Z] 	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:244)
[2024-11-07T15:34:04.976Z] 	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)
[2024-11-07T15:34:04.976Z] 	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
[2024-11-07T15:34:04.976Z] 	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115)
[2024-11-07T15:34:04.976Z] 	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:127)
[2024-11-07T15:34:04.976Z] 	at io.jenkins.infra.repository_permissions_updater.ArtifactoryPermissionsUpdater.generateTokens(ArtifactoryPermissionsUpdater.groovy:452)
[2024-11-07T15:34:04.976Z] 	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
[2024-11-07T15:34:04.976Z] 	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
[2024-11-07T15:34:04.976Z] 	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:101)
[2024-11-07T15:34:04.976Z] 	at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite$StaticMetaMethodSiteNoUnwrapNoCoerce.invoke(StaticMetaMethodSite.java:149)
[2024-11-07T15:34:04.976Z] 	at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.callStatic(StaticMetaMethodSite.java:100)
[2024-11-07T15:34:04.976Z] 	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallStatic(CallSiteArray.java:55)
[2024-11-07T15:34:04.976Z] 	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:196)
[2024-11-07T15:34:04.976Z] 	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:208)
[2024-11-07T15:34:04.976Z] 	at io.jenkins.infra.repository_permissions_updater.ArtifactoryPermissionsUpdater.main(ArtifactoryPermissionsUpdater.groovy:518)

=> Looks like the Artifactory token is invalid or expired, as they were no code change in RPU

@dduportal
Copy link
Contributor

Confirmed the Artifactory admin token was expired. It looks like I did forget to put a reminder in #4203.

Capture d’écran 2024-11-08 à 08 00 36

@dduportal
Copy link
Contributor

  • Created a new token in Artifactory with the following settings (with s/YYYY-MM-DD/2024-11-08/g):
Capture d’écran 2024-11-08 à 08 09 53
  • Credential updated in trusted.ci with the new value, tokenID and TTL:
Capture d’écran 2024-11-08 à 08 10 08
  • Triggered an RPU build (38744)

@dduportal
Copy link
Contributor

Update:

  • Last RPU Job ran successfully (took ~ 30 min as usual instead of the suspicious ~ 5 min on failing jobs)
  • Token are updated

=> CD is back

@dduportal
Copy link
Contributor

Confirmed it worked for https://github.com/jenkinsci/jenkins-infra-test-plugin/actions/runs/11732201191

@timja timja mentioned this issue Nov 8, 2024
Open
@timja
Copy link
Member

timja commented Nov 8, 2024

created #4383 for monitoring

@dduportal
Copy link
Contributor

created #4383 for monitoring

Ah ah I was writing an issue for this, thanks Tim! ❤️

Also Added a calendar event for the team

@jglick
Copy link
Author

jglick commented Nov 8, 2024 •
edited
Loading

Created a new token in Artifactory…

Yuck, does it not support OIDC?

@dduportal
Copy link
Contributor

Created a new token in Artifactory…

Yuck, does it not support OIDC?

In theory yes: https://jfrog.com/help/r/jfrog-platform-administration-documentation/configure-an-oidc-integration

I'm not sure about the effort required to have it used on such a sensitive system though (compared to rotating a token every 3 months)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dduportal dduportal

Labels
artifactory
Projects
None yet
Milestone
infra-team-sync-2024-11-12
Development

No branches or pull requests
3 participants
@jglick @dduportal @timja