Failure in TUF-on-CI publish

[sigstore-bot]

Workflow run failed for TUF-on-CI publish.

Failed run: https://github.com/sigstore/root-signing-staging/actions/runs/10503325582

CC @sigstore/tuf-root-signing-staging-codeowners, please have a look.

[jku]

Analysis

The signing event #156 failed to publish because sigstore-java does not like the metadata.

• The target hashes in metadata changed to use only sha256 instead of both sha256 and sha512 -- this is a tuf-on-ci decision that became visible in root-signing-staging now since this is the first artifact modification. My belief was that this change had already happened in the metadata but that was not the case
• sigstore-java theoretically supports either hash but has a bug where it always tries to download the file using sha512 regardless of what the repository provides: var versionedTargetName = targetData.getHashes().getSha512() + "." + targetName;
• we need at least a temporary solution for the hash issue for actual root-signing signing event: we can't just break a client with no warning even if the bug is in sigstore-java. In staging we could break things but in this case that does not seem useful since we need a solution for the production signing event

Next steps

the current published timestamp expires 2024-08-28, usual signing schedule would be imply signing on 2024-08-24.
If we revert we should do it by Friday 2024-08-23. Same issue will come up in root-signing signing event so we need solutions for both: the difference is that here the signing event has been merged so clock is ticking.

These are the options for solving the root-signing-staging failure that I can see at the moment:

• revert to the merge of 78539ea: this means we won't get the staging test for items in Update to targets #157 in time for current root-signing signing event
• skip the java client test temporarily: this will make java client fail on staging which is not the end of the world -- this does nothing to help solve issue for root-signing
• Change tuf-on-ci to include sha512 in the metadata as well, release, deploy the release on root-signing-staging. Probably makes sense to revert and re-do the signing event in this case.
• Temporarily change tuf-on-ci upload-repository so it creates the sha512-prefixed target files too. These would be for sigstore-java specifically for a short time

Options for the ongoing root-signing signing event:

• Avoid making artifact changes: If we don't change artifacts, then this issue does not come up yet. This gives time to sigstore-java to fix the bug
• Make artifact changes, let sigstore-java break: highly undesirable, let's not do this
• Change tuf-on-ci to include sha512 as well, release, deploy the release on root-signing before making artifact changes in the signing event -- this could have side effects, I would like to test this longer
• Temporarily change tuf-on-ci upload-repository so it creates the sha512-prefixed target files too. These would be for sigstore-java specifically for a short time -- this is a hack but should have no side effects as metadata does not change

[kommendorkapten]

Great analysis.

I would like to know more about a timeline for fixing the Javaclient. It seems the fix would be small, but would we be able to distribute it in time? For prod TUF root, we have at least a week.

If we don't think the time is enough, my vote is to not updated the targets (i.e omit sigstore/root-signing#1268).

One question though, we do need to update targets.json, as the npm delegate's key will change. Will this change render in updated hashes in the targets metadata @jku?

[jku]

It seems the fix would be small, but would we be able to distribute it in time?

Without breaking users, I don't think so. sigstore-java would have to release a new version and all users would have to upgrade to it before the repository change happens.

[jku]

One question though, we do need to update targets.json, as the npm delegate's key will change. Will this change render in updated hashes in the targets metadata @jku?

No, only actual changes in the artifacts will trigger rebuilding the hashes

[kommendorkapten]

Perfect, then I think we should wait for next signing to modify targets to avoid breaking Java, and make sure the Java client is updated ASAP so we can perform any changes next time.

[kommendorkapten]

For staging, I'm tempted to keep as is? Assuming we can get a fix for the Java client pretty soon so at least the tip is working.

[jku]

Yeah that does seem reasonable. Summing up:

• We can add a hack to repository machinery to make things work for sigstore-java (linked just above) but...
• Since we were surprised by the hash change (or rather that it had not happened already), the safer choice is
  • for root-signing: Avoid making artifact changes in the initial signing event: this should mean no hash changes
  • for root-signing-staging: let sigstore-java break for now, remove it temporarily from tests

This means

• we can evaluate the hash change safety in good time
• the artifact changes get a little time for clients to test: that's good
• we will need a signing event a bit later for the artifact changes -- but that should be less painful now with tuf-on-ci so I think that's fine

[sigstore-bot]

Workflow run succeeded for TUF-on-CI publish.

Successful run: https://github.com/sigstore/root-signing-staging/actions/runs/10508655581

Closing issue based on this success.

[jku]

sigstore-java test was disabled temporarily:

• staging operates as usual
• sigstore-java on staging is broken until a (likely simple) client fix is deployed

Avoiding artifact changes in production root-signing still seems like a good call for the migration signing event that's going on