WIP Implement rotation of service-account signing key

[johngmyers]

Needs documentation, but this is the basic idea.

The multiple-id-per-keypair design of VFS isn't useful because I can't identify the subkeys as being next, previous, or current. It only seems to be making calling the VFS API unnecessarily complicated. Any objections to my removing that feature?

/kind feature

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: johngmyers
To complete the pull request process, please assign geojaz
You can assign the PR to them by writing /assign @geojaz in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[justinsb]

Hand-waving, the advantage of the keyset approach is that we don't have to implement logic for each key, and the relationship between the keys in the set is (maybe) easier than the -next / -previous convention - at least for computers. Though the suffixes are also elegant :-)

With using multiple VFS files, we might have to be careful about atomicity of filesystem updates, whereas the keyset is a single file so should be easier to reason about...

We could alternatively add a state field to KeysetItem, so it applies to each key in the keyset. We can also look at the NotBefore / NotAfter or other metadata on each certificate to establish an ordering.

Whichever approach we choose, I think we should also try to tackle rotation of the CA certificate in the same pattern (or explicitly choose to follow a different pattern).

[johngmyers]

kubernetes/kubernetes#91070 has a sketch of what has to happen between key rotations. It's unfortunate the serviceaccount controller doesn't regenerate the tokens on a key change.

It's normal to have either current/next without previous or previous/current without next and the code needs to know the difference. So a simple ordering of keys in a keyset is insufficient.

As you can see, the rotation code handles atomicity issues by doing checks and updates in a particular order. It does rely on VFS committing an operation before returning.

The previous/current/next pattern could work for CAs, though the need to distribute trust further can complicate things. Another thing that can be done with CAs is to have the older CAs issue the newer ones.

[k8s-ci-robot]

@johngmyers: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

@johngmyers: The following tests failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
pull-kops-verify-cloudformation	7f02472	link	/test pull-kops-verify-cloudformation
pull-kops-verify-hashes	7f02472	link	/test pull-kops-verify-hashes
pull-kops-e2e-k8s-containerd	7f02472	link	/test pull-kops-e2e-k8s-containerd

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[olemarkus]

I did another rotate command here. I also tried to look at the usefulness of multiple id-per-keypair, but couldn't find a good way of making that work for this purpose. See #10516.

[johngmyers]

I think we need to add a concept of "next" (or "current") to the keyset code, so we can tell whether it is the newest or second newest key that should be used to mint things. I remember from looking at the keyset code earlier that the requirement that key ids be numeric was troublesome, but perhaps we can just add metadata for identifying the "current" key.

I also remember the legacy on-disk format being troublesome, especially since the legacy format was authoritative.

[johngmyers]

I remember the keystore interface lacking methods for atomically updating multiple key versions in a keyset.

[olemarkus]

I did another rotate command here. I also tried to look at the usefulness of multiple id-per-keypair, but couldn't find a good way of making that work for this purpose. See #10516.

since I made this comment, I made the keyset implementation work well for me. It allows for graceful rotation of the CA without risk of breaking the cluster if one has to break the rotation. I believe the same approach can be used here.

Once #10516 has been vetted, I am happy to try that approach with the signing key.

[johngmyers]

Superseded by #11204