Kubewarden policy to validate cosign images

[mudler]

In order to validate images used for upgrades in kubernetes, we can have a kubewarden policy to validate cosign images and prevent upgrades from being pulled and hitting nodes.

Ping Kubewarden team to see if they have already a policy for it and we can just re-use it (and document it)

Action Items

• See if there is a Kubewarden policy for validating cosign-ed images and see if fits our case, or either create a new one.
• QA and check if fits our scenarios
• Update our documentation with a practical example of building an image, signing it, and upgrading it in kubernetes with the kubewarden policy. Showing off also how an unsigned one would fail miserably in upgrading is a plus!

[mudler]

I've pinged the kubewarden team as I could not find any policy for cosign in https://hub.kubewarden.io/, if that's the case we should implement our own policy

[mudler]

I've checked in with @flavio and at the moment this doesn't seems to be technically possible with Kubewarden but it's in the roadmap.

Let's postpone this as there is no pressure for it right now, in the case we need a temporary solution to verify cosign images we can write a custom admission controller for the purpose or use https://artifacthub.io/packages/helm/sigstore/cosigned and replace it later with a kubewarden policy.

[mudler]

This seems to be possible now

[flavio]

This is the kubewarden policy that implements image verification: https://github.com/kubewarden/verify-image-signatures

Feel free to reach out if something is missing or if you want to propose improvements

[mudler]

From a preliminary test the policy seems sufficient as it is gating upgrades. I'm checking out now if I can make an upgrade going through it and documenting all the steps. So far looks promising

[mudler]

Bug was found along the way. Blocked by kubewarden/verify-image-signatures#20

[mudler]

Seems to be fixed upstream now, having a look at it again

[mudler]

With latest kubewarden I get a strange error on the validation policy pod:

 2022-07-06T09:08:50.479016Z ERROR validation{host="policy-server-default-8b84b74c8-2skhx" policy_id="namespaced-system-upgrade-gate-upgrades" kind="Pod" kind_group="" kind_version="v1" name="apply-os-upgrade-on-localhost-with-4d7cde11576e7ba6998d35-rdbpz" namespace="system-upgrade" operation="CREATE" request_uid="1c691265-11d1-4fb3-83fb-8d8c1b27874f" resource="pods" resource_group="" resource_version="v1" subresource=""}:policy_eval:validate{self=PolicyEvaluator {  id: "namespaced-system-upgrade-gate-upgrades", settings: {"modifyImagesWithDigest": Bool(true), "signatures": Array([Object({"image": String("quay.io/c3os/*"), "keyless": Array([Object({"issuer": String("https://token.actions.githubusercontent.com"), "subject": String("https://github.com/c3os-io/c3os/.github/workflows/image.yaml@refs/heads/master")})])})])} }}: policy_evaluator::runtimes::wapc: callback evaluation failed policy_id=1 binding="kubewarden" operation="v1/verify" error="Cannot pull quay.io/c3os/c3os:sha256-4a6f272b46db9a99998a5eeb335c061d106d129abe9194f3396641ef1c04260d.sig: error sending request for url (https://cdn02.quay.io/sha256/66/66e3dfcad3ef0692442eb17cd9eaccee2599bef6406ec3fe3a7711ad1381a26c?Expires=1657099125&Signature=P3oFjZ48~0Y8uR07LXYM9-ZjMSmxiNaecTTDLowy98jWCinvi5hs5-zBfVACAx9KVwwZoUgrfKyetmEerpliU4HsvObvuPDXOjFVlPRPE4TZ-DY4Y6SSg5gFJAkDcCFLirwkRM3dVw0d5s~5zU7~cn7AQ2~GMgsWoa0XRHhctJP4Ovv3F3UQSl19NQsCeKqbzz-pDWUgZCEtiavZru2dgrNfpj4EbkUWWgVfiUgIGbz2FS55csAcPol3Q3EzilBK7UxLoFOJVutDs5c7uX5HXYFoRCnLYwKKl4jgPCIFtI7nZZCjGTXlkcEKhi89FWLtHkRti1qZG57x~nf0eb5~lw__&Key-Pair-Id=APKAJ67PQLWGCSP66DGA): error trying to connect: dns error: failed to lookup address information: Try again" 

Weirdly enough, dns resolution seem to work just fine in the cluster.

When I describe the job that creates the pod, I see in the events:

  Warning  FailedCreate  10s (x2 over 20s)  job-controller  Error creating: admission webhook "namespaced-system-upgrade-gate-upgrades.kubewarden.admission" denied the request: Request rejected by policy namespaced-system-upgrade-gate-upgrades. The policy attempted to mutate the request, but it is currently configured to not allow mutation

Digging still 👀 ...

[viccuad]

The DNS resolution failure seems weird to be fair. I wonder if it is a fluke.

We just released v0.1.5 of the policy. It has a new github_actions signature setting, that is more secure (prevents about attackers consuming GHA reusable workflows and passing signatures as them) and also simpler to use (blog post in kubewarden.io/blog)

The policy with this settings should work:

apiVersion: policies.kubewarden.io/v1alpha2
kind: ClusterAdmissionPolicy
metadata:
  name: verify-image-signatures-policy
spec:
  module: registry://ghcr.io/kubewarden/policies/verify-image-signatures:v0.1.5
  rules:
  - apiGroups: [""]
    apiVersions: ["v1"]
    resources: ["pods"]
    operations:
    - CREATE
    - UPDATE
  mutating: true
  settings:
    signatures:
        - image: "quay.io/c3os/*"
          github_actions: 
              owner: "c3os-io"
              repo: "c3os" # optional

The second error, the mutation one, may have been because the ClusterAdmissionPolicy was missing .spec.mutating: true.
By default, the policy will validate the container tag and by default mutate the container definition to append the digest of the verified tag (digests are immutable, tags are mutable).

If one doesn't want the policy to mutate, they can set .spec.settings.modifyImagesWithDigest: false (default is true), and then it is irrelevant if the policy has .spec.mutating: true.

[viccuad]

Just tried with a different policy, and I get the dns error too:

2022-07-21T09:43:13.952451Z ERROR policy_server: error while downloading policy clusterwide-psp-capabilities from registry://ghcr.io/kubewarden/policies/capabilities-psp:v0.1.9: the policy registry://ghcr.io/kubewarden/policies/capabilities-psp:v0.1.9 could not be downloaded due to error: error sending request for url (https://ghcr.io/v2/): error trying to connect: dns error: failed to lookup address information: Try again

I will look into it.

[viccuad]

The DNS error seems to be here because I started the cluster with VPN enabled, and then took it off (thanks to @raulcabello for the hint). I wonder if that was it too in your side @mudler.

[frelon]

Elemental-toolkit does not provide images anymore, closing this.