Bug: Panic when verifying policy, error "unknown variant ecdsa"

[viccuad]

When running kwctl verify as follows, mirroring a failure shown on integration tests in CI, I get:

2024-03-20T10:36:23.248610Z  WARN kwctl: Cannot fetch TUF repository: TufError(ParseMetadata { role: Root, source: Error("unknown variant `ecdsa`, expected one of `rsa`, `ed25519`, `ecdsa-sha2-nistp256`", line: 9, column: 22), backtrace: Backtrace(()) })

This happens for test policy registry://ghcr.io/kubewarden/tests/capabilities-psp:v0.1.9 and GA registry://ghcr.io/kubewarden/policies/capabilities-psp:v0.1.15

Full output:

Click me

$ kwctl --version
kwctl 1.11.0-rc5

Use the `info` command to display system information.

$ kwctl verify --verification-config-path verification-config.yml registry://ghcr.io/kubewarden/tests/capabilities-psp:v0.1.9
2024-03-20T10:36:23.248610Z  WARN kwctl: Cannot fetch TUF repository: TufError(ParseMetadata { role: Root, source: Error("unknown variant `ecdsa`, expected one of `rsa`, `ed25519`, `ecdsa-sha2-nistp256`", line: 9, column: 22), backtrace: Backtrace(()) })
2024-03-20T10:36:23.248644Z  WARN policy_fetcher::verify: Sigstore Verifier created without Fulcio data: keyless signatures are going to be discarded because they cannot be verified
2024-03-20T10:36:23.248649Z  WARN policy_fetcher::verify: Sigstore Verifier created without Rekor data: transparency log data won't be used
2024-03-20T10:36:23.248652Z  WARN policy_fetcher::verify: Sigstore capabilities are going to be limited
2024-03-20T10:36:23.248656Z  INFO sigstore::cosign::client_builder: Rekor public key not provided. Rekor integration disabled
2024-03-20T10:36:23.248660Z  INFO sigstore::cosign::client_builder: No Fulcio cert has been provided. Fulcio integration disabled
2024-03-20T10:36:25.214557Z  INFO sigstore::cosign::signature_layers: Ignoring bundle, rekor public key not provided to verification client bundle="{\"SignedEntryTimestamp\":\"MEUCIQCTsSmCHAnPYzjENHAUceoUqqjzNYyamfmXfmtjwszsDwIgHxzHN5c476RJZfS5xy6k7e1NiwUvqkr2w+Kd9u6/f2c=\",\"Payload\":{\"body\":\"eyJhcGlWZXJzaW9uIjoiMC4wLjEiLCJraW5kIjoiaGFzaGVkcmVrb3JkIiwic3BlYyI6eyJkYXRhIjp7Imhhc2giOnsiYWxnb3JpdGhtIjoic2hhMjU2IiwidmFsdWUiOiIwODFlZDJjYzJhYTZhYjA1YTQ0MzFkNDM1ZTNkNWViOGVhMzFjYTJjNTZiNjQwNDM4NGQxMjRjMmMwYWNkYzcxIn19LCJzaWduYXR1cmUiOnsiY29udGVudCI6Ik1FVUNJQk5JQ1Q3ZGtoVkNHd3ZyTk45RlZPRVhQWUxyWVA0elRaSGpqVHRVazVZTkFpRUE1SkxmS1NSK05XUUZlUytXbjE3aFl3eTNML2lFSFpod0dNS3AzV1VkSWxvPSIsInB1YmxpY0tleSI6eyJjb250ZW50IjoiTFMwdExTMUNSVWRKVGlCRFJWSlVTVVpKUTBGVVJTMHRMUzB0Q2sxSlNVUlRWRU5EUVhNclowRjNTVUpCWjBsVlFVNUZlVzR4WjJKWVFVdFJVR0ZYVW5CUVIxTmxNMDVGTjIxbmQwTm5XVWxMYjFwSmVtb3dSVUYzVFhjS1MycEZWazFDVFVkQk1WVkZRMmhOVFdNeWJHNWpNMUoyWTIxVmRWcEhWakpOVWtWM1JIZFpSRlpSVVVSRmQyaDZZVmRrZW1SSE9YbGFWRUZsUm5jd2VRcE5ha0Y0VFdwamVFNTZVWGROZW14aFJuY3dlVTFxUVhoTmFtTjRUbnBWZDAxNmFHRk5RazE0UlZSQlVFSm5UbFpDUVc5VVEwaE9jRm96VGpCaU0wcHNDazFHYTNkRmQxbElTMjlhU1hwcU1FTkJVVmxKUzI5YVNYcHFNRVJCVVdORVVXZEJSVEZxU0RKMmJtbHlkVkYxUzFNMWFFZHJkSGhSZFRVdmVWVlRaM2dLWTFreVEyc3pXR2hOVFRKdlVWSnhObVJPZWsxaE5GaERZVFpFUW1oWWVTdGFURkpUVURJMGFqSmhlVkY1WjNaWU5IRlBVRnBNYUVaNFlVOURRV1ZuZHdwblowaHJUVUUwUjBFeFZXUkVkMFZDTDNkUlJVRjNTVWhuUkVGVVFtZE9Wa2hUVlVWRVJFRkxRbWRuY2tKblJVWkNVV05FUVhwQlRVSm5UbFpJVWsxQ0NrRm1PRVZCYWtGQlRVSXdSMEV4VldSRVoxRlhRa0pSWTNaQlNsbFZaM293WmxWSUwxY3JTMlkwYkVaR1pETXJSRWhxUVdaQ1owNVdTRk5OUlVkRVFWY0taMEpTV1hkQ05XWnJWVmRzV25Gc05ucEtRMmhyZVV4UlMzTllSaXRxUWpSQ1owNVdTRkpGUldOVVFuWm9iVEZ2WkVoU2QyTjZiM1pNTW1Sd1pFZG9NUXBaYVRWcVlqSXdkbUV6Vm1sYVdHUm9ZMjFTYkdKcE9XNWhXRkp2WkZkSmRGbFhUakJoVnpsMVkzazRkVm95YkRCaFNGWnBURE5rZG1OdGRHMWlSemt6Q21ONU9YbGFXRlo2V1ZkS2MxcFRNWGxhVjNoc1dWaE9iRXhZUW5aaVIyeHFaVk14ZVdSWVRqQk1ibXgwWWtWQ2VWcFhXbnBNTW1oc1dWZFNla3d6V1hnS1RVSTBSME5wYzBkQlVWRkNaemM0ZDBGUldVVkZTRXBzV201TmRtUkhSbTVqZVRreVRVTTBlRXhxYTNkRloxbExTM2RaUWtKQlIwUjJla0ZDUVdkUlJRcGpTRlo2WVVSQlkwSm5iM0pDWjBWRlFWbFBMMDFCUlVWQ1FUVlRXbGQ0YkZsWVRteEpTRUoyWWtkc2FtVlVRVEpDWjI5eVFtZEZSVUZaVHk5TlFVVkVDa0pEYUdsTmFteG9Xa2RKTVZscVdUTlBWMVpwVGtSU2FFOVhSbXRaTWxFeFdUSlplazVxVVhkYVZFMDFXa2RXYTAxVWEzZFplbGw1VFVSclIwTnBjMGNLUVZGUlFtYzNPSGRCVVVWRlN6Sm9NR1JJUW5wUGFUaDJaRWM1Y2xwWE5IVlpWMDR3WVZjNWRXTjVOVzVoV0ZKdlpGZEtNV015Vm5sWk1qbDFaRWRXZFFwa1F6VnFZakl3ZDAxQldVdExkMWxDUWtGSFJIWjZRVUpDVVZGcFlUTldhVnBZWkdoamJWSnNZbWs1YWxsWVFtaFpiV3h6WVZoU2NGcFlUWFJqU0U1M0NreFlRblppUjJ4cVpWUkJTMEpuWjNGb2EycFBVRkZSUkVGM1RtOUJSRUpzUVdwRlFXeG9ia0UzTDJGWGVVaGtTMFEwY1ZKbWRHWm5hRmh4U1ZoQ1NWWUtaV2d5VFZWTWFrOU5aRVpLWmxOU2MzSjVPWGRpZVdsNlFpdHRlWGgxZG1SMFpEaE1RV3BDWVhsS2JVY3pLekYzT0VOcE9HVldOMFZoWmtWUVZ6TnlNd3BPT1VsSlJ6QnZTVmhTYjNRMlRWazNWVTVUUTNsQ2NHTmFTMUJqTXpZd2VsbGxPSG8xTjJzOUNpMHRMUzB0UlU1RUlFTkZVbFJKUmtsRFFWUkZMUzB0TFMwSyJ9fX19\",\"integratedTime\":1643305239,\"logIndex\":1181814,\"logID\":\"c0d23d6ad406973f9559f3ba2d1ca01f84147d8ffc5b8445c224f98b9591801d\"}}"
2024-03-20T10:36:25.214765Z  INFO sigstore::cosign::signature_layers: Ignoring certificate annotation reason="fulcio certificates not provided"
Error: Policy registry://ghcr.io/kubewarden/tests/capabilities-psp:v0.1.9 cannot be validated
Image verification failed: missing signatures
The following constraints were not satisfied:
kind: githubAction
owner: kubewarden
repo: null
annotations: null


Stack backtrace:
   0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
   1: kwctl::verify::verify::{{closure}}
   2: kwctl::main::{{closure}}
   3: tokio::runtime::park::CachedParkThread::block_on
   4: tokio::runtime::runtime::Runtime::block_on
   5: kwctl::main
   6: std::sys_common::backtrace::__rust_begin_short_backtrace
   7: std::rt::lang_start::{{closure}}
   8: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/ops/function.rs:284:13
   9: std::panicking::try::do_call
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:552:40
  10: std::panicking::try
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:516:19
  11: std::panic::catch_unwind
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panic.rs:142:14
  12: std::rt::lang_start_internal::{{closure}}
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/rt.rs:148:48
  13: std::panicking::try::do_call
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:552:40
  14: std::panicking::try
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:516:19
  15: std::panic::catch_unwind
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panic.rs:142:14
  16: std::rt::lang_start_internal
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/rt.rs:148:20
  17: main

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
   1: kwctl::main::{{closure}}
   2: tokio::runtime::park::CachedParkThread::block_on
   3: tokio::runtime::runtime::Runtime::block_on
   4: kwctl::main
   5: std::sys_common::backtrace::__rust_begin_short_backtrace
   6: std::rt::lang_start::{{closure}}
   7: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/ops/function.rs:284:13
   8: std::panicking::try::do_call
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:552:40
   9: std::panicking::try
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:516:19
  10: std::panic::catch_unwind
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panic.rs:142:14
  11: std::rt::lang_start_internal::{{closure}}
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/rt.rs:148:48
  12: std::panicking::try::do_call
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:552:40
  13: std::panicking::try
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:516:19
  14: std::panic::catch_unwind
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panic.rs:142:14
  15: std::rt::lang_start_internal
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/rt.rs:148:20
  16: main

The verification-config.yml is straight from kwctl scaffold verification-config > verification-config.yml:

# Default Kubewarden verification config
#
# With this config, the only valid policies are those signed by Kubewarden
# infrastructure.
#
# This config can be saved to its default location (for this OS) with:
#   kwctl scaffold verification-config > /home/vic/.config/kubewarden/verification-config.yml
#
# Providing a config in the default location enables Sigstore verification.
# See https://docs.kubewarden.io for more Sigstore verification options.
apiVersion: v1
allOf:
- kind: githubAction
  owner: kubewarden
  repo: null
  annotations: null
anyOf: null

Note:

While this makes image verification fail in kwctl and policy-server, we fail-closed, meaning that even if images are correctly signed, Kubewarden will fail the image verification and report "Image verification failed: missing signatures".

[flavio]

Seems to be caused by sigstore/sigstore-rs#338

[viccuad]

This needs:
a. Patched awslabs/touch: https://github.com/viccuad/tough/tree/patch-edcsa, see compare

b. Either use a [patch.crates-io] approach:

c. Or consume manually:

1. Patched sigstore-rs: https://github.com/viccuad/sigstore-rs/tree/patch-edcs see compare
2. Patched policy-fetcher
3. Patched policy-evaluator

d. Releases for kwctl and policy-server.

[viccuad]

Trying approach b, [patch.crates-io] is not reasonable.

The patch.crates.io Rust feature doesn't support crate features nor default features. We need to consume awslabs/touch with the http feature enabled.
One can't set it as default, hence one needs to remove the feature and hardcode it (same for awslabs/touch, awslabs/tough-ssm as they expect that http feature) (tried here).
Still, doing that means patching sigstore-rs to not consume awslabs/tought with the feature, and that brings the same problem again.

Simpler to fork the dependency chain, approach c.

[viccuad]

Consumed the pointed forks above in policy-fetcher, policy-evaluator, kwctl, and the upstream tough fix seems to be incomplete.

Moving to blocked.