Misleading error message when trust store is not configured

[yizha1]

What is the areas you experience the issue in?

Notation CLI

What is not working as expected?

I didn't configure trust store and run notation verify command directly. The error message was too general and misleading, see the following

$ notation verify $IMAGE
Error: signature verification failed for all the signatures associated with localhost:5001/net-monitor@sha256:8456f085dd609fd12cdebc5f80b6f33f25f670a7a9a03c8fa750b8aee0c4d657

With -v flag used, the detailed reason was shown.

$ notation verify $IMAGE -v
INFO Checking whether signature verification should be skipped or not
INFO Trust policy configuration: &{Name:policy-1 RegistryScopes:[localhost:5001/net-monitor] SignatureVerification:{VerificationLevel:strict Override:map[]} TrustStores:[ca:mystore] TrustedIdentities:[*]}
INFO Check over. Trust policy is not configured to skip signature verification
INFO Processing signature with manifest mediaType: application/vnd.oci.image.manifest.v1+json and digest: sha256:7aa4005f01c913531e1ac58176d2cc25ba9f5849ab07eb430dab2e1e04ddeff0
INFO Trust policy configuration: &{Name:policy-1 RegistryScopes:[localhost:5001/net-monitor] SignatureVerification:{VerificationLevel:strict Override:map[]} TrustStores:[ca:mystore] TrustedIdentities:[*]}
ERRO authenticity validation failed. Failure reason: error while loading the trust store, "/home/yizha1/.config/notation/truststore/x509/ca/mystore" does not exist
WARN Signature sha256:7aa4005f01c913531e1ac58176d2cc25ba9f5849ab07eb430dab2e1e04ddeff0 failed verification with error: error while loading the trust store, "/home/yizha1/.config/notation/truststore/x509/ca/mystore" does not exist
Error: signature verification failed for all the signatures associated with localhost:5001/net-monitor@sha256:8456f085dd609fd12cdebc5f80b6f33f25f670a7a9a03c8fa750b8aee0c4d657

What did you expect to happen?

The proposed error messages could be

$ notation verify $IMAGE
Error: the trust store with the name mystore of type ca doesn't exist. Use command "notation cert add" to add the trust store to verify $IMAGE

How can we reproduce it?

1. Configure trust policy with non-existing trust store for verifying the image
2. Run notation verify $image command

Describe your environment

Windows WSL2

What is the version of your Notation CLI or Notation Library?

Notation v1.0.0-rc.7

[priteshbandi]

$ notation verify $IMAGE
Error: the trust store '${truststore name}' of type ca doesn't exist. Use command "notation cert add" to create and add trusted certificates to trust store

cc: @iamsamirzon

[iamsamirzon]

@yizha1 - The above message needs changes.

Error: the trust store '${truststore name}' of type ca/signingAuthority doesn't exist. Use command "notation cert add" to create and add trusted certificates to the trust store

[JeyJeyGao]

In our current design, we log errors that are triggered during the signature verification process. Each signature may cause a unique error due to its association with a different trust store. We lack a comprehensive check of all configurations before initiating signature verification, which means these errors are only logged when processing each individual signature.

The challenge arises when dealing with multiple signatures. If a single signature causes an error, we can easily output that to the user. But what happens when multiple signatures trigger different trust store errors? How should we report these errors? Should we only output one error, or should we provide a detailed error report for each signature?

We need to define our UX strategy for handling these scenarios. Also discussed with @yizha1 and @priteshbandi, we acknowledge that it might not be feasible to implement these improvements prior to version 1.0.0 due to the need for substantial refactoring.

[yizha1]

@shizhMSFT @sajayantony @iamsamirzon Any comments on #699 (comment)? It seems we cannot fix it by v1, since it requires refactoring work. We can add these scenarios to troubleshooting guide for v1 release, that users need to enable -v flag to view error details.

[shizhMSFT]

After reviewing the notation-go code, the verification outcome of failed cases is not exposed. Therefore, it is not possible for CLI to know the detailed error of each signature verification. Therefore, we have to improve notation-go in post v1 (with no breaking changes) first to address this issue.

[yizha1]

@priteshbandi @iamsamirzon @FeynmanZhou @shizhMSFT @JeyJeyGao Since we cannot solve this issue ultimately in v1. I suggest adding additional text Run the command with "--verbose" or "-v" flag for details to the general error message, like

$ notation verify $IMAGE
Error: signature verification failed for all the signatures associated with localhost:5001/net-monitor@sha256:8456f085dd609fd12cdebc5f80b6f33f25f670a7a9a03c8fa750b8aee0c4d657
<new line>
Run the command with "--verbose" or "-v" flag for details

It should be a very simple fix, but it will help users to find the details of this error. Any comments?

[FeynmanZhou]

I think it's acceptable to remind users to use -v in the output to get detailed logs for troubleshooting in v1.0.0 and design a comprehensive check to cover multiple signatures scenario after v1.0.0.

[iamsamirzon]

Well, I will say we can just document the "-v" reference in our quick start guide, Vs changing code.

[yizha1]

Well, I will say we can just document the "-v" reference in our quick start guide, Vs changing code.

@iamsamirzon I created an issue to document the failure cases in troubleshooting guide. But it's a good point that in quick start, we can mention that users can use -v flag to get more logs.