Add support for multiple attestations at once

[jonjohnsonjr]

We should bikeshed exact format and names here.

Combined with: chainguard-dev/terraform-publisher-apko#18

I instrumented terraform output how long each WriteState invocation took, then ran a few things locally before and after.

terraform apply -auto-approve -target=module.go

With just a single module, things look not that different...

Total WriteState latency: 5.7s -> 4.185s (~27% drop)

Number of WriteState calls: 344 -> 260 (~24% drop)

But if we pull in some more modules:

terraform apply -auto-approve -target=module.go -target=module.dotnet -target=module.python -target=module.python-fips -target=module.go-fips

Total WriteState latency: 50s -> 33s (~34% drop)

Number of WriteState calls: 1078 -> 796 (~26% drop)

If we look plot each point (milliseconds), we see something like this:

The latency for each WriteState grows ~linearly with the the size of the terraform.tfstate file, which is proportional to the number of resources, so the overall latency will grow quadratically.

Terraform also allocates two new buffer that end up growing to be the size of the terraform.tfstate file (and the previous file), so this also starts to cause a lot of GC pressure as the size increases.

[mattmoor]

I think we're leaving ~3x on the table by not enabling attesting to multiple images simultaneously.

This is still going to be ~3x the WriteState volume as cosign_sign simply because we can't recursively attest (because we want to specialize the attestation body per architecture).

I'd be fine with adding this, but similar to wanting a form of oci_tag that accepts digest -> []tag, I think we want something like digest -> []statement here.

[mattmoor]

I think I'd call the above resources something different cosign_multi_attest or oci_multi_tag?

So I'd be fine with landing this as-is and trying to measure the impact of this on our mega module 🤞

[jonjohnsonjr]

So I'd be fine with landing this as-is and trying to measure the impact of this on our mega module

SGTM -- collapsing at this level is nice because we get both fewer resources but also fewer HTTP requests to the registry (we do the whole .att tag at once), so this should make a nice dent.

Agree with adding a new resource if we want to collapse multiple archs/index into a single resource.

[mattmoor]

@jonjohnsonjr yeah, I also think it would be neat to consider having this "set" the attestations vs. augment them. Or at least have an option to strip ~other attestations of the same predicate type, so it doesn't grow monotonically.