Failing security scan workflows due to Trivy rate-limiting

[dandelany]

Description

The Aerie release process runs our Publish workflow in GH actions, which runs a Trivy security scan on all docker images. On recent releases, this scan often fails on some of the container scans. While it usually succeeds on subsequent runs, sometimes it takes many tries - last release I had to run it 12 times.

This appears to be a known issue that other projects are experiencing related to the way Trivy serves the DB of security issues on ghcr.io.

Fixes/workarounds

Not sure on the best fix for this yet, some possibilities:

• Can we download the DB once and use it for all container scans, instead of downloading once per container? This doesn't solve the issue but might make it occur less often
• Or we can cache the DB separately (see linked issue), but then we have to maintain it/host it somewhere.
• Or we could ditch Trivy entirely and use something else for security scans - need to track down what our requirements are for these.

[akshaykrjain]

I wanted to share an adjustment I made that allows referencing the Trivy database from the ECR Public Gallery, which simplifies setup for environments without needing to modify GitHub token permissions or add additional AWS ECR login steps.

Here’s what I added in the action’s configuration:

  - name: Run Trivy vulnerability scanner
    uses: aquasecurity/trivy-action@0.10.0
    env:
      TRIVY_DB_REPOSITORY: public.ecr.aws/aquasecurity/trivy-db
      TRIVY_JAVA_DB_REPOSITORY: public.ecr.aws/aquasecurity/trivy-java-db
    with:
      image-ref: ${{ steps.set-image-tag.outputs.image-tag }}
      ignore-unfixed: true
      vuln-type: 'os,library'
      severity: 'CRITICAL,HIGH'
      format: 'table'

This setup works seamlessly without extra configuration changes or permissions updates.

One question I have is about the update frequency of the Trivy database on ECR Public. Based on my checks, it appears actively maintained, but it would be great to know if there’s a set schedule for updates, or if users should be aware of any potential delays.

Thanks again, and hope this helps others looking to streamline their Trivy setup!

[dandelany]

Thanks for the tip @akshaykrjain - so if I understand correctly, that's an alternate repository (ECR Public Gallery) hosting the same Trivy DB. I was also concerned about update frequency of this and wanted to make sure it was really supported & maintained by the Trivy folks, so I looked into it - it appears so, and in fact they (aquasecurity) use the same Github action in the official Trivy repo to regularly publish the updates to both places at the same time, see:

https://github.com/aquasecurity/trivy-db/blob/main/.github/workflows/cron.yml#L71
https://github.com/aquasecurity/trivy-db/actions/runs/11943968765

I'll experiment with this and consider using it in our GH workflows 👍