Ratelimiting error when downloading vulnerability db from ghcr.io

[HenrikDK]

Hi, we're using trivy to scan our containers, lately we've been seeing an increase number of rate-limiting errors when trivy is downloading the vulnerability database.

"2024-09-18T10:40:44Z FATAL Fatal error init error: DB error: failed to download vulnerability DB: database download error: oci download error: failed to fetch the layer: GET https://ghcr.io/v2/aquasecurity/trivy-db/blobs/sha256:11c57f2012b2ac112256f94aa404e1feb7e1b7a5787598946b87149115cdb43d: TOOMANYREQUESTS: retry-after: 129.163µs, allowed: 44000/minute"

My guess is this is a global ratelimit as i can't imagine our low number of devs are causing 700+ requests a second.

I have in the meantime discovered that these scans are only used for SBOM generation on our end so we don't need to download the vulnerability database everytime, but i though this issue should be raised as i can't imagine we are the only ones seeing these errors.

[simar7]

Thanks for the report, we will look into it.

[benglewis]

I also saw this right now :/ Any ideas why?

[billhammond-dev]

I believe this is currently causing problems with anyone using the trivy action. We have had to turn it off on some workflows. I'm not sure what the long term solution might be - if GH cannot increase the global rate limit for the artifact pull then maybe it needs to be in a public AWS S3 bucket or something similar?

[billhammond-dev]

Also see https://github.com/orgs/community/discussions/139074

[billhammond-dev]

From My PR above, a workaround suggested by someone else:

- uses: aquasecurity/trivy-action@0.24.0
  with:
    ...
  env:
    TRIVY_DB_REPOSITORY: <something else than ghcr.io>
    TRIVY_JAVA_DB_REPOSITORY: <something else than ghcr.io>```

[nelsonleblanc-rl]

Does anyone know how to get trivy-action to auth with a privately hosted trivy-db repo? I can get it working fine with normal trivy on local, but trivy-action does not work with either docker/login-action or the usual echo $GITHUB_TOKEN | docker login ghcr.io -u USERNAME --password-stdin

2024-09-20T16:39:01Z FATAL Fatal error init error: DB error: failed to download vulnerability DB: database download error: OCI repository error: 1 error occurred: * GET https://ghcr.io/token?scope=repository%3Aprivate-github-org%2Ftrivy-db%3Apull&service=ghcr.io: UNAUTHORIZED: authentication required

[billhammond-dev]

I was able to get it to work with ECR only using an OIDC login via the configure-aws-credentials action used right before the trivy action. It is not using docker to pull the artifact as it is not a docker image.

[9838183063]

I am poor student

[srenatus]

I have no long-term tests yet, but from my understanding of GH's rate limiting, just providing a token of any sort will give you higher quotas? If that's the case, the following should help:

      - name: Run Trivy scan on image
        uses: aquasecurity/trivy-action@0.24.0
        with:
          [... your config ...]
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

[baksetercx]

I have no long-term tests yet, but from my understanding of GH's rate limiting, just providing a token of any sort will give you higher quotas? If that's the case, the following should help:

      - name: Run Trivy scan on image
        uses: aquasecurity/trivy-action@0.24.0
        with:
          [... your config ...]
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

I've tried logging in to GHCR via docker/login-action before running Trivy CLI (not action), and I am still getting lots of 429 errors.

[nnellanspdl]

From My PR above, a workaround suggested by someone else:

- uses: aquasecurity/trivy-action@0.24.0
  with:
    ...
  env:
    TRIVY_DB_REPOSITORY: <something else than ghcr.io>
    TRIVY_JAVA_DB_REPOSITORY: <something else than ghcr.io>```

So, if I understand this correctly:

I, as the consumer of this action, must download copies of these DBs and store them on my own registry. Then, I must pass environment variables to the action which point at my copies of the DBs. Is that correct?

How often are these DBs updated?

[ybelMekk]

@nnellanspdl think its at 00:00 every day? but im not sure.

But anyway this workaround is a hustle to host them self if u need to update them every day

[NicholasFiorentini]

I have no long-term tests yet, but from my understanding of GH's rate limiting, just providing a token of any sort will give you higher quotas? If that's the case, the following should help:

      - name: Run Trivy scan on image
        uses: aquasecurity/trivy-action@0.24.0
        with:
          [... your config ...]
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

I've tried logging in to GHCR via docker/login-action before running Trivy CLI (not action), and I am still getting lots of 429 errors.

Same for me, it doesn't seem to have significant effects.

[NicholasFiorentini]

I have no long-term tests yet, but from my understanding of GH's rate limiting, just providing a token of any sort will give you higher quotas? If that's the case, the following should help:

      - name: Run Trivy scan on image
        uses: aquasecurity/trivy-action@0.24.0
        with:
          [... your config ...]
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

I'm trying with:

env:
    ACTIONS_RUNTIME_TOKEN: ${{ secrets.GITHUB_TOKEN }}

I spawned multiple parallel ci/cd actions, and this seems more reliable.

[BRONSOLO]

If anyone is going the route of uploading the Trivy DB to their own registry, I've had success using https://github.com/oras-project/setup-oras

Something like:

  vendor-trivy-db:
    runs-on: ubuntu-latest
    steps:
      - name: Vendor latest trivy db
        uses: oras-project/setup-oras@v1
      - run: |
          oras pull ghcr.io/aquasecurity/trivy-db:2
          oras login -u ${{ secrets.REGISTRY_USERNAME }} -p ${{ secrets.REGISTRY_TOKEN }} YOUR_REGISTRY
          oras push YOUR_REGISTRY \
            db.tar.gz:application/vnd.aquasec.trivy.db.layer.v1.tar+gzip \
            --artifact-type application/vnd.aquasec.trivy.config.v1+json

[eugentius]

I setup AWS ECR pull-throuhg cache for trivy-db and trivy-java-db , modified action:

      - name: Run Trivy vulnerability scanner
        uses: aquasecurity/trivy-action@0.24.0
        with:
          image-ref: ${{ env.DOCKER_IMAGE_TO_SCAN }}
          format: 'table'
          exit-code: '1' 
          ignore-unfixed: true
          vuln-type: 'os,library'
          severity: 'CRITICAL,HIGH'
        env:
          TRIVY_DB_REPOSITORY: <ECR_ID>.dkr.ecr.us-east-1.amazonaws.com/github/ghcr.io/aquasecurity/trivy-db
          TRIVY_JAVA_DB_REPOSITORY: <ECR_ID>.dkr.ecr.us-east-1.amazonaws.com/github/ghcr.io/aquasecurity/trivy-java-db
          TRIVY_DEBUG: true

but pulling of trivy-db fails with:

2024-09-23T16:16:12Z	INFO	Downloading DB...	repository="<ECR_ID>.dkr.ecr.us-east-1.amazonaws.com/github/ghcr.io/aquasecurity/trivy-db"
2024-09-23T16:16:12Z	DEBUG	No metadata file
2024-09-23T16:16:17Z	DEBUG	Credential error	err="failed to get authorization token: operation error ECR: GetAuthorizationToken, get identity: get credentials: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, canceled, context deadline exceeded"
2024-09-23T16:16:17Z	FATAL	Fatal error	init error: DB error: failed to download vulnerability DB: database download error: OCI repository error: 1 error occurred:
	* GET https://<ECR_ID>.dkr.ecr.us-east-1.amazonaws.com/v2/github/ghcr.io/aquasecurity/trivy-db/manifests/2: unexpected status code 401 Unauthorized: Not Authorized

Docker is logged-in.
If I run trivy binary locally or on runner, it works fine:

runner@runner-set-xs-djfqb-0:/tmp$ export TRIVY_DB_REPOSITORY=<ECR_ID>.dkr.ecr.us-east-1.amazonaws.com/github/ghcr.io/aquasecurity/trivy-db
runner@runner-set-xs-djfqb-0:/tmp$ export TRIVY_JAVA_DB_REPOSITORY=<ECR_ID>.dkr.ecr.us-east-1.amazonaws.com/github/ghcr.io/aquasecurity/trivy-java-db
runner@runner-set-xs-djfqb-0:/tmp$ trivy image    --format table --exit-code  1 --ignore-unfixed --vuln-type  os,library --severity  CRITICAL,HIGH  <ECR_ID>.dkr.ecr.us-east-1.amazonaws.com/my-awesome-app:1.23.0
2024-09-23T16:35:04Z	WARN	'--vuln-type' is deprecated. Use '--pkg-types' instead.
2024-09-23T16:35:04Z	INFO	Adding schema version to the DB repository for backward compatibility	repository="<ECR_ID>.dkr.ecr.us-east-1.amazonaws.com/github/ghcr.io/aquasecurity/trivy-db:2"
2024-09-23T16:35:04Z	INFO	Adding schema version to the Java DB repository for backward compatibility	repository="<ECR_ID>.dkr.ecr.us-east-1.amazonaws.com/github/ghcr.io/aquasecurity/trivy-java-db:1"
2024-09-23T16:35:04Z	INFO	[db] Need to update DB
2024-09-23T16:35:04Z	INFO	[db] Downloading DB...	repository="<ECR_ID>.dkr.ecr.us-east-1.amazonaws.com/github/ghcr.io/aquasecurity/trivy-db:2"
53.56 MiB / 53.56 MiB [------------------------------------------------------------------------------------------------------ 

Has somebody tried to pull trivy-db from AWS ECR using action?

[billhammond-dev]

Yes so you can pull from ECR pull through but only if you do an OIDC set-aws-credentials action first before the trivy action. Im not sure why yet that you cannot use anything but OIDC, or at least I can't seem to get regular role assumption to work. Docker login doesnt help you as the container doesnt try to pull the DB using docker commands.

If you try a docker pull you will get the unsupported media type error as the above post, as the artifact isnt an 'image'

[nbenmoody-tesouro]

Ah, thanks. I was logged in under the incorrect account when I posted originally. That's what I was wondering, @billhammond-dev !

[nbenmoody-tesouro]

This was my error, for anyone else who runs into it:

latest: Pulling from github/ghcr.io/aquasecurity/trivy-db
unsupported media type application/vnd.aquasec.trivy.config.v1+json

[nnellanspdl]

@nnellanspdl think its at 00:00 every day? but im not sure.

But anyway this workaround is a hustle to host them self if u need to update them every day

Thanks. Yes, this is a lot to ask of consumers of your action.

[Nava-JoshLong]

I'm guessing it would be too much work to update the logic for pulling the file to allow passing it the file directly? We could setup a workflow to pull and stash the image every X hours, and then in the workflow that uses the image, we pull the file from the stash to use. It'd lower the amount of hits by users, and we wouldn't need to host it in AWS and pay

[simar7]

ACTIONS_RUNTIME_TOKEN

@NicholasFiorentini that's interesting, would you mind creating a PR to document this in the repo? If possible, could you also reference where this environment variable is documented?

[jpalomaki]

FWIW, here's a sample snippet for using AWS ECR pull through cache repositories using OIDC for AWS auth.

Pull through cache ECR repositories (for hosting the cached trivy DB artifacts) must be configured prior to running this workflow, see documentation.

- name: Setup AWS credentials
  uses: aws-actions/configure-aws-credentials@v4
  with:
    aws-region: ...
    role-to-assume: <role, assumable through OIDC, that can pull from the cache ECR repositories>

- id: ecr-login
  name: Login to ECR
  uses: aws-actions/amazon-ecr-login@v2

...

- name: Run trivy scan
  uses: aquasecurity/trivy-action@0.24.0
  with:
    ...
  env:
    TRIVY_DB_REPOSITORY: ${{ steps.ecr-login.outputs.registry }}/github/aquasecurity/trivy-db:2
    TRIVY_JAVA_DB_REPOSITORY: ${{ steps.ecr-login.outputs.registry }}/github/aquasecurity/trivy-java-db:1

Per AWS documentation:

When a cached image is pulled through the Amazon ECR private registry URI, Amazon ECR checks the upstream repository at least once every 24 hours to verify whether the cached image is the latest version. If there is a newer image in the upstream registry, Amazon ECR attempts to update the cached image. This timer is based off the last pull of the cached image.

[sp3c1]

does TRIVY_CHECKS_BUNDLE_REPOSITORY work like TRIVY_DB_REPOSITORY accepting comma separated list after 0.28.0 ?release

[adambirds]

Surely it can t be hard @simar7 to have automatic fallback to other registrys as a feature! We shouldn't be having to edit our actions to add this config otherwise its a breaking change! How hard could it be to have it mirrored in multiple locations and then just have a list of these it trys first inside the actual action!

[rv0lt]

This is an open source project we use for free, not a paid proprietary software you are paying to use, so there is no point in being offensive.

Go is not my strength, but I have made a small PR with the changes (I believe) needed to add this other AWS mirror Registry as part of the default registries along with GHRC. Hopefully it can pass.

aquasecurity/trivy#7697

[mvdkleijn]

This will probably get me down voted, but (as a long time open source dev myself) I don't think what @adambirds wrote is offensive at all.

I just read a fairly understandable expression of frustration from a fellow human about the fact that the vulnerability DB availability is unstable.

Anyway, having said that and to come back to the heart of the matter and to summarize:

• The DB is not available from Aquasec's own servers, only Github;
• The DB is mirrored to ECR (though unclear to me if that's an official mirror);
• The DB should now be available again on DockerHub (didn't check);
• The problem is not specific to trivy-action, rather to trivy proper;

While I agree this is an open source project, it is also an open source project backed by Aquasec themselves and used as a backbone for their commercial products. Or did I get that wrong?

Anyway, all of the above sounds like workarounds to me and not final solutions. While it is perfectly feasible for companies (and even larger open source projects) to setup caching proxies for the DB themselves, small open source projects / devs generally don't have that luxury.

edit: so the most practical solution so far is to use a Github workflow to cache an updated DB once a day.. though that's far from ideal.

[adambirds]

Couldn't have put it better @mvdkleijn! As you say small developers don't have the luxury of setting up proxies or extra caches or keeping their own mirror! As @nvuillam also said the other week, having people host their own mirrors becomes a vulnerability in itself!

My comment was just frustration because it shouldn't be that hard to have more official mirrors and a list built into the code that it loops round and tries which would solve the issue! And as far as my research goes there is apparently a couple of mirrors but no official list of them anywhere and we should by required to edit all our action to include an argument list of mirrors ourselves because they could become out of date and it is a breaking change which based off the version change it shouldn't be as they only did a minor version number change.

At the moment it's still broken, causing hundreds probably of failures for people using it wasting so many hours and minutes of actions!