feat: Update registry fallbacks #7679

simar7 · 2024-10-08T23:18:25Z

Description

Adds public mirror.gcr.io registry as a fallback for both vuln-db and java-db.

Issues

Close Mitigate rate limit issues by utilizing mirror.gcr.io as default DB repository #7938

Related PRs

feat: support multiple DB repositories for vulnerability and Java DB #7605

Checklist

I've read the guidelines for contributing to this repository.
I've followed the conventions in the PR title.
I've added tests that prove my fix is effective or that my feature works.
I've updated the documentation with the relevant information (if needed).
I've added usage information (if the PR introduces new options)
I've included a "before" and "after" example to the description (if the PR is a user interface change).

knqyf263 · 2024-10-09T03:24:24Z

We should document how to authenticate with ECR. Otherwise, ECR Public has only 500 GB free tier for anonymous users. Also, it's calculated by IP address. I suppose we'll run out of the free tier on CI service as it shares IP addresses.

DmitriyLewen · 2024-10-09T03:49:24Z

I'm worried about adding ecr as the default registry.
I think most users will ignore the authentication recommendation, and some users just don't read the docs.

Instead of adding ecr - I suggest adding warning if downloading db from ghcr returns error.
We will show information about using ecr (with authentication information) + link to the docs.

nvuillam · 2024-10-09T21:31:01Z

We read the docs but... having to authenticate to aws would be a breaking change 🥹

DmitriyLewen · 2024-10-10T04:30:42Z

hello @nvuillam
Authentication is necessary to preserve aws limits for unauthenticated users as much as possible.
But we don't require it. We just ask users to do it if possible - it will help avoid rate limiting errors like ghcr.io.

simar7 · 2024-10-10T06:10:58Z

We should document how to authenticate with ECR. Otherwise, ECR Public has only 500 GB free tier for anonymous users. Also, it's calculated by IP address. I suppose we'll run out of the free tier on CI service as it shares IP addresses.

Yes - I wanted to add more info on this topic. Added fa03189

DmitriyLewen · 2024-10-10T07:48:01Z

docs/docs/configuration/db.md

+Please see more info on how to authenticate with ECR [auth-ecr] and GHCR [auth-ghcr].
+
+#### Caching DBs
+Trivy DB and Trivy Java DB are published every 24 hours. If you are running Trivy scans more often than this, you can significantly benefit from caching the DBs on each run and updating them as needed.


24 hours is incorrect.
We publish trivy-db every 6 hours, but update interval is 24 hours.
for trivy-java-db: publish every 24 hours, update interval is 3 days.

knqyf263 · 2024-10-10T08:06:07Z

We will show information about using ecr (with authentication information) + link to the docs.

I'm also leaning toward this approach. I'm afraid that users downloading container images from ECR Public (e.g., Kubernetes cluster) may suddenly reach rate limits due to Trivy DB and be unable to deploy, etc.

Of course, it is best if users can continue to use DBs as before without being aware of it, but the implicit use of ECR Public may have other negative effects. We should be careful about the use of ECR Public. If users explicitly set this up, they will understand these trade-offs.

simar7 · 2024-10-17T04:55:24Z

We will show information about using ecr (with authentication information) + link to the docs.

I'm also leaning toward this approach. I'm afraid that users downloading container images from ECR Public (e.g., Kubernetes cluster) may suddenly reach rate limits due to Trivy DB and be unable to deploy, etc.

Yes but the same can be said for GHCR today.

Of course, it is best if users can continue to use DBs as before without being aware of it, but the implicit use of ECR Public may have other negative effects. We should be careful about the use of ECR Public. If users explicitly set this up, they will understand these trade-offs.

I added some docs here.

Ultimately we can reach rate limits even with authenticated use of container registries. The fallback to using a secondary registry only mitigates the risk up to an extent.

knqyf263 · 2024-10-17T05:46:42Z

Yes but the same can be said for GHCR today.

The rate limits in GHCR are on the organization, not the user, unlike ECR Public. In other words, heavy use of GHCR may make aquasecurity images unavailable, but not other images. Users may not be able to use Trivy, but they will be able to pull any other images except aquasecurity.

It should also be improved once this PR is merged: aquasecurity/trivy#7679

goatwu1993 · 2024-10-22T07:12:56Z

checks bundle also needs ecr.

        env:
          TRIVY_DB_REPOSITORY: public.ecr.aws/aquasecurity/trivy-db
          TRIVY_JAVA_DB_REPOSITORY: public.ecr.aws/aquasecurity/trivy-java-db
          TRIVY_CHECKS_BUNDLE_REPOSITORY: public.ecr.aws/aquasecurity/trivy-checks

bgoareguer · 2024-10-24T09:37:11Z

The trivy.dbRepository parameter in the helm/trivy/values.yaml file also needs to be updated.

afdesk · 2024-10-28T06:21:26Z

The trivy.dbRepository parameter in the helm/trivy/values.yaml file also needs to be updated.

thanks for the good idea.
it should be add after the PR is merged and released.

afdesk · 2024-10-28T06:27:07Z

@simar7
Now trivy-db pushes the database to Docker Hub: aquasecurity/trivy-db#448
maybe it makes sense to add Docker Hub as a default repository too?
just idea. thanks!

jeanmarcmoeckel · 2024-11-13T07:24:41Z

Any updates here?

knqyf263 · 2024-11-14T19:36:04Z

mirror.gcr.io seems the best for our use case. Please let us know if anyone knows about rate limits on that.
#7538 (comment)

benji78 · 2024-11-15T09:08:08Z

I see 8 incorrect mentions of public.ecr.aws/aquasecurity/trivy-db:1 which should be trivy-java-db !

DmitriyLewen · 2024-11-15T09:11:17Z

pkg/javadb/client.go

@@ -29,6 +29,9 @@ const (
 var (
 	// GitHub Container Registry
 	DefaultGHCRRepository = fmt.Sprintf("%s:%d", "ghcr.io/aquasecurity/trivy-java-db", SchemaVersion)
+
+	// AWS ECR
+	DefaultECRRepository = fmt.Sprintf("%s:%d", "public.ecr.aws/aquasecurity/trivy-db", SchemaVersion)


Suggested change

DefaultECRRepository = fmt.Sprintf("%s:%d", "public.ecr.aws/aquasecurity/trivy-db", SchemaVersion)

DefaultECRRepository = fmt.Sprintf("%s:%d", "public.ecr.aws/aquasecurity/trivy-java-db", SchemaVersion)

Thanks @benji78 !

knqyf263 · 2024-11-18T07:32:36Z

pkg/remote/remote.go

-	token := registry.GetToken(ctx, domain, option)
-	if !lo.IsEmpty(token) {
-		opts = append(opts, remote.WithAuth(&token))
+	// The default trivy-db and trivy-java-db repositories are public.


If I understand correctly, there is no chance that mirror.gcr.io requires authentication. We should skip mirror.gcr.io here.

trivy/pkg/fanal/image/registry/google/google.go

Line 32 in 94791f8

if domain != gcrURLDomain && !strings.HasSuffix(domain, gcrURLSuffix) && !strings.HasSuffix(domain, garURLSuffix) {

yeah, you're probably right, I didn't think about it
Changed in 99e96bb

knqyf263 · 2024-11-18T08:08:36Z

docs/docs/configuration/db.md

+
+#### Authenticated use of Registries
+By authenticating with the registries that Trivy hosts its DBs on can significantly increase the limit for users. For Amazon ECR, the details for rate limits can be found [ecr-limits].
+For GitHub GHCR, the rate limits can be found [ghcr-limits]


It's not correct. GHCR doesn't respect the GitHub token now.

But some users wrote that GH token helps them.
Do you think that for GHCR it makes no difference at all whether the user is authorized or not when downloading the image?

But some users wrote that GH token helps them.

Trivy has a bug that doesn't reflect GITHUB_TOKEN now, so I'm sure those users luckily didn't see rate limit errors.

I saw the same rate limit error when I set the token, so I believe it has no effect on GHCR. If we confirm it helps, we can add it back.

Got it!
Removed info about GHCR in c18a511

knqyf263 · 2024-11-18T08:09:55Z

pkg/javadb/client.go

@@ -29,6 +29,9 @@ const (
 var (
 	// GitHub Container Registry
 	DefaultGHCRRepository = fmt.Sprintf("%s:%d", "ghcr.io/aquasecurity/trivy-java-db", SchemaVersion)
+
+	// GHC mirrors


nit

Suggested change

// GHC mirrors

// GCR mirrors

thanks! fixed in 272b1a0

benji78 · 2024-11-18T08:46:52Z

docs/docs/configuration/db.md

+#### Authenticated use of Registries
+By authenticating with the registries that Trivy hosts its DBs on can significantly increase the limit for users. For Amazon ECR, the details for rate limits can be found [ecr-limits].
+
+Please see more info on how to authenticate with ECR [auth-ecr].


Is Amazon ECR still relevant here if it has been replaced by GCR?

We still push trivy-db and trivy-java-db into ECR.
So this is still relevant for users using ECR.

benji78 · 2024-11-18T08:56:58Z

docs/docs/configuration/db.md

+
+#### Caching DBs
+Trivy DB and Trivy Java DB are published every 6 hours and 24 hours, respectively. If you are running Trivy scans more often than this, you can significantly benefit from caching the DBs on each run and updating them as needed.
+Once example of this can be seen in Trivy Action, where with caching multiple CI invocations can be performed with a single download of the DBs. More on info Trivy Action caching can be found [trivy-action-cache].


There is an extra c in Once which should be One example of this …

Thanks! We will fix that in another PR.

@benji78 - #7942

DmitriyLewen · 2024-11-18T09:54:59Z

@aqua-bot backport release/v0.57

Co-authored-by: DmitriyLewen <dmitriy.lewen@smartforce.io>

aqua-bot · 2024-11-18T09:55:45Z

Backport PR created: #7944

simar7 added 2 commits October 8, 2024 17:11

feat(db): Add public ECR registry as a fallback

6e1b679

update docs

74c53f8

simar7 requested a review from nikpivkin October 8, 2024 23:18

simar7 self-assigned this Oct 8, 2024

simar7 mentioned this pull request Oct 9, 2024

feat(db): Allow by default the mirror AWS registry as well as GHCR #7697

Closed

6 tasks

add docs

fa03189

simar7 force-pushed the update-registry-fallbacks branch from c4d3c8d to fa03189 Compare October 10, 2024 06:15

simar7 marked this pull request as ready for review October 10, 2024 06:16

simar7 requested a review from knqyf263 as a code owner October 10, 2024 06:16

simar7 requested a review from DmitriyLewen October 10, 2024 06:16

DmitriyLewen reviewed Oct 10, 2024

View reviewed changes

qligier added a commit to ahdis/matchbox that referenced this pull request Oct 21, 2024

Make trivy scanning more reliable

3626580

It should also be improved once this PR is merged: aquasecurity/trivy#7679

knqyf263 mentioned this pull request Nov 5, 2024

fix: Add additional mirror repositories as defaults #7864

Closed

3 tasks

mkurapov mentioned this pull request Nov 6, 2024

chore: fix vulnerabilities interledger/rafiki#3087

Merged

6 tasks

P4sca1 mentioned this pull request Nov 8, 2024

CI complains hitting rate limit on trivy scan fkie-cad/Logprep#700

Closed

DmitriyLewen reviewed Nov 15, 2024

View reviewed changes

DmitriyLewen added 5 commits November 15, 2024 18:03

Merge branch 'master' into 'update-registry-fallbacks'

c6c9bc2

refactor: use mirror.gcr.io instead of ECR

ed69c0e

mage docs:generate

8b1b973

docs(db): update publish time for trivy-db and trivy-java-db

dcb8248

fix: don't check credential of mirror.gcr.io/aquasec repos for DBs

a8807ac

benji78 mentioned this pull request Nov 16, 2024

doc: fix incorrect default value of some parameters in helm README goharbor/harbor-scanner-trivy#6

Open

chore: add todo

fb09465

knqyf263 reviewed Nov 18, 2024

View reviewed changes

DmitriyLewen added 2 commits November 18, 2024 13:43

refactor: skip auth for mirror.gcr.io domain

99e96bb

docs: add comment about skipping mirror.gcr.io

1c967e9

knqyf263 reviewed Nov 18, 2024

View reviewed changes

DmitriyLewen added 2 commits November 18, 2024 14:19

chore: fix typo in comment

272b1a0

docs: remove info about GitHub

c18a511

knqyf263 approved these changes Nov 18, 2024

View reviewed changes

knqyf263 enabled auto-merge November 18, 2024 08:30

knqyf263 added this pull request to the merge queue Nov 18, 2024

benji78 suggested changes Nov 18, 2024

View reviewed changes

Merged via the queue into aquasecurity:main with commit 5ba9a83 Nov 18, 2024
17 checks passed

aqua-bot mentioned this pull request Nov 18, 2024

release: v0.58.0 [main] #7874

Merged

DmitriyLewen mentioned this pull request Nov 18, 2024

Mitigate rate limit issues by utilizing mirror.gcr.io as default DB repository #7938

Closed

github-actions bot pushed a commit that referenced this pull request Nov 18, 2024

feat: Update registry fallbacks (#7679)

ea7a880

Co-authored-by: DmitriyLewen <dmitriy.lewen@smartforce.io>

aqua-bot mentioned this pull request Nov 18, 2024

feat: Update registry fallbacks [backport: release/v0.57] #7944

Merged

jinja2 mentioned this pull request Nov 18, 2024

[chore] use newer trivy version with registry fallback feature signalfx/splunk-otel-collector#5622

Merged

stgdns mentioned this pull request Nov 20, 2024

Add support for multiple vulnerability database repositories goharbor/harbor-scanner-trivy#7

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Update registry fallbacks #7679

feat: Update registry fallbacks #7679

simar7 commented Oct 8, 2024 •

edited by DmitriyLewen

Loading

knqyf263 commented Oct 9, 2024

DmitriyLewen commented Oct 9, 2024

nvuillam commented Oct 9, 2024

DmitriyLewen commented Oct 10, 2024

simar7 commented Oct 10, 2024 •

edited

Loading

DmitriyLewen Oct 10, 2024

knqyf263 commented Oct 10, 2024

simar7 commented Oct 17, 2024

knqyf263 commented Oct 17, 2024

goatwu1993 commented Oct 22, 2024

bgoareguer commented Oct 24, 2024

afdesk commented Oct 28, 2024 •

edited

Loading

afdesk commented Oct 28, 2024

jeanmarcmoeckel commented Nov 13, 2024

knqyf263 commented Nov 14, 2024 •

edited

Loading

benji78 commented Nov 15, 2024

DmitriyLewen Nov 15, 2024

DmitriyLewen Nov 15, 2024

knqyf263 Nov 18, 2024

DmitriyLewen Nov 18, 2024

knqyf263 Nov 18, 2024

DmitriyLewen Nov 18, 2024

knqyf263 Nov 18, 2024

DmitriyLewen Nov 18, 2024

knqyf263 Nov 18, 2024

DmitriyLewen Nov 18, 2024

benji78 Nov 18, 2024

DmitriyLewen Nov 18, 2024

benji78 Nov 18, 2024

DmitriyLewen Nov 18, 2024

DmitriyLewen Nov 18, 2024

DmitriyLewen commented Nov 18, 2024

aqua-bot commented Nov 18, 2024

	DefaultECRRepository = fmt.Sprintf("%s:%d", "public.ecr.aws/aquasecurity/trivy-db", SchemaVersion)
	DefaultECRRepository = fmt.Sprintf("%s:%d", "public.ecr.aws/aquasecurity/trivy-java-db", SchemaVersion)

feat: Update registry fallbacks #7679

feat: Update registry fallbacks #7679

Conversation

simar7 commented Oct 8, 2024 • edited by DmitriyLewen Loading

Description

Issues

Related PRs

Checklist

knqyf263 commented Oct 9, 2024

DmitriyLewen commented Oct 9, 2024

nvuillam commented Oct 9, 2024

DmitriyLewen commented Oct 10, 2024

simar7 commented Oct 10, 2024 • edited Loading

Choose a reason for hiding this comment

knqyf263 commented Oct 10, 2024

simar7 commented Oct 17, 2024

knqyf263 commented Oct 17, 2024

goatwu1993 commented Oct 22, 2024

bgoareguer commented Oct 24, 2024

afdesk commented Oct 28, 2024 • edited Loading

afdesk commented Oct 28, 2024

jeanmarcmoeckel commented Nov 13, 2024

knqyf263 commented Nov 14, 2024 • edited Loading

benji78 commented Nov 15, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DmitriyLewen commented Nov 18, 2024

aqua-bot commented Nov 18, 2024

simar7 commented Oct 8, 2024 •

edited by DmitriyLewen

Loading

simar7 commented Oct 10, 2024 •

edited

Loading

afdesk commented Oct 28, 2024 •

edited

Loading

knqyf263 commented Nov 14, 2024 •

edited

Loading