feat: support multiple DB repositories for vulnerability and Java DB

[nikpivkin]

Description

Usage:

./trivy image alpine:3.19 --db-repository "ghcr.io/aquasecurity/trivy-db:2","public.ecr.aws/aquasecurity/trivy-db"
./trivy image alpine:3.19 --db-repository ghcr.io/aquasecurity/trivy-db:2 --db-repository public.ecr.aws/aquasecurity/trivy-db

Example:

Downloading an artifact from another repository when receiving error code 429:

./trivy image alpine:3.19 --db-repository "ghcr.io/aquasecurity/trivy-db:2","public.ecr.aws/aquasecurity/trivy-db"
2024-10-01T14:00:39+06:00       INFO    [db] Need to update DB
2024-10-01T14:00:39+06:00       INFO    Downloading vulnerability DB...
2024-10-01T14:00:39+06:00       INFO    Downloading  artifact...        repo="ghcr.io/aquasecurity/trivy-db:2"
2024-10-01T14:00:41+06:00       ERROR   Failed to download artifact     repo="ghcr.io/aquasecurity/trivy-db:2" err="oci download error: failed to fetch the layer: GET https://ghcr.io/v2/aquasecurity/trivy-db/blobs/sha256:23d1b901e7534020d5ac5f238b090ad66dcb78afef7301ed7d8ebe6b974ab5f1: TOOMANYREQUESTS: retry-after: 1.254886ms, allowed: 44000/minute"
2024-10-01T14:00:41+06:00       INFO    Trying to download artifact from other repository...
2024-10-01T14:00:41+06:00       INFO    Downloading  artifact...        repo="public.ecr.aws/aquasecurity/trivy-db:2"
53.85 MiB / 53.85 MiB [--------------------------------------------------------------------------------------] 100.00% 1.45 MiB p/s 37s
2024-10-01T14:01:22+06:00       INFO    Artifact successfully downloaded        repo="public.ecr.aws/aquasecurity/trivy-db:2"

Related issues

• Close feat: support multiple DB repositories for vulnerability and Java DB #7603

Related PRs

• ci: publish trivy-java-db to ECR Public trivy-java-db#37

Checklist

• I've read the guidelines for contributing to this repository.
• I've followed the conventions in the PR title.
• I've added tests that prove my fix is effective or that my feature works.
• I've updated the documentation with the relevant information (if needed).
• I've added usage information (if the PR introduces new options)
• I've included a "before" and "after" example to the description (if the PR is a user interface change).

[simar7]

Should we also have a debug log to say where the DB was downloaded from?

[nikpivkin]

It's logged https://github.com/aquasecurity/trivy/pull/7605/files#diff-5da3684806835246d262230050593f460b12b6c0e3966df174e6061be0e9e575R235

[simar7]

I meant to say log the success of the DB. Something along the lines of "Successfully download from ... " as a debug print. But it's just a nit.

[nikpivkin]

Got it now, I added d75a584

[DmitriyLewen]

Do we need this?
We do same in initOCIArtifact:

trivy/pkg/db/db.go

Lines 202 to 204 in 95df470

	if c.artifact != nil {
	return c.artifact, nil
	}

+

trivy/pkg/db/db.go

Lines 237 to 239 in 95df470

	if err := a.Download(ctx, dst, downloadOpt); err != nil {
	log.Error("Failed to download DB", log.String("repo", repo.String()), log.Err(err))
	if i < len(c.dbRepositories)-1 {

[nikpivkin]

If we already have one artifact initialized with a repository via WithOCIArtifact, there is no point in looping through repositories.

[DmitriyLewen]

Maybe I didn't make it clear.

You have duplicate code.
Can we keep c.artifact != nil check and a.Download function in just one place?

trivy/pkg/db/db.go

Lines 226 to 228 in 95df470

	if c.artifact != nil {
	return c.artifact.Download(ctx, dst, downloadOpt)
	}

trivy/pkg/db/db.go

Lines 202 to 204 in 95df470

	if c.artifact != nil {
	return c.artifact, nil
	}

and

trivy/pkg/db/db.go

Line 227 in 95df470

return c.artifact.Download(ctx, dst, downloadOpt)

trivy/pkg/db/db.go

Line 237 in 95df470

if err := a.Download(ctx, dst, downloadOpt); err != nil {

[nikpivkin]

@DmitriyLewen If the artifact already exists (manually created), then we don't need to create it here and we can just download it instead of trying to load it from possible repositories.

[nikpivkin]

@DmitriyLewen Done b08004a

[simar7]

lgtm, just left one nit comment.

[knqyf263]

I'll review it tomorrow.

[DmitriyLewen]

LGTM

[knqyf263]

We should try the next repository only when the error is 429 or 5xx, like this. Can we extract status code by using transport.Error with errors.As? Temporary may help.

[nikpivkin]

Agreed, I think we can move this error handling when downloading an artifact.

[knqyf263]

All OCI artifacts are always initialized now, even if the primary registry works correctly. It means we waste HTTP calls. What if delaying initialization when needed? Please let me know if I'm missing something.

diff --git a/pkg/db/db.go b/pkg/db/db.go
index 46c2a0584..716dd5d71 100644
--- a/pkg/db/db.go
+++ b/pkg/db/db.go
@@ -42,7 +42,7 @@ var (
 )

 type options struct {
-       artifact       *oci.Artifact
+       artifact       *oci.Artifact // For testing purpose only
        dbRepositories []name.Reference
 }

@@ -199,6 +199,10 @@ func (c *Client) updateDownloadedAt(ctx context.Context, dbDir string) error {
 }

 func (c *Client) initOCIArtifact(repository name.Reference, opt types.RegistryOptions) (*oci.Artifact, error) {
+       if c.artifact != nil {
+               return c.artifact, nil // For unit tests
+       }
+
        art, err := oci.NewArtifact(repository.String(), c.quiet, opt)
        // TODO: NewArtifact never returns an error
        if err != nil {
@@ -218,30 +222,12 @@ func (c *Client) initOCIArtifact(repository name.Reference, opt types.RegistryOp
        return art, nil
 }

-func (c *Client) initArtifacts(opt types.RegistryOptions) ([]*oci.Artifact, error) {
-       if c.artifact != nil {
-               return []*oci.Artifact{c.artifact}, nil
-       }
-
-       artifacts := make([]*oci.Artifact, 0, len(c.dbRepositories))
-
-       for _, repo := range c.dbRepositories {
-               a, err := c.initOCIArtifact(repo, opt)
+func (c *Client) downloadDB(ctx context.Context, opt types.RegistryOptions, dst string) error {
+       for i, repo := range c.dbRepositories {
+               art, err := c.initOCIArtifact(repo, opt)
                if err != nil {
-                       return nil, err
+                       return xerrors.Errorf("failed to initialize OCI artifact: %w", err)
                }
-               artifacts = append(artifacts, a)
-       }
-       return artifacts, nil
-}
-
-func (c *Client) downloadDB(ctx context.Context, opt types.RegistryOptions, dst string) error {
-       arts, err := c.initArtifacts(opt)
-       if err != nil {
-               return err
-       }
-
-       for i, art := range arts {
                log.Info("Downloading vulnerability DB...", log.String("repo", art.Repository()))
                if err := art.Download(ctx, dst, oci.DownloadOption{MediaType: dbMediaType}); err != nil {
                        log.Error("Failed to download DB", log.String("repo", art.Repository()), log.Err(err))

[knqyf263]

Client.artifact is for testing purposes only. We don't need to expand it to a slice. It means it's okay to return the same instance, but we can change it if we really need it.

[nikpivkin]

@knqyf263 Artifact initialization does not cause http requests and never returns an error. https://github.com/aquasecurity/trivy/blob/main/pkg/oci/artifact.go#L60-L70 If necessary, I can do some refactoring in this PR.

[knqyf263]

Hmm. We forgot to fix the error handling. I'll open a PR now.

[nikpivkin]

artifact can be passed through the WithOCIArtifact option, which is public. Can anyone use it?

[knqyf263]

Opened #7615

[knqyf263]

Merged. #7615

[nikpivkin]

artifact can be passed through the WithOCIArtifact option, which is public. Can anyone use it?

@knqyf263

[knqyf263]

@nikpivkin Yes, anybody can use it, and it's actually used in several places.

trivy/pkg/db/db_test.go

Line 175 in fc6b3a7

client := db.NewClient(dbDir, true, db.WithOCIArtifact(art))

trivy/pkg/rpc/server/listen_test.go

Line 97 in 4c6e8ca

client := db.NewClient(dbDir, true, db.WithOCIArtifact(art))

[knqyf263]

We're unsure how the free tier in ECR Public works, so we should probably avoid adding ECR for now. Instead, we'll document how to use ECR Public in another PR.

[knqyf263]

I want to improve logging a bit. I'll open another PR soon.

[beltran-rubo]

The 2nd default repository is wrong, it should be "public.ecr.aws/aquasecurity/trivy-db:2"

[stewartcampbell]

If I add a typo to the first URL, it fails to try the second. Is that expected?

trivy --version
Version: 0.56.1

trivy image alpine:3.19 --db-repository "ghcrx.io/aquasecuridty/trivy-db:2","public.ecr.aws/aquasecurity/trivy-db:2" --debug
2024-10-08T09:07:29+01:00       DEBUG   No plugins loaded
2024-10-08T09:07:29+01:00       DEBUG   Default config file "file_path=trivy.yaml" not found, using built in values
2024-10-08T09:07:29+01:00       DEBUG   Cache dir       dir="/home/stewart/.cache/trivy"
2024-10-08T09:07:29+01:00       DEBUG   Cache dir       dir="/home/stewart/.cache/trivy"
2024-10-08T09:07:30+01:00       DEBUG   Parsed severities       severities=[UNKNOWN LOW MEDIUM HIGH CRITICAL]
2024-10-08T09:07:30+01:00       DEBUG   Ignore statuses statuses=[]
2024-10-08T09:07:30+01:00       DEBUG   [vulndb] There is no valid metadata file        err="unable to open a file: open /home/stewart/.cache/trivy/db/metadata.json: no such file or directory"
2024-10-08T09:07:30+01:00       INFO    [vulndb] Need to update DB
2024-10-08T09:07:30+01:00       DEBUG   [vulndb] No metadata file
2024-10-08T09:07:30+01:00       INFO    [vulndb] Downloading vulnerability DB...
2024-10-08T09:07:30+01:00       INFO    [vulndb] Downloading artifact...        repo="ghcrx.io/aquasecuridty/trivy-db:2"
2024-10-08T09:07:30+01:00       FATAL   Fatal error
  - init error:
    github.com/aquasecurity/trivy/pkg/commands/artifact.Run
        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:367
  - DB error:
    github.com/aquasecurity/trivy/pkg/commands/artifact.NewRunner
        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:119
  - failed to download vulnerability DB:
    github.com/aquasecurity/trivy/pkg/commands/operation.DownloadDB
        /home/runner/work/trivy/trivy/pkg/commands/operation/operation.go:40
  - OCI artifact error:
    github.com/aquasecurity/trivy/pkg/db.(*Client).Download
        /home/runner/work/trivy/trivy/pkg/db/db.go:158
  - failed to download vulnerability DB:
    github.com/aquasecurity/trivy/pkg/db.(*Client).downloadDB
        /home/runner/work/trivy/trivy/pkg/db/db.go:207
  - failed to download artifact from ghcrx.io/aquasecuridty/trivy-db:2:
    github.com/aquasecurity/trivy/pkg/oci.Artifacts.Download
        /home/runner/work/trivy/trivy/pkg/oci/artifact.go:236
  - OCI repository error:
    github.com/aquasecurity/trivy/pkg/oci.(*Artifact).populate
        /home/runner/work/trivy/trivy/pkg/oci/artifact.go:96
  - 1 error occurred:
        * Get "https://ghcrx.io/v2/": dial tcp: lookup ghcrx.io on 10.255.255.254:53: no such host

trivy image alpine:3.19 --db-repository ghcrx.io/aquasecurity/trivy-db:2 --db-repository public.ecr.aws/aquasecurity/trivy-db:2 --debug
2024-10-08T09:10:02+01:00       DEBUG   No plugins loaded
2024-10-08T09:10:02+01:00       DEBUG   Default config file "file_path=trivy.yaml" not found, using built in values
2024-10-08T09:10:02+01:00       DEBUG   Cache dir       dir="/home/stewart/.cache/trivy"
2024-10-08T09:10:02+01:00       DEBUG   Cache dir       dir="/home/stewart/.cache/trivy"
2024-10-08T09:10:02+01:00       DEBUG   Parsed severities       severities=[UNKNOWN LOW MEDIUM HIGH CRITICAL]
2024-10-08T09:10:02+01:00       DEBUG   Ignore statuses statuses=[]
2024-10-08T09:10:02+01:00       DEBUG   [vulndb] There is no valid metadata file        err="unable to open a file: open /home/stewart/.cache/trivy/db/metadata.json: no such file or directory"
2024-10-08T09:10:02+01:00       INFO    [vulndb] Need to update DB
2024-10-08T09:10:02+01:00       DEBUG   [vulndb] No metadata file
2024-10-08T09:10:02+01:00       INFO    [vulndb] Downloading vulnerability DB...
2024-10-08T09:10:02+01:00       INFO    [vulndb] Downloading artifact...        repo="ghcrx.io/aquasecurity/trivy-db:2"
2024-10-08T09:10:02+01:00       FATAL   Fatal error
  - init error:
    github.com/aquasecurity/trivy/pkg/commands/artifact.Run
        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:367
  - DB error:
    github.com/aquasecurity/trivy/pkg/commands/artifact.NewRunner
        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:119
  - failed to download vulnerability DB:
    github.com/aquasecurity/trivy/pkg/commands/operation.DownloadDB
        /home/runner/work/trivy/trivy/pkg/commands/operation/operation.go:40
  - OCI artifact error:
    github.com/aquasecurity/trivy/pkg/db.(*Client).Download
        /home/runner/work/trivy/trivy/pkg/db/db.go:158
  - failed to download vulnerability DB:
    github.com/aquasecurity/trivy/pkg/db.(*Client).downloadDB
        /home/runner/work/trivy/trivy/pkg/db/db.go:207
  - failed to download artifact from ghcrx.io/aquasecurity/trivy-db:2:
    github.com/aquasecurity/trivy/pkg/oci.Artifacts.Download
        /home/runner/work/trivy/trivy/pkg/oci/artifact.go:236
  - OCI repository error:
    github.com/aquasecurity/trivy/pkg/oci.(*Artifact).populate
        /home/runner/work/trivy/trivy/pkg/oci/artifact.go:96
  - 1 error occurred:
        * Get "https://ghcrx.io/v2/": dial tcp: lookup ghcrx.io on 10.255.255.254:53: no such host

[DmitriyLewen]

Hello @stewartcampbell
This is expected behavior.
We only skip the repository for 429 and 5xx errors - #7605 (comment)

[stewartcampbell]

Thanks @DmitriyLewen