Refactor source API

[wagoodman]

Refactors the single uber-object of source.Source into multiple single responsibility objects (e.g. source.DirectorySource, source.StereoscopeImageSource, etc) that all fall under the same (new) source.Source abstraction:

type Source interface {
	artifact.Identifiable
	FileResolver(Scope) (file.Resolver, error)
	Describe() Description
	io.Closer
}

There are two ways to create a source object:

1. directly
2. via detection (considering scheme + a content analysis of the reference if it's a local file)

The first is new with this PR:

src, err := NewFromDirectory(
	DirectoryConfig{
		Path: "/somewhere/to/look/at",
	},
)

The second was a simplification of the existing approach:

detection, err := source.Detect("/somewhere/to/look/at", app.DefaultImagePullSource)
if err != nil {
	return err
}

src, err := detection.NewSource(
	&source.Alias{
		Name:    app.SourceName,
		Version: app.SourceVersion,
	},
	app.Registry.ToOptions(),
	app.Platform,
	app.Exclusions,
)
if src != nil {
	defer src.Close()
}

Open questions:

1. Today we do NOT have the source.target section JSON output captured in the JSON schema -- should we start to do this?

Breaking changes:

• JSON schema v9 bump (see [1]). Note: there are some structural changes to the source.target section in the JSON output (now called source.metadata), however, we have never made guarantees about this section of the JSON document.
• The new source.Source interface replaces the existing object, which breaks the existing API.

Main changes:

• Added new FileSource, DirectorySource, and StereoscopeImageSource concrete types that implement the new source.Source interface
• There is a lot more testing around how the source ID is generated and what is used to define that ID for each source object.
• Simplified much of the source.Input and detection approach to a source.Detect() call which you can create a source object from.
• [1] Overall attempts to move the serialized source object to act like the package object: there is a top level declaration of name and version and nested metadata, and all metadata is mapped 1:1 in the output. This PR removes much of the "select a field and map it with special logic on encoding/decoding" for the source descriptions. Now it is a much simpler pass through for all fields on source.Description. To keep patterns in line with how packages are done, the target field in the JSON output of the source block has been renamed to metadata. This means a new JSON schema has been generated (v9).
• Elevates source.Name and source.Version to be outside of the specific source metadata. Instead it is at the top level, next to ID, as this is common to all sources.
• Adds a new file.ChrootContext for the root+cwd+base logic from the directory resolver (simplifies the dir resolver and allows for reuse of the chroot-like logic).
• FileSource used to not respond to FilesByPath, however, now it does (the parent dir is indexed with all other files excluded now)

Auxiliary changes:

• Testing snapshots for formats now store the redacted output instead of the original output
• A new internal.AllSourceMetadataReflectTypes collection to aid in testing, ensuring completion testing for source.*Metadata types
• Encapsulates the source name and version into an alias object

Deferred work:

• any more adjustments to source.Detect have been deferred until adding the new --from option Allow source type input via CLI flag (not scheme) #1783

Follow up PRs:

• remove pkg metadata type reflect type helpers and use packagemetadata helpers with what was introduced with sourcemetadata. Related to Remove MetadataType from the core package struct #1735
• unexport source models if possible
• move JSON schema version to the syftjson package
• Can we figure a better source ID for the directory source? (Today it's based on path, can we make it content driven?)

JSON Schema Diff for reviewers:

# $ diff ./schema/json/schema-8.0.1.json ./schema/json/schema-9.0.0.json
<   "$id": "https://github.com/anchore/syft/syft/formats/syftjson/model/document",
---
>   "$id": "anchore.io/schema/syft/json/9.0.0/document",
1851a1852,1857
>           "type": "string"
>         },
>         "name": {
>           "type": "string"
>         },
>         "version": {
1857c1863
<         "target": true
---
>         "metadata": true
1861a1868,1869
>         "name",
>         "version",
1863c1871
<         "target"
---
>         "metadata"

Addresses #558 (comment)

Closes: #1866

[github-actions]

Benchmark Test Results

Benchmark results from the latest changes vs base branch

goos: linux%0Agoarch: amd64%0Apkg: github.com/anchore/syft/test/integration%0Acpu: Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz%0A                                                          │ ./.tmp/benchmark-d8a15f2.txt │%0A                                                          │            sec/op            │%0AImagePackageCatalogers/alpmdb-cataloger-2                                    12.05m ± 1%25%0AImagePackageCatalogers/apkdb-cataloger-2                                     732.4µ ± 1%25%0AImagePackageCatalogers/binary-cataloger-2                                    207.0µ ± 2%25%0AImagePackageCatalogers/dpkgdb-cataloger-2                                    589.9µ ± 3%25%0AImagePackageCatalogers/dotnet-deps-cataloger-2                               1.228m ± 2%25%0AImagePackageCatalogers/go-module-binary-cataloger-2                          96.36µ ± 2%25%0AImagePackageCatalogers/java-cataloger-2                                      13.13m ± 1%25%0AImagePackageCatalogers/graalvm-native-image-cataloger-2                      94.82µ ± 3%25%0AImagePackageCatalogers/javascript-package-cataloger-2                        415.7µ ± 2%25%0AImagePackageCatalogers/nix-store-cataloger-2                                 280.3µ ± 7%25%0AImagePackageCatalogers/php-composer-installed-cataloger-2                    801.0µ ± 2%25%0AImagePackageCatalogers/portage-cataloger-2                                   465.5µ ± 3%25%0AImagePackageCatalogers/python-package-cataloger-2                            3.206m ± 1%25%0AImagePackageCatalogers/r-package-cataloger-2                                 201.7µ ± 1%25%0AImagePackageCatalogers/rpm-db-cataloger-2                                    529.9µ ± 1%25%0AImagePackageCatalogers/ruby-gemspec-cataloger-2                              916.1µ ± 1%25%0AImagePackageCatalogers/sbom-cataloger-2                                      119.2µ ± 1%25%0Ageomean                                                                      615.1µ%0A%0A                                                          │ ./.tmp/benchmark-d8a15f2.txt │%0A                                                          │             B/op             │%0AImagePackageCatalogers/alpmdb-cataloger-2                                   5.127Mi ± 0%25%0AImagePackageCatalogers/apkdb-cataloger-2                                    205.0Ki ± 0%25%0AImagePackageCatalogers/binary-cataloger-2                                   30.20Ki ± 0%25%0AImagePackageCatalogers/dpkgdb-cataloger-2                                   169.1Ki ± 0%25%0AImagePackageCatalogers/dotnet-deps-cataloger-2                              405.6Ki ± 0%25%0AImagePackageCatalogers/go-module-binary-cataloger-2                         9.906Ki ± 0%25%0AImagePackageCatalogers/java-cataloger-2                                     2.826Mi ± 0%25%0AImagePackageCatalogers/graalvm-native-image-cataloger-2                     8.594Ki ± 0%25%0AImagePackageCatalogers/javascript-package-cataloger-2                       100.9Ki ± 0%25%0AImagePackageCatalogers/nix-store-cataloger-2                                49.14Ki ± 0%25%0AImagePackageCatalogers/php-composer-installed-cataloger-2                   186.6Ki ± 0%25%0AImagePackageCatalogers/portage-cataloger-2                                  120.0Ki ± 0%25%0AImagePackageCatalogers/python-package-cataloger-2                           1.004Mi ± 0%25%0AImagePackageCatalogers/r-package-cataloger-2                                53.30Ki ± 0%25%0AImagePackageCatalogers/rpm-db-cataloger-2                                   180.9Ki ± 0%25%0AImagePackageCatalogers/ruby-gemspec-cataloger-2                             144.1Ki ± 0%25%0AImagePackageCatalogers/sbom-cataloger-2                                     14.20Ki ± 0%25%0Ageomean                                                                     132.8Ki%0A%0A                                                          │ ./.tmp/benchmark-d8a15f2.txt │%0A                                                          │          allocs/op           │%0AImagePackageCatalogers/alpmdb-cataloger-2                                    87.75k ± 0%25%0AImagePackageCatalogers/apkdb-cataloger-2                                     4.182k ± 0%25%0AImagePackageCatalogers/binary-cataloger-2                                     830.0 ± 0%25%0AImagePackageCatalogers/dpkgdb-cataloger-2                                    3.002k ± 0%25%0AImagePackageCatalogers/dotnet-deps-cataloger-2                               6.338k ± 0%25%0AImagePackageCatalogers/go-module-binary-cataloger-2                           281.0 ± 0%25%0AImagePackageCatalogers/java-cataloger-2                                      39.88k ± 0%25%0AImagePackageCatalogers/graalvm-native-image-cataloger-2                       228.0 ± 0%25%0AImagePackageCatalogers/javascript-package-cataloger-2                        1.404k ± 0%25%0AImagePackageCatalogers/nix-store-cataloger-2                                  895.0 ± 0%25%0AImagePackageCatalogers/php-composer-installed-cataloger-2                    4.079k ± 0%25%0AImagePackageCatalogers/portage-cataloger-2                                   2.269k ± 0%25%0AImagePackageCatalogers/python-package-cataloger-2                            16.44k ± 0%25%0AImagePackageCatalogers/r-package-cataloger-2                                  929.0 ± 0%25%0AImagePackageCatalogers/rpm-db-cataloger-2                                    3.989k ± 0%25%0AImagePackageCatalogers/ruby-gemspec-cataloger-2                              2.447k ± 0%25%0AImagePackageCatalogers/sbom-cataloger-2                                       394.0 ± 0%25%0Ageomean                                                                      2.583k

[wagoodman]

note from sync review: this is a good time to make all breaking changes now, can we finalize detection.NewSource now?

[wagoodman]

from sync review:

• are we using the file.Base? Should we allow this to be configuable for a file source?
• should we name syftjson.model.source.type to metadatatype?

[kzantow]

Left some initial feedback, overall looking like a 👍 change

[kzantow]

why is CWD part of this?

[wagoodman]

it looks like it was needed for the chroot object, but I'm not using it to determine the ID here. I've updated this to extract cwd entirely from this execution path.

[wagoodman]

I wanted to call out this change explicitly -- I still haven't been able to reconcile what a "good" ID is for a directory (that isn't too expensive to figure). But I've added this warning to at least let folks know that if you're doing a directory scan an are not setting --source-name or --source-version explicitly then this is most likely wrong. I didn't call out the specific options since this is the lib path and not in the cmd pkg.

[kzantow]

should this be an Info or Debug message? This will show for every directory scan otherwise, I think

[wagoodman]

I think that's the problem with the directory source ID -- it's really not adequate for any use case where you're trying to cross correlate SBOMs historically since it's brittle and may overlap with entirely unrelated scans. It feels like it should be a warning for that reason.

[wagoodman]

I did find that this wasn't showing up though, and I discovered why: we can't assume that a pointer to alias means that the alias is optional, we still need to check individual fields. I've updated the API to not take alias pointers in config objects (any spot on the exported API where they were function args it is still a pointer though). This allows for the zero value to be used in configs (or for the caller to fully specify like with the cmd package) and we need to check the name and version explicitly, which was already being done in some circumstances.

[kzantow]

question: I missed this in the sync review, but where the "incorrect" names being handled?

[wagoodman]

this PR doesn't go all the way with the packagemetadata, that's reserved for #1844 . In short, this isn't used yet (except for a test), only AllTypes() is used and is consistent with the sourcemetadata package approach. The value add for adding this in this PR was to add the completion test around packagemetadata.AllNames() to detect when there is drift with the ast analysis and the generated code... a test that will be extended to include the JSON name mapping in #1844 (as done in the sourcemetadata.AllNames() completion test).

[kzantow]

should this be an Info or Debug message? This will show for every directory scan otherwise, I think