Loading curations from ClearlyDefined in the analyzer step is inefficient

[oheger-bosch]

When analyzing a larger project, I noticed that about half of the time was consumed by loading curation data from ClearlyDefined via ClearlyDefinedPackageCurationProvider. So optimizing this class should actually have a measurable effect on analyzer runs.

Points to optimize could be:

• Packages that are referenced by multiple projects are queried multiple times.
• Requests for single packages are sent one by one. It could be checked whether the ClearlyDefined API supports bulk requests, or some parallelization could be done on client side.

[sschuberth]

It could be checked whether the ClearlyDefined API supports bulk requests

Looks good: https://api.clearlydefined.io/api-docs/#/curations/post_curations_

[sschuberth]

Reopening this to discuss with @oheger-bosch about:

Packages that are referenced by multiple projects are queried multiple times.

[sschuberth]

Reopening this to discuss with @oheger-bosch about:

Packages that are referenced by multiple projects are queried multiple times.

After looking at

ort/analyzer/src/main/kotlin/AnalyzerResultBuilder.kt

Lines 81 to 86 in 58dfab3

	/**
	* Add the given [packageSet] to this builder. This function can be used for packages that have been obtained
	* independently of a [ProjectAnalyzerResult].
	*/
	fun addPackages(packageSet: Set<Package>): AnalyzerResultBuilder {
	val (curations, duration) = measureTimedValue { curationProvider.getCurationsFor(packageSet.map { it.id }) }

and

ort/analyzer/src/main/kotlin/Analyzer.kt

Line 161 in 58dfab3

.addPackages(managerResult.sharedPackages)

and

ort/analyzer/src/main/kotlin/PackageManagerResult.kt

Lines 46 to 51 in 58dfab3

	/**
	* A set with [Package]s shared across the projects analyzed by this [PackageManager]. Package managers that
	* produce a shared [DependencyGraph] typically do not collect packages on a project-level, but globally. Such
	* packages can be stored in this property.
	*/
	val sharedPackages: Set<Package> = sortedSetOf()

@oheger-bosch and I came to the conclusion that this issue will disappear once all package manager are migrated to the dependency graph format (see #3825) which uses a shared list of packages for all projects of the same type.

So I'm closing this in favor of keeping only #3825.