refine dependency resolution design

[webern]

In the design document, #3, the section entitled Twoliter Update needs refinement.

This may require prototyping and implementation to figure out.
Let's not forget to update the design doc when we have a better handle on this.

The section in its original for read as follows:

Twoliter Update

TODO - This section is a bit hand-wavy and will have its design further refined.

twoliter update is another inspiration from cargo, this time inspired by cargo update.
This command makes it easy for maintainers to update the packages (kits actually) they are using to the latest versions.

In the original Bottlerocket build system, we used Cargo crates to model RPM dependencies.
In Twoliter, we will again lean on Cargo to manage dependencies for us.
We will create a local Cargo registry to represent kit and SDK versions, then use cargo update to resolve the latest dependencies.

Before describing this procedure, we should consider that no two versions of the same kit can be in the dependency tree.
Since they are Yum repositories, this would mess things up.
Cargo allows multiple major versions of a package to exist in a build, but this is behavior we do not want.
We will manipulate version numbers to so that Cargo gives us the desired behavior.

First Twoliter will read the kit dependencies from each variant and add any external kits to its graph.
Next Twoliter will read the kit definitions in the kits directory, and add all of their external dependencies to its graph.

Once it has this list, Twoliter will begin a traversal.
For each kit it will pull all the -metadata images for each external kit JSON files locally.
Each time it finds a new external kit (i.e. an external kit that is depended on by an external kit), it will add this new dependency to the graph.

Once it has all the dependency and version information, Twoliter will build a local cargo registry.
Each kit will be represented by an empty Cargo lib.rs project with the same name as the kit.
The Cargo package version will be given by deterministically adding the kit's version to 1.x (TODO - hand-waved).

The SDK will be a special Cargo package.
Two SDKs may have the same version, but come from different container registries.
To ensure a single SDK is used, it will be given a version in Cargo like this (TODO - or some hash of the necessary unique information):

[dependencies]
sdk = "=0.50.0-public.ecr.aws_bottlerocket_bottlerocket-sdk-x86_64"

At this point Twoliter will build a package that represents the maintainers top level dependencies.

[dependencies]
bottlerocket-core-kit = "1.1.15" # TODO - best way to normalize this into 1.x
bottlerocket-aws-kit = "1.1.15"  # in this example the patch version from 1.15.1 has been lost

Twoliter will run cargo check --locked to make sure the current state of the project is good.
If that succeeds then cargo update will change the Cargo.toml file to the highest versions of all kits possible.
The results of this will be parsed and propagated back to the project.
(TODO - how)

Definition of Done

A PR updating the design doc should answer the following questions and eliminate TODOs in the doc that relate to kit and SDK dependency resolution and updates:

• Where will the maintainer specify external kit dependencies?
• How will twoliter discover the complete list of external depenencies?
• How will local kit dependencies be defined?
• How will a maintainer lock a certain kit dependency to a certain version?
• How will Twoliter record the results of its dependency resolution?
• Does Twoliter need to communicate the resolved depenencies to buildsys?

Depends on

• design document #1

[webern]

This gave me some confidence that it will work:

cargo.tar.gz

[webern]

It would be good to update the design doc in GitHub with the new design.

[webern]

Done other than updating the design doc if desired.