Enable independent crate versioning for SDK crates #1540

jdisanti · 2022-07-07T17:14:14Z

Motivation and Context

This enables SDK crates to version independently of each other so that a release only has to publish changed crates rather than every single crate even if it has no changes. This also enables multiple SDK releases per smithy-rs release since manual version bump is no longer required.

Testing

Do a dry-run smithy-rs release with the changes and verify the artifacts
Do a dry-run SDK release with the changes and verify the artifacts

Checklist

I have updated CHANGELOG.next.toml if I made changes to the AWS SDK, generated SDK code, or SDK runtime crates

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

The release tag should be added to the versions manifest at the very end of the release process rather than for each individual synced commit.

github-actions · 2022-07-07T18:32:47Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)
No codegen difference in the Server Test
No codegen difference in the Server Test Python

A new doc preview is ready to view.

github-actions · 2022-07-07T19:24:46Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)
No codegen difference in the Server Test
No codegen difference in the Server Test Python

A new doc preview is ready to view.

…t-sdk-versions

github-actions · 2022-07-07T20:06:35Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)
No codegen difference in the Server Test
No codegen difference in the Server Test Python

A new doc preview is ready to view.

rcoh

only blocker is figuring out the aws-config version stuff, otherwise LGTM

rcoh · 2022-07-11T19:34:41Z

tools/publisher/src/main.rs

+        }
+    } else {
+        // TODO(https://github.com/awslabs/smithy-rs/issues/1531): Remove V1 args
+        println!("Failed to match new arg format. Trying to parse the old arg format.");


rcoh · 2022-07-11T19:35:08Z

tools/publisher/src/main.rs

+#[derive(Parser, Debug)]
+#[clap(author, version, about)]
+enum ArgsV1 {


this is exactly the same right (except for HydrateReadmeArgsV1?) In general, probably nice to have a module of arg versions or something. V1 being old and Args being the latest was a bit confusing

This doesn't seem worth it to me since the V1 will be short lived. I agree, it is slightly confusing, but it's not terrible.

rcoh · 2022-07-11T22:38:42Z

tools/publisher/src/subcommand/generate_version_manifest.rs

+fn hash_models(projection: &SmithyBuildProjection) -> Result<String> {
+    // Must match `hashModels` in `CrateVersioner.kt`
    let mut hashes = String::new();
    for import in &projection.imports {
        hashes.push_str(&sha256::digest_file(import).context("hash model")?);


how does this get used from the Smithy side vs. Rust side?

The publisher tool's generate-version-manifest subcommand creates the versions.toml which includes the model hashes, where this Rust implementation is used.

The Gradle build system then reads the prior release's versions.toml and determines if models were changed by doing its own model hashing.

An alternative we could consider is creating a tool that does model hashing and use it from both.

rcoh · 2022-07-11T22:40:12Z

tools/ci-build/scripts/generate-aws-sdk

@@ -27,12 +27,17 @@ mv aws-doc-sdk-examples/rust_dev_preview smithy-rs/aws/sdk/examples
 rm -rf smithy-rs/aws/sdk/examples/.cargo
 rm smithy-rs/aws/sdk/examples/Cargo.toml

+echo -e "${C_YELLOW}Creating empty model metadata file since we don't have model update information...${C_RESET}"
+MODEL_METADATA_PATH="$(pwd)/model-metadata.toml"


what's model-metadata and how does this compare with the versions manifest?

The model metadata is produced by the automation that brings in the latest SDK service models, and includes information such as what kind of change it was (documentation only, or APIs), and a changelog entry describing the change.

The generate-aws-sdk script is only used in CI, and validates that a SDK can be successfully generated. It's not used to generate the SDK for releases since the sdk-sync tool does that, and there, model metadata and codegen changes get incorporated.

rcoh · 2022-07-11T22:41:33Z

aws/sdk-codegen/src/main/kotlin/software/amazon/smithy/rustsdk/AwsReadmeDecorator.kt

@@ -62,7 +63,7 @@ class AwsReadmeDecorator : RustCodegenDecorator<ClientCodegenContext> {

                    ```toml
                    [dependencies]
-                    aws-config = "${codegenContext.settings.moduleVersion}"
+                    aws-config = "$awsConfigVersion"


this is special because we don't have access to the versions manifest at this point?

Exactly. The service crates get generated before the version manifest is created. This is OK though since service crate generation only needs the version numbers of runtime crates.

rcoh · 2022-07-11T22:42:56Z

aws/sdk-codegen/src/main/kotlin/software/amazon/smithy/rustsdk/SdkSettings.kt

+    val awsConfigVersion: String get() =
+        awsSdk?.getStringMember("awsConfigVersion")?.orNull()?.value
+            ?: throw IllegalStateException("missing `awsConfigVersion` codegen setting")


probably worth noting that this is only for docs? Is this actually used during codegen? (I think it's not?) We should probably use it otherwise it may be surprising when it doesn't actually change the aws-config version when we generate the aws-config dependency normally

It uses a path dependency when it generates the service crate, so that should always be correct. The fix-manifests tool inserts version numbers after the fact. I think this all should be fine until we make runtime crates version independently, at which point this all needs to be revisited anyway. The mechanism for that revisiting will be the removal of the smithy.rs.runtime.crate.version Gradle property, which should reveal all the locations that rely on it, including this one.

rcoh · 2022-07-11T22:44:10Z

aws/sdk/build.gradle.kts

@@ -72,6 +72,8 @@ fun eventStreamAllowList(): Set<String> {
 }

 fun generateSmithyBuild(services: AwsServices): String {
+    val awsConfigVersion = properties.get("smithy.rs.runtime.crate.version")


is this under the assumption that the aws-config version always matches the smithy.rs runtime crate version? I think those are different numbers—aws-smithy-http is on 0.44, eg.

This PR makes all the runtime crates use the same version numbers. They all use smithy.rs.runtime.crate.version now. This will result in a version number jump from ~0.15.0 to ~0.46.0 for the AWS runtime crates on the first release that has these changes, but that should be fine.

…t-sdk-versions

github-actions · 2022-07-19T01:07:48Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)
No codegen difference in the Server Test
No codegen difference in the Server Test Python

A new doc preview is ready to view.

github-actions · 2022-07-19T18:39:11Z

A new generated diff is ready to view.

AWS SDK (ignoring whitespace)
No codegen difference in the Server Test
No codegen difference in the Server Test Python

A new doc preview is ready to view.

jdisanti added 9 commits July 7, 2022 10:11

Update hydrate-readmes to take versions from versions.toml

7512640

Add cleanup TODO comments to changelogger

44392ec

Update SDK readme template

2f0f5c8

Split up generate-version-manifest subcommand

6d2af16

The release tag should be added to the versions manifest at the very end of the release process rather than for each individual synced commit.

Eliminate the aws.sdk.version property

de276cd

Fallback to model hash comparison if model metadata claims no changes

69c546d

Update changelog

3da4ca8

Add acquire-base-image to release.yml

1733d0a

Fix changelogger invocation

e9f62ba

Use empty model metadata for SDK generation in CI

a2f21ec

jdisanti added 3 commits July 7, 2022 12:24

Fix the aws-config version number in SDK crate readmes

92399b9

Merge remote-tracking branch 'upstream/main' into jdisanti-independen…

165a943

…t-sdk-versions

Fix typo

94bb3e2

rcoh reviewed Jul 11, 2022

View reviewed changes

jdisanti added 2 commits July 18, 2022 17:26

Output error message to stderr

1b95eab

Merge remote-tracking branch 'upstream/main' into jdisanti-independen…

3e9674a

…t-sdk-versions

rcoh approved these changes Jul 19, 2022

View reviewed changes

Merge branch 'main' into jdisanti-independent-sdk-versions

fda06aa

rcoh requested a review from a team as a code owner July 19, 2022 17:43

jdisanti merged commit 6143d90 into main Jul 19, 2022

jdisanti deleted the jdisanti-independent-sdk-versions branch July 19, 2022 19:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable independent crate versioning for SDK crates #1540

Enable independent crate versioning for SDK crates #1540

jdisanti commented Jul 7, 2022 •

edited

Loading

github-actions bot commented Jul 7, 2022

github-actions bot commented Jul 7, 2022

github-actions bot commented Jul 7, 2022

rcoh left a comment

rcoh Jul 11, 2022

rcoh Jul 11, 2022

jdisanti Jul 19, 2022

rcoh Jul 11, 2022

jdisanti Jul 12, 2022

rcoh Jul 11, 2022

jdisanti Jul 12, 2022

rcoh Jul 11, 2022

jdisanti Jul 12, 2022

rcoh Jul 11, 2022

jdisanti Jul 12, 2022

rcoh Jul 11, 2022

jdisanti Jul 12, 2022

github-actions bot commented Jul 19, 2022

github-actions bot commented Jul 19, 2022

Enable independent crate versioning for SDK crates #1540

Enable independent crate versioning for SDK crates #1540

Conversation

jdisanti commented Jul 7, 2022 • edited Loading

Motivation and Context

Testing

Checklist

github-actions bot commented Jul 7, 2022

github-actions bot commented Jul 7, 2022

github-actions bot commented Jul 7, 2022

rcoh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jul 19, 2022

github-actions bot commented Jul 19, 2022

jdisanti commented Jul 7, 2022 •

edited

Loading