feat: populate `.repo-metadata.json` from highest version #2890

JoeWang1127 · 2024-06-13T20:30:16Z

In this PR:

populate .repo-metadata.json from highest stable version of proto paths.

The .repo-metadata.json is only generated once per library and the first item in proto paths is used, so the proto paths with the highest stable version should be the first item after sorting the proto paths in the library.

The method used to compare two proto paths, a and b:

if both of a and b don't have a version, the one with lower depth is smaller.
if either a or b has a version, the one with the version is smaller.
if either a or b has a stable version, but not both, the one with the stable version is smaller, e.g., google/spanner/v2 is smaller than google/spanner/v3beta.
if both a and b have a non-stable version, the one with higher version is smaller, e.g., google/spanner/v2beta1 is smaller than google/spanner/v2alpha2.
if both a and b have a stable version, the one with lower depth is smaller, e.g., google/spanner/v1 is smaller than google/spanner/admin/v1, google/spanner/v1 is smaller than google/spanner/admin/v2
if both a and b have a stable version and the depth is the same, the one with higher version is smaller, e.g., google/spanner/v2 is smaller than google/spanner/v1.

Test the change:

build the image locally:

docker build -f .cloudbuild/library_generation/library_generation.Dockerfile -t local:latest .

build the generator locally:

mvn -V -B -ntp clean install -DskipTests

run the image:

docker run --rm -u "$(id -u):$(id -g)" -v "$(pwd):/workspace" -v "$HOME/.m2:/home/.m2" local:latest

Result:

no change in common protos
transport changed to both in java-iam/.repo-metadata.json. This is expected since the transport in google/iam/v2/BUILD.bazel is grpc+rest. Without this change, the transport is parsed from google/iam/v2/BUILD.bazel, which doesn't have a java_gapic_library target, thus the value is grpc by default.

diegomarquezp

QQ, Don't we have .repo-metadata golden files for our units? Maybe we can change one of these to test the proto_path sorting criteria

diegomarquezp · 2024-06-17T19:01:53Z

library_generation/model/gapic_config.py

+        if len(self_dirs) != len(other_dirs):
+            return len(self_dirs) < len(other_dirs)
+        # otherwise, the one with higher version is smaller.
+        return self_version > other_version


I'm a bit ignorant to string comparison in python. Is v2 greater than v1 because "2" has a higher ASCII value than "1"?

Is v2 greater than v1 because "2" has a higher ASCII value than "1"?

Yes.

I released that string comparison has a bug here: v10 is smaller than v2. I'll change the algorithm.

library_generation/utils/proto_path_utils.py

library_generation/model/gapic_config.py

blakeli0 · 2024-06-18T02:37:33Z

library_generation/test/model/library_config_unit_tests.py

@@ -37,3 +38,29 @@ def test_get_library_returns_api_shortname(self):
            gapic_configs=list(),
        )
        self.assertEqual("secret", library.get_library_name())
+
+    def test_get_sorted_gapic_configs_returns_correct_order(self):


Can we create a test for each of the scenario? [v1alpha1, v1], [v1, v2] etc. Because

It would be much more readable. For example, just by reading the tests, I don't know why v1alpha1, v1, v2, admin_v2, non_versioned, v1beta1 is sorted to v2, v1, admin_v2, v1beta1, v1alpha1, non_versioned.

We can also easily test more scenarios. e.g. scenarios of two non versioned libraries.

I added unit tests to verify comparison result of two gapic_configs.

I kept this test case though.

sonarcloud · 2024-06-18T19:04:48Z

Quality Gate passed for 'gapic-generator-java-root'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

sonarcloud · 2024-06-18T19:07:54Z

Quality Gate passed for 'java_showcase_integration_tests'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

JoeWang1127 · 2024-06-18T20:26:45Z

QQ, Don't we have .repo-metadata golden files for our units? Maybe we can change one of these to test the proto_path sorting criteria

The golden files are used in testing .repo-metadata.json generation, generate_prerequisite_files to be specific.

If we want to test generating this file with a library_config (from which contains multiple proto_path), we need test generate_composed_library, which is not easy to do since this function aggregates many "small" functions and is tested in integration test.

🤖 I have created a release *beep* *boop* --- <details><summary>2.42.0</summary> ## [2.42.0](v2.41.0...v2.42.0) (2024-06-25) ### Features * Allow Adding Client Level Attributes to MetricsTracerFactory ([#2614](#2614)) ([f122c6f](f122c6f)) * gapic-generator-java to perform a no-op when no services are detected ([#2460](#2460)) ([c0b5646](c0b5646)) * Make Layout Parser generally available in V1 ([e508ae6](e508ae6)) * populate `.repo-metadata.json` from highest version ([#2890](#2890)) ([f587541](f587541)) * push SNAPSHOT versions of the hermetic build docker image ([#2888](#2888)) ([81df866](81df866)) ### Bug Fixes * **deps:** update the Java code generator (gapic-generator-java) to 1.2.3 ([e508ae6](e508ae6)) * Expose Gax meter name ([#2865](#2865)) ([6c5d6ce](6c5d6ce)) * Move the logic of getting systemProductName from static block to static method ([#2874](#2874)) ([536f1eb](536f1eb)) * Update default Otel Attribute from method_name to method ([#2833](#2833)) ([af10a9e](af10a9e)) ### Dependencies * update dependency com.google.auto.value:auto-value to v1.11.0 ([#2842](#2842)) ([dd27fdf](dd27fdf)) * update dependency com.google.auto.value:auto-value-annotations to v1.11.0 ([#2843](#2843)) ([bf8e67f](bf8e67f)) * update dependency com.google.cloud:grpc-gcp to v1.6.1 ([#2943](#2943)) ([9f16b40](9f16b40)) * update dependency org.checkerframework:checker-qual to v3.44.0 ([#2848](#2848)) ([7a99c50](7a99c50)) * update dependency org.easymock:easymock to v5.3.0 ([#2871](#2871)) ([c243f7d](c243f7d)) * update google api dependencies ([#2846](#2846)) ([b5ef698](b5ef698)) * update googleapis/java-cloud-bom digest to 17cc5ec ([#2882](#2882)) ([d6abd8e](d6abd8e)) * update netty dependencies to v4.1.111.final ([#2877](#2877)) ([b5f10b9](b5f10b9)) * update opentelemetry-java monorepo to v1.39.0 ([#2863](#2863)) ([9d1f3a8](9d1f3a8)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com> Co-authored-by: Joe Wang <joewa@google.com>

JoeWang1127 added 4 commits June 13, 2024 16:29

feat: populate `.repo-metadata.json using the highest version

57f1a13

add unit tests

bd77781

Merge branch 'main' into feat/populate-metadate-from-highest-version

aaf65a9

Merge branch 'main' into feat/populate-metadate-from-highest-version

07f5172

JoeWang1127 marked this pull request as ready for review June 17, 2024 16:19

JoeWang1127 requested a review from a team as a code owner June 17, 2024 16:19

JoeWang1127 requested review from blakeli0 and diegomarquezp June 17, 2024 16:19

diegomarquezp reviewed Jun 17, 2024

View reviewed changes

blakeli0 reviewed Jun 17, 2024

View reviewed changes

library_generation/utils/proto_path_utils.py Outdated Show resolved Hide resolved

library_generation/model/gapic_config.py Outdated Show resolved Hide resolved

blakeli0 reviewed Jun 18, 2024

View reviewed changes

JoeWang1127 added 2 commits June 18, 2024 14:57

change comparsion alg

b9611ff

add unit tests

c2e7b1c

product-auto-label bot added the size: m Pull request size is medium. label Jun 18, 2024

JoeWang1127 requested a review from blakeli0 June 18, 2024 19:35

JoeWang1127 requested a review from diegomarquezp June 18, 2024 20:27

blakeli0 approved these changes Jun 19, 2024

View reviewed changes

JoeWang1127 merged commit f587541 into main Jun 19, 2024
34 checks passed

JoeWang1127 deleted the feat/populate-metadate-from-highest-version branch June 19, 2024 12:19

release-please bot mentioned this pull request Jun 19, 2024

chore(main): release 2.42.0 #2847

Merged

blakeli0 mentioned this pull request Jun 20, 2024

chore: add iam logging #2932

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: populate `.repo-metadata.json` from highest version #2890

feat: populate `.repo-metadata.json` from highest version #2890

JoeWang1127 commented Jun 13, 2024 •

edited

Loading

diegomarquezp left a comment

diegomarquezp Jun 17, 2024

JoeWang1127 Jun 17, 2024

JoeWang1127 Jun 18, 2024

blakeli0 Jun 18, 2024

JoeWang1127 Jun 18, 2024 •

edited

Loading

sonarcloud bot commented Jun 18, 2024

sonarcloud bot commented Jun 18, 2024

JoeWang1127 commented Jun 18, 2024

feat: populate .repo-metadata.json from highest version #2890

feat: populate .repo-metadata.json from highest version #2890

Conversation

JoeWang1127 commented Jun 13, 2024 • edited Loading

diegomarquezp left a comment

Choose a reason for hiding this comment

diegomarquezp Jun 17, 2024

Choose a reason for hiding this comment

JoeWang1127 Jun 17, 2024

Choose a reason for hiding this comment

JoeWang1127 Jun 18, 2024

Choose a reason for hiding this comment

blakeli0 Jun 18, 2024

Choose a reason for hiding this comment

JoeWang1127 Jun 18, 2024 • edited Loading

Choose a reason for hiding this comment

sonarcloud bot commented Jun 18, 2024

Quality Gate passed for 'gapic-generator-java-root'

sonarcloud bot commented Jun 18, 2024

Quality Gate passed for 'java_showcase_integration_tests'

JoeWang1127 commented Jun 18, 2024

feat: populate `.repo-metadata.json` from highest version #2890

feat: populate `.repo-metadata.json` from highest version #2890

JoeWang1127 commented Jun 13, 2024 •

edited

Loading

JoeWang1127 Jun 18, 2024 •

edited

Loading