chore: bake the grpc plugin in the hermetic library generation image #3045

diegomarquezp · 2024-07-11T13:45:58Z

This PR bakes the gRPC plugin in the hermetic library generation image.
The approach is very similar to #2707

The docker image will have a configurable ARG for the gRPC plugin
The docker image will download the gRPC plugin at the version specified in the previous point
The docker image will set the variables DOCKER_GRPC_VERSION and DOCKER_GRPC_LOCATION
generate_library.sh will now be able to avoid downloading the gRPC plugin if the computed plugin version is the same as DOCKER_GRPC_VERSION (all this via utilities.sh).

diegomarquezp · 2024-07-11T13:53:53Z

library_generation/utils/utilities.sh

@@ -282,7 +315,7 @@ get_proto_path_from_preprocessed_sources() {
  pushd "${sources}" > /dev/null
  local proto_library=$(find . -maxdepth 1 -type d -name 'proto-*' | sed 's/\.\///')
  local found_libraries=$(echo "${proto_library}" | wc -l)
-  if [ -z ${proto_library} ]; then
+  if [[ -z ${proto_library} ]]; then


I noticed that the shell unit tests were expecting the binary brackets because proto_library could be expanded to a string with spaces.

+ '[' -z proto-2 proto-1 ']' /usr/local/google/home/diegomarquezp/google/sdk-platform-java/library_generation/test/../utils/utilities.sh: line 318: [: proto-2: binary operator expected

library_generation/utils/utilities.sh

blakeli0 · 2024-07-16T22:38:45Z

library_generation/utils/utilities.sh

  fi
+  echo "$(pwd)/${grpc_filename}"


Is this a debug log?

No, it's the return statement of the function. Many functions in utilities.sh are made this way. The way we use is variable=$(my_function "arg1" "arg2"). The only possible return values for return are integers from 0 to 255 (and consumed via $?) so it's not useful.

blakeli0 · 2024-07-16T22:42:40Z

.cloudbuild/library_generation/library_generation.Dockerfile

 # we indicate protoc is available in the container via env vars
 ENV DOCKER_PROTOC_LOCATION=/protoc
 ENV DOCKER_PROTOC_VERSION="${PROTOC_VERSION}"

+# install grpc
+WORKDIR /grpc
+RUN source /src/utils/utilities.sh \


source /src/utils/utilities.sh is not needed anymore. We should probably split

RUN source /src/utils/utilities.sh \ && download_protoc "${PROTOC_VERSION}" "${OS_ARCHITECTURE}"

to

RUN source /src/utils/utilities.sh RUN download_protoc "${PROTOC_VERSION}" "${OS_ARCHITECTURE}"

Then here we can just do

RUN download_grpc_plugin "${GRPC_VERSION}" "${OS_ARCHITECTURE}"

This is something I considered initially when working on the Dockerfile. The effects of source are contained in the RUN statement only, because it is translated into /bin/sh -c "source script.sh" (see SO)

Cool, that's good to know!

blakeli0 · 2024-07-17T18:32:46Z

.cloudbuild/library_generation/library_generation.Dockerfile

@@ -19,7 +19,9 @@ SHELL [ "/bin/bash", "-c" ]

 ARG OWLBOT_CLI_COMMITTISH=ac84fa5c423a0069bbce3d2d869c9730c8fdf550
 ARG PROTOC_VERSION=25.3
+ARG GRPC_VERSION=1.62.2


Can we configure this with renovate bot, and group it with other grpc update? e.g. There is an existing PR that updates gRPC to the latest.

Done

sdk-platform-java/renovate.json

Lines 55 to 67 in 9795e23

{

"customType": "regex",

"fileMatch": [

"^gax-java/dependencies\\.properties$",

"^\\.cloudbuild/library_generation/library_generation\\.Dockerfile$"

],

"matchStrings": [

"version\\.io_grpc=(?<currentValue>.+?)\\n",

"ARG GRPC_VERSION=[\"']?(?<currentValue>.+?)[\"']?\\s+"

],

"depNameTemplate": "io.grpc:grpc-core",

"datasourceTemplate": "maven"

},

This version can be updated to 1.65.1 now since this PR is merged..

blakeli0

We can still pass grpc_version from generation_config.yaml, I guess that would still override the baked in grpc version and the default grpc version from the pom?

Co-authored-by: Blake Li <blakeli@google.com>

diegomarquezp · 2024-07-22T20:22:16Z

We can still pass grpc_version from generation_config.yaml, I guess that would still override the baked in grpc version and the default grpc version from the pom?

@blakeli0 Yes, if --grpc_version is not passed via the config yaml, then it will infer it via utilities.sh/get_protoc_version.

sdk-platform-java/library_generation/generate_library.sh

Lines 89 to 91 in 0a95554

    
           if [ -z "${grpc_version}" ]; then 
        
             grpc_version=$(get_grpc_version "${gapic_generator_version}") 
        
           fi

sdk-platform-java/library_generation/generate_composed_library.py

Lines 124 to 133 in 0a95554

    
           def __construct_tooling_arg(config: GenerationConfig) -> List[str]: 
        
               """ 
        
               Construct arguments of tooling versions used in generate_library.sh 
        
               :param config: the generation config 
        
               :return: arguments containing tooling versions 
        
               """ 
        
               arguments = [] 
        
               arguments += util.create_argument("gapic_generator_version", config) 
        
               arguments += util.create_argument("grpc_version", config) 
        
               arguments += util.create_argument("protoc_version", config)

sdk-platform-java/library_generation/utils/utilities.py

Lines 31 to 39 in 0a95554

    
           def create_argument(arg_key: str, arg_container: object) -> List[str]: 
        
               """ 
        
               Generates a list of two elements [argument, value], or returns 
        
               an empty array if arg_val is None 
        
               """ 
        
               arg_val = getattr(arg_container, arg_key, None) 
        
               if arg_val is not None: 
        
                   return [f"--{arg_key}", f"{arg_val}"] 
        
               return []

sonarcloud · 2024-07-25T16:07:35Z

Quality Gate passed for 'gapic-generator-java-root'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

sonarcloud · 2024-07-25T16:10:43Z

Quality Gate passed for 'java_showcase_integration_tests'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

diegomarquezp added 2 commits July 11, 2024 13:41

chore: bake the grpc plugin in the hermetic library generation image

8de1c01

improve comments

15b2fc5

product-auto-label bot added the size: m Pull request size is medium. label Jul 11, 2024

diegomarquezp commented Jul 11, 2024

View reviewed changes

diegomarquezp marked this pull request as ready for review July 11, 2024 13:54

blakeli0 reviewed Jul 16, 2024

View reviewed changes

blakeli0 reviewed Jul 17, 2024

View reviewed changes

diegomarquezp and others added 2 commits July 22, 2024 16:02

Update library_generation/utils/utilities.sh

0a95554

Co-authored-by: Blake Li <blakeli@google.com>

add grpc to renovate.json

034edf9

add GRPC_VERSION to renovate config

9795e23

diegomarquezp requested a review from blakeli0 July 24, 2024 20:54

diegomarquezp mentioned this pull request Jul 24, 2024

feat: bake gapic-generator-java into the hermetic build docker image #3067

Merged

diegomarquezp added 2 commits July 25, 2024 15:59

Merge remote-tracking branch 'origin/main' into bake-grpc-hermetic-build

ecfe8cd

update grpc to 1.65.1

bfac93f

blakeli0 approved these changes Jul 25, 2024

View reviewed changes

diegomarquezp merged commit 4bbb8ac into main Jul 25, 2024
48 of 50 checks passed

diegomarquezp deleted the bake-grpc-hermetic-build branch July 25, 2024 21:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: bake the grpc plugin in the hermetic library generation image #3045

chore: bake the grpc plugin in the hermetic library generation image #3045

diegomarquezp commented Jul 11, 2024

diegomarquezp Jul 11, 2024

blakeli0 Jul 16, 2024

diegomarquezp Jul 22, 2024

blakeli0 Jul 16, 2024

diegomarquezp Jul 22, 2024

blakeli0 Jul 25, 2024

blakeli0 Jul 17, 2024

diegomarquezp Jul 22, 2024

blakeli0 Jul 25, 2024

diegomarquezp Jul 25, 2024

blakeli0 left a comment

diegomarquezp commented Jul 22, 2024

sonarcloud bot commented Jul 25, 2024

sonarcloud bot commented Jul 25, 2024

	{
	"customType": "regex",
	"fileMatch": [
	"^gax-java/dependencies\\.properties$",
	"^\\.cloudbuild/library_generation/library_generation\\.Dockerfile$"
	],
	"matchStrings": [
	"version\\.io_grpc=(?<currentValue>.+?)\\n",
	"ARG GRPC_VERSION=[\"']?(?<currentValue>.+?)[\"']?\\s+"
	],
	"depNameTemplate": "io.grpc:grpc-core",
	"datasourceTemplate": "maven"
	},

chore: bake the grpc plugin in the hermetic library generation image #3045

chore: bake the grpc plugin in the hermetic library generation image #3045

Conversation

diegomarquezp commented Jul 11, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blakeli0 left a comment

Choose a reason for hiding this comment

diegomarquezp commented Jul 22, 2024

sonarcloud bot commented Jul 25, 2024

Quality Gate passed for 'gapic-generator-java-root'

sonarcloud bot commented Jul 25, 2024

Quality Gate passed for 'java_showcase_integration_tests'