403 when try to use grpc com.google.endpoints.examples.hello.HelloWorldClient

[zhihuawensc]

In which file did you encounter the issue?

com.google.endpoints.examples.hello.HelloWorldClient

https://github.com/GoogleCloudPlatform/java-docs-samples/blob/master/endpoints/getting-started-grpc/client/src/main/java/com/google/endpoints/examples/hello/HelloWorldClient.java

Did you change the file? If so, how?

NO.

Describe the issue

I followed all the steps in https://github.com/GoogleCloudPlatform/java-docs-samples/tree/master/endpoints/getting-started-grpc, it can run locally. I started the docker, enabled the HTTP traffic on the gce instance, created the api key on my project. But I still got 403 error as below. Please advise me how to debug this issue.

C02S715FFVH6:getting-started-grpc zhihua.wen$ java -jar client/build/libs/client.jar --host external_ip:80 --api_key <api_key>
Sep 27, 2017 3:27:18 PM io.grpc.internal.ManagedChannelImpl

INFO: [ManagedChannelImpl@4cdbe50f] Created with target external_ip:80

Sep 27, 2017 3:27:18 PM com.google.endpoints.examples.hello.HelloWorldClient greet
INFO: Will try to greet world ...
Sep 27, 2017 3:27:18 PM com.google.endpoints.examples.hello.HelloWorldClient$Interceptor interceptCall
INFO: Intercepted helloworld.Greeter/SayHello
Sep 27, 2017 3:27:18 PM com.google.endpoints.examples.hello.HelloWorldClient$Interceptor$1 start
INFO: Attaching API Key: <api_key>

Sep 27, 2017 3:27:19 PM com.google.endpoints.examples.hello.HelloWorldClient greet

WARNING: RPC failed: Status{code=UNAVAILABLE, description=Service control request failed with HTTP response code 403, cause=null}

Sep 27, 2017 3:27:19 PM io.grpc.internal.ManagedChannelImpl maybeTerminateChannel

[zhihuawensc]

Here is the esp log I got, it seems servicemanagement api is denied on my service, I did enable the google servicemanagement api in my project.

Container	Timestamp	Message
esp	2017-09-27T22:48:12.087421253Z	[libprotobuf ERROR external/servicecontrol_client_git/src/service_control_client_impl.cc:182] Failed in Report call: Service control request failed with HTTP response code 403
esp	2017-09-27T22:48:12.087381581Z	2017/09/27 22:48:12[error]9#9: Failed to call https://servicecontrol.googleapis.com/v1/services/hellogrpc.endpoints.tidal-datum-763.cloud.goog:report, Error: FORBIDDEN: server response status code: 403, Response body: ���rPermission 'servicemanagement.services.report' denied on service 'hellogrpc.endpoints.tidal-datum-763.cloud.goog'.
esp	2017-09-27T22:48:10.057063139Z	10.128.0.4 - - [27/Sep/2017:22:48:10 +0000] "POST /helloworld.Greeter/SayHello HTTP/2.0" 503 0 "-" "grpc-java-netty"
esp	2017-09-27T22:48:10.056599829Z	[libprotobuf ERROR external/servicecontrol_client_git/src/service_control_client_impl.cc:213] Failed in Check call: Service control request failed with HTTP response code 403
esp	2017-09-27T22:48:10.056520659Z	2017/09/27 22:48:10[error]9#9: Failed to call https://servicecontrol.googleapis.com/v1/services/hellogrpc.endpoints.tidal-datum-763.cloud.goog:check, Error: FORBIDDEN: server response status code: 403, Response body: ���qPermission 'servicemanagement.services.check' denied on service 'hellogrpc.endpoints.tidal-datum-763.cloud.goog'.

[jeffmendoza]

@zhihuawensc does the service account that is associated with your GCE VM have Read scopes on "Service Management" and enabled for "Service Control"? This is what I get by deafult:

[jeffmendoza]

Also make sure your service account has "Editor" permissions in https://console.cloud.google.com/iam-admin/iam/project

[zhihuawensc]

@jeffmendoza
Yes, I think have both set up. I saw the same 403 error in both GCE and GKE steps as shown in https://github.com/GoogleCloudPlatform/java-docs-samples/tree/master/endpoints/getting-started-grpc, please advise me as how I can furher debug it.

Is there a simple way I can just test the servicemanagement api permission from inside the VM?

[jeffmendoza]

It is a bit tedious, but you can. Get an access token for your compute engine default service account:

curl "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token" -H "Metadata-Flavor: Google"

{"access_token":"asdf","expires_in":3217,"token_type":"Bearer"}

Then make a request to service management.

curl https://servicemanagement.googleapis.com/v1/services/$SERVICE_NAME/configs/$SERVICE_CONFIG_ID -H "Authorization":"Bearer asdf"

.....

Note I used the $SERVICE_NAME and $SERVICE_CONFIG env vars the sample has you set.

The reference for that call is here: https://cloud.google.com/service-management/reference/rest/v1/services.configs/get

I see that your errors are for service control check and report, the reference for that is here: https://cloud.google.com/service-control/reference/rest/ There might be a problem with the settings on that API, but it looks good in your screenshots.

[zhihuawensc]

@jeffmendoza

I can make a request to service management api from the GCE vm using the access token from the metadata server.

curl https://servicemanagement.googleapis.com/v1/services/$SERVICE_NAME/configs/$SERVICE_CONFIG_ID -H "Authorization":"Bearer ya29.xxxx"
{
"name": "hellogrpc.endpoints.tidal-datum-763.cloud.goog",
"title": "Hello gRPC API",
"apis": [
{
"name": "helloworld.Greeter",
"methods": [
{
...

[zhihuawensc]

@jeffmendoza
My issue is solve now, in my previous attempt
the command "gcloud service-management deploy out.pb api_config.yaml" crashed after showing some successful message, but maybe it was incomplete.

I updated gcloud util and changed the service name, this time everything went through.

[jeffmendoza]

Good to know. Do you have the log from the original crash of the gcloud command?

[ernsheong]

Wow, it's 2019 and this was probably the cause of my issue. I redeployed my service again and the error went away (or changed, rather).

Please see:

• https://groups.google.com/forum/#!msg/google-cloud-endpoints/UYReDuKNoc0/BjO-GIhZCgAJ
• https://issuetracker.google.com/issues/122241615

There is a very painful bug that needs to be addressed by the Cloud Endpoints team.

UPDATE: Looks like my original issue was is not the exact same as this issue, but this issue lent clues anyway.