ComputeEngineCredentials does not handle error response from the metadata server correctly

[jianglai]

ComputeEngineCredentials assumes that the response from from the metadata server is either 200 or 503:

try {
      response = request.execute();
    } catch (UnknownHostException exception) {
      throw new IOException(
          "ComputeEngineCredentials cannot find the metadata server. This is"
              + " likely because code is not running on Google Compute Engine.",
          exception);
    }

    if (response.getStatusCode() == 503) {
      throw GoogleAuthException.createWithTokenEndpointResponseException(
          new HttpResponseException(response));
    }

    return response;

This is not necessarily true. For example, on GKE, when Workload Identity Federation is used, one needs to bind the k8s service account to an IAM service account in order to receive an OIDC token by calling idTokenWithAudience. Without the binding, the metadata server returns a 404 with the following body:

Your Kubernetes service account (default/nomulus) is not annotated with a target Google service account, which is a requirement for retrieving Identity Tokens using Workload Identity.
Please add the [iam.gke.io/gcp-service-account=[GSA_NAME]@[PROJECT_ID](http://iam.gke.io/gcp-service-account=[GSA_NAME]@[PROJECT_ID)] annotation to your Kubernetes service account.
Refer to [https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity?authuser=0)

The 404 code is silently ignored and the response body (i. e. the error message above) was then treated as the base64-encoded ID token, which causes the decoding step to fail:

Caused by: java.lang.IllegalArgumentException: com.google.common.io.BaseEncoding$DecodingException: Unrecognized character: 0x20
	at com.google.common.io.BaseEncoding.decode(BaseEncoding.java:222)
	at com.google.api.client.util.Base64.decodeBase64(Base64.java:106)
	at com.google.api.client.json.webtoken.JsonWebSignature$Parser.parse(JsonWebSignature.java:545)
	at com.google.api.client.json.webtoken.JsonWebSignature.parse(JsonWebSignature.java:479)
	at com.google.auth.oauth2.IdToken.create(IdToken.java:80)
	at com.google.auth.oauth2.IdToken.create(IdToken.java:68)
	at com.google.auth.oauth2.ComputeEngineCredentials.idTokenWithAudience(ComputeEngineCredentials.java:335)
	at google.registry.util.OidcTokenUtils.createOidcToken(OidcTokenUtils.java:57)
	at google.registry.proxy.ProxyModule.lambda$provideOidcToken$1(ProxyModule.java:278)
	at com.google.common.base.Suppliers$ExpiringMemoizingSupplier.get(Suppliers.java:300)
	at google.registry.proxy.handler.HttpsRelayServiceHandler.decodeFullHttpRequest(HttpsRelayServiceHandler.java:105)
	at google.registry.proxy.handler.WhoisServiceHandler.decodeFullHttpRequest(WhoisServiceHandler.java:49)
	at google.registry.proxy.handler.HttpsRelayServiceHandler.decode(HttpsRelayServiceHandler.java:129)
	at io.netty.handler.codec.ByteToMessageCodec$1.decode(ByteToMessageCodec.java:42)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
	... 30 more
Caused by: com.google.common.io.BaseEncoding$DecodingException: Unrecognized character: 0x20
	at com.google.common.io.BaseEncoding$Alphabet.decode(BaseEncoding.java:541)
	at com.google.common.io.BaseEncoding$Base64Encoding.decodeTo(BaseEncoding.java:1037)
	at com.google.common.io.BaseEncoding.decodeChecked(BaseEncoding.java:237)
	at com.google.common.io.BaseEncoding.decode(BaseEncoding.java:220)
	... 45 more

I had to manually send recreate the HTTP request to the metadata server to obtain the response body and understand the problem, as java.lang.IllegalArgumentException: com.google.common.io.BaseEncoding$DecodingException: Unrecognized character: 0x20 is not helpful at all.

In my opinion getMetadataResponse should throw (with the message body, if present) whenever the response code is not 200, instead of only throwing for 503.

[clundin25]

That seems reasonable to me, and (mostly) matches the Python implementation https://github.com/googleapis/google-auth-library-python/blob/main/google/auth/compute_engine/_metadata.py#L151.