Missing Instance ID with exportSpannerClientMetrics

[joshbautista]

Current Behavior

We're testing v0.42.2 having upgraded from v0.37.0, and we're now receiving the following warning:

[WARNING] (com.google.cloud.spanner.SpannerCloudMonitoringExporter exportSpannerClientMetrics): Metric data has missing instanceId. Skipping export.

We haven't made any changes to our PGAdapter setup aside from bumping versions.

Context (Environment)

• Running PGAdapter 0.42.2 as a sidecar in GKE
• PGAdapter configured with 1 vCPU and 2GB Memory limits (actual CPU use hovers at ~ 100mCPU)
• PGAdapter executed with the following args:

- args:
  - -p
  - <REDACTED>
  - -i
  - <REDACTED>
  - -d
  - <REDACTED>
  - -enable_otel
  - -otel_trace_ratio=0.05

Environment Variables:
JDK_JAVA_OPTIONS = "-Xmx1600M -Xms1600M -XshowSettings:vm" 

Looks like v0.42.2 is using libraries-bom 26.52.0, which uses google-cloud-spanner 6.83.0 and we're hitting this check that was introduced in 6.81.1.

[olavloite]

@joshbautista Thanks for the report.

The problem occurs when a request is being sent by PGAdapter (or more correct: By the underlying Java client) but never really leaves the client, for example due to a network problem. The reason is that:

1. The RPC is collected as a failed attempt and included in the metrics.
2. However, the instance_id is not set before the request is being sent. Instead, that happens in this interceptor when the headers are being is sent.
3. If the request is never being sent due to a network problem, then no instance_id will be set, and the metric will be added to the collection without an instance_id.
4. Once that has happened once, it will continue to log the warning, as the entire export is being skipped (instead of only the metric without an instance_id).

I'll open an issue for the Java client and ensure that a fix is applied there.

[olavloite]

googleapis/java-spanner#3612