detectors/gcp: support Cloud Run jobs

[zchee]

Support Cloud Run jobs using CLOUD_RUN_JOB and CLOUD_RUN_EXECUTION.

[damemi]

/gcbrun

[zchee]

Note that Cloud Run jobs do not add K_XXX environment variables automatically. Instead of CLOUD_RUN_JOB and CLOUD_RUN_EXECUTION.

[zchee]

@damemi not running gcbrun...?

[damemi]

/gcbrun

[damemi]

hm that seemed to work

[zchee]

thanks

[damemi]

cc @dashpole

[dashpole]

I don't see those set in https://cloud.google.com/run/docs/container-contract#jobs-env-vars. Are the docs incorrect?

[zchee]

@dashpole I think doc is lacking. I was tested on real environment.

[zchee]

@dashpole PTAL

[dashpole]

/gcbrun

[zchee]

@dashpole gentle ping

[dashpole]

Yes, sorry. We are discussing as a team to make sure this is the right approach.

[zchee]

@dashpole Ah, I see. I waiting your discuss result.

[dashpole]

Hey @zchee, if you make a job with multiple tasks, what is the value in CLOUD_RUN_EXECUTION? Does each task end up with the same job name and execution? If so, we may need some additional information to prevent collisions between executions.

[zchee]

@dashpole I'll investigate it.

Just in case, in the case of the gcloud bate run jobs create command, it means --tasks= flag, not --parallelism flag, right?

ref:

• https://cloud.google.com/run/docs/create-jobs

[dashpole]

If you can, try both, and also look at what CLOUD_RUN_TASK_INDEX and CLOUD_RUN_TASK_ATTEMPT are set to. Really appreciate the help

[zchee]

@dashpole okay, will check both situations today.

Also, I'll report Cloud Run jobs docs lacking that's 2 envs via Mercari's Google TAM (I work for Mercari, Inc).

[zchee]

@dashpole replied from Google support, said "seems to serverless team forgot update document", So I think would be better to waiting to update docs. Maybe your concern is also cleared.

---

Also, not yet checked real results (sorry, I was busy a bit) but will check.

[zchee]

@dashpole I found otel side hardcoded K_SERVICE, need fix it also.

• https://github.com/open-telemetry/opentelemetry-go-contrib/blob/v1.9.0/detectors/gcp/cloud-run.go#L110

[dashpole]

Note that the old cloud run (which that code references) is deprecated in favor of this code base.

[zchee]

@dashpole updated document

[zchee]

@dashpole gentle ping

I want to hear your opinions.

[dashpole]

What were CLOUD_RUN_TASK_INDEX and CLOUD_RUN_TASK_ATTEMPT set to?

Also, what is in d.metadata.InstanceID() (faas.id)?

Basically, we want to make sure that faas.name, faas.version, and faas.id uniquely identify an instance of a cloud run job. Otherwise, if two things are running in parallel with the same faas.* attributes, there could be conflicts.

[zchee]

@dashpole I'll reply details later.

[zchee]

@dashpole

Sorry for the late reply.

Use https://github.com/GoogleCloudPlatform/golang-samples/tree/main/run/jobs

diff --git a/run/jobs/main.go b/run/jobs/main.go
index 05e3195..a8a6d0a 100644
--- a/run/jobs/main.go
+++ b/run/jobs/main.go
@@ -21,6 +21,8 @@ import (
 	"os"
 	"strconv"
 	"time"
+
+	"cloud.google.com/go/compute/metadata"
 )
 
 type Config struct {
@@ -99,6 +101,12 @@ func main() {
 	}
 
 	log.Printf("Completed Task #%s, Attempt #%s", config.taskNum, config.attemptNum)
+
+	instanceID, err := metadata.InstanceID()
+	if err != nil {
+		log.Fatal(err)
+	}
+	log.Printf("instanceID: %s\n", instanceID)
 }
 
 // Throw an error based on fail rate

$ gcloud alpha run jobs create --tasks=5 --parallelism=3 --region=asia-northeast1 jobs
# execute...

labels: {
run.googleapis.com/execution_name: "jobs-ph5p9"
run.googleapis.com/task_attempt: "0"
run.googleapis.com/task_index: "0"
}

labels: {
run.googleapis.com/execution_name: "jobs-ph5p9"
run.googleapis.com/task_attempt: "0"
run.googleapis.com/task_index: "1"
}

labels: {
run.googleapis.com/execution_name: "jobs-ph5p9"
run.googleapis.com/task_attempt: "0"
run.googleapis.com/task_index: "2"
}

textPayload: "2022/09/10 08:28:37 instanceID: 00c527f6d4ab9f193b199268ccb9d99d05ebc33d8983a56d2faef9169e9b42adf8fb6caaf62a35d1bff6df6b375c4282310b634da8218f77e706845ac6f4d5f27714"

[damemi]

@zchee sorry for the delay in this. The spec changes just merged and from the discussion in open-telemetry/opentelemetry-specification#3378 this will be slightly different than we talked about here.

Instead, we will have separate GCP-specific resource attributes for Cloud Run execution and Job ID (open-telemetry/opentelemetry-go#4079 will add these to the Go semconv package).

I think we should probably just make these available via their own functions, and maybe add 2 new functions IsJob() and IsService(). wdyt?

[zchee]

make sense. I'll implements.

[damemi]

Thanks for updating this @zchee. We just need to make a few changes to match the spec for Cloud Run Jobs that was approved upstream.

They did not like the idea of using execution+task as faas.version, so those will be their own individual attributes instead. That means we need separate functions for each of them.

To know that we need to call those functions, Cloud Run Jobs have to be treated like their own environment, separate from Services. So a new detector function for that as well.

imo, faas.name is fine to be shared between Jobs and Services. These are semantically providing the same info, even though it is technically different environment variables.

To summarize, we need the following new functions:

• onCloudRunJob() - detect if we are running on Cloud Run Jobs
• FaasJobExecution() - return the Execution ID of the Job
• FaasJobTaskIndex() - return the Task Index for the execution

Our Collector resource detection processor will be updated to write to the new gcp.cloud_run.* attributes when running on Cloud Run Jobs, and our exporter's monitored resource mapping will be updated to parse execution+task into a unique task ID.

[zchee]

@damemi All of done. Also, thanks for polite details reply.
PTAL.

---

BTW, I'm concerned about the inconsistencies in the notation of Cloud Run jobs. Like CloudFunctions vs CloudFunction.
There are

• Cloud Run jobs (I assume this is the official service name, references from GCP docs
• Cloud Run Job (lack of end of "s"
• Cloud Run Jobs (means camel case. For documentation, not for the source code cuz Go's naming convention

For now, I have updated this PR according to otel spec's changes. But certainly not unified between the source code and GCP documentationd. such as CloudRunJob, FaaSJobExecution, FaaSJobTaskIndex, and etc.
WDYT?

[zchee]

We should at least leave a comment here about ...

#465 (comment)

Should I include that to this PR?

[damemi]

We should at least leave a comment here about ...

#465 (comment)

Should I include that to this PR?

yes please do, this will prevent the same headache in the future :)

for the naming inconsistencies, I don't really have a good answer. I think the usage depends on the context, ie when referring to the product "Cloud Run jobs" makes sense, when referring to multiple jobs the plural makes sense vs a singular job. Either way, I would always prefer the cloudRunJob camel case vs cloudRunjob in code

[damemi]

on second thought, the new functions should probably match the resource attribute they are updating. So CloudRunJobExecution() and CloudRunJobTaskIndex(). Then it's:

• FaaSName() -> faas.name
• FaaSVersion() -> faas.version
• CloudRunJobExecution() -> gcp.cloud_run.job.execution
• CloudRunJobTaskIndex() -> gcp.cloud_run.job.task_index

sorry to ask you to change this again. does that sound good to you?

[damemi]

Something shorter like JobExecution() and JobTaskIndex() are probably fine too

[zchee]

I chose the first one cuz mapped to otel spec is readable

[damemi]

sounds good to me!

[zchee]

@damemi PTAL.

[damemi]

/gcbrun

[damemi]

thanks @zchee for working on this!