Using custom auto-instrumentation distributions/images

[svrnm]

Auto-instrumentation injection is a great feature for people to get started with OpenTelemetry easily (because of that it's also mentioned in the docs now 😄 ).

It would be great if the end-user would have more flexibility on configuring the different injections:

• Add additional instrumentation packages (or extensions) in package.json / requirements.txt or via -Dotel.javaagent.extensions for java
• Load a different distribution (my own, vendor-specific) for the javaagent.jar, opentelemetry-distro, @opentelemetry/sdk-node

Would this be an extension to the operator that would be worth pursuing?

[jpkrohling]

cc @anuraaga, @pavolloffay

[anuraaga]

Hi @svrnm - The operator allows using a different docker image for sourcing instrumentation, so I think that the operator probably has the needed extension point for this.

We probably need better documentation on how to build a custom docker image for user needs. For first point, Add additional instrumentation packages, I think it's relatively simple by using our published image as a base and just calling npm install or pip install or copying in javaagent extensions. javaagent extensions would also involve adding a JVM flag in the k8s config with the flag (this could potentially be a native instrumentation config option if enough demand).

Replacing with an entirely different distro is less obvious how to make it user friendly. I'm only really familiar with Java distros, and I think some docs that use our very trivial Dockerfile would be fine. Any thoughts on how to customize the actual build scripts for the other packages?

[svrnm]

We probably need better documentation on how to build a custom docker image for user needs.

Agreed, as you can see from me opening this issue people might not be aware of this.

I think a quick fix is just writing that in simple words at the end of the relevant section in the docs, e.g. (reusing your words):

The operator allows using a different docker image for sourcing instrumentation. 
Use the existing published images as base and call `npm install` or `pop install` 
to add additional instrumentation packages.

Next step would be adding some simple examples.

Replacing with an entirely different distro is less obvious how to make it user friendly

So, far java someone would replace the URL in this line:

ADD https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v$version/opentelemetry-javaagent.jar /javaagent.jar

with something like

ADD https://vendor.url/opentelemetry-javaagent.jar /javaagent.jar

Correct?

For python & node.js it depends on how the distribution is created:

• if there are just extensions building on top of @opentelemetry/sdk-node and opentelemetry-distro it is similar to adding additional instrumentations
• if there are replacements for those 2 packages like @vendor/sdk-node and vendor-opentelemetry-distro they might need to
  • overwrite package.json and autoinstrumentation.ts for node.js
  • overwrite requirements.txt and ??? for python

It sounds like there might be some language-specific best practices for creating distributions, so that they are compatible with this.

[pavolloffay]

@svrnm your assumptions are correct.

As @anuraaga mentioned, using custom images is something that users can do right now, it just needs to be documented.

In some parts of the operator code we are also checking for things like what variable names are specified in the CR and if the variable name does not have the right prefix (at the moment OTEL_, SPLUNK_) then an error is thrown, however we are open to generalize that for any vendor distribution.

[pavolloffay]

@svrnm are you willing to open a PR and document how a custom image can be built?

[cuichenli]

i will try to document it.

[svrnm]

@pavolloffay , @anuraaga: @cuichenli took care of it, see #762, this looks good to me.