Java agent file name in Dockerfile not working with OTEL Operator

[riveryc]

As the operator has the agent file hard coded as:
https://github.com/open-telemetry/opentelemetry-operator/blob/main/pkg/instrumentation/javaagent.go#L69

I think the easiest way to adopt this would be changing the default file name to javaagent.jar if possible in
https://github.com/signalfx/splunk-otel-java/blob/main/Dockerfile#L3

So we can directly use your built image with the OTEL operator.

Is there any hard rule we have to use existing name?

[mateuszrzeszutek]

@owais can you take a look at this?

[owais]

Splunk does not officially support the community operator but you can use it with a custom image derived from official Splunk image(s) or configure operator differently.

Splunk does have it's own operator for Splunk OpenTelemetry Collector which supports Java auto-instrumentation. It is in beta though.

1. https://github.com/signalfx/splunk-otel-operator
2. https://github.com/signalfx/splunk-otel-collector-operator#automatically-instrumenting-k8s-pods

[riveryc]

Thanks for the sharing @owais .

Could you please introduce the major difference between opensource version otel operator vs splunk otel operator?
As a Splunk customer, we'd like to leverage the best practise from vendor, however we don't want to be too far away from the open source project, what would be your recommendation?

[owais]

Biggest difference is that Splunk Operator deploys Splunk's distributions of agents (Collector, Java instrumentation, etc) with default configuration that makes more sense for Splunk customers.

Another major difference is the CRDs supported by the operators. Community operator lets people deploy a general purpose otel deployment which can work with any vendor but with considerable effort on behalf of the user (configuration, etc). Splunk operator CRDs are optimized to simplify deployment for Splunk customers as much as possible. For example, basic CRD from community operator looks like the following where YOU have to configure everything by hand.

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: simplest
spec:
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
    processors:

    exporters:
      logging:

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: []
          exporters: [logging]

While as Splunk operator CRD looks like the following:

apiVersion: otel.splunk.com/v1alpha1
kind: Agent
metadata:
  name: splunk-otel
  namespace: splunk-otel-operator-system
spec:
  clusterName: <MY_CLUSTER_NAME>
  realm: <SPLUNK_REALM>

As you can see, Splunk operator knows how to configure the collector and instrumentation for Splunk customers and needs just the most important bits of info from you (realm, clustername and token). Splunk operator knows how many agents to deploy (one agent per host, one collector and a gateway receiver) for ideal user experience. It knows which components to configure on each component and how to make the data flow between them e.g, it will auto-configure instrumentation agents to send data either to agent or collector depending on what is deployed, etc. All this knowledge is encoded into the operator that you'll need to learn on your own if you do a custom deployment (community operator). (note that most of these things can be overridden in the Splunk operator if desired).

Hope this helps.

[riveryc]

@owais that makes sense, and thanks for sharing, but it doesn't block you guys to use a generic name and to be aligned with Open source version at all, right?

Open source: javaagent.jar
Splunk version: splunk-otel-javaagent-all.jar

I don't see changing a file name will make everything extremely incompatible or increase any complexity.
Unless I miss anything from here?

As Splunk's customer, I'd like to use Splunk version operator, however due yours is not ready (v1 or stable) yet, can you make it easier to be consumed by open source version otel operator?

As I can see the filename is the only block here.

We are only using the Instrumentation at the moment to automatically inject the agent into the pod, as it'll have otel collector address specified as part of the configuration, so it doesn't really have any Splunk specific things there.

Do you mind to make it easier for us to use?

Thanks
River

[owais]

A different filename was used on purpose to make it easier to differentiate what is being used. It is the same reason why the JARs are released under different names. Also a lot of other Splunk components/code rely on such such naming scheme and forcing one component (operator) to break away from convention would be very hard to justify. While the change itself is simple enough to make in Docker, it can have a cascading effect on a number of other components.

I think the proper solution to this problem is the community operator allowing users to custom images/agents by not hard-coding the agent names/paths. I do realize that might take time be built into the community operator though. I'll see if we can contribute it as a feature to the project. In the meanwhile, could you use self-made custom images derived from official splunk ones?

FROM quay.io/signalfx/splunk-otel-instrumentation-java:latest

RUN mv /splunk-otel-javaagent-all.jar /javaagent.jar

Then built and publish this image and use it with operator.

[owais]

Looks like the official solution from OpenTelemetry Operator for this problem is also to use a custom image: open-telemetry/opentelemetry-operator#752