The "otlp-http-legacy" port breaks ingresses

[acramsay]

I'm trying to use the otel operator to manage a collector ingress automatically. Unfortunately, the ingress does not work. At least, it does not work when backed by ALB controller in AWS. Here's a relevant part of my otelcol resource:

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata: ...
spec:
  ingress:
    hostname: gateway.redacted.com
    type: ingress
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
            cors:
              allowed_origins:
                - "https://*"

And here are snippets of the service and ingress resources created by the operator.

apiVersion: v1
kind: Service
metadata: ...
spec:
  ...
  ports:
  - appProtocol: grpc
    name: otlp-grpc
    port: 4317
    protocol: TCP
    targetPort: 4317
  - appProtocol: http
    name: otlp-http
    port: 4318
    protocol: TCP
    targetPort: 4318
  - appProtocol: http
    name: otlp-http-legacy
    port: 55681
    protocol: TCP
    targetPort: 4318

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata: ...
spec:
  rules:
  - host: gateway.redacted.com
    http:
      paths:
      - backend:
          service:
            name: opentelemetry-deployment-collector
            port:
              name: otlp-grpc
        path: /otlp-grpc
        pathType: Prefix
      - backend:
          service:
            name: opentelemetry-deployment-collector
            port:
              name: otlp-http
        path: /otlp-http
        pathType: Prefix
      - backend:
          service:
            name: opentelemetry-deployment-collector
            port:
              name: otlp-http-legac
        path: /otlp-http-legacy
        pathType: Prefix

Notice that the service's .spec.ports[2].name is otlp-http-legacy while the ingress' .spec.rules[0].http.paths[2].backend.service.port.name is otlp-http-legac. In other words, the name is truncated in the ingress, missing the final y character.

I believe that I found the source of this issue in collector/reconcile/ingress.go, which limits the name to 15 characters. However, the service is parsed and limited to 63 characters.

Is there a reason that the ingress string is limited to 15 characters? Can we simply change it to 63 characters? Alternatively, can we limit the string in the service resource to the same size?

[acramsay]

Note issue 4565 where it was decided to remove support for the legacy port. Maybe it makes more sense to continue PR 4916 and just remove this port from the ingress and service resources?

[TylerHelmuth]

We should remove the legacy http port stuff.

[pavolloffay]

hi @lodrus did you manage to use Ingress created by the operator in your setup? For example did you mange to report to that ingress OTLP or OTLP HTTP data? If so you could please share your config and any additional config that you applied?

[acramsay]

I did not find a way to use the operator managed ingress.

I have a tiny helm chart to deploy the otel collector resource. I simply added an ingress to that chart as a short term solution until we upgrade to a new release.

Edit: A little clarification. I took this approach because after looking through the operator code, I didn't think it was possible to work around the issue without changing the operator code.

[pavolloffay]

@lodrus could you please point out what changes you needed on the ingress managed by the operator?

[acramsay]

Here's the same yaml I originally posted but I removed ~90% of the content.

apiVersion: v1
kind: Service
spec:
  ports:
  - name: otlp-http-legacy

---

piVersion: networking.k8s.io/v1
kind: Ingress
spec:
  rules:
  - http:
      paths:
      - backend:
          service:
            port:
              name: otlp-http-legac

Note that both objects are created by the otel operator. There are two basic problems here. First is the existence of code to handle legacy ports which is solved by #1971. Unfortunately, this only masks the second issue: the two strings highlighted in this yaml need to match exactly but they do not.

In the service resource, the spec.ports[].name string is limited to 63 characters. But that same string in the ingress resource (spec.rules[].http.paths[[.backend.service.port.name) is truncated by the operator to 15 characters. If the service port name can be as long as 63 characters, then the ingress should allow this as well.

[pavolloffay]

I think the port name in the ingress should not be used, it should rather use the port number to match with the service. WDYT?

[acramsay]

Personally, I think either solution would be fine. I suppose the named port (e.g. otlp-http-legacy) is a bit more of a clear connection to the service. Is either a numbered or named port generally preferred over the other?