Change default OTLP endpoint to use http:// scheme

[anuraaga]

Changes

Changes the default scheme for OTLP exporters to http:// because the default host is localhost.

With the additional language that SDKs may choose a different default, actually it stops being a breaking change to update this default. I argue that the current default has no use case - it is extremely rare to use TLS with a localhost connection. This prevents the defaults to provide an out-of-the-box experience for any practical situation. It is a better experience if they can provide an OOB experience for something at least. Or otherwise I don't think we would want to have any defaults at all.

While this still seemed relatively harmless, I'm finding that it can cause difficulty in reasoning out other changes, such as in #1895. All the variables should be defined with some target experience in mind - currently there isn't, but the use of localhost seems to strongly indicate a target of sidecar.

[johnnyshields]

See this comment: #1895 (comment)

[anuraaga]

Note, I considered adding some explicit language like "Default values are chosen to provide an out-of-the-box experience for sidecar usage" but left it out. Let me know if something like that makes sense / could be helpful.

[johnnyshields]

@anuraaga I suggest we should mark this PR as a "Draft" until we get clarity on #1895. It seems we need direction as to whether to use Sidecar or Non-sidecar as the default setting, and then raise a single PR which makes it consistent.

[anuraaga]

The point of this PR was to get more clarity to help with #1895 since it's smaller scope, a merge or not merge on this would clear up the path to your PR either way.

[tigrannajaryan]

Note, I considered adding some explicit language like "Default values are chosen to provide an out-of-the-box experience for sidecar usage" but left it out. Let me know if something like that makes sense / could be helpful.

I think it is not necessary because the sidecar is not the only deployment configuration when this is suitable. For example http to localhost is also the correct default for apps deployed on a VM/physical host with a local Collector installed as an agent.

I think it is a reasonable approach to make sure the default values are consistent around the assumption that there is a Collector running on the localhost. However I believe it is important to ensure it is not easy to accidentally send insecurely to a non-local host.

If we accept this PR the defaults will be:

Setting	Default Value	Notes
Endpoint	http://localhost
Protocol	http/protobuf	grpc allowed as a default in SDKs for backward compatibility reasons
Insecure	false	Ignored, because http protocol is used

When the destination is not the localhost (i.e. sending directly to a backend or to an intermediary Collector on a different host) then at least the Endpoint must be specified, which forces the user to choose http or https as the scheme in the URL (assuming they keep the default http/protobuf as the Protocol). This is good, they need to consciously choose a non-secure transport.

If instead they decide to use grpc to send to a non-local host, then they will have to specify at least Endpoint and Protocol. Once the Protocol is switched to grpc, the Insecure=false option kicks in. Again, this is good, can't accidentally send insecurely.

So, it looks good to me.

[anuraaga]

For example http to localhost is also the correct default for apps deployed on a VM/physical host with a local Collector installed as an agent.

Thanks, definitely - I've generally meant both container and non-container uses of localhost when using the term sidecar. One reason is probably because in Java land the word agent as a daemon concept causes some grief vs javaagent so avoid it ;) But we're on the same page I think.