OTEL_RESOURCE_ATTRIBUTES does not address scoped environment variables

[RassK]

Theory

If environment variable is used like dictionary and elements position logically in the different scope, then it makes maintaining of OTEL_RESOURCE_ATTRIBUTES very hard.

Example of available scopes: machine level, hosting process (console, web server, etc), process.
Example of elements in different logical scope: service.name, development.environment

In practice

OTEL_RESOURCE_ATTRIBUTES=service.name < this is logically close to the process as possible.
OTEL_RESOURCE_ATTRIBUTES=development.environment < this is logically close to the biggest scope possible (eg machine scope).

The issue presents only if there are multiple scopes defined. The root cause usually is multiple services per hosting instance, typically a VM based host.

In Windows environment there is the IIS web server, that is used to host multiple web application on single server. It was also common practise to have multiple Windows Service based services in the same machine.

The only known fixes to me are:

1. Add new environment variable for problematic high scoped entry (eg OTEL_ENVIRONMENT).
2. Manually merge scopes down to the process level. So each process scope will define development.environment which makes high level changes very error prone (same environment can report different environment names in case of an error). If each website is having their own CI/CD pipeline, it makes individual changes extremely hard.

[pellared]

As rule of thumb I think one env var should be used to set one setting.
It could result in "SDK env var explosion", but I am not sure if it is bad (I think it is easier to use and more flexible).
Therefore I would like to have an env var like OTEL_RESOURCE_DEVELOPMENT_ENVIRONMENT.

We could allow setting resource attributes via following pattern OTEL_RESOURCE_[KEY]=[value] where the
"KEY" would have to be transformed by making it lowercase and replacing _ with .. A similar pattern is used in OTel Java Automatic Instrumentation for disabling instrumentations: https://opentelemetry.io/docs/instrumentation/java/automatic/agent-config/#suppressing-specific-agent-instrumentation

[RassK]

Even if OTEL_RESOURCE_ATTRIBUTES has to be kept, then OTEL_RESOURCE_[KEY]=[value] could be with higher priority.
This should work then very well!

[pellared]

Related #2891