[otel] Add basic configuration samples

[michalpristas]

Written in a way validate passes on them now using placeholders

Closes #4785

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[strawgate]

afaik these will need to follow the format provided in the ticket with the specified placeholders present

[michalpristas]

will we replace placeholders by any means using template engine?
reason i'm asking is that at elastic we always had a good onboarding experience when it comes to default local environment. you just start processes and everything works.
i dont think somebody will do sed on config file to see what's going on, especially not on windows

[strawgate]

indeed, the onboarding flow will replace the placeholders

we will also need to update the placeholders with the infraprocessor included

[strawgate]

I updated the issue #4785 with updated configs that should work with the infra processor, just waiting on if the index names need to change

[michalpristas]

updated configs

[strawgate]

we may remove the logs_index and metrics_index parameters, I guess there's some discussion about having the index pulled from the datastream values: open-telemetry/opentelemetry-collector-contrib#33755

edit: yeah confirmed we'll want to pull the metrics_index and logs_index settings as that is planned to merge shortly

[flash1293]

@strawgate In the snippet for the onboarding flow, we try to get the otel_samples/patformlogs_hostmetrics.yml for both linux and mac - on this PR it looks like it only exists for mac though? What's the right file for linux in the onboarding flow? Is it logs_hostmetrics.yml? Not sure why the difference

[michalpristas]

@flash1293 can you get me everything you need, filenames, content etc.in the meantime i will add what you requested here

[flash1293]

@michalpristas Here is the PR for the onboarding flow: elastic/kibana#183732

The snippet for Linux looks like this:

arch=$(if ([[ $(arch) == "arm" || $(arch) == "aarch64" ]]); then echo "arm64"; else echo $(arch); fi)

curl --output elastic-distro-8.15.0-SNAPSHOT-linux-$arch.tar.gz --url https://snapshots.elastic.co/8.15.0-af44b0a5/downloads/beats/elastic-agent/elastic-agent-8.15.0-SNAPSHOT-linux-$arch.tar.gz --proto '=https' --tlsv1.2 -fOL && mkdir elastic-distro-8.15.0-SNAPSHOT-linux-$arch && tar -xvf elastic-distro-8.15.0-SNAPSHOT-linux-$arch.tar.gz -C "elastic-distro-8.15.0-SNAPSHOT-linux-$arch" --strip-components=1 && cd elastic-distro-8.15.0-SNAPSHOT-linux-$arch 
        
rm ./otel.yml && cp ./otel_samples/platformlogs_hostmetrics.yml ./otel.yml && sed -i 's#<<ES_ENDPOINT>>#https://myinstance.es.eu-west-1.aws.qa.elastic.cloud:443#g' ./otel.yml && sed -i 's/<<ES_API_KEY>>/base64encodedkey==/g' ./otel.yml

it should prepare everything to collect and ship host logs and metrics

[strawgate]

I'm probably missing some context here, the content in the files is a bit repetitive, is there a reason for this?

There should be 2 configs for each platform:

1. Logs and Infrastructure-only Host Metrics
2. Logs, Host Metrics, and Traces to APM

I need to double check if we need to include the resource detection processor in these

[AlexanderWert]

I need to double check if we need to include the resource detection processor in these

Yes, we need the resource detection processor to populate the resource attributes properly.

[flash1293]

@michalpristas In the latest state, these samples now use env variables for ES host and API key: https://github.com/elastic/elastic-agent/pull/5002/files#diff-f10f8ed3393ab509de41e1eaea7fc6bbced0a461041b3fd79ad0eb53165ae030R25

However, the quickstart snippet in the Kibana UI is looking for placeholders: <<ES_ENDPOINT>> and <<ES_API_KEY>> which are replaced with the actual values.

How should we bring them together?
Options:

• Switch it back to <<ES_ENDPOINT>> and <<ES_API_KEY>>
• Change the snippet to replace ${env:ELASTIC_ENDPOINT} and ${env:ELASTIC_API_KEY} with the actual values instead
• Change the snippet to set the env vars via export instead of using sed to search/replace in the file

I would prefer the first option, the second one seems like a weird middle ground and the third one makes the host and api key a little too ephemeral to me - as they are not persisted anywhere, it would be very easy for the user to lose them by closing the terminal session. Writing them to the config file seems like the right call.

Any thoughts?

[michalpristas]

@flash1293 my issue with first option is that it provides terrible experience for user, while env is standard when it comes to otel config, you can validate and use config right away, with weird placeholders we force users to do something that is not standard way of handling configuration.

providing them with sed may work for linux/mac users, but this won't work on windows

i'd be for 2nd or 3rd option and as this is implementation detail on your side, i don't really have a preference, 3rd one being more standard. my goal is to have usable config packed with agent

[flash1293]

@michalpristas fair points, let's go with the second option for now. I will adjust the snippet

[flash1293]

@michalpristas On testing with https://snapshots.elastic.co/8.15.0-af44b0a5/downloads/beats/elastic-agent, I'm getting

failed to build pipelines: failed to create "elasticsearch" exporter for data type "metrics": telemetry type is not supported

is this expected at the moment?

[AlexanderWert]

is this expected at the moment?

Yes, because the other PR with metrics support is not included yet

[strawgate]

providing them with sed may work for linux/mac users, but this won't work on windows

sed won't work, but this is a trivial transformation on Windows with powershell -- I have a powershell script already for this but didn't include it because we aren't launching/testing windows support with this release anyway.

it provides terrible experience for user, while env is standard when it comes to otel config

Can you help me understand why this is a terrible experience for users? I understand it is a less fun experience for developers, but a user is only going to be impacted by this once, as they only go through onboarding once. A user who wants to use environment variables can easily do so?

I also believe it depends on the use-case. I believe that using environment variables for credentials is standard for application environments, but is not standard (would be considered insecure), for endpoint systems.

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

[michalpristas]

i agree on the second point, but I believe these configs are not meant for production environments.
as I understand them is this scenario:
i'm normal dev asked or want to try collection done by elastic, i set up local ES and Kibana and here's agent tar package. they say they have logs, metrics and traces and they provided me even a config file, great.
what do i need to do to have perfect experience? ideally ./elastic-agent otel --config=logs_metrics_traces.yml
if i don't have a default then modifying the file or providing ENV, these are still standard for trying out.

I havent' seen onboarding flow Joe is working on, but copy/paste script generated by that I find fine.
what i would like to have still is effortless experience for that basic scenario I described above, conscious of our out of the box deployment defaults, just like we had with any other product.

so my ideal behavior would require this to be implemented upstream

[michalpristas]

I understood that @flash1293 implemented replace of env, which is great. I enable auto merge and as soon as somebody from my team hit approve, this gets in.

[ycombinator]

I'm +1 to the environment variables approach as it provides a standard way to pass information to the config for replacement while also not precluding sed-style replacements if someone chooses to do that instead.