[logstash] align mappings with metricbeat

[klacabane]

Summary

Closes #3916

Align the logstash data streams' mappings with the metricbeat ones. The change also defines a logstash service that can be started by running elastic-package service up in the logstash directory.

Testing

First make sure you have elastic-package installed.

When testing we have to install the elasticsearch package as well otherwise the logstash section in Stack Monitoring won't show up.

• We can automate the package installation by providing the right fleet configuration to kibana, and we can use elastic-package profiles to do that. Let's download a profile that specifically does that. You can skip this step if you want to install the packages manually

curl https://drive.google.com/uc\?export\=download\&id\=18ZhDYugqGUgxT0zE68DQuI9jOdtcvyri -L -o /tmp/stack_monitoring-logstash.zip && \
unzip -o /tmp/stack_monitoring-logstash.zip -d ~/.elastic-package/profiles && \
rm /tmp/stack_monitoring-logstash.zip

• Now cd at the root of the integrations repository and let's build elasticsearch/logstash package, start the stack with the downloaded profile and also start a logstash service with some predefined pipelines. The command may take a moment to complete and will be done once the logstash service is started, this log message should appear: Service is up, please use ctrl+c to take it down

(cd packages/elasticsearch && elastic-package build) && \
(
  cd packages/logstash && elastic-package build && \
  elastic-package stack up -v -d --profile stack_monitoring-logstash --version 8.5.0-SNAPSHOT && \
  elastic-package service up -v
)

• load kibana at https://localhost:5601, navigate to Stack Monitoring and inspect logstash section. These two pipelines should show up under the elasticsearch cluster while the standalone-pipeline should appear in the Standalone cluster.
  Note: If the Standalone Cluster is not showing up it means that this change [Stack Monitoring] support logstash datastream in standalone query kibana#138689 is not available in 8.5 snapshot yet. You can follow the Testing steps from this PR [Stack Monitoring] support for integration package kibana#138224 to connect a local kibana containing the changes

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-08-16T09:32:05.885+0000

• Duration: 17 min 56 sec

Test stats 🧪

Test	Results
Failed	0
Passed	15
Skipped	0
Total	15

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (0/0)	💚
Files	100.0% (0/0)	💚 2.879
Classes	100.0% (0/0)	💚 2.879
Methods	42.857% (6/14)	👎 -47.201
Lines	100.0% (0/0)	💚 8.173
Conditionals	100.0% (0/0)	💚

[matschaffer]

We can automate the package installation by providing the right fleet configuration to kibana, and we can use elastic-package profiles

That's a really interesting step. Nice work figuring out how to package/distribute the profile. It makes me wonder if there should be something a little more "first class" for automating installation (for example elastic-package stack install logstash). But not something I'd consider in-scope for this PR.

[matschaffer]

Also scratching my head at what coverage counting tool is flagging -46.357 for a PR that's all configuration as far as I can see.

[matschaffer]

My initial attempt to run your instructions didn't give me the agent package, just the filebeat/metricbeat "instructional" (? not sure if right word) integrations.

Trying this:

 ~/elastic/integrations/packages/logstash
❯ elastic-package build; elastic-package stack up -v -d --services package-registry

(I think I'm also warming regional caches this morning, so far 20min in total for all the docker-compose ups 😆 )

Update: I think I broke it

Guess I'll use this as an excuse to try to learn more about how the package registry works...

[matschaffer]

Looks like that command used a different profile, so kibana was failing to query to refreshed package service:

❯ docker logs elastic-package-stack_kibana_1
[2022-08-16T00:28:50.000+00:00][ERROR][plugins.fleet] Error connecting to package registry: request to https://package-registry:8080/search?experimental=true&kibana.version=8.5.0 failed, reason: self signed certificate in certificate chain

Trying this now:

❯ elastic-package stack up -v -d --profile stack_monitoring-logstash --services package-registry --version 8.5.0-SNAPSHOT

Update: error is gone, still no logstash package. I'm gonna retry with default profiles and see if that changes anything.

[matschaffer]

While waiting for stacks to do stack things I think I found the coverage report https://beats-ci.elastic.co/job/Ingest-manager/job/integrations/job/main/1196/cobertura/logstash/node/

Looks like it's borrowing the cobertura format so Methods here seems to translate to files that get exercised?

[matschaffer]

So I ended up needing to switch the package to release: ga but then I realized, maybe the package was actually installed I just can't see it in the UI. I'll try the description steps again and see if that's the case.

[matschaffer]

I wasn't able to get the elasticsearch package to show up in the UI, so I enabled internal monitoring and was able to see logstash.

The setup mode N/A seems to indicate that the data is coming in via agent.

To cross check I looked at the health api but I think maybe it doesn't support the new indices yet?

Would be nice if there were an easy way to pull a source-clone kibana into the stack to be able to fix and test the code accordingly. (Update: oh nice, you have that in elastic/kibana#138224)

[matschaffer]

Okay, so confusion resolved here. Looks like the profile works, but since they're not ga they only show in "installed integrations" not in the "browse integrations"?

Aside: the ES logo looks broken, so I opened elastic/kibana#138865 to fix

[matschaffer]

After a bit of head-desking this seems to work. I had a couple of questions, so just commenting for now.

I'd also like to see if I can test CFS & config reloading to get the reload counters or pipeline versioning to do something other than just be blank.

[matschaffer]

Any idea if we can parameterize this to the latest stack snapshot? Thinking something like what we do on the beats repos.

[klacabane]

Would that be the master-SNAPSHOT image ?

[matschaffer]

Worth a try at least :) I did the same in https://github.com/elastic/beats/blob/main/docs/devguide/modules-dev-guide.asciidoc#test

I think my ideal would be some sort of automation like elastic/beats#32720 that explicitly bumps to updated versions. That way we avoid spurious main branch failures on this repo if something is broken in an updated stack version.

Either way I think we can defer that work to a later PR.

[matschaffer]

I opened up #4013 to follow up on this and added it to our integration meta.

[matschaffer]

Do these get validated via a test or something? The cgroup graphs are blank when I test it manually, but I think that might be just due to no quota being set. Will see if I can add one.

[matschaffer]

Oh, also is there anything I can do with https://github.com/klacabane/metrics-mappings-to-monitoring to cross-check the mapping diffs?

[matschaffer]

heh, I ran into another funny "sharp edge" on elastic-package. Looks like I was running an old docker-compose via pip so I removed that figuring v2 might be easier to set cpu quotas with, but that causes the hostnames to change so it's elastic-package-service-logstash-1 not elastic-package-service_logstash_1

[matschaffer]

Well, I can't seem to get logstash to return any cgroup stats even with a quota set:

❯ docker exec -it $(docker ps -qlf Name=logstash) curl localhost:9600'/_node/stats/os?pretty'
{
  "host" : "5d112bb6717c",
  "version" : "8.5.0",
  "http_address" : "0.0.0.0:9600",
  "id" : "5630eddf-666b-42ec-8085-54b87bab8c13",
  "name" : "5d112bb6717c",
  "ephemeral_id" : "ca910dd5-bf0d-4165-8c74-3713f9c4a891",
  "status" : "green",
  "snapshot" : true,
  "pipeline" : {
    "workers" : 3,
    "batch_size" : 125,
    "batch_delay" : 50
  },
  "os" : { }
}

❯ docker inspect $(docker ps -qlf Name=logstash) --format '{{json .HostConfig.CpuQuota}}'
50000

[matschaffer]

I can see cpu periods/throttling info in /sys so maybe we're just hitting a logstash bug here:

❯ docker exec -it $(docker ps -qf Name=logstash) cat /sys/fs/cgroup/cpu.stat
usage_usec 127507926
user_usec 119128284
system_usec 8379642
nr_periods 39180
nr_throttled 465
throttled_usec 6879286

[matschaffer]

Also confirmed standalone using your instructions from elastic/kibana#138224 - thanks for that! Thinking we should probably start an integrations.md dev doc for stack monitoring to capture some of these techniques.

[klacabane]

My initial attempt to run your instructions didn't give me the agent package, just the filebeat/metricbeat "instructional" (? not sure if right word) integrations.

When not GA the integrations won't appear in the UI, but you can still find them with the global search bar. if you type elasticsearch/kibana/logstash it should show an integration option. You can also navigate to the integration through the metricbeat tutorial, they have a link to the agent package.

Looks like that command used a different profile, so kibana was failing to query to refreshed package service:

Ah got bit by that several time and added it to the (still local) docs. profile have their own certificates and CAs so the package-registry needs to be started with the --profile as well to be able to connect back.

To cross check I looked at the health api but I think maybe it doesn't support the new indices yet?

Correct, health api does not support metrics-* yet

Well, I can't seem to get logstash to return any cgroup stats even with a quota set

Let me confirm that on my side and I'll create a ticket

Do these get validated via a test or something?

We can implement system tests (https://github.com/elastic/elastic-package/blob/main/docs/howto/system_testing.md) that verifies that the mapped properties are correctly ingested. I wanted to push their implementation a bit later since I expect those to fail and spot properties that are not populated (see cgroup) and increase the scope of the PR. Our testing strategy for now is to rely on Stack Monitoring UI behaving similarly when loaded with agent and metricbeat, then we can add these tests as a mapping testing and optimization ticket, wdyt ?

[matschaffer]

if you type elasticsearch/kibana/logstash it should show an integration option. You can also navigate to the integration through the metricbeat tutorial, they have a link to the agent package.

ahhh... big TIL on that one for me. Thanks!

Our testing strategy for now is to rely on Stack Monitoring UI behaving similarly when loaded with agent and metricbeat, then we can add these tests as a mapping testing and optimization ticket, wdyt ?

Makes sense to me. Looks like #4008 has that covered 👍🏻

[matschaffer]

Good to merge I think. Definitely some outstanding questions, but we can address them as work carries on in the main meta(s).