Use elastic-package for APM package development

[ycombinator]

Looking through https://github.com/elastic/apm-server/tree/master/apmpackage, I think many of the steps in the Developer Documentation for the apm package could be updated to use relevant elastic-package commands.

Similarly, the CI configuration for this repository could also run certain elastic-package commands on the apm package. Here are relevant steps from the CI configuration for the https://github.com/elastic/integrations repository that might be useful to incorporate into this repository's CI configuration:

• For static linting of the package contents:
  • https://github.com/elastic/integrations/blob/master/.ci/Jenkinsfile#L69
• For testing the package:
  • https://github.com/elastic/integrations/blob/master/.ci/Jenkinsfile#L70
  • https://github.com/elastic/integrations/blob/f34360ec84f7a9cd41fc5fe6ec2f2642788b6893/.ci/Jenkinsfile#L74-L75
• For collecting test results and logs (for debugging):
  • https://github.com/elastic/integrations/blob/f34360ec84f7a9cd41fc5fe6ec2f2642788b6893/.ci/Jenkinsfile#L82-L85
• For final cleanup:
  • https://github.com/elastic/integrations/blob/master/.ci/Jenkinsfile#L86

[axw]

@ycombinator thanks for opening! I'd appreciate your thoughts on the below.

For static linting of the package contents:

This was done in #4953, I think we're good here.

system tests:

We have a couple of Fleet-integration system tests in https://github.com/elastic/apm-server/blob/master/systemtest/fleet_test.go, and we'll add more over time as the server and package evolves. These tests are part of our existing system test framework, which covers both the Fleet and non-Fleet modes of running APM Server.

The Fleet-integration system tests involve:

1. building and injecting apm-server into the Elastic Agent container
2. injecting the APM integration package into the Package Registry container
3. setting up a policy and waiting for APM Server to be started
4. running functional tests against the APM Server. e.g. send some events using the Elastic APM Go agent, and then checking the events indexed into Elasticsearch are as expected (golden file testing, with test-specific filtering of dynamic fields).

So I think we're covered for system tests. The one thing we don't do with the integration yet is cross check indexed documents with field mappings. We do have unit tests around this, and have #4408 open to reimplement this as a functional test. I've added a note to that to look into using the static tests.

pipeline tests:

I'm not keen on adding separate pipeline tests, preferring to focus on the feature being tested which will indirectly exercise the ingest pipeline. The fact that a feature is implemented in the ingest pipeline vs. in APM Server code is not relevant to users.

asset tests:

I don't understand what these tests are doing. The docs say:

As a package developer, you do not need to do any work to define an asset loading test for your package. All the necessary information is already present in the package's files.

If the package is known to be valid and you don't specify any expectations, what do the asset tests check? Shouldn't we be covered by running elastic-package check?

static tests:

We have some sample events in our docs, but they're not using the elastic-package convention (I think they should, per my comment about using elastic-package to build docs in #4953). Anyway our sample events are golden test files, so I think we're good here too, aside from the note about checking field mappings above.

[jalvz]

Re. docs generation: the APM package lists all its fields with a flag indicating whether that field is ECS or not (we do the same in the main docs, see eg. https://www.elastic.co/guide/en/apm/server/7.11/exported-fields-apm-transaction.html)

@ycombinator @ruflin I imagine that for other integrations most fields are ECS so not sure what value is for others, but would you like to have such ECS flag for all the packages?

[ycombinator]

I'm not keen on adding separate pipeline tests, preferring to focus on the feature being tested which will indirectly exercise the ingest pipeline. The fact that a feature is implemented in the ingest pipeline vs. in APM Server code is not relevant to users.

Sure, no problem. I tend to think of the pipelines tests as unit tests in that they test a smaller unit of the overall feature, but having the pipelines exercised by the broader system tests is cool if that works for you!

asset tests:

I don't understand what these tests are doing.

These tests try to install the package using Fleet APIs and then ensure that all Elastic Stack assets defined by the package (e.g. Kibana visualizations, Kibana dashboards, Elasticsearch ingest pipelines, Elasticsearch ILM policies, etc.) are loaded up into the Stack as expected.

If the package is known to be valid and you don't specify any expectations, what do the asset tests check? Shouldn't we be covered by running elastic-package check?

Not quite. elastic-package check is a shortcut for elastic-package format, elastic-package lint, and elastic-package build. To run the asset tests, you'll want to run elastic-package test asset or just elastic-package test; the latter would run all types of tests, not just asset tests.

[ycombinator]

Re. docs generation: the APM package lists all its fields with a flag indicating whether that field is ECS or not (we do the same in the main docs, see eg. https://www.elastic.co/guide/en/apm/server/7.11/exported-fields-apm-transaction.html)

Thanks for mentioning this. This is neat, IMO, especially since it links over to the ECS field reference!

@ycombinator @ruflin I imagine that for other integrations most fields are ECS so not sure what value is for others, but would you like to have such ECS flag for all the packages?

Actually it's a mix. For metrics data streams the fields tend not to be mostly ECS but for log data streams the fields do tend to be mostly ECS. I'll defer to @ruflin as to whether we want to flag fields in documentation as being ECS or not when it comes to all packages.

[ruflin]

I wasn't aware of the ECS flag but I think that is a pretty neat thing to also advertise ECS. Currently it seems to link to ECS general doc page, would be even nicer to link to the field itself (wishlist).

[ycombinator]

I've created elastic/elastic-package#293 for bringing the ECS field flagging feature to all packages in general.

[mtojek]

I believe we should figure out first a place for common fields in general, right?

[axw]

We have been using elastic-package for linting, building, and building docs for quite a while now.