Adding beat module with stats metricset

[ycombinator]

This PR adds a new beat module to Metricbeat. It also implements the first (of two) metricsets for this module, stats.

More specifically, this PR:

• Introduces the beat module.
• Implements the stats metricset for this module.
• Implements the non-xpack code path for this metricset.
• Implements the xpack code path for this metricset.
• Introduces the modules.d/beat.yml and modules.d/beat-xpack.yml module configuration files.
• Implements integration tests for the module.
• Implements system tests for the module.

Testing this PR

X-Pack code path (for Stack Monitoring UI)

1. Build metricbeat with this PR.

   cd $GOPATH/src/github.com/elastic/beats/metricbeat
   mage build
   

2. Using Kibana > Console, clear out existing Beats monitoring data.

   DELETE .monitoring-beats-*
   

3. Start Metricbeat with internal collection.

   ./metricbeat -e -E monitoring.enabled=true
   

4. Wait 10 seconds or so for monitoring data to be collected.

5. Using Kibana > Console, retrieve the latest document from the .monitoring-beats-* index and save it.

   POST .monitoring-beats-*/_search?filter_path=hits.hits._source
   {
     "query": {
       "bool": {
         "filter": {
           "term": {
             "type": "beats_stats"
           }
         }
       }
     },
     "sort": [
       {
         "timestamp": {
           "order": "desc"
         }
       }
     ]
   }
   

6. Stop Metricbeat.

7. Using Kibana > Console, clear out existing Beats monitoring data.

   DELETE .monitoring-beats-*
   

8. Start Metricbeat with HTTP API enabled for external collection.

   ./metricbeat -e -E http.enabled=true
   

9. In a new window enable the beats-xpack Metricbeat module for external collection and start this Metricbeat instance.

   ./metricbeat modules enable beat-xpack
   ./metricbeat -e
   

10. Wait 10 seconds or so for monitoring data to be collected.

11. Using Kibana > Console, retrieve the latest document from the .monitoring-beats-* index and save it.

    POST .monitoring-beats-*/_search?filter_path=hits.hits._source
    {
      "query": {
        "bool": {
          "filter": {
            "term": {
              "type": "beats_stats"
            }
          }
        }
      },
      "sort": [
        {
          "timestamp": {
            "order": "desc"
          }
        }
      ]
    }
    

12. Compare the JSON document saved from internal collection (step 5) with the JSON document saved from external collection (step 11). You might want to use a tool like jsondiff.com for this purpose.

    The only structural difference between the two documents should be that the externally-collected JSON document should have 7 extra top-level fields: @timestamp, event, ecs, agent, metricset, service, host. These are expected extra fields that are always inserted by Metricbeat. Other than these expected structural differences, the rest of the structure should be identical; however, values of corresponding fields may be different.

[ruflin]

With this the stats endpoint basically becomes a "documented" endpoint and the API and it's content should not change anymore. Perhaps a good time to revised the output we have to check for example if it follows ECS etc. At the same time we use this event already for the monitoring reporting, so we should not break too many things here?

[sayden]

Sorry for the early review. I'm stuck at the airport 😄

[ycombinator]

jenkins, test this

[elasticmachine]

Pinging @elastic/stack-monitoring

[ycombinator]

Reviewers: there is a LOT of new code / files in this PR. However, they are all in service of a single logical end-goal: introducing the beats module with the stats metricset within it. Still, if you'd prefer that I break up this PR into smaller PRs, I'm happy to do that — perhaps the implementation can be one PR and all the test code/files a follow up PR?

[ycombinator]

With this the stats endpoint basically becomes a "documented" endpoint and the API and it's content should not change anymore. Perhaps a good time to revised the output we have to check for example if it follows ECS etc. At the same time we use this event already for the monitoring reporting, so we should not break too many things here?

I would not change this API now, in a minor version. Perhaps we can do that in 8.0.0.

However, what I would do now is make sure the non-xpack code path of the beats/stats metricset indexes documents into metricbeat-* that follow ECS conventions. This can be checked by looking at the generated data.json file.

[cachedout]

I tested this according to the steps outlined by @ycombinator and it worked as described!

[ruflin]

Same as above, beat.* instead?

[ycombinator]

Done.

[ruflin]

Why not use the new testing mechanism here with the golden files?

[ycombinator]

Sorry, I think I already forgot about this 😞. Could you point me to an example module that's using this?

[ruflin]

this should help: https://github.com/elastic/beats/tree/master/metricbeat/mb/testing/data

[ycombinator]

Okay, I started to use this new testdata framework in 627ecf2bb. However, as you will see the generated stats.800.json-expected.json file is bad. It has a error.message field with value as "HTTP error 404 in : 404 Not Found".

Can you help me figure out what I'm doing wrong? Better yet, can you help me figure out how to debug what's going on with this new framework 🎣?

[ycombinator]

Okay, I did some digging and I think I know why this is happening. The beats/stats metricset makes 2 HTTP API calls: GET / and GET /stats.

I'm investigating now how to mock both these calls with the testdata framework.

[ycombinator]

AFAICT, there's no way currently for the testdata framework to handle the scenario where a metricset needs to call multiple HTTP APIs. Can you confirm this?

If this is true, I'd rather not hold up this PR (and the one coming after it for the beat/state metricset: #12615) for this enhancement to be implemented in the new test framework. Once the enhancement is there I can make a follow up PR to use it with these metricsets. WDYT?

[ruflin]

I can confirm unfortunately, see #11425 SGTM to move forward.

[ycombinator]

I've merged #12621 and rebased this PR on master now. @ruflin this PR is ready for another round of code review. In particular, you might want to see #12181 (comment). Thanks!

[ruflin]

Skimmed through it and looks good to me. What I didn't check in detail is naming fields. It could be an opportunity to rethink how we report beats metrics (structure), at the same time this should not hold back this.

[ruflin]

Changelog?

[ycombinator]

Re: API change (naming of fields), I mentioned it earlier in the PR (#12181 (comment)):

I would not change this API now, in a minor version. Perhaps we can do that in 8.0.0.

However, what I would do now is make sure the non-xpack code path of the beat/stats metricset indexes documents into metricbeat-* that follow ECS conventions. This can be checked by looking at the generated data.json file.

[ycombinator]

@ruflin I'm thinking I should wait to include the CHANGELOG entry until the follow up PR to this one is also merged: #12615. Then we will have a complete beat module. WDYT?