[Metricbeat][Kubernetes] Remove repeated KSM metrics files

[constanca-m]

What does this PR do?

1. Remove all kube state metrics files from each state_* metricset folder testdata.
2. Rename the KSM files inside kubernetes/_meta/test so they have a prefix that can be used with testdata.
3. Change the default path for the testdata.

Why is it important?

Updating kube state metrics files would require changing it in every state_* metricset. With one general folder, this will no longer be necessary.

Description

Every state_* metricset required the KSM files to be inside testdata if running the testing function TestDataFiles. Changing the config.yml file of each of these metricsets to set the path to one read folder:

type: http
url: "/metrics"
suffix: plain
path: "../_meta/test"

We no longer need to have multiple repeated files. Since the folder kubernetes/_meta/test had previous files in use, these files were renamed to include the extension .plain. This is important as it allows us to use these files for two different testing functions: TestMetricSet and TestDataFiles (which requires a suffix, plain).

A new file was added to the kubernetes/_meta/test: docs.plain. This file has the same content as ksm.v2.7.0.plain. It needs to be inside the folder, so the data.json file inside each state_* metricset gets created (as of now, this file is generated from one that needs to have the name docs.plain). ksm.v2.7.0.plain still exists, so we know the metrics of that KSM version. docs.plain should therefore be updated to be compatible with a version we support.

Checklist

• I have made corresponding change to the default configuration files
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

1. Clone this branch.
2. Run each state_*_test file.

Related issues

• Relates to [Metricbeat] Testing: Allow use of different directory to test data instead of the expected one #34467

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @constanca-m? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-02-21T09:02:21.696+0000

• Duration: 64 min 16 sec

Test stats 🧪

Test	Results
Failed	0
Passed	4335
Skipped	876
Total	5211

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[ChrsMark]

I'm a bit confused why we test this changes with MODULE="kubernetes" mage goIntegTest. This command executes the integration tests but here we are changing the golden files that are used for the unit tests.
So with this command I don't think that these changes are tested at all. See

beats/metricbeat/module/kubernetes/state_service/state_service_test.go

Line 19 in 9e174fa

// +build !integration

.

I also tried to make these tests to break with this change:

diff --git a/metricbeat/module/kubernetes/state_service/_meta/testdata/ksm.v2.4.2.plain-expected.json b/metricbeat/module/kubernetes/state_service/_meta/testdata/ksm.v2.4.2.plain-expected.json
index 6b2d1a1d07..b6a686021e 100644
--- a/metricbeat/module/kubernetes/state_service/_meta/testdata/ksm.v2.4.2.plain-expected.json
+++ b/metricbeat/module/kubernetes/state_service/_meta/testdata/ksm.v2.4.2.plain-expected.json
@@ -14,7 +14,7 @@
                 "cluster_ip": "10.96.0.10",
                 "created": "2023-01-16T14:32:42.000Z",
                 "name": "kube-dns",
-                "type": "ClusterIP"
+                "type": "ClusterIP42"

but without success. @constanca-m could you double check how we should verify these changes in this PR? I guess you need sth like go test -v ./... under the state_service directory, this worked for me.

[ChrsMark]

I tested the changes locally with go test -v ./... (under a metricset directory) and works as expected.

Before merging could you document the details about the docs.plain and ksm.v2.7.0.plain files similarly with what you explain the PR description?

[constanca-m]

I tested the changes locally with go test -v ./... (under a metricset directory) and works as expected.

Before merging could you document the details about the docs.plain and ksm.v2.7.0.plain files similarly with what you explain the PR description?

I tried to look for an appropriate file for that, but I couldn't find any yesterday. I will check today again and commit that change before merging.

[constanca-m]

but without success. @constanca-m could you double check how we should verify these changes in this PR? I guess you need sth like go test -v ./... under the state_service directory, this worked for me.

You are right, that's not the right command. mage build unitTest tests one of the functions, I will update the description accordingly.

[ChrsMark]

Thanks for adding the docs!