Introduce Beat CRD #3041

david-kow · 2020-05-12T08:08:38Z

This PR introduces first iteration of CRD for Beats and it's updated to reflect most recent changes.

Look & feel

Filebeat

$ kubectl get beat filebeat-sample
NAME              HEALTH   AVAILABLE   EXPECTED   TYPE       VERSION   AGE
filebeat-sample   green    3           3          filebeat   7.6.2     2m7s

$ kubectl tree beat filebeat-sample
NAMESPACE  NAME                                                             READY  REASON  AGE
default    Beat/filebeat-sample                                             -              19s
default    ├─DaemonSet/filebeat-sample-beat-filebeat                        -              16s
default    │ ├─ControllerRevision/filebeat-sample-beat-filebeat-7f7667d58b  -              16s
default    │ ├─Pod/filebeat-sample-beat-filebeat-pnx85                      True           16s
default    │ ├─Pod/filebeat-sample-beat-filebeat-wmnvb                      True           16s
default    │ └─Pod/filebeat-sample-beat-filebeat-x5rtf                      True           16s
default    ├─Secret/filebeat-sample-beat-es-ca                              -              17s
default    ├─Secret/filebeat-sample-beat-filebeat-config                    -              17s
default    ├─Secret/filebeat-sample-beat-user                               -              17s
default    └─ServiceAccount/elastic-operator-beat-filebeat-sample-d7fb      -              17s

Heartbeat

$ kubectl get beat heartbeat-sample
NAME               HEALTH   AVAILABLE   EXPECTED   TYPE        VERSION   AGE
heartbeat-sample   green    1           1          heartbeat   7.6.2     10h

$ kubectl tree beat heartbeat-sample
NAMESPACE  NAME                                                          READY  REASON  AGE
default    Beat/heartbeat-sample                                         -              10h
default    ├─Deployment/heartbeat-sample-beat-otherbeat                  -              10h
default    │ └─ReplicaSet/heartbeat-sample-beat-otherbeat-c8645b56       -              10h
default    │   └─Pod/heartbeat-sample-beat-otherbeat-c8645b56-cr4f8      True           10h
default    ├─Secret/heartbeat-sample-beat-es-ca                          -              10h
default    ├─Secret/heartbeat-sample-beat-otherbeat-config               -              10h
default    ├─Secret/heartbeat-sample-beat-user                           -              10h
default    └─ServiceAccount/elastic-operator-beat-heartbeat-sample-8d7e  -              10h

Features

native support for Filebeat and Metricbeat
initial support for other Beats
support for daemonsets and deployments
providing Beat status
basic webhook/non-webhook validation
rollout for config/version changes
labelling for all created resources
support for pod template/config overriding
basic apm/events/logging
RBAC for autodiscovery setup (can be disabled via operator flag, separate issue at Improve setup of ClusterRole and ClusterRoleBinding for autodiscover #3134)

Additional notes

State

This is the first pass, meta-issue was created (#3040) for additional context and work tracking. There is couple of open issues to discuss PSP/SCC treatment (#3133), default/merging config (#3042) and autodiscover RBAC (#3134).

Testing

I was able to run Filebeat, Metricbeat and Heartbeat successfully, change config, upgrade versions. Some E2E are added for couple basic scenarios and SmokeTest runs with Filebeat now. Tested on OpenShift as well. There is a facility introduced that helps with querying ES and checking whether Beats events really appear in ES. In E2E tests autodiscover RBAC is managed by the operator, while the correct PSP is granted to the operator similarly to other stack products. This will be improved to grant additional permissions only to Beat pods and avoid escalating operator privileges unnecessarily.

Also, at some point, tested manually with:

Filebeat config targeting specific namespaces, labels
output overriden
multiple Beats, multiple Beats per namespace

Beat support

Only filebeat and metricbeat are supported natively. Other beats can be deployed, but only when both image and config are provided.

Config handling

As this deserves an issue of it's own to discuss (#3042), I'll just sum up what is currently in the code: if no spec.config is provided we use a static, default config from controller/common/beat/filebeat/config.go, otherwise we use the provided config. The only merging we do is the output if elasticsearchRef was provided.

cmd/manager/main.go

barkbay

Just did a first review, I'll do some additional tests.

config/operator/global/rbac.yaml

config/samples/beat/filebeat_es_kibana.yaml

pkg/apis/beat/v1beta1/beat_types.go

pkg/apis/beat/v1beta1/webhook.go

pkg/controller/common/beat/otherbeat/otherbeat.go

pkg/controller/common/beat/autodiscovery.go

First pass at Beats in ECK, features include: - native support for Filebeat (daemonset only) - initial support for other Beats (deployment only) - RBAC for autodiscovery setup (can be disabled via operator flag) - providing Beat status - basic webhook/non-webhook validation - rollout for config/version changes - labelling for all created resources - support for pod template/config overriding - basic apm/events/logging

sebgl

Great work! The code is very clean now with the recent refactoring 👍
I left some comments, mostly for minor cosmetic changes. I'm fine with merging early and creating follow-up issues where necessary.

cmd/manager/main.go

sebgl · 2020-06-05T09:43:05Z

config/crds/all-crds.yaml

+  - JSONPath: .status.expectedNodes
+    description: Expected nodes
+    name: expected
+    type: integer


I think we have a single nodes (corresponding to available nodes) instead of expectedNodes and availableNodes in other CRDs. Let's be consistent?

I think we did want to change that in the past (#1786). I'd be for keeping both and improving the status on other resources. It's following the pattern of built-in resources, gives the users more information and it comes free. I'd leave it as is unless you feel strongly about it. (I'm open to naming changes.)

config/crds/all-crds.yaml

config/dev/elastic-psp.yaml

pkg/controller/common/association/association.go

test/e2e/test/beat/pod_builder.go

test/e2e/test/beat/steps.go

anyasabo · 2020-06-05T20:17:36Z

Just to track from our out of band conversation yesterday, I think we do want to release this as v1alpha1 at first

This is to avoid needing more permissions than the pod has through autodiscovery Improve checks in e2e tests

sebgl

LGTM. Awesome work @david-kow!
I left a few comments again. I think I'd prefer we merge soon and build on top with smaller PRs, rather than iterate here for too long.

test/e2e/test/beat/pod_builder.go

sebgl · 2020-06-08T08:05:34Z

pkg/controller/beat/metricbeat/config.go

-    hosts:
-    - https://${HOSTNAME}:10249
-    metricsets:
-    - proxy


Can you clarify why we remove those? Is it moving us away from https://raw.githubusercontent.com/elastic/beats/7.7/deploy/kubernetes/metricbeat-kubernetes.yaml?

Filebeat doesn't need any API permissions other than autodiscover provides, but Metricbeat does. Removing the modules/metricsets that need those permissions. This will be addressed by implementation of #3190, but not fixing it here to avoid solving the same problem twice. Those modules/metricsets will be brought back in another PR.

config/samples/beat/filebeat_es_kibana_apm.yaml

pkg/controller/beat/common/pod.go

sebgl · 2020-06-08T08:22:05Z

pkg/apis/beat/v1beta1/webhook.go

+				field.NewPath("spec").Child("image"),
+				"Image is required if Beat type is not one of [filebeat, metricbeat]"),
+		}
+	}


Discussed on Slack: it's about what we support or not. If users specify type: packetbeat they may feel like ECK supports running packetbeat, which is not true.

sebgl · 2020-06-08T08:23:07Z

pkg/controller/common/container/container.go

@@ -24,6 +24,8 @@ const (
 	ElasticsearchImage    Image = "elasticsearch/elasticsearch"
 	KibanaImage           Image = "kibana/kibana"
 	EnterpriseSearchImage Image = "enterprise-search/enterprise-search"
+	FilebeatImage         Image = "beats/filebeat"
+	MetricbeatImage       Image = "beats/metricbeat"


Discussed on Slack: it's about what we "support". If the user is allowed to specify type: packetbeat they may feel like we officially support it, which isn't true.

david-kow · 2020-06-08T10:53:12Z

Thanks for reviews everyone!

david-kow added >feature Adds or discusses adding a feature to the product v1.2.0 labels May 12, 2020

david-kow mentioned this pull request May 12, 2020

[META] Beat CRD #3040

Closed

anyasabo reviewed May 12, 2020

View reviewed changes

cmd/manager/main.go Outdated Show resolved Hide resolved

barkbay reviewed May 13, 2020

View reviewed changes

David Kowalski added 14 commits May 21, 2020 06:57

Exclude BeatList from doc generation

b8ee209

Add autodiscovery rbac flag to manager cmd, fix rbac for the operator

c68cfed

Change autodiscover RBAC flag from disable... to enable...

3f0f842

Improve godoc wording

69184cc

Rename beat_controller.go to controller.go

b538333

Fix comment wording

9ae99a4

Fix update path for autodiscover resources

98745b4

Add image validation if Beat type is not well known

b2f898e

Regenerate CRDs and api docs

469f67c

Add metricbeat driver

be128a2

Expand and improve Beat samples

40a35c5

Refactor config and pod vehicle reconcilation

4305780

Make new linter happy

d4c538e

david-kow force-pushed the beat_crd branch 2 times, most recently from 08c6598 to d4c538e Compare May 21, 2020 05:09

David Kowalski added 9 commits May 23, 2020 14:26

Add e2e tests, update smoke test

e88b8c1

Add UTs

20f2af3

Move writing derefed secret to hash to common

df0357c

Split config.go file

566feb8

Set default replica count for unknown beats

897bbfb

Fix setting labels on daemonset

66e29ae

Fix not returning error when association secret key isn't found

fcb8002

Add UTs for beat common and config

b45481f

Fix imports

b2760b1

David Kowalski added 3 commits June 5, 2020 00:50

Clean up comments on Beat type

ac1ff77

Generated files

328a833

Fixes for OpenShift E2E tests

9133ecd

david-kow force-pushed the beat_crd branch from a0bad10 to 9133ecd Compare June 5, 2020 00:29

david-kow requested review from pebrc, sebgl and barkbay June 5, 2020 00:58

david-kow mentioned this pull request Jun 5, 2020

Improve setup of ClusterRole and ClusterRoleBinding for autodiscover #3134

Closed

sebgl mentioned this pull request Jun 5, 2020

Proposal: Beat "presets" #3190

Closed

sebgl reviewed Jun 5, 2020

View reviewed changes

Set template hash label to ClusterRoleBinding

ae69269

David Kowalski added 10 commits June 5, 2020 22:26

Rename roles.yaml to beat-roles.yaml, common.go to driver.go

ad6f674

Conditionally add empty namespace to cached namespaces

8941ce8

Reword comments, logs

22b940b

Fix comments

595610a

Drop 'operator' from autodiscover resource names

ea57b80

Move validating Beat spec to a separate func

ae1e37e

Tighten beat PSP, fix comments, regenerate

84a6345

Unit test derefing secret for config hash

eca6654

Fix beat-roles.yaml file name in Makefile

d793e62

Remove some metricsets from default metricbeat cfg

0ce0903

This is to avoid needing more permissions than the pod has through autodiscovery Improve checks in e2e tests

david-kow force-pushed the beat_crd branch from 1781174 to 0ce0903 Compare June 7, 2020 08:30

sebgl approved these changes Jun 8, 2020

View reviewed changes

PR fixes

4a679c7

pebrc approved these changes Jun 8, 2020

View reviewed changes

david-kow merged commit 2c6b487 into elastic:master Jun 8, 2020

david-kow deleted the beat_crd branch June 8, 2020 10:53

david-kow mentioned this pull request Jun 8, 2020

PSP/SCC for Beats #3133

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce Beat CRD #3041

Introduce Beat CRD #3041

david-kow commented May 12, 2020 •

edited

Loading

barkbay left a comment

sebgl left a comment

sebgl Jun 5, 2020

david-kow Jun 5, 2020

anyasabo commented Jun 5, 2020

sebgl left a comment •

edited

Loading

sebgl Jun 8, 2020

david-kow Jun 8, 2020

sebgl Jun 8, 2020

sebgl Jun 8, 2020

david-kow commented Jun 8, 2020

Introduce Beat CRD #3041

Introduce Beat CRD #3041

Conversation

david-kow commented May 12, 2020 • edited Loading

Look & feel

Filebeat

Heartbeat

Features

Additional notes

State

Testing

Beat support

Config handling

barkbay left a comment

Choose a reason for hiding this comment

sebgl left a comment

Choose a reason for hiding this comment

sebgl Jun 5, 2020

Choose a reason for hiding this comment

david-kow Jun 5, 2020

Choose a reason for hiding this comment

anyasabo commented Jun 5, 2020

sebgl left a comment • edited Loading

Choose a reason for hiding this comment

sebgl Jun 8, 2020

Choose a reason for hiding this comment

david-kow Jun 8, 2020

Choose a reason for hiding this comment

sebgl Jun 8, 2020

Choose a reason for hiding this comment

sebgl Jun 8, 2020

Choose a reason for hiding this comment

david-kow commented Jun 8, 2020

david-kow commented May 12, 2020 •

edited

Loading

sebgl left a comment •

edited

Loading