Add logrotation section on Running Filebeat on k8s

[ChrsMark]

What does this PR do?

This PR adds logrotation section on docs about Running Filebeat on Kubernetes. This is part of action items to cover logrotation issues occurring to users running Filebeat on Kubernetes.

Why is it important?

Users that are not aware of logrotation on their k8s clusters might face issue with lost or duplicate events.

@jsoriano @kvch let me know what you think.

[elasticmachine]

Pinging @elastic/integrations (Team:Platforms)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: Pull request #24120 updated

• Start Time: 2021-02-19T11:15:56.876+0000

• Duration: 21 min 25 sec

• Commit: 2c8440e

Trends 🧪

❕ Flaky test report

No test was executed to be analysed.

[jsoriano]

Should we add to the reference config the wildcard recommended in the linked docs?
From:

beats/deploy/kubernetes/filebeat/filebeat-configmap.yaml

Lines 12 to 14 in 96346ad

	- type: container
	paths:
	- /var/log/containers/*.log

To:

    - type: container
      paths:
        - /var/log/containers/*.log*

[jsoriano]

Let the docs builder build the url to the proper version and document, use a placeholder like this one:

Suggested change

	https://www.elastic.co/guide/en/beats/filebeat/current/file-log-rotation.html#file-log-rotation[logrotation specific documentation].
	<<file-log-rotation,log rotation specific documentation>>.

[ChrsMark]

Should we add to the reference config the wildcard recommended in the linked docs?
From:

beats/deploy/kubernetes/filebeat/filebeat-configmap.yaml

Lines 12 to 14 in 96346ad

	- type: container
	paths:
	- /var/log/containers/*.log

To:

    - type: container
      paths:
        - /var/log/containers/*.log*

I don't think it will work. /var/log/containers/ contains symlinks to /var/log/pods/.../....log and then to /var/lib/docker/container/abcd/abcd.log (in case of docker) so a log file inside /var/log/containers/ will only point to the current "active" log file through the symlink. In case of docker, rotated files are under /var/lib/docker/containers/:

ll -la /var/lib/docker/containers/**
...
...
/var/lib/docker/containers/fbc13e9d289fcc15fbc2c60154ec16623e9db6a22f3c53c3f1d83c2c5f63a12e:
total 46816
drwx------  4 root root     4096 Feb 10 02:17 .
drwx------ 50 root root    12288 Feb 10 13:20 ..
drwx------  2 root root     4096 Feb  5 11:25 checkpoints
-rw-------  1 root root    12448 Feb  5 11:25 config.v2.json
-rw-r-----  1 root root  7862893 Feb 10 13:20 fbc13e9d289fcc15fbc2c60154ec16623e9db6a22f3c53c3f1d83c2c5f63a12e-json.log
-rw-r-----  1 root root 10000134 Feb 10 02:17 fbc13e9d289fcc15fbc2c60154ec16623e9db6a22f3c53c3f1d83c2c5f63a12e-json.log.1
-rw-r-----  1 root root 10000076 Feb  9 12:16 fbc13e9d289fcc15fbc2c60154ec16623e9db6a22f3c53c3f1d83c2c5f63a12e-json.log.2
-rw-r-----  1 root root 10000029 Feb  8 22:18 fbc13e9d289fcc15fbc2c60154ec16623e9db6a22f3c53c3f1d83c2c5f63a12e-json.log.3
-rw-r-----  1 root root 10000117 Feb  8 08:33 fbc13e9d289fcc15fbc2c60154ec16623e9db6a22f3c53c3f1d83c2c5f63a12e-json.log.4
-rw-r--r--  1 root root     2233 Feb  5 11:25 hostconfig.json
drwx------  2 root root     4096 Feb  5 11:25 mounts

Switching the manifest to /var/lib/docker/containers/*/*.log* is not a good idea since it will not cover runtimes other than docker 😞 , so better to leave it as is?

[jsoriano]

Switching the manifest to `/var/lib/docker/containers//.log* is not a good idea since it will not cover runtimes other than docker disappointed , so better to leave it as is?

Ok, lets leave it as is, we may need to review this in any case soon after the removal of docker from kubernetes and possible upcoming changes in logs managemeent in K8s.