Log rotation keeps writing to rotated files

[beiske]

Kibana version: 7.6.0 and 7.6.1

Original install method (e.g. download page, yum, from source, etc.): Elastic Cloud

Describe the bug:

Steps to reproduce:

1. Start a 7.6 deployment on ESS
2. Create some traffic to make Kibana chatty in log files
3. Wait for Kibana to eventually run out of disk space as /app/kibana.log.0 reached 2GB-

Expected behavior:
Rotated files should not go much above the configured size of 10MB

Provide logs and/or server output (if relevant):

ls -lh /app/logs/
total 2.0G
-rw-r--r-- 1 founduser founduser 8.0K Mar 23 09:11 boot.log
-rw-r--r-- 1 founduser founduser 912K Mar 23 09:11 kibana.log
-rw-r--r-- 1 founduser founduser 2.0G Mar 22 17:31 kibana.log.0

Any additional context:

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)

[mshustov]

I see some inconsistencies in rotation behavior.

Kibana rotate files.

• Set a small threshold for logging.rotate.everyBytes. ~ 150kB - 200kb

logging.verbose: true
logging.dest: "/Users/mikhailshustov/work/kibana/logs/kibana.log"
logging.rotate:
  enabled: true
  everyBytes: 200000
  keepFiles: 5

• run Kibana with yarn start
• Kibana generates tons of logs during start and initiate rotation due to this check

  kibana/src/legacy/server/logging/rotate/log_rotator.ts

  Line 81 in 6d5c8ca

  await this._callRotateOnStartup();

• run ls -lh logs/ and note it contains 2 log files: kibana.log & kibana.log.0. Kibana continue writing to both of them.

Kibana does not rotate files.

• Set bigger threshold for logging.rotate.everyBytes. ~ 500kB

logging.verbose: true
logging.dest: "/Users/mikhailshustov/work/kibana/logs/kibana.log"
logging.rotate:
  enabled: true
  everyBytes: 500000
  keepFiles: 5

• run Kibana with yarn start
• Kibana doesn't rotate on start condition

  kibana/src/legacy/server/logging/rotate/log_rotator.ts

  Line 81 in 6d5c8ca

  await this._callRotateOnStartup();

• wait for some time and run ls -lh logs/. You can see that kibana.log grows in size and bigger than everyBytes, but rotation is not happening. Although if you run touch logs/kibana.log, Kibana rotates the file and writes logs only to the newly created file.

I expected logging.rotate.usePolling could help to detect the file size changes, but this check prevents the polling mechanism to be used

kibana/src/legacy/server/logging/rotate/log_rotator.ts

Line 155 in 6d5c8ca

if (this.usePolling && this.usePolling !== this.config.get('logging.rotate.usePolling')) {

.
Changing code to always use polling logic via usePolling: true

kibana/src/legacy/server/logging/rotate/log_rotator.ts

Line 166 in 6d5c8ca

usePolling: this.usePolling,

addresses the issue. But I'm not sure it's the right solution.

@elastic/kibana-operations any ideas what could go wrong? Probably it's not the root of the problem but doesn't allow us to debug it locally.

[mistic]

@beiske after #60956 getting merged, do u think you could try it with

logging.rotate:
....
  usePolling: true

I believe that would make it work as expected 😃

[mshustov]

done in #60956
backported to
7.x: 192d348
7.6: d23a726