Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RPM package sets deprecated logging.dest cli flag #111263

Closed
rudolf opened this issue Sep 6, 2021 · 12 comments
Closed

RPM package sets deprecated logging.dest cli flag #111263

rudolf opened this issue Sep 6, 2021 · 12 comments
Labels
blocker bug Fixes for quality problems that affect the customer experience impact:critical This issue should be addressed immediately due to a critical level of impact on the product. loe:small Small Level of Effort Team:Operations Team label for Operations Team v7.16.0

Comments

@rudolf
Copy link
Contributor

rudolf commented Sep 6, 2021

Description:
The RPM package service description sets the deprecated --logging.dest flag which causes the following problems:

  1. Kibana generates a deprecation warning which users are unable to resolve
    @mshustov suggested setting deprecation.skip_deprecated_settings='logging.dest' to suppress the warning
  2. RPM package keeps writing to rotated log files when using the new logging configuration
    When using the RPM package and the new logging configuration, Kibana will keep writing legacy request logs to log files that have already been rotated (exceeded size or interval duration).

This is because legacy logging is enabled in the service description by setting --logging.dest=
https://github.com/elastic/kibana/blob/7.15/src/dev/build/tasks/os_packages/service_templates/systemd/etc/systemd/system/kibana.service#L18

Versions affected: 7.15, 7.14, 7.13 and probably all of 7.x

@rudolf rudolf added bug Fixes for quality problems that affect the customer experience Team:Operations Team label for Operations Team labels Sep 6, 2021
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-operations (Team:Operations)

@rudolf
Copy link
Contributor Author

rudolf commented Sep 6, 2021

To fix this in 7.x we will probably have to set logging.dest=/var/log/kibana/kibana.log in kibana.yml instead of doing this in the service file. That way the legacy logging system is still the default when installing the RPM, but it's a bit more obvious to users that Kibana is using the legacy logging configuration and that they should remove these legacy configuration options.

@jbudz
Copy link
Member

jbudz commented Sep 6, 2021

We merged a change for this recently - #98213

@rudolf
Copy link
Contributor Author

rudolf commented Sep 7, 2021

Closed in #98213

Any reason we're not backporting that to 7.15.1?

@rudolf rudolf closed this as completed Sep 7, 2021
@rudolf
Copy link
Contributor Author

rudolf commented Sep 28, 2021

#98213 has been reverted in #111966

@rudolf rudolf reopened this Sep 28, 2021
@tylersmalley tylersmalley added 1 and removed 1 labels Oct 11, 2021
@exalate-issue-sync exalate-issue-sync bot added impact:low Addressing this issue will have a low level of impact on the quality/strength of our product. loe:small Small Level of Effort labels Oct 12, 2021
@rudolf rudolf added the v7.16.0 label Dec 2, 2021
@rudolf
Copy link
Contributor Author

rudolf commented Dec 2, 2021

This would affect users in 7.16 because they will get a deprecation warning for using logging.dest but not be able to resolve it without changing the service definition manually.

@rudolf rudolf changed the title RPM package keeps writing to rotated log files when using the new logging configuration RPM package sets deprecated logging.dest cli flag Dec 2, 2021
@rudolf rudolf added impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. blocker labels Dec 2, 2021
@LeeDr
Copy link

LeeDr commented Dec 2, 2021

FYI, on my CentOS 7 install it's here;

[root@localhost vagrant]# grep -n logging.dest /etc/init.d/kibana 
38:args="--logging.dest=/var/log/kibana/kibana.log"

@rashmivkulkarni
Copy link
Contributor

++, FYI: after having changed the logging.dest=/var/log/kibana/kibana.log in my CentOS7_rpm, and upon Kibana restart, it fails to come up.

jbudz added a commit to jbudz/kibana that referenced this issue Dec 2, 2021
OS packages manage Kibana's logging configuration and will be updated
automatically to use the new logging configuration in 8.0. This skips
the critical deprecation error that will be logged by the upgrade
assistant for a package managed configuration.

Related to elastic#111263
Related to elastic#109159
jbudz added a commit that referenced this issue Dec 17, 2021
OS packages manage Kibana's logging configuration and will be updated
automatically to use the new logging configuration in 8.0. This skips
the critical deprecation error that will be logged by the upgrade
assistant for a package managed configuration.

Related to #111263
Related to #109159

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
@akash1810
Copy link

akash1810 commented Jan 24, 2022

I've come up against a problem that I believe is caused by this too.

When running Kibana (v7.16.2) via systemd logs are written in json format to /var/log/kibana/kibana.log via the service file.

I've recently had an incident where Kibana became unavailable due to disk exhaustion - /var/log/kibana/kibana.log was consuming a quarter of all disk space. I'd like to configure log rotation as described here, my kibana.yml looks like this:

logging:
  appenders:
    rolling-file:
      type: rolling-file
      fileName: /var/log/kibana.log
      layout:
        type: pattern
      policy:
        type: size-limit
        size: 100mb
      strategy:
        type: numeric
        pattern: '-%i'
        max: 2
  root:
    appenders:
      - default
      - rolling-file

However, this is not having the desired effect. Whilst /var/log/kibana.log is rotating as desired, /var/log/kibana/kibana.log continues to be written to too, and w/out rotation. In fact I'm now in a worse situation as more disk space is being used with two log files!

Attempts to disable the root logger (logging.root.level: off) has no effect either.

How can one stop Kibana writing logs to /var/log/kibana/kibana.log when running Kibana via systemd? Alternatively, how can one rotate /var/log/kibana/kibana.log using the logging configuration syntax introduced in v7.13?

@exalate-issue-sync exalate-issue-sync bot added impact:critical This issue should be addressed immediately due to a critical level of impact on the product. and removed impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. impact:low Addressing this issue will have a low level of impact on the quality/strength of our product. labels Mar 2, 2022
@tylersmalley tylersmalley removed loe:small Small Level of Effort impact:critical This issue should be addressed immediately due to a critical level of impact on the product. EnableJiraSync labels Mar 16, 2022
@jbudz
Copy link
Member

jbudz commented Mar 16, 2022

Hey @akash1810 sorry you ran into this. I'd recommend removing logging.dest from ExecStart in /usr/lib/systemd/system/kibana.service followed by systemctl daemon-reload.

The logging.dest setting in the service file wasn't implemented as well as it could have been, but we ended up stuck with it for the 7.x release line due to the fix being a breaking change. In 8.0+ we've removed logging settings from the systemd service file and use only kibana.yml.

@jbudz
Copy link
Member

jbudz commented Mar 16, 2022

Related to the deprecated logging.dest flag in use - this still exists in 7.17 due to the new logger outputting logs in a different format. We added more documentation to 7.17 to address this and avoid a breaking change.

#120306
#120273

Closing this out.

@jbudz jbudz closed this as completed Mar 16, 2022
@exalate-issue-sync exalate-issue-sync bot reopened this Mar 16, 2022
@exalate-issue-sync exalate-issue-sync bot added impact:critical This issue should be addressed immediately due to a critical level of impact on the product. loe:small Small Level of Effort labels Mar 16, 2022
@akash1810
Copy link

akash1810 commented Mar 16, 2022

@jbudz thanks for the reply, it's come at a great time too as we have been upgrading our cluster this week! Great to hear the issue is resolved in the v8 release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
blocker bug Fixes for quality problems that affect the customer experience impact:critical This issue should be addressed immediately due to a critical level of impact on the product. loe:small Small Level of Effort Team:Operations Team label for Operations Team v7.16.0
Projects
None yet
Development

No branches or pull requests

7 participants