Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filebeat elasticsearch module sending gc.log even if only server submodule is enabled #30995

Closed
leweafan opened this issue Mar 24, 2022 · 3 comments · Fixed by #32781
Closed
Assignees
Labels
Team:Infra Monitoring UI - DEPRECATED Infrastructure Monitoring UI team - DEPRECATED - Use Team:Monitoring Team:Service-Integrations Label for the Service Integrations team

Comments

@leweafan
Copy link
Contributor

leweafan commented Mar 24, 2022

Bug description

Filebeat elasticsearch module sending gc.log even if only server submodule is enabled and gc submodule is disabled.

/etc/filebeat/modules.d/elasticsearch.yml

- module: elasticsearch
  server:
    enabled: true

  gc:
    enabled: false

  audit:
    enabled: true

  slowlog:
    enabled: true

  deprecation:
    enabled: true

gc.log added cause *.log pattern

/usr/share/filebeat/module/elasticsearch/server/manifest.yml

module_version: 1.0

var:
  - name: paths
    default:
      - /var/log/elasticsearch/*.log
      - /var/log/elasticsearch/*_server.json
    os.darwin:
      - /usr/local/var/lib/elasticsearch/*.log
      - /usr/local/var/lib/elasticsearch/*_server.json
    os.windows:
      - c:/ProgramData/Elastic/Elasticsearch/logs/*.log
      - c:/ProgramData/Elastic/Elasticsearch/logs/*_server.json

ingest_pipeline:
  - ingest/pipeline.yml
  - ingest/pipeline-plaintext.yml
  - ingest/pipeline-json.yml
input: config/log.yml

Files like access.log, slowlog.log and deprecation.log excluded but not gc.log

/usr/share/filebeat/module/elasticsearch/server/config/log.yml

type: log
paths:
{{ range $i, $path := .paths }}
 - {{$path}}
{{ end }}
exclude_files: [".gz$","_slowlog.log$","_access.log$","_deprecation.log$"]
multiline:
  pattern: '^(\[[0-9]{4}-[0-9]{2}-[0-9]{2}|{)'
  negate: true
  match: after

processors:
# Locale for time zone is only needed in non-json logs
- add_locale.when.not.regexp.message: "^{"
- add_fields:
    target: ''
    fields:
      ecs.version: 1.9.0

Solution

gc.log should be added to exclude_files:

exclude_files: [".gz$","_slowlog.log$","_access.log$","_deprecation.log$", "gc.log$"]

Version

Filebeat 7.17.1

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Mar 24, 2022
@leweafan
Copy link
Contributor Author

leweafan commented Mar 24, 2022

Hot fix:

/etc/filebeat/modules.d/elasticsearch.yml

- module: elasticsearch
  # Server log
  server:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:
    input:
      exclude_files: ['gc.log$']

@leweafan
Copy link
Contributor Author

Seems the same issue described a year ago but without solution - #23583

@leweafan
Copy link
Contributor Author

Also pattern *.log will grab _audit.log which is part of audit module and also should be added to exclude_files like access.log which is part of audit module too.

@endorama endorama added the Team:Service-Integrations Label for the Service Integrations team label Apr 12, 2022
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Apr 12, 2022
@matschaffer matschaffer self-assigned this Sep 13, 2022
@matschaffer matschaffer added the Team:Infra Monitoring UI - DEPRECATED Infrastructure Monitoring UI team - DEPRECATED - Use Team:Monitoring label Sep 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Team:Infra Monitoring UI - DEPRECATED Infrastructure Monitoring UI team - DEPRECATED - Use Team:Monitoring Team:Service-Integrations Label for the Service Integrations team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants