25 使用filebeat收集iis,apache日志，通过grafana显示

本文介绍如何使用 filebeat 收集 iis, apache 的日志送往 elk, 然后通过 Grafana 监控访问状况

安装 filebeat

Linux可以参考：

https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation.html

win可以在下面地址下载压缩包：

https://www.elastic.co/downloads/beats/filebeat

配置 filebeat module

通过package安装的filebeat，档案位置在：

/etc/filebeat

启用 modules

filebeat modules enable apache
filebeat modules enable iis

完成之后，可以在 modules.d 目录下找到 module 的配置文件：iis.yml, apache.yml，可以配置log路径

示例：

# Module: apache
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.0/filebeat-module-apache.html

- module: apache
  # Access logs
  access:
    enabled: true
    var.paths: ["/var/log/apache2/access.log*","/etc/filebeat/apachelog/access.log*"]
    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    # var.paths: ["/proj/mtk40329/filebeat/apachelog/*.log"]

  # Error logs
  error:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

配置 filebeat

filebeat的配置文件为 filebeat.yml

需要配置的参数主要为：

• reload.enabled: true，可以重新读取配置信息
• output.elasticsearch.hosts，elk地址
• setup.ilm，配置index名称及rollover后缀

#============================= Filebeat modules ===============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: true

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["xxx:9200"]

#============================== Setup ILM =====================================

# Configure Index Lifecycle Management Index Lifecycle Management creates a
# write alias and adds additional settings to the template.
# The elasticsearch.output.index setting will be replaced with the write alias
# if ILM is enabled.

# Enabled ILM support. Valid values are true, false, and auto. The beat will
# detect availabilty of Index Lifecycle Management in Elasticsearch and enable
# or disable ILM support.
setup.ilm.enabled: auto

# Configure the ILM write alias name.
setup.ilm.rollover_alias: "access.xxxweb"

# Configure rollover index pattern.
setup.ilm.pattern: "001"

对于支持ILM(index lifecycle management)的elk，想要更改filebeat创建的index名称，配置output.elasticsearch.index是不起作用的，需要修改ILM的配置。

启动 filebeat

可以通过 service filebeat start，或者 filebeat run 来启动。

配置 Grafana

配置好 Grafana 的 elk search source 源之后，就可以通过 dashboard 查看 filebeat 送往 elk 的日志了。

apache

获取 apache 的 pv:

_index:access.xxxweb* AND url.original:*.html AND NOT http.response.status_code:401

获取 apache 的 uv:

_index:access.xxxweb* AND url.original:*.html AND NOT http.response.status_code:401

metric修改为 source.ip 的unique count

获取热门page：

_index:access.xxxweb* AND url.original:*.html AND NOT http.response.status_code:401

groupby: Terms, field: url.original, top: 10

iis

获取 iis 的 pv:

_index:access.xxxweb* AND url.path:*.html AND NOT http.response.status_code:401

获取 iis 的 uv:

_index:access.xxxweb* AND url.path:*.html AND NOT http.response.status_code:401

metric修改为 user.name 的unique count

获取热门page：

_index:access.xxxweb* AND url.path:*.html AND NOT http.response.status_code:401

groupby: Terms, field: url.path, top: 10

参考

• filebeat
• ILM
• elasticsearch-output