Log monitoring and alerts

[anandslab]

I am new to the topic of log monitoring. I would like to add something simple to do the following:

1. Check the logs (from various sources) for specific phrases or events
2. If found, send a notification (Telegram, email, etc.)

Does anyone know of something that can do this. I have read around and there is Graylog, Grafana/Loki, etc. Just wondering if there is simpler and opensource/free for homelab use.

EDIT1: Since i already have grafana in the stack, I am wondering if I should just build on it and add Loki and eventually Prometheus for monitoring needs.

Sorry, just looping in few frequent visitors here: @robflate @DennisGaida @shawnthompson @uasdj25 @gzecchi

[robflate]

It’s something I keep meaning to do but never get round to. For 95% of the issues I have, Dozzle & Portainer help me find the problem. Interested in a more complete solution though.

[DennisGaida]

Thanks for looping me in! I'll just put my elasticsearch-stack branch here, which I set up alright, but I tore it down again because in my experience it was freaking slow / lots of resource usage. I have heard that logstash would be what you are asking, but again: I didn't get it running smoothly and easily. Startup was like 30s-60s.

Without further ado (mind you I am using a cleaner named traefik_proxy network name and some other variable names such as $DOCKERDIR or $DOMAIN_INTERNAL [instead of that nondescriptive DOMAIN0], besides that everything should work just like in your config):

  # ELK - Stack
  elasticsearch:
    container_name: elasticsearch
    image: elasticsearch:7.16.3
    restart: unless-stopped
    volumes:
      - $DOCKERDIR/elasticsearch:/usr/share/elasticsearch/data
    deploy:
      resources:
        limits:
          memory: 512M
    networks:
      - default
    environment:
      - discovery.type=single-node
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    # ports:
    #   - 9200:9200
    logging:
      driver: gelf
      options:
        gelf-address: udp://localhost:12201
        tag: "elasticsearch"

  logstash:
    container_name: logstash
    image: logstash:7.16.3
    restart: unless-stopped
    # links:
    #   - elasticsearch
    volumes:
      - $DOCKERDIR/logstash:/etc/logstash
    environment:
      - "LS_JAVA_OPTS=-Xms512m -Xmx512m"
    command: logstash -f /etc/logstash/logstash.conf
    networks:
      - traefik_proxy
      - default
    ports:
      - 12201:12201/udp
    # depends_on:
    #   - elasticsearch
    logging:
      driver: gelf
      options:
        gelf-address: udp://localhost:12201
        tag: "logstash"

  kibana:
    container_name: kibana
    image: kibana:7.16.3
    restart: unless-stopped
    networks:
      - traefik_proxy
      - default
    # ports:
    #   - 5601:5601
    # depends_on:
    #   - elasticsearch
    #   - logstash
    #   - traefik
    labels:
      traefik.enable: "true"
      ## HTTP Routers
      traefik.http.routers.kibana-rtr.entrypoints: "https"
      traefik.http.routers.kibana-rtr.rule: "Host(`kibana.${DOMAIN_INTERNAL}`)"
      ## Middlewares
      traefik.http.routers.kibana-rtr.middlewares: "chain-authelia@file"
      ## HTTP Services
      traefik.http.routers.kibana-rtr.service: "kibana-svc"
      traefik.http.services.kibana-svc.loadbalancer.server.port: "5601"
    logging:
      driver: gelf
      options:
        gelf-address: udp://localhost:12201
        tag: "kibana"

As for Prometheus - yes I have that running and use it quite heavily. I'm using Prometheus with quite a few services: matusnovak/prometheus-smartctl, prom/blackbox-exporter, miguelndecarvalho/speedtest-exporter, netdata/netdata. Especially netdata is insane in the amount of information you get. Also a resource hog. I haven't played with log ingestion yet for Prometheus and now you set me on another path :)

  # Prometheus - monitoring and alerting toolkit
  # https://github.com/prometheus/prometheus
  prometheus:
    image: prom/prometheus
    container_name: prometheus
    <<: *common-keys-unstopped
    networks:
      - traefik_proxy
      - default
    user: ${PUID}:${PGID}
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - $DOCKERDIR/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - $DOCKERDIR/prometheus/data:/prometheus # TODO: Storage policies
    # to enable web admin api:
    # command: [ "--web.enable-admin-api",
    #          "--config.file=/etc/prometheus/prometheus.yml",
    #          "--storage.tsdb.path=/prometheus",
    #          "--web.console.libraries=/usr/share/prometheus/console_libraries",
    #          "--web.console.templates=/usr/share/prometheus/consoles" ]
    labels:
      traefik.enable: "true"
      ## HTTP Routers
      traefik.http.routers.prometheus-rtr.entrypoints: "https"
      traefik.http.routers.prometheus-rtr.rule: "Host(`prometheus.${DOMAIN_INTERNAL}`)"
      ## Middlewares
      traefik.http.routers.prometheus-rtr.middlewares: "chain-authelia@file"
      ## HTTP Services
      traefik.http.routers.prometheus-rtr.service: "prometheus-svc"
      traefik.http.services.prometheus-svc.loadbalancer.server.port: "9090"

As far as "figure out where a problem comes from", logs is one thing and I usually just check those logs first. I would love to do end-to-end request tracing and thought that Jaeger for Traefik would be nice, but that was very complicated to set up and I couldn't get it working easily https://traefik.io/blog/application-request-tracing-with-traefik-and-jaeger-on-kubernetes/.

[PrplHaz4]

As far as "figure out where a problem comes from", logs is one thing and I usually just check those logs first. I would love to do end-to-end request tracing and thought that Jaeger for Traefik would be nice, but that was very complicated to set up and I couldn't get it working easily https://traefik.io/blog/application-request-tracing-with-traefik-and-jaeger-on-kubernetes/.

Most observability stacks require components IN the applications you're monitoring, so true traceability of a single request will be tough to come by without contributing that code to the projects you're hosting.

[DennisGaida]

Totally agreed and logging / tracing is unfortunately VERY different per project. What I meant with request tracing is pretty much network package inspection - with all the web stacks the network traffic can already tell a lot and since everything goes through Traefik it could be a very nice collection point next to just running tcpdump on the host.

[PrplHaz4]

Label-based log indexing (InfluxDB, Loki) requires some level of analysis and config for each log input vs full text search (ElasticStack, Graylog).

Plugins can help provide more consistent structure to the data in both cases, but adding a data source (and being able to find stuff in logs) has been easier for me with ElasticStack (because non-labeled data is still searchable).

If all you need is alerting I'd guess Loki + Grafana would be the "easiest" next step - but may not be the most helpful if you want to troubleshoot issues through your stack (out of the box).

Loki + Prometheus + Grafana also seems to be where most of the medium-sized world is headed.

IMO, SIEM is important for a homelab too - ElasticStack "includes" it and Loki has no plans to support that use case: grafana/loki#2047 (comment)

So, ElasticStack seems to be the only one supporting all my needs in "one" product (logs, metrics, alerts, dashboards, SIEM...etc) but it requires way more care and feeding than anything in a homelab should.

My thoughts (TLDR; No clear "winner"):

ElasticStack w/Beats (instead of LogStash)

• CONS: resource hog, lots to learn, running in containers was an afterthought, tremendous effort to keep this stack running
• PROS: full text search, useful plugins, pre-built dashboards and consistent schema for metrics/logs, flexible alerting, SIEM

InfluxDB v2

• CONS: v2 incomplete for log viewing, no full-text search, each input needs setup
• PROS: pre-built dashboards, lots of useful telegraf plugins, less moving parts

Graylog

• CONS: limited plugins/dashboards, each input needs setup, open core
• PROS: middle of the road in most aspects

Loki (no first-hand experience)

• CONS: each input needs setup, no full-text search, no plans for SIEM integration, BYO dashboards/views
• PROS: easy to scale/setup, easy to run, more relevant in workplace, better license

[anandslab]

Thanks for all the input everyone. A lot of these are way over my head. I think I will try Loki first and see how i feel about it.

[anandslab]

Successfully shipped all of my logs (proxmox host logs, 2 guest machine logs, script logs) to Loki and I can now query them on Grafana. Just need to setup alerts :-)

[DennisGaida]

Wow, you are quick! Or you have a lot of time on your hands ;-)

[anandslab]

Now you are making feel guilty about neglecting some of my other household chores :-(

[robflate]

Looking forward to seeing the setup! For container logs are you just aggregating the logs as seen with docker logs (basically, what you see in Dozzle) or are you also aggregating additional application logs? Plex for example and many other containers I use have lots of other internal logs not exposed to docker logs. Interested in what exactly you're indexing. Thanks.

[Maynardnaze]

Splunk. The driver is already in docker

…

On Wed, Mar 23, 2022, 11:22 AM SmartHomeBeginner ***@***.***> wrote: I am new to the topic of log monitoring. I would like to add something simple to do the following: 1. Check the logs (from various sources) for specific phrases or events 2. If found, send a notification (Telegram, email, etc.) Does anyone know of something that can do this. I have read around and there is Graylog, Grafana/Loki, etc. Just wondering if there is simpler and opensource/free for homelab use. Sorry, just looping in few frequent visitors here: @robflate <https://github.com/robflate> @DennisGaida <https://github.com/DennisGaida> @shawnthompson <https://github.com/shawnthompson> @uasdj25 <https://github.com/uasdj25> @gzecchi <https://github.com/gzecchi> — Reply to this email directly, view it on GitHub <#262>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJIJNRDMBBAMDBR2QQCZDXDVBMZM3ANCNFSM5ROLPVQA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[DennisGaida]

Would you care to elaborate? You just drop a paid product in here without any explanation, reference or anything.

[uasdj25]

For my homelab use, I only roll with Dozzle and manually checking it and Portainer. I've used various for work purposes. Graylog is fairly easy to setup, but a bit cumbersome. I think I'd add in Loki too. Netdata is fantastic, but it will consume a lot of resources.

In general, I think I'm running fewer containers than most here, so it's a bit easier for me to manage. That may change if I convert to cloud/seedbox though.