A Docker service discovery to Promtail

[jeschkies]

Is your feature request related to a problem? Please describe.
We have use cases when it is not desirable to ship Docker container logs via the Docker driver but with Promtail. See e.g. #2361.

Describe the solution you'd like
We could offer Docker container discovery alongside the Kubernetes service discovery. The discovery would be pointed at the Docker daemon and fetch all containers and then their logs.

Describe alternatives you've considered
It is possible to point Promtail to the Docker logs files. However, the Promtail process would require access to the log files.

[slim-bean]

This work was fairly recently merged into Prometheus: prometheus/prometheus#8629 which is where Promtail gets its service discovery code which I believe could fairly easily be supported in Promtail.

[jeschkies]

This is awesome! Do you think it would be feasible to scrape the logs from the API then?

[durcon]

FYI:

At the moment I use following work-around:

scrape_configs:
- job_name: docker
  static_configs:
  - targets:
      - localhost
    labels:
      job: docker
      host: my-host
      __path__: /var/lib/docker/containers/*/*-json.log
  pipeline_stages:
  - json:
      expressions:
        message:   log
        stream:    stream
        timestamp: time
        attrs:
  - json:
      expressions:
        container_name: 
      source: attrs
  - timestamp:
      source: timestamp
      format: RFC3339Nano
  - labels:
      stream:
      container_name:
  - output:
      source: message  

Disadvantages

• It is not recommended to use Docker's log files, see JSON File logging driver
• Promtail has to run with root user to read Docker's log files.
• container_name label is not added automatically like Loki Docker Driver does, see Labels
• host label is not added automatically like Loki Docker Driver does, see Labels
• JSON is not automatically extracted like Loki Docker Driver does
• Log rotation has to be handled, see Usage.

[stale]

Hi! This issue has been automatically marked as stale because it has not had any
activity in the past 30 days.

We use a stalebot among other tools to help manage the state of issues in this project.
A stalebot can be very useful in closing issues in a number of cases; the most common
is closing issues or PRs where the original reporter has not responded.

Stalebots are also emotionless and cruel and can close issues which are still very relevant.

If this issue is important to you, please add a comment to keep it open. More importantly, please add a thumbs-up to the original issue entry.

We regularly sort for closed issues which have a stale label sorted by thumbs up.

We may also:

• Mark issues as revivable if we think it's a valid issue but isn't something we are likely
  to prioritize in the future (the issue will still remain closed).
• Add a keepalive label to silence the stalebot if the issue is very common/popular/important.

We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task,
our sincere apologies if you find yourself at the mercy of the stalebot.

[jeschkies]

Work is in progress.

[lucasra1]

@jeschkies Sorry if this is the wrong place to post something like this, but as this isn't in the latest release yet, a bug report would be inappropriate I think.

I was just testing the new docker_sd_config. Everything worked like a charm, until I noticed something weird. When I restart a container, from which I scrape logs with the docker_sd_config, promtail seems to not send the logs to loki. All other containers still work fine. When I then restart promtail, all the missing logs get delivered at once. So nothing is missing, but after the restart it seems to stop sending the logs of this specific container to loki. Is this something you can maybe reproduce?

[jeschkies]

Thanks for testing the implementation 🙏 Could you file a new issue?

Interesting. I thought I covered that case. How long did you wait? The new container should be discovered after some time. Did the container change its id? There might be an issue if the container keeps its id.

[lucasra1]

#5259 Created an issue.

• At one point i did wait like 90min and nothing happened. After restarting promtail instantly all logs were present. Even the ones in the 90 min period. Nothing was lost.
• This happens when the container gets restarted, so the ID stays the same...

[durcon]

Today I tried Promtail 2.5.0 and configured it for Docker Containers. Unfortunately, it didn't work. Promtail service didn't start.

Promtail configuration:

  - job_name: docker 
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)'
        target_label: 'container'

Service configuration:

[Unit]
Description=Promtail service
After=network.target

[Service]
Type=simple
User=promtail
Group=promtail

ExecStart=/usr/local/bin/promtail -config.file=/etc/promtail/promtail-local-config.yaml
SyslogIdentifier=promtail
Restart=always

Unix socket:

srw-rw----  1 root          docker           0 Jun 29 08:06 docker.sock

I tested it manually with sudo -u promtail curl --unix-socket /var/run/docker.sock http://localhost/info and it worked.

Groups:

uid=996(promtail) gid=100(users) groups=100(users),4(adm),120(elasticsearch),998(docker)

Unfortunately Promtail always shows that it restarted too quickly. This happens if the configuration is wrong or a permission is missing. A more meaningful error message would help.

Any hint, what I did wrong?

[durcon]

I fixed my problem with Docker configuration, but now I get:

Jul 06 12:05:06 durcon-test promtail[6986]: level=warn ts=2022-07-06T12:05:06.519826266Z caller=target.go:120 target=docker/c83bd24d2db2d661817baffcab544d9efa514364dc7f60d4bcbb4506e801e791 msg="could not transfer logs" written=0 container=c83bd24d2db2d661817baffcab544d9efa514364dc7f60d4bcbb4506e801e791 err="context canceled"

@jeschkies Do you have any hint for me?

[jeschkies]

🤔 that seems like a Loki connection error. Could you try another target to verify that your client setup is correct?

[durcon]

@jeschkies

🤔 that seems like a Loki connection error. Could you try another target to verify that your client setup is correct?

After restarting the docker container it worked. Maybe the message comes if no new logs are available?

Thank you.

[jeschkies]

Ah, that could be. I've never tested it without logs I think.

[Bamboer]

I got the same problem.
Promtail configuration

scrape_configs:
- job_name: docker
  docker_sd_configs:
    - host: unix:///var/run/docker.sock
      refresh_interval: 5s
      filters:
        - name: expose
          values: ["9080/TCP"]
  relabel_configs:
    - source_labels: ['__meta_docker_container_name']
      regex: '/(.*)'
      target_label: 'container'
  pipeline_stages:
  - docker: {}

The Promtail dashboard

I will get containers info if I drop the filters option. but I coud't got the container log file path in the Promtail dashboard even if I configure the path label .

    - source_labels: ['__meta_docker_container_id']
      replacement: /var/lib/docker/containers/$1/*.log
      target_label: __path__

The other problem is seems that the relabel_configs is not work with docker_sd_configs.

[jeschkies]

@Bamboer could you file a new ticket? That'd be a little easier to track.

[Bamboer]

@Bamboer could you file a new ticket? That'd be a little easier to track.

yes 8388

[hanyce]

I fixed my problem with Docker configuration, but now I get:
Excuse me，Can you tell me how you resolved it?

[Bamboer]

I fixed my problem with Docker configuration, but now I get:
Excuse me，Can you tell me how you resolved it?
Here is my solution 8600

I writed a new docker model for promtail to fix container label lose problem.
you can use dockerd_sd_configs model to tail containers logs.

scrape_configs:
- job_name: node-exporter
  dockerd_sd_configs:
    - host: unix:///var/run/docker.sock
      refresh_interval: 30s
  pipeline_stages:
    - drop:
        expression:  .*INFO.*|.*test.*|.*WARNING.*
  relabel_configs:
  - replacement: node-exporter
    target_label: name
  - source_labels: [__meta_docker_container_id]
    target_label: container
  - source_labels: [__meta_docker_container_id, __meta_docker_container_id]
    separator: /
    target_label: __path__
    replacement: /var/lib/docker/containers/$1-json.log
    action: replace
  - source_labels: [__meta_docker_container_label_io_kubernetes_pod_name]
    regex: .*node-exporter.*
    action: keep
  - source_labels: [__meta_docker_container_label_io_kubernetes_docker_type]
    regex: podsandbox
    action: drop