Basic Auth causes Caddy to use an extraordinary amount of CPU while viewing the NetData Dashboard

[douglasparker]

Environment

Operating System

OS: Ubuntu 20.04.2 LTS
CPU: 8 Cores
RAM: 8 GB RAM
Swap: 4 GB Swap
Disk: 160 GB NVMe SSD

Applications

Caddy: v2.0.0-rc.3 h1:z2H/QnaRscip6aZJxwTbghu3zhC88Vo8l/K57WUce4Q=
NetData: v1.31.0

Configuration

Caddy - Caddyfile

https://monitor.douglas-parker.com {

    basicauth * {
        douglasparker <redacted>
    }
        reverse_proxy 127.0.0.1:19999
}

NetData - netdata.conf

[global]
    debug log = none
    error log = none
    access log = none

    memory mode = dbengine
    dbengine multihost disk space = 64
    page cache size = 32
    update every = 1

[web]
    enable gzip compression = no

Summary

Caddy will use extreme amounts of CPU usage when you enable basic auth and have many requests being made. In my case, CPU usage spikes to 75%-100% while I am viewing the NetData Dashboard. I can only assume that this has to do with the overhead of the bcrypt algorithm plus many metrics being updated in the dashboard in real time.

I had assumed that this was resolved with commit cc81de9 which resolved issue #3462, but it appears that this is still a major issue with basic auth.

Workarounds

I can work around this issue by lowering the frequency that metrics are collected. For example, updating the dashboard every 5 seconds instead of every 1 second. However, there is still a massive CPU spike when the data is finally sent.

Disabling basic auth completely resolves this issue, but isn't a viable solution for many.

Switching to another web server such as nginx also resolves this issue.

Screenshots

Basic Auth Disabled

Basic Auth Enabled

[francislavoie]

Caddy: v2.0.0-rc.3 h1:z2H/QnaRscip6aZJxwTbghu3zhC88Vo8l/K57WUce4Q=

That's a very old version. Please use v2.4.3.

The fix to basicauth performance landed in v2.1.0. You're using an older version than that.

[douglasparker]

@francislavoie I am using the official Caddy docker image. Docker Hub says that caddy:latest was last updated 4 days ago.

docker-compose.yml

services:
  caddy:
    image: caddy/caddy:latest
    container_name: caddy
    network_mode: host
    volumes:
      - /opt/caddy/Caddyfile:/etc/caddy/Caddyfile:ro
      - /opt/caddy/data:/data
      - /opt/caddy/config:/config
      - /var/www:/var/www
    restart: unless-stopped

root@node01:/opt/netdata# docker exec caddy caddy version
v2.0.0-rc.3 h1:z2H/QnaRscip6aZJxwTbghu3zhC88Vo8l/K57WUce4Q=

[francislavoie]

caddy/caddy is not the official docker image, that's our CI target. Use the caddy image, which is the official image. Also, make sure to run docker pull to get the latest docker images. See the docs on Docker Hub https://hub.docker.com/_/caddy

[douglasparker]

caddy/caddy is not the official docker image, that's our CI target. Use the caddy image, which is the official image. Also, make sure to run docker pull to get the latest docker images. See the docs on Docker Hub https://hub.docker.com/_/caddy

Good to know, thank you. I might suggest that the documentation on docker hub make this clear.

As you can see, it states that 2.4.3 is a shared tag with latest.

I will update the docker image and then update with my results here.

Thanks again!

[douglasparker]

Just a quick update, updating to the latest caddy version does indeed resolve this issue. Thank you so much for the clarification.

I highly recommend making this clear on docker hub. In my experience the latest tag tends to be the latest stable release, and docker hub even explicitly states that it is a shared tag with the latest version. This will lead to a lot of confusion, and even worse many people running outdated versions of caddy that may have security vulnerabilities.

[francislavoie]

Good to know, thank you. I might suggest that the documentation on docker hub make this clear.

That documentation is auto-generated by Docker's tooling. We can't change it.

If you use a shared tag, then it's up to you, the user, to run docker pull to periodically keep your pulled images up to date. Docker doesn't pull images for you on its own.

Using :latest is an anti-pattern, and not recommended. You should always target a specific tag (pin versions), else you risk breaking changes in the container to cause an unattended outage (should you automate pulling new versions of containers)

[douglasparker]

If you use a shared tag, then it's up to you, the user, to run docker pull to periodically keep your pulled images up to date. Docker doesn't pull images for you on its own.

If you pull caddy/caddy:latest, it pulls the extremely outdated version, which I thought was odd.

Using :latest is an anti-pattern, and not recommended. You should always target a specific tag (pin versions), else you risk breaking changes in the container to cause an unattended outage (should you automate pulling new versions of containers)
Most of my services are non production. I have backups and monitoring setup for the off chance that something breaks.

It's a risk trade off that I am willing to take. I am not a business running a production environment. I need to stay up to date with minimal input from me.

EDIT: I suppose it might be wise to pin the major version at least.

[francislavoie]

If you pull caddy/caddy:latest, it pulls the extremely outdated version, which I thought was odd.

We don't tag latest on that repo (I suppose since v2.0.0-rc.3). Like I said, it's our CI target, it should not be used by the public.

[douglasparker]

If you pull caddy/caddy:latest, it pulls the extremely outdated version, which I thought was odd.

We don't tag latest on that repo (I suppose since v2.0.0-rc.3). Like I said, it's our CI target, it should not be used by the public.

Okay, I appreciate your help in identifying what was wrong. I am very happy with the basic auth performance now.

[powerman]

I have this issue on latest caddy:2.6.2. Here is what happens on caddy container restart with basic auth enabled:

And here is same 1-minute window with caddy restart without basic auth:

[mholt]

@powerman Could you please open an issue and fill out this bug report template? It will help us identify whether it is a bug or simply a configuration issue.

Template

## 1. Environment

### 1a. Operating system and version

```
paste here
```


### 1b. Caddy version (run `caddy version` or paste commit SHA)

```
paste here
```


### 1c. Go version (if building Caddy from source; run `go version`)

```
paste here
```


## 2. Description

### 2a. What happens (briefly explain what is wrong)




### 2b. Why it's a bug (if it's not obvious)




### 2c. Log output

```
paste terminal output or logs here
```



### 2d. Workaround(s)




### 2e. Relevant links




## 3. Tutorial (minimal steps to reproduce the bug)

Instructions -- please heed otherwise we cannot help you (help us help you!)

1. Environment: Please fill out your OS and Caddy versions, even if you don't think they are relevant. (They are always relevant.) If you built Caddy from source, provide the commit SHA and specify your exact Go version.

2. Description: Describe at a high level what the bug is. What happens? Why is it a bug? Not all bugs are obvious, so convince readers that it's actually a bug.

   • 2c) Log output: Paste terminal output and/or complete logs in a code block. DO NOT REDACT INFORMATION except for credentials. Please enable debug and access logs.
   • 2d) Workaround: What are you doing to work around the problem in the meantime? This can help others who encounter the same problem, until we implement a fix.
   • 2e) Relevant links: Please link to any related issues, pull requests, docs, and/or discussion. This can add crucial context to your report.

3. Tutorial: What are the minimum required specific steps someone needs to take in order to experience the same bug? Your goal here is to make sure that anyone else can have the same experience with the bug as you do. You are writing a tutorial, so make sure to carry it out yourself before posting it. Please:

   • Start with an empty config. Add only the lines/parameters that are absolutely required to reproduce the bug.
   • Do not run Caddy inside containers.
   • Run Caddy manually in your terminal; do not use systemd or other init systems.
   • If making HTTP requests, avoid web browsers. Use a simpler HTTP client instead, like curl.
   • Do not redact any information from your config (except credentials). Domain names are public knowledge and often necessary for quick resolution of an issue!
   • Note that ignoring this advice may result in delays, or even in your issue being closed. 😞 Only actionable issues are kept open, and if there is not enough information or clarity to reproduce the bug, then the report is not actionable.

Example of a tutorial:

Create a config file:

{ ... }

Open terminal and run Caddy:

$ caddy ...

Make an HTTP request:

$ curl ...

Notice that the result is ___ but it should be ___.