Docker API /events Endpoint won't stop streaming

[dangrie158]

Description

The /events/ endpoint has a until parameter that according to the docs

Show events created until this timestamp then stop streaming.

(emphasis mine).

However in recent versions there seems to be a regression where the stream will not be closed by the server. I relied on this behaviour previously so I know it worked at least up until some time back in September 2023.

Specifying any other API version in the URL does not seem to make a difference

Reproduce

1. make sure Docker Desktop is running
   run curl --unix-socket /var/run/docker.sock 'http://localhost/v1.27/events?until=1726805738'

Expected behavior

The stream should be closed (i.e. the curl process should exit)

docker version

Client:
 Version:           27.2.0
 API version:       1.47
 Go version:        go1.21.13
 Git commit:        3ab4256
 Built:             Tue Aug 27 14:14:45 2024
 OS/Arch:           darwin/arm64
 Context:           default

Server: Docker Desktop 4.34.2 (167172)
 Engine:
  Version:          27.2.0
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.21.13
  Git commit:       3ab5c7d
  Built:            Tue Aug 27 14:15:41 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.7.20
  GitCommit:        8fc6bcff51318944179630522a095cc9dbf9f353
 runc:
  Version:          1.1.13
  GitCommit:        v1.1.13-0-g58aa920
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    27.2.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.16.2-desktop.1
    Path:     /Users/griesshaber/.config/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.2-desktop.2
    Path:     /Users/griesshaber/.config/docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.34
    Path:     /usr/local/lib/docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Alpha) (Docker Inc.)
    Version:  v0.0.15
    Path:     /usr/local/lib/docker/cli-plugins/docker-desktop
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/griesshaber/.config/docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.25
    Path:     /usr/local/lib/docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     /usr/local/lib/docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.3.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/griesshaber/.config/docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.13.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-scout

Server:
 Containers: 29
  Running: 7
  Paused: 0
  Stopped: 22
 Images: 48
 Server Version: 27.2.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8fc6bcff51318944179630522a095cc9dbf9f353
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.10.4-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 10
 Total Memory: 15.6GiB
 Name: docker-desktop
 ID: 602950af-5b7d-441a-aad9-54ba1801129d
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/griesshaber/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

Diagnostics ID

1E3E03C8-5A9C-4B69-9376-8FF133A841D2/20240920085405

Additional Info

This was previously reported im moby/moby#48536 but diagnosed to be an issue with the API Proxy within docker desktop.

[thaJeztah]

Thanks! Let me copy over my triage from the moby ticket for easier discoverability;

---

Thanks for reporting; this looks to be a bug in the API proxy that's part of Docker Desktop. Docker Desktop proxies the Docker Engine API to both bridge between the VM (in which the Engine is running), and to perform some transformations that are needed to map resources between the host and the VM (for example, converting paths on your host to their corresponding location inside the VM).

I tried reproducing this issue on Docker Engine runnine on Linux, which didn't show the problem;

curl -s --unix-socket /var/run/docker.sock 'http://localhost/v1.27/events?until=1726805738'
{"status":"create","id":"1c7b60bdf66be86f8abf2a149c8bcf4e30103f01a8cbac4ac162c3ebc20cba08","from":"nginx:alpine","Type":"container","Action":"create","Actor":{"ID":"1c7b60bdf66be86f8abf2a149c8bcf4e30103f01a8cbac4ac162c3ebc20cba08","Attributes":{"image":"nginx:alpine","maintainer":"NGINX Docker Maintainers \u003cdocker-maint@nginx.com\u003e","name":"foo"}},"scope":"local","time":1726768739,"timeNano":1726768739373677981}
{"Type":"network","Action":"connect","Actor":{"ID":"9fa82f4bc8aeb0d04978d20c7271e4ce5f2f3a6cd41a9ee738449c803fb3519c","Attributes":{"container":"1c7b60bdf66be86f8abf2a149c8bcf4e30103f01a8cbac4ac162c3ebc20cba08","name":"bridge","type":"bridge"}},"scope":"local","time":1726768739,"timeNano":1726768739465549183}
{"status":"start","id":"1c7b60bdf66be86f8abf2a149c8bcf4e30103f01a8cbac4ac162c3ebc20cba08","from":"nginx:alpine","Type":"container","Action":"start","Actor":{"ID":"1c7b60bdf66be86f8abf2a149c8bcf4e30103f01a8cbac4ac162c3ebc20cba08","Attributes":{"image":"nginx:alpine","maintainer":"NGINX Docker Maintainers \u003cdocker-maint@nginx.com\u003e","name":"foo"}},"scope":"local","time":1726768739,"timeNano":1726768739739040445}

When doing the same on Docker Desktop (on macOS in my case), I was able to reproduce the issue; curl showed the response but did not disconnect.

To confirm the issue was with the API, I ran a container that bind-mounted both the proxied (/var/run/docker.sock) and the non-proxied (/var/run/docker.sock.raw) API socket;

docker run -it --rm \
    -v /var/run/docker.sock.raw:/var/run/docker.sock.raw \
    -v /var/run/docker.sock:/var/run/docker.sock \
    alpine

Then, inside the container, I installed curl, and tried the same request against both;

apk add --no-cache -qq curl

# connect to the raw (non-proxied) socket;
curl -sS --unix-socket /var/run/docker.sock.raw 'http://localhost/v1.27/events?until=1726805738'
# {"Type":"network",...
# {"Type":"container",,..


# connect to the proxied socket;
curl -sS --unix-socket /var/run/docker.sock 'http://localhost/v1.27/events?until=1726805738'
# {"Type":"network",...
# {"Type":"container",,..
^C