Postgres 16 client support with wolfi-base

[jasaltvik]

Which image/versions are related to this issue/feature request?

The latest postgres image (ish)

Issue/Feature description

Hi,

I used the postgres image as inspiration to create a really simple Postgres 15 client image like this

FROM cgr.dev/chainguard/wolfi-base

RUN apk update && apk add --no-cache --update-cache glibc-locale-en busybox su-exec bash postgresql-15-client libpq-15

ENV DURATION=30m
CMD sleep ${DURATION}

It's just used as a way for devs to spin up a container that can be used to connect to a Postgres server, and has been working fine.

Then Postgres 16 was released, and I thought I would try to upgrade the client image to 16 as well

RUN apk update && apk add --no-cache --update-cache glibc-locale-en busybox su-exec bash postgresql-16-client libpq-16

This works, but when connecting to the server I get

~ $ psql -U <user>
psql: error: connection to server at "<server>" (<ip>), port 5432 failed: FATAL:  no PostgreSQL user name specified in startup packet
connection to server at "<server>" (<ip>), port 5432 failed: FATAL:  no PostgreSQL user name specified in startup packet
double free or corruption (out)
Aborted (core dumped)

I've not been able to understand the cause of this. Do you have any input on what could be missing to make it work with Postgres 16?

If this issue doesn't belong here, feel free to close it.

Thanks in advance.

[camden-xage]

@jasaltvik I'm experiencing a similar issue, but I'm trying to create a minimal postgres image to run as a server to reproduce without all of our extra "fluff" I'm struggling to get it to throw this error for some reason (e.g. basic postgres image with just a single username/password set up). Are you able to share how you are running your postgres server? Is it just a simple docker setup or something similar?

[mattmoor]

Digging into this we found an issue with the way provider_priority was being handled (it gets dropped when generating the index) which was leading to an older version of libcrypt1 being installed by the solver (r7 vs. r5 for the rest of the stuff from glibc).

We're fixing that regardless, but if you have an easy repro, I am curious whether installing libcrypt1=2.38-r7 explicitly resolves this for you.

ref: chainguard-dev/melange#878

[camden-xage]

@mattmoor I was able to find out that it has to do with TLS/SSL being enabled on the server. I've been going back and forth in a ticket (but did eventually end up in this issue after hours of trying to figure out specific server config was causing the issue lol) and did find a way to reproduce it. Installing that package specifically doesn't seem to help. I'll post the steps here as well for anyone that needs it.

To start, you have to generate a certificate and key and then set the permissions for the postgres user in the container.

openssl genrsa -out server.key 2048
openssl rsa -in server.key -out server.key
openssl req -sha256 -new -key server.key -out server.csr -subj '/CN=localhost'
openssl x509 -req -sha256 -days 365 -in server.csr -signkey server.key -out server.crt

sudo chown 70:70 server.crt server.key
sudo chmod 600 server.crt server.key

Then you can use this compose file to launch postgres using that cert/key and with TLS enabled. The chainguard postgres image can work as the server here, but it has some issues after the first login (likely due to the same issue, it will go into recovery mode due to some tls issue), so just use the docker hub one.

services:
  postgres:
    container_name: postgres
    image: postgres:15.5-alpine
    command: -c ssl=on -c ssl_cert_file=/cert.pem -c ssl_key_file=/key.pem
    ports:
      - "5432:5432"
    volumes:
      - ./server.crt:/cert.pem
      - ./server.key:/key.pem
    environment:
      - POSTGRES_USER=sensify
      - POSTGRES_PASSWORD=password

Even when installing that package you mentioned explicitly it seems to still be an issue

❯ docker run --rm -it cgr.dev/chainguard/wolfi-base:latest
705e3607b3c8:/# apk add --no-cache --quiet libcrypt1=2.38-r7 postgresql-15-client libpq-15

705e3607b3c8:/# apk list --installed | grep -E 'postgres|pq|crypt'
WARNING: opening from cache https://packages.wolfi.dev/os: No such file or directory
libcrypt1-2.38-r7 x86_64 {glibc} (GPL-3.0-or-later) [installed]
libcrypto3-3.2.0-r0 x86_64 {openssl} (Apache-2.0) [installed]
libpq-15-15.5-r3 x86_64 {postgresql-15} (BSD) [installed]
postgresql-15-client-15.5-r3 x86_64 {postgresql-15} (BSD) [installed]

705e3607b3c8:/# psql --host 172.16.3.236 --port 5432 -U sensify
psql: error: connection to server at "172.16.3.236", port 5432 failed: FATAL:  no PostgreSQL user name specified in startup packet
connection to server at "172.16.3.236", port 5432 failed: FATAL:  no PostgreSQL user name specified in startup packet
Segmentation fault (core dumped)

[mattmoor]

@camden-xage One of the other things I noticed diffing the good/bad digests you shared is that the one meaningful dependency change (other than epoch bumps) was openssl went from 3.1 to 3.2, so I am curious (given that this relates to crypto) whether you've tried pinning / downgrading to openssl 3.1?

[mattmoor]

This seems very similar to the issue here (h/t Natacha for the pointer): Homebrew/homebrew-core#155651

[mattmoor]

Yeah, so it definitely seems like Postgres needs a patch to work properly with openssl 3.2.

Thanks for the super detailed repro case above, I'll work with the folks on our side to reproduce this, confirm a fix, and get it turned into a test case for this particular code path. 🙏

[camden-xage]

@mattmoor Downgrading everything to 3.1 works

❯ docker run --rm -it cgr.dev/chainguard/wolfi-base:latest
8ce7743c2d84:/# apk add --no-cache --quiet libcrypt1=2.38-r7 postgresql-15-client libpq-15 openssl~3.1 openssl-config~3.1 openssl-provider-legacy~3.1 libssl3~3.1 libcrypto3~3.1
8ce7743c2d84:/# psql --host 172.16.3.236 --port 5432 -U sensify
Password for user sensify: 
psql (15.5)
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, compression: off)
Type "help" for help.

sensify=# exit

[mattmoor]

Awesome. Seems like we have our smoking gun. Will sync up with our folks tomorrow to get the postgres patch in place, and the above turned into a test case 🤞

[jasaltvik]

@camden-xage / @mattmoor Thanks guys! Nice catch with the openssl upgrade.

@camden-xage I guess this is not very relevant anymore, but the client was used to connect to a Azure Database for PostgreSQL flexible server.

[joshrwolf]

hey folks, I believe the pieces are in place now to close this. since opening, we have:

1. pulled in openssl3.2 patches for postgresql-{12,13,14,15,16}

• 16: wolfi-dev/os@d3d7256
• 15: wolfi-dev/os@c2c9475
• ...

2. added some basic tests to the postgres image to ensure we do not regress on tls again

as the ticket is written (1) should be sufficient. @jasaltvik can you verify and close if so? if not please lmk what else we need 🙏

[jasaltvik]

@joshrwolf Sorry for the late reply. Just verified that the client works now, thanks!