Post-quantum key agreement X25519Kyber768 disabled in Go 1.23

[bwesterb]

Go 1.23 adds support for and enables post-quantum key agreement:

The experimental post-quantum key exchange mechanism X25519Kyber768Draft00 is now enabled by default when Config.CurvePreferences is nil. The default can be reverted by adding tlskyber=0 to the GODEBUG environment variable.

Go 1.23 doesn't expose a CurveID, so setting CurvePreferences (as Caddy does) disables Kyber.

(The reason for not exposing the CurveID, is that X25519Kyber768 is a preliminary key agreement that doesn't use ML-KEM, the final version of Kyber, that wasn't out when X25519Kyber768 was proposed. X25519Kyber768 will be phased out, but that will takes months if not more than a year.)

I see two options.

1. Ignore X25519Kyber768.
2. Modify Caddy to not set CurvePreferences when the user doesn't specify any curves.

I prefer the second and am happy to write a PR.

cc @sam-bee @FiloSottile

[sam-bee]

I agree that it would be good if Caddy could offer post-quantum cryptography support. Ignoring X25519Kyber768 is also not my preferred option.

For the benefit of anyone joining the conversation, here is some background information:

There is a blog post here explaining why this issue is important, why PQC is being adopted now, and emphasising that adoption is increasingly widespread.

Compiling Caddy with Cloudflare's fork of Go results in working PQC, as demonstrated in this walkthrough. This is a result of @bwesterb's PR to Caddy last year. As mentioned at Gophercon UK, it seems that Caddy compiled with Go 1.23 provides no way of accessing the X25519Kyber768 functionality, because it is unexported in the Go runtime. As noted in this issue above, however, it is the default 'curve' for Go 1.23, so allowing a user of Caddy use Go's defaults would give out-of-the-box access to PQC.

It seems from looking at the Go source code in 1.22.6 that the Go defaults would be X25519, CurveP256, CurveP384, CurveP521, and in 1.23.0 they are x25519Kyber768Draft00, X25519, CurveP256, CurveP384, CurveP521. At present Caddy's default curves are X25519 and CurveP256. Using Go's defaults instead would therefore result in adding PQC where Caddy is compiled with Go 1.23.

I'm afraid I'm not sure at present what behaviour we should prefer where TLS 1.2 has been specified. Should Caddy's default curves depend on the TLS version in future? In the current implementation the two are not interdependent, but I'm unsure whether that would need to change.

[mholt]

Thanks for opening this discussion!

Modify Caddy to not set CurvePreferences when the user doesn't specify any curves.

This sounds logical to me. We currently have a hard-coded default of those which are "fast by design (e.g. X25519) and those for which an optimized assembly implementation exists (e.g. P256)" (so those are the 2 defaults).

I think post-quantum key agreements are worth the tradeoff though, especially since it sounds like it will be a temporary restriction where we can't set CurvePreferences to get it.

[bwesterb]

Yeah, I assume X25519MLKEM768 when it lands in either 1.24 or 1.25 it'll have a CurveID. I think the current Caddy default is great. P-521 is very much overkill and much slower. But it's not that slow it's a huge DoS vector. I'll fashion a PR.

[bwesterb]

Unfortunately Go 1.23 only enables Kyber if the go directive is 1.23 or higher. This is at odds with #6318

[francislavoie]

We prefer to keep two Go versions supported in Caddy to ease the transition for plugins and users, otherwise users may suddenly be stuck having to upgrade Go before they're otherwise ready to. So I think this should be shelved until 1.24 is out when we can bump Caddy's minimum to 1.23 (i.e. roughly 6 months from now). You can bump it in a fork to 1.23 if you need it though, pretty easy to do and you can still use xcaddy to build.

[FiloSottile]

Unfortunately Go 1.23 only enables Kyber if the go directive is 1.23 or higher. This is at odds with #6318

You can force it on with

//go:debug tlskyber=1

The go.mod version just sets the defaults of the GODEBUGs.

See https://go.dev/doc/godebug.

[mholt]

Oh nice, if that ^ works, maybe we can accept this change sooner than 1.24.

[bwesterb]

Seems to work. Thanks @FiloSottile.

[crrodriguez]

This stopped working as the tlskyber experiment ended. breaks compile with go1.24rc1

[mholt]

If that's the case, then I guess we can remove the line and just expect 1.23 users to upgrade for the latest features (which is reasonable). I'll commit it soon.

[bwesterb]

If you revert the changes, you'll need to make a few additional changes to support X25519MLKEM768 in Caddy in 1.24: add X25519MLKEM768 (gated on 1.24) to these two maps.

[mholt]

Yeah; we'll just have to do it behind a build tag if we want to continue supporting older versions of Go.

@francislavoie @hairyhenderson @carlwgeorge Do we still need to support old versions of Go? It makes cutting edge features tedious, and either we have to deal with that tedium or be 6 months behind.

[carlwgeorge]

This will always be a balancing act. The minimum go version determines which platforms can build caddy with the system version of go. The current minimum of 1.22 excludes several prominent distro versions that are still maintained:

• Ubuntu 20.04
• Ubuntu 22.04
• Debian 11
• Debian 12

Raising that minimum to 1.23 will further exclude:

• Ubuntu 24.04
• Fedora 40
• RHEL 8
• RHEL 9 (until Q2 next year, when it is expected to get go 1.23)

I understand the desire to raise the minimum go version, my only ask is that the drawbacks are considered and an intentional decision is made.

[FiloSottile]

The minimum go version determines which platforms can build caddy with the system version of go.

Apologies for the drive-by comment, but Go 1.21 shipped a forwards-compatiblity system. In short, if your go.mod says go 1.23.4 and the system toolchain is too old, it will download the new one and run it.

Now, some distributions disable this for ideological reasons (I don't understand why it'd be any different than downloading other go.mod dependencies, but anyway), but it can be reenabled with GOTOOLCHAIN=auto.

You could just add that to the build instructions, and enjoy compatibility with any Go 1.21+ system, while picking your schedule to upgrade, and only supporting one version at a time ✨

[carlwgeorge]

Distro build systems typically disable all external downloads during the build process, which limits distro packagers to building with the system version of go. Setting GOTOOLCHAIN=auto won't change that. These build systems also don't permit downloading dependencies during the build process, they must be packaged separately or vendored into the source tarball.

[FiloSottile]

I think we are talking about two different things: I thought the question was "can people who installed Go through their package manager install Caddy" and I think the answer is yes with GOTOOLCHAIN=auto. If any distribution disables forward-compatibility so hard that users can't turn it back on for their own builds, we'd like to know so we can plead for them to stop.

I agree that GOTOOLCHAIN=auto doesn't solve the constraints of upstream packagers in building official packages, as those are hermetic. Is that a target Caddy cares about? Aren't those always out of date anyway, since just like they freeze the version of Go they freeze the version of Caddy? It doesn't seem like a documented install method, judging from https://caddyserver.com/docs/install (where all non-rolling distro repos are downstream).

[carlwgeorge]

The question was about setting the minimum version of go allowed to build caddy. I was pointing out that this affects distros that want to ship caddy as a distro package. I'm not talking about people building their own local caddy binary. My comment about GOTOOLCHAIN=auto was specific to distro build systems. It can't work because without internet access ("hermetic") it can't download newer toolchains. I'm not aware of anyone disabling that feature outright.

I hope distro packages is a target the Caddy project cares about. It's why I participate in Caddy, and likely why @mholt tagged me in this issue. I can't speak for other distros, but in Fedora, CentOS, and RHEL packages are not frozen, there is just care given to how and when they're upgraded. A new minimum version of go sets a hard limit for the distro caddy package. It also causes additional work in the Caddy copr (which I maintain on behalf of the Caddy project). The idea is that the default distro package is pretty current but may lag slightly behind to comply with the update policies of the distro, and users can opt-in to the copr if they absolutely need the latest version without delay.

[FiloSottile]

I see, I think we understand each other now, and I agree GOTOOLCHAIN=auto doesn't solve your problem.

My personal opinion is that distros are free to apply the update policies they like, but it should be a given that if the toolchain is old then some of the packages built with it will also be old. Meanwhile backport repos that wish to provide recent packages need some way to use recent toolchains. Otherwise we are pushing extra work (like the work discussed here) onto every upstream project to let non-rolling distros have it both ways. But! This is @mholt's project, not mine, so my opinion is irrelevant and I shall bow out :)

[mholt]

Thanks for chiming in @FiloSottile!

I realize I brought this issue off-topic, technically; but I think it's worth finishing the discussion while we're here.

We run into this issue (of supporting latest-minus-one Go version) every ~6-12 months, and each time we have to jump through hoops to make the program compile on an older version while releasing new features, which often enhance security/privacy. In this case, the latest Go version offers post-quantum protection, as well as ECH; both are huge privacy benefits that we should start gaining production experience for ASAP.

I understand and respect various distro's policies regarding stability and not always jumping into the latest versions of things. Some people consider that a feature. (I do sometimes.) One of our objectives/values is to advance and enhance the state of privacy and security on the Web; so waiting at least 6 months to release features that can do that kind of runs contrary to our objectives.

Sure, we can go to extra work to refactor things, put certain code/features behind build tags, try to document that the feature works on the latest versions available at some repositories but not others, then undo it all 6-12 months later... meanwhile having people think they are getting the feature when they're really not because their choice of distro gave them the latest release that ... doesn't have the feature !?

After a brief discussion with maintainers on Slack, I think we might just drop the "keep working for 11.9-months-old-Go-versions" thing, and only support the latest version of Go if that's what we need to do in order to efficiently release new features/patches.

This means that if someone chooses a distro that values stability and does not offer the latest packages, then they will get a version of Caddy that may be a few months old. But at least it should not look like the latest release since the latest release won't package for those distros until later.

Those are the consequences of choosing a distro with those values.

One thing I want to do in the future is strive to have more affinity between Caddy version and Go version. I think one step in that process.

So, hey, maybe we can get post-quantum and ECH out with 2.9. @francislavoie Are we OK with potentially releasing on a Go RC? (I think I am. That's kind of the point of RC's. We can always release patches later to use the final release.)

@carlwgeorge I greatly appreciate your continued maintenance of Caddy packages for our community. We want to continue that, it just might be that in some cases -- and not all, for we don't always use new Go features right away -- Caddy isn't released for a few months on those distros.

[francislavoie]

@mholt if you mean Caddy RC using Go RC, sure I'm fine with that but if you mean full release of 2.9 I think we should wait until full Go release too.

[mholt]

Fair, I was considering that; the only thing is that very few people, if anyone, use the Caddy pre-releases 😅

I'm actually quite confident in Go's RCs. They recommend using them in production. We can do a 2.9.1 to upgrade to the final Go 1.24 release and include a few of our own fixes with it.

But if there is opposition to the idea, then I'm flexible on this. I just know many people (and businesses) waiting for 2.9. Doesn't feel right to stick ECH and post-quantum in 2.9.1... or 2.10, seeing how close it is.

[francislavoie]

I don't like it because convincing people to use Go RC to make their xcaddy builds for 2.9 is thorny. I'd rather we push 2.9 stable out until Feb when 1.24 stable is out, if you're tied to having features that require 1.24 merged right away.

We have lots of people waiting for 2.9. Do we really need to have this stuff in 2.9? Can't we just do a quick 2.10 once 1.24 is out and we've done enough testing, or something?

[mholt]

That's a good point... Requiring developers, plugin authors, and compiling-from-source users to install a Go RC, which is not a stock Go installation (in package managers, etc), will be surprising and inconvenient to most. We will probably have a lot of pushback in issues and frustrated users who can't build Caddy without upgrading their Go version to a prerelease.

Let's go with your idea then. I'll try to get 2.9 out ASAP, and then we can do 2.10 sooner than I had expected.