deps: switch openssl to quictls/openssl #37601

jasnell · 2021-03-04T16:07:35Z

/cc @nodejs/tsc @nodejs/crypto

Yesterday, Akamai and Microsoft announced that they are providing, and will be supporting, a temporary supported fork of OpenSSL 1.1.1 and 3.0.0 that includes the necessary bits to support QUIC implementations.

See @richsalz's tweet here for details: https://twitter.com/RichSalz/status/1367349918671773697

Critically, this fork is intended only as a stopgap to support platforms that wish to deliver QUIC support while the main OpenSSL project is focused on the release of 3.0.0. The only differences this fork will have relative to the main OpenSSL project is the additional pieces necessary for QUIC.

Because both Akamai and Microsoft are using this code in production, they are committing to keeping it up to date with the main OpenSSL releases, so there should not be any lag time between upstream updates and updates in the fork.

Further, Akamai and Microsoft will be requiring that any contributions made to this fork are limited to the QUIC support, and that all other non-QUIC related changes have to be made upstream in the main project. All potential contributors will have to sign and submit the OpenSSL Contributors License Agreement (which I will be doing myself later today).

Now to the point: This pull request replaces the vendored-in OpenSSL distribution in core with the Akamai/Microsoft fork. This does not yet re-introduce the QUIC implementation in Node.js but does make the underlying APIs to do so available by default. If this gets approval and lands, I will start reintroducing the QUIC subsystem in Node.js incrementally over the next few weeks (I have a number of implementation improvements that I've been working on in the interim).

With this version of OpenSSL vendored into core, the QUIC support would be enabled by default as opposed to being opt-in as it was before. For operating systems that use their own distros of Node.js that use the operating system provided openssl (such as RedHat), the QUIC support would not be available unless those distributions provide the option to install the QUIC-enabled openssl alternative. ~~I'm not sure if it's in there yet, but if not I will be opening~~ I have opened a PR ~~soon to add it, but~~ that adds a #DEFINE flag ~~will~~ to be used to conditionally disable building the QUIC support on such platforms.

As for ongoing support, as I mentioned, I will be submitting a signed OpenSSL CLA later today, and will be committing to supporting this for as long as necessary, helping to resolve any issues that may come up while using the fork.

It should be highlighed that for as long as the QUIC subsystem in Node.js is considered experimental, direct use by any Node.js Native Addons of the additional QUIC APIs offered by the fork would also be considered experimental and At Your Own Risk.

Obviously, Long Term Support is a consideration here and I have raised that as a concern in discussions with Akamai and Microsoft. I have communicated to them that if this fork ends up going out in an LTS covered release line that we would need to commit to 30 months of maintenance. While they have not made any guarantees, they are willing to work with us on this, especially if we contribute back to help with that maintenance (thus, part of the reason I'll be signing the CLA). Also, as I said, both companies are using this in production systems today and have their own support and maintenance requirements. Bottom line is that LTS should not be a problem here, especially given that the QUIC support will be considered experimental for quite some time as it matures.

The eventual long term goal is to switch back to the official OpenSSL release once it does include the necessary QUIC support. That will be some time after the 3.0.0 release, with no clear estimate on when that will be.

Source: https://github.com/quictls/openssl/tree/OpenSSL_1_1_1j+quic
Signed-off-by: James M Snell jasnell@gmail.com

nodejs-github-bot · 2021-03-04T16:30:42Z

CI: https://ci.nodejs.org/job/node-test-pull-request/36471/

jasnell · 2021-03-04T16:31:07Z

@richardlau ... I'm not sure I would label this semver-major, actually. semver-minor yes, but there are no breaking changes other than, perhaps, the version number identifier.

mcollina

lgtm

richsalz · 2021-03-04T16:55:41Z

This is great news, thanks @jasnell and the whole Node team!

MylesBorins · 2021-03-04T17:03:42Z

@jasnell if this is semver-major will replacing the fork also be semver-major?

jasnell · 2021-03-04T17:13:40Z

@MylesBorins I don't think it should be semver major.

richardlau · 2021-03-04T17:19:56Z

@jasnell I've replaced the semver-major tag with semver-minor + dont-land-on-* tags.

nodejs-github-bot · 2021-03-04T17:24:29Z

CI: https://ci.nodejs.org/job/node-test-pull-request/36472/ 💚

richardlau · 2021-03-04T17:40:56Z

https://github.com/nodejs/node/blob/master/doc/guides/maintaining-openssl.md probably should be updated.

It would also be good to point to a specific commit/tag rather than the branch https://github.com/quictls/openssl/tree/OpenSSL_1_1_1j+quic in case of divergence (e.g. more quic commits added to the branch).

jasnell · 2021-03-04T17:50:07Z

@richardlau ... yeah, I've got those edits pending, I'm just making sure that CI passes in case there are any other bits that need to be changed (there shouldn't be but just in case).

jasnell · 2021-03-04T18:04:20Z

@richardlau one point... the repo is not yet tagging specific releases, and for now, we absolutely want to work forward with the tip of the development in the branch, so I'll avoid pointing to a specific commit or tag for now. Later that will anchor to a specific tag, however.

richardlau · 2021-03-04T18:08:58Z

@jasnell The reason to prefer specific commit/tag is to make it easier to determine if any upstream (quic) fixes are in our code or are waiting for another update (i.e. we're now behind). So as long as that's done before this lands I'm good.

mhdawson · 2021-03-04T21:10:57Z

A few things I think we need to answer/address before landing this include:

Have you thought about how that is going to work for distros which I think all dynamically link against an openssl library that is shipped with the operating system
the level of support behind the fork in terms of speed of releases etc.
documenting that for Node.js the QUIC part of OpenSSL is experimental.
ie if your native add-on uses the additional APIs that are available in the
fork, beware that it could go away

jasnell · 2021-03-04T21:28:03Z

Have you thought about how that is going to work for distros which I think all dynamically link against an openssl library that is shipped with the operating system

Yes, the idea there would be to default set the OPENSSL_NO_QUIC define when shared-openssl is selected in the build so that the QUIC subsystem is disabled. It would be possible to selectively turn that back on if the distro chose to build against a build of the quictls/openssl version as a shared library.

the level of support behind the fork in terms of speed of releases etc.

See the discussion on this in the original PR post. @richsalz might be able to weigh in a bit more here.

documenting that for Node.js the QUIC part of OpenSSL is experimental. ie if your native add-on uses the additional APIs that are available in the fork, beware that it could go away

Discussed in the PR post. I'm not sure how/where to add documentation to that effect.

jasnell · 2021-03-04T21:29:04Z

The reason to prefer specific commit/tag is to make it easier to determine if any upstream (quic) fixes are in our code or are waiting for another update (i.e. we're now behind). So as long as that's done before this lands I'm good.

Ah right, yeah, will note that for sure. I'm actually waiting for another commit to land before moving forward with this to land so once that lands up stream and I pull it in here I'll note the commit sha reference in the commit message.

richsalz · 2021-03-04T22:01:23Z

Please take a look at the README/FAQ at https://github.com/quictls/openssl and see if it answers your questions. If not, please open an issue. It also talks about different SONAME and openssl version output.

jasnell · 2021-03-08T15:58:57Z

I will be updating the quictls/openssl commit in this PR later on today with the commit sha

mhdawson

Rubber Stamp LGTM

jasnell · 2021-03-11T20:47:02Z

I will give it another couple of days for review, then if there are no objections, I'll land on Monday.

As of quictls/openssl@0c70d48 Source: https://github.com/quictls/openssl/tree/OpenSSL_1_1_1j+quic Signed-off-by: James M Snell <jasnell@gmail.com> PR-URL: #37601 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Gus Caplan <me@gus.host> Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com> Reviewed-By: Jiawen Geng <technicalcute@gmail.com> Reviewed-By: Michael Dawson <midawson@redhat.com>

Signed-off-by: James M Snell <jasnell@gmail.com> PR-URL: #37601 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Gus Caplan <me@gus.host> Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com> Reviewed-By: Jiawen Geng <technicalcute@gmail.com> Reviewed-By: Michael Dawson <midawson@redhat.com>

PR-URL: #37601 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Gus Caplan <me@gus.host> Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com> Reviewed-By: Jiawen Geng <technicalcute@gmail.com> Reviewed-By: Michael Dawson <midawson@redhat.com>

jasnell · 2021-03-15T14:34:49Z

Landed in 66f0eb7...7abdc3e

Heads up to all @nodejs/crypto folks... the changes the source of openssl updates! Please see the updating guide for details.

As of quictls/openssl@0c70d48 Source: https://github.com/quictls/openssl/tree/OpenSSL_1_1_1j+quic Signed-off-by: James M Snell <jasnell@gmail.com> PR-URL: #37601 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Gus Caplan <me@gus.host> Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com> Reviewed-By: Jiawen Geng <technicalcute@gmail.com> Reviewed-By: Michael Dawson <midawson@redhat.com>

Signed-off-by: James M Snell <jasnell@gmail.com> PR-URL: #37601 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Gus Caplan <me@gus.host> Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com> Reviewed-By: Jiawen Geng <technicalcute@gmail.com> Reviewed-By: Michael Dawson <midawson@redhat.com>

PR-URL: #37601 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Gus Caplan <me@gus.host> Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com> Reviewed-By: Jiawen Geng <technicalcute@gmail.com> Reviewed-By: Michael Dawson <midawson@redhat.com>

Notable changes: * crypto: * add optional callback to crypto.sign and crypto.verify (Filip Skokan) #37500 * support JWK objects in create\*Key (Filip Skokan) #37254 * deps: * update archs files for OpenSSL-1.1.1+quic (James M Snell) #37601 * switch openssl to quictls/openssl (James M Snell) #37601 * doc: * update maintaining-openssl guide (James M Snell) #37601 * fs: * improve fsPromises writeFile performance (Nitzan Uziely) #37610 * add promisified readFile benchmark (Nitzan Uziely) #37608 * improve fsPromises readFile performance (Nitzan Uziely) #37608 * lib: * implement AbortSignal.abort() (James M Snell) #37693 * node-api: * define version 8 (Gabriel Schulhof) #37652 * test: * update dom/abort tests (James M Snell) #37693 * fixup test to account for quic openssl version (James M Snell) #37601 * worker: * add setEnvironmentData/getEnvironmentData (James M Snell) #37486

PR-URL: #37766 Notable changes: * crypto: * add optional callback to crypto.sign and crypto.verify (Filip Skokan) #37500 * support JWK objects in create\*Key (Filip Skokan) #37254 * deps: * update archs files for OpenSSL-1.1.1+quic (James M Snell) #37601 * switch openssl to quictls/openssl (James M Snell) #37601 * doc: * update maintaining-openssl guide (James M Snell) #37601 * fs: * improve fsPromises writeFile performance (Nitzan Uziely) #37610 * add promisified readFile benchmark (Nitzan Uziely) #37608 * improve fsPromises readFile performance (Nitzan Uziely) #37608 * lib: * implement AbortSignal.abort() (James M Snell) #37693 * node-api: * define version 8 (Gabriel Schulhof) #37652 * test: * update dom/abort tests (James M Snell) #37693 * fixup test to account for quic openssl version (James M Snell) #37601 * worker: * add setEnvironmentData/getEnvironmentData (James M Snell) #37486

As of quictls/openssl@0c70d48 Source: https://github.com/quictls/openssl/tree/OpenSSL_1_1_1j+quic Signed-off-by: James M Snell <jasnell@gmail.com> PR-URL: #37601 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Gus Caplan <me@gus.host> Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com> Reviewed-By: Jiawen Geng <technicalcute@gmail.com> Reviewed-By: Michael Dawson <midawson@redhat.com>

Signed-off-by: James M Snell <jasnell@gmail.com> PR-URL: #37601 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Gus Caplan <me@gus.host> Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com> Reviewed-By: Jiawen Geng <technicalcute@gmail.com> Reviewed-By: Michael Dawson <midawson@redhat.com>

PR-URL: #37601 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Gus Caplan <me@gus.host> Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com> Reviewed-By: Jiawen Geng <technicalcute@gmail.com> Reviewed-By: Michael Dawson <midawson@redhat.com>

PR-URL: #37766 Notable changes: * crypto: * add optional callback to crypto.sign and crypto.verify (Filip Skokan) #37500 * support JWK objects in create\*Key (Filip Skokan) #37254 * deps: * update archs files for OpenSSL-1.1.1+quic (James M Snell) #37601 * switch openssl to quictls/openssl (James M Snell) #37601 * doc: * update maintaining-openssl guide (James M Snell) #37601 * fs: * improve fsPromises writeFile performance (Nitzan Uziely) #37610 * add promisified readFile benchmark (Nitzan Uziely) #37608 * improve fsPromises readFile performance (Nitzan Uziely) #37608 * lib: * implement AbortSignal.abort() (James M Snell) #37693 * node-api: * define version 8 (Gabriel Schulhof) #37652 * test: * update dom/abort tests (James M Snell) #37693 * fixup test to account for quic openssl version (James M Snell) #37601 * worker: * add setEnvironmentData/getEnvironmentData (James M Snell) #37486

PR-URL: #37766 Notable changes: * crypto: * add optional callback to crypto.sign and crypto.verify (Filip Skokan) #37500 * support JWK objects in create\*Key (Filip Skokan) #37254 * deps: * switch openssl to quictls/openssl (James M Snell) #37601 * fs: * improve fsPromises writeFile performance (Nitzan Uziely) #37610 * improve fsPromises readFile performance (Nitzan Uziely) #37608 * lib: * implement AbortSignal.abort() (James M Snell) #37693 * node-api: * define version 8 (Gabriel Schulhof) #37652 * worker: * add setEnvironmentData/getEnvironmentData (James M Snell) #37486

PR-URL: #37766 Notable changes: * crypto: * add optional callback to crypto.sign and crypto.verify (Filip Skokan) #37500 * support JWK objects in create\*Key (Filip Skokan) #37254 * deps: * update to cjs-module-lexer@1.1.0 (Guy Bedford) #37712 * switch openssl to quictls/openssl (James M Snell) #37601 * fs: * improve fsPromises writeFile performance (Nitzan Uziely) #37610 * improve fsPromises readFile performance (Nitzan Uziely) #37608 * lib: * implement AbortSignal.abort() (James M Snell) #37693 * node-api: * define version 8 (Gabriel Schulhof) #37652 * worker: * add setEnvironmentData/getEnvironmentData (James M Snell) #37486

nodejs-github-bot added needs-ci PRs that need a full CI run. openssl Issues and PRs related to the OpenSSL dependency. labels Mar 4, 2021

This comment has been minimized.

Sign in to view

richardlau added the semver-major PRs that contain breaking changes and should be released in the next major version. label Mar 4, 2021

mcollina approved these changes Mar 4, 2021

View reviewed changes

richardlau added dont-land-on-v10.x semver-minor PRs that contain new features and should be released in the next minor version. and removed semver-major PRs that contain breaking changes and should be released in the next major version. labels Mar 4, 2021

jasnell force-pushed the the-quic-comeback-tour-act-1 branch from 3c1fd9e to abc2768 Compare March 4, 2021 18:07

jasnell added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Mar 4, 2021

mhdawson added the tsc-agenda Issues and PRs to discuss during the meetings of the TSC. label Mar 4, 2021

mhdawson mentioned this pull request Mar 8, 2021

Node.js Technical Steering Committee (TSC) Meeting 2021-03-11 nodejs/TSC#980

Closed

mhdawson approved these changes Mar 11, 2021

View reviewed changes

mhdawson mentioned this pull request Mar 15, 2021

Node.js Technical Steering Committee (TSC) Meeting 2021-03-18 nodejs/TSC#993

Closed

jasnell removed tsc-agenda Issues and PRs to discuss during the meetings of the TSC. needs-ci PRs that need a full CI run. labels Mar 15, 2021

jasnell closed this Mar 15, 2021

danielleadams mentioned this pull request Mar 16, 2021

v15.12.0 proposal #37766

Merged

Ayase-252 mentioned this pull request Mar 23, 2021

Add riscv64 backend for node.js #37856

Closed

brodycj mentioned this pull request Mar 31, 2021

nit: update OpenSSL-Strategy.md nodejs/TSC#1008

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

deps: switch openssl to quictls/openssl #37601

deps: switch openssl to quictls/openssl #37601

jasnell commented Mar 4, 2021 •

edited

Loading

This comment has been minimized.

nodejs-github-bot commented Mar 4, 2021

jasnell commented Mar 4, 2021

mcollina left a comment

richsalz commented Mar 4, 2021

MylesBorins commented Mar 4, 2021

jasnell commented Mar 4, 2021

richardlau commented Mar 4, 2021

nodejs-github-bot commented Mar 4, 2021 •

edited by jasnell

Loading

richardlau commented Mar 4, 2021

jasnell commented Mar 4, 2021

jasnell commented Mar 4, 2021

richardlau commented Mar 4, 2021

mhdawson commented Mar 4, 2021

jasnell commented Mar 4, 2021

jasnell commented Mar 4, 2021

richsalz commented Mar 4, 2021

jasnell commented Mar 8, 2021

mhdawson left a comment

jasnell commented Mar 11, 2021

jasnell commented Mar 15, 2021

deps: switch openssl to quictls/openssl #37601

deps: switch openssl to quictls/openssl #37601

Conversation

jasnell commented Mar 4, 2021 • edited Loading

This comment has been minimized.

nodejs-github-bot commented Mar 4, 2021

jasnell commented Mar 4, 2021

mcollina left a comment

Choose a reason for hiding this comment

richsalz commented Mar 4, 2021

MylesBorins commented Mar 4, 2021

jasnell commented Mar 4, 2021

richardlau commented Mar 4, 2021

nodejs-github-bot commented Mar 4, 2021 • edited by jasnell Loading

richardlau commented Mar 4, 2021

jasnell commented Mar 4, 2021

jasnell commented Mar 4, 2021

richardlau commented Mar 4, 2021

mhdawson commented Mar 4, 2021

jasnell commented Mar 4, 2021

jasnell commented Mar 4, 2021

richsalz commented Mar 4, 2021

jasnell commented Mar 8, 2021

mhdawson left a comment

Choose a reason for hiding this comment

jasnell commented Mar 11, 2021

jasnell commented Mar 15, 2021

jasnell commented Mar 4, 2021 •

edited

Loading

nodejs-github-bot commented Mar 4, 2021 •

edited by jasnell

Loading