openssl 3.0 support

[satya-rajesh]

In continuation to PowerShell/PowerShell#14410

PowerShell-7.0.3$ grep -Rns "ERR_put_error" ./
Binary file ./src/powershell-unix/bin/Linux/netcoreapp3.1/linux-x64/System.Security.Cryptography.Native.OpenSsl.a matches
Binary file ./src/powershell-unix/bin/Linux/netcoreapp3.1/linux-x64/System.Security.Cryptography.Native.OpenSsl.so matches
Binary file ./bin/System.Security.Cryptography.Native.OpenSsl.a matches
Binary file ./bin/System.Security.Cryptography.Native.OpenSsl.so matches

System.Security.Cryptography.Native.OpenSsl.so contains deprecated APIs of Openssl 3.0
Reference: https://www.openssl.org/news/changelog.html

This bug is created for support of openssl 3.0 in dotnet

---

Project Plan (updated as data is known, checkmarks mean done locally, not necessarily merged)

• Convert RSA shim from RSA* to EVP_PKEY*
  • Keygen
  • Sign
  • Verify
  • Encrypt
  • Decrypt
  • Import
  • Export
• Rebuild the interop exception model.
  • Each operation should reset the error queue, and we want to throw the first error, not the last one.
• Convert ECDSA shim to fully EVP_PKEY*
• Convert ECDH shim to fully EVP_PKEY*
• Convert DSA shim to fully EVP_PKEY*
• Write translation table for X509ChainStatusCodes (some #define values changed in OpenSSL 3.0)

[ghost]

Tagging subscribers to this area: @bartonjs, @vcsjones, @krwq
See info in area-owners.md if you want to be subscribed.

Issue Details

---

In continuation to PowerShell/PowerShell#14410

PowerShell-7.0.3$ grep -Rns "ERR_put_error" ./
Binary file ./src/powershell-unix/bin/Linux/netcoreapp3.1/linux-x64/System.Security.Cryptography.Native.OpenSsl.a matches
Binary file ./src/powershell-unix/bin/Linux/netcoreapp3.1/linux-x64/System.Security.Cryptography.Native.OpenSsl.so matches
Binary file ./bin/System.Security.Cryptography.Native.OpenSsl.a matches
Binary file ./bin/System.Security.Cryptography.Native.OpenSsl.so matches

System.Security.Cryptography.Native.OpenSsl.so contains deprecated APIs of Openssl 3.0
Reference: https://www.openssl.org/news/changelog.html

This bug is created for support of openssl 3.0 in dotnet

Author:	satya-rajesh
Assignees:	-
Labels:	area-System.Security, untriaged
Milestone:	-

[bartonjs]

I thought we already had an issue tracking this, but apparently it was just on my implicit TODO. So now this is the issue tracking working with OpenSSL 3. (Which we may have to service... at least, I think we did for 1.0/1.1 hybridization)

[jeffhandley]

@bartonjs I just found that we simply had a note card on our project board for it (hence you thinking we had something tracking it). I've updated the board to use this issue instead of the note card.

[omajid]

Fedora 34, tracked via dotnet/core#5851, will have OpenSSL 3.0: https://fedoraproject.org/wiki/Changes/OpenSSL3.0, though there are plans for a OpenSSL 1.1 package for compatibility.

[Lxiamail]

Sorry for accidently closed the issue.

[omajid]

Hey folks! Is there some feature branch or personal repo that I can try out to see what the latest status of this implementation is? I was trying out building runtime with (a pre-release of) OpenSSL 3.0 today and I get a couple of pages worth of build errors. I would like to help out with the port/testing if possible.

[bartonjs]

@omajid I have a tinkering branch at https://github.com/bartonjs/runtime/tree/support_openssl3

My recollection is that bartonjs@459d274#diff-9f438d89e563743defcabf3a647a5be42683be66ad8d381683c9967d85d4f214 was the only part required to compile against 3.0, but since I commented out a static assert and haven't written code to deal with it yet there are obviously runtime errors expected.

Moving asymmetric operations off of primitives and onto EVP_PKEY got too big, so I pulled that off as its own thing, but I keep getting distracted by other things.

[omajid]

Thanks! Unfortunately, that doesn't seem to address the compiler errors I am seeing.

Building runtime's main branch leads me to errors like https://gist.github.com/omajid/03106c961a4cc39a2e456aea13302943. I tried support_openssl3 branch, but that leads me to essentially the same set of errors.

The first few errors seem to be fixed by something like this in opensslshim.h:

+#if OPENSSL_VERSION_NUMBER >= OPENSSL_VERSION_3_0_0_RTM
+
+// Remove problematic #defines
+#undef CRYPTO_num_locks
+#undef ERR_load_crypto_strings
+#undef OPENSSL_add_all_algorithms_conf
+#undef SSL_load_error_strings
+#undef SSL_library_init
+#undef CRYPTO_set_locking_callback
+
+#endif

I will keep working away at this.

[bartonjs]

Hmmm.. I probably only tried building non-portable when building against 3.0

[omajid]

Thanks, non-portable worked for me too using support_openssl3 branch, once I had bypassed the non-portable build errors.

[omajid]

Moving asymmetric operations off of primitives and onto EVP_PKEY got too big, so I pulled that off as its own thing, but I keep getting distracted by other things.

Not sure what your schedule or the scope of this implementation change looks like, but I wonder if it we should get runtime building/running against OpenSSL 3.0 first (even if that means using deprecated (but not removed) primitives) and then look at moving to EVP_KEY methods?

[bartonjs]

which release is the openssl 3.0.0 support targeted ?

.NET 6

[omajid]

I am interested in getting this support merged into 3.1 as well. I have a draft PR here: dotnet/corefx#43078

[bartonjs]

@omajid Can you describe the need for supporting OpenSSL 3.0 in 3.1? 6.0 will come out in ~November, and then 3.1 will go out of support about a year later.

At minimum, I don't think we'd accept the PR into release/3.1 while OpenSSL 3.0 is still calling their builds alpha, so it's unclear how much time this would grant if the problem space is just ".NET LTS using OpenSSL 3".

It's also worth noting that I'm/we're not done with the OpenSSL 3.0 project... we want to get the rest of the runtime onto EVP_PKEY and off of the primitives to avoid dependencies on now-deprecated API.

[omajid]

@omajid Can you describe the need for supporting OpenSSL 3.0 in 3.1? 6.0 will come out in ~November, and then 3.1 will go out of support about a year later.

Happy to! I am interested in this because a couple of Linux distributions that I care about will be moving to OpenSSL 3.0 soon.

Fedora is planning to use it as soon as 3.0 is declared GA. https://fedoraproject.org/wiki/Changes/OpenSSL3.0 was initially for Fedora 34, but got pushed to Fedora 35 due to OpenSSL upstream being late. There is a plan to have a compatibility package but I am not sure how long that will be supported or if it would be usable for building.

It's also looking like RHEL 9 will include (only?) OpenSSL 3.0: https://bugzilla.redhat.com/show_bug.cgi?id=1955873

At minimum, I don't think we'd accept the PR into release/3.1 while OpenSSL 3.0 is still calling their builds alpha

Yeah, that's completely fair. I was motivated by the distros above and figured it's better to share my work than keep it to myself.

it's unclear how much time this would grant if the problem space is just ".NET LTS using OpenSSL 3".

True. Maybe the better move for these distributions would be waiting for the next LTS (6.0) and not shipping/supporting .NET Core 3.1?

It's also worth noting that I'm/we're not done with the OpenSSL 3.0 project... we want to get the rest of the runtime onto EVP_PKEY and off of the primitives to avoid dependencies on now-deprecated API.

Thanks! I didn't realize that - even though I now see that this issue is still open. I built portable and non portable builds against OpenSSL 3.0 alpha 15 and assumed this was fully completed.

[satya-rajesh]

$ grep -r "EVP_MD_size" .
dotnet-sdk-6.0.100/shared/Microsoft.NETCore.App/6.0.0-preview.5.21301.5/libSystem.Security.Cryptography.Native.OpenSsl.so

EVP_MD_size API seems to be causing an issue for powershell build.

As per https://www.openssl.org/docs/manmaster/man3/EVP_MD_size.html, this function is renamed in openssl 3.0.0

Error observed with https://dotnet.microsoft.com/download/dotnet/6.0, https://github.com/PowerShell/PowerShell/releases/tag/v7.2.0-preview.7 and openssl 3.0.0 beta1:

Welcome to .NET 6.0!^M
---------------------^M
SDK Version: 6.0.100-preview.5.21302.13

Telemetry^M
---------^M
The .NET tools collect usage data in order to help us improve your experience. It is collected by Microsoft and shared with the community. You can opt-out of telemetry by setting the DOTNET_CLI_TELEMETRY_OPTOUT environment variable to '1' or 'true' using your favorite shell.^M
^M
Read more about .NET CLI Tools telemetry: https://aka.ms/dotnet-cli-telemetry

----------------^M
Installed an ASP.NET Core HTTPS development certificate.^M
To trust the certificate run 'dotnet dev-certs https --trust' (Windows and macOS only).^M
Learn about HTTPS: https://aka.ms/dotnet-https^M
----------------^M
Write your first app: https://aka.ms/dotnet-hello-world^M
Find out what's new: https://aka.ms/dotnet-whats-new^M
Explore documentation: https://aka.ms/dotnet-docs^M
Report issues and find source on GitHub: https://github.com/dotnet/core^M
Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli^M

Cannot get required symbol EVP_MD_size from libssl
./build.sh: line 6: 17097 Aborted (core dumped) dotnet restore $f

Any suggestions ?

[bartonjs]

Looks like EVP_MD_size got renamed to EVP_MD_get_size for Beta1, and our support was based on Alpha....14? So we'll need to do another sweep.

[bartonjs]

.NET 6 Preview 7 will work with OpenSSL 3.0 Beta 1. Hopefully there aren't any more renames now that they got to Beta 😄.

[satya-rajesh]

.NET 6 Preview 7 will work with OpenSSL 3.0 Beta 1. Hopefully there aren't any more renames now that they got to Beta 😄.

Thanks for the update. When will be the preview 7, i didnt find it yet at https://dotnet.microsoft.com/download/dotnet/6.0 ?

[bartonjs]

Preview 7 will come out in about a month. (Preview 6 came out two days ago)

[bartonjs]

Moving the issue to 7.0 to track that we still have some plumbing/infrastructure work to do to get off of deprecated APIs. While doing the DSA* => EVP_PKEY* change the tests revealed that we probably want to clean up how we interact with the error queue (#55973), which is too concerning to do with the limited available time in 6.0.

6.0 will still ship with OSSL3 support, it's just not as clean as I'd like.

[adamijak]

My Razor server fails to start on Fedora 36 and VsCode, with error message Cannot get required symbol EVP_MD_size from libssl

> dotnet --list-sdks
3.1.417 [/usr/lib64/dotnet/sdk]
6.0.103 [/usr/lib64/dotnet/sdk]

> openssl version
OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)

I should have openssl1.1 installed, is there any way I can fix it?

[omajid]

Hey, @adamijak do you have a set of steps I can follow to reproduce the issue you are seeing?

I ran the following steps and I couldn't see a problem:

podman run -it registry.fedoraproject.org/fedora:36 /bin/bash
dnf install dotnet-sdk-3.1 dotnet-sdk-6.0 openssl1.1
mkdir -p foobar
cd foobar
dotnet new razor
dotnet run

Are you seeing errors running via dotnet run? Or is it the vscode instance the one showing errors? Does the error say anything about OmniSharp? Can you share the error messages (preferably the complete set of logs and/or stack trace) here?

[adamijak]

Hey, @adamijak do you have a set of steps I can follow to reproduce the issue you are seeing?

I ran the following steps and I couldn't see a problem:

podman run -it registry.fedoraproject.org/fedora:36 /bin/bash
dnf install dotnet-sdk-3.1 dotnet-sdk-6.0 openssl1.1
mkdir -p foobar
cd foobar
dotnet new razor
dotnet run

Are you seeing errors running via dotnet run? Or is it the vscode instance the one showing errors? Does the error say anything about OmniSharp? Can you share the error messages (preferably the complete set of logs and/or stack trace) here?

Hi I opened separate issue #67375 for this

[jeffhandley]

@bartonjs Do you have any objections to us punting this out of .NET 8 (into Future for the time-being), and revisiting the remaining work as a candidate for .NET 9 planning?