action runner does not work with CentOS Stream 9 #1902

henrywang · 2022-05-18T08:33:09Z

Describe the bug
I’m registering a CentOS Stream 9 VM as self hosted runner. I got the following error when I run command ./config.sh --url https://github.com/xxxx --token xxxx --name runner-centos-stream-9-large --labels centos-stream-9,large --ephemeral --disableupdate --unattended

Error: The SSL connection could not be established, see inner exception.

The latest version of ca-certificates-2020.2.50-94.el9.noarch.rpm is installed.

I also tried on CentOS Stream 8 VM with same version of runner, register works without error.

To Reproduce
Steps to reproduce the behavior:

Deploy CentOS Stream 9 VM from https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/BaseOS/x86_64/images/CentOS-Stream-GenericCloud-9-20220516.0.x86_64.qcow2
Install dependencies: acl, lttng-ust, openssl-libs, krb5-libs, zlib, libicu
Download latest runner from https://github.com/actions/runner/releases/download/v2.291.1/actions-runner-linux-x64-2.291.1.tar.gz and extract the installer file
Run command: ./config.sh --url https://github.com/xxxx --token xxxx --name runner-centos-stream-9-large --labels centos-stream-9,large --ephemeral --disableupdate --unattended
See the error

Expected behavior
Register successful.

Runner Version and Platform

v2.291.1

OS of the machine running the runner?

CentOS Stream 9

What's not working?

The SSL connection could not be established, see inner exception.

[2022-05-17 13:54:33Z ERR  ConfigurationManager] Failed to get tenant credentials -- Atempt: 1
[2022-05-17 13:54:33Z ERR  ConfigurationManager] System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid because of errors in the certificate chain: NotSignatureValid
   at System.Net.Security.SslStream.SendAuthResetSignal(ProtocolToken message, ExceptionDispatchInfo exception)
   at System.Net.Security.SslStream.CompleteHandshake(SslAuthenticationOptions sslAuthenticationOptions)
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request)
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at GitHub.Runner.Listener.Configuration.ConfigurationManager.GetTenantCredential(String githubUrl, String githubToken, String runnerEvent)

Job Log Output

--------------------------------------------------------------------------------
|        ____ _ _   _   _       _          _        _   _                      |
|       / ___(_) |_| | | |_   _| |__      / \   ___| |_(_) ___  _ __  ___      |
|      | |  _| | __| |_| | | | | '_ \    / _ \ / __| __| |/ _ \| '_ \/ __|     |
|      | |_| | | |_|  _  | |_| | |_) |  / ___ \ (__| |_| | (_) | | | \__ \     |
|       \____|_|\__|_| |_|\__,_|_.__/  /_/   \_\___|\__|_|\___/|_| |_|___/     |
|                                                                              |
|                       Self-hosted runner registration                        |
|                                                                              |
--------------------------------------------------------------------------------

# Authentication

The SSL connection could not be established, see inner exception.

Runner and Worker's Diagnostic Logs

[2022-05-17 13:54:33Z ERR  ConfigurationManager] Failed to get tenant credentials -- Atempt: 1
[2022-05-17 13:54:33Z ERR  ConfigurationManager] System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid because of errors in the certificate chain: NotSignatureValid
   at System.Net.Security.SslStream.SendAuthResetSignal(ProtocolToken message, ExceptionDispatchInfo exception)
   at System.Net.Security.SslStream.CompleteHandshake(SslAuthenticationOptions sslAuthenticationOptions)
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsync(SslClientAuthenticationOptions sslOptions, HttpRequestMessage request, Boolean async, Stream stream, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.AddHttp11ConnectionAsync(HttpRequestMessage request)
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at GitHub.Runner.Listener.Configuration.ConfigurationManager.GetTenantCredential(String githubUrl, String githubToken, String runnerEvent)

To debug this issue I also tried openssl s_client -connect github.com:443 to check system ca certificate and trust status. Here's the output.

CONNECTED(00000003)
depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
verify return:1
depth=1 C = US, O = DigiCert Inc, CN = DigiCert TLS Hybrid ECC SHA384 2020 CA1
verify return:1
depth=0 C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = github.com
verify return:1
---
Certificate chain
 0 s:C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = github.com
   xxxxx
 1 s:C = US, O = DigiCert Inc, CN = DigiCert TLS Hybrid ECC SHA384 2020 CA1
   xxxxx
---
Server certificate
-----BEGIN CERTIFICATE-----
xxxxx
-----END CERTIFICATE-----
subject=C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = github.com
issuer=C = US, O = DigiCert Inc, CN = DigiCert TLS Hybrid ECC SHA384 2020 CA1
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2805 bytes and written 378 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_128_GCM_SHA256
    Session-ID: FCF1A7989FAB082EF5B9A285BA3070AE1447C1FD1346A6DBE12C7A5A856F230A
    Session-ID-ctx:
    Resumption PSK: C42947DD5081D1279D871EF9406409213F783228F020A219248AE564A1F4FF69
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - 7c 32 2a f5 da 2e db be-7f 4d e8 a0 d7 0f bc ef   |2*......M......
    0010 - bc 22 b9 19 b5 61 b9 71-2a b6 14 67 07 48 f0 aa   ."...a.q*..g.H..

    Start Time: 1652840443
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_128_GCM_SHA256
    Session-ID: 91D77EC84BAC6E4CE4EFA5995F8500B90A0C6E4153CA74CF98A4B208F690FEA7
    Session-ID-ctx:
    Resumption PSK: 323AE0644E3A097D1BDD6F5E26B6612FA9FE6D4185ED2BE1F98E427EF62E8182
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - 40 a4 38 a6 b1 e8 e5 61-2e 30 d0 02 66 37 1c e3   @.8....a.0..f7..
    0010 - f9 12 13 2f 54 a3 0b 84-2b e4 31 9d c1 fe 6c 69   .../T...+.1...li

    Start Time: 1652840443
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK

The text was updated successfully, but these errors were encountered:

martinpitt · 2022-05-19T06:56:57Z

I tried this on a RHEL 9.1 nightly VM, which is by and large the same as CentOS 9 stream.

sudo dnf install -y acl lttng-ust openssl-libs krb5-libs zlib libicu
curl -L -O https://github.com/actions/runner/releases/download/v2.291.1/actions-runner-linux-x64-2.291.1.tar.gz
tar xf actions-runner-linux-x64-2.291.1.tar.gz
./config.sh --url https://github.com/xxxx --token xxxx --name runner-centos-stream-9-large --labels centos-stream-9,large --ephemeral --disableupdate --unattended

Note I literally used the xxx here -- I didn't set up any project or token, the error seems to happen before already.

This gets the same error. The relevant part seems to be this:

System.Security.Authentication.AuthenticationException: The remote certificate is invalid because of errors in the certificate chain: NotSignatureValid

But unfortunately no details. Is this not using the system OpenSSL config?

TingluoHuang · 2022-05-24T21:21:12Z

might related centos stream-9 use openssl 3.0 by default and dotnet is not playing well with it.
dotnet/sdk#25582

henrywang · 2022-06-28T13:41:54Z

@martinpitt @TingluoHuang There's a workaround for this issue. Enable SHA-1 in RHEL 9 and CentOS Stream 9 by following command:
# update-crypto-policies --set DEFAULT:SHA1
But as we all know, SHA-1 is not considered secure any more.

A workaround for issue actions/runner#1902

nikola-jokic · 2022-07-14T09:42:04Z

Hi @henrywang,

I reproduced the issue and it seems to me it is really related to the .NET Core. The line that throws an exception is during the POST request call on:

runner/src/Runner.Listener/Configuration/ConfigurationManager.cs

Line 631 in 407a347

    
           var response = await httpClient.PostAsync(githubApiUrl, new StringContent(string.Empty));

So, if you ask me, the workaround is the way to go until this issue is fixed inside the dotnet 😞

nikola-jokic · 2022-10-12T12:38:02Z

I will close this issue since there is nothing we can change about it. Thank you for providing a workaround!

henrywang added the bug Something isn't working label May 18, 2022

henrywang added a commit to virt-s1/kite-action that referenced this issue Jun 28, 2022

proxy: Enable SHA-1 on RHEL9 and CS9 to work with action runner

40305d4

A workaround for issue actions/runner#1902

nikola-jokic self-assigned this Jul 12, 2022

nikola-jokic closed this as completed Oct 12, 2022

vascoguita mentioned this issue Feb 27, 2023

Acceptance tests fail on GitHub Actions due to mtime truncation cs3org/reva#3689

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

action runner does not work with CentOS Stream 9 #1902

action runner does not work with CentOS Stream 9 #1902

henrywang commented May 18, 2022 •

edited

Loading

martinpitt commented May 19, 2022

TingluoHuang commented May 24, 2022

henrywang commented Jun 28, 2022

nikola-jokic commented Jul 14, 2022

nikola-jokic commented Oct 12, 2022

action runner does not work with CentOS Stream 9 #1902

action runner does not work with CentOS Stream 9 #1902

Comments

henrywang commented May 18, 2022 • edited Loading

Runner Version and Platform

OS of the machine running the runner?

What's not working?

Job Log Output

Runner and Worker's Diagnostic Logs

martinpitt commented May 19, 2022

TingluoHuang commented May 24, 2022

henrywang commented Jun 28, 2022

nikola-jokic commented Jul 14, 2022

nikola-jokic commented Oct 12, 2022

henrywang commented May 18, 2022 •

edited

Loading