WCF on Linux - Kerberos Authentication with UpnEndpointIdentity seems to be sending the wrong name-type to the KDC. #31040
Comments
|
We made a great deal of fixes for Kerberos related scenarios in .NET Core 3.0. These fixes are not all backported to .NET Core 2.1/2.2. What exact version of .NET Core are you using? Please show I suggest you try .NET Core 3.0 to see if it works better. However, we still have some limitations with UPNs compared with our SPN support. See: #16576. Also, in order to fully diagnose this, please attach a repro client/server solution so that we can try it out. |
|
Hi @davidsh and thanks for the quick response. We're on .NET Core 3.0. We noticed a lot of changes for Windows Auth in 3.0 so we immediately upgraded. I'll work on getting a repro put together. In the meantime, should we try |
Yes, that would be a good workaround potentially. Also, please show a KRB5_TRACE for your application. It will show useful output. Run this command before running your application: export KRB5_TRACE=/dev/stdout |
|
cc: @mconnew |
|
Here is the output from KR5_TRACE (I've replaced our domain with Corp.Test.Lcl): 10.10.7.3 is the KDC. svcAXBCPDV@corp.test.lcl is the UPN that the WCF service is running under, and the UPN we're providing on the client. This matches what I'm seeing in Wireshark almost exactly. One note: the additional query for krbtgt/test.lcl@test.lcl is strange. That's not happening on windows. |
|
The name-type is only ever really used as a hint in the AD KDC. Can you confirm the |
|
WCF can't implement this for the HTTP transport because of a missing feature in HttpClientHandler compared with .NET Framework. The issue I've opened to request this missing feature is #25320. I suggest adding a thumbs up on the first comment to register that you need this. Currently SpnIdentity won't work either. Recently a sort of workaround was added to HttpClientHandler where you can indirectly specify an SPN by adding a HOST header with that SPN. But that is just a hack which won't work in some scenarios. Rather than WCF implement a partial solution which will only work sometimes and will cause confusion, I would prefer that issue #25320 is addressed. If you want to do this yourself, WCF has an extensibility point so you can do this yourself. There's a blog post which outlines the general approach that someone wrote up. There's also a pretty good example here of how to use this by providing a delegating handler. In your delegating handler you would add a Host header before you call SendAsync on the wrapped handler. |
|
They're slightly different. On Ubuntu, it's sending this in the request-body: On Windows, it's: Could it be the extra sname string for the realm? I'm using the same UPN in both (svcAXBCPDV@corp.test.lcl). |
|
That is most likely why AD is complaining. You will need to add an additional SPN on the service account for it to match up against.
…________________________________
From: Jon Wingfield <notifications@github.com>
Sent: Wednesday, October 2, 2019 5:38:08 PM
To: dotnet/corefx <corefx@noreply.github.com>
Cc: Steve Syfuhs <steve@syfuhs.net>; Mention <mention@noreply.github.com>
Subject: Re: [dotnet/corefx] WCF on Linux - Kerberos Authentication with UpnEndpointIdentity seems to be sending the wrong name-type to the KDC. (#41489)
@SteveSyfuhs<https://github.com/SteveSyfuhs>
They're slightly different. On Ubuntu, it's sending this in the request-body:
Frame 55: 1018 bytes on wire (8144 bits), 1018 bytes captured (8144 bits) on interface 0
Ethernet II, Src: Cisco_3c:7a:00 (00:05:9a:3c:7a:00), Dst: Cimsys_33:44:55 (00:11:22:33:44:55)
Internet Protocol Version 4, Src: 10.10.1.43, Dst: 10.10.7.3
Transmission Control Protocol, Src Port: 50029, Dst Port: 88, Seq: 5841, Ack: 1, Len: 964
[9 Reassembled TCP Segments (6804 bytes): dotnet/corefx#47(1366), dotnet/runtime#13860(94), dotnet/runtime#13861(1366), dotnet/corefx#50(94), dotnet/corefx#51(1366), dotnet/runtime#13862(94), dotnet/corefx#53(1366), dotnet/runtime#13863(94), dotnet/corefx#55(964)]
Kerberos
Record Mark: 6800 bytes
tgs-req
pvno: 5
msg-type: krb-tgs-req (12)
padata: 2 items
req-body
Padding: 0
kdc-options: 50010000 (forwardable, proxiable, canonicalize)
realm: CORP.TEST.LCL
sname
name-type: kRB5-NT-SRV-HST (3)
sname-string: 2 items
SNameString: svcAXBCPDV
SNameString: corp.test.lcl
till: 2019-10-01 10:56:05 (UTC)
nonce: 1122014012
etype: 8 items
ENCTYPE: eTYPE-AES256-CTS-HMAC-SHA1-96 (18)
ENCTYPE: eTYPE-AES128-CTS-HMAC-SHA1-96 (17)
ENCTYPE: eTYPE-AES256-CTS-HMAC-SHA384-192 (20)
ENCTYPE: eTYPE-AES128-CTS-HMAC-SHA256-128 (19)
ENCTYPE: eTYPE-DES3-CBC-SHA1 (16)
ENCTYPE: eTYPE-ARCFOUR-HMAC-MD5 (23)
ENCTYPE: eTYPE-CAMELLIA128-CTS-CMAC (25)
ENCTYPE: eTYPE-CAMELLIA256-CTS-CMAC (26)
On Windows, it's:
Frame 18: 1299 bytes on wire (10392 bits), 1299 bytes captured (10392 bits) on interface 0
Ethernet II, Src: Cisco_3c:7a:00 (00:05:9a:3c:7a:00), Dst: Cimsys_33:44:55 (00:11:22:33:44:55)
Internet Protocol Version 4, Src: 10.10.1.43, Dst: 10.10.7.5
Transmission Control Protocol, Src Port: 50004, Dst Port: 88, Seq: 5465, Ack: 1, Len: 1245
[5 Reassembled TCP Segments (6709 bytes): dotnet/corefx#14(1366), dotnet/corefx#15(1366), dotnet/corefx#16(1366), dotnet/runtime#13850(1366), dotnet/runtime#13851(1245)]
Kerberos
Record Mark: 6705 bytes
tgs-req
pvno: 5
msg-type: krb-tgs-req (12)
padata: 2 items
req-body
Padding: 0
kdc-options: 40810000 (forwardable, renewable, canonicalize)
realm: CORP.TEST.LCL
sname
name-type: kRB5-NT-PRINCIPAL (1)
sname-string: 1 item
SNameString: svcAXBCPDV
till: 2037-09-13 02:48:05 (UTC)
nonce: 544141559
etype: 5 items
enc-authorization-data
Could it be the extra sname string for the realm? I'm using the same UPN in both (svcAXBCPDV@corp.test.lcl<mailto:svcAXBCPDV@corp.test.lcl>).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://github.com/dotnet/corefx/issues/41489?email_source=notifications&email_token=AAJHTYM6YIHQJWZA2T36AM3QMU5HBA5CNFSM4I4RDR5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAGTSHA#issuecomment-537737500>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAJHTYPJIVHZQUUBMVBZJ43QMU5HBANCNFSM4I4RDR5A>.
|
|
@SteveSyfuhs but why is it being sent this way? Are there details in GSSAPI that are causing it to be handled differently? |
|
That I can't answer without debugging or reviewing code. The fully qualified name is most likely just what's being passed to the API and is _that_ value because of what @mconnew is suggesting.
…________________________________
From: Jon Wingfield <notifications@github.com>
Sent: Wednesday, October 2, 2019 7:05:26 PM
To: dotnet/corefx <corefx@noreply.github.com>
Cc: Steve Syfuhs <steve@syfuhs.net>; Mention <mention@noreply.github.com>
Subject: Re: [dotnet/corefx] WCF on Linux - Kerberos Authentication with UpnEndpointIdentity seems to be sending the wrong name-type to the KDC. (#41489)
@SteveSyfuhs<https://github.com/SteveSyfuhs> but why is it being sent this way? Are there details in GSSAPI that are causing it to be handled differently?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://github.com/dotnet/corefx/issues/41489?email_source=notifications&email_token=AAJHTYI4SOIGDDHUP5KX5J3QMVHONA5CNFSM4I4RDR5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAGXVII#issuecomment-537754273>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAJHTYMU7RZPIMPKBU2WS5TQMVHONANCNFSM4I4RDR5A>.
|
|
In the Windows dump, you can see it's using the name-type "kRB5-NT-PRINCIPAL" which can be either an SPN or a UPN. In the Linux dump, it's using the type "kRB5-NT-SRV-HST" which I believe can't be used for UPN's.
|
We currently always use this type instead of kRB5-NT-PRINCIPAL. That is because it is the only thing that works correctly with SPNEGO protocol and can handle Kerberos to NTLM fallback. Long term, we need to probably adjust this if we know that we are using a UPN and not SPN. |
|
@davidsh Is this known to work with UPNs though? I'll have a minimal project with client and server to share tomorrow. This is really the most basic situation ever though. Generate new Wcf Service, host under a service account, bind a corefx client to it, run corefx project under Linux. I just want to make sure it reproduces in this minimal form on our infrastructure before sending it over. |
|
@davidsh, on the capture from Windows, it's using kRB5-NT-PRINCIPAL. Are you saying this isn't using SPNEGO? Or is this a limitation of gssapi on Linux? |
Windows is using SPNEGO. But it's always using that particular KRB5 principal format. Windows actually doesn't do a real job of using SPNEGO to downgrade from Kerberos to NTLM. If you look at the Wireshark traces on Windows, you will see that Windows uses "raw" NTLM packets. It does NOT use SPNEGO wrapped NTLM packets. In some ways, that is a violation of the spirit of "Negotiate" protocol (SPNEGO). On Windows, this is achieved because the Negotiate SSPI has logic in it that decides to use either the Kerberos SSPI or the NTLM SSPI directly. But Linux will use SPNEGO wrapped NTLM packets when it detects it needs to downgrade from Kerberos to NTLM. However, to get that to work properly on Linux with the current SPNEGO implementation, the kRB5-NT-SRV-HST needs to be used. |
|
@davidsh I've created a minimal project to reproduce this. It's almost completely just an out-of-the-box WCF Client/Server with a few details:
The client works fine on windows but fails on Ubuntu. The only thing I did on Ubuntu to run it is: The same error as above is returned. Here's the github repo: https://github.com/jonwingfield/Corefx3WcfKerberosFailure |
Thanks for the repro. Does this require any other configuration such as DNS or other Windows Active Directory / Linux Kerberos configurations? Using any UPNs right now doesn't work as per #16576. |
|
@jonwingfield, I know it's a pain, but could you create a new SPN in your domain, give permissions to that SPN to the user that the service is running as and configure your WCF service to use that SPN? For example, use the spn nettcp/hostname. The ability to use a UPN is to allow you to avoid the complication of creating an SPN for a particular service and doesn't require any special permissions to do so. As it might be a while before your scenario is supported on Linux, being pragmatic it might be worth doing the work to get things working with an SPN instead. |
|
@davidsh It does require setup of a krb5.conf file, but this is environment dependent so I can't really provide an example file. There isn't anything extra to do other than specify the KDC and realms as outlined in this article: https://docs.microsoft.com/en-us/sql/connect/jdbc/using-kerberos-integrated-authentication-to-connect-to-sql-server?view=sql-server-ver15. |
This is the first of several PRs that add Enterprise Scenarios Testing capability to the repo. This PR focusses on Linux which allows for docker containers to be used in an enterprise network configuration. I focussed on 2 workflows: 1) The 'dev' workflow, and 2) The PR/CI workflow. The dev workflow works well since it's using containers in a docker-compose environment along with volume mounting your current dev's repo enlistment. The PR/CI workflow gives us an Azure DevOps pipeline to automate verification. I still need to work with the infra team to add a real pipeline that will run. I can't do that until this is merged. In the meantime, I have my own DevOps pipeline that verified this PR. See: https://dev.azure.com/systemnetncl/Enterprise%20Testing/_build/results?buildId=141 I will be linking a follow-up GitHub issue describing the roadmap for building on this system including adding Windows environments, NTLM protocol, proxies, and other libraries such as System.Net.Mail and System.Data.SqlClient. Those libraries also use Negotiate/Kerberos/NTLM enterprise-oriented protocols. Contributes to: https://github.com/dotnet/corefx/issues/41652 https://github.com/dotnet/corefx/issues/41489 https://github.com/dotnet/corefx/issues/36896 https://github.com/dotnet/corefx/issues/30150 https://github.com/dotnet/corefx/issues/24707 https://github.com/dotnet/corefx/issues/10041 https://github.com/dotnet/corefx/issues/6606 https://github.com/dotnet/corefx/issues/6161
This is the first of several PRs that add Enterprise Scenarios Testing capability to the repo. This PR focusses on Linux which allows for docker containers to be used in an enterprise network configuration. I focussed on 2 workflows: 1) The 'dev' workflow, and 2) The PR/CI workflow. The dev workflow works well since it's using containers in a docker-compose environment along with volume mounting your current dev's repo enlistment. The PR/CI workflow gives us an Azure DevOps pipeline to automate verification. I still need to work with the infra team to add a real pipeline that will run. I can't do that until this is merged. In the meantime, I have my own DevOps pipeline that verified this PR. See: https://dev.azure.com/systemnetncl/Enterprise%20Testing/_build/results?buildId=141 I will be linking a follow-up GitHub issue describing the roadmap for building on this system including adding Windows environments, NTLM protocol, proxies, and other libraries such as System.Net.Mail and System.Data.SqlClient. Those libraries also use Negotiate/Kerberos/NTLM enterprise-oriented protocols. Contributes to: https://github.com/dotnet/corefx/issues/41652 https://github.com/dotnet/corefx/issues/41489 https://github.com/dotnet/corefx/issues/36896 https://github.com/dotnet/corefx/issues/30150 https://github.com/dotnet/corefx/issues/24707 https://github.com/dotnet/corefx/issues/10041 https://github.com/dotnet/corefx/issues/6606 https://github.com/dotnet/corefx/issues/6161 * Address PR feedback * Change pipeline *.yml to only run on selected filepaths for PRs * Change kdc container Dockerfile to be based on ubuntu:18.04 * Fix typo in README.md * Update .yml file * Link (instead of copy) apache kerb module to the right place
|
Hi! I was just wondering if there is a solution for this problem somewhere? Although I do not have the knowledge on display in this thread, I do have very similar problems. I have a very simple .netcore (3.1.401) console app, comparable to the app that is referenced above - trying to make requests to a WCF service (net.tcp/AX) from a CentOS 7 environment. Kerberos is setup - to my knowledge - correctly (not by me, by professionals within this area of expertise) and the code itself is quite similar to @jonwingfield 's. Just a simple "authenticate and get some data, please".
Of course I will supply logs & code if necessary, I was just wondering if there is a magic solution or at least further development that I can explore. Cheers, Joakim |
|
As mentioned in the comment above: #31040 (comment) Server not found in Kerberos database means the SPN of the target server used by the client doesn't match anything on record in AD. Find out what SPN is being used and update the client to use it, or update the AD service account with the SPN that's actually being used. |
|
@joakimobstfelder, look at the generated code for where an EndpointAddress is constructed. One of the parameters passed to the EndpointAddress constructor will be an EndpointIdentity. Depending on how the server is being run, it could be either an SpnEndpointIdentity or a UpnEndpointIdentity. This needs to be set correctly. If it's using the EndpointAddress overload where an identity isn't provided, the the spn that the client will use is based on the hostname your provide to WCF. |
Hi,
So I'm really struggling with getting a .NET Core app running on Linux because of Kerberos. The app is a client that is communicating with a WCF service running on a windows server. The Service runs under a service account, so the client specifies a UpnEndpointIdentity. This works great on windows, but when trying to run it on Ubuntu, I'm getting GSSAPI errors. Specifically this:
I'm no Kerberos expert, but I've started to dig into the Wireshark traces and it looks like on windows it's sending a request to the KDC with an sname name-type of
kRB5-NT-SRV-HST (3). On Ubuntu, it's instead sending name-type:kRB5-NT-PRINCIPAL (1). The KDC is rejecting the request withKRB5KDC_ERR_S_PRINCIPAL_UNKNOWN, which makes sense since the client is asking for a host instead of a principal.Also, I'm able to correctly acquire a service ticket using
kgetcredfor the UPN, and I can see it correctly sent in the wireshark trace. Is this a bug in .NET Cores interaction with GSSAPI, or WCF, or am I just missing something? @davidsh Wrote some of this code so I'm hoping he can help.I'm happy to provide the Wireshark traces if that will help, or any more details needed.
Thanks!
The text was updated successfully, but these errors were encountered: