External Credential using OAuth 2.0 with JWT Bearer Flow deployed to Scratch Org does not work

[jclark-dot-org]

Summary

I have an sf project that integrates with the Google API using Named Credentials, which in turn use an External Credential which authenticates as a Google Service Account using OAuth2.0 with JWT Bearer Flow. It works in the original scratch, and I can configure it manually in another scratch org, but I cannot successfully build a working new scratch with sf project deploy start. The source deploys, but the integration does not work because the certificate is not imported correctly ("Unable to complete the JWT token exchange. Error: invalid_grant. Error description: Invalid JWT Signature.")

Steps To Reproduce

I cannot provide a repository, because I would need to expose my Google certificate.

Here are the basic steps to setup the Named Credential/External Credential/etc. I can provide full details with links, etc, but it's not required to understand the issue. Mostly I have followed the steps in this Salesforce StackExchange answer. See "Additional Information" below for a concise explanation of the issue.

1. In Google Cloud Console: Create a Google Cloud Application, add a Service Account, and create a Key for the service account. Download as a p12 file. Make a note of the Service Account's Email Address.
2. Use keytool to convert the p12 file to a JKS file.
3. Create a scratch org.
4. In order to successfully import the JKS file to Salesforce, you must first setup an Identity Provider as follows: Create a Self-signed Cert (Setup > Security > Certificate and Key Management > Create Self-Signed Certificat), then Enable Identity Provider (Setup > Identity > Identity Provider > Click "Enable Identity provider", and choose the cert you created. Once you are done, view the new cert; note that it expires "31 Dec 9999 23:59:59 GMT", and only specifies a CN, as a long series of numbers.
5. Now you can upload the JKS file: Setup > Security > Certificate and Key Management > Import from Keystore. Select the JKS file and provide the PW. The new certificate will be named "privatekey" by default; you may choose to rename it. I did not.
6. Create an External Credential: Setup > Security > Named Credentials > External Credentials > New. Set a Name and Label, and choose "OAuth 2.0" as the auth protocol. Set Auth Flow Type to JWT Bearer Flow, and set identity provider url to "https://oauth2.googleapis.com/token". In the claims section, set the "iss" and "sub" claims to the Service Account email address from Step 1, and set "aud" to
   "https://oauth2.googleapis.com/token". Add a claim, name it "scope", and set it, including one or more Google API scopes; I am using "openid email https://www.googleapis.com/auth/drive".
7. Add a Principal, as a Named Principal. Leave Scope blank here.
8. Create a Named Credential: Setup > Security > Named Credentials > New. Set a name and label, and set the URL to "https://www.googleapis.com/drive/v3". Set External Credential to the credential you created in step 6.

That's all the manual config, and any apex code that uses the Named Credential will work. As to the issue:

1. Retrieve all of the project metadata using sf project retrieve start. This should include certs for both the IDP cert from step 4 and the imported Google cert from step 5, the external credential from step 6 and the named credential from step7, as well as settings/Security.settings-meta.xml. I'm pulling Security Settings because it showed as a modified file, and I assumed that was to reflect the changes from `Enable Identity Provider" in step 4 above. As you will see below, that seems to be wrong.
2. Create a new scratch org, and push the metadata with sf project deploy start. This will succeed.
3. Open the new scratch org, and go to Setup > Security > Certificate and Key Management. View the "privatekey" cert, and note that it expires in a year, and specifies CN=privatekey and "O=Salesforce.com" among other values; compare to step 4 above. This is not the google key imported in the first scratch org; it appears to be a Salesforce-self signed cert. See screenshots below.
4. Now go to Setup > Identity > Identity Provider, and note that the "Enable Identity Provider" is available.

Expected result

The correct Google Certificate should have been created in the new scratch org. Identity Provider should already be enabled.

Actual result

A cert with the same name is created, but it is a different cert, and appears to be a Salesforce self-signed cert, not the Google Service Account key. Identity Provider is not enabled.

Additional information

The issue is that a certificate imported into salesforce in one org, retrieved via sf project retrieve, and pushed to another org, is not created correctly in the second org. This may be a side effect of the fact that enabling an Identity Provider in one scratch doesn't seem to be transferred to another scratch.

As an additional test, I ensured that my project was completely in sync with the second scratch org (sf project retrieve preview showed no differences), then Enabled Identity Provider, then checked org state with sf project retrieve preview again; it showed no differences.

Here's what the import key looks like in the original scratch org:

And here's what it looks like after being deployed to a new scratch org:

System Information

Using zsh on Mac OS 14.6.1

sf cli info:

{
  "architecture": "darwin-arm64",
  "cliVersion": "@salesforce/cli/2.58.7",
  "nodeVersion": "node-v20.17.0",
  "osVersion": "Darwin 23.6.0",
  "rootPath": "/Users/jclark/.nvm/versions/node/v20.17.0/lib/node_modules/@salesforce/cli",
  "shell": "zsh",
  "pluginVersions": [
    "@oclif/plugin-autocomplete 3.2.2 (core)",
    "@oclif/plugin-commands 4.0.13 (core)",
    "@oclif/plugin-help 6.2.10 (core)",
    "@oclif/plugin-not-found 3.2.18 (core)",
    "@oclif/plugin-plugins 5.4.6 (core)",
    "@oclif/plugin-search 1.2.7 (core)",
    "@oclif/plugin-update 4.5.5 (core)",
    "@oclif/plugin-version 2.2.11 (core)",
    "@oclif/plugin-warn-if-update-available 3.1.13 (core)",
    "@oclif/plugin-which 3.2.12 (core)",
    "@salesforce/cli 2.58.7 (core)",
    "apex 3.4.8 (core)",
    "api 1.2.1 (core)",
    "auth 3.6.54 (core)",
    "data 3.6.5 (core)",
    "deploy-retrieve 3.12.3 (core)",
    "info 3.4.3 (core)",
    "limits 3.3.29 (core)",
    "marketplace 1.2.25 (core)",
    "org 4.5.7 (core)",
    "packaging 2.8.2 (core)",
    "schema 3.3.26 (core)",
    "settings 2.3.16 (core)",
    "sobject 1.4.33 (core)",
    "source 3.5.18 (core)",
    "telemetry 3.6.10 (core)",
    "templates 56.3.17 (core)",
    "trust 3.7.27 (core)",
    "user 3.5.29 (core)"
  ]
}

[github-actions]

Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support.

[WillieRuemmele]

Hey @jclark-dot-org - that's quite a complex setup and a lot to dig through... however, I'm not sure Named Credentials are transferrable between orgs when setup that way, they might have specific ids associated with a specific org that would not deploy.

Have you seen the documentation mentioning a transition to Named Credentials Schema?

furthermore, there's additional setup guidance here and here

[jclark-dot-org]

@WillieRuemmele I believe I am using Named Credentials Schema. This isn't the legacy Named Credentials; it uses an External Credential, Named Principal, and Named Credentials, as outlined in my ticket. They are easily retrieved and deployed; the only issue seems to be the Certificate, which I thought might be tied to the Identity Provider issue.

However, As another test, I created a new scratch, manually created a self-signed cert and enabled Identity Provider, and then pushed my google cert; it still turns into a Salesforce cert (C=USA, ST=CA, L=San Francisco, O=Salesforce.com, OU=ORGID, CN=privatekey). So perhaps the real issue is just retrieving and deploying the certificate with the SF CLI. Would you like me to close this ticket and open a separate, more focused ticket?

[WillieRuemmele]

I guess I'm not sure if this is a CLI issue, or a server-side issue

then pushed my google cert; it still turns into a Salesforce cert

can you set the SF_MDAPI_TEMP_DIR env var, retry that deploy, inspect the directory to see if it's been changed into a Salesforce cert, or if it's still a google cert.

if it's a google cert, something on the server is changing it.

if it's a salesforce cert, then the bug lies in the CLI's domain, and then we can fix it.

[jclark-dot-org]

@WillieRuemmele After doing some more research:

The JKS file used to import the keys includes both a private key and the auth provider certificate for the key. The certificate (*.pem) file retrieved from the org contains only the certificate, not the key. Finally, the Metadata API Reference for the Certificate metadata type notes, "Salesforce doesn’t allow the import or export of the private key via the API."

I'm a bit confused by how the imported cert appears in the org, but that's not a CLI issue, and now I don't think any of this this is a CLI issue, at least as I've written it up, so I'm closing this ticket. I do think there needs to be some way to programmatically/declaratively import keys into a new scratch org, but I'll do some more research and open a new ticket if it makes sense.

Thanks for your time.

[jclark-dot-org]

Closing; see previous comment.