Proof-of-concept: HSM support for keys (#21)

[ximon18]

This PR adds proof-of-concept support for the generation and storage of, and signing with, RSA keys managed by an external "cyptographic token" (whether it be physical or virtual, local or remote) via a PKCS#11 or KMIP interface. For simplicity we collectively refer to this as the HSM feature in Krill, as interfacing with Hardware Security Modules is the primary use case for this feature.

PKCS#11 is a run-time interface to a provider supplied .so dynamic library file which Krill loads into its own process and invokes the functions it provides per the PKCS#11 specification. The dynamic library can either be a complete token local virtual implementation such as SoftHSMv2, or it could be an interface to a local or remote commercial HSM vendor appliance (or perhaps even a cluster of appliances).

KMIP similarly standardizes a similar interface for interfacing with "cryptographic tokens" but the interface is TCP based rather than dynamic library based. KMIP requires servers to implement a binary TCP+"TTLV" protocol, and optionally supports HTTPS+"TTLV"/JSON/XML. This PR only implements support for TCP+"TTLV".

This PR implements support for a subset of PKCS#11 v2.20 and KMIP v1.2 (the first version to support the sign operation). Both PKCS#11 and KMIP offer a wide array of functionality of which we use only the pieces required to generate and manage RSA keys and to sign data with those keys, to generate random numbers, plus any required administrative functions (e.g. login, logout, etc). The functionality is implemented as an alternate but compatible "signer" provider to the OpenSSL based "signer" that is already built-in to Krill.

This PR is NOT ready to be merged yet. Instead it is intended to enable early integration testing and review feedback and to act as the basis on which architectural decisions can be made regarding how to implement HSM support suitable for inclusion in a Krill release.

Tested manually with SoftHSMv2 v2.6.1, PyKMIP v0.10.0, AWS CloudHSM (via PKCS#11) and Kryptus kNET (via both PKCS#11 v1.22.0 and KMIP TCP+TTLV).

A note about the X.509 SubjectPublicKeyInfo KeyIdentifier:

Krill uses the key identifier of the public key in various places. We therefore need to be able to generate this on data extracted from a SubjectPublicKeyInfo representation, or generate it from raw RSA key components (modulus & public exponent).

PKCS#11 v2.40 introduced the CKA_PUBLIC_KEY_INFO attribute which in theory can be used to obtain the SubjectPublicKeyInfo for a key pair, but the PKCS#11 implementation does not have to support it and in fact SoftHSMv2 returns an empty value for this attribute.

Instead we use the older PKCS#11 support for fetching the RSA modulus and public exponent attribute values for the created key pair and construct the public key from these with support from a fork of the bcder Rust crate. A fork is need in order to support manual construction of a PublicKey object from the RSA modulus arbitrary length PKCS#11 "Big Integer" value and public exponent value as bcder lacked support for encoding arbitrary length unsigned integers. The forked changes have been proposed for inclusion in bcder upstream but even if that is accepted, Krill does not currently compile against newer bcder and rpki crates and would need to be updated to use the new feature in the new create versions.

Testing:

Manual testing with various providers (see above) and a full run of cargo test with use of PKCS#11 with SoftHSMv2 have been performed, but as yet there are no automated tests of the functionality in this PR. Manual verification of the signing has also been done by logging the signing inputs to and outputs from Krill and using command line openssl to verify data signed with the same key extracted from SoftHSMv2, PyKMIP, Kryptus kNET and AWS CloudHSM.

We need to decide how we want to automatically test this functionality. Options include a mock PKCS#11 implementation and using SoftHSMv2. For the latter we would need to decide if that should be part of the integration test suite (e.g. as is currently done for the ui-tests feature using Cypress via Docker for headless browser based testing) and/or as part of the Krill e2e test.

All testing was done on Ubuntu 18.04 LTS and Ubuntu 21.04. This code has NOT been tested on MacOS or Windows or other Linuxes.

Usage:

The feature is hidden behind a Rust "hsm" feature flag. This PR supports extensions to krill.conf to load a PKCS#11 library or to interface with a KMIP server. See the comments in krill-hsm.conf.

Via configuration it is also possible to tell Krill that one signer is to be used for key rolls such that krillc keyroll initiate will create the new key in the specified signer instead of in the "default" signer (which is also configurable).

Krill has also been extended to include the Key Identifier, signer name and HSM key ID in the REST API and krillc output when "show"ing a CA which can be useful when trying to correlate the keys in Krill with those seen in the HSM.

Currently when using PKCS#11 one must know the "slot" ID, Krill does not enumerate them and pick the first nor is it able to find them by some sort of "label". The slot ID must be found using an external tool such as pkcs11-tool or softhsm2-util --show-slots.

Architecture:

Currently this work has been implemented by making the existing OpenSslSigner and the new Pkcs11Signer and KmipSigner variants of a SignerImpl enum. The implementation currently uses duplication of code in signing.rs which is ugly and hopefully can be avoided. There is also an ongoing discussion about whether to use the HSM for all keys or only for a master key with on-disk "child" keys.

As the Key Identifier cannot be known until after the key is created and as AWS CloudHSM does not permit setting attributes on keys post-creation, this implementation instead generates a random 20 byte identifier for "tagging" created keys with and uses that to look them up in the HSM when it needs to use them to do signing or to delete them. This requires that a mapping be maintained of Key Identifiers to HSM key identifiers. This mapping is currently implemented as a simple in-memory HashMap with writes being persisted by appending to an on-disk simple CSV file which is loaded on Krill startup and used to reconstruct the HashMap contents. The mapping ensures that as long as a signer name remains constant that a key can be looked up in and used with the correct "cryptographic token".

As noted above this work depends on forks of the bcder and rpki-rs Rust crates and on new kmip crates created by NLnet Labs as part of this work.

Additionally we may want to consider using runtime signer configuration via krillc instead of krill.conf startup time based configuration, for example to ensure that changes possible to signer configuration after a key has been created (e.g. to point it to a complete different "cryptographic token") are limited and to use a signer ID based approach for the key mapping such that it is possible to rename signers without breaking Krills ability to find and use the keys associated with that signer.

Observations:

Krill doesn't currently ever delete keys. This PR implements key deletion capability but only the one off key signing code uses it.

[ximon18]

I'm going to close this PR as the majority of the exploratory work has been done and reviewing and maintaining and merging a large PR like this is undesirable, difficult and risky (as significant conflicting changes have been made to Krill since this work started and it becomes harder to know if some valuable change is not going to be accidentally reverted by merging the prototype code in). Instead a series of smaller PRs will be used to add final support for HSMs to Krill. The branch backing this PR will remain for now as a source for basing the actual final code on.