Add DeriveKeyPair API

[Eddy-M-K]

Add derandomized keypair generation and encapsulation API for all KEM schemes (only ML-KEM has an internal implementation at the moment, all other schemes will throw an error).

Closes #1206

• Does this PR change the input/output behaviour of a cryptographic algorithm (i.e., does it change known answer test values)? (If so, a version bump will be required from x.y.z to x.(y+1).0.)
• Does this PR change the list of algorithms available -- either adding, removing, or renaming? Does this PR otherwise change an API? (If so, PRs in fully supported downstream projects dependent on these, i.e., oqs-provider will also need to be ready for review and merge by the time this is merged.)

keypair_derand() and encaps_derand() are added to the API of all KEM schemes.

[baentsch]

First off: Thanks for the work making this available, @Eddy-M-K -- and for more than just one algorithm!

If I understand it right, you're adding two new functions to the KEM algs, thus requiring all consumers of the library that want to make use of this to write lots of new code (essentially replacing/augmenting keygen/encaps with their new de-randomizing twins): Did I get this right? Is there no way to avoid that (say, by creating a setup function changing the semantics of keygen/encaps)? Is it really necessary to set "randomness"/well, "seeds" at every API invocation?

[baentsch]

OK -- now I understand some more: The underlying implementation asks for the seed/coins to be passed in each call. This however begs the question: How many "coins" are needed to correctly operate this API? Shouldn't this be documented & checked here? Equivalent question: How many coins are needed for the keygen API call? Beyond the ability to improve API robustness (check this at invocation), if this were known(?) wouldn't it be possible to pass a sufficient number of "coins" to a "setup_random_coins" call that is invoked once and then used "under the covers" for each subsequent call to (API-wise unchanged) keygen and encaps calls? That way the many "OQS_ERROR" return responses for the KEM algs not supporting that could be saved: They simply wouldn't use "prepared coins" when asked to operate keygen&encaps.

[SWilson4]

Sorry it's taken me so long to follow up on this, Michael.

For the first question, length_keypair_coins and length_encaps_coins are added to the OQS_KEM struct; the caller can use them similarly to length_shared_secret, length_ciphertext, etc.

For the second, first let me confirm I'm understanding correctly: setup_random_coins would be a one-time call to set up (i.e., seed) a source of randomness which would then be used as RNG for OQS_randombytes in subsequent KEM operations? This would be similar in functionality to the OQS_randombytes_custom_algorithm call but wouldn't require the caller to define their own function and pass it as a pointer—correct?

The purpose of this PR is to achieve something different. The keypair_derand call is intended to implement (well, expose) the DeriveKeyPair KEM function defined in RFC 9180. This is used notably in the recent Messaging Layer Security standard. By design, this is basically a KDF for KEM keys—the caller provides input keying material and gets back a KEM keypair—which needs to be called multiple times rather than having a one-time setup. The intent is to supplement the existing keygen function, not replace it: RFC 9180 requires that the KEM supports both operations. ML-KEM already supports the DeriveKeyPair operation as part of its API via keypair_derand, as well as a similar encaps_derand function. (I don't know an application for the latter off the top of my head.) I imagine it would have been nice to have this background before you looked at the PR; sorry for the poor coordination on my part.

[baentsch]

Thanks for the explanations. I now understand some more -- but clearly not everything: I thought this PR is meant to enable open-quantum-safe/oqs-provider#447? If this were so (?), I'm afraid this API will require substantial changes to oqsprovider to facilitate this, no? But then again, maybe you or @Eddy-M-K can show me wrong by doing a simple PR implementing open-quantum-safe/oqs-provider#447 based on this new API (maybe a "tracking-..." branch to trigger CI as we also do with new algs?). Again, will be online only by smart phone until Aug 19, so please do not expect serious further feedback from me until then.

[bifurcation]

@Eddy-M-K @baentsch - What do we need to do to get this PR un-stuck? Support for HPKE is also blocked on the lack of DeriveKeyPair.

FWIW: I would simplify this PR some:

• Remove the encaps_derand() parts, since they're not required for DeriveKeyPair
• Don't change documentation for non-upgraded KEMs; if DeriveKeyPair isn't supported, there's no need to talk about its parameters

Also, a naming nit: Instead of _keypair_derand, I would use _derive_keypair. AFAIK, the derand terminology is not widely used; _derive_keypair would at least align terminology with HPKE, which is the primary consumer and deployed on some scale.

[SWilson4]

@Eddy-M-K @baentsch - What do we need to do to get this PR un-stuck? Support for HPKE is also blocked on the lack of DeriveKeyPair.

Truthfully, this has stalled mostly to our limited developer time and other features being higher priority (it's not a coincidence that the last activity on this PR was the Friday before the NIST final standards were released). If you are interested in getting this PR over the finish line, the help would be more than welcome.

One open question is the oqs-provider integration: I had hoped that we could kill two birds with one stone and use the new API to solve open-quantum-safe/oqs-provider#447. For HPKE, would you be looking to use liboqs APIs directly or go via the OpenSSL3 provider?

FWIW: I would simplify this PR some:

* Remove the `encaps_derand()` parts, since they're not required for DeriveKeyPair

* Don't change documentation for non-upgraded KEMs; if DeriveKeyPair isn't supported, there's no need to talk about its parameters

I would have no objection to removing encaps_derand. I'm not sure about the latter point, as we have yet to implement an API function that is only supported by one algorithm and generally aim for uniform support (see the spirited discussion on #1919). I don't believe that we would be able to achieve the latter goal for this feature, however.

Also, a naming nit: Instead of _keypair_derand, I would use _derive_keypair. AFAIK, the derand terminology is not widely used; _derive_keypair would at least align terminology with HPKE, which is the primary consumer and deployed on some scale.

I have no objection to this, either.

I do have one question as I revisit this PR after the publication of FIPS 203: can we expose the "derand" (which I suppose is now "internal") API without running afoul of the final standard? This paragraph at the top of page 16 makes me wary:

Controlled access to internal functions. The key-encapsulation mechanism ML-KEM makes use
of internal, “derandomized” functions ML-KEM.KeyGen_internal, ML-KEM.Encaps_internal, and
ML-KEM.Decaps_internal, specified in Section 6. The interfaces for these functions should not
be made available to applications other than for testing purposes. In particular, the sampling of
random values required for key generation (as specified in ML-KEM.KeyGen) and encapsulation
(as specified in ML-KEM.Encaps) shall be performed by the cryptographic module.

[baentsch]

I do have one question as I revisit this PR after the publication of FIPS 203: can we expose the "derand" (which I suppose is now "internal") API without running afoul of the final standard?

Welcome to the club @SWilson4 : See e.g., openssl/openssl#25938 (comment) now put into documentation in openssl/openssl#26037. Looks like there really is an advantage contributing to different projects aiming at solving the same problem... :-)