shim-15.8 for WAXAR

[Jurij-Ivastsuk]

Confirm the following are included in your repo, checking each box:

• completed README.md file with the necessary information
• shim.efi to be signed
• public portion of your certificate(s) embedded in shim (the file passed to VENDOR_CERT_FILE)
• binaries, for which hashes are added to vendor_db ( if you use vendor_db and have hashes allow-listed )
• any extra patches to shim via your own git tree or as files (we use no extra patches)
• any extra patches to grub via your own git tree or as files (no extra patches, we use Canonical upstream for GRUB)
• build logs
• a Dockerfile to reproduce the build of the provided shim EFI binaries

---

What is the link to your tag in a repo cloned from rhboot/shim-review?

---

https://github.com/Jurij-Ivastsuk/WAXAR-shim-review/tree/waxar-shim-x86_64-aarch64-20240601

---

What is the SHA256 hash of your final SHIM binary?

---

SHA256 (shimx64.efi): 1894c7e467991c117162e1e40dd61ed85a5f790a68216fdbee93b2619b70df67
SHA256 (shimaa64.efi): 5c7238473401d791c7f376efee90cf1e9fe23e2bf1814f4795474d57920bdfbf

---

What is the link to your previous shim review request (if any, otherwise N/A)?

---

N/A

[aronowski]

Hello.

I'll start reviewing the application soon, but the first requirement is that all the checkboxes are checked and render well.

Please, fix the formatting and check the missing one - it's supposed to mean "if we have any extra patches, then they have been included."

[Jurij-Ivastsuk]

@aronowski Hi, many thanks for any tips!
I literally understood point 6 as "if you have extra patches for GRUB..." Since we use the GRUB software upstream from Canonical, and don't make any changes there, with patches or otherwise, I figured I shouldn't mark this point as done.

[aronowski]

@Jurij-Ivastsuk, thank you!

I agree that some part in here may not be clear, but I'll try my best to address them. Some documents I've been working on improving during my free time, but haven't requested to incorporate them so far.

Reviewing also may not be easy especially when dealing with environments I'm not that familiar with or analyzing patches for correctness (took me about 8 hours to spot one major error, but I did it). If there's no activity e.g. by the end of next week, ping me.

[Jurij-Ivastsuk]

@aronowski, I understand and I thank you for the work you do in your spare time!

[aronowski]

Hello.

I did scratch the surface of the application. There are some curiosities and things that need to be fixed. I address them as some loose thoughts below.

In the meantime, I'm sending contact verification emails soon. Please, post the random words, that I'm emailing you, in the comments.

---

The NX compatibility bit should be enabled only if the whole bootchain is NX compatible - that's the requirement Microsoft has, as I'm aware right now. Therefore, the patch 530 shall not be used.

Furthermore, the validation function patch does its job in regard to older requirements, which have changed - there's no need to use it today.

Please, update the binaries and the building process as described - I'll then try to rebuild it and check if everything is correct.

---

The 626.patch PR hasn't been merged and debugging the code in my head is out of the question. It's just a one-line change, but no idea about the consequences and possible corner-cases, especially since I'm not a low-level programmer.

I'm not saying that it's unable to be approved as-is - just that I'll need some help with it and wondering, how to get some.

---

Linux required upstream commits - are they applied for the 5.15.1 Debian kernel?

The commit eadb2f47a3ced5c64b23b90fd2a3463f63726066 as far as I know got introduced in Linux v5.19. Is it backported in Debian?

---

There's no ephemeral key - is the strategy sufficient, as it's described right now?

It would be preferable to mention the usage of CONFIG_MODVERSIONS=y and CONFIG_MODULE_SIG_FORCE=y. Furthermore, is a new key being used to sign each new kernel build?

---

The CA certificate waxar.ceris valid for almost 100 years and it's only 2048 bit.

I don't know if Microsoft would be willing to approve this - any chances of generating one with a shorter lifespan and/or using 4096 bit public modulus?

[aronowski]

Verification emails sent.

[Jurij-Ivastsuk]

I have received your email. Here is the decrypted text:
summerhouses ants regionally announces unmasking fusing addling bantered
sprouted chiropractic

[Jurij-Ivastsuk]

@aronowski Hi, thank you very much for the first hints!
Now I have changed the following:

1. patches 530 and 531 (NX compatibility) have been removed
2. docker file adapted accordingly
3. CA certificate waxar.cer is now valid for 50 years and its size is now 4096 byte
4. for kernel building we use CONFIG_MODVERSIONS=y and CONFIG_MODULE_SIG_FORCE=y
5. the new certificate is already used for signing kernel build as well as for signing all modules
   Regarding commit eadb2f47a3ced5c64b23b90fd2a3463f63726066 (CVE-2022-21499) in Debian, I have found the following information:
   bullseye 5.10.197-1 fixed
   bullseye (security) 5.10.205-2 fixed
   bullseye 5.10.120-1 fixed
   bookworm 6.1.66-1 fixed
   Exactly for kernel 5.15.1 no Information could be found...
   As for 626.patch, I understand that you cannot be responsible for all matters. Maybe you can ping someone who does low-level programming and who can properly assess our PR request. Actually I explained the problem and the solution pretty well. Without this patch we can't use shim like this because it keeps giving us errors (screenshot and explanation in PR 626).

[Jurij-Ivastsuk]

@aronowski Hi, should we now switch to 15.8 ?

[aronowski]

Updating should extend the lifecycle of the review if something happened and if reviews from other people from the committee got delayed. I'd do so, especially since I just scratched the surface and will perform a further review, after the latest fixes have been applied.

[Jurij-Ivastsuk]

@aronowski Hi,
the second contact (Lukas Kienbaum) have received your email (shim-words x2). Here is the decrypted text:
hauled multiplication Collins reemerge exhilarates fathers Vijayawada
stubbiest fête refile

[Jurij-Ivastsuk]

@aronowski Hi, did I understand you correctly, we are waiting until the last reviews from the committee are closed, then you will continue with your review with us? I still don't understand if we will stay with version 15.7 or will we logically switch to 15.8. In this case, if I understand correctly, a new review "shim-15.8 for WAXAR" will be requested. Is that the case?

[aronowski]

@Jurij-Ivastsuk

Please, update all the required things to reflect shim 15.8. I'll then perform further review and ask other reviewers for help (you can ping them as well).

---

we are waiting until the last reviews from the committee are closed, then you will continue with your review with us?

I'll try my best when there's optimal environment for me - for instance, the comment from January 22 I posted in the middle of the night in my timezone, as I had some free time and no other things on my mind.

Still, there are other things that can make some applications being reviewed ASAP and some that wait for, let's say, a month or even more. The main thing is that I'm just more familiar with certain distros and their environments.

Some things I gathered as part of the improvements I'd like to introduce to the project, so they are blatantly visible, e.g. here.

---

CA certificate [...] size is now 4096 byte

4096 byte? ;-)

[Jurij-Ivastsuk]

@aronowski Hi, thank you very much for the tips and explanations! I really like your suggestions for improvements. I have to apologize for the incorrect representation of the certificate size: of course it's bits and not bytes. So the CA certificate size is now 4096 bits... ;-)))
I'll let you know when I've made all the adjustments for 15.8.

[Jurij-Ivastsuk]

@aronowski Hi, I have a question, can you please explain to me why the CA certificate must be 4096 bit in size and not 2048 bit?

[aronowski]

@Jurij-Ivastsuk, it's not like it must be this or that, but instead a good practice to follow the recommendations from NIST.SP.800-57pt1r5 - table no. 2 on page 54 and table no. 4 on page 59.

I'll review all the updates soon, once I get more free time and some well-deserved sleep.

[Jurij-Ivastsuk]

@aronowski Hi, please only continue with review if you have slept enough. i have made all adjustments for version 15.8 ;-))

[aronowski]

@Jurij-Ivastsuk,

Exactly for kernel 5.15.1 no Information could be found...

Apologies, that's the Ubuntu Jammy kernel, rather than Debian's as I thought earlier.

So I think our best bet is to wait for an official answer from Canonical kernel maintainers. Please see #368 (comment).

---

please only continue with review if you have slept enough

Let's say 6 hours is enough for now. :-P

I was checking things out and got stuck with the kernel-related thing. Let's hope it gets resolved soon.

[Jurij-Ivastsuk]

@aronowski , @THS-on Hello, I wanted to kindly ask if there is any progress in our case ?

[aronowski]

@Jurij-Ivastsuk,

Hello, I wanted to kindly ask if there is any progress in our case ?

I suppose it will be more helpful if I start raising both positive feedback, as well as potential issues immediately, rather than doing one big review. The latter is tough for several reasons, one of them being the lack of having a comfortable office with air conditioner - doing an analysis in over 30 degrees Celsius does not help at all.

The curiosity I noticed:

Next, we set the parameter: CONFIG_MODULE_SIG_KEY="certs/signing_key.pem" (see config file).
The string provides the path to a file containing both a private key and its corresponding X.509 certificate in PEM form.
For security reasons, these files are not included in current Sources (linux-source-515.tar.gz).

According to https://www.kernel.org/doc/html/v5.15/admin-guide/module-signing.html:

File name or PKCS#11 URI of module signing key (CONFIG_MODULE_SIG_KEY)

Setting this option to something other than its default of certs/signing_key.pem will disable the autogeneration of signing keys and allow the kernel modules to be signed with a key of your choosing.

Was this intentional? Is it possible to check the compiled modules through static analysis, if they were signed with the appropriate signing key?

[Jurij-Ivastsuk]

@aronowski Hello, and thank you for your questions.
I can well understand that it is indeed difficult to work at temperatures above 30 degrees Celsius.
I hope that the temperature conditions will be a little more bearable for you in the next few days.

To your questions, during the compilation with CONFIG_MODULE_SIG_ALL=y we noticed that if there are no pub- and priv-keys
in the certs directory, they are not created automatically as the documentation says, no idea why.
But if we generate both keys beforehand and then insert them into the certs directory, then all modules are signed.
That is the reason for this procedure.

To check whether modules are actually signed we have 2 possibilities (as an example on the ext4.ko module):

1. We see a line in the building log: SIGN debian/linux-image/lib/modules/5.15.1-devimg-waxar/kernel/fs/ext4/ext4.ko
2. With the help of modinfo:
   modinfo ext4.ko
   filename: /usr/src/Test_Modules_Signature/ext4.ko
   softdep: pre: crypto-crc32c
   license: GPL
   description: Fourth Extended Filesystem
   author: Remy Card, Stephen Tweedie, Andrew Morton, Andreas Dilger, Theodore Ts'o and others
   alias: fs-ext4
   alias: ext3
   alias: fs-ext3
   alias: ext2
   alias: fs-ext2
   depends: mbcache,jbd2,crc16
   retpoline: Y
   intree: Y
   name: ext4
   vermagic: 5.15.1-devimg-waxar SMP mod_unload modversions
   sig_id: PKCS#7
   signer: Build time autogenerated kernel key
   sig_key: 30:00:4F:89:A6:9D:0E:78:07:FB:35:C5:A8:07:17:E7:8E:C4:39:C1
   sig_hashalgo: sha256
   signature: 04:16:59:88:3C:30:C2:19:12:CB:74:DE:AF:B8:25:49:4A:EE:F0:6A: .....

I hope I have answered all your questions. If you have any further questions, please do not hesitate to contact me.

[aronowski]

@Jurij-Ivastsuk,

To your questions, during the compilation with CONFIG_MODULE_SIG_ALL=y we noticed that if there are no pub- and priv-keys
in the certs directory, they are not created automatically as the documentation says, no idea why.
But if we generate both keys beforehand and then insert them into the certs directory, then all modules are signed.
That is the reason for this procedure.

As far as I can see the process does seem to work like the kernel documentation says. The modinfo listing says:

[...]
signer: Build time autogenerated kernel key
[...]

This matches the process described in the "Generating signing keys" section of the v5.15 kernel documentation. In particular the x509.genkey file should be generated by your certs/Makefile.

For extra safety I'd also check your kernel's keyring on a booted system. Please see, if anything is printed by running:

# grep "Build time autogenerated kernel key" /proc/keys

on the current compilation of your system. If the same happens, then the usage of an ephemeral key is actually present here, most likely accidentally, as your shim application says otherwise (No worries - things like that may happen and that's why the peer-review process is here to help).

What I'd also recommend is to edit the kernel's config file, change the current line:

CONFIG_MODULE_SIG_KEY="certs/signing_key.pem"

to something like

CONFIG_MODULE_SIG_KEY="certs/waxar_signing_key.pem"

and try again by renaming the file with your private key and its corresponding certificate to waxar_signing_key.pem. Once the kernel's compiled, see if modinfo prints something different, as intended in the application.

[Jurij-Ivastsuk]

@aronowski Hi, we have checked your suggestions:
# grep "Build time autogenerated kernel key" /proc/keys
there is no output, the string does not exist

After new compilation we have checked the same module (modinfo ext4.ko)
[...] signer: Build time autogenerated kernel key [...]
No changes in this line.

[Jurij-Ivastsuk]

@aronowski Hello, I have now created a commit with the current Shim version 15.8.
I have compiled this Shim variant without using the improvement I suggested earlier (Improving the robustness of value retention for the variable second_stage #626). The main change: in this variant I use the parameter DISABLE_REMOVABLE_LOAD_OPTIONS for compiling (see https://github.com/rhboot/shim/blob/main/BUILDING).
With this parameter I can solve my problem without using the patch #626.

Changes in this commit:

1. current link to the current tag: waxar-shim-x86_64-aarch64-20240531
2. current SHA256 hashes for both files shimx64.efi and shimaa64.efi
3. Dockerfile now reflects the changes in the building process
4. build.log file now reflects the changes in the building process
5. both files shimx64.efi and shimaa64.efi are now up to date

You can access this commit under the following link:
https://github.com/Jurij-Ivastsuk/WAXAR-shim-review/tree/waxar-shim-x86_64-aarch64-20240531

[Jurij-Ivastsuk]

@aronowski Please don't be surprised, I have forgotten something:

6. Inserted changes and adjustments to the README.md file to reflect the latest changes in relation to recompiling without patches

Now all the documentation is complete... I think :-)
You can access this commit under the following link:
https://github.com/Jurij-Ivastsuk/WAXAR-shim-review/tree/waxar-shim-x86_64-aarch64-20240601

[aronowski]

@Jurij-Ivastsuk, I'll take a look at the updated shim no sooner than this Sunday. Please let me know everything I need to know by then.

[Jurij-Ivastsuk]

@aronowski Hello, I'm sure you are responsible for many application reviews and each application has its own special features. Therefore, in order to simplify things and time savings for you with our application, so that you have a better overview, I am writing a short summary of what we have done together so far.

1. We have checked email contacts they are OK
2. We have created a new certificate with 4096 bit and a validity period of 50 years (instead of 2048 bit, 100 years) and implemented it for the compilation of the current Shim version (15.8) and signing of our boot components. You have checked this certificate and it was OK
3. I had quite a long discussion with @Metabolix about the usefulness of the PR 626 (Improving the robustness of value retention for the variable second_stage shim#626). In the end, although the solution we proposed in PR 626 turned out to be viable, we found a solution that solves our problem and does not change the code (using the parameter DISABLE_REMOVABLE_LOAD_OPTIONS=y). So I think using this parameter will make the review process easier for you, because there are no more code changes to discuss.
4. According to your recommendation, we have changed the config file for kernel compilation and activated the parameters CONFIG_MODULE_SIG_ALL and CONFIG_MODULE_SIG_FORCE. So that the modules are automatically signed with a certificate during kernel compilation. This increases the security that the kernel cannot load any foreign modules. We can also prove that all modules are signed accordingly after kernel compilation.

All changes and adjustments are now included in the last commit (https://github.com/Jurij-Ivastsuk/WAXAR-shim-review/tree/waxar-shim-x86_64-aarch64-20240601)
Please let me know if something is missing.

[THS-on]

By just looking at the provided kernel config and setup description this contradicts the statement made in the README.
If you have the private key in a file, it is not secured by an HSM. Please clarify the following:

• How are your signing keys stored for the kernel modules?
• Why are you not able to use ephemeral key signing?
• How do you ensure that kernel modules from older kernels cannot be loaded on newer ones?

@aronowski if I see it correctly the cert is only newly generated if the x509.genkey changes. See https://github.com/torvalds/linux/blob/83a7eefedc9b56fe7bfeff13b6c7356688ffa670/certs/Makefile#L52-L53

[Jurij-Ivastsuk]

@THS-on , @aronowski Hi, thank you for questions.
In HSM only our private key for our certificate is stored which we only use for our CA certificate, which in turn is used for signing kernel and grub.

• If we compile a new Linux kernel with the options CONFIG_MODULE_SIG_ALL and CONFIG_MODULE_SIG_FORCE, the signature keys are handled as follows:
  During the kernel compilation process (or, in our case before compilation), a new key pair (private and public key) is generated for signing the kernel modules. The private key is used to sign the modules during the compilation. The public key is integrated into the compiled kernel. This is then used when loading the modules to check their signature. The generated keys are not stored persistently. A new key pair will be generated with each new kernel compilation. We delete the private key immediately after kernel compilation to prevent misuse. The public key is contained in the compiled kernel and does not need to be stored separately.
• As I wrote earlier in this chat, we can't use the ephemeral key because building and signing are done on different machines and we cannot rule out the possibility that there will be a need to build out-of-tree modules.
• In order to ensure that kernel modules from older kernels cannot be loaded on newer ones, We always remove all old modules after upgrading to a new kernel version.
  The simplest way to prohibit a kernel to load older kernel modules, is to use an ephemeral key during build time, but due to the restrictions in our case, as I already mentioned, we cannot use them.

[steve-mcintyre]

@Jurij-Ivastsuk Simply deleting old kernel modules from a system is not enough for revocation. You need to have technical measures in place to block them from loading.

There's a few issues here where it's not clear from your submission that you understand how things are meant to work.

[Jurij-Ivastsuk]

@THS-on , @aronowski
Dear @steve-mcintyre thank you for your comments. Our Linux environment is located on the external boot device like CD, USB stick or USB drive (like live-CD). The boot process is initiated by selecting Boot Device from the boot menu.

• When we upgrade our entire system, we have full control over kernel upgrade, modules upgrade in system and modules upgrade in initramfs. Therefore, when we upgrade - we upgrade the entire system, starting from bootloader stage1 and stage2, kernel, initramfs up to all system components and modules. All modules in initramfs and in the system tree are linked to the corresponding kernel. Therefore, I think the treatment of modules from the security perspective is slightly different in our case.

• As I have already mentioned, the simplest mechanism to prohibit a kernel to load older kernel modules, is to use an ephemeral key during build time, but due to the restrictions in our case we cannot use them. Theoretically, there is another possibility to block the loading of old modules, namely to use the blacklisting procedure for modules. But in our case this procedure would not be applicable because the names of most modules remain the same during the transition to a new version.

• Finally, we have a mechanism designed to exclude unwanted modules from loading - signing the modules with a unique license key at compile time. This is the purpose of signing. All modules that do not have the correct license key are ignored.

There's a few issues here where it's not clear from your submission that you understand how things are meant to work

I can't say that I know everything, but unfortunately I can't do anything with this statement of yours. I would be very happy to learn something new from you. If you think you know something more in this context, please share your knowledge with everyone here who is in a technically similar situation and would like to improve their solutions.

[Jurij-Ivastsuk]

Hello dear reviewer @aronowski @THS-on @steve-mcintyre, to answer once again your question about the prevention of loading unwanted modules: we use our own certificate to check the signatures of the modules to be loaded. Our kernel can also be delivered with new keys, if we need it, to prevent old modules from being loaded.

[aronowski]

I'll raise the issue during the next call.

---

If you think you know something more in this context, please share your knowledge with everyone here who is in a technically similar situation and would like to improve their solutions.

I myself was thinking about writing a guide, how to start the shim-related venue from scratch and implement appropriate measures for reasonable security, but taking into account each company's distinct situation, but there's no way I'll be able to account for so many different cases and limitations. Especially doing this in a free time after job hours and considering my own knowledge, experience and limitations.

I could think of something similar to the venue from this thread about smaller companies (one referred there as a fictional MyCompany) and their products deriving from the larger ones, but in the context of shim implementation and the whole bootloader chain (referred as Using downstream implementations from Canonical in your application). Still, as said earlier, I can see too many distinct cases to make it reasonable.

[THS-on]

@Jurij-Ivastsuk regarding the kernel module signing:

• If you need the way to build kernel modules after the initial build, you can include additional kernel specific certificates per each build. Note that it is highly recommended to protect those keys in an HSM or similar. The main reason right now for accepting submissions without ephemeral key signing, is already products that have been signed in the past and when they have a sufficient mitigation strategy.
• How your build is currently documented (https://github.com/Jurij-Ivastsuk/WAXAR-shim-review/blob/waxar-shim-x86_64-aarch64-20240601/Setup_for_module_signing.txt) it looks like the signing keys are in plain text on some build machine and emphermal, which is not sufficient.
• Please elaborate on the implementation with the license key and module signing. I currently don't see the connection to the kernel module loading mechanism

[Jurij-Ivastsuk]

@THS-on Hello, thank you very much for your comments. In our current kernel-mudule signing-setup we currently use two different certificates: 1. our own CA certificate and 2. kernel specific certificate for each compilation.
Can you please explain what you mean by “license key”. Is this a third variant of the certificate?

[THS-on]

In our current kernel-mudule signing-setup we currently use two different certificates: 1. our own CA certificate and 2. kernel specific certificate for each compilation

Can you update your documentation to reflect that? And also describe how your procedure with those keys are in detail?

Can you please explain what you mean by “license key”. Is this a third variant of the certificate?

This refers to your third point from this comment #362 (comment)

[Jurij-Ivastsuk]

Hello @THS-on
Thank you very much for your comments and suggestions.

In order to implement the realization of the “completely secure boot” principle, we have developed an approach with double signing of certain boot components using HSM. We combine the signing of the kernel and modules once with our CA certificate (for kernel) and additionally with unique private key + CA certificate (for modules). This approach for signing the modules is more secure compared to signing only with a unique private key. Additional use of HSM gives us an even higher protection for security because both private keys are only in HSM and never appear as physical files.

We have adapted and updated our setup documentation in line with your comments. Please see here: (https://github.com/Jurij-Ivastsuk/WAXAR-shim-review/blob/waxar-shim-x86_64-aarch64-20240804/Setup_for_module_signing.txt)

[THS-on]

Thanks for the update. Some questions:

• What kind of HSM are you using and who has access to it?
• You seem to sign the installed files, which seems weird. How do you package your kernel? As you are Debian based, have a look at what either Debian or Ubuntu is doing for their kernel packaging
• How does your CA + leaf certificates structure look like?
• Are you rotating the keys or only certificates? Because if I recall correctly the kernel checks on a key basis and not certificate (unless something has changed).

[dbnicholson]

In order to implement the realization of the “completely secure boot” principle, we have developed an approach with double signing of certain boot components using HSM. We combine the signing of the kernel and modules once with our CA certificate (for kernel) and additionally with unique private key + CA certificate (for modules). This approach for signing the modules is more secure compared to signing only with a unique private key. Additional use of HSM gives us an even higher protection for security because both private keys are only in HSM and never appear as physical files.

The kernel only allows and validates a single signature at the end of the module. So, it's not valid to say that modules are signed with both a unique key and your CA certificate's key.

Furthermore, it's more secure to sign modules only with the ephemeral key generated during the kernel build. That way, only modules built for that particular kernel build can be loaded. It ensures that you can't take a vulnerable module from an older kernel build and load it with a current kernel. In that scheme, modules only need to be signed with the ephemeral key during the kernel build. There's no need to keep the ephemeral key or re-sign the modules later. If you did sign the modules with your CA key, then any module ever signed by you would be valid for loading in the current kernel.

The only place this causes an issue is with out of tree modules like the nvidia drivers. In that case, you obviously need to sign them outside of the kernel build. This is what CONFIG_SYSTEM_TRUSTED_KEYS is for and where you'd put your CA certificate. For out of tree modules, you'd sign them with your CA key (or one for a certificate issued by the CA) in an HSM like you would for the kernel itself. This is a weaker but necessary process since you can now load any older signed out of tree module into the current kernel.

Following the standard process (CONFIG_MODULE_SIG_ALL=y) is more secure for handling in tree modules. Do you have out of tree modules that you need to sign? If not, then you don't need to do anything with CONFIG_SYSTEM_TRUSTED_KEYS or do any module signing involving an HSM.

[Jurij-Ivastsuk]

Hello @dbnicholson,
thank you very much for your advice and constructive recommendations.
The use of ephemeral key is certainly the best alternative from a security point of view. However, as I have already explained in an earlier discussion, this is unfortunately technically not an option for us:
#362 (comment)

[Jurij-Ivastsuk]

Hello @THS-on,
Thank you for your questions.

• We use General Purpose HSM, which provides broad cryptographic functionality and can be used for a wide range of applications.
• We can easily replace the double signing of the modules with a single one. This is carried out automatically during kernel compilation with a leaf certificate. This procedure is already described in our setup. The second signing of the modules with our CA certificate will be omitted.
• Kernel compilation and packaging is initiated with:
  make -j6 deb-pkg LOCALVERSION=-devimg-waxar KDEB_PKGVERSION=$(make kernelversion)-2
  The overall process initiated consists of the following steps:

1. kernel compilation
2. module compilation
3. module signing with leaf certificate
4. Debian-packaging of the kernel package for installation
   After the whole process is successfully completed, we get the Debian package for the new version of Kernel:
   linux-image-5.15.1-devimg-waxar_5.15.1-2_amd64.deb
   After the installation of this Debian package we get the kernel and all signed modules installed.

• As I do not have access to my IT-infrastructure at the moment, please have a look at
  https://github.com/Jurij-Ivastsuk/WAXAR-shim-review/tree/waxar-shim-x86_64-aarch64-20240804
  where you will find the file waxar.der, our DER coded public CA certificate.
  Please use the command:
  openssl x509 -in waxar.der -inform der -text -noout
  This will allow you to read out the structure of our certificate.
  I can provide the structure of our leaf certificate later if required.

• Rotation only takes place with Key. With each new compilation, a new key is created accordingly, so that the modules are signed with a new leaf certificate each time they are compiled.

[dbnicholson]

Hi @Jurij-Ivastsuk, I'm trying to understand why this is a blocker. Quoting from that comment:

• If we compile a new Linux kernel with the options CONFIG_MODULE_SIG_ALL and CONFIG_MODULE_SIG_FORCE, the signature keys are handled as follows:
  During the kernel compilation process (or, in our case before compilation), a new key pair (private and public key) is generated for signing the kernel modules. The private key is used to sign the modules during the compilation. The public key is integrated into the compiled kernel. This is then used when loading the modules to check their signature. The generated keys are not stored persistently. A new key pair will be generated with each new kernel compilation. We delete the private key immediately after kernel compilation to prevent misuse. The public key is contained in the compiled kernel and does not need to be stored separately.
• As I wrote earlier in this chat, we can't use the ephemeral key because building and signing are done on different machines and we cannot rule out the possibility that there will be a need to build out-of-tree modules.
• In order to ensure that kernel modules from older kernels cannot be loaded on newer ones, We always remove all old modules after upgrading to a new kernel version.
  The simplest way to prohibit a kernel to load older kernel modules, is to use an ephemeral key during build time, but due to the restrictions in our case, as I already mentioned, we cannot use them.

In this comment it sounds like you're doing exactly what I suggested. An ephemeral key is created during the kernel build and the private key is thrown away after the build. Unless you're following a different procedure, this is what's stored in CONFIG_MODULE_SIG_KEY. If you want to add a 2nd public key for verifying out of tree modules, then add it to CONFIG_SYSTEM_TRUSTED_KEYS.

The in tree modules are already signed with the ephemeral key and will be validated against it with the public certificate stored in CONFIG_MODULE_SIG_KEY. There's no need to sign them again later. The only reason to sign modules outside of the kernel build process is out of tree modules. In that case, build the kernel with that certificate in CONFIG_SYSTEM_TRUSTED_KEYS. So, I don't understand what's unique about your setup that requires you to sign the in tree modules again during the signing phase.

[Jurij-Ivastsuk]

Hi @dbnicholson,
I think I need to explain something here. As you can see, in a short summary on June 7 (ttps://github.com//issues/362#issuecomment-2154452597) I mentioned that we have implemented the recommendations of @aronowski in the activation of the parameters CONFIG_MODULE_SIG_ALL and CONFIG_MODULE_SIG_FORCE.
I didn't know at that time that this change would lead to the creation of ephemeral keys. Under ephemeral key I understood one of the other implementations. That is why I mentioned that the use of ephemeral key is not the option for us. So, at this point, perhaps there was a misunderstanding in the communication.

Because some reviewers (#362 (comment)) have pointed out that you have to create a procedure to prevent old modules from being loaded, we have proposed an idea with double signing. Although, if the ephemeral key is already in use, this measure is completely unnecessary. The only point the reviewer @THS-on pointed out was the use of HSM when implementing automatic signing during kernel build. Because the argument that the signature key is physically accessible for a certain time during the kernel build, I think his argument is perfectly fine. After all, you want to have the highest possible security so that you can prevent other perhaps unwanted modules from being loaded. We have implemented HSM for the creation of ephemeral keys (https://github.com/Jurij-Ivastsuk/WAXAR-shim-review/blob/waxar-shim-x86_64-aarch64-20240804/Setup_for_module_signing.txt). So, now we hope that all security requirements are fulfilled by us and we would be happy that the review process can be decided positively for us at this point.

[dbnicholson]

@Jurij-Ivastsuk Thanks very much for that explanation. So, the purpose of the HSM within the kernel build is to generate the key and do the signing rather than doing it all with a key generated on disk from the OS random source? That is certainly more secure.

However, this part seems to undo all of that:

Step 2: Additionally signing all modules with our CA-Certificate using HSM
To be sure that modules from older kernels or unwanted modules cannot be loaded, we have used the strategy of double signing of all modules. On the one hand with the unique kernel certificate and on the other hand with our CA certificate. Taking into account that our CA certificate is located in the HSM, this gives us a high level of security in terms of preventing the possibility of infiltration of foreign and unknown modules into the boot process.

If you sign the modules with your CA certificate, then you lose the ability to prevent old modules from being loaded since all modules from every kernel build will be valid. Furthermore, the linux module signature verification process only validates a single signature. It does not support having multiple signatures on a module. Looking at the validation process, it simply copies a single signature of bytes from the end of the module for verification. So, it's only validating the last signature, which is your CA signature. The ephemeral key signature isn't being used at all.

I feel like you should just drop this step. You just said "Although, if the ephemeral key is already in use, this measure is completely unnecessary." The only thing this step is going to do is decrease security.

[Jurij-Ivastsuk]

Hi @dbnicholson,
Thank you for your comment.
I completely agree with you and will skip step 2. Because as you say, this will only reduce security instead of increasing it.
I will remove this step from our setup as soon as possible.

[Jurij-Ivastsuk]

Hi @THS-on, @aronowski and @dbnicholson,

according to our last discussions with @dbnicholson, the area concerning the double signing of the modules has been completely removed from our signing setup. Please have a look at the updated version of our setup here:
https://github.com/Jurij-Ivastsuk/WAXAR-shim-review/blob/waxar-shim-x86_64-aarch64-20240915/Setup_for_module_signing.txt