Post v2.21.0 release updates to main (#3670)

* docs: release 2.21

* chore: update version.txt to v2.22.0-pre

* chore: update CI for v2.21.0

---------

Co-authored-by: edgelessci <edgelessci@users.noreply.github.com>

Author: edgelessci

.github/workflows/e2e-test-release.yml

@@ -409,7 +409,7 @@ jobs:
       fail-fast: false
       max-parallel: 1
       matrix:
-        fromVersion: ["v2.20.1"]
+        fromVersion: ["v2.21.0"]
         attestationVariant: ["gcp-sev-snp", "azure-sev-snp", "azure-tdx", "aws-sev-snp"]
     name: Run upgrade tests
     secrets: inherit

.github/workflows/e2e-test-weekly.yml

@@ -420,7 +420,7 @@ jobs:
       fail-fast: false
       max-parallel: 1
       matrix:
-        fromVersion: ["v2.20.1"]
+        fromVersion: ["v2.21.0"]
         attestationVariant: ["gcp-sev-snp", "azure-sev-snp", "azure-tdx", "aws-sev-snp"]
     name: Run upgrade tests
     secrets: inherit

docs/versioned_docs/version-2.21/_media/SLSA-Badge-full-level3.svg

@@ -0,0 +1,62 @@
+# Encrypted persistent storage
+
+Confidential VMs provide runtime memory encryption to protect data in use.
+In the context of Kubernetes, this is sufficient for the confidentiality and integrity of stateless services.
+Consider a front-end web server, for example, that keeps all connection information cached in main memory.
+No sensitive data is ever written to an insecure medium.
+However, many real-world applications need some form of state or data-lake service that's connected to a persistent storage device and requires encryption at rest.
+As described in [Use persistent storage](../workflows/storage.md), cloud service providers (CSPs) use the container storage interface (CSI) to make their storage solutions available to Kubernetes workloads.
+These CSI storage solutions often support some sort of encryption.
+For example, Google Cloud [encrypts data at rest by default](https://cloud.google.com/security/encryption/default-encryption), without any action required by the customer.
+
+## Cloud provider-managed encryption
+
+CSP-managed storage solutions encrypt the data in the cloud backend before writing it physically to disk.
+In the context of confidential computing and Constellation, the CSP and its managed services aren't trusted.
+Hence, cloud provider-managed encryption protects your data from offline hardware access to physical storage devices.
+It doesn't protect it from anyone with infrastructure-level access to the storage backend or a malicious insider in the cloud platform.
+Even with "bring your own key" or similar concepts, the CSP performs the encryption process with access to the keys and plaintext data.
+
+In the security model of Constellation, securing persistent storage and thereby data at rest requires that all cryptographic operations are performed inside a trusted execution environment.
+Consequently, using CSP-managed encryption of persistent storage usually isn't an option.
+
+## Constellation-managed encryption
+
+Constellation provides CSI drivers for storage solutions in all major clouds with built-in encryption support.
+Block storage provisioned by the CSP is [mapped](https://guix.gnu.org/manual/en/html_node/Mapped-Devices.html) using the [dm-crypt](https://www.kernel.org/doc/html/latest/admin-guide/device-mapper/dm-crypt.html), and optionally the [dm-integrity](https://www.kernel.org/doc/html/latest/admin-guide/device-mapper/dm-integrity.html), kernel modules, before it's formatted and accessed by the Kubernetes workloads.
+All cryptographic operations happen inside the trusted environment of the confidential Constellation node.
+
+Note that for integrity-protected disks, [volume expansion](https://kubernetes.io/blog/2018/07/12/resizing-persistent-volumes-using-kubernetes/) isn't supported.
+
+By default the driver uses data encryption keys (DEKs) issued by the Constellation [*KeyService*](microservices.md#keyservice).
+The DEKs are in turn derived from the Constellation's key encryption key (KEK), which is directly derived from the [master secret](keys.md#master-secret).
+This is the recommended mode of operation, and also requires the least amount of setup by the cluster administrator.
+
+Alternatively, the driver can be configured to use a key management system to store and access KEKs and DEKs.
+
+Refer to [keys and cryptography](keys.md) for more details on key management in Constellation.
+
+Once deployed and configured, the CSI driver ensures transparent encryption and integrity of all persistent volumes provisioned via its storage class.
+Data at rest is secured without any additional actions required by the developer.
+
+## Cryptographic algorithms
+
+This section gives an overview of the libraries, cryptographic algorithms, and their configurations, used in Constellation's CSI drivers.
+
+### dm-crypt
+
+To interact with the dm-crypt kernel module, Constellation uses [libcryptsetup](https://gitlab.com/cryptsetup/cryptsetup/).
+New devices are formatted as [LUKS2](https://gitlab.com/cryptsetup/LUKS2-docs/-/tree/master) partitions with a sector size of 4096 bytes.
+The used key derivation function is [Argon2id](https://datatracker.ietf.org/doc/html/rfc9106) with the [recommended parameters for memory-constrained environments](https://datatracker.ietf.org/doc/html/rfc9106#section-7.4) of 3 iterations and 64 MiB of memory, utilizing 4 parallel threads.
+For encryption Constellation uses AES in XTS-Plain64. The key size is 512 bit.
+
+### dm-integrity
+
+To interact with the dm-integrity kernel module, Constellation uses [libcryptsetup](https://gitlab.com/cryptsetup/cryptsetup/).
+When enabled, the used data integrity algorithm is [HMAC](https://datatracker.ietf.org/doc/html/rfc2104) with SHA256 as the hash function.
+The tag size is 32 Bytes.
+
+## Encrypted S3 object storage
+
+Constellation comes with a service that you can use to transparently retrofit client-side encryption to existing applications that use S3 (AWS or compatible) for storage.
+To learn more, check out the [s3proxy documentation](../workflows/s3proxy.md).

docs/versioned_docs/version-2.21/_media/benchmark_fio_azure_bw.png

@@ -0,0 +1,49 @@
+# Constellation images
+
+Constellation uses a minimal version of Fedora as the operating system running inside confidential VMs. This Linux distribution is optimized for containers and designed to be stateless.
+The Constellation images provide measured boot and an immutable filesystem.
+
+## Measured boot
+
+```mermaid
+flowchart LR
+  Firmware   --> Bootloader
+  Bootloader --> uki
+  subgraph uki[Unified Kernel Image]
+  Kernel[Kernel]
+  initramfs[Initramfs]
+  cmdline[Kernel Command Line]
+  end
+  uki --> rootfs[Root Filesystem]
+```
+
+Measured boot uses a Trusted Platform Module (TPM) to measure every part of the boot process. This allows for verification of the integrity of a running system at any point in time. To ensure correct measurements of every stage, each stage is responsible to measure the next stage before transitioning.
+
+### Firmware
+
+With confidential VMs, the firmware is the root of trust and is measured automatically at boot. After initialization, the firmware will load and measure the bootloader before executing it.
+
+### Bootloader
+
+The bootloader is the first modifiable part of the boot chain. The bootloader is tasked with loading the kernel, initramfs and setting the kernel command line. The Constellation bootloader measures these components before starting the kernel.
+
+### initramfs
+
+The initramfs is a small filesystem loaded to prepare the actual root filesystem. The Constellation initramfs maps the block device containing the root filesystem with [dm-verity](https://www.kernel.org/doc/html/latest/admin-guide/device-mapper/verity.html). The initramfs then mounts the root filesystem from the mapped block device.
+
+dm-verity provides integrity checking using a cryptographic hash tree. When a block is read, its integrity is checked by verifying the tree against a trusted root hash. The initramfs reads this root hash from the previously measured kernel command line. Thus, if any block of the root filesystem's device is modified on disk, trying to read the modified block will result in a kernel panic at runtime.
+
+After mounting the root filesystem, the initramfs will switch over and start the `init` process of the integrity-protected root filesystem.
+
+## State disk
+
+In addition to the read-only root filesystem, each Constellation node has a disk for storing state data.
+This disk is mounted readable and writable by the initramfs and contains data that should persist across reboots.
+Such data can contain sensitive information and, therefore, must be stored securely.
+To that end, the state disk is protected by authenticated encryption.
+See the section on [keys and encryption](keys.md#storage-encryption) for more information on the cryptographic primitives in use.
+
+## Kubernetes components
+
+During initialization, the [*Bootstrapper*](microservices.md#bootstrapper) downloads and verifies the [Kubernetes components](https://kubernetes.io/docs/concepts/overview/components/) as configured by the user.
+They're stored on the state partition and can be updated once new releases need to be installed.

docs/versioned_docs/version-2.21/_media/benchmark_fio_azure_iops.png

@@ -0,0 +1,130 @@
+# Key management and cryptographic primitives
+
+Constellation protects and isolates your cluster and workloads.
+To that end, cryptography is the foundation that ensures the confidentiality and integrity of all components.
+Evaluating the security and compliance of Constellation requires a precise understanding of the cryptographic primitives and keys used.
+The following gives an overview of the architecture and explains the technical details.
+
+## Confidential VMs
+
+Confidential VM (CVM) technology comes with hardware and software components for memory encryption, isolation, and remote attestation.
+For details on the implementations and cryptographic soundness, refer to the hardware vendors' documentation and advisories.
+
+## Master secret
+
+The master secret is the cryptographic material used for deriving the [*clusterID*](#cluster-identity) and the *key encryption key (KEK)* for [storage encryption](#storage-encryption).
+It's generated during the bootstrapping of a Constellation cluster.
+It can either be managed by [Constellation](#constellation-managed-key-management) or an [external key management system](#user-managed-key-management).
+In case of [recovery](#recovery-and-migration), the master secret allows to decrypt the state and recover a Constellation cluster.
+
+## Cluster identity
+
+The identity of a Constellation cluster is represented by cryptographic [measurements](attestation.md#runtime-measurements):
+
+The **base measurements** represent the identity of a valid, uninitialized Constellation node.
+They depend on the node image, but are otherwise the same for every Constellation cluster.
+On node boot, they're determined using the CVM's attestation mechanism and [measured boot up to the read-only root filesystem](images.md).
+
+The **clusterID** represents the identity of a single initialized Constellation cluster.
+It's derived from the master secret and a cryptographically random salt and unique for every Constellation cluster.
+The [Bootstrapper](microservices.md#bootstrapper) measures the *clusterID* into its own PCR before executing any code not measured as part of the *base measurements*.
+See [Node attestation](attestation.md#node-attestation) for details.
+
+The remote attestation statement of a Constellation cluster combines the *base measurements* and the *clusterID* for a verifiable, unspoofable, unique identity.
+
+## Network encryption
+
+Constellation encrypts all cluster network communication using the [container network interface (CNI)](https://github.com/containernetworking/cni).
+See [network encryption](networking.md) for more details.
+
+The Cilium agent running on each node establishes a secure [WireGuard](https://www.wireguard.com/) tunnel between it and all other known nodes in the cluster.
+Each node creates its own [Curve25519](http://cr.yp.to/ecdh.html) encryption key pair and distributes its public key via Kubernetes.
+A node uses another node's public key to decrypt and encrypt traffic from and to Cilium-managed endpoints running on that node.
+Connections are always encrypted peer-to-peer using [ChaCha20](http://cr.yp.to/chacha.html) with [Poly1305](http://cr.yp.to/mac.html).
+WireGuard implements [forward secrecy with key rotation every 2 minutes](https://lists.zx2c4.com/pipermail/wireguard/2017-December/002141.html).
+
+## Storage encryption
+
+Constellation supports transparent encryption of persistent storage.
+The Linux kernel's device mapper-based encryption features are used to encrypt the data on the block storage level.
+Currently, the following primitives are used for block storage encryption:
+
+* [dm-crypt](https://www.kernel.org/doc/html/latest/admin-guide/device-mapper/dm-crypt.html)
+* [dm-integrity](https://www.kernel.org/doc/html/latest/admin-guide/device-mapper/dm-integrity.html)
+
+Adding primitives for integrity protection in the CVM attacker model are under active development and will be available in a future version of Constellation.
+See [encrypted storage](encrypted-storage.md) for more details.
+
+As a cluster administrator, when creating a cluster, you can use the Constellation [installation program](orchestration.md) to select one of the following methods for key management:
+
+* Constellation-managed key management
+* User-managed key management
+
+### Constellation-managed key management
+
+#### Key material and key derivation
+
+During the creation of a Constellation cluster, the cluster's master secret is used to derive a KEK.
+This means creating two clusters with the same master secret will yield the same KEK.
+Any data encryption key (DEK) is derived from the KEK via HKDF.
+Note that the master secret is recommended to be unique for every cluster and shouldn't be reused (except in case of [recovering](../workflows/recovery.md) a cluster).
+
+#### State and storage
+
+The KEK is derived from the master secret during the initialization.
+Subsequently, all other key material is derived from the KEK.
+Given the same KEK, any DEK can be derived deterministically from a given identifier.
+Hence, there is no need to store DEKs. They can be derived on demand.
+After the KEK was derived, it's stored in memory only and never leaves the CVM context.
+
+#### Availability
+
+Constellation-managed key management has the same availability as the underlying Kubernetes cluster.
+Therefore, the KEK is stored in the [distributed Kubernetes etcd storage](https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/) to allow for unexpected but non-fatal (control-plane) node failure.
+The etcd storage is backed by the encrypted and integrity protected [state disk](images.md#state-disk) of the nodes.
+
+#### Recovery
+
+Constellation clusters can be recovered in the event of a disaster, even when all node machines have been stopped and need to be rebooted.
+For details on the process see the [recovery workflow](../workflows/recovery.md).
+
+### User-managed key management
+
+User-managed key management is under active development and will be available soon.
+In scenarios where constellation-managed key management isn't an option, this mode allows you to keep full control of your keys.
+For example, compliance requirements may force you to keep your KEKs in an on-prem key management system (KMS).
+
+During the creation of a Constellation cluster, you specify a KEK present in a remote KMS.
+This follows the common scheme of "bring your own key" (BYOK).
+Constellation will support several KMSs for managing the storage and access of your KEK.
+Initially, it will support the following KMSs:
+
+* [AWS KMS](https://aws.amazon.com/kms/)
+* [GCP KMS](https://cloud.google.com/security-key-management)
+* [Azure Key Vault](https://azure.microsoft.com/en-us/services/key-vault/#product-overview)
+* [KMIP-compatible KMS](https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=kmip)
+
+Storing the keys in Cloud KMS of AWS, Azure, or GCP binds the key usage to the particular cloud identity access management (IAM).
+In the future, Constellation will support remote attestation-based access policies for Cloud KMS once available.
+Note that using a Cloud KMS limits the isolation and protection to the guarantees of the particular offering.
+
+KMIP support allows you to use your KMIP-compatible on-prem KMS and keep full control over your keys.
+This follows the common scheme of "hold your own key" (HYOK).
+
+The KEK is used to encrypt per-data "data encryption keys" (DEKs).
+DEKs are generated to encrypt your data before storing it on persistent storage.
+After being encrypted by the KEK, the DEKs are stored on dedicated cloud storage for persistence.
+Currently, Constellation supports the following cloud storage options:
+
+* [AWS S3](https://aws.amazon.com/s3/)
+* [GCP Cloud Storage](https://cloud.google.com/storage)
+* [Azure Blob Storage](https://azure.microsoft.com/en-us/services/storage/blobs/#overview)
+
+The DEKs are only present in plaintext form in the encrypted main memory of the CVMs.
+Similarly, the cryptographic operations for encrypting data before writing it to persistent storage are performed in the context of the CVMs.
+
+#### Recovery and migration
+
+In the case of a disaster, the KEK can be used to decrypt the DEKs locally and subsequently use them to decrypt and retrieve the data.
+In case of migration, configuring the same KEK will provide seamless migration of data.
+Thus, only the DEK storage needs to be transferred to the new cluster alongside the encrypted data for seamless migration.