How to allocate unused disk space to Kubernetes storage

[tz-torchai]

Hi folks,

Based on the Storage section of the Talos documentation (link):

... because Talos Linux assumes complete control of the disk it is installed on, so that it can control the partition table for image based upgrades, you cannot partition the rest of the disk for use by workloads.

I have a few questions regarding this:

1. Understanding Talos disk usage

For example, if the "Disk" size is 512GiB, and i set the maxSize in EPHEMERAL VolumeConfig as 100GiB, then all the space Talos controls are "partitions" with label EPHEMERAL STATE META, which is ~100GiB+100MiB, is my understanding correct?

2. Partitioning unused space

I also heard that we can still create a partition on the unused space and allocate it to K8s Storage solution like Mayastor, which sounds contradictory to what docs says? How does this work in practice?

3. Missing partition in discovered volumes

When I set 30Gib for EPHEMERAL in VolumeConfig and applied:

machine:
  disks:
    - device: nvme0n1
      partitions:
        - mountpoint: /var/mnt/extra1
          size: 256 GiB

to all my nodes with the same storage hardware, each with 2 x 512G nvme SSD. Here is one of them:

❯  talosctl get discoveredvolumes  --nodes  talos-3ug-qaq
NODE            NAMESPACE   TYPE               ID          VERSION   TYPE        SIZE     DISCOVERED   LABEL       PARTITIONLABEL
talos-3ug-qaq   runtime     DiscoveredVolume   loop2       1         disk        74 MB    squashfs
talos-3ug-qaq   runtime     DiscoveredVolume   nvme0n1     1         disk        512 GB   gpt
talos-3ug-qaq   runtime     DiscoveredVolume   nvme0n1p1   1         partition   105 MB   vfat         EFI         EFI
talos-3ug-qaq   runtime     DiscoveredVolume   nvme0n1p2   1         partition   1.0 MB                            BIOS
talos-3ug-qaq   runtime     DiscoveredVolume   nvme0n1p3   1         partition   1.0 GB   xfs          BOOT        BOOT
talos-3ug-qaq   runtime     DiscoveredVolume   nvme0n1p4   1         partition   1.0 MB                            META
talos-3ug-qaq   runtime     DiscoveredVolume   nvme0n1p5   1         partition   105 MB   xfs          STATE       STATE
talos-3ug-qaq   runtime     DiscoveredVolume   nvme0n1p6   1         partition   30 GB    xfs          EPHEMERAL   EPHEMERAL
talos-3ug-qaq   runtime     DiscoveredVolume   nvme1n1     1         disk        512 GB

Why I don't see the 256Gib partition? I know EPHEMERAL can't be modified without a full node wipe. Is this issue related to that?
Simply put, if I add, modify, or delete a partition, I need to wipe the node and reinstall Talos for that node.

4. Allocating unused disk space to Kubernetes storage

If I am using K8s storage(not hostPath or local path etc.) like Mayastor, do I still need to partition and set the mountpoint in machine.disks?

5. Best approach for partition of file system and Kubernetes storage

I know that if I want to read/write directly to the filesystem, I must create partitions and set mount points. Based on this:
If the answer to 4 is no, I imagine a good approach is to
a. Create a good size of partition and set the mountpoint for deployments who want read / write directly to the filesystem or hostPath
b. Assign all the remaining partitions or the whole disk (unpartitioned) to K8s storage like Mayastor

Sorry for a lot of questions, thank you in advance!

[smira]

For example, if the "Disk" size is 512GiB, and i set the maxSize in EPHEMERAL VolumeConfig as 100GiB, then all the space Talos controls are "partitions" with label EPHEMERAL STATE META, which is ~100GiB+100MiB, is my understanding correct?

yes

I also heard that we can still create a partition on the unused space and allocate it to K8s Storage solution like Mayastor, which sounds contradictory to what docs says? How does this work in practice?

The full support for user volumes hasn't come yet (it is expected in Talos 1.10).

Some people partition the rest of the disk manually/using other means besides Talos.

3. Missing partition in discovered volumes

This section is a question about legacy way which doesn't work quite well. Short version - you need to pick an empty disk if you want to use it right now (nvme1n1).

If I am using K8s storage(not hostPath or local path etc.) like Mayastor, do I still need to partition and set the mountpoint in machine.disks?

It depends on what you're looking for, you can perfect use EPHEMERAL (/var) for it as well, or you might allocate a user volume.

Some storages (e.g. Ceph) can allocate unused disks on their own.

5. Best approach for partition of file system and Kubernetes storage

Yes, in general that is correct, but details depend on specifics.