v1.4.0

KubeOne v1.4.0

Today, we are pleased to announce that KubeOne 1.4 is now generally available. With this release, we introduce our new KubeOneCluster API version with many new features that simplify configuration management. Additionally, we have added support for Kubernetes 1.23 and Cilium CNI and facilitated CCM/CSI migration, among other features. KubeOne 1.4 also provides alpha-level support for Nutanix.

Major Highlights

We recommend checking out the Upgrading from KubeOne 1.3 to 1.4 tutorial, as well as, the changelog for more information about upgrading and the latest features and improvements.

Attention Needed

• KubeOne 1.4.0-beta.0 introduces the new KubeOneCluster v1beta2 API
  • The existing KubeOneCluster v1beta1 manifests can be migrated by using the kubeone config migrate command
  • The kubeone config print command now uses the new v1beta2 API
  • The existing KubeOneCluster v1beta1 API is considered deprecated and will be removed in KubeOne 1.6+
  • Highlights:
    • The API group has been changed from kubeone.io to kubeone.k8c.io
    • The AssetConfiguration API has been removed from the v1beta2 API. The AssetConfiguration API can still be used with the v1beta1 API, but we highly recommend migrating away because the v1beta1 API is deprecated
    • The PodPresets feature has been removed from the v1beta2 API because Kubernetes removed support for PodPresets in Kubernetes 1.20
    • Packet (packet) cloud provider has been rebranded to Equinix Metal (equinixmetal). The existing Packet cluster will work with equinixmetal cloud provider, however, manual migration steps are required if you want to use new Terraform configs for Equinix Metal
    • A new ContainerRuntime API has been added to the v1beta2 API in order to support configuring mirror registries
• kubeone install and kubeone upgrade commands are considered deprecated in favor of kubeone apply
  • install and upgrade commands will be removed in KubeOne 1.6+
  • We highly encourage switching to kubeone apply. The apply command has the same semantics and works in the same way as install/upgrade, with some additional checks to ensure each requested operation is safe for the cluster
• Unconditionally deploy AWS, AzureDisk, AzureFile, and vSphere CSI drivers if the Kubernetes version is 1.23 or newer (#1831)
  • Those providers have the CSI migration enabled by default in Kubernetes 1.23, so the CSI driver will be used for all volumes operations
• Unconditionally deploy DigitalOcean, Hetzner, Nutanix, and OpenStack Cinder CSI drivers (#1831)
  • OpenStack has the CSI migration enabled by default since Kubernetes 1.18, so the CSI driver will be used for all operations
• CentOS 8 has reached End-Of-Life (EOL) on January 31st, 2022. It will no longer receive any updates (including security updates). Support for CentOS 8 in KubeOne is deprecated and will be removed in a future release. We strongly recommend migrating to another operating system or RHEL/CentOS distribution as soon as possible.

Breaking changes / Action Required

• The default AMI for CentOS in Terraform configs for AWS has been changed to Rocky Linux. If you use the new Terraform configs with an existing cluster, make sure to bind the AMI as described in the production recommendations document (#1809)
• The cloud-provider-credentials Secret is removed by KubeOne because KubeOne does not use it any longer. If you have any workloads NOT created by KubeOne that use this Secret, please migrate before upgrading KubeOne. Instead, KubeOne now creates kubeone-machine-controller-credentials and kubeone-ccm-credentials Secrets used by machine-controller and external CCM
• Support for Amazon EKS-D clusters has been removed starting from this release
• GCP: Default operating system for control plane instances is now Ubuntu 20.04 (#1576)
  • Make sure to bind control_plane_image_family to the image you're currently using or Terraform might recreate all your control plane instances
• Azure: Default VM type is changed to Standard_F2 (#1528)
  • Make sure to bind control_plane_vm_size and worker_vm_size to the VM size you're currently using or Terraform might recreate all your instances

Known Issues

• It's not possible to run kube-proxy in IPVS mode on Kubernetes 1.23 clusters using Canal/Calico CNI. Trying to upgrade existing 1.22 clusters using IPVS to 1.23 will result in a validation error from KubeOne

Checksums

SHA256 checksums can be found in the kubeone_1.4.0_checksums.txt file.

v1.4.0-rc.1

Attention Needed

• Unconditionally deploy AWS, AzureDisk, AzureFile, and vSphere CSI drivers if the Kubernetes version is 1.23 or newer (#1831)
  • Those providers have the CSI migration enabled by default in Kubernetes 1.23, so the CSI driver will be used for all volumes operations
• Unconditionally deploy DigitalOcean, Hetzner, Nutanix, and OpenStack Cinder CSI drivers (#1831)
  • OpenStack has the CSI migration enabled by default since Kubernetes 1.18, so the CSI driver will be used for all operations
• [BREAKING] The default AMI for CentOS in Terraform configs for AWS has been changed to Rocky Linux. If you use the new Terraform configs with an existing cluster, make sure to bind the AMI as described in the production recommendations document (#1809)

Added

• Include darwin/arm64 and linux/arm64 builds in release artifacts (#1821)
• Allow providing operating system via the API (#1809)

Changed

General

• Increase the minimum Kubernetes version to 1.20 (#1818)
• Validate the Kubernetes version against supported versions constraints (#1808)
• Allow Docker as a container runtime up to Kubernetes v1.24 (previously up to v1.22) (#1826)
• Unconditionally deploy AWS, AzureDisk, AzureFile, and vSphere CSI drivers if the Kubernetes version is 1.23 or newer (#1831)
  • Those providers have the CSI migration enabled by default in Kubernetes 1.23, so the CSI driver will be used for all volumes operations
• Unconditionally deploy DigitalOcean, Hetzner, Nutanix, and OpenStack Cinder CSI drivers (#1831)
  • OpenStack has the CSI migration enabled by default since Kubernetes 1.18, so the CSI driver will be used for all operations

Fixed

• Restore missing addons deploy after containerd migration (#1824)
• Select correct CSR to approve (#1813)

Terraform Configs

• [BREAKING] The default AMI for CentOS in Terraform configs for AWS has been changed to Rocky Linux. If you use the new Terraform configs with an existing cluster, make sure to bind the AMI as described in the production recommendations document (#1809)
• Add the control_plane_vm_count variable to the AWS configs used to control the number of control plane nodes (defaults to 3) (#1810)
• Update the Terraform provider for OpenStack to version 1.47.0 (#1816)
• Set Ubuntu 20.04 as the default image for OpenStack (#1816)
• Add example Terraform configs for Flatcar on vSphere (#1838)

Updated

• Update DigitalOcean CSI to v4.0.0 (#1820)
• Update machine-controller to v1.43.0 (#1834)

Checksums

SHA256 checksums can be found in the kubeone_1.4.0-rc.1_checksums.txt file.

v1.4.0-rc.0

Attention Needed

• CentOS 8 has reached End-Of-Life (EOL) on January 31st, 2022. It will no longer receive any updates (including security updates). Support for CentOS 8 in KubeOne is deprecated and will be removed in a future release. We strongly recommend migrating to another operating system or CentOS distribution as soon as possible.

Added

• Add experimental/alpha-level support for Kubermatic Operating System Manager (OSM) (#1748)
• Add ability to change the container log maximum size (defaults to 100Mi) (#1644)
• Add ability to change the container log maximum files (defaults to 5) (#1759)
• Add the DigitalOcean CSI driver. The CSI driver is deployed automatically if .cloudProvider.external is enabled (#1754)
• Add the default StorageClass and VolumeSnapshotClass for the DigitalOcean CSI driver. The StorageClass and VolumeSnapshotClass can be deployed by enabling the default-storage-class embedded addon (#1754)
• Generate and approve CSRs for control plane and static workers nodes. Enable the server TLS bootstrap for control plane and static worker nodes (#1750, #1758)
• Source .cloudProvider.csiConfig from the credentials file if present (#1739)
• Fetch containerd auth config from the credentials file if present (#1745)

Changed

Fixed

• Change baseurl to vault.centos.org for CentOS 8 (#1767)
• Fix Docker to containerd migration on non-Flatcar operating systems (#1743)
• Fix propagation of proxy config to machines and Kubernetes components (#1746)

Addons

• Replace Hubble static certificate with CronJob generation (#1752)
• Make template function required available to addons manifest templates (#1737)
• Ensure unattended-upgrades in dpkg is active (#1756)

Terraform Configs

• Create a placement group for control plane nodes in Terraform configs for Hetzner (#1762)

Updated

• Update Canal CNI to v3.22.0 (#1797)
• Update Cilium to v1.11.1 (#1752)
• Update Calico VXLAN addon to v3.22.0 (#1797)
• Update images in order to support Kubernetes 1.23 (#1751, #1753)
  • Update AWS External Cloud Controller Manager (CCM) to v1.23.0-alpha.0 for Kubernetes 1.23 clusters
  • Update Azure External Cloud Controller Manager (CCM) to v1.23.2 for Kubernetes 1.23 clusters
  • Update AWS EBS CSI driver to v1.5.0
  • Update AzureFile CSI driver to v1.9.0
  • Update AzureDisk CSI driver to v1.10.0
  • Update OpenStack External Cloud Controller Manager (CCM) to v1.23.0 for Kubernetes 1.23 clusters
  • Update the DigitalOcean External Cloud Controller Manager (CCM) to v0.1.36
  • Update the Hetzner External Cloud Controller Manager (CCM) to v1.12.1
• Update machine-controller to v1.42.2 (#1748)

Checksums

SHA256 checksums can be found in the kubeone_1.4.0-rc.0_checksums.txt file.

v1.4.0-beta.1

Attention Needed

• [BREAKING] The cloud-provider-credentials Secret is removed by KubeOne because KubeOne does not use it any longer. If you have any workloads NOT created by KubeOne that use this Secret, please migrate before upgrading KubeOne. Instead, KubeOne now creates kubeone-machine-controller-credentials and kubeone-ccm-credentials Secrets used by machine-controller and external CCM (#1717, #1718)

Added

• Add experimental/alpha support for Nutanix (#1723, #1725, #1733)
  • Support for Nutanix is experimental, so implementation and relevant addons might be changed until it doesn't graduate to beta/stable
• Add the Nutanix CSI driver addon. The addon is deployed manually, on-demand, by enabling the csi-nutanix embedded addon (see the PR description for more details and examples) (#1733, #1734)
• Add the default StorageClass for the Nutanix CSI driver. The StorageClass can be deployed by enabling the default-storage-class embedded addon (see the PR description for more details and examples) (#1733)
• Add the Registry Credentials configuration to the RegistryConfiguration API (#1724)
• Add support for different credentials for machine-controller and CCM. Environment variables can be prefixed with MC_ for machine-controller credentials and CCM_ for CCM credentials (#1717)

Changed

General

• [BREAKING] The cloud-provider-credentials Secret is removed by KubeOne because KubeOne does not use it any longer. If you have any workloads NOT created by KubeOne that use this Secret, please migrate before upgrading KubeOne. Instead, KubeOne now creates kubeone-machine-controller-credentials and kubeone-ccm-credentials Secrets used by machine-controller and external CCM (#1717, #1718)

Fixed

• Fix a bug with the addons applier applying all files when the addons path is not provided (#1733)

Addons

• Fix control plane tolerations in Azure CCM and CSI addons (node-role.kubernetes.io/master doesn't have a value) (#1733)
• Add node affinity to the cluster-autoscaler addon (#1716)

Terraform Configs

• Remove centos choice from the GCE Terraform example configs as it's unsupported (#1712)

Updated

• Update machine-controller to v1.42.0 (#1733)

Checksums

SHA256 checksums can be found in the kubeone_1.4.0-beta.1_checksums.txt file.

v1.4.0-beta.0

Attention Needed

• KubeOne 1.4.0-beta.0 introduces the new KubeOneCluster v1beta2 API
  • The new v1beta2 API is still under-development and might be changed before the KubeOne 1.4.0 release
  • We recommend and highly encourage testing the new API, but considering that the API might be changed before the final release, we don't recommend migrating production clusters to the new API yet
  • The migration for existing KubeOneCluster manifests is not yet available
  • The kubeone config print command now uses the new v1beta2 API
  • The existing KubeOneCluster v1beta1 API is considered as deprecated and will be removed in KubeOne 1.6+
  • Highlights:
    • The API group has been changed from kubeone.io to kubeone.k8c.io
    • The AssetConfiguration API has been removed from the v1beta2 API. The AssetConfiguration API can still be used with the v1beta1 API, but we highly recommend migrating away because the v1beta1 API is deprecated
    • The PodPresets feature has been removed from the v1beta2 API because Kubernetes removed support for PodPresets in Kubernetes 1.20
    • Packet (packet) cloud provider has been rebranded to Equinix Metal (equinixmetal). The existing Packet cluster will work with equinixmetal cloud provider, however, manual migration steps are required if you want to use new Terraform configs for Equinix Metal
    • A new ContainerRuntime API has been added to the v1beta2 API in order to support configuring mirror registries. This API is still work-in-progress and will mostly like be extended before the final release
• kubeone install and kubeone upgrade commands are considered as deprecated in favor of kubeone apply
  • install and upgrade commands will be removed in KubeOne 1.6+
  • We highly encourage switching to kubeone apply. The apply command has the same semantics and works in the same way as install/upgrade, with some additional checks to ensure each requested operation is safe for the cluster
• Support for Amazon EKS-D clusters has been removed starting from this release

Known Issues

• It's not possible to run kube-proxy in IPVS mode on Kubernetes 1.23 clusters using Canal/Calico CNI. Trying to upgrade existing 1.22 clusters using IPVS to 1.23 will result in a validation error from KubeOne
  • More information about this issue can be found in the following Calico ticket: projectcalico/calico#5011

Added

API

• Add the KubeOneCluster v1beta2 API and change the API group to kubeone.k8c.io (#1649)
  • Make kubeone config print command use the new kubeone.k8c.io/v1beta2 API (#1651)
  • Add the new ContainerRuntime API with support for mirror registries (#1674)
  • Addons directory path (.addons.path) is not required when using only embedded addons (#1668)
  • Addons directory path (.addons.path) is not defaulted to ./addons any longer (#1668)
  • Add the KubeletConfig API used to configure systemReserved, kubeReserved, and evictionHard Kubelet options (#1698)
  • Remove the PodPresets feature (#1662)
  • Remove the AssetConfiguration API (#1699)
  • Rebrand Packet (packet) to Equinix Metal (equinixmetal) and support migrating existing Packet clusters to Equinix Metal
    clusters (#1663)

Features

• Add support for Kubernetes 1.23 (#1678)
• Add kubeone addons list command used to list available and enabled addons (#1642)
• Add support for OpenStack Application Credentials (#1666)
• Add a new --kubernetes-version flag to the kubeone config images command (#1671)
  • This flag is used to filter images for a particular Kubernetes version. The flag cannot be used along with the KubeOneCluster manifest (--manifest flag)
• Addon parameters can be resolved into environment variable contents if the env: prefix is set in the parameter value (#1691)

Changed

General

• Improve installation scripts used to install container runtime (#1664)

Fixed

• Fix issues when disabling nm-cloud-setup on RHEL (#1706)
• cri-tools is now installed automatically as a dependency of kubeadm on Amazon Linux 2. This fixes provisioning issues on Amazon Linux 2 with newer Kubernetes versions. (#1701)
• Fix the image loader script to support KubeOne 1.3+ and Kubernetes 1.22+ (#1671)
• The kubeone config images command now shows images for the latest Kubernetes version (instead of for the oldest) (#1671)
• Allow pods with the seccomp profile defined to get scheduled if the PodSecurityPolicy (PSP) feature is enabled (#1686)

Addons

• Update the cluster-autoscaler addon to match the upstream manifest (#1713)

Terraform Configs

• Automatically determine GCE zone for the initial MachineDeployment (#1703)
• Fix AMI filter in Terraform configs for AWS to always use x86_64 images (#1692)

Updated

• Update Cilium CNI addon to v1.11.0 (#1681)
• Update vSphere CSI driver addon to v2.4.0. This change introduces Kubernetes 1.22 support for vSphere clusters (#1675)
• Update Go to 1.17.5 (#1689)

Removed

• Remove support for Amazon EKS-D clusters (#1699)

Checksums

SHA256 checksums can be found in the kubeone_1.4.0-beta.0_checksums.txt file.

v1.3.3

Changed

Fixed

• Allow pods with the seccomp profile defined to get scheduled if the PodSecurityPolicy (PSP) feature is enabled (#1687)
• Fix the image loader script to support KubeOne 1.3+ and Kubernetes 1.22+ (#1672)
• The kubeone config images command now shows images for the latest Kubernetes version (instead of for the oldest) (#1672)
• Add a new --kubernetes-version flag to the kubeone config images command (#1672)
  • This flag is used to filter images for a particular Kubernetes version. The flag cannot be used along with the KubeOneCluster manifest (--manifest flag)

Addons

• Deploy default StorageClass for GCP clusters if the default-storage-class addon is enabled (#1639)

Updated

• Update machine-controller to v1.37.2 (#1654)
  • machine-controller is now using Ubuntu 20.04 instead of 18.04 by default for all newly-created Machines on AWS, Azure, DO, GCE, Hetzner, Openstack, and Equinix Metal
  • This release defaults the provisioning utility for Flatcar machines on AWS to cloud-init (previously ignition). Ignition is currently not working on AWS because of the user data limit
  • If you have the provisioning utility explicitly set to Ignition, you'll not be able to provision new Flatcar machines on AWS. In that case, manually changing the provisioning utility to cloud-init is required

Checksums

SHA256 checksums can be found in the kubeone_1.3.3_checksums.txt file.

v1.4.0-alpha.0

Attention Needed

• [BREAKING] GCP: Default operating system for control plane instances is now Ubuntu 20.04 (#1576)
  • Make sure to bind control_plane_image_family to the image you're currently using or Terraform might recreate all your control plane instances
• [BREAKING] Azure: Default VM type is changed to Standard_F2 (#1528)
  • Make sure to bind control_plane_vm_size and worker_vm_size to the VM size you're currently using or Terraform might recreate all your instances

Added

Features

• Add CCM/CSI migration support for clusters with the static worker nodes (#1544)
• Add CCM/CSI migration support for the Azure clusters (#1610)
• Automatically create cloud-config Secret for all providers if external cloud controller manager (.cloudProvider.external) is enabled (#1575)
• Add support for Cilium CNI (#1560, #1629)
• Add support for additional Subject Alternative Names (SANs) for the Kubernetes API server (#1599, #1603, #1606)
• Add a new MachineAnnotations field in the API used to define annotations in MachineDeployment.Spec.Template.Spec.Annotations (#1601)
• Add a new --create-machine-deployments flag to the kubeone apply command used to control should KubeOne create initial MachineDeployment objects when provisioning the cluster (default is true) (#1617)

Addons

• Integrate the AWS CCM addon with KubeOne (#1585)
  • The AWS CCM is now deployed if the external cloud provider (.cloudProvider.external) is enabled
  • This option cannot be enabled for existing AWS clusters running in-tree cloud provider, instead, those clusters must go through the CCM/CSI migration process
• Add the AWS EBS CSI driver addon (#1597)
  • Automatically deploy the AWS EBS CSI driver addon if external cloud controller manager (.cloudProvider.external) is enabled
  • Add default StorageClass for AWS EBS CSI driver to the default-storage-class embedded addon
• Integrate the Azure CCM addon with KubeOne (#1561, #1579)
  • The Azure CCM is now deployed if the external cloud provider (.cloudProvider.external) is enabled
  • This option cannot be enabled for existing Azure clusters running in-tree cloud provider, instead, those clusters must go through the CCM/CSI migration process
• Add the AzureFile CSI driver addon (#1575, #1579)
  • Automatically deploy the AzureFile CSI driver addon if external cloud controller manager (.cloudProvider.external) is enabled
  • Add default StorageClass for AzureFile CSI driver to the default-storage-class embedded addon
• Add the AzureDisk CSI driver addon (#1577)
  • Automatically deploy the AzureDisk CSI driver addon if external cloud controller manager (.cloudProvider.external) is enabled
  • Add default StorageClass for AzureDisk CSI driver to the default-storage-class embedded addon

Other

• Add a deprecation warning for PodSecurityPolicies (#1595)

Changed

General

• Validate the cluster name to ensure it's a correct DNS subdomain (RFC 1123) (#1641, #1646, #1648)
• Create MachineDeployments only for newly-provisioned clusters (#1627)
• Show warning about LBs on CCM migration for OpenStack clusters (#1627)
• Change default Kubernetes version in the example configuration to v1.22.3 (#1605)

Fixed

• Force drain nodes to remove standalone pods (#1627)
• Check for minor version when choosing kubeadm API version (#1627)
• Provide --cluster-name flag to the OpenStack external CCM (read PR description for more details) (#1619)
• Enable ip_tables related kernel modules and disable nm-cloud-setup tool on AWS for RHEL machines (#1607)
• Properly pass machine-controllers args (#1594)
  • This fixes the issue causing machine-controller and machine-controller-webhook deployments to run with incorrect flags
  • If you created your cluster with KubeOne 1.2 or older, and already upgraded to KubeOne 1.3, we recommend running kubeone apply again with KubeOne 1.3.2 or newer to properly reconcile machine-controller deployments
• Fix yum versionlock delete containerd.io error (#1600)
• Ensure containerd/docker be upgraded automatically when running kubeone apply (#1589)
• Edit SELinux config file only if file exists (#1532)

Addons

• Add new "required" addons template function (#1618)
• Replace critical-pod annotation with priorityClassName (#1627)
• Default image in the cluster-autoscaler addon and allow the image to be overridden using addon parameters (#1552)
• Minor improvements to OpenStack CCM and CSI addons. OpenStack CSI controller can now be scheduled on control plane nodes (#1531)
• Deploy default StorageClass for GCP clusters if the default-storage-class addon is enabled (#1638)

Terraform Configs

• [BREAKING] GCP: Default operating system for control plane instances is now Ubuntu 20.04 (#1576)
  • Make sure to bind control_plane_image_family to the image you're currently using or Terraform might recreate all your control plane instances
• [BREAKING] Azure: Default VM type is changed to Standard_F2 (#1528)
  • Make sure to bind control_plane_vm_size and worker_vm_size to the VM size you're currently using or Terraform might recreate all your instances
• OpenStack: Open NodePorts by default (#1530)
• AWS: Open NodePorts by default (#1535)
• GCE: Open NodePorts by default (#1529)
• Hetzner: Create Firewall by default (#1533)
• Azure: Open NodePorts by default (#1528)
• Fix keepalived script in Terraform configs for vSphere to assume yes when updating repos (#1537)
• Add additional Availability Set used for worker nodes to Terraform configs for Azure (#1556)
  • Make sure to check the production recommendations for Azure clusters for more information about how this additional availability set is used

Updated

• Update machine-controller to v1.37.0 (#1647)
  • machine-controller is now using Ubuntu 20.04 instead of 18.04 by default for all newly-created Machines on AWS, Azure, DO, GCE, Hetzner, Openstack and Equinix Metal
• Update Hetzner Cloud Controller Manager to v1.12.0 (#1583)
• Update Go to 1.17.1 (#1534, #1541, #1542, #1545)

Removed

• Remove the PodPresets feature (#1593)
  • If you're still using this feature, make sure to migrate away before upgrading to this KubeOne release
• Remove Ansible examples (#1633)

Checksums

SHA256 checksums can be found in the kubeone_1.4.0-alpha.0_checksums.txt file.

v1.3.2

Changed

General

• Create MachineDeployments only for newly-provisioned clusters (#1628)
• Show warning about LBs on CCM migration for OpenStack clusters (#1628)

Fixed

• Force drain nodes to remove standalone pods (#1628)
• Check for minor version when choosing kubeadm API version (#1628)
• Provide --cluster-name flag to the OpenStack external CCM (read PR description for more details) (#1632)
• Enable ip_tables related kernel modules and disable nm-cloud-setup tool on AWS for RHEL machines (#1616)
• Properly pass machine-controllers args (#1596)
  • This fixes the issue causing machine-controller and machine-controller-webhook deployments to run with incorrect flags
  • If you created your cluster with KubeOne 1.2 or older, and already upgraded to KubeOne 1.3, we recommend running kubeone apply again to properly reconcile machine-controller deployments
• Edit SELinux config file only if file exists (#1592)
• Fix yum versionlock delete containerd.io error (#1602)
• Ensure containerd/docker be upgraded automatically when running kubeone apply (#1590)

Addons

• Add new "required" addons template function (#1624)
• Replace critical-pod annotation with priorityClassName (#1628)
• Update Hetzner Cloud Controller Manager to v1.12.0 (#1592)
• Default image in the cluster-autoscaler addon and allow the image to be overridden using addon parameters (#1553)
• Minor improvements to OpenStack CCM and CSI addons. OpenStack CSI controller can now be scheduled on control plane nodes (#1536)

Terraform Configs

• OpenStack: Open NodePorts by default (#1592)
• GCE: Open NodePorts by default (#1592)
• Azure: Open NodePorts by default (#1592)
• Azure: Default VM type is changed to Standard_F2 (#1592)
• Add additional Availability Set used for worker nodes to Terraform configs for Azure (#1562)
  • Make sure to check the production recommendations for Azure clusters for more information about how this additional availability set is used
• Fix keepalived script in Terraform configs for vSphere to assume yes when updating repos (#1538)

Removed

• Remove Ansible examples (#1634)

Checksums

SHA256 checksums can be found in the kubeone_1.3.2_checksums.txt file.

v1.3.0

KubeOne v1.3.0

Today, we are pleased to announce that KubeOne 1.3 is now generally available. The previous release paved a road for many new features and this time we are excited to present those features to you. KubeOne 1.3 brings a brand new Addons API, managed support for encryption providers, automated Docker to containerd migration, and more!

Major Highlights

We recommend checking out the Upgrading from KubeOne 1.2 to 1.3 tutorial, as well as, the changelog for more information about upgrading and the latest features and improvements.

Attention Needed

Breaking changes / Action Required

• Increase the minimum Kubernetes version to v1.19.0. If you have Kubernetes clusters running v1.18 or older, you need to use an older KubeOne release to upgrade them to v1.19, and then upgrade to KubeOne 1.3.
• Increase the minimum Terraform version to 1.0.0.
• Remove support for Debian and RHEL 7 clusters. If you have Debian clusters, we recommend migrating to another operating system, for example Ubuntu. If you have RHEL 7 clusters, you should consider migrating to RHEL 8 which is supported.
• Automatically deploy CSI plugins for Hetzner, OpenStack, and vSphere clusters using external cloud provider. If you already have the CSI plugin deployed, you need to make sure that your CSI plugin deployment is compatible with the KubeOne CSI plugin addon.
• The kubeone reset command requires an explicit confirmation like the apply command starting with this release. The command can be automatically approved by using the --auto-approve flag.

Deprecations

• KubeOne Addons can now be organized into subdirectories. It currently remains possible to put addons in the root of the addons directory, however, this is option is considered as deprecated as of this release. We highly recommend all users to reorganize their addons into subdirectories, where each subdirectory is for YAML manifests related to one addon.
• We're deprecating support for CentOS 8 because it's reaching End-of-Life (EOL) on December 31, 2021. CentOS 7 remains supported by KubeOne for now.

Checksums

SHA256 checksums can be found in the kubeone_1.3.0_checksums.txt file.

v1.3.0-rc.0

Attention Needed

• [BREAKING/ACTION REQUIRED] Increase the minimum Kubernetes version to v1.19.0 (#1466)
  • If you have Kubernetes clusters running v1.18 or older, you need to use an older KubeOne release to upgrade them to v1.19, and then upgrade to KubeOne 1.3.
  • Check out the Compatibility guide for more information about supported Kubernetes versions for each KubeOne release.
• [BREAKING/ACTION REQUIRED] Add support for CSI plugins for clusters using external cloud provider (i.e. .cloudProvider.external is enabled)
  • The Cinder CSI plugin is deployed by default for OpenStack clusters (#1465)
  • The Hetzner CSI plugin is deployed by default for Hetzner clusters
  • The vSphere CSI plugin is deployed by default if the CSI plugin configuration is provided via newly-added cloudProvider.csiConfig field
    • More information about the CSI plugin configuration can be found in the vSphere CSI docs: https://vsphere-csi-driver.sigs.k8s.io/driver-deployment/installation.html#create_csi_vsphereconf
    • Note: the vSphere CSI plugin requires vSphere version 6.7U3.
  • The default StorageClass is not deployed by default. It can be deployed via new Addons API by enabling the default-storage-class addon, or manually.
  • ACTION REQUIRED: If you already have the CSI plugin deployed, you need to make sure that your CSI plugin deployment is compatible with the KubeOne CSI plugin addon.
    • You can find the CSI addons in the addons directory: https://github.com/kubermatic/kubeone/tree/master/addons
    • If your CSI plugin deployment is incompatible with the KubeOne CSI addon, you can resolve it in one of the following ways:
      • Delete your CSI deployment and let KubeOne install the CSI driver for you. Note: you'll not be able to mount volumes until you don't install the CSI driver again.
      • Override the appropriate CSI addon with your deployment manifest. With this way, KubeOne will install the CSI plugin using your manifests. To do this, you need to:
        • Enable addons in the KubeOneCluster manifest (.addons.enable) and provide the path to addons directory (.addons.path, for example: ./addons)
        • Create a subdirectory in the addons directory named same as the CSI addon used by KubeOne, for example ./addons/csi-openstack-cinder or ./addons/csi-vsphere (see https://github.com/kubermatic/kubeone/tree/master/addons for addon names)
        • Put your CSI deployment manifests in the newly created subdirectory

Known Issues

• It's currently not possible to provision or upgrade to Kubernetes 1.22 for clusters running on vSphere. This is because vSphere CCM and CSI don't support Kubernetes 1.22. We'll introduce Kubernetes 1.22 support for vSphere as soon as new CCM and CSI releases with support for Kubernetes 1.22 are out.
• Clusters provisioned with Kubernetes 1.22 or upgraded from 1.21 to 1.22 using KubeOne 1.3.0-alpha.1 use a metrics-server version incompatible with Kubernetes 1.22. This might cause issues with deleting Namespaces that manifests by the Namespace being stuck in the Terminating state. This can be fixed by upgrading the metrics-server by running kubeone apply.
• The new Addons API requires the addons directory path (.addons.path) to be provided and the directory must exist (it can be empty), even if only embedded addons are used. If the path is not provided, it'll default to ./addons.

Added

Features

• Implement the Addons API used to manage addons deployed by KubeOne (#1462, #1486)
  • The new Addons API can be used to deploy the addons embedded in the KubeOne binary.
  • Currently available addons are: backups-restic, default-storage-class, and unattended-upgrades.
  • More information about the new API can be found by running kubeone config print --full.
• [BREAKING/ACTION REQUIRED] Add support for the Cinder CSI plugin (#1465)
  • The plugin is deployed by default for OpenStack clusters using the external cloud provider.
  • Check out the Attention Needed section of the changelog for more information.
• Add support for the vSphere CSI plugin (#1484)
  • Deploying the CSI plugin requires providing the CSI configuration using a newly added .cloudProvider.csiConfig field
    • More information about the CSI plugin configuration can be found in the vSphere CSI docs: https://vsphere-csi-driver.sigs.k8s.io/driver-deployment/installation.html#create_csi_vsphereconf
  • The CSI plugin is deployed automatically if .cloudProvider.csiConfig is provided and .cloudProvider.external is enabled
  • Check out the Attention Needed section of the changelog for more information.
• Implement the CCM/CSI migration for OpenStack and vSphere (#1468, #1469, #1472, #1482, #1487, #1494)
  • The CCM/CSI migration is used to migrate clusters running in-tree cloud provider (i.e. with .cloudProvider.external set to false) to the external CCM (cloud-controller-manager) and CSI plugin.
  • The migration is implemented with the kubeone migrate to-ccm-csi command.
  • The CCM/CSI migration for vSphere is currently experimental and not tested.
  • More information about how the CCM/CSI migration works can be found by running kubeone migrate to-ccm-csi --help.

Addons

• Add a new optional embedded addon default-storage-class used to deploy default StorageClass for AWS, Azure, GCP, OpenStack, vSphere, or Hetzner clusters (#1488)

Changed

General

• [BREAKING/ACTION REQUIRED] Increase the minimum Kubernetes version to v1.19.0 (#1466)
  • If you have Kubernetes clusters running v1.18 or older, you need to use an older KubeOne release to upgrade them to v1.19, and then upgrade to KubeOne 1.3.
  • Check out the Compatibility guide for more information about supported Kubernetes versions for each KubeOne release.
• Improve the kubeone reset output to include more information about the target cluster (#1474)

Fixed

• Make kubeone apply skip already provisioned static worker nodes (#1485)
• Fix NPE when migrating to containerd (#1499)

Updated

• OpenStack CCM version now depends on the Kubernetes version (#1465)
  • Kubernetes 1.19 clusters use OpenStack CCM v1.19.2
  • Kubernetes 1.20 clusters use OpenStack CCM v1.20.2
  • Kubernetes 1.21 clusters use OpenStack CCM v1.21.0
  • Kubernetes 1.22+ clusters use OpenStack CCM v1.22.0
• vSphere CCM (CPI) version now depends on the Kubernetes version (#1489)
  • Kubernetes 1.19 clusters use vSphere CPI v1.19.0
  • Kubernetes 1.20 clusters use vSphere CPI v1.20.0
  • Kubernetes 1.21 clusters use vSphere CPI v1.21.0
  • Kubernetes 1.22+ clusters are currently unsupported on vSphere (see Known Issues for more details)
• Update metrics-server to v0.5.0 (#1483)
  • This fixes support for Kubernetes 1.22 clusters.
  • The metrics-server now uses serving certificates signed by the Kubernetes CA instead of the self-signed certificates.
• Update machine-controller to v1.35.2 (#1489)
• Update Hetzner CSI driver to v1.6.0 (#1491)

Removed

• Remove CSIMigration and CSIMigrationComplete fields from the API (#1473)
  • Those two fields were non-functional since they were added, so this change shouldn't affect users.
  • If you have any of those those two fields set in the KubeOneCluster manifest, make sure to remove them or otherwise the validation will fail.

Checksums

SHA256 checksums can be found in the kubeone_1.3.0-rc.0_checksums.txt file.