Skip to content

Commit

Permalink
Rename master to control plane - non-breaking changes only (part of k…
Browse files Browse the repository at this point in the history
  • Loading branch information
bogd committed Sep 2, 2024
1 parent 03372d8 commit 5d2bee1
Show file tree
Hide file tree
Showing 37 changed files with 172 additions and 135 deletions.
10 changes: 5 additions & 5 deletions contrib/terraform/openstack/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,17 +60,17 @@ You can create many different kubernetes topologies by setting the number of
different classes of hosts. For each class there are options for allocating
floating IP addresses or not.

- Master nodes with etcd
- Master nodes without etcd
- Control plane nodes with etcd
- Control plane nodes without etcd
- Standalone etcd hosts
- Kubernetes worker nodes

Note that the Ansible script will report an invalid configuration if you wind up
with an even number of etcd instances since that is not a valid configuration. This
restriction includes standalone etcd nodes that are deployed in a cluster along with
master nodes with etcd replicas. As an example, if you have three master nodes with
etcd replicas and three standalone etcd nodes, the script will fail since there are
now six total etcd replicas.
control plane nodes with etcd replicas. As an example, if you have three control plane
nodes with etcd replicas and three standalone etcd nodes, the script will fail since
there are now six total etcd replicas.

### GlusterFS shared file system

Expand Down
5 changes: 3 additions & 2 deletions docs/ansible/ansible.md
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,7 @@ The following tags are defined in playbooks:
| container_engine_accelerator | Enable nvidia accelerator for runtimes |
| container-engine | Configuring container engines |
| container-runtimes | Configuring container runtimes |
| control-plane | Configuring K8s control plane node role |
| coredns | Configuring coredns deployment |
| crio | Configuring crio container engine for hosts |
| crun | Configuring crun runtime |
Expand Down Expand Up @@ -199,7 +200,7 @@ The following tags are defined in playbooks:
| local-path-provisioner | Configure External provisioner: local-path |
| local-volume-provisioner | Configure External provisioner: local-volume |
| macvlan | Network plugin macvlan |
| master | Configuring K8s master node role |
| master (DEPRECATED) | Deprecated - see `control-plane` |
| metallb | Installing and configuring metallb |
| metrics_server | Configuring metrics_server |
| netchecker | Installing netchecker K8s app |
Expand All @@ -210,7 +211,7 @@ The following tags are defined in playbooks:
| node | Configuring K8s minion (compute) node role |
| nodelocaldns | Configuring nodelocaldns daemonset |
| node-label | Tasks linked to labeling of nodes |
| node-webhook | Tasks linked to webhook (grating access to resources) |
| node-webhook | Tasks linked to webhook (granting access to resources)|
| nvidia_gpu | Enable nvidia accelerator for runtimes |
| oci | Cloud provider: oci |
| persistent_volumes | Configure csi volumes |
Expand Down
2 changes: 1 addition & 1 deletion docs/operations/etcd.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Installs docker in etcd group members and runs etcd on docker containers. Only u

### Kubeadm

This deployment method is experimental and is only available for new deployments. This deploys etcd as a static pod in master hosts.
This deployment method is experimental and is only available for new deployments. This deploys etcd as a static pod on control plane hosts.

## Metrics

Expand Down
2 changes: 1 addition & 1 deletion extra_playbooks/migrate_openstack_provider.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
tasks:
- name: Include kubespray-default variables
include_vars: ../roles/kubespray-defaults/defaults/main/main.yml
- name: Copy get_cinder_pvs.sh to master
- name: Copy get_cinder_pvs.sh to first control plane node
copy:
src: get_cinder_pvs.sh
dest: /tmp
Expand Down
2 changes: 1 addition & 1 deletion extra_playbooks/upgrade-only-k8s.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@
- { role: kubespray-defaults}
- { role: kubernetes/preinstall, tags: preinstall }

- name: Handle upgrades to master components first to maintain backwards compat.
- name: Handle upgrades to control plane components first to maintain backwards compat.
hosts: kube_control_plane
any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
serial: 1
Expand Down
4 changes: 2 additions & 2 deletions inventory/sample/group_vars/all/all.yml
Original file line number Diff line number Diff line change
Expand Up @@ -75,8 +75,8 @@ loadbalancer_apiserver_healthcheck_port: 8081
# skip_http_proxy_on_os_packages: false

## Since workers are included in the no_proxy variable by default, docker engine will be restarted on all nodes (all
## pods will restart) when adding or removing workers. To override this behaviour by only including master nodes in the
## no_proxy variable, set below to true:
## pods will restart) when adding or removing workers. To override this behaviour by only including control plane nodes
## in the no_proxy variable, set below to true:
no_proxy_exclude_workers: false

## Certificate Management
Expand Down
2 changes: 1 addition & 1 deletion inventory/sample/group_vars/k8s_cluster/k8s-cluster.yml
Original file line number Diff line number Diff line change
Expand Up @@ -272,7 +272,7 @@ default_kubelet_config_dir: "{{ kube_config_dir }}/dynamic_kubelet_dir"
# kube_cpu_reserved: 100m
# kube_ephemeral_storage_reserved: 2Gi
# kube_pid_reserved: "1000"
# Reservation for master hosts
# Reservation for control plane hosts
# kube_master_memory_reserved: 512Mi
# kube_master_cpu_reserved: 200m
# kube_master_ephemeral_storage_reserved: 2Gi
Expand Down
2 changes: 1 addition & 1 deletion playbooks/remove_node.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@
- { role: remove-node/remove-etcd-node }
- { role: reset, tags: reset, when: reset_nodes | default(True) | bool }

# Currently cannot remove first master or etcd
# Currently cannot remove first control plane node or first etcd node
- name: Post node removal
hosts: "{{ node | default('kube_control_plane[1:]:etcd[1:]') }}"
gather_facts: false
Expand Down
4 changes: 2 additions & 2 deletions playbooks/upgrade_cluster.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@
- name: Install etcd
import_playbook: install_etcd.yml

- name: Handle upgrades to master components first to maintain backwards compat.
- name: Handle upgrades to control plane components first to maintain backwards compat.
gather_facts: false
hosts: kube_control_plane
any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
Expand All @@ -60,7 +60,7 @@
- { role: kubernetes-apps, tags: csi-driver }
- { role: upgrade/post-upgrade, tags: post-upgrade }

- name: Upgrade calico and external cloud provider on all masters, calico-rrs, and nodes
- name: Upgrade calico and external cloud provider on all control plane nodes, calico-rrs, and nodes
hosts: kube_control_plane:calico_rr:kube_node
gather_facts: false
any_errors_fatal: "{{ any_errors_fatal | default(true) }}"
Expand Down
11 changes: 8 additions & 3 deletions roles/etcd/handlers/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,19 +13,19 @@
service:
name: etcd
state: restarted
when: is_etcd_master
when: 'etcd' in group_names
listen: Restart etcd

- name: Reload etcd-events
service:
name: etcd-events
state: restarted
when: is_etcd_master
when: 'etcd' in group_names
listen: Restart etcd-events

- name: Wait for etcd up
uri:
url: "https://{% if is_etcd_master %}{{ etcd_address }}{% else %}127.0.0.1{% endif %}:2379/health"
url: "https://{% if 'etcd' in group_names %}{{ etcd_address }}{% else %}127.0.0.1{% endif %}:2379/health"
validate_certs: false
client_cert: "{{ etcd_cert_dir }}/member-{{ inventory_hostname }}.pem"
client_key: "{{ etcd_cert_dir }}/member-{{ inventory_hostname }}-key.pem"
Expand All @@ -40,8 +40,13 @@

- name: Wait for etcd-events up
uri:
<<<<<<< HEAD
url: "https://{% if is_etcd_master %}{{ etcd_address }}{% else %}127.0.0.1{% endif %}:2383/health"
validate_certs: false
=======
url: "https://{% if 'etcd' in group_names %}{{ etcd_address }}{% else %}127.0.0.1{% endif %}:2383/health"
validate_certs: no
>>>>>>> bd6cf5de5 (Rename master to control plane - non-breaking changes only (part of #11353). See also https://github.com/kubernetes/community/blob/65d886bb3029e73d9729e1d4f27422a7985233ed/sig-architecture/naming/recommendations/001-master-control-plane.md)
client_cert: "{{ etcd_cert_dir }}/member-{{ inventory_hostname }}.pem"
client_key: "{{ etcd_cert_dir }}/member-{{ inventory_hostname }}-key.pem"
register: result
Expand Down
39 changes: 28 additions & 11 deletions roles/etcd/tasks/configure.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
check_mode: false
run_once: true
when:
- is_etcd_master
- 'etcd' in group_names
- etcd_cluster_setup
tags:
- facts
Expand All @@ -30,7 +30,7 @@
check_mode: false
run_once: true
when:
- is_etcd_master
- 'etcd' in group_names
- etcd_events_cluster_setup
tags:
- facts
Expand All @@ -43,7 +43,7 @@

- name: Configure | Refresh etcd config
include_tasks: refresh_config.yml
when: is_etcd_master
when: 'etcd' in group_names

- name: Configure | Copy etcd.service systemd file
template:
Expand All @@ -54,7 +54,9 @@
# FIXME: check that systemd version >= 250 (factory-reset.target was introduced in that release)
# Remove once we drop support for systemd < 250
validate: "sh -c '[ -f /usr/bin/systemd/system/factory-reset.target ] || exit 0 && systemd-analyze verify %s:etcd-{{ etcd_deployment_type }}.service'"
when: is_etcd_master and etcd_cluster_setup
when:
- 'etcd' in group_names
- etcd_cluster_setup

- name: Configure | Copy etcd-events.service systemd file
template:
Expand All @@ -65,12 +67,14 @@
validate: "sh -c '[ -f /usr/bin/systemd/system/factory-reset.target ] || exit 0 && systemd-analyze verify %s:etcd-events-{{ etcd_deployment_type }}.service'"
# FIXME: check that systemd version >= 250 (factory-reset.target was introduced in that release)
# Remove once we drop support for systemd < 250
when: is_etcd_master and etcd_events_cluster_setup
when:
- 'etcd' in group_names
- etcd_events_cluster_setup

- name: Configure | reload systemd
systemd_service:
daemon_reload: true
when: is_etcd_master
when: 'etcd' in group_names

# when scaling new etcd will fail to start
- name: Configure | Ensure etcd is running
Expand All @@ -79,7 +83,9 @@
state: started
enabled: true
ignore_errors: "{{ etcd_cluster_is_healthy.rc == 0 }}" # noqa ignore-errors
when: is_etcd_master and etcd_cluster_setup
when:
- 'etcd' in group_names
- etcd_cluster_setup

# when scaling new etcd will fail to start
- name: Configure | Ensure etcd-events is running
Expand All @@ -88,7 +94,9 @@
state: started
enabled: true
ignore_errors: "{{ etcd_events_cluster_is_healthy.rc != 0 }}" # noqa ignore-errors
when: is_etcd_master and etcd_events_cluster_setup
when:
- 'etcd' in group_names
- etcd_events_cluster_setup

- name: Configure | Wait for etcd cluster to be healthy
shell: "set -o pipefail && {{ bin_dir }}/etcdctl endpoint --cluster status && {{ bin_dir }}/etcdctl endpoint --cluster health 2>&1 | grep -v 'Error: unhealthy cluster' >/dev/null"
Expand All @@ -102,7 +110,7 @@
check_mode: false
run_once: true
when:
- is_etcd_master
- 'etcd' in group_names
- etcd_cluster_setup
tags:
- facts
Expand All @@ -125,7 +133,7 @@
check_mode: false
run_once: true
when:
- is_etcd_master
- 'etcd' in group_names
- etcd_events_cluster_setup
tags:
- facts
Expand All @@ -141,8 +149,15 @@
register: etcd_member_in_cluster
ignore_errors: true # noqa ignore-errors
changed_when: false
<<<<<<< HEAD
check_mode: false
when: is_etcd_master and etcd_cluster_setup
=======
check_mode: no
when:
- 'etcd' in group_names
- etcd_cluster_setup
>>>>>>> bd6cf5de5 (Rename master to control plane - non-breaking changes only (part of #11353). See also https://github.com/kubernetes/community/blob/65d886bb3029e73d9729e1d4f27422a7985233ed/sig-architecture/naming/recommendations/001-master-control-plane.md)
tags:
- facts
environment:
Expand All @@ -158,7 +173,9 @@
ignore_errors: true # noqa ignore-errors
changed_when: false
check_mode: false
when: is_etcd_master and etcd_events_cluster_setup
when:
- 'etcd' in group_names
- etcd_events_cluster_setup
tags:
- facts
environment:
Expand Down
26 changes: 17 additions & 9 deletions roles/etcd/tasks/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,8 @@
- kube_network_plugin != "calico" or calico_datastore == "etcd"
- inventory_hostname in groups['k8s_cluster']
tags:
- master
- master # master tag is deprecated and replaced by control-plane
- control-plane
- network

- name: Set etcd_client_cert_serial
Expand All @@ -50,7 +51,8 @@
- kube_network_plugin != "calico" or calico_datastore == "etcd"
- inventory_hostname in groups['k8s_cluster']
tags:
- master
- master # master tag is deprecated and replaced by control-plane
- control-plane
- network

- name: Install etcdctl and etcdutl binary
Expand All @@ -61,36 +63,42 @@
- etcdutl
- upgrade
when:
- inventory_hostname in groups['etcd']
- 'etcd' in group_names
- etcd_cluster_setup

- name: Install etcd
include_tasks: "install_{{ etcd_deployment_type }}.yml"
when: is_etcd_master
when: 'etcd' in group_names
tags:
- upgrade

- name: Configure etcd
include_tasks: configure.yml
when: is_etcd_master
when: 'etcd' in group_names

- name: Refresh etcd config
include_tasks: refresh_config.yml
when: is_etcd_master
when: 'etcd' in group_names

- name: Restart etcd if certs changed
command: /bin/true
notify: Restart etcd
when: is_etcd_master and etcd_cluster_setup and etcd_secret_changed | default(false)
when:
- 'etcd' in group_names
- etcd_cluster_setup
- etcd_secret_changed | default(false)

- name: Restart etcd-events if certs changed
command: /bin/true
notify: Restart etcd
when: is_etcd_master and etcd_events_cluster_setup and etcd_secret_changed | default(false)
when:
- 'etcd' in group_names
- etcd_events_cluster_setup
- etcd_secret_changed | default(false)

# After etcd cluster is assembled, make sure that
# initial state of the cluster is in `existing`
# state instead of `new`.
- name: Refresh etcd config again for idempotency
include_tasks: refresh_config.yml
when: is_etcd_master
when: 'etcd' in group_names
8 changes: 6 additions & 2 deletions roles/etcd/tasks/refresh_config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,16 @@
dest: /etc/etcd.env
mode: "0640"
notify: Restart etcd
when: is_etcd_master and etcd_cluster_setup
when:
- 'etcd' in group_names
- etcd_cluster_setup

- name: Refresh config | Create etcd-events config file
template:
src: etcd-events.env.j2
dest: /etc/etcd-events.env
mode: "0640"
notify: Restart etcd-events
when: is_etcd_master and etcd_events_cluster_setup
when:
- 'etcd' in group_names
- etcd_events_cluster_setup
6 changes: 3 additions & 3 deletions roles/kubernetes-apps/metrics_server/tasks/main.yml
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
---
# If all masters have node role, there are no tainted master and toleration should not be specified.
- name: Check all masters are node or not
# If all control plane nodes have the node role, there are no tainted control plane nodes and toleration should not be specified.
- name: Check all control plane nodes are node or not
set_fact:
masters_are_not_tainted: "{{ groups['kube_node'] | intersect(groups['kube_control_plane']) == groups['kube_control_plane'] }}"
control_plane_nodes_are_not_tainted: "{{ groups['kube_node'] | intersect(groups['kube_control_plane']) == groups['kube_control_plane'] }}"

- name: Metrics Server | Delete addon dir
file:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -85,9 +85,9 @@ spec:
volumes:
- name: tmp
emptyDir: {}
{% if not masters_are_not_tainted or metrics_server_extra_tolerations is defined %}
{% if not control_plane_nodes_are_not_tainted or metrics_server_extra_tolerations is defined %}
tolerations:
{% if not masters_are_not_tainted %}
{% if not control_plane_nodes_are_not_tainted %}
- key: node-role.kubernetes.io/control-plane
effect: NoSchedule
{% endif %}
Expand Down
Loading

0 comments on commit 5d2bee1

Please sign in to comment.