Make etcdctl copy command work on cri-o/containerd

[mattymo]

This is a follow-on to #5777 where etcd_kubeadm_enabled mode used to force using etcdctl from binary sources. Now it uses a Docker image, but the fix only included Docker container engine support. This adds support for cri-o and containerd.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mattymo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [mattymo]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[floryut]

Are you sure about this ? if container_manager is crio you are using docker run ?

[mattymo]

Ooh, you're right. I copied this from helm role. We should use crictl here. ctr is containerd-specific.

[EppO]

Just a shell alias wrapping up a ctr command on the master nodes would do the job too, right? Why do we need the binary installed on the hosts to achieve this use case? Running etcdctl inside the etcd container seems good enough to me although it needs the etcd container to be running if we just attach to it, but if it's not running for whatever reasons (etcd container failing), you wouldn't be able to run the etcdctl on the host anyway.

[mattymo]

@EppO Then you have to mount all the certs if you go with the shell wrapper. It's just more effort than it is worth.

[EppO]

@EppO Then you have to mount all the certs if you go with the shell wrapper. It's just more effort than it is worth.

You don't have to mount anything if you "exec" in the container. I was thinking about something like this (using docker, need to port it to crictl)

alias etcdctl='docker exec -it $(docker ps -f "name=k8s_etcd" --format "{{.ID}}") etcdctl --cert-file=/etc/kubernetes/ssl/etcd/peer.crt --key-file=/etc/kubernetes/ssl/etcd/peer.key --ca-file /etc/kubernetes/ssl/etcd/ca.crt --endpoints=https://127.0.0.1:2379'

with that alias installed on every master nodes, you can run any commands without specifying the certs or endpoints on any masters:

$ etcdctl cluster-health
member 5c04bf204def0eda is healthy: got healthy result from https://10.1.2.1:2379
member e949065838d88228 is healthy: got healthy result from https://10.1.2.2:2379
member fdc65c5087f1d04f is healthy: got healthy result from https://10.1.2.3:2379
cluster is healthy
$ etcdctl member list
5c04bf204def0eda: name=master01 peerURLs=https://10.1.2.1:2380 clientURLs=https://10.1.2.1:2379 isLeader=true
e949065838d88228: name=master02 peerURLs=https://10.1.2.2:2380 clientURLs=https://10.1.2.2:2379 isLeader=false
fdc65c5087f1d04f: name=master03 peerURLs=https://10.1.2.3:2380 clientURLs=https://10.1.2.3:2379 isLeader=false

[EppO]

On the other hand, if you want to "run" instead of "exec", because the etcd is failing for example, you would need to do something like this:

alias etcdctl='docker run -it --rm  --net=host --volume=/etc/kubernetes/ssl/etcd:/etc/kubernetes/ssl/etcd etcd:v3.3.12 etcdctl --cert-file=/etc/kubernetes/ssl/etcd/peer.crt --key-file=/etc/kubernetes/ssl/etcd/peer.key --ca-file /etc/kubernetes/ssl/etcd/ca.crt --endpoints=https://10.1.2.1:2379'

$ etcdctl member list
5c04bf204def0eda: name=master01 peerURLs=https://10.1.2.1:2380 clientURLs=https://10.1.2.1:2379 isLeader=true
e949065838d88228: name=master02 peerURLs=https://10.1.2.2:2380 clientURLs=https://10.1.2.2:2379 isLeader=false
fdc65c5087f1d04f: name=master03 peerURLs=https://10.1.2.3:2380 clientURLs=https://10.1.2.3:2379 isLeader=false

[didacog]

In case of a proxy scenario ctr command does not use the ENV setup in systemd/containerd unit, so this will fail. Also, the compare command fails with a file not found error if etcdctl binary does not exist , that is what is expected, as the previous version just copies the binary from the docker image.

I made it work with proxy ajdusting few things, maybe you'll find a better way, but at least the following worked for me, while this commit is failing now.
Also replaced command by shell to make ctr command work
I've modified the roles/etcd/tasks/install_etcdctl_docker.yml like this:

---
- name: Set commands for etcdctl container tasks
  set_fact:
    etcdctl_compare_command: >-
      {%- if container_manager in ['docker', 'crio'] %}
      {{ docker_bin_dir }}/docker run --rm -v {{ bin_dir }}:/systembindir --entrypoint /usr/bin/cmp {{ etcd_image_repo }}:{{ etcd_image_tag }} /usr/local/bin/etcdctl /systembindir/helm
      {%- elif container_manager == "containerd" %}
      [[ -f {{ bin_dir }}/etcdctl ]] && ctr run --rm --mount type=bind,src={{ bin_dir }},dst=/systembindir,options=rbind:rw {{ etcd_image_repo }}:{{ etcd_image_tag }} etcdctl-compare sh -c 'cmp /usr/local/bin/etcdctl /systembindir/etcdctl'
      {%- endif %}
    etcdctl_copy_command: >-
      {%- if container_manager in ['docker', 'crio'] %}
      {{ docker_bin_dir }}/docker run --rm -v {{ bin_dir }}:/systembindir --entrypoint /bin/cp {{ etcd_image_repo }}:{{ etcd_image_tag }} -f /usr/local/bin/etcdctl /systembindir/etcdctl
      {%- elif container_manager == "containerd" %}
      ctr run --rm --mount type=bind,src={{ bin_dir }},dst=/systembindir,options=rbind:rw {{ etcd_image_repo }}:{{ etcd_image_tag }} etcdctl-copy sh -c '/bin/cp -f /usr/local/bin/etcdctl /systembindir/etcdctl'
      {%- endif %}

- name: Ensure image is pulled for containerd
  shell: "HTTPS_PROXY={{ https_proxy }} ctr i pull {{ etcd_image_repo }}:{{ etcd_image_tag }}"
  when: container_manager == "containerd"

- name: Compare host etcdctl with etcdctl image
  shell: "{{ etcdctl_compare_command }}"
  register: etcdctl_task_compare_result
  until: etcdctl_task_compare_result.rc in [0,1,2]
  retries: 4
  delay: "{{ retry_stagger | random + 3 }}"
  changed_when: false
  failed_when: "etcdctl_task_compare_result.rc not in [0,1,2]"

- name: Copy etcdctl from etcdctl container
  shell: "{{ etcdctl_copy_command }}"
  when: etcdctl_task_compare_result.rc != 0
  register: etcdctl_task_result
  until: etcdctl_task_result.rc == 0
  retries: 4
  delay: "{{ retry_stagger | random + 3 }}"

[hafe]

[root@master-1 ~]# alias etcdctl="crictl exec -it $(crictl ps --name etcd -q) etcdctl --cacert /etc/kubernetes/ssl/etcd/ca.crt --cert /etc/kubernetes/ssl/etcd/healthcheck-client.crt --key /etc/kubernetes/ssl/etcd/healthcheck-client.key"
[root@master-1 ~]# etcdctl member list
302028076f67b4ea, started, master-1, https://10.0.2.23:2380, https://10.0.2.23:2379, false

works fine!

[k8s-ci-robot]

@mattymo: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[hafe]

I've got a patch with a shell wrapper for etcdctl. Are you interested?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[mattymo]

I haven't had time to clean up my PR. I merged #6857 in favor of this.
/close

[k8s-ci-robot]

@mattymo: Closed this PR.

In response to this:

I haven't had time to clean up my PR. I merged #6857 in favor of this.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.