download_cache_dir permission errors

[Jeroen0494]

Environment:

• OS (printf "$(uname -srm)\n$(cat /etc/os-release)\n"):
  Linux 5.4.0-148-generic x86_64

NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

• Version of Ansible (ansible --version):

ansible [core 2.12.5]
  config file = /var/lib/jenkins/workspace/k8s/06_add_node/ansible.cfg
  configured module search path = ['/var/lib/jenkins/workspace/k8s/06_add_node/module/kubespray/library']
  ansible python module location = /var/lib/jenkins/workspace/k8s/06_add_node/python-env/ansible/lib/python3.8/site-packages/ansible
  ansible collection location = /var/lib/jenkins/.ansible/collections:/usr/share/ansible/collections
  executable location = /var/lib/jenkins/workspace/k8s/06_add_node/python-env/ansible/bin/ansible
  python version = 3.8.10 (default, Mar 13 2023, 10:26:41) [GCC 9.4.0]
  jinja version = 2.11.3
  libyaml = True

• Version of Python (python --version):
  Python 3.8.10

Kubespray version (commit) (git rev-parse --short HEAD):
c553912f9

Bug report:
Files in the Kubespray cache directory are saved with the root user instead of the user executing Ansible.

jenkins@phcds274:~$ ls -al kubespray_cache/
total 1269420
drwxrwsr-x  3 root    jenkins      4096 May  8 10:23 .
drwxr-x--- 20 jenkins jenkins      4096 May  2 10:32 ..
-rw-r--r--  1 jenkins jenkins  59499254 May  8 10:23 calicoctl
-rw-r--r--  1 jenkins jenkins  36939510 Mar 22 10:56 cni-plugins-linux-amd64-v1.0.1.tgz
-rw-r--r--  1 jenkins jenkins  36336160 Apr 13 16:33 cni-plugins-linux-amd64-v1.1.1.tgz
-rw-rw-r--  1 jenkins jenkins  40517426 May  8 10:22 cni-plugins-linux-amd64-v1.2.0.tgz
-rw-rw-r--  1 jenkins jenkins  43433356 May  8 10:22 containerd-1.6.15-linux-amd64.tar.gz
-rw-r--r--  1 jenkins jenkins  44436699 Apr 24 11:10 containerd-1.6.4-linux-amd64.tar.gz
-rw-rw-r--  1 jenkins jenkins  44458241 May  1 10:28 containerd-1.6.8-linux-amd64.tar.gz
-rw-r--r--  1 jenkins jenkins  18510359 Apr 24 11:09 crictl-v1.23.0-linux-amd64.tar.gz
-rw-rw-r--  1 jenkins jenkins  14522128 May  1 10:28 crictl-v1.24.0-linux-amd64.tar.gz
-rw-rw-r--  1 jenkins jenkins  21819185 May  8 10:22 crictl-v1.25.0-linux-amd64.tar.gz
-rw-r--r--  1 jenkins jenkins  19444820 Apr 24 11:09 etcd-v3.5.3-linux-amd64.tar.gz
-rw-rw-r--  1 jenkins jenkins  19432359 May  1 10:28 etcd-v3.5.4-linux-amd64.tar.gz
-rw-rw-r--  1 jenkins jenkins  19492576 May  8 10:22 etcd-v3.5.6-linux-amd64.tar.gz
drwxr-xr-x  2 jenkins jenkins      4096 May  8 10:26 images
-rw-r--r--  1 jenkins jenkins  45858816 Mar 22 10:55 kubeadm-v1.22.8-amd64
-rw-r--r--  1 jenkins jenkins  45219840 Apr 13 16:34 kubeadm-v1.23.7-amd64
-rw-rw-r--  1 jenkins jenkins  44388352 May  1 10:28 kubeadm-v1.24.6-amd64
-rw-rw-r--  1 jenkins jenkins  43810816 May  8 10:22 kubeadm-v1.25.6-amd64
-rw-r--r--  1 jenkins jenkins  46940160 Mar 22 10:56 kubectl-v1.22.8-amd64
-rw-r--r--  1 jenkins jenkins  46596096 Apr 24 11:10 kubectl-v1.23.7-amd64
-rw-rw-r--  1 jenkins jenkins  45723648 May  1 10:28 kubectl-v1.24.6-amd64
-rw-rw-r--  1 jenkins jenkins  45031424 May  8 10:22 kubectl-v1.25.6-amd64
-rw-r--r--  1 jenkins jenkins 121250040 Mar 22 10:56 kubelet-v1.22.8-amd64
-rw-r--r--  1 jenkins jenkins 124542016 Apr 13 16:34 kubelet-v1.23.7-amd64
-rw-rw-r--  1 jenkins jenkins 116066776 May  1 10:28 kubelet-v1.24.6-amd64
-rw-rw-r--  1 jenkins jenkins 114245880 May  8 10:22 kubelet-v1.25.6-amd64
-rw-r--r--  1 jenkins jenkins  10479236 Apr 24 11:09 nerdctl-0.19.0-linux-amd64.tar.gz
-rw-rw-r--  1 jenkins jenkins  10685899 May  1 10:28 nerdctl-0.22.2-linux-amd64.tar.gz
-rw-rw-r--  1 jenkins jenkins  10715986 May  8 10:22 nerdctl-1.0.0-linux-amd64.tar.gz
-rw-r--r--  1 jenkins jenkins   9431456 May  1 10:28 runc
jenkins@phcds274:~$ ls -al kubespray_cache/images/
total 2018120
drwxr-xr-x 2 jenkins jenkins      4096 May  8 10:26 .
drwxrwsr-x 3 root    jenkins      4096 May  8 10:23 ..
-rwxr-xr-x 1 jenkins jenkins  76228929 Mar 22 10:59 docker.io_kubernetesui_dashboard-amd64_v2.4.0.tar
-rwxr-xr-x 1 jenkins jenkins  68018688 Apr 24 11:12 docker.io_kubernetesui_dashboard-amd64_v2.5.0.tar
-rwxr-xr-x 1 jenkins jenkins  75796480 May  1 10:31 docker.io_kubernetesui_dashboard-amd64_v2.6.1.tar
-rw-r--r-- 1 root    root     75798016 May  8 10:25 docker.io_kubernetesui_dashboard_v2.7.0.tar
-rwxr-xr-x 1 jenkins jenkins  16103349 Mar 22 10:59 docker.io_kubernetesui_metrics-scraper_v1.0.7.tar
-rwxr-xr-x 1 jenkins jenkins  19756032 May  1 10:32 docker.io_kubernetesui_metrics-scraper_v1.0.8.tar
-rwxr-xr-x 1 jenkins jenkins  59812628 Mar 22 11:01 docker.io_library_nginx_1.21.4.tar
-rwxr-xr-x 1 jenkins jenkins  10207232 May  1 10:30 docker.io_library_nginx_1.23.0-alpine.tar
-rw-r--r-- 1 root    root     10245632 May  8 10:24 docker.io_library_nginx_1.23.2-alpine.tar
-rwxr-xr-x 1 jenkins jenkins  10545632 Mar 22 10:59 k8s.gcr.io_addon-resizer_1.8.11.tar
-rwxr-xr-x 1 jenkins jenkins  14736869 Mar 22 10:58 k8s.gcr.io_coredns_coredns_v1.8.0.tar
-rwxr-xr-x 1 jenkins jenkins  13593600 Apr 24 11:12 k8s.gcr.io_coredns_coredns_v1.8.6.tar
-rwxr-xr-x 1 jenkins jenkins  16410932 Mar 22 10:59 k8s.gcr.io_cpa_cluster-proportional-autoscaler-amd64_1.8.5.tar
-rwxr-xr-x 1 jenkins jenkins  45127985 Mar 22 10:58 k8s.gcr.io_dns_k8s-dns-node-cache_1.21.1.tar
-rwxr-xr-x 1 jenkins jenkins  36783966 Mar 22 11:00 k8s.gcr.io_kube-apiserver_v1.22.8.tar
-rwxr-xr-x 1 jenkins jenkins  32610816 Apr 24 11:13 k8s.gcr.io_kube-apiserver_v1.23.7.tar
-rwxr-xr-x 1 jenkins jenkins  34987816 Mar 22 11:00 k8s.gcr.io_kube-controller-manager_v1.22.8.tar
-rwxr-xr-x 1 jenkins jenkins  30184448 Apr 24 11:13 k8s.gcr.io_kube-controller-manager_v1.23.7.tar
-rwxr-xr-x 1 jenkins jenkins  39049938 Mar 22 11:01 k8s.gcr.io_kube-proxy_v1.22.8.tar
-rwxr-xr-x 1 jenkins jenkins  39280128 Apr 24 11:13 k8s.gcr.io_kube-proxy_v1.23.7.tar
-rwxr-xr-x 1 jenkins jenkins  17189312 Mar 22 11:00 k8s.gcr.io_kube-scheduler_v1.22.8.tar
-rwxr-xr-x 1 jenkins jenkins  15145984 Apr 24 11:13 k8s.gcr.io_kube-scheduler_v1.23.7.tar
-rwxr-xr-x 1 jenkins jenkins  27847559 Mar 22 10:59 k8s.gcr.io_metrics-server_metrics-server_v0.5.0.tar
-rwxr-xr-x 1 jenkins jenkins  26031616 Apr 24 11:12 k8s.gcr.io_metrics-server_metrics-server_v0.5.2.tar
-rwxr-xr-x 1 jenkins jenkins    309337 Mar 22 10:58 k8s.gcr.io_pause_3.3.tar
-rwxr-xr-x 1 jenkins jenkins  54323077 Mar 22 10:57 quay.io_calico_cni_v3.20.3.tar
-rwxr-xr-x 1 jenkins jenkins  86902931 Apr 13 16:35 quay.io_calico_cni_v3.22.3.tar
-rwxr-xr-x 1 jenkins jenkins 107995648 May  1 10:30 quay.io_calico_cni_v3.23.3.tar
-rw-r--r-- 1 root    root     87473664 May  8 10:23 quay.io_calico_cni_v3.24.5.tar
-rwxr-xr-x 1 jenkins jenkins  28118835 Mar 22 10:58 quay.io_calico_kube-controllers_v3.20.3.tar
-rwxr-xr-x 1 jenkins jenkins  57444908 Apr 13 16:35 quay.io_calico_kube-controllers_v3.22.3.tar
-rwxr-xr-x 1 jenkins jenkins  53784576 May  1 10:30 quay.io_calico_kube-controllers_v3.23.3.tar
-rw-r--r-- 1 root    root     31154688 May  8 10:24 quay.io_calico_kube-controllers_v3.24.5.tar
-rwxr-xr-x 1 jenkins jenkins  69167046 Mar 22 10:56 quay.io_calico_node_v3.20.3.tar
-rwxr-xr-x 1 jenkins jenkins  79830907 Apr 13 16:34 quay.io_calico_node_v3.22.3.tar
-rwxr-xr-x 1 jenkins jenkins  73463296 May  1 10:29 quay.io_calico_node_v3.23.3.tar
-rw-r--r-- 1 root    root     81593856 May  8 10:23 quay.io_calico_node_v3.24.5.tar
-rwxr-xr-x 1 jenkins jenkins   9917047 Mar 22 10:57 quay.io_calico_pod2daemon-flexvol_v3.20.3.tar
-rwxr-xr-x 1 jenkins jenkins   9140169 Apr 13 16:35 quay.io_calico_pod2daemon-flexvol_v3.22.3.tar
-rwxr-xr-x 1 jenkins jenkins   8689152 May  1 10:30 quay.io_calico_pod2daemon-flexvol_v3.23.3.tar
-rw-r--r-- 1 root    root      7089664 May  8 10:24 quay.io_calico_pod2daemon-flexvol_v3.24.5.tar
-rwxr-xr-x 1 jenkins jenkins  44061204 Mar 22 10:55 quay.io_coreos_etcd_v3.5.0.tar
-rwxr-xr-x 1 jenkins jenkins  13593600 May  1 10:31 registry.k8s.io_coredns_coredns_v1.8.6.tar
-rw-r--r-- 1 root    root     14846464 May  8 10:24 registry.k8s.io_coredns_coredns_v1.9.3.tar
-rwxr-xr-x 1 jenkins jenkins  15216128 May  1 10:31 registry.k8s.io_cpa_cluster-proportional-autoscaler-amd64_1.8.5.tar
-rwxr-xr-x 1 jenkins jenkins  42458112 May  1 10:31 registry.k8s.io_dns_k8s-dns-node-cache_1.21.1.tar
-rwxr-xr-x 1 jenkins jenkins  33821696 May  1 10:32 registry.k8s.io_kube-apiserver_v1.24.6.tar
-rw-r--r-- 1 root    root     34263552 May  8 10:25 registry.k8s.io_kube-apiserver_v1.25.6.tar
-rwxr-xr-x 1 jenkins jenkins  31053824 May  1 10:32 registry.k8s.io_kube-controller-manager_v1.24.6.tar
-rw-r--r-- 1 root    root     31275008 May  8 10:26 registry.k8s.io_kube-controller-manager_v1.25.6.tar
-rwxr-xr-x 1 jenkins jenkins  39528448 May  1 10:33 registry.k8s.io_kube-proxy_v1.24.6.tar
-rw-r--r-- 1 root    root     20285952 May  8 10:26 registry.k8s.io_kube-proxy_v1.25.6.tar
-rwxr-xr-x 1 jenkins jenkins  15496704 May  1 10:32 registry.k8s.io_kube-scheduler_v1.24.6.tar
-rw-r--r-- 1 root    root     15809536 May  8 10:26 registry.k8s.io_kube-scheduler_v1.25.6.tar
-rwxr-xr-x 1 jenkins jenkins  28067328 May  1 10:31 registry.k8s.io_metrics-server_metrics-server_v0.6.1.tar
-rw-r--r-- 1 root    root     28144640 May  8 10:25 registry.k8s.io_metrics-server_metrics-server_v0.6.2.tar
-rwxr-xr-x 1 jenkins jenkins    310272 May  1 10:30 registry.k8s.io_pause_3.6.tar
-rw-r--r-- 1 root    root       319488 May  8 10:24 registry.k8s.io_pause_3.8.tar

Relevant inventory variables:

download_run_once: True
download_localhost: True
download_cache_dir: "/var/lib/jenkins/kubespray_cache"
download_keep_remote_cache: True

Command used to invoke ansible:

Output of ansible run:

[2023-05-09T06:11:18.489Z] TASK [download : prep_download | Create local cache for files and images on control node] ***
[2023-05-09T06:11:18.489Z] Tuesday 09 May 2023  08:11:18 +0200 (0:00:00.333)       0:02:04.195 *********** 
[2023-05-09T06:11:18.489Z] An exception occurred during task execution. To see the full traceback, use -vvv. The error was: PermissionError: [Errno 1] Operation not permitted: b'/var/lib/jenkins/kubespray_cache/images/quay.io_calico_node_v3.24.5.tar'
[2023-05-09T06:11:18.489Z] fatal: [dov-dris-a-k8s008 -> localhost]: FAILED! => changed=false 
[2023-05-09T06:11:18.489Z]   module_stderr: |-
[2023-05-09T06:11:18.489Z]     Traceback (most recent call last):
[2023-05-09T06:11:18.489Z]       File "/var/lib/jenkins/.ansible/tmp/ansible-tmp-1683612678.2734106-966970-87852861360441/AnsiballZ_file.py", line 107, in <module>
[2023-05-09T06:11:18.489Z]         _ansiballz_main()
[2023-05-09T06:11:18.489Z]       File "/var/lib/jenkins/.ansible/tmp/ansible-tmp-1683612678.2734106-966970-87852861360441/AnsiballZ_file.py", line 99, in _ansiballz_main
[2023-05-09T06:11:18.489Z]         invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
[2023-05-09T06:11:18.489Z]       File "/var/lib/jenkins/.ansible/tmp/ansible-tmp-1683612678.2734106-966970-87852861360441/AnsiballZ_file.py", line 47, in invoke_module
[2023-05-09T06:11:18.489Z]         runpy.run_module(mod_name='ansible.modules.file', init_globals=dict(_module_fqn='ansible.modules.file', _modlib_path=modlib_path),
[2023-05-09T06:11:18.489Z]       File "/usr/lib/python3.8/runpy.py", line 207, in run_module
[2023-05-09T06:11:18.489Z]         return _run_module_code(code, init_globals, run_name, mod_spec)
[2023-05-09T06:11:18.489Z]       File "/usr/lib/python3.8/runpy.py", line 97, in _run_module_code
[2023-05-09T06:11:18.489Z]         _run_code(code, mod_globals, init_globals,
[2023-05-09T06:11:18.489Z]       File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
[2023-05-09T06:11:18.489Z]         exec(code, run_globals)
[2023-05-09T06:11:18.489Z]       File "/tmp/ansible_file_payload_fyqu7fza/ansible_file_payload.zip/ansible/modules/file.py", line 972, in <module>
[2023-05-09T06:11:18.489Z]       File "/tmp/ansible_file_payload_fyqu7fza/ansible_file_payload.zip/ansible/modules/file.py", line 958, in main
[2023-05-09T06:11:18.489Z]       File "/tmp/ansible_file_payload_fyqu7fza/ansible_file_payload.zip/ansible/modules/file.py", line 678, in ensure_directory
[2023-05-09T06:11:18.489Z]       File "/tmp/ansible_file_payload_fyqu7fza/ansible_file_payload.zip/ansible/modules/file.py", line 353, in recursive_set_attributes
[2023-05-09T06:11:18.489Z]       File "/tmp/ansible_file_payload_fyqu7fza/ansible_file_payload.zip/ansible/module_utils/basic.py", line 1178, in set_fs_attributes_if_different
[2023-05-09T06:11:18.489Z]       File "/tmp/ansible_file_payload_fyqu7fza/ansible_file_payload.zip/ansible/module_utils/basic.py", line 938, in set_mode_if_different
[2023-05-09T06:11:18.489Z]     PermissionError: [Errno 1] Operation not permitted: b'/var/lib/jenkins/kubespray_cache/images/quay.io_calico_node_v3.24.5.tar'
[2023-05-09T06:11:18.489Z]   module_stdout: ''
[2023-05-09T06:11:18.489Z]   msg: |-
[2023-05-09T06:11:18.489Z]     MODULE FAILURE
[2023-05-09T06:11:18.489Z]     See stdout/stderr for the exact error
[2023-05-09T06:11:18.489Z]   rc: 1
[2023-05-09T06:11:18.489Z] 
[2023-05-09T06:11:18.489Z] NO MORE HOSTS LEFT *************************************************************

[Jeroen0494]

Kind reminder

[joes]

I am also encountering this issue at times and I need to fix the permissions manually or re-run the playbooks.

I started to suspect that running any other command with sudo prior to starting kubespray deployment the issue would arise more often. I think I saw some tasks in the playbooks that were used to ascertain if passwordless sudo was possible on a host that might be the culprit here. That check in turn somehow then determined what permissions was applied to the downloaded files. The checks made sense in the context of a remote host but not so much when done on localhost (when download_localhost: true).

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[Jeroen0494]

/remove-lifecycle stale
This is still an issue

[VannTen]

Can you try to see if the linked PR fix the issue ?