it's really really really hard run kubespray in china! #6207

willzhang · 2020-05-29T06:34:06Z

i have change many things ,but i still failed when change the kubelet kubectl kubeadm url, it need check sum, i really have no idea ,so i must give up, it's too hard !

TASK [download : download_file | Download item] ***************************************************************************************************************************************************************
fatal: [node1 -> 192.168.93.60]: FAILED! => {"attempts": 4, "changed": true, "checksum_dest": null, "checksum_src": "28619090913443987880712a3388fa26ce50bac1", "dest": "/tmp/releases/kubeadm-v1.18.2-amd64", "elapsed": 3, "msg": "The checksum for /tmp/releases/kubeadm-v1.18.2-amd64 did not match 290bb6acb12c844f76affbab1ce374903bd97c4f19ac8cd3e6fdb7208d638ac8; it was a60974e9840e006076d204fd4ddcba96213beba10fb89ff01882095546c9684d.", "src": "/root/.ansible/tmp/ansible-moduletmp-1590761713.14-2YfN66/tmpPfQqLR", "url": "https://kube-binary.oss-cn-shenzhen.aliyuncs.com/V1.18.3/kubeadm"}
fatal: [node2 -> 192.168.93.61]: FAILED! => {"attempts": 4, "changed": true, "checksum_dest": null, "checksum_src": "28619090913443987880712a3388fa26ce50bac1", "dest": "/tmp/releases/kubeadm-v1.18.2-amd64", "elapsed": 3, "msg": "The checksum for /tmp/releases/kubeadm-v1.18.2-amd64 did not match 290bb6acb12c844f76affbab1ce374903bd97c4f19ac8cd3e6fdb7208d638ac8; it was a60974e9840e006076d204fd4ddcba96213beba10fb89ff01882095546c9684d.", "src": "/root/.ansible/tmp/ansible-moduletmp-1590761717.34-iDdOsp/tmp1ZEN9B", "url": "https://kube-binary.oss-cn-shenzhen.aliyuncs.com/V1.18.3/kubeadm"}

修改k8s版本及镜像仓库，使用阿里云镜像仓库

# cat inventory/mycluster/group_vars/k8s-cluster/k8s-cluster.yml
kube_version: v1.18.2
kube_image_repo: "registry.aliyuncs.com/google_containers"

修改docker镜像加速，使用阿里云镜像加速

# cat inventory/mycluster/group_vars/all/docker.yml
docker_registry_mirrors:
   - https://uyah70su.mirror.aliyuncs.com

修改docker-ce yum源，使用阿里云yum源安装docker-ce

$ vim roles/container-engine/docker/defaults/main.yml

# CentOS/RedHat docker-ce repo
docker_rh_repo_base_url: 'https://mirrors.aliyun.com/docker-ce/linux/centos/7/$basearch/stable'
docker_rh_repo_gpgkey: 'https://mirrors.aliyun.com/docker-ce/linux/centos/gpg'

# CentOS/RedHat Extras repo
extras_rh_repo_base_url: "http://mirrors.aliyun.com/centos/$releasever/extras/$basearch/"
extras_rh_repo_gpgkey: "http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7"

修改kube download url

$ vim roles/download/defaults/main.yml
kubelet_download_url: "https://kube-binary.oss-cn-shenzhen.aliyuncs.com/V1.18.3/kubelet"
kubectl_download_url: "https://kube-binary.oss-cn-shenzhen.aliyuncs.com/V1.18.3/kubectl"
kubeadm_download_url: "https://kube-binary.oss-cn-shenzhen.aliyuncs.com/V1.18.3/kubeadm"

The text was updated successfully, but these errors were encountered:

willzhang · 2020-05-29T06:35:44Z

and the offline docs not clearly, i can't get any help from it
https://github.com/kubernetes-sigs/kubespray/blob/master/docs/downloads.md#offline-environment

i want a step by step offline docs.

floryut · 2020-05-29T06:51:34Z

@EppO Maybe can give you some hint on a full offline installation

EppO · 2020-06-02T14:43:04Z

you're mixing 2 kubernetes versions so checksums don't match:

https://kube-binary.oss-cn-shenzhen.aliyuncs.com/V1.18.3/kubeadm
/tmp/releases/kubeadm-v1.18.2-amd64

I guess you're setting the version in the kubeadm_download_url. In order to avoid that checksum mismatch, you should set it that way:

kubeadm_download_url: "http://yourhttpfilesserver/path/{{ kube_version }}/kubeadm"

and use the kube_version of your inventory.

rahulwagh · 2020-06-04T09:13:00Z

@floryut
May be this offline guide can help you to do the kubespray installation
Guide - https://jhooq.com/kubespray-12-steps-for-installing-a-production-ready-kubernetes-cluster/

I pretty much followed all the 12 steps mentioned in the guide, i hope it helps you.

floryut · 2020-06-04T09:27:02Z

@floryut
May be this offline guide can help you to do the kubespray installation
Guide - https://jhooq.com/kubespray-12-steps-for-installing-a-production-ready-kubernetes-cluster/

I pretty much followed all the 12 steps mentioned in the guide, i hope it helps you.

You mean @willzhang

EppO · 2020-06-04T13:28:59Z

@floryut
May be this offline guide can help you to do the kubespray installation
Guide - https://jhooq.com/kubespray-12-steps-for-installing-a-production-ready-kubernetes-cluster/

I pretty much followed all the 12 steps mentioned in the guide, i hope it helps you.

This guide is very detailed but it's not intended to be used for an offline installation:

git clone the kubespray repo on github
download pip packages from PyPI
doesn't modify the image repos for all container images used by kubespray
download kubectl from google storage

For now the instructions for offline instructions are pretty succinct: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/downloads.md#offline-environment
I commented in that issue the recipe I was using: #5973 (comment)

If you're struggling with these instructions, let us know we'll improve the docs.

willzhang · 2020-06-07T13:22:41Z

you're mixing 2 kubernetes versions so checksums don't match:
https://kube-binary.oss-cn-shenzhen.aliyuncs.com/V1.18.3/kubeadm
/tmp/releases/kubeadm-v1.18.2-amd64
I guess you're setting the version in the kubeadm_download_url. In order to avoid that checksum mismatch, you should set it that way:
kubeadm_download_url: "http://yourhttpfilesserver/path/{{ kube_version }}/kubeadm"
and use the kube_version of your inventory.

you are right, the version is wrong, i have change it and it download success.

and there have no kubeadm kubelet kubectl binary mirror in china, i must download and upload it to some place like https://kube-binary.oss-cn-shenzhen.aliyuncs.com/V1.18.3/kubeadm, a object storage of aliyun.it's so troublesome

i can install them with yum install , there have yum repo mirror in china ,so can kubespray support yum install kubelet kubeadm kubectl ?

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg \
https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF


#yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

willzhang · 2020-06-07T13:36:06Z

other problems:
1.docker images
can i change all the registry url to my local registry ,maybe a local harbor, but the problem is what docker images should i save to my local registry first?

$ vim inventory/mycluster/group_vars/k8s-cluster/k8s-cluster.yml

kube_image_repo: "registry.aliyuncs.com/google_containers"

$ vim roles/download/defaults/main.yml
# gcr and kubernetes image repo define
gcr_image_repo: "gcr.io"
kube_image_repo: "k8s.gcr.io"

# docker image repo define
docker_image_repo: "docker.io"

# quay image repo define
#quay_image_repo: "quay.io"
quay_image_repo: "quay-mirror.qiniu.com"

# alauda.cn image repo (for kube-ovn...)
alauda_image_repo: "index.alauda.cn"

can kubespray support a comand like

ansible-playbook -i inventory/mycluster/hosts.yaml image list
ansible-playbook -i inventory/mycluster/hosts.yaml image pull

long long ago ,in china ,we can use gcr.azk8s.cn as replice of gcr.io and k8s.gcr.io ,But now it's not available. Microsoft has canceled it。 so it's really hard to find a suitable registry mirror.

so tell me what image i need , i can pull them use other way， and save them to my local harbor registry first.

2.github files
download file from github in china is very slow ,just about 50kb/s， so can i download this file to some local directory like /tmp/kubespray/xxxfile before run kubespray?

[root@node1 kubespray]# cat roles/download/defaults/main.yml | grep github.com
# https://github.com/ansible/ansible/issues/32750. Set this variable to False to disable
etcd_download_url: "https://github.com/coreos/etcd/releases/download/{{ etcd_version }}/etcd-{{ etcd_version }}-linux-{{ image_arch }}.tar.gz"
cni_download_url: "https://github.com/containernetworking/plugins/releases/download/{{ cni_version }}/cni-plugins-linux-{{ image_arch }}-{{ cni_version }}.tgz"
calicoctl_download_url: "https://github.com/projectcalico/calicoctl/releases/download/{{ calico_ctl_version }}/calicoctl-linux-{{ image_arch }}"
crictl_download_url: "https://github.com/kubernetes-sigs/cri-tools/releases/download/{{ crictl_version }}/crictl-{{ crictl_version }}-{{ ansible_system | lower }}-{{ image_arch }}.tar.gz"

EppO · 2020-06-09T01:33:58Z

i can install them with yum install , there have yum repo mirror in china ,so can kubespray support yum install kubelet kubeadm kubectl ?

No currently kubespray doesn't support this mode of installation for kubelet/kubeadm/kubectl

can kubespray support a comand like
ansible-playbook -i inventory/mycluster/hosts.yaml image list
ansible-playbook -i inventory/mycluster/hosts.yaml image pull
long long ago ,in china ,we can use gcr.azk8s.cn as replice of gcr.io and k8s.gcr.io ,But now it's not available. Microsoft has canceled it。 so it's really hard to find a suitable registry mirror.

so tell me what image i need , i can pull them use other way， and save them to my local harbor registry first.

First, the complete list of container images you need depends on your setup (network plugin, addons, ...), so look at https://github.com/kubernetes-sigs/kubespray/blob/master/roles/download/defaults/main.yml and try to determine which one you'll use.
For the kubernetes related images, you can run kubeadm config images list, it will tell about you kube-apiserver, kube-scheduler, kube-proxy, kube-controller-manager, pause and etcd images.
There is no easier way than trial and error: you run kubespray, it fails because you miss an image in your local registry, you add it, you reset your cluster and try again, until all images needed are finally there.

2.github files
download file from github in china is very slow ,just about 50kb/s， so can i download this file to some local directory like /tmp/kubespray/xxxfile before run kubespray?

[root@node1 kubespray]# cat roles/download/defaults/main.yml | grep github.com
# https://github.com/ansible/ansible/issues/32750. Set this variable to False to disable
etcd_download_url: "https://github.com/coreos/etcd/releases/download/{{ etcd_version }}/etcd-{{ etcd_version }}-linux-{{ image_arch }}.tar.gz"
cni_download_url: "https://github.com/containernetworking/plugins/releases/download/{{ cni_version }}/cni-plugins-linux-{{ image_arch }}-{{ cni_version }}.tgz"
calicoctl_download_url: "https://github.com/projectcalico/calicoctl/releases/download/{{ calico_ctl_version }}/calicoctl-linux-{{ image_arch }}"
crictl_download_url: "https://github.com/kubernetes-sigs/cri-tools/releases/download/{{ crictl_version }}/crictl-{{ crictl_version }}-{{ ansible_system | lower }}-{{ image_arch }}.tar.gz"

I would suggest to put them at a similar spot than kubeadm/kubectl/kubelet if that's faster for you

DancingFFFire · 2020-06-23T09:21:45Z

https://github.com/DancingFFFire/kubespray-offline-china

I write it today。maybe can help you。

willzhang · 2020-06-28T03:01:05Z

https://github.com/DancingFFFire/kubespray-offline-china

I write it today。maybe can help you。

thanks, But it's still not easy，hope some day kubespray have a solution。

EppO · 2020-06-29T13:31:13Z

We improved the docs regarding offline environments: https://github.com/kubernetes-sigs/kubespray/blob/master/docs/offline-environment.md, please have a look and let us know if that helps.

fejta-bot · 2020-09-27T13:47:40Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot · 2020-10-27T14:31:25Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

fejta-bot · 2020-11-26T15:14:18Z

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

k8s-ci-robot · 2020-11-26T15:14:23Z

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

willzhang added the kind/feature Categorizes issue or PR as related to a new feature. label May 29, 2020

EppO mentioned this issue Jun 4, 2020

Improve air-gap installation instructions #6234

Merged

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 27, 2020

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Oct 27, 2020

k8s-ci-robot closed this as completed Nov 26, 2020

yankay mentioned this issue Oct 17, 2022

[Docs] support the download mirror for China #9396

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

it's really really really hard run kubespray in china! #6207

it's really really really hard run kubespray in china! #6207

willzhang commented May 29, 2020

willzhang commented May 29, 2020 •

edited

Loading

floryut commented May 29, 2020

EppO commented Jun 2, 2020 •

edited

Loading

rahulwagh commented Jun 4, 2020 •

edited

Loading

floryut commented Jun 4, 2020

EppO commented Jun 4, 2020 •

edited

Loading

willzhang commented Jun 7, 2020

willzhang commented Jun 7, 2020 •

edited

Loading

EppO commented Jun 9, 2020 •

edited

Loading

DancingFFFire commented Jun 23, 2020

willzhang commented Jun 28, 2020

EppO commented Jun 29, 2020

fejta-bot commented Sep 27, 2020

fejta-bot commented Oct 27, 2020

fejta-bot commented Nov 26, 2020

k8s-ci-robot commented Nov 26, 2020

it's really really really hard run kubespray in china! #6207

it's really really really hard run kubespray in china! #6207

Comments

willzhang commented May 29, 2020

willzhang commented May 29, 2020 • edited Loading

floryut commented May 29, 2020

EppO commented Jun 2, 2020 • edited Loading

rahulwagh commented Jun 4, 2020 • edited Loading

floryut commented Jun 4, 2020

EppO commented Jun 4, 2020 • edited Loading

willzhang commented Jun 7, 2020

willzhang commented Jun 7, 2020 • edited Loading

EppO commented Jun 9, 2020 • edited Loading

DancingFFFire commented Jun 23, 2020

willzhang commented Jun 28, 2020

EppO commented Jun 29, 2020

fejta-bot commented Sep 27, 2020

fejta-bot commented Oct 27, 2020

fejta-bot commented Nov 26, 2020

k8s-ci-robot commented Nov 26, 2020

willzhang commented May 29, 2020 •

edited

Loading

EppO commented Jun 2, 2020 •

edited

Loading

rahulwagh commented Jun 4, 2020 •

edited

Loading

EppO commented Jun 4, 2020 •

edited

Loading

willzhang commented Jun 7, 2020 •

edited

Loading

EppO commented Jun 9, 2020 •

edited

Loading