Problems of install v1.6.0

[siaimes]

Organization Name:

Short summary about the issue/question: TASK [container-engine/docker : ensure docker-ce repository is enabled] failed

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 2: ordinal not in range(128)
failed: [rzxt-openpai-100] (item=deb https://download.docker.com/linux/ubuntu xenial stable
) => {"ansible_loop_var": "item", "changed": false, "item": "deb https://download.docker.com/linux/ubuntu xenial stable\n", "module_stderr": "Traceback (most recent call last):\n  File \"<stdin>\", line 102, in <module>\n  File \"<stdin>\", line 94, in _ansiballz_main\n  File \"<stdin>\", line 40, in invoke_module\n  File \"/usr/lib/python3.5/runpy.py\", line 196, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib/python3.5/runpy.py\", line 96, in _run_module_code\n    mod_name, mod_spec, pkg_name, script_name)\n  File \"/usr/lib/python3.5/runpy.py\", line 85, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_apt_repository_payload_jnt_14wj/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py\", line 564, in <module>\n  File \"/tmp/ansible_apt_repository_payload_jnt_14wj/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py\", line 515, in main\n  File \"/tmp/ansible_apt_repository_payload_jnt_14wj/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py\", line 397, in __init__\n  File \"/tmp/ansible_apt_repository_payload_jnt_14wj/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py\", line 176, in __init__\n  File \"/tmp/ansible_apt_repository_payload_jnt_14wj/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py\", line 276, in load\n  File \"/usr/lib/python3.5/encodings/ascii.py\", line 26, in decode\n    return codecs.ascii_decode(input, self.errors)[0]\nUnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 2: ordinal not in range(128)\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

Brief what process you are following:

How to reproduce it:

OpenPAI Environment: v1.6.0

• OpenPAI version: v1.6.0
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release): ubuntu 16.04
• Kernel (e.g. uname -a):
• Hardware (e.g. core number, memory size, storage size, GPU type etc.):
• Others:

Anything else we need to know:

[siaimes]

It seems that the ansible version has been updated? Is this the cause of the failure of v1.6.0 deployment?

[siaimes]

new problem. on ubuntu 20.04.

TASK [../roles/docker-cache/install : run add_docker_cache_config.py script to add docker cache config] *************************************************************************************
changed: [rzxt-openpai-090]
fatal: [rzxt-openpai-092]: FAILED! => {"changed": true, "cmd": ["python3", "/tmp/add_docker_cache_config.py", "172.17.175.92:30500"], "delta": "0:00:00.308062", "end": "2021-04-25 03:18:14.548182", "msg": "non-zero return code", "rc": 1, "start": "2021-04-25 03:18:14.240120", "stderr": "Traceback (most recent call last):\n  File \"/tmp/add_docker_cache_config.py\", line 46, in <module>\n    main()\n  File \"/tmp/add_docker_cache_config.py\", line 24, in main\n    current_config = json.load(f);\n  File \"/usr/lib/python3.8/json/__init__.py\", line 293, in load\n    return loads(fp.read(),\n  File \"/usr/lib/python3.8/json/__init__.py\", line 357, in loads\n    return _default_decoder.decode(s)\n  File \"/usr/lib/python3.8/json/decoder.py\", line 337, in decode\n    obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n  File \"/usr/lib/python3.8/json/decoder.py\", line 355, in raw_decode\n    raise JSONDecodeError(\"Expecting value\", s, err.value) from None\njson.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)", "stderr_lines": ["Traceback (most recent call last):", "  File \"/tmp/add_docker_cache_config.py\", line 46, in <module>", "    main()", "  File \"/tmp/add_docker_cache_config.py\", line 24, in main", "    current_config = json.load(f);", "  File \"/usr/lib/python3.8/json/__init__.py\", line 293, in load", "    return loads(fp.read(),", "  File \"/usr/lib/python3.8/json/__init__.py\", line 357, in loads", "    return _default_decoder.decode(s)", "  File \"/usr/lib/python3.8/json/decoder.py\", line 337, in decode", "    obj, end = self.raw_decode(s, idx=_w(s, 0).end())", "  File \"/usr/lib/python3.8/json/decoder.py\", line 355, in raw_decode", "    raise JSONDecodeError(\"Expecting value\", s, err.value) from None", "json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)"], "stdout": "", "stdout_lines": []}

[Binyang2014]

@hzy46 Any comments for this?

[hzy46]

Caused by https://github.com/microsoft/pai/blob/master/contrib/kubespray/roles/docker-cache/install/files/add_docker_cache_config.py. If /etc/docker/daemon.json doesn't exist or is an empty file, the script will fail.

@siaimes, for a workaround, please create the file /etc/docker/daemon.json with content {} on 172.17.175.92.

@debuggy please take a look at this issue.

[hzy46]

It seems that the ansible version has been updated? Is this the cause of the failure of v1.6.0 deployment?

Ansible 2.7.12 doesn't support ubuntu 20.04. So we use ansible 2.9.7.

See the comments here: https://github.com/microsoft/pai/blob/master/contrib/kubespray/script/environment.sh#L46

[siaimes]

It seems that the ansible version has been updated? Is this the cause of the failure of v1.6.0 deployment?

Ansible 2.7.12 doesn't support ubuntu 20.04. So we use ansible 2.9.7.

See the comments here: https://github.com/microsoft/pai/blob/master/contrib/kubespray/script/environment.sh#L46

But now I installed on ubuntu 16.04 ansible also be upgraded to 2.9.7, and then unable to deploy the service.

[siaimes]

Caused by https://github.com/microsoft/pai/blob/master/contrib/kubespray/roles/docker-cache/install/files/add_docker_cache_config.py. If /etc/docker/daemon.json doesn't exist or is an empty file, the script will fail.

@siaimes, for a workaround, please create the file /etc/docker/daemon.json with content {} on 172.17.175.92.

@debuggy please take a look at this issue.

So, I think this is a bug, the master node does not have a GPU, so there is no /etc/docker/daemon.json.

[siaimes]

It seems that the ansible version has been updated? Is this the cause of the failure of v1.6.0 deployment?

Ansible 2.7.12 doesn't support ubuntu 20.04. So we use ansible 2.9.7.
See the comments here: https://github.com/microsoft/pai/blob/master/contrib/kubespray/script/environment.sh#L46

But now I installed on ubuntu 16.04 ansible also be upgraded to 2.9.7, and then unable to deploy the service.

on ubuntu 16.04, this error occurred.

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 2: ordinal not in range(128)
failed: [csip-090] (item=deb https://download.docker.com/linux/ubuntu xenial stable
) => {"ansible_loop_var": "item", "changed": false, "item": "deb https://download.docker.com/linux/ubuntu xenial stable\n", "module_stderr": "Traceback (most recent call last):\n  File \"<stdin>\", line 102, in <module>\n  File \"<stdin>\", line 94, in _ansiballz_main\n  File \"<stdin>\", line 40, in invoke_module\n  File \"/usr/lib/python3.5/runpy.py\", line 196, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib/python3.5/runpy.py\", line 96, in _run_module_code\n    mod_name, mod_spec, pkg_name, script_name)\n  File \"/usr/lib/python3.5/runpy.py\", line 85, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_apt_repository_payload_ofapqotz/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py\", line 564, in <module>\n  File \"/tmp/ansible_apt_repository_payload_ofapqotz/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py\", line 515, in main\n  File \"/tmp/ansible_apt_repository_payload_ofapqotz/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py\", line 397, in __init__\n  File \"/tmp/ansible_apt_repository_payload_ofapqotz/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py\", line 176, in __init__\n  File \"/tmp/ansible_apt_repository_payload_ofapqotz/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py\", line 276, in load\n  File \"/usr/lib/python3.5/encodings/ascii.py\", line 26, in decode\n    return codecs.ascii_decode(input, self.errors)[0]\nUnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 2: ordinal not in range(128)\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

[siaimes]

I remove the line sudo python3 -m pip install ansible==2.9.7, then new error occurred.

ERROR! no action detected in task. This often indicates a misspelled module name, or incorrect module path.

The error appears to have been in '/home/csip/pai/contrib/kubespray/docker-cache-config-distribute.yml': line 11, column 7, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

  tasks:
    - name: Restart service docker config from /etc/docker/daemon.json after update
      ^ here

[siaimes]

I remove the line sudo python3 -m pip install ansible==2.9.7, then new error occurred.

ERROR! no action detected in task. This often indicates a misspelled module name, or incorrect module path.

The error appears to have been in '/home/csip/pai/contrib/kubespray/docker-cache-config-distribute.yml': line 11, column 7, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

  tasks:
    - name: Restart service docker config from /etc/docker/daemon.json after update
      ^ here

I remove line ansible-playbook -i ${HOME}/pai-deploy/cluster-cfg/hosts.yml docker-cache-config-distribute.yml || exit $?

the error reappeard.

TASK [container-engine/docker : ensure docker-ce repository is enabled] *********************************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 2: ordinal not in range(128)
failed: [csip-093] (item=deb https://download.docker.com/linux/ubuntu xenial stable
) => {"changed": false, "item": "deb https://download.docker.com/linux/ubuntu xenial stable\n", "module_stderr": "Traceback (most recent call last):\n  File \"<stdin>\", line 113, in <module>\n  File \"<stdin>\", line 105, in _ansiballz_main\n  File \"<stdin>\", line 48, in invoke_module\n  File \"/usr/lib/python3.5/imp.py\", line 234, in load_module\n    return load_source(name, filename, file)\n  File \"/usr/lib/python3.5/imp.py\", line 170, in load_source\n    module = _exec(spec, sys.modules[name])\n  File \"<frozen importlib._bootstrap>\", line 626, in _exec\n  File \"<frozen importlib._bootstrap_external>\", line 665, in exec_module\n  File \"<frozen importlib._bootstrap>\", line 222, in _call_with_frames_removed\n  File \"/tmp/ansible_apt_repository_payload_66bhmoxx/__main__.py\", line 549, in <module>\n  File \"/tmp/ansible_apt_repository_payload_66bhmoxx/__main__.py\", line 509, in main\n  File \"/tmp/ansible_apt_repository_payload_66bhmoxx/__main__.py\", line 391, in __init__\n  File \"/tmp/ansible_apt_repository_payload_66bhmoxx/__main__.py\", line 175, in __init__\n  File \"/tmp/ansible_apt_repository_payload_66bhmoxx/__main__.py\", line 275, in load\n  File \"/usr/lib/python3.5/encodings/ascii.py\", line 26, in decode\n    return codecs.ascii_decode(input, self.errors)[0]\nUnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 2: ordinal not in range(128)\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 2: ordinal not in range(128)
failed: [csip-090] (item=deb https://download.docker.com/linux/ubuntu xenial stable
) => {"changed": false, "item": "deb https://download.docker.com/linux/ubuntu xenial stable\n", "module_stderr": "Traceback (most recent call last):\n  File \"<stdin>\", line 113, in <module>\n  File \"<stdin>\", line 105, in _ansiballz_main\n  File \"<stdin>\", line 48, in invoke_module\n  File \"/usr/lib/python3.5/imp.py\", line 234, in load_module\n    return load_source(name, filename, file)\n  File \"/usr/lib/python3.5/imp.py\", line 170, in load_source\n    module = _exec(spec, sys.modules[name])\n  File \"<frozen importlib._bootstrap>\", line 626, in _exec\n  File \"<frozen importlib._bootstrap_external>\", line 665, in exec_module\n  File \"<frozen importlib._bootstrap>\", line 222, in _call_with_frames_removed\n  File \"/tmp/ansible_apt_repository_payload_or6rcjpa/__main__.py\", line 549, in <module>\n  File \"/tmp/ansible_apt_repository_payload_or6rcjpa/__main__.py\", line 509, in main\n  File \"/tmp/ansible_apt_repository_payload_or6rcjpa/__main__.py\", line 391, in __init__\n  File \"/tmp/ansible_apt_repository_payload_or6rcjpa/__main__.py\", line 175, in __init__\n  File \"/tmp/ansible_apt_repository_payload_or6rcjpa/__main__.py\", line 275, in load\n  File \"/usr/lib/python3.5/encodings/ascii.py\", line 26, in decode\n    return codecs.ascii_decode(input, self.errors)[0]\nUnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 2: ordinal not in range(128)\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

[siaimes]

Organization Name:

Short summary about the issue/question: TASK [container-engine/docker : ensure docker-ce repository is enabled] failed

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 2: ordinal not in range(128)
failed: [rzxt-openpai-100] (item=deb https://download.docker.com/linux/ubuntu xenial stable
) => {"ansible_loop_var": "item", "changed": false, "item": "deb https://download.docker.com/linux/ubuntu xenial stable\n", "module_stderr": "Traceback (most recent call last):\n  File \"<stdin>\", line 102, in <module>\n  File \"<stdin>\", line 94, in _ansiballz_main\n  File \"<stdin>\", line 40, in invoke_module\n  File \"/usr/lib/python3.5/runpy.py\", line 196, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib/python3.5/runpy.py\", line 96, in _run_module_code\n    mod_name, mod_spec, pkg_name, script_name)\n  File \"/usr/lib/python3.5/runpy.py\", line 85, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_apt_repository_payload_jnt_14wj/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py\", line 564, in <module>\n  File \"/tmp/ansible_apt_repository_payload_jnt_14wj/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py\", line 515, in main\n  File \"/tmp/ansible_apt_repository_payload_jnt_14wj/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py\", line 397, in __init__\n  File \"/tmp/ansible_apt_repository_payload_jnt_14wj/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py\", line 176, in __init__\n  File \"/tmp/ansible_apt_repository_payload_jnt_14wj/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py\", line 276, in load\n  File \"/usr/lib/python3.5/encodings/ascii.py\", line 26, in decode\n    return codecs.ascii_decode(input, self.errors)[0]\nUnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 2: ordinal not in range(128)\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

Brief what process you are following:

How to reproduce it:

OpenPAI Environment: v1.6.0

• OpenPAI version: v1.6.0
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release): ubuntu 16.04
• Kernel (e.g. uname -a):
• Hardware (e.g. core number, memory size, storage size, GPU type etc.):
• Others:

Anything else we need to know:

The most original error occurred in this line, which may be caused by incompatibility between python3 and python2.

ansible-playbook -i inventory/pai/hosts.yml -e "ansible_python_interpreter=/usr/bin/python3" cluster.yml --become --become-user=root -e "@inventory/pai/openpai.yml" || exit $?

[siaimes]

Even if I did not modify this part, /etc/docker/daemon.json was also modified to point the mirror to the master node.

#######################################################################
#                    OpenPAI Customized Settings                      #
#######################################################################
# enable_hived_scheduler: true
# enable_docker_cache: false
# docker_cache_storage_backend: "azure" # or "filesystem"
# docker_cache_azure_account_name: ""
# docker_cache_azure_account_key: ""
# docker_cache_azure_container_name: "dockerregistry"
# docker_cache_fs_mount_path: "/var/lib/registry"
# docker_cache_remote_url: "https://registry-1.docker.io"
# docker_cache_htpasswd: ""
# enable_marketplace: "false"

[siaimes]

Organization Name:
Short summary about the issue/question: TASK [container-engine/docker : ensure docker-ce repository is enabled] failed

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 2: ordinal not in range(128)
failed: [rzxt-openpai-100] (item=deb https://download.docker.com/linux/ubuntu xenial stable
) => {"ansible_loop_var": "item", "changed": false, "item": "deb https://download.docker.com/linux/ubuntu xenial stable\n", "module_stderr": "Traceback (most recent call last):\n  File \"<stdin>\", line 102, in <module>\n  File \"<stdin>\", line 94, in _ansiballz_main\n  File \"<stdin>\", line 40, in invoke_module\n  File \"/usr/lib/python3.5/runpy.py\", line 196, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib/python3.5/runpy.py\", line 96, in _run_module_code\n    mod_name, mod_spec, pkg_name, script_name)\n  File \"/usr/lib/python3.5/runpy.py\", line 85, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_apt_repository_payload_jnt_14wj/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py\", line 564, in <module>\n  File \"/tmp/ansible_apt_repository_payload_jnt_14wj/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py\", line 515, in main\n  File \"/tmp/ansible_apt_repository_payload_jnt_14wj/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py\", line 397, in __init__\n  File \"/tmp/ansible_apt_repository_payload_jnt_14wj/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py\", line 176, in __init__\n  File \"/tmp/ansible_apt_repository_payload_jnt_14wj/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py\", line 276, in load\n  File \"/usr/lib/python3.5/encodings/ascii.py\", line 26, in decode\n    return codecs.ascii_decode(input, self.errors)[0]\nUnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 2: ordinal not in range(128)\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

Brief what process you are following:
How to reproduce it:
OpenPAI Environment: v1.6.0

• OpenPAI version: v1.6.0
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release): ubuntu 16.04
• Kernel (e.g. uname -a):
• Hardware (e.g. core number, memory size, storage size, GPU type etc.):
• Others:

Anything else we need to know:

The most original error occurred in this line, which may be caused by incompatibility between python3 and python2.

ansible-playbook -i inventory/pai/hosts.yml -e "ansible_python_interpreter=/usr/bin/python3" cluster.yml --become --become-user=root -e "@inventory/pai/openpai.yml" || exit $?

Solve this problem by changing python3 to python2.

[siaimes]

succeed after changing python3 to python2.

But the final output has no master node IP.

OpenPAI is successfully deployed, please check the following information:
Kubernetes cluster config :     ~/pai-deploy/kube/config
OpenPAI cluster config    :     ~/pai-deploy/cluster-cfg
OpenPAI cluster ID        :     pai
Default username          :     admin
Default password          :     admin-password

You can go to http:, then use the default username and password to log in.

[yiyione]

succeed after changing python3 to python2.

But the final output has no master node IP.

OpenPAI is successfully deployed, please check the following information:
Kubernetes cluster config :     ~/pai-deploy/kube/config
OpenPAI cluster config    :     ~/pai-deploy/cluster-cfg
OpenPAI cluster ID        :     pai
Default username          :     admin
Default password          :     admin-password

You can go to http:, then use the default username and password to log in.

After the deploy, can you visit the webportal ? or just the final output has problem

[siaimes]

succeed after changing python3 to python2.
But the final output has no master node IP.

OpenPAI is successfully deployed, please check the following information:
Kubernetes cluster config :     ~/pai-deploy/kube/config
OpenPAI cluster config    :     ~/pai-deploy/cluster-cfg
OpenPAI cluster ID        :     pai
Default username          :     admin
Default password          :     admin-password

You can go to http:, then use the default username and password to log in.

After the deploy, can you visit the webportal ? or just the final output has problem

just the final output has problem.

[siaimes]

The paictl version is inconsistent.

Please input the cluster-id which you wanna operate: pai
2021-04-27 05:33:06,471 [INFO] - deployment.confStorage.download : Congratulations: Cluster-id checking passed.
2021-04-27 05:33:07,085 [INFO] - deployment.utility.pai_version : Cluster version: v1.6.0

2021-04-27 05:33:07,088 [INFO] - deployment.utility.pai_version : paictl version: v1.5.0

[siaimes]

maybe duel to docker-cache service do not start after /bin/bash quick-start-service.sh

But the job cannot be started. This is the log of kubelet, it keeps outputting the image pull completion. The openpai system did not output any logs.

csip@csip-090:~$ sudo systemctl status kubelet.service 
● kubelet.service - Kubernetes Kubelet Server
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2021-04-27 13:19:30 CST; 4min 39s ago
     Docs: https://github.com/GoogleCloudPlatform/kubernetes
 Main PID: 173276 (kubelet)
    Tasks: 0
   Memory: 32.5M
      CPU: 1.107s
   CGroup: /system.slice/kubelet.service
           ‣ 173276 /usr/local/bin/kubelet --logtostderr=true --v=2 --node-ip=172.17.175.90 --hostname-override=csip-090 --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --config=/etc/kubernetes/kubelet-config.yaml --kubeconfig=/etc/kubernetes/kubelet.conf 

Apr 27 13:23:31 csip-090 kubelet[173276]: I0427 13:23:31.097675  173276 kube_docker_client.go:342] Pulling image "openpai/standard:python_3.6-pytorch_1.2.0-gpu": "7b872974e97c: Pull complete "
Apr 27 13:23:31 csip-090 kubelet[173276]: I0427 13:23:31.776293  173276 setters.go:73] Using node IP: "172.17.175.90"
Apr 27 13:23:41 csip-090 kubelet[173276]: I0427 13:23:41.097618  173276 kube_docker_client.go:342] Pulling image "openpai/standard:python_3.6-pytorch_1.2.0-gpu": "7b872974e97c: Pull complete "
Apr 27 13:23:41 csip-090 kubelet[173276]: I0427 13:23:41.788918  173276 setters.go:73] Using node IP: "172.17.175.90"
Apr 27 13:23:51 csip-090 kubelet[173276]: I0427 13:23:51.097632  173276 kube_docker_client.go:342] Pulling image "openpai/standard:python_3.6-pytorch_1.2.0-gpu": "7b872974e97c: Pull complete "
Apr 27 13:23:51 csip-090 kubelet[173276]: I0427 13:23:51.382029  173276 kubelet_getters.go:177] status for pod nginx-proxy-csip-090 updated to {Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2021-04-27 11:48:40 +0800 CST  } {Ready True 0001-01-01 00:00:00 +000
Apr 27 13:23:51 csip-090 kubelet[173276]: I0427 13:23:51.662060  173276 endpoint.go:111] State pushed for device plugin github.com/fuse
Apr 27 13:23:51 csip-090 kubelet[173276]: I0427 13:23:51.800339  173276 setters.go:73] Using node IP: "172.17.175.90"
Apr 27 13:24:01 csip-090 kubelet[173276]: I0427 13:24:01.097620  173276 kube_docker_client.go:342] Pulling image "openpai/standard:python_3.6-pytorch_1.2.0-gpu": "7b872974e97c: Pull complete "
Apr 27 13:24:01 csip-090 kubelet[173276]: I0427 13:24:01.813667  173276 setters.go:73] Using node IP: "172.17.175.90"

[Binyang2014]

@siaimes Do you use our dev-box to run ansible playbook, or you run the playbook in the host env.
From your description, seems there are several issues:

1. Can not deploy PAI successfully. After you change to use python2 this issue is fixed . This is for deployment issue.
2. Another issue is your job can not start successfully, for this issue, we provide some events can be seen in web portal, or you can use kubectl describe po <your-job-pod> to get more details

[Binyang2014]

Thanks this information. I updated the previous comments. I want to confirm the deployment issue you already fixed by using python2? (We need to figure out why this error is happened )

For job can not start issue, can you run kubecrl describe po <your-job-pod-name> to give us more info? If it caused by docker pull error. @debuggy Can you help to take a look for this docker cache issue?

[siaimes]

Thanks this information. I updated the previous comments. I want to confirm the deployment issue you already fixed by using python2? (We need to figure out why this error is happened )

Indeed it is.

[siaimes]

For job can not start issue, can you run kubecrl describe po <your-job-pod-name> to give us more info? If it caused by docker pull error. @debuggy Can you help to take a look for this docker cache issue?

./paictl.py service start -n docker-cache

The last reply has explained the situation.

[Binyang2014]

Thanks @siaimes, so everything works good after you start the docker-cache? And your job can be started now? If so, we will figure out why docker-cache not started automatically

[siaimes]

Thanks @siaimes, so everything works good after you start the docker-cache? And your job can be started now? If so, we will figure out why docker-cache not started automatically

OK.

[debuggy]

Hi, I will check the issue about docker-cache, adding @SwordFaith in this thread.

[siaimes]

The market is not added to the homepage by default, but the related container is started.

[siaimes]

One comment was deleted for privacy reasons, and it contains the following information that may be useful.

@siaimes Do you use our dev-box to run ansible playbook, or you run the playbook in the host env.

I used a brand new ubuntu 16.04 virtual machine as a dev-box and configured it exactly according to the installation guide except docker cache change as follows.

#######################################################################
#                    OpenPAI Customized Settings                      #
#######################################################################
# enable_hived_scheduler: true
enable_docker_cache: true
docker_cache_storage_backend: "filesystem"
# docker_cache_azure_account_name: ""
# docker_cache_azure_account_key: ""
# docker_cache_azure_container_name: "dockerregistry"
docker_cache_fs_mount_path: "/var/lib/registry"
# docker_cache_remote_url: "https://registry-1.docker.io"
# docker_cache_htpasswd: "" 
# enable_marketplace: "true"

In this way, the docker cache service should be started after the /bin/bash quick-start-service.sh, but it is actually not started.

I started a dev-box container, and then I ran this command and it was all right.

./paictl.py service start -n docker-cache

docker ps shows that the container was indeed started manually.

csip@csip-093:/var/lib$ sudo docker ps
CONTAINER ID   IMAGE                                              COMMAND                   CREATED             STATUS             PORTS     NAMES
27dae951a6cf   1fd8e1b0bb7e                                       "/entrypoint.sh /etc…"    52 seconds ago      Up 52 seconds                k8s_docker-registry-cache_docker-registry-cache-6cc556f5bd-vm82s_default_1496d699-8567-4853-958f-3156b65c72b6_0
f937d64f3fbe   gcr.io/google_containers/pause-amd64:3.1   "/pause"                  53 seconds ago      Up 52 seconds                k8s_POD_docker-registry-cache-6cc556f5bd-vm82s_default_1496d699-8567-4853-958f-3156b65c72b6_0
fc0e1a86a7c6   openpai/kubernetes-dashboard-amd64                 "/dashboard --insecu…"    54 minutes ago      Up About an hour             k8s_kubernetes-dashboard-openpai_kubernetes-dashboard-openpai-6bb47894d8-k2qxs_kube-system_261638fa-ecda-4933-b48d-661381edf2df_0
900224203a9d   gcr.io/google_containers/pause-amd64:3.1   "/pause"                  54 minutes ago      Up About an hour             k8s_POD_kubernetes-dashboard-openpai-6bb47894d8-k2qxs_kube-system_261638fa-ecda-4933-b48d-661381edf2df_0
54622b411d5a   openpai/fluentd                                    "tini -- /bin/entryp…"    54 minutes ago      Up About an hour             k8s_fluentd_fluentd-ds-cwxpx_default_a84fc92e-86c4-40a8-86f7-380db48d30d1_0
1a197724daf0   gcr.io/google_containers/pause-amd64:3.1   "/pause"                  55 minutes ago      Up About an hour             k8s_POD_fluentd-ds-cwxpx_default_a84fc92e-86c4-40a8-86f7-380db48d30d1_0
4398b921887a   openpai/pai-marketplace-webportal                  "docker-entrypoint.s…"    56 minutes ago      Up About an hour             k8s_marketplace-webportal_marketplace-webportal-ds-wwvjt_default_def0e5b0-7c72-4646-b45f-d816517cbe9f_0

[SwordFaith]

Hi, have you tried to use quick-start install in our dev-box docker container? I wonder if those errors are due to not use our dev box to deploy. I'm a little confused by your info. It seems you start a brand new ubuntu 16.04 machine to reinstall openpai v1.6.0, and encountered several issues. The ansible version issue is about python version or ansible version, it will be a big help to understand the error if you share the detailed error message with us. Kubectl service info and the log will help a lot as well.

[SwordFaith]

Caused by https://github.com/microsoft/pai/blob/master/contrib/kubespray/roles/docker-cache/install/files/add_docker_cache_config.py. If /etc/docker/daemon.json doesn't exist or is an empty file, the script will fail.
@siaimes, for a workaround, please create the file /etc/docker/daemon.json with content {} on 172.17.175.92.
@debuggy please take a look at this issue.

So, I think this is a bug, the master node does not have a GPU, so there is no /etc/docker/daemon.json.

It‘s my bad for not consider this situation. We will fix this later.

[siaimes]

Hi, have you tried to use quick-start install in our dev-box docker container? I wonder if those errors are due to not use our dev box to deploy. I'm a little confused by your info. It seems you start a brand new ubuntu 16.04 machine to reinstall openpai v1.6.0, and encountered several issues.

First, there are three machines, master, worker, and dev-box. My dev-box is a ubuntu 16.04 virtual machine. Then follow this process exactly except docker cache section.

This document does not require /bin/bash quick-start-kubespray.sh and /bin/bash quick-start-service.sh to be executed in the dev-box container, so I run this command in dev-box virtual machine.

[SwordFaith]

Hi, have you tried to use quick-start install in our dev-box docker container? I wonder if those errors are due to not use our dev box to deploy. I'm a little confused by your info. It seems you start a brand new ubuntu 16.04 machine to reinstall openpai v1.6.0, and encountered several issues.

First, there are three machines, master, worker, and dev-box. My dev-box is a ubuntu 16.04 virtual machine. Then follow this process exactly except docker cache section.

This document does not require /bin/bash quick-start-kubespray.sh and /bin/bash quick-start-service.sh to be executed in the dev-box container, so I run this command in dev-box virtual machine.

Can you run the failed playbook with -vvv to get the detailed log? It will be a big help

[siaimes]

The ansible version issue is about python version or ansible version, it will be a big help to understand the error if you share the detailed error message with us. Kubectl service info and the log will help a lot as well.

This error should come from ansibel, maybe it is not compatible with python3.

After change this line

ansible-playbook -i inventory/pai/hosts.yml -e "ansible_python_interpreter=/usr/bin/python3" cluster.yml --become --become-user=root -e "@inventory/pai/openpai.yml" || exit $?

to this

ansible-playbook -i inventory/pai/hosts.yml -e "ansible_python_interpreter=/usr/bin/python" cluster.yml --become --become-user=root -e "@inventory/pai/openpai.yml" || exit $?

it worked.

[siaimes]

Can you run the failed playbook with -vvv to get the detailed log? It will be a big help

I'm sorry, I just successfully installed v1.6.0, so I can no longer run these commands.

[SwordFaith]

Can you run the failed playbook with -vvv to get the detailed log? It will be a big help

I'm sorry, I just successfully installed v1.6.0, so I can no longer run these commands.

It's OK, we will go through your situation by ourselves in order to find the bug, thanks for using our project again.

[siaimes]

TASK [container-engine/docker : ensure docker-ce repository is enabled] failed
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 2: ordinal not in range(128)
failed: [rzxt-openpai-100] (item=deb https://download.docker.com/linux/ubuntu xenial stable
) => {"ansible_loop_var": "item", "changed": false, "item": "deb https://download.docker.com/linux/ubuntu xenial stable\n", "module_stderr": "Traceback (most recent call last):\n  File \"<stdin>\", line 102, in <module>\n  File \"<stdin>\", line 94, in _ansiballz_main\n  File \"<stdin>\", line 40, in invoke_module\n  File \"/usr/lib/python3.5/runpy.py\", line 196, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib/python3.5/runpy.py\", line 96, in _run_module_code\n    mod_name, mod_spec, pkg_name, script_name)\n  File \"/usr/lib/python3.5/runpy.py\", line 85, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_apt_repository_payload_jnt_14wj/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py\", line 564, in <module>\n  File \"/tmp/ansible_apt_repository_payload_jnt_14wj/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py\", line 515, in main\n  File \"/tmp/ansible_apt_repository_payload_jnt_14wj/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py\", line 397, in __init__\n  File \"/tmp/ansible_apt_repository_payload_jnt_14wj/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py\", line 176, in __init__\n  File \"/tmp/ansible_apt_repository_payload_jnt_14wj/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py\", line 276, in load\n  File \"/usr/lib/python3.5/encodings/ascii.py\", line 26, in decode\n    return codecs.ascii_decode(input, self.errors)[0]\nUnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 2: ordinal not in range(128)\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

Is this information sufficient to locate the problem?
It contains the task name. Although I tried to locate the problem by searching related keywords in the pai project, I didn't find any useful information, so I finally found a higher-level command, which is to change python3 to python2.

[SwordFaith]

TASK [container-engine/docker : ensure docker-ce repository is enabled] failed
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 2: ordinal not in range(128)
failed: [rzxt-openpai-100] (item=deb https://download.docker.com/linux/ubuntu xenial stable
) => {"ansible_loop_var": "item", "changed": false, "item": "deb https://download.docker.com/linux/ubuntu xenial stable\n", "module_stderr": "Traceback (most recent call last):\n File "", line 102, in \n File "", line 94, in _ansiballz_main\n File "", line 40, in invoke_module\n File "/usr/lib/python3.5/runpy.py", line 196, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File "/usr/lib/python3.5/runpy.py", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File "/usr/lib/python3.5/runpy.py", line 85, in _run_code\n exec(code, run_globals)\n File "/tmp/ansible_apt_repository_payload_jnt_14wj/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py", line 564, in \n File "/tmp/ansible_apt_repository_payload_jnt_14wj/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py", line 515, in main\n File "/tmp/ansible_apt_repository_payload_jnt_14wj/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py", line 397, in init\n File "/tmp/ansible_apt_repository_payload_jnt_14wj/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py", line 176, in init\n File "/tmp/ansible_apt_repository_payload_jnt_14wj/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py", line 276, in load\n File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode\n return codecs.ascii_decode(input, self.errors)[0]\nUnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 2: ordinal not in range(128)\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

This seems an apt install task failed. kubernetes-sigs/kubespray#6231

[siaimes]

TASK [container-engine/docker : ensure docker-ce repository is enabled] failed
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 2: ordinal not in range(128)
failed: [rzxt-openpai-100] (item=deb https://download.docker.com/linux/ubuntu xenial stable
) => {"ansible_loop_var": "item", "changed": false, "item": "deb https://download.docker.com/linux/ubuntu xenial stable\n", "module_stderr": "Traceback (most recent call last):\n File "", line 102, in \n File "", line 94, in _ansiballz_main\n File "", line 40, in invoke_module\n File "/usr/lib/python3.5/runpy.py", line 196, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File "/usr/lib/python3.5/runpy.py", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File "/usr/lib/python3.5/runpy.py", line 85, in _run_code\n exec(code, run_globals)\n File "/tmp/ansible_apt_repository_payload_jnt_14wj/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py", line 564, in \n File "/tmp/ansible_apt_repository_payload_jnt_14wj/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py", line 515, in main\n File "/tmp/ansible_apt_repository_payload_jnt_14wj/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py", line 397, in init\n File "/tmp/ansible_apt_repository_payload_jnt_14wj/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py", line 176, in init\n File "/tmp/ansible_apt_repository_payload_jnt_14wj/ansible_apt_repository_payload.zip/ansible/modules/packaging/os/apt_repository.py", line 276, in load\n File "/usr/lib/python3.5/encodings/ascii.py", line 26, in decode\n return codecs.ascii_decode(input, self.errors)[0]\nUnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 2: ordinal not in range(128)\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

This seems an apt install task failed. kubernetes-sigs/kubespray#6231

I have always had a question. Since users are required to install docker in advance in the document, why does the openpai installation script reinstall it again? Because the way users add docker sources is different from openpai, this often leads to apt conflicts.

[siaimes]

The paictl version is inconsistent.

Please input the cluster-id which you wanna operate: pai
2021-04-27 05:33:06,471 [INFO] - deployment.confStorage.download : Congratulations: Cluster-id checking passed.
2021-04-27 05:33:07,085 [INFO] - deployment.utility.pai_version : Cluster version: v1.6.0

2021-04-27 05:33:07,088 [INFO] - deployment.utility.pai_version : paictl version: v1.5.0

@SwordFaith This may be another bug. This is output when running ./paictl.py service start.

[siaimes]

The paictl version is inconsistent.

Please input the cluster-id which you wanna operate: pai
2021-04-27 05:33:06,471 [INFO] - deployment.confStorage.download : Congratulations: Cluster-id checking passed.
2021-04-27 05:33:07,085 [INFO] - deployment.utility.pai_version : Cluster version: v1.6.0

2021-04-27 05:33:07,088 [INFO] - deployment.utility.pai_version : paictl version: v1.5.0

@SwordFaith This may be another bug. This is output when running ./paictl.py service start.

Related to issue 5465.