Skip to content
This repository has been archived by the owner on Jan 11, 2023. It is now read-only.

Cluster provisioning fails due to invalid nvidia keys, during apt-get update in the cse for agents and master #3933

Closed
nikolajbrinch opened this issue Oct 2, 2018 · 14 comments
Closed

Cluster provisioning fails due to invalid nvidia keys, during apt-get update in the cse for agents and master #3933

nikolajbrinch opened this issue Oct 2, 2018 · 14 comments
Assignees
@CecileRobertMichon

Comments

@nikolajbrinch
Copy link

Is this a request for help?:
yes

Is this an ISSUE or FEATURE REQUEST? (choose one):
ISSUE

What version of acs-engine?:
0.22.2

kubernetes.json

{
  "apiVersion": "vlabs",
  "properties": {
    "orchestratorProfile": {
      "orchestratorType": "Kubernetes",
      "orchestratorRelease": "1.11",
      "orchestratorVersion": "1.11.2",
      "kubernetesConfig": {
        "enableRbac": true,
        "dockerEngineVersion": "17.05.*",
        "etcdDiskSizeGB": "256"
      }
    },
    "masterProfile": {
      "count": 1,
      "dnsPrefix": "...",
      "vmSize": "Standard_DS1_v2",
      "storageProfile": "ManagedDisks"
    },
    "agentPoolProfiles": [
      {
        "name": "agentpool1",
        "count": 3,
        "vmSize": "Standard_DS1_v2",
        "availabilityProfile": "AvailabilitySet",
        "storageProfile": "ManagedDisks"
      }
    ],
    "linuxProfile": {
      "adminUsername": "",
      "ssh": {
        "publicKeys": [
          {
            "keyData": "ssh-rsa ...""
          }
        ]
      }
    },
    "servicePrincipalProfile": {
      "clientId": "",
      "secret": ""
    }
  }
}

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm)

What happened:
Command output:

INFO[0547] Finished ARM Deployment (nine-test-cluster_rg-715616485). Error: Code="DeploymentFailed" Message="At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-debug for usage details." Details=  [
    {
      "code": "Conflict",
      "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"VMExtensionProvisioningError\",\r\n        \"message\": \"VM has reported a failure when processing extension 'cse-agent-2'. Error message: \\\"Enable failed: failed to execute command: command terminated with exit status=99\\n[stdout]\\n\\n[stderr]\\n\\\".\"\r\n      }\r\n    ]\r\n  }\r\n}"
    },
    {
      "code": "Conflict",
      "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"VMExtensionProvisioningError\",\r\n        \"message\": \"VM has reported a failure when processing extension 'cse-agent-1'. Error message: \\\"Enable failed: failed to execute command: command terminated with exit status=99\\n[stdout]\\n\\n[stderr]\\n\\\".\"\r\n      }\r\n    ]\r\n  }\r\n}"
    },
    {
      "code": "Conflict",
      "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"VMExtensionProvisioningError\",\r\n        \"message\": \"VM has reported a failure when processing extension 'cse-agent-0'. Error message: \\\"Enable failed: failed to execute command: command terminated with exit status=99\\n[stdout]\\n\\n[stderr]\\n\\\".\"\r\n      }\r\n    ]\r\n  }\r\n}"
    },
    {
      "code": "Conflict",
      "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"VMExtensionProvisioningError\",\r\n        \"message\": \"VM has reported a failure when processing extension 'cse-master-0'. Error message: \\\"Enable failed: failed to execute command: command terminated with exit status=99\\n[stdout]\\n\\n[stderr]\\n\\\".\"\r\n      }\r\n    ]\r\n  }\r\n}"
    }
  ] 
FATA[0547] Code="DeploymentFailed" Message="At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-debug for usage details." Details=[
    {
      "code": "Conflict",
      "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"VMExtensionProvisioningError\",\r\n        \"message\": \"VM has reported a failure when processing extension 'cse-agent-2'. Error message: \\\"Enable failed: failed to execute command: command terminated with exit status=99\\n[stdout]\\n\\n[stderr]\\n\\\".\"\r\n      }\r\n    ]\r\n  }\r\n}"
    },
    {
      "code": "Conflict",
      "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"VMExtensionProvisioningError\",\r\n        \"message\": \"VM has reported a failure when processing extension 'cse-agent-1'. Error message: \\\"Enable failed: failed to execute command: command terminated with exit status=99\\n[stdout]\\n\\n[stderr]\\n\\\".\"\r\n      }\r\n    ]\r\n  }\r\n}"
    },
    {
      "code": "Conflict",
      "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"VMExtensionProvisioningError\",\r\n        \"message\": \"VM has reported a failure when processing extension 'cse-agent-0'. Error message: \\\"Enable failed: failed to execute command: command terminated with exit status=99\\n[stdout]\\n\\n[stderr]\\n\\\".\"\r\n      }\r\n    ]\r\n  }\r\n}"
    },
    {
      "code": "Conflict",
      "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"VMExtensionProvisioningError\",\r\n        \"message\": \"VM has reported a failure when processing extension 'cse-master-0'. Error message: \\\"Enable failed: failed to execute command: command terminated with exit status=99\\n[stdout]\\n\\n[stderr]\\n\\\".\"\r\n      }\r\n    ]\r\n  }\r\n}"
    }
  ]

/var/log/azure/cluster-provisioning.log

++ date
++ hostname
+ echo Tue Oct 2 10:56:43 UTC 2018,k8s-master-17869660-0, startcustomscript
+ source /opt/azure/containers/provision_source.sh
++ ERR_SYSTEMCTL_START_FAIL=4
++ ERR_CLOUD_INIT_TIMEOUT=5
++ ERR_FILE_WATCH_TIMEOUT=6
++ ERR_HOLD_WALINUXAGENT=7
++ ERR_RELEASE_HOLD_WALINUXAGENT=8
++ ERR_APT_INSTALL_TIMEOUT=9
++ ERR_ETCD_DATA_DIR_NOT_FOUND=10
++ ERR_ETCD_RUNNING_TIMEOUT=11
++ ERR_ETCD_DOWNLOAD_TIMEOUT=12
++ ERR_ETCD_VOL_MOUNT_FAIL=13
++ ERR_ETCD_START_TIMEOUT=14
++ ERR_ETCD_CONFIG_FAIL=15
++ ERR_DOCKER_INSTALL_TIMEOUT=20
++ ERR_DOCKER_DOWNLOAD_TIMEOUT=21
++ ERR_DOCKER_KEY_DOWNLOAD_TIMEOUT=22
++ ERR_DOCKER_APT_KEY_TIMEOUT=23
++ ERR_DOCKER_START_FAIL=24
++ ERR_K8S_RUNNING_TIMEOUT=30
++ ERR_K8S_DOWNLOAD_TIMEOUT=31
++ ERR_KUBECTL_NOT_FOUND=32
++ ERR_KUBELET_START_FAIL=34
++ ERR_IMG_DOWNLOAD_TIMEOUT=33
++ ERR_CNI_DOWNLOAD_TIMEOUT=41
++ ERR_MS_PROD_DEB_DOWNLOAD_TIMEOUT=42
++ ERR_MS_PROD_DEB_PKG_ADD_FAIL=43
++ ERR_MODPROBE_FAIL=49
++ ERR_OUTBOUND_CONN_FAIL=50
++ ERR_KATA_KEY_DOWNLOAD_TIMEOUT=60
++ ERR_KATA_APT_KEY_TIMEOUT=61
++ ERR_KATA_INSTALL_TIMEOUT=62
++ ERR_CONTAINERD_DOWNLOAD_TIMEOUT=70
++ ERR_CUSTOM_SEARCH_DOMAINS_FAIL=80
++ ERR_GPU_DRIVERS_START_FAIL=84
++ ERR_GPU_DRIVERS_INSTALL_TIMEOUT=85
++ ERR_APT_DAILY_TIMEOUT=98
++ ERR_APT_UPDATE_TIMEOUT=99
++ ERR_CSE_PROVISION_SCRIPT_NOT_READY_TIMEOUT=100
+++ awk '{print toupper($0)}'
+++ tr -d 'ID="'
+++ grep '^ID='
+++ cat /etc/lsb-release /etc/os-release
++ OS=UBUNTU
++ UBUNTU_OS_NAME=UBUNTU
++ RHEL_OS_NAME=RHEL
++ COREOS_OS_NAME=COREOS
++ KUBECTL=/usr/local/bin/kubectl
++ DOCKER=/usr/bin/docker
++ GPU_DV=396.26
++ GPU_DEST=/usr/local/nvidia
++ NVIDIA_DOCKER_VERSION=2.0.3
++ DOCKER_VERSION=1.13.1-1
++ NVIDIA_CONTAINER_RUNTIME_VERSION=2.0.0
+ source /opt/azure/containers/provision_installs.sh
++ CC_SERVICE_IN_TMP=/opt/azure/containers/cc-proxy.service.in
++ CC_SOCKET_IN_TMP=/opt/azure/containers/cc-proxy.socket.in
++ CNI_CONFIG_DIR=/etc/cni/net.d
++ CNI_BIN_DIR=/opt/cni/bin
++ CNI_DOWNLOADS_DIR=/opt/cni/downloads
+ source /opt/azure/containers/provision_configs.sh
+ CUSTOM_SEARCH_DOMAIN_SCRIPT=/opt/azure/containers/setup-custom-search-domains.sh
+ set +x
+ [[ UBUNTU == COREOS ]]
+ '[' -f /var/run/reboot-required ']'
+ REBOOTREQUIRED=false
+ '[' -f /var/log/azure/golden-image-install.complete ']'
+ echo 'detected golden image pre-install'
detected golden image pre-install
+ FULL_INSTALL_REQUIRED=false
+ testOutboundConnection
+ retrycmd_if_failure 40 1 3 nc -vz www.google.com 443
+ retries=40
+ wait_sleep=1
+ timeout=3
+ shift
+ shift
+ shift
++ seq 1 40
+ for i in '$(seq 1 $retries)'
+ timeout 3 nc -vz www.google.com 443
Connection to www.google.com 443 port [tcp/https] succeeded!
+ '[' 0 -eq 0 ']'
+ break
+ echo Executed '"nc' -vz www.google.com '443"' 1 times
Executed "nc -vz www.google.com 443" 1 times
+ [[ ! -z true ]]
+ installEtcd
++ tr -d '[:space:]'
++ cut -d : -f 2
++ grep 'etcd Version'
++ etcd --version
+ CURRENT_VERSION=3.2.23
+ [[ 3.2.23 == \3\.\2\.\2\3 ]]
+ echo 'etcd version 3.2.23 is already installed, skipping download'
etcd version 3.2.23 is already installed, skipping download
+ false
+ echo 'Golden image; skipping dependencies installation'
Golden image; skipping dependencies installation
+ installContainerRuntime
+ [[ docker == \d\o\c\k\e\r ]]
+ installDocker
++ cut -d , -f 1
++ cut -d ' ' -f 3
++ docker --version
+ CURRENT_VERSION=1.13.1
+ [[ 1.13.1 = 17.05.* ]]
+ retrycmd_if_failure_no_stats 20 1 5 curl -fsSL https://aptdocker.azureedge.net/gpg
+ retries=20
+ wait_sleep=1
+ timeout=5
+ shift
+ shift
+ shift
++ seq 1 20
+ for i in '$(seq 1 $retries)'
+ timeout 5 curl -fsSL https://aptdocker.azureedge.net/gpg
+ '[' 0 -eq 0 ']'
+ break
+ retrycmd_if_failure 10 5 10 apt-key add /tmp/aptdocker.gpg
+ retries=10
+ wait_sleep=5
+ timeout=10
+ shift
+ shift
+ shift
++ seq 1 10
+ for i in '$(seq 1 $retries)'
+ timeout 10 apt-key add /tmp/aptdocker.gpg
OK
+ '[' 0 -eq 0 ']'
+ break
+ echo Executed '"apt-key' add '/tmp/aptdocker.gpg"' 1 times
Executed "apt-key add /tmp/aptdocker.gpg" 1 times
+ sudo tee /etc/apt/sources.list.d/docker.list
+ echo 'deb https://aptdocker.azureedge.net/repo ubuntu-xenial main'
deb https://aptdocker.azureedge.net/repo ubuntu-xenial main
+ printf 'Package: docker-engine\nPin: version 17.05.*\nPin-Priority: 550\n'
+ apt_get_update
+ retries=10
+ apt_update_output=/tmp/apt-get-update.out
++ seq 1 10
+ for i in '$(seq 1 $retries)'
+ timeout 30 dpkg --configure -a
+ timeout 30 apt-get -f -y install
Reading package lists...
Building dependency tree...
Reading state information...
0 upgraded, 0 newly installed, 0 to remove and 23 not upgraded.
+ grep -E '^([WE]:.*)|([eE]rr.*)$'
+ tee /tmp/apt-get-update.out
+ timeout 120 apt-get update
E: Could not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable)
E: Unable to lock directory /var/lib/apt/lists/
+ '[' 0 -ne 0 ']'
+ cat /tmp/apt-get-update.out
Reading package lists...
E: Could not get lock /var/lib/apt/lists/lock - open (11: Resource temporarily unavailable)
E: Unable to lock directory /var/lib/apt/lists/
+ '[' 1 -eq 10 ']'
+ sleep 30
+ for i in '$(seq 1 $retries)'
+ timeout 30 dpkg --configure -a
dpkg: error: dpkg status database is locked by another process
+ timeout 30 apt-get -f -y install
E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)
E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?
+ grep -E '^([WE]:.*)|([eE]rr.*)$'
+ tee /tmp/apt-get-update.out
+ timeout 120 apt-get update
Err:9 https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64  InRelease
Err:10 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease
Err:11 https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Some index files failed to download. They have been ignored, or old ones used instead.
+ '[' 0 -ne 0 ']'
+ cat /tmp/apt-get-update.out
Hit:1 http://azure.archive.ubuntu.com/ubuntu xenial InRelease
Hit:2 http://azure.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:3 http://azure.archive.ubuntu.com/ubuntu xenial-backports InRelease
Ign:4 http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04  InRelease
Hit:5 http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04  Release
Hit:6 http://security.ubuntu.com/ubuntu xenial-security InRelease
Hit:7 https://packages.microsoft.com/ubuntu/16.04/prod xenial InRelease
Hit:8 https://aptdocker.azureedge.net/repo ubuntu-xenial InRelease
Get:10 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease [1,136 B]
Get:11 https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease [1,129 B]
Err:9 https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64  InRelease
  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
Err:10 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease
  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
Err:11 https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease
  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
Fetched 2,265 B in 1s (1,567 B/s)
Reading package lists...
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Some index files failed to download. They have been ignored, or old ones used instead.
+ '[' 2 -eq 10 ']'
+ sleep 30
+ for i in '$(seq 1 $retries)'
+ timeout 30 dpkg --configure -a
dpkg: error: dpkg status database is locked by another process
+ timeout 30 apt-get -f -y install
E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)
E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?
+ grep -E '^([WE]:.*)|([eE]rr.*)$'
+ tee /tmp/apt-get-update.out
+ timeout 120 apt-get update
Err:8 https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64  InRelease
Err:10 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease
Err:11 https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Some index files failed to download. They have been ignored, or old ones used instead.
+ '[' 0 -ne 0 ']'
+ cat /tmp/apt-get-update.out
Hit:1 http://azure.archive.ubuntu.com/ubuntu xenial InRelease
Hit:2 http://azure.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:3 http://azure.archive.ubuntu.com/ubuntu xenial-backports InRelease
Hit:4 http://security.ubuntu.com/ubuntu xenial-security InRelease
Ign:5 http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04  InRelease
Hit:6 http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04  Release
Hit:7 https://packages.microsoft.com/ubuntu/16.04/prod xenial InRelease
Hit:9 https://aptdocker.azureedge.net/repo ubuntu-xenial InRelease
Get:10 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease [1,136 B]
Get:11 https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease [1,129 B]
Err:8 https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64  InRelease
  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
Err:10 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease
  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
Err:11 https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease
  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
Fetched 2,265 B in 1s (1,197 B/s)
Reading package lists...
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Some index files failed to download. They have been ignored, or old ones used instead.
+ '[' 3 -eq 10 ']'
+ sleep 30
+ for i in '$(seq 1 $retries)'
+ timeout 30 dpkg --configure -a
+ timeout 30 apt-get -f -y install
Reading package lists...
Building dependency tree...
Reading state information...
0 upgraded, 0 newly installed, 0 to remove and 6 not upgraded.
+ grep -E '^([WE]:.*)|([eE]rr.*)$'
+ tee /tmp/apt-get-update.out
+ timeout 120 apt-get update
Err:8 https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64  InRelease
Err:10 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease
Err:11 https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Some index files failed to download. They have been ignored, or old ones used instead.
+ '[' 0 -ne 0 ']'
+ cat /tmp/apt-get-update.out
Hit:1 http://azure.archive.ubuntu.com/ubuntu xenial InRelease
Hit:2 http://azure.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:3 http://azure.archive.ubuntu.com/ubuntu xenial-backports InRelease
Hit:4 http://security.ubuntu.com/ubuntu xenial-security InRelease
Ign:5 http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04  InRelease
Hit:6 http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04  Release
Hit:7 https://packages.microsoft.com/ubuntu/16.04/prod xenial InRelease
Hit:9 https://aptdocker.azureedge.net/repo ubuntu-xenial InRelease
Get:10 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease [1,136 B]
Get:11 https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease [1,129 B]
Err:8 https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64  InRelease
  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
Err:10 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease
  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
Err:11 https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease
  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
Fetched 2,265 B in 0s (3,202 B/s)
Reading package lists...
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Some index files failed to download. They have been ignored, or old ones used instead.
+ '[' 4 -eq 10 ']'
+ sleep 30
+ for i in '$(seq 1 $retries)'
+ timeout 30 dpkg --configure -a
+ timeout 30 apt-get -f -y install
Reading package lists...
Building dependency tree...
Reading state information...
0 upgraded, 0 newly installed, 0 to remove and 6 not upgraded.
+ grep -E '^([WE]:.*)|([eE]rr.*)$'
+ tee /tmp/apt-get-update.out
+ timeout 120 apt-get update
Err:9 https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64  InRelease
Err:10 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease
Err:11 https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Some index files failed to download. They have been ignored, or old ones used instead.
+ '[' 0 -ne 0 ']'
+ cat /tmp/apt-get-update.out
Hit:1 http://azure.archive.ubuntu.com/ubuntu xenial InRelease
Hit:2 http://azure.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:3 http://azure.archive.ubuntu.com/ubuntu xenial-backports InRelease
Hit:4 http://security.ubuntu.com/ubuntu xenial-security InRelease
Ign:5 http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04  InRelease
Hit:6 http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04  Release
Hit:7 https://packages.microsoft.com/ubuntu/16.04/prod xenial InRelease
Hit:8 https://aptdocker.azureedge.net/repo ubuntu-xenial InRelease
Get:10 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease [1,136 B]
Get:11 https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease [1,129 B]
Err:9 https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64  InRelease
  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
Err:10 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease
  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
Err:11 https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease
  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
Fetched 2,265 B in 0s (2,417 B/s)
Reading package lists...
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Some index files failed to download. They have been ignored, or old ones used instead.
+ '[' 5 -eq 10 ']'
+ sleep 30
+ for i in '$(seq 1 $retries)'
+ timeout 30 dpkg --configure -a
+ timeout 30 apt-get -f -y install
Reading package lists...
Building dependency tree...
Reading state information...
0 upgraded, 0 newly installed, 0 to remove and 6 not upgraded.
+ grep -E '^([WE]:.*)|([eE]rr.*)$'
+ tee /tmp/apt-get-update.out
+ timeout 120 apt-get update
Err:9 https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64  InRelease
Err:10 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease
Err:11 https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Some index files failed to download. They have been ignored, or old ones used instead.
+ '[' 0 -ne 0 ']'
+ cat /tmp/apt-get-update.out
Hit:1 http://azure.archive.ubuntu.com/ubuntu xenial InRelease
Hit:2 http://security.ubuntu.com/ubuntu xenial-security InRelease
Hit:3 http://azure.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:4 http://azure.archive.ubuntu.com/ubuntu xenial-backports InRelease
Ign:5 http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04  InRelease
Hit:6 http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04  Release
Hit:7 https://packages.microsoft.com/ubuntu/16.04/prod xenial InRelease
Hit:8 https://aptdocker.azureedge.net/repo ubuntu-xenial InRelease
Get:10 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease [1,136 B]
Get:11 https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease [1,129 B]
Err:9 https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64  InRelease
  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
Err:10 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease
  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
Err:11 https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease
  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
Fetched 2,265 B in 1s (2,207 B/s)
Reading package lists...
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Some index files failed to download. They have been ignored, or old ones used instead.
+ '[' 6 -eq 10 ']'
+ sleep 30
+ for i in '$(seq 1 $retries)'
+ timeout 30 dpkg --configure -a
+ timeout 30 apt-get -f -y install
Reading package lists...
Building dependency tree...
Reading state information...
0 upgraded, 0 newly installed, 0 to remove and 6 not upgraded.
+ grep -E '^([WE]:.*)|([eE]rr.*)$'
+ tee /tmp/apt-get-update.out
+ timeout 120 apt-get update
Err:8 https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64  InRelease
Err:10 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease
Err:11 https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Some index files failed to download. They have been ignored, or old ones used instead.
+ '[' 0 -ne 0 ']'
+ cat /tmp/apt-get-update.out
Hit:1 http://azure.archive.ubuntu.com/ubuntu xenial InRelease
Hit:2 http://azure.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:3 http://azure.archive.ubuntu.com/ubuntu xenial-backports InRelease
Ign:4 http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04  InRelease
Hit:5 http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04  Release
Hit:6 http://security.ubuntu.com/ubuntu xenial-security InRelease
Hit:7 https://packages.microsoft.com/ubuntu/16.04/prod xenial InRelease
Hit:9 https://aptdocker.azureedge.net/repo ubuntu-xenial InRelease
Get:10 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease [1,136 B]
Get:11 https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease [1,129 B]
Err:8 https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64  InRelease
  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
Err:10 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease
  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
Err:11 https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease
  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
Fetched 2,265 B in 1s (2,200 B/s)
Reading package lists...
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Some index files failed to download. They have been ignored, or old ones used instead.
+ '[' 7 -eq 10 ']'
+ sleep 30
+ for i in '$(seq 1 $retries)'
+ timeout 30 dpkg --configure -a
+ timeout 30 apt-get -f -y install
Reading package lists...
Building dependency tree...
Reading state information...
0 upgraded, 0 newly installed, 0 to remove and 6 not upgraded.
+ grep -E '^([WE]:.*)|([eE]rr.*)$'
+ tee /tmp/apt-get-update.out
+ timeout 120 apt-get update
Err:8 https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64  InRelease
Err:10 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease
Err:11 https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Some index files failed to download. They have been ignored, or old ones used instead.
+ '[' 0 -ne 0 ']'
+ cat /tmp/apt-get-update.out
Hit:1 http://azure.archive.ubuntu.com/ubuntu xenial InRelease
Hit:2 http://azure.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:3 http://azure.archive.ubuntu.com/ubuntu xenial-backports InRelease
Ign:4 http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04  InRelease
Hit:5 http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04  Release
Hit:6 http://security.ubuntu.com/ubuntu xenial-security InRelease
Hit:7 https://packages.microsoft.com/ubuntu/16.04/prod xenial InRelease
Hit:9 https://aptdocker.azureedge.net/repo ubuntu-xenial InRelease
Get:10 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease [1,136 B]
Get:11 https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease [1,129 B]
Err:8 https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64  InRelease
  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
Err:10 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease
  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
Err:11 https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease
  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
Fetched 2,265 B in 1s (2,107 B/s)
Reading package lists...
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Some index files failed to download. They have been ignored, or old ones used instead.
+ '[' 8 -eq 10 ']'
+ sleep 30
+ for i in '$(seq 1 $retries)'
+ timeout 30 dpkg --configure -a
+ timeout 30 apt-get -f -y install
Reading package lists...
Building dependency tree...
Reading state information...
0 upgraded, 0 newly installed, 0 to remove and 6 not upgraded.
+ grep -E '^([WE]:.*)|([eE]rr.*)$'
+ tee /tmp/apt-get-update.out
+ timeout 120 apt-get update
Err:9 https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64  InRelease
Err:10 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease
Err:11 https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Some index files failed to download. They have been ignored, or old ones used instead.
+ '[' 0 -ne 0 ']'
+ cat /tmp/apt-get-update.out
Hit:1 http://azure.archive.ubuntu.com/ubuntu xenial InRelease
Hit:2 http://azure.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:3 http://azure.archive.ubuntu.com/ubuntu xenial-backports InRelease
Get:4 http://security.ubuntu.com/ubuntu xenial-security InRelease [107 kB]
Ign:5 http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04  InRelease
Hit:6 http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04  Release
Hit:7 https://aptdocker.azureedge.net/repo ubuntu-xenial InRelease
Hit:8 https://packages.microsoft.com/ubuntu/16.04/prod xenial InRelease
Get:10 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease [1,136 B]
Get:11 https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease [1,129 B]
Err:9 https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64  InRelease
  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
Err:10 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease
  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
Err:11 https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease
  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
Fetched 109 kB in 1s (95.4 kB/s)
Reading package lists...
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Some index files failed to download. They have been ignored, or old ones used instead.
+ '[' 9 -eq 10 ']'
+ sleep 30
+ for i in '$(seq 1 $retries)'
+ timeout 30 dpkg --configure -a
+ timeout 30 apt-get -f -y install
Reading package lists...
Building dependency tree...
Reading state information...
0 upgraded, 0 newly installed, 0 to remove and 6 not upgraded.
+ grep -E '^([WE]:.*)|([eE]rr.*)$'
+ tee /tmp/apt-get-update.out
+ timeout 120 apt-get update
Err:8 https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64  InRelease
Err:9 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease
Err:10 https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Some index files failed to download. They have been ignored, or old ones used instead.
+ '[' 0 -ne 0 ']'
+ cat /tmp/apt-get-update.out
Hit:1 http://azure.archive.ubuntu.com/ubuntu xenial InRelease
Hit:2 http://azure.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:3 http://azure.archive.ubuntu.com/ubuntu xenial-backports InRelease
Get:4 http://security.ubuntu.com/ubuntu xenial-security InRelease [107 kB]
Ign:5 http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04  InRelease
Hit:6 https://aptdocker.azureedge.net/repo ubuntu-xenial InRelease
Hit:7 https://packages.microsoft.com/ubuntu/16.04/prod xenial InRelease
Get:9 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease [1,136 B]
Get:10 https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease [1,129 B]
Hit:11 http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04  Release
Err:8 https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64  InRelease
  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
Err:9 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease
  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
Err:10 https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease
  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
Fetched 109 kB in 2s (52.8 kB/s)
Reading package lists...
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Some index files failed to download. They have been ignored, or old ones used instead.
+ '[' 10 -eq 10 ']'
+ return 1
+ exit 99

What you expected to happen:

A new cluster should be built

How to reproduce it (as minimally and precisely as possible):

acs-engine-0.22.2 deploy --subscription-id ... --location northeurope --resource-group ... --api-model ./kubernetes.json --output-directory ./arm-templates_1.11.2 --debug

Anything else we need to know:

I logged in to the master and executed

sudo apt-get update

with the following result

Hit:1 http://azure.archive.ubuntu.com/ubuntu xenial InRelease
Hit:2 http://azure.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:3 http://azure.archive.ubuntu.com/ubuntu xenial-backports InRelease
Get:4 http://security.ubuntu.com/ubuntu xenial-security InRelease [107 kB]
Hit:5 https://packages.microsoft.com/ubuntu/16.04/prod xenial InRelease
Hit:7 https://aptdocker.azureedge.net/repo ubuntu-xenial InRelease                  
Get:8 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease [1136 B]
Get:9 https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease [1129 B]  
Ign:10 http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04  InRelease
Hit:11 http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/xUbuntu_16.04  Release
Err:6 https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64  InRelease
  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
Err:8 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease
  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
Err:9 https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease
  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
Fetched 109 kB in 0s (136 kB/s)
Reading package lists... Done
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64  InRelease: The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/libnvidia-container/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Failed to fetch https://nvidia.github.io/nvidia-docker/ubuntu16.04/amd64/InRelease  The following signatures were invalid: KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745  KEYEXPIRED 1538166745
W: Some index files failed to download. They have been ignored, or old ones used instead.

As can be seen from the above, something is wrong with the nvidia repository keys.
Please advice as I'm stuck at the moment.
@nikolajbrinch
Copy link
Author

nikolajbrinch commented Oct 2, 2018 •
edited
Loading

Ok here is a work around tested and verified:

Deploy using ARM templates (by using acs-engine generate first, and using az group deployment create), and pass an extra parameters file with the content (corresponding to the above kubernetes.json):

  "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "osImageOffer": {
      "value": "UbuntuServer"
    },
    "osImagePublisher": {
      "value": "Canonical"
    },
    "osImageSKU": {
      "value": "16.04-LTS"
    },
    "osImageVersion": {
      "value": "16.04.201809120"
    },
    "agentpool1osImageOffer": {
      "value": "UbuntuServer"
    },
    "agentpool1osImagePublisher": {
      "value": "Canonical"
    },
    "agentpool1osImageSKU": {
      "value": "16.04-LTS"
    },
    "agentpool1osImageVersion": {
      "value": "16.04.201809120"
    }
  }
}

This will use the latest 16.04-LTS image which seems to work. Currently acs-engine uses by default an old image: 16.04.201804050

From the ID it seems to be from 5th of April, which I think is rather old.
Do not use 18.04-LTS as the Microsoft package repository does not change so it will still be the package repository for 16.04??? And several packages fails to install.

@CecileRobertMichon
Copy link
Contributor

@nikolajbrinch thanks for reporting. What do you mean by "Currently acs-engine uses by default an old image: 16.04.201804050"? acs-engine in fact uses 16.04.201809120 as default https://github.com/Azure/acs-engine/blob/master/pkg/acsengine/defaults.go#L77. However, starting with acs-engine 0.22.0, we use a custom distro called "aks" to pre-install software, which is based on Ubuntu 16.04 "latest". I'd be interested to know if you have an indication that your cluster was built with a older version of the image.
An easier workaround for your case for now is to specify "distro": "ubuntu" in your master and agent profiles in your apimodel. This will ensure that the vanilla ubuntu image gets used while we figure out what is wrong. At the moment I suspect the issue is arising from the fact that you specified a newer Docker version in your apimodel.

@CecileRobertMichon CecileRobertMichon self-assigned this Oct 2, 2018
@grenzr
Copy link

grenzr commented Oct 2, 2018 •
edited
Loading

does the aks 0.16.0 image version resolve the key problem?
#3913

Unfortunately this isn't in an acs-engine release yet, but you could always build your own binary in the interim (or better yet, just hack the image version in parameters.json)

@CecileRobertMichon CecileRobertMichon mentioned this issue Oct 2, 2018
Closed
@CecileRobertMichon
Copy link
Contributor

@grenzr good call it most likely does as our tests are not seeing the same error (running from the master branch). Also, as @davidmatson pointed out in the expiration points to unit timestamp 2018-09-28T20:32:25+00:00. We will release with the new image soon, in the meantime you can try either a) building from master b) specifying "distro": "ubuntu" as I described above or c) manually change the osImageVersion to 0.16.0 in parameters

@grenzr
Copy link

grenzr commented Oct 2, 2018

confirmed - using distro: 'ubuntu' in masterProfile and agentProfile gets you a working cluster.

@davidmatson
Copy link

Also confirming that I was able to get a working cluster by switching osImageVersion to 0.16.0 in azuredeploy.parameters.json. (Just had to switch from acs-engine deploy to acs-engine generate to have the option to tweak this value.)

@CecileRobertMichon
Copy link
Contributor

Great thanks @davidmatson and @grenzr. Patch release is on its way.

@jackfrancis
Copy link
Member

Fixed here: https://github.com/Azure/acs-engine/releases/tag/v0.22.3

@nikolajbrinch
Copy link
Author

@CecileRobertMichon I mean using acs-engine 0.22.3

$ bin/acs-engine-0.22.2 version
Version: v0.22.2
GitCommit: 03f15953f
GitTreeState: clean

and generating templates, the following is in my azuredeploy.json

    "osImageVersion": {
      "defaultValue": "16.04.201804050",
      "metadata": {
        "description": "OS image version."
      },
      "type": "string"
    },

and I was of the impression that azuredeploy.json states the defaulValues for the current acs-engine version. That would be nice though, so I wouldn't have to go to the code...
Even better if this was documented somewhere.

@nikolajbrinch
Copy link
Author

@CecileRobertMichon on the docker version thing. The reason for actually deploying this cluster again, was that I crashed the API server, and decided to just rebuild the cluster (it is a test cluster).
So the whole build process was a repeat of an earlier built - also using docker 17.05.
So it was running for days, until I wiped the Resource Group and ran my script again.

BTW: The Troubleshooting documentation helped me figure out the problem, so many thanks for writing that up.

@CecileRobertMichon
Copy link
Contributor

@nikolajbrinch my initial intuition about the newer docker version was wrong, the problem was definitely that the nvidia key used in the 0.15.0 VHD image was expired. The only way that your docker version was affecting that is that because that version is not preinstalled on our AKS curated image it was being pulled at deployment which included running an apt-get update and that was causing the nvidia key error.

Thanks for the feedback on the default value in the template, that version is actually not being used but I will make sure it's removed or updated to prevent confusion.

@nikolajbrinch
Copy link
Author

@CecileRobertMichon Thank you for clarifying everything and making a quick fix. Tested with a small cluster provisioning and it works fine now (0.22.3).

We use the newer docker version, as it support multistage builds, and other newer stuff. Our Jenkins servers run on this and does the images builds, and we like using multistage builds and setting ownership and group during copy :-)

@CecileRobertMichon CecileRobertMichon mentioned this issue Oct 3, 2018
Merged
3 tasks
@d-demirci
Copy link

This is already closed, but another way maybe to update the keys as told here https://nvidia.github.io/nvidia-docker/

@nikolajbrinch
Copy link
Author

@d-demirci Yes that would be ideal, but that is not up to the user, but up to the maintainer. The CSE is failing at provisioning, nothing I can do, as far as I know?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@CecileRobertMichon CecileRobertMichon

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@grenzr @jackfrancis @davidmatson @nikolajbrinch @CecileRobertMichon @d-demirci