[BUG] ssh runner times out #3334
Comments
|
Nested virtualization setups aren't supported. Note: @code-ready/crc-team we could add a timeout override that will just ignore these values until the user decides to break out. |
gbraad
added
nested-virt
|
We have discussed to ignore or extend the ssh-timeout, but this is not a common issue. |
|
@LucienBrule Can you share |
|
I am having more or less the same problem during starting crc. crc start -c 6 -m 11500 -p pull-secret.txtError: INFO Checking if running as non-root
INFO Checking if running inside WSL2
INFO Checking if crc-admin-helper executable is cached
INFO Checking for obsolete admin-helper executable
INFO Checking if running on a supported CPU architecture
INFO Checking minimum RAM requirements
INFO Checking if crc executable symlink exists
INFO Checking if Virtualization is enabled
INFO Checking if KVM is enabled
INFO Checking if libvirt is installed
INFO Checking if user is part of libvirt group
INFO Checking if active user/process is currently part of the libvirt group
INFO Checking if libvirt daemon is running
INFO Checking if a supported libvirt version is installed
INFO Checking if crc-driver-libvirt is installed
INFO Checking crc daemon systemd socket units
INFO Checking if systemd-networkd is running
INFO Checking if NetworkManager is installed
INFO Checking if NetworkManager service is running
INFO Checking if /etc/NetworkManager/conf.d/crc-nm-dnsmasq.conf exists
INFO Checking if /etc/NetworkManager/dnsmasq.d/crc.conf exists
INFO Checking if libvirt 'crc' network is available
INFO Checking if libvirt 'crc' network is active
INFO Loading bundle: crc_libvirt_4.11.1_amd64...
INFO Creating CRC VM for openshift 4.11.1...
INFO Generating new SSH key pair...
INFO Generating new password for the kubeadmin user
INFO Starting CRC VM for openshift 4.11.1...
INFO CRC instance is running with IP 192.168.130.11
INFO CRC VM is running
INFO Updating authorized keys...
INFO Configuring shared directories
INFO Check internal and public DNS query...
INFO Check DNS query from host...
INFO Verifying validity of the kubelet certificates...
INFO Starting kubelet service
INFO Waiting for kube-apiserver availability... [takes around 2min]
Temporary error: ssh command error:
command : timeout 5s oc get cm --context admin --cluster crc --kubeconfig /opt/kubeconfig
err : Process exited with status 124
(x13)General information
CRC versionCRC version: 2.8.0+217b3bd
OpenShift version: 4.11.1
Podman version: 4.1.1CRC statusDEBU CRC version: 2.8.0+217b3bd
DEBU OpenShift version: 4.11.1
DEBU Podman version: 4.1.1
DEBU Running 'crc status'
DEBU Checking file: /home/student/.crc/machines/crc/.crc-exist
DEBU Checking file: /home/student/.crc/machines/crc/.crc-exist
DEBU Found binary path at /home/student/.crc/bin/crc-driver-libvirt
DEBU Launching plugin server for driver libvirt
DEBU Plugin server listening at address 127.0.0.1:42519
DEBU () Calling .GetVersion
DEBU Using API Version 1
DEBU () Calling .SetConfigRaw
DEBU () Calling .GetMachineName
DEBU (crc) Calling .GetBundleName
DEBU (crc) Calling .GetState
DEBU (crc) DBG | time="2022-09-10T08:40:07+02:00" level=debug msg="Getting current state..."
DEBU (crc) DBG | time="2022-09-10T08:40:07+02:00" level=debug msg="Fetching VM..."
DEBU (crc) Calling .GetIP
DEBU (crc) DBG | time="2022-09-10T08:40:07+02:00" level=debug msg="GetIP called for crc"
DEBU (crc) DBG | time="2022-09-10T08:40:07+02:00" level=debug msg="Getting current state..."
DEBU (crc) DBG | time="2022-09-10T08:40:07+02:00" level=debug msg="IP address: 192.168.130.11"
DEBU (crc) Calling .GetIP
DEBU (crc) DBG | time="2022-09-10T08:40:07+02:00" level=debug msg="GetIP called for crc"
DEBU (crc) DBG | time="2022-09-10T08:40:07+02:00" level=debug msg="Getting current state..."
DEBU Running SSH command: df -B1 --output=size,used,target /sysroot | tail -1
DEBU (crc) DBG | time="2022-09-10T08:40:07+02:00" level=debug msg="IP address: 192.168.130.11"
DEBU Using ssh private keys: [/home/student/.crc/machines/crc/id_ecdsa /home/student/.crc/cache/crc_libvirt_4.11.1_amd64/id_ecdsa_crc]
DEBU SSH command results: err: <nil>, output: 32737570816 12010016768 /sysroot
DEBU cannot get OpenShift status: stat /home/student/.crc/machines/crc/kubeconfig: no such file or directory
DEBU Making call to close driver server
DEBU (crc) Calling .Close
DEBU (crc) DBG | time="2022-09-10T08:40:33+02:00" level=debug msg="Closing plugin on server side"
DEBU Successfully made call to close driver server
DEBU Making call to close connection to plugin binary
CRC VM: Running
OpenShift: Unreachable (v4.11.1)
Podman:
Disk Usage: 12.01GB of 32.74GB (Inside the CRC VM)
Cache Usage: 17.04GB
Cache Directory: /home/student/.crc/cacheCRC config- consent-telemetry : noHost Operating SystemNAME="Red Hat Enterprise Linux"
VERSION="8.6 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.6"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.6" |
|
@praveenkumar journalctl output Also here's the output of Take 1Highlights
From the recording above, the crc start command got stuck waiting for the machine to come up under libvirt (ending at at 3:20). It got to retry attempt 60/60, called GetConfigRaw, re checked if the machine was up and in that check it was up. So now the instance is assumed to be up because it has an ip address assigned to it. It then attempts to ssh into the machine, entering a retry loop. I exited the crc start command at 12:53, and eventually terminated the machine. There's no recording of this, but here's what I did: because crc stop hung too. Take 2Highlights
Work AroundMy current work around is:
@gbraad I think you can get a sense of where the boot is hanging from the asciinema. I understand nested virtualization is not a supported use case, however it does appear to work when the timeouts are increased. It would be cool if the timeouts could be set via config though. |
|
@LucienBrule Thank you for providing all the details, we will discuss it in coming week and try to put a plan in place. |
General information
crc setupbefore starting it (Yes/No)? YesCRC version
CRC status
CRC config
Host Operating System
Steps to reproduce
Expected
crc start doesn't time out.
Actual
crc start hangs when attempting to get ssh access to the VM.
The timeout is only 300 seconds as per https://github.com/code-ready/crc/blob/main/pkg/crc/machine/start.go#L335
Solution
I pulled the code and made the timeout 10000 seconds. After waiting about 15 minutes, it came up and finished installing successfully.
Etc
So I monitored the crc vm with cockpit, when booting it hangs for about 10 minutes at 'Booting RHEL FCOS' (the first line before it goes into console spam mode/systemd). Of interest, other VMs run under KVM don't have this long boot time problem, including the stock FCOS image. So my fix of increasing the timeout works for now but perhaps there's something that can be done about the boot time.
The text was updated successfully, but these errors were encountered: