Running colima on GitLab macOS runners

[mitar]

Description

I am attempting to run colima on GitLab macOS CI runners and I simply does not start. But the issue is that I cannot find any information about why it does not start. So I am opening this issue for two reasons:

• To ask that colima provides better logging information (or maybe just point me on where to find that) so that it would be clearer to me why it is not running. Is this something I can fix or something which is the issue with GitLab runners?
• To have an issue for others to comment about their experience running colima on GitLab macOS runners.

My goal is to be able to run arm64 Docker images inside macOS.

My GitLab CI configuration is as follows:

test_mac:
  tags:
    - saas-macos-medium-m1

  image: macos-13-xcode-14

  variables:
    HOMEBREW_NO_AUTO_UPDATE: 1
    HOMEBREW_NO_ENV_HINTS: 1
    HOMEBREW_NO_INSTALL_CLEANUP: 1

  before_script:
    - brew update
    - brew install colima docker
    - softwareupdate --install-rosetta --agree-to-license
    - colima start --vm-type vz --arch aarch64 --vz-rosetta --very-verbose --cpu 4 --memory 4 --disk 15 || true
    - cat /Users/gitlab/.lima/colima/serial*.log
    - cat /Users/gitlab/.lima/colima/ha.stderr.log
    - docker info

  script:
    - "true"

``` $ brew install colima docker Warning: Treating docker as a formula. For the cask, use homebrew/cask/docker ==> Fetching dependencies for colima: capstone, dtc, glib, ca-certificates, openssl@3, gnutls, jpeg-turbo, libpng, libslirp, libssh, lzo, pixman, snappy, vde, xz, qemu and lima ==> Fetching capstone ==> Downloading https://ghcr.io/v2/homebrew/core/capstone/manifests/4.0.2 ==> Downloading https://ghcr.io/v2/homebrew/core/capstone/blobs/sha256:4a7f3c31862791da2c3e9436880a84953ad61a7764fbac208fb6b6999ee139dc ==> Fetching dtc ==> Downloading https://ghcr.io/v2/homebrew/core/dtc/manifests/1.7.0 ==> Downloading https://ghcr.io/v2/homebrew/core/dtc/blobs/sha256:9ca326b92b46108692e2f27bf20e83877bf772650f0e6912be5ce3934df284a5 ==> Fetching glib ==> Downloading https://ghcr.io/v2/homebrew/core/glib/manifests/2.76.4 ==> Downloading https://ghcr.io/v2/homebrew/core/glib/blobs/sha256:e5995be18147cbc766ee495c2d142f79b80cc18ac95c8eb3bce6613bcad88fe3 ==> Fetching ca-certificates ==> Downloading https://ghcr.io/v2/homebrew/core/ca-certificates/manifests/2023-08-22 ==> Downloading https://ghcr.io/v2/homebrew/core/ca-certificates/blobs/sha256:a331e92e7a759571296581f029e5cc2ec7cee70cd92dc0b5f8eb76095f94a21a ==> Fetching openssl@3 ==> Downloading https://ghcr.io/v2/homebrew/core/openssl/3/manifests/3.1.2-1 ==> Downloading https://ghcr.io/v2/homebrew/core/openssl/3/blobs/sha256:904a07d94bd731815d2db4c92abec9b90132cb9b0fb1c83a2906c364f9665b04 ==> Fetching gnutls ==> Downloading https://ghcr.io/v2/homebrew/core/gnutls/manifests/3.8.1 ==> Downloading https://ghcr.io/v2/homebrew/core/gnutls/blobs/sha256:e3c3e0156bee79b8af745e9d4587ccca48afe3a50a25071bfb185b3cfe41a215 ==> Fetching jpeg-turbo ==> Downloading https://ghcr.io/v2/homebrew/core/jpeg-turbo/manifests/3.0.0 ==> Downloading https://ghcr.io/v2/homebrew/core/jpeg-turbo/blobs/sha256:1971c1fa66c2580fa0bfafe5350c6170bfe7395a4e503e7bfe0c69ec2e353010 ==> Fetching libpng ==> Downloading https://ghcr.io/v2/homebrew/core/libpng/manifests/1.6.40 ==> Downloading https://ghcr.io/v2/homebrew/core/libpng/blobs/sha256:c309cf133ab08f4fd25210da897eaaff2603e9a7e1bdc178821c7e186fb9ee69 ==> Fetching libslirp ==> Downloading https://ghcr.io/v2/homebrew/core/libslirp/manifests/4.7.0 ==> Downloading https://ghcr.io/v2/homebrew/core/libslirp/blobs/sha256:3e461de89cdcc48cd88ed8065700575cc3d4921213f2023cce1c1c56853c9117 ==> Fetching libssh ==> Downloading https://ghcr.io/v2/homebrew/core/libssh/manifests/0.10.5_1 ==> Downloading https://ghcr.io/v2/homebrew/core/libssh/blobs/sha256:5b5925ae5e0d31dd7b2c508eb1f6a19585a87cb66310d5a658e772666500191d ==> Fetching lzo ==> Downloading https://ghcr.io/v2/homebrew/core/lzo/manifests/2.10 ==> Downloading https://ghcr.io/v2/homebrew/core/lzo/blobs/sha256:a565c627b13f2dc7fc4550aa8290a4c3feb2f48fcaa45c9f7f4bc4fe4535aa66 ==> Fetching pixman ==> Downloading https://ghcr.io/v2/homebrew/core/pixman/manifests/0.42.2-1 ==> Downloading https://ghcr.io/v2/homebrew/core/pixman/blobs/sha256:e27867c503bd9cf858159261e053184d19ae00357dc89426810f80734aaaefd0 ==> Fetching snappy ==> Downloading https://ghcr.io/v2/homebrew/core/snappy/manifests/1.1.10 ==> Downloading https://ghcr.io/v2/homebrew/core/snappy/blobs/sha256:ca95915a51bed09a5e70ebb6f253eabe4df5b00e87ebe49aea0124f8bb51bc3c ==> Fetching vde ==> Downloading https://ghcr.io/v2/homebrew/core/vde/manifests/2.3.3 ==> Downloading https://ghcr.io/v2/homebrew/core/vde/blobs/sha256:982a56825cbd1bd374001330e3492e83a1d82825ed7228d33705ffad3b927e8e ==> Fetching xz ==> Downloading https://ghcr.io/v2/homebrew/core/xz/manifests/5.4.4 ==> Downloading https://ghcr.io/v2/homebrew/core/xz/blobs/sha256:ba318d89eea54f33cc3613b1cb69ca4217a8f961e59026418e569e8421afbb8c ==> Fetching qemu ==> Downloading https://ghcr.io/v2/homebrew/core/qemu/manifests/8.1.0_1-1 ==> Downloading https://ghcr.io/v2/homebrew/core/qemu/blobs/sha256:9b96017b8a7d8235275108a23224a388cb8b9a7276c76dcf876dc11eae372562 ==> Fetching lima ==> Downloading https://ghcr.io/v2/homebrew/core/lima/manifests/0.17.2 ==> Downloading https://ghcr.io/v2/homebrew/core/lima/blobs/sha256:b769e329437f18161ac7074812ffcf58c0c5058db0896385888f31fb221b093e ==> Fetching colima ==> Downloading https://ghcr.io/v2/homebrew/core/colima/manifests/0.5.5 ==> Downloading https://ghcr.io/v2/homebrew/core/colima/blobs/sha256:f6934ad6a852b6dfef89d1c70f1054f57bb8d73829273d2bf2a81b54d40b209c ==> Fetching dependencies for docker: docker-completion ==> Fetching docker-completion ==> Downloading https://ghcr.io/v2/homebrew/core/docker-completion/manifests/24.0.5 ==> Downloading https://ghcr.io/v2/homebrew/core/docker-completion/blobs/sha256:b77b6d4980dd2c9345cf64b9a790c0ac177358195d19202a06112b19b1533a1d ==> Fetching docker ==> Downloading https://ghcr.io/v2/homebrew/core/docker/manifests/24.0.5 ==> Downloading https://ghcr.io/v2/homebrew/core/docker/blobs/sha256:535cbf2123170068ee315fdf5e8ceeacedc22e9a89cd17d45d2194b07dce3ea0 ==> Installing dependencies for colima: capstone, dtc, glib, ca-certificates, openssl@3, gnutls, jpeg-turbo, libpng, libslirp, libssh, lzo, pixman, snappy, vde, xz, qemu and lima ==> Installing colima dependency: capstone ==> Pouring capstone--4.0.2.arm64_ventura.bottle.tar.gz 🍺 /opt/homebrew/Cellar/capstone/4.0.2: 25 files, 14.7MB ==> Installing colima dependency: dtc ==> Pouring dtc--1.7.0.arm64_ventura.bottle.tar.gz 🍺 /opt/homebrew/Cellar/dtc/1.7.0: 18 files, 619.2KB ==> Installing colima dependency: glib ==> Pouring glib--2.76.4.arm64_ventura.bottle.tar.gz 🍺 /opt/homebrew/Cellar/glib/2.76.4: 455 files, 22.2MB ==> Installing colima dependency: ca-certificates ==> Pouring ca-certificates--2023-08-22.arm64_ventura.bottle.tar.gz ==> Regenerating CA certificate bundle from keychain, this may take a while... 🍺 /opt/homebrew/Cellar/ca-certificates/2023-08-22: 3 files, 221.7KB ==> Installing colima dependency: openssl@3 ==> Pouring openssl@3--3.1.2.arm64_ventura.bottle.1.tar.gz 🍺 /opt/homebrew/Cellar/openssl@3/3.1.2: 6,495 files, 28.4MB ==> Installing colima dependency: gnutls ==> Pouring gnutls--3.8.1.arm64_ventura.bottle.tar.gz 🍺 /opt/homebrew/Cellar/gnutls/3.8.1: 1,284 files, 10.7MB ==> Installing colima dependency: jpeg-turbo ==> Pouring jpeg-turbo--3.0.0.arm64_ventura.bottle.tar.gz 🍺 /opt/homebrew/Cellar/jpeg-turbo/3.0.0: 44 files, 3.4MB ==> Installing colima dependency: libpng ==> Pouring libpng--1.6.40.arm64_ventura.bottle.tar.gz 🍺 /opt/homebrew/Cellar/libpng/1.6.40: 27 files, 1.3MB ==> Installing colima dependency: libslirp ==> Pouring libslirp--4.7.0.arm64_ventura.bottle.tar.gz 🍺 /opt/homebrew/Cellar/libslirp/4.7.0: 11 files, 385.1KB ==> Installing colima dependency: libssh ==> Pouring libssh--0.10.5_1.arm64_ventura.bottle.tar.gz 🍺 /opt/homebrew/Cellar/libssh/0.10.5_1: 23 files, 1.3MB ==> Installing colima dependency: lzo ==> Pouring lzo--2.10.arm64_ventura.bottle.tar.gz 🍺 /opt/homebrew/Cellar/lzo/2.10: 31 files, 566.3KB ==> Installing colima dependency: pixman ==> Pouring pixman--0.42.2.arm64_ventura.bottle.1.tar.gz 🍺 /opt/homebrew/Cellar/pixman/0.42.2: 11 files, 1.2MB ==> Installing colima dependency: snappy ==> Pouring snappy--1.1.10.arm64_ventura.bottle.tar.gz 🍺 /opt/homebrew/Cellar/snappy/1.1.10: 18 files, 164.3KB ==> Installing colima dependency: vde ==> Pouring vde--2.3.3.arm64_ventura.bottle.tar.gz 🍺 /opt/homebrew/Cellar/vde/2.3.3: 66 files, 1.3MB ==> Installing colima dependency: xz ==> Pouring xz--5.4.4.arm64_ventura.bottle.tar.gz 🍺 /opt/homebrew/Cellar/xz/5.4.4: 163 files, 2.6MB ==> Installing colima dependency: qemu ==> Pouring qemu--8.1.0_1.arm64_ventura.bottle.1.tar.gz 🍺 /opt/homebrew/Cellar/qemu/8.1.0_1: 162 files, 533.9MB ==> Installing colima dependency: lima ==> Pouring lima--0.17.2.arm64_ventura.bottle.tar.gz 🍺 /opt/homebrew/Cellar/lima/0.17.2: 111 files, 167.4MB ==> Installing colima ==> Pouring colima--0.5.5.arm64_ventura.bottle.tar.gz ==> Caveats zsh completions have been installed to: /opt/homebrew/share/zsh/site-functions ==> Summary 🍺 /opt/homebrew/Cellar/colima/0.5.5: 8 files, 14.5MB ==> Installing dependencies for docker: docker-completion ==> Installing docker dependency: docker-completion ==> Pouring docker-completion--24.0.5.arm64_ventura.bottle.tar.gz 🍺 /opt/homebrew/Cellar/docker-completion/24.0.5: 9 files, 337.8KB ==> Installing docker ==> Pouring docker--24.0.5.arm64_ventura.bottle.tar.gz 🍺 /opt/homebrew/Cellar/docker/24.0.5: 13 files, 31.6MB ==> Upgrading 4 dependents of upgraded formulae: awscli 2.13.2 -> 2.13.13, openldap 2.6.5 -> 2.6.6, curl 8.2.0 -> 8.2.1, python@3.11 3.11.4_1 -> 3.11.5 ==> Fetching openldap ==> Downloading https://ghcr.io/v2/homebrew/core/openldap/manifests/2.6.6 ==> Downloading https://ghcr.io/v2/homebrew/core/openldap/blobs/sha256:29426f6dd15964b20f8561f9f0a7a332a80086a0b480ac739719213c76f2d5e9 ==> Fetching curl ==> Downloading https://ghcr.io/v2/homebrew/core/curl/manifests/8.2.1 ==> Downloading https://ghcr.io/v2/homebrew/core/curl/blobs/sha256:901965403320a6aacdac96e06bc5f140db827565ba7c2cc7bc1e4337756fda2e ==> Fetching dependencies for python@3.11: sqlite ==> Fetching sqlite ==> Downloading https://ghcr.io/v2/homebrew/core/sqlite/manifests/3.43.0 ==> Downloading https://ghcr.io/v2/homebrew/core/sqlite/blobs/sha256:e6a7d708ca874fc883696409b8319d9ed9376ebfbe911739b50a202d3392fdad ==> Fetching python@3.11 ==> Downloading https://ghcr.io/v2/homebrew/core/python/3.11/manifests/3.11.5 ==> Downloading https://ghcr.io/v2/homebrew/core/python/3.11/blobs/sha256:55f29263615ffcb1a9097c71fd3ce42647d4f40f6eeecda1317f43cd2a3fcef5 ==> Fetching awscli ==> Downloading https://ghcr.io/v2/homebrew/core/awscli/manifests/2.13.13 ==> Downloading https://ghcr.io/v2/homebrew/core/awscli/blobs/sha256:f09021f9c2cc8bc0819ff871e3c60c3b1443cc4304e757b00c1a6625b7a2a7b4 ==> Upgrading openldap 2.6.5 -> 2.6.6 ==> Pouring openldap--2.6.6.arm64_ventura.bottle.tar.gz 🍺 /opt/homebrew/Cellar/openldap/2.6.6: 83 files, 6.9MB ==> Upgrading curl 8.2.0 -> 8.2.1 ==> Pouring curl--8.2.1.arm64_ventura.bottle.tar.gz ==> Caveats curl is keg-only, which means it was not symlinked into /opt/homebrew, because macOS already provides this software and installing another version in parallel can cause all kinds of trouble. If you need to have curl first in your PATH, run: echo 'export PATH="/opt/homebrew/opt/curl/bin:$PATH"' >> ~/.zshrc For compilers to find curl you may need to set: export LDFLAGS="-L/opt/homebrew/opt/curl/lib" export CPPFLAGS="-I/opt/homebrew/opt/curl/include" For pkg-config to find curl you may need to set: export PKG_CONFIG_PATH="/opt/homebrew/opt/curl/lib/pkgconfig" zsh completions have been installed to: /opt/homebrew/opt/curl/share/zsh/site-functions ==> Summary 🍺 /opt/homebrew/Cellar/curl/8.2.1: 514 files, 4.3MB ==> Upgrading python@3.11 3.11.4_1 -> 3.11.5 ==> Installing dependencies for python@3.11: sqlite ==> Installing python@3.11 dependency: sqlite ==> Pouring sqlite--3.43.0.arm64_ventura.bottle.tar.gz 🍺 /opt/homebrew/Cellar/sqlite/3.43.0: 11 files, 4.6MB ==> Installing python@3.11 ==> Pouring python@3.11--3.11.5.arm64_ventura.bottle.tar.gz ==> /opt/homebrew/Cellar/python@3.11/3.11.5/bin/python3.11 -Im ensurepip ==> /opt/homebrew/Cellar/python@3.11/3.11.5/bin/python3.11 -Im pip install -v -- 🍺 /opt/homebrew/Cellar/python@3.11/3.11.5: 3,289 files, 61.8MB ==> Upgrading awscli 2.13.2 -> 2.13.13 ==> Pouring awscli--2.13.13.arm64_ventura.bottle.tar.gz ==> Caveats The "examples" directory has been installed to: /opt/homebrew/share/awscli/examples zsh completions and functions have been installed to: /opt/homebrew/share/zsh/site-functions ==> Summary 🍺 /opt/homebrew/Cellar/awscli/2.13.13: 12,918 files, 110.8MB ==> Checking for dependents of upgraded formulae... ==> No broken dependents found! ==> Caveats ==> colima zsh completions have been installed to: /opt/homebrew/share/zsh/site-functions ==> curl curl is keg-only, which means it was not symlinked into /opt/homebrew, because macOS already provides this software and installing another version in parallel can cause all kinds of trouble. If you need to have curl first in your PATH, run: echo 'export PATH="/opt/homebrew/opt/curl/bin:$PATH"' >> ~/.zshrc For compilers to find curl you may need to set: export LDFLAGS="-L/opt/homebrew/opt/curl/lib" export CPPFLAGS="-I/opt/homebrew/opt/curl/include" For pkg-config to find curl you may need to set: export PKG_CONFIG_PATH="/opt/homebrew/opt/curl/lib/pkgconfig" zsh completions have been installed to: /opt/homebrew/opt/curl/share/zsh/site-functions ==> awscli The "examples" directory has been installed to: /opt/homebrew/share/awscli/examples zsh completions and functions have been installed to: /opt/homebrew/share/zsh/site-functions $ softwareupdate --install-rosetta --agree-to-license 2023-08-29 10:36:55.458 softwareupdate[5696:13831] Package Authoring Error: 042-01896: Package reference com.apple.pkg.RosettaUpdateAuto is missing installKBytes attribute By using the agreetolicense option, you are agreeing that you have run this tool with the license only option and have read and agreed to the terms. If you do not agree, press CTRL-C and cancel this process immediately. Install of Rosetta 2 finished successfully ```

$ colima start --vm-type vz --arch aarch64 --vz-rosetta --very-verbose --cpu 4 --memory 4 --disk 15 || true
time="2023-08-29T10:36:57Z" level=trace msg="cmd [\"limactl\" \"info\"]"
time="2023-08-29T10:36:57Z" level=trace msg="cmd [\"limactl\" \"list\" \"colima\" \"--json\"]"
time="2023-08-29T10:36:57Z" level=trace msg="error retrieving running instance: instance 'colima' does not exist"
time="2023-08-29T10:36:57Z" level=info msg="starting colima"
time="2023-08-29T10:36:57Z" level=info msg="runtime: docker"
time="2023-08-29T10:36:57Z" level=info msg="creating and starting ..." context=vm
time="2023-08-29T10:36:57Z" level=trace msg="cmd [\"pgrep\" \"oahd\"]"
time="2023-08-29T10:36:57Z" level=info msg="Terminal is not available, proceeding without opening an editor"
time="2023-08-29T10:36:57Z" level=trace msg="cmd [\"limactl\" \"start\" \"--tty=false\" \"/var/folders/lr/kycxps496119mlqd5wf0wcr40000gn/T/colima.yaml\"]"
time="2023-08-29T10:36:57Z" level=warning msg="`vmType: vz` is experimental"
time="2023-08-29T10:36:57Z" level=warning msg="Ignoring: vmType vz: [OS]"
time="2023-08-29T10:36:57Z" level=info msg="Attempting to download the image" arch=aarch64 digest="sha512:84c93e8aaa09446618bf87daa993e260da69b50e95670aed5df6671b2cff9464810752cbf70f6ee5ddf9d3e1c91d98104b3c573cc024c5f0687ad3f4d2e93ebc" location="https://github.com/abiosoft/alpine-lima/releases/download/colima-v0.5.5/alpine-lima-clm-3.18.0-aarch64.iso"
Downloading the image (alpine-lima-clm-3.18.0-aarch64.iso)
297.37 MiB / 297.37 MiB (100.00%) ? p/stime="2023-08-29T10:37:02Z" level=info msg="Downloaded the image from \"https://github.com/abiosoft/alpine-lima/releases/download/colima-v0.5.5/alpine-lima-clm-3.18.0-aarch64.iso\""
time="2023-08-29T10:37:05Z" level=info msg="[hostagent] Starting VZ (hint: to watch the boot progress, see \"/Users/gitlab/.lima/colima/serial*.log\")"
time="2023-08-29T10:37:05Z" level=info msg="[hostagent] new connection from  to "
time="2023-08-29T10:37:05Z" level=info msg="[hostagent] Setting up Rosetta share"
time="2023-08-29T10:37:05Z" level=fatal msg="exiting, status={Running:false Degraded:false Exiting:true Errors:[] SSHLocalPort:0} (hint: see \"/Users/gitlab/.lima/colima/ha.stderr.log\")"
$ cat /Users/gitlab/.lima/colima/serial*.log
$ cat /Users/gitlab/.lima/colima/ha.stderr.log
{"level":"debug","msg":"executing [ssh-keygen -t ed25519 -q -N  -f /Users/gitlab/.lima/_config/user]","time":"2023-08-29T10:37:02Z"}
{"level":"debug","msg":"Creating iso file /Users/gitlab/.lima/colima/cidata.iso","time":"2023-08-29T10:37:04Z"}
{"level":"debug","msg":"Using /var/folders/lr/kycxps496119mlqd5wf0wcr40000gn/T/diskfs_iso370350914 as workspace","time":"2023-08-29T10:37:04Z"}
{"level":"debug","msg":"Failed to detect CPU features. Assuming that AES acceleration is available on this Apple silicon.","time":"2023-08-29T10:37:04Z"}
{"level":"debug","msg":"OpenSSH version 9.0.1 detected","time":"2023-08-29T10:37:04Z"}
{"level":"debug","msg":"AES accelerator seems available, prioritizing aes128-gcm@openssh.com and aes256-gcm@openssh.com","time":"2023-08-29T10:37:05Z"}
{"level":"info","msg":"Starting VZ (hint: to watch the boot progress, see \"/Users/gitlab/.lima/colima/serial*.log\")","time":"2023-08-29T10:37:05Z"}
{"level":"debug","msg":"Start tcp DNS listening on: 127.0.0.1:49269","time":"2023-08-29T10:37:05Z"}
{"level":"debug","msg":"Using search domains: [ec2.internal]","time":"2023-08-29T10:37:05Z"}
{"level":"debug","msg":"Start udp DNS listening on: 127.0.0.1:51476","time":"2023-08-29T10:37:05Z"}
{"level":"info","msg":"new connection from  to ","time":"2023-08-29T10:37:05Z"}
{"level":"info","msg":"Setting up Rosetta share","time":"2023-08-29T10:37:05Z"}
time="2023-08-29T10:37:05Z" level=fatal msg="error starting vm: error at 'creating and starting': exit status 1"

My current hypothesis is that GitLab runs macOS runners inside VM and that prevents another VM to run inside it (that nested VMs are not supported on M1). But it would be great if this would be somehow reported by colima so that next me does not spend a day debugging this.

I think this hypothesis is confirmed when I try QEMU:

``` $ brew install colima docker Warning: Treating docker as a formula. For the cask, use homebrew/cask/docker ==> Fetching dependencies for colima: capstone, glib, jpeg-turbo, libpng, libslirp, libssh, lzo, pixman, snappy, vde, qemu and lima ==> Fetching capstone ==> Downloading https://ghcr.io/v2/homebrew/core/capstone/manifests/4.0.2 ==> Downloading https://ghcr.io/v2/homebrew/core/capstone/blobs/sha256:4a7f3c31862791da2c3e9436880a84953ad61a7764fbac208fb6b6999ee139dc ==> Fetching glib ==> Downloading https://ghcr.io/v2/homebrew/core/glib/manifests/2.76.4 ==> Downloading https://ghcr.io/v2/homebrew/core/glib/blobs/sha256:e5995be18147cbc766ee495c2d142f79b80cc18ac95c8eb3bce6613bcad88fe3 ==> Fetching jpeg-turbo ==> Downloading https://ghcr.io/v2/homebrew/core/jpeg-turbo/manifests/2.1.5.1 ==> Downloading https://ghcr.io/v2/homebrew/core/jpeg-turbo/blobs/sha256:24657182233e7669e645d675a7a954f277510c9de37fdbeedbeee777246cdaea ==> Fetching libpng ==> Downloading https://ghcr.io/v2/homebrew/core/libpng/manifests/1.6.40 ==> Downloading https://ghcr.io/v2/homebrew/core/libpng/blobs/sha256:c309cf133ab08f4fd25210da897eaaff2603e9a7e1bdc178821c7e186fb9ee69 ==> Fetching libslirp ==> Downloading https://ghcr.io/v2/homebrew/core/libslirp/manifests/4.7.0 ==> Downloading https://ghcr.io/v2/homebrew/core/libslirp/blobs/sha256:3e461de89cdcc48cd88ed8065700575cc3d4921213f2023cce1c1c56853c9117 ==> Fetching libssh ==> Downloading https://ghcr.io/v2/homebrew/core/libssh/manifests/0.10.5_1 ==> Downloading https://ghcr.io/v2/homebrew/core/libssh/blobs/sha256:5b5925ae5e0d31dd7b2c508eb1f6a19585a87cb66310d5a658e772666500191d ==> Fetching lzo ==> Downloading https://ghcr.io/v2/homebrew/core/lzo/manifests/2.10 ==> Downloading https://ghcr.io/v2/homebrew/core/lzo/blobs/sha256:a565c627b13f2dc7fc4550aa8290a4c3feb2f48fcaa45c9f7f4bc4fe4535aa66 ==> Fetching pixman ==> Downloading https://ghcr.io/v2/homebrew/core/pixman/manifests/0.42.2-1 ==> Downloading https://ghcr.io/v2/homebrew/core/pixman/blobs/sha256:e27867c503bd9cf858159261e053184d19ae00357dc89426810f80734aaaefd0 ==> Fetching snappy ==> Downloading https://ghcr.io/v2/homebrew/core/snappy/manifests/1.1.10 ==> Downloading https://ghcr.io/v2/homebrew/core/snappy/blobs/sha256:ca95915a51bed09a5e70ebb6f253eabe4df5b00e87ebe49aea0124f8bb51bc3c ==> Fetching vde ==> Downloading https://ghcr.io/v2/homebrew/core/vde/manifests/2.3.3 ==> Downloading https://ghcr.io/v2/homebrew/core/vde/blobs/sha256:982a56825cbd1bd374001330e3492e83a1d82825ed7228d33705ffad3b927e8e ==> Fetching qemu ==> Downloading https://ghcr.io/v2/homebrew/core/qemu/manifests/8.0.3 ==> Downloading https://ghcr.io/v2/homebrew/core/qemu/blobs/sha256:019b99b1608ab59d75492252f2304649391fe61e947ed40d4c47e274d8bf869b ==> Fetching lima ==> Downloading https://ghcr.io/v2/homebrew/core/lima/manifests/0.16.0 ==> Downloading https://ghcr.io/v2/homebrew/core/lima/blobs/sha256:a8fc0d86b84683fb635bc4bb22ce6936b31c6b2aa02e495aa040f55dee8fa14c ==> Fetching colima ==> Downloading https://ghcr.io/v2/homebrew/core/colima/manifests/0.5.5 ==> Downloading https://ghcr.io/v2/homebrew/core/colima/blobs/sha256:f6934ad6a852b6dfef89d1c70f1054f57bb8d73829273d2bf2a81b54d40b209c ==> Fetching dependencies for docker: docker-completion ==> Fetching docker-completion ==> Downloading https://ghcr.io/v2/homebrew/core/docker-completion/manifests/24.0.4-1 ==> Downloading https://ghcr.io/v2/homebrew/core/docker-completion/blobs/sha256:632bdfc758fb664784f80ae5e63037220352d5316b9b4bf9c041ef745b01add8 ==> Fetching docker ==> Downloading https://ghcr.io/v2/homebrew/core/docker/manifests/24.0.4-1 ==> Downloading https://ghcr.io/v2/homebrew/core/docker/blobs/sha256:993a7739c3097bad48f5020581d92d6d84bede595d920d601da96ed9c8d6d6a6 ==> Installing dependencies for colima: capstone, glib, jpeg-turbo, libpng, libslirp, libssh, lzo, pixman, snappy, vde, qemu and lima ==> Installing colima dependency: capstone ==> Pouring capstone--4.0.2.arm64_ventura.bottle.tar.gz 🍺 /opt/homebrew/Cellar/capstone/4.0.2: 25 files, 14.7MB ==> Installing colima dependency: glib ==> Pouring glib--2.76.4.arm64_ventura.bottle.tar.gz 🍺 /opt/homebrew/Cellar/glib/2.76.4: 455 files, 22.2MB ==> Installing colima dependency: jpeg-turbo ==> Pouring jpeg-turbo--2.1.5.1.arm64_ventura.bottle.tar.gz 🍺 /opt/homebrew/Cellar/jpeg-turbo/2.1.5.1: 44 files, 2.5MB ==> Installing colima dependency: libpng ==> Pouring libpng--1.6.40.arm64_ventura.bottle.tar.gz 🍺 /opt/homebrew/Cellar/libpng/1.6.40: 27 files, 1.3MB ==> Installing colima dependency: libslirp ==> Pouring libslirp--4.7.0.arm64_ventura.bottle.tar.gz 🍺 /opt/homebrew/Cellar/libslirp/4.7.0: 11 files, 385KB ==> Installing colima dependency: libssh ==> Pouring libssh--0.10.5_1.arm64_ventura.bottle.tar.gz 🍺 /opt/homebrew/Cellar/libssh/0.10.5_1: 23 files, 1.3MB ==> Installing colima dependency: lzo ==> Pouring lzo--2.10.arm64_ventura.bottle.tar.gz 🍺 /opt/homebrew/Cellar/lzo/2.10: 31 files, 566.3KB ==> Installing colima dependency: pixman ==> Pouring pixman--0.42.2.arm64_ventura.bottle.1.tar.gz 🍺 /opt/homebrew/Cellar/pixman/0.42.2: 11 files, 1.2MB ==> Installing colima dependency: snappy ==> Pouring snappy--1.1.10.arm64_ventura.bottle.tar.gz 🍺 /opt/homebrew/Cellar/snappy/1.1.10: 18 files, 164.3KB ==> Installing colima dependency: vde ==> Pouring vde--2.3.3.arm64_ventura.bottle.tar.gz 🍺 /opt/homebrew/Cellar/vde/2.3.3: 66 files, 1.3MB ==> Installing colima dependency: qemu ==> Pouring qemu--8.0.3.arm64_ventura.bottle.tar.gz 🍺 /opt/homebrew/Cellar/qemu/8.0.3: 162 files, 527.4MB ==> Installing colima dependency: lima ==> Pouring lima--0.16.0.arm64_ventura.bottle.tar.gz 🍺 /opt/homebrew/Cellar/lima/0.16.0: 105 files, 131MB ==> Installing colima ==> Pouring colima--0.5.5.arm64_ventura.bottle.tar.gz ==> Caveats zsh completions have been installed to: /opt/homebrew/share/zsh/site-functions ==> Summary 🍺 /opt/homebrew/Cellar/colima/0.5.5: 8 files, 14.5MB ==> Installing dependencies for docker: docker-completion ==> Installing docker dependency: docker-completion ==> Pouring docker-completion--24.0.4.arm64_ventura.bottle.1.tar.gz 🍺 /opt/homebrew/Cellar/docker-completion/24.0.4: 9 files, 337.6KB ==> Installing docker ==> Pouring docker--24.0.4.arm64_ventura.bottle.1.tar.gz 🍺 /opt/homebrew/Cellar/docker/24.0.4: 13 files, 31.6MB ==> Caveats ==> colima zsh completions have been installed to: /opt/homebrew/share/zsh/site-functions $ softwareupdate --install-rosetta --agree-to-license 2023-08-29 13:49:03.113 softwareupdate[3953:13143] Package Authoring Error: 042-01896: Package reference com.apple.pkg.RosettaUpdateAuto is missing installKBytes attribute By using the agreetolicense option, you are agreeing that you have run this tool with the license only option and have read and agreed to the terms. If you do not agree, press CTRL-C and cancel this process immediately. Install of Rosetta 2 finished successfully ```

$ colima start --vm-type qemu --arch aarch64 --very-verbose --cpu 4 --memory 4 --disk 15 || true
time="2023-08-29T13:49:07Z" level=trace msg="cmd [\"limactl\" \"info\"]"
time="2023-08-29T13:49:07Z" level=trace msg="cmd [\"limactl\" \"list\" \"colima\" \"--json\"]"
time="2023-08-29T13:49:07Z" level=trace msg="error retrieving running instance: instance 'colima' does not exist"
time="2023-08-29T13:49:07Z" level=info msg="starting colima"
time="2023-08-29T13:49:07Z" level=info msg="runtime: docker"
time="2023-08-29T13:49:07Z" level=info msg="preparing network ..." context=vm
time="2023-08-29T13:49:07Z" level=trace msg="cmd [\"ln\" \"-sf\" \"/opt/homebrew/bin/colima\" \"/Users/gitlab/.colima/_wrapper/3a9197e1ca3cd2da076da2b473d7a7eb118e2cca/bin/qemu-system-aarch64\"]"
time="2023-08-29T13:49:07Z" level=trace msg="cmd [\"ln\" \"-sf\" \"/opt/homebrew/bin/colima\" \"/Users/gitlab/.colima/_wrapper/3a9197e1ca3cd2da076da2b473d7a7eb118e2cca/bin/qemu-system-x86_64\"]"
time="2023-08-29T13:49:07Z" level=trace msg="cmd [\"ln\" \"-sf\" \"/opt/homebrew/share\" \"/Users/gitlab/.colima/_wrapper/3a9197e1ca3cd2da076da2b473d7a7eb118e2cca/share\"]"
time="2023-08-29T13:49:07Z" level=trace msg="cmd [\"/opt/homebrew/bin/colima\" \"daemon\" \"status\" \"default\"]"
time="2023-08-29T13:49:07Z" level=trace msg="cmd [\"/opt/homebrew/bin/colima\" \"daemon\" \"start\" \"default\" \"--gvproxy\" \"--very-verbose\"]"
time="2023-08-29T13:49:07Z" level=info msg="creating and starting ..." context=vm
time="2023-08-29T13:49:07Z" level=trace msg="cmd [\"limactl\" \"start\" \"--tty=false\" \"/var/folders/lr/kycxps496119mlqd5wf0wcr40000gn/T/colima.yaml\"]"
time="2023-08-29T13:49:07Z" level=info msg="Terminal is not available, proceeding without opening an editor"
time="2023-08-29T13:49:07Z" level=info msg="Attempting to download the image" arch=aarch64 digest="sha512:84c93e8aaa09446618bf87daa993e260da69b50e95670aed5df6671b2cff9464810752cbf70f6ee5ddf9d3e1c91d98104b3c573cc024c5f0687ad3f4d2e93ebc" location="https://github.com/abiosoft/alpine-lima/releases/download/colima-v0.5.5/alpine-lima-clm-3.18.0-aarch64.iso"
Downloading the image (alpine-lima-clm-3.18.0-aarch64.iso)
222.28 MiB / 297.37 MiB (74.75%) ? p/s
297.37 MiB / 297.37 MiB (100.00%) 85.37 MiB/stime="2023-08-29T13:49:17Z" level=info msg="Downloaded the image from \"https://github.com/abiosoft/alpine-lima/releases/download/colima-v0.5.5/alpine-lima-clm-3.18.0-aarch64.iso\""
time="2023-08-29T13:49:21Z" level=info msg="[hostagent] Starting QEMU (hint: to watch the boot progress, see \"/Users/gitlab/.lima/colima/serial.log\")"
time="2023-08-29T13:49:21Z" level=info msg="SSH Local Port: 49215"
time="2023-08-29T13:49:21Z" level=info msg="[hostagent] Driver stopped due to error: \"exit status 255\""
time="2023-08-29T13:49:22Z" level=info msg="[hostagent] Waiting for the essential requirement 1 of 5: \"ssh\""
time="2023-08-29T13:49:22Z" level=info msg="[hostagent] Shutting down the host agent"
time="2023-08-29T13:49:22Z" level=warning msg="[hostagent] failed to exit SSH master"
time="2023-08-29T13:49:22Z" level=info msg="[hostagent] Shutting down QEMU with ACPI"
time="2023-08-29T13:49:22Z" level=warning msg="[hostagent] failed to open the QMP socket \"/Users/gitlab/.lima/colima/qmp.sock\", forcibly killing QEMU"
time="2023-08-29T13:49:22Z" level=info msg="[hostagent] QEMU has already exited"
time="2023-08-29T13:49:22Z" level=fatal msg="exiting, status={Running:false Degraded:false Exiting:true Errors:[] SSHLocalPort:0} (hint: see \"/Users/gitlab/.lima/colima/ha.stderr.log\")"
$ cat /Users/gitlab/.lima/colima/serial*.log
$ cat /Users/gitlab/.lima/colima/ha.stderr.log
{"level":"debug","msg":"executing [ssh-keygen -t ed25519 -q -N  -f /Users/gitlab/.lima/_config/user]","time":"2023-08-29T13:49:18Z"}
{"level":"debug","msg":"Creating iso file /Users/gitlab/.lima/colima/cidata.iso","time":"2023-08-29T13:49:19Z"}
{"level":"debug","msg":"Using /var/folders/lr/kycxps496119mlqd5wf0wcr40000gn/T/diskfs_iso3152737646 as workspace","time":"2023-08-29T13:49:19Z"}
{"level":"debug","msg":"Failed to detect CPU features. Assuming that AES acceleration is available on this Apple silicon.","time":"2023-08-29T13:49:19Z"}
{"level":"debug","msg":"OpenSSH version 9.0.1 detected","time":"2023-08-29T13:49:20Z"}
{"level":"debug","msg":"AES accelerator seems available, prioritizing aes128-gcm@openssh.com and aes256-gcm@openssh.com","time":"2023-08-29T13:49:20Z"}
{"level":"debug","msg":"QEMU version 8.0.3 detected","time":"2023-08-29T13:49:21Z"}
{"level":"debug","msg":"firmware candidates = [/Users/gitlab/.local/share/qemu/edk2-aarch64-code.fd /Users/gitlab/.colima/_wrapper/3a9197e1ca3cd2da076da2b473d7a7eb118e2cca/share/qemu/edk2-aarch64-code.fd /usr/share/AAVMF/AAVMF_CODE.fd /usr/share/qemu-efi-aarch64/QEMU_EFI.fd]","time":"2023-08-29T13:49:21Z"}
{"level":"info","msg":"Starting QEMU (hint: to watch the boot progress, see \"/Users/gitlab/.lima/colima/serial.log\")","time":"2023-08-29T13:49:21Z"}
{"level":"debug","msg":"qCmd.Args: [/Users/gitlab/.colima/_wrapper/3a9197e1ca3cd2da076da2b473d7a7eb118e2cca/bin/qemu-system-aarch64 -m 4096 -cpu host -machine virt,accel=hvf -smp 4,sockets=1,cores=4,threads=1 -drive if=pflash,format=raw,readonly=on,file=/Users/gitlab/.colima/_wrapper/3a9197e1ca3cd2da076da2b473d7a7eb118e2cca/share/qemu/edk2-aarch64-code.fd -boot order=d,splash-time=0,menu=on -drive file=/Users/gitlab/.lima/colima/basedisk,format=raw,media=cdrom,readonly=on -drive file=/Users/gitlab/.lima/colima/diffdisk,if=virtio,discard=on -drive id=cdrom0,if=none,format=raw,readonly=on,file=/Users/gitlab/.lima/colima/cidata.iso -device virtio-scsi-pci,id=scsi0 -device scsi-cd,bus=scsi0.0,drive=cdrom0 -netdev user,id=net0,net=192.168.5.0/24,dhcpstart=192.168.5.15,hostfwd=tcp:127.0.0.1:49215-:22 -device virtio-net-pci,netdev=net0,mac=52:55:55:59:df:93 -device virtio-rng-pci -display none -vga none -device ramfb -device qemu-xhci,id=usb-bus -device usb-kbd,bus=usb-bus.0 -device usb-mouse,bus=usb-bus.0 -parallel none -chardev socket,id=char-serial,path=/Users/gitlab/.lima/colima/serial.sock,server=on,wait=off,logfile=/Users/gitlab/.lima/colima/serial.log -serial chardev:char-serial -chardev socket,id=char-qmp,path=/Users/gitlab/.lima/colima/qmp.sock,server=on,wait=off -qmp chardev:char-qmp -name lima-colima -pidfile /Users/gitlab/.lima/colima/qemu.pid]","time":"2023-08-29T13:49:21Z"}
{"level":"debug","msg":"qemu[stderr]: qemu-system-aarch64: Error: HV_UNSUPPORTED","time":"2023-08-29T13:49:21Z"}
{"level":"info","msg":"Driver stopped due to error: \"exit status 255\"","time":"2023-08-29T13:49:21Z"}
{"level":"info","msg":"Waiting for the essential requirement 1 of 5: \"ssh\"","time":"2023-08-29T13:49:21Z"}
{"level":"info","msg":"Shutting down the host agent","time":"2023-08-29T13:49:22Z"}
{"level":"debug","msg":"shutting down the SSH master","time":"2023-08-29T13:49:22Z"}
{"level":"debug","msg":"executing script \"ssh\"","time":"2023-08-29T13:49:22Z"}
{"level":"debug","msg":"executing ssh for exiting the master: /usr/bin/ssh [ssh -F /dev/null -o IdentityFile=\"/Users/gitlab/.lima/_config/user\" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o NoHostAuthenticationForLocalhost=yes -o GSSAPIAuthentication=no -o PreferredAuthentications=publickey -o Compression=no -o BatchMode=yes -o IdentitiesOnly=yes -o Ciphers=\"^aes128-gcm@openssh.com,aes256-gcm@openssh.com\" -o User=gitlab -o ControlMaster=auto -o ControlPath=\"/Users/gitlab/.lima/colima/ssh.sock\" -o ControlPersist=5m -O exit -p 49215 127.0.0.1]","time":"2023-08-29T13:49:22Z"}
{"level":"debug","msg":"executing ssh for script \"ssh\": /usr/bin/ssh [ssh -F /dev/null -o IdentityFile=\"/Users/gitlab/.lima/_config/user\" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o NoHostAuthenticationForLocalhost=yes -o GSSAPIAuthentication=no -o PreferredAuthentications=publickey -o Compression=no -o BatchMode=yes -o IdentitiesOnly=yes -o Ciphers=\"^aes128-gcm@openssh.com,aes256-gcm@openssh.com\" -o User=gitlab -o ControlMaster=auto -o ControlPath=\"/Users/gitlab/.lima/colima/ssh.sock\" -o ControlPersist=5m -p 49215 127.0.0.1 -- /bin/bash]","time":"2023-08-29T13:49:22Z"}
{"error":"failed to execute `ssh -O exit -p 49215 127.0.0.1`, out=\"Control socket connect(/Users/gitlab/.lima/colima/ssh.sock): No such file or directory\\r\\n\": exit status 255","level":"warning","msg":"failed to exit SSH master","time":"2023-08-29T13:49:22Z"}
{"level":"debug","msg":"stdout=\"\", stderr=\"ssh: connect to host 127.0.0.1 port 49215: Connection refused\\r\\n\", err=failed to execute script \"ssh\": stdout=\"\", stderr=\"ssh: connect to host 127.0.0.1 port 49215: Connection refused\\r\\n\": exit status 255","time":"2023-08-29T13:49:22Z"}
{"level":"info","msg":"Shutting down QEMU with ACPI","time":"2023-08-29T13:49:22Z"}
{"error":"dial unix /Users/gitlab/.lima/colima/qmp.sock: connect: connection refused","level":"warning","msg":"failed to open the QMP socket \"/Users/gitlab/.lima/colima/qmp.sock\", forcibly killing QEMU","time":"2023-08-29T13:49:22Z"}
{"level":"info","msg":"QEMU has already exited","time":"2023-08-29T13:49:22Z"}
time="2023-08-29T13:49:22Z" level=fatal msg="error starting vm: error at 'creating and starting': exit status 1"

See debug-only message qemu[stderr]: qemu-system-aarch64: Error: HV_UNSUPPORTED. My understanding is that this is because of nested virtualization?

But this could be an error message when regularly running colima without --very-verbose flag and with vz VM type as well.

[rfay]

It's broken right now, has been on and off for a couple of weeks. You'll see many issues in this queue about qemu failing at start on amd64. that's what is going on here.

You can find your way to most of the drama via #786 (comment)

The Github Runners issue is actions/runner-images#8104 (comment)

[mitar]

But vz VM type also does not work?

[rfay]

Well, you're using qemu here, that's what's broken.

[rfay]

It's possible that you have a completely different problem; I don't know much/anything about Gitlab, but IIRC it doesn't have an actually easily usable docker provider.