Revert "qemu 8.0.4" #139409

AkihiroSuda · 2023-08-12T22:56:58Z

Reverts #139356 , as the release seems broken

This release is broken: https://gitlab.com/qemu-project/qemu/-/issues/1828

v8.0.3:
$ qemu-system-x86_64 -accel hvf
(shows iPXE screen, as expected)
v8.0.4:
$ qemu-system-x86_64 -accel hvf
qemu-system-x86_64: -accel hvf: Unknown Error
Abort trap: 6
(Tested on macOS 13.5 Intel)

Originally posted by @AkihiroSuda in #139356 (comment)

chenrui333 · 2023-08-12T23:33:00Z

is it just a patch thing rather than a complete release revert?

AkihiroSuda · 2023-08-12T23:35:39Z

is it just a patch thing rather than a complete release revert?

Ideally we should just have a patch, but the issue is still unanalyzed and critical for most QEMU users on macOS, so it would be nice to just revert the complete release.

AkihiroSuda · 2023-08-12T23:38:11Z

I also noticed that QEMU v8.0.4 still works with brew install --from-source qemu.

So this might be a regression on the Homebrew side, not on QEMU side.

alebcay · 2023-08-12T23:40:14Z

Generally, reverting a version bump doesn't go well (the entire versioning/update mechanism is only designed to increment). There is a mechanism available to "force" a "downgrade" but it's usually reserved for tricking the versioning system to accept upstream "long-term"/permanent changes, e.g. project changes its versioning scheme, not for a hotfix on a single version.

Agree we should look further into why it may work when built from source but not bottled.

AkihiroSuda · 2023-08-12T23:53:34Z

It looks like re-attaching the com.apple.security.hypervisor entitlement works:

$ codesign -d --entitlements - --xml /usr/local/Cellar/qemu/8.0.3/bin/qemu-system-x86_64 
Executable=/usr/local/Cellar/qemu/8.0.3/bin/qemu-system-x86_64
<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "https://www.apple.com/DTDs/PropertyList-1.0.dtd"><plist version="1.0"><dict><key>com.apple.security.hypervisor</key><true/></dict></plist>

$ codesign -d --entitlements - --xml /usr/local/Cellar/qemu/8.0.4/bin/qemu-system-x86_64 
Executable=/usr/local/Cellar/qemu/8.0.4/bin/qemu-system-x86_64
<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "https://www.apple.com/DTDs/PropertyList-1.0.dtd"><plist version="1.0"><dict><key>com.apple.security.hypervisor</key><true/></dict></plist>

$ /usr/local/Cellar/qemu/8.0.4/bin/qemu-system-x86_64 -accel hvf
qemu-system-x86_64: -accel hvf: Unknown Error
Abort trap: 6

$ cat >entitlements.xml <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>com.apple.security.hypervisor</key>
    <true/>
</dict>
</plist>
EOF

$ codesign -s - --entitlements entitlements.xml --force /usr/local/Cellar/qemu/8.0.4/bin/qemu-system-x86_64
/usr/local/Cellar/qemu/8.0.4/bin/qemu-system-x86_64: replacing existing signature

$ /usr/local/Cellar/qemu/8.0.4/bin/qemu-system-x86_64 -accel hvf
(shows iPXE screen, as expected)

A weird thing is that com.apple.security.hypervisor seems already attached to the binary, and the entitlement stuff doesn't seem changed in #139356

alebcay · 2023-08-13T00:17:20Z

It's possible some that some code changed in https://github.com/Homebrew/brew regarding codesign/entitlements in the bottle generation process in the time between the 8.0.3 and 8.0.4.

Bo98 · 2023-08-13T00:24:07Z

Can you obtain a crash log? "Unknown error" is of little use on its own.

AkihiroSuda · 2023-08-13T00:31:07Z

Can you obtain a crash log? "Unknown error" is of little use on its own.

$ lldb qemu-system-x86_64
(lldb) target create "qemu-system-x86_64"
Current executable set to 'qemu-system-x86_64' (x86_64).
(lldb) process handle SIGUSR2 -s false -p true
NAME         PASS     STOP     NOTIFY
===========  =======  =======  =======
SIGUSR2      true     false    not set
(lldb) run -accel hvf
Process 31391 launched: '/usr/local/bin/qemu-system-x86_64' (x86_64)
Process 31391 stopped and restarted: thread 1 received signal: SIGUSR2
qemu-system-x86_64: -accel hvf: Unknown Error
Process 31391 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
    frame #0: 0x00007ff8036a7202 libsystem_kernel.dylib`__pthread_kill + 10
libsystem_kernel.dylib`:
->  0x7ff8036a7202 <+10>: jae    0x7ff8036a720c            ; <+20>
    0x7ff8036a7204 <+12>: movq   %rax, %rdi
    0x7ff8036a7207 <+15>: jmp    0x7ff8036a0ceb            ; cerror_nocancel
    0x7ff8036a720c <+20>: retq   
Target 0: (qemu-system-x86_64) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
  * frame #0: 0x00007ff8036a7202 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007ff8036deee6 libsystem_pthread.dylib`pthread_kill + 263
    frame #2: 0x00007ff803605b45 libsystem_c.dylib`abort + 123
    frame #3: 0x00000001004d1a38 qemu-system-x86_64`assert_hvf_ok.cold.1 + 16
    frame #4: 0x000000010033e331 qemu-system-x86_64`assert_hvf_ok + 101
    frame #5: 0x000000010033e45f qemu-system-x86_64`hvf_accel_init + 22
    frame #6: 0x0000000100204392 qemu-system-x86_64`accel_init_machine + 82
    frame #7: 0x00000001001bd416 qemu-system-x86_64`do_configure_accelerator + 201
    frame #8: 0x0000000100497183 qemu-system-x86_64`qemu_opts_foreach + 91
    frame #9: 0x00000001001bac76 qemu-system-x86_64`qemu_init + 11335
    frame #10: 0x00000001003472e7 qemu-system-x86_64`main + 9
    frame #11: 0x00007ff80338541f dyld`start + 1903
(lldb)

alebcay · 2023-08-13T00:38:12Z

I tried with lldb with run -accel hvf and also just run and got the same error. However, running the binary without -accel hvf works fine for me. So my guess is that the error encountered there might be a red herring.

Bo98 · 2023-08-13T00:40:01Z

Does codesign --verify flag anything?

We don't really support codesigning on Intel, so I'm guessing something's perhaps changed to invalidate it.

AkihiroSuda · 2023-08-13T00:47:30Z

Yes, the sign seems broken:

$ codesign --verify /usr/local/Cellar/qemu/8.0.3/bin/qemu-system-x86_64 
(no output)

$ codesign --verify /usr/local/Cellar/qemu/8.0.4/bin/qemu-system-x86_64 
/usr/local/Cellar/qemu/8.0.4/bin/qemu-system-x86_64: invalid signature (code or signature have been modified)
In architecture: x86_64

Bo98 · 2023-08-13T00:49:35Z

We could potentially modify https://github.com/Homebrew/brew/blob/7044f50f889f8ec7d8656ac176e960e53bee1a0a/Library/Homebrew/extend/os/mac/keg.rb#L29 to apply on Intel on the specific conditions that an existing signature has been invalidated (we won't codesign everything like on arm64).

AkihiroSuda · 2023-08-13T01:34:33Z

We could potentially modify https://github.com/Homebrew/brew/blob/7044f50f889f8ec7d8656ac176e960e53bee1a0a/Library/Homebrew/extend/os/mac/keg.rb#L29 to apply on Intel on the specific conditions that an existing signature has been invalidated (we won't codesign everything like on arm64).

Thanks for taking a look, could you modify that file and rebuild the bottle?

alebcay · 2023-08-13T03:28:41Z

One interesting thing I noticed, the following sequence on my local machine (Intel, Monterey) produces a binary that doesn't have its signature broken:

$ brew install --build-bottle qemu
$ brew bottle qemu
$ brew rm qemu
$ brew install ./qemu--8.0.4.monterey.bottle.1.tar.gz
$ qemu-system-x86_64 -accel hvf
( window pops open as expected )
$ codesign --verify /usr/local/Cellar/qemu/8.0.4/bin/qemu-system-x86_64
( no output, exit code 0 )

So not sure what the CI is doing differently that causes its bottles to not work when installed on my machine.

HOMEBREW_VERSION: 4.1.4-40-g2cfea60
ORIGIN: https://github.com/Homebrew/brew
HEAD: 2cfea600d4bded2e59c4cb97dcb3b5805f3bc5e2
Last commit: 87 minutes ago
Core tap origin: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 2b685d8cd2b94e7dbb54fe2e80427ab357bd3156
Core tap last commit: 4 hours ago
Core tap branch: master
Core tap JSON: 13 Aug 03:13 UTC
HOMEBREW_PREFIX: /usr/local
HOMEBREW_BOOTSNAP: set
HOMEBREW_CASK_OPTS: []
HOMEBREW_DISPLAY: /private/tmp/com.apple.launchd.y85x2GqnId/org.macosforge.xquartz:0
HOMEBREW_EDITOR: /Applications/VSCodium.app/Contents/Resources/app/bin/codium
HOMEBREW_MAKE_JOBS: 4
Homebrew Ruby: 2.6.10 => /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby
CPU: quad-core 64-bit ivybridge
Clang: 14.0.0 build 1400
Git: 2.41.0 => /usr/local/bin/git
Curl: 7.87.0 => /usr/bin/curl
macOS: 12.6.6-x86_64
CLT: 14.2.0.0.1.1668646533
Xcode: 14.2

AkihiroSuda · 2023-08-13T15:51:21Z

$ file /usr/local/Cellar/lima/0.17.1/bin/limactl
/usr/local/Cellar/lima/0.17.1/bin/limactl: Mach-O 64-bit executable x86_64

$ codesign -d --entitlements - --xml /usr/local/Cellar/lima/0.17.1/bin/limactl 
Executable=/usr/local/Cellar/lima/0.17.1/bin/limactl
<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "https://www.apple.com/DTDs/PropertyList-1.0.dtd"><plist version="1.0"><dict><key>com.apple.security.network.client</key><true/><key>com.apple.security.network.server</key><true/><key>com.apple.security.virtualization</key><true/></dict></plist>

$ codesign --verify /usr/local/Cellar/lima/0.17.1/bin/limactl
(no output)

Lima v0.17.1 was bottled just 16 hours ago but it seems still properly signed 🤔

Wondering if the QEMU v8.0.4 signature was just lost due to a transient error?

AkihiroSuda · 2023-08-13T15:56:20Z

Would it be possible to trigger the bot to rebuild QEMU v8.0.4 bottle and see if it works?

rfay · 2023-08-14T00:14:43Z

I do hope we can find some resolution soon, even if it's a short-term one. This problem causes all lima and colima installs to fail OOTB, so the support load will continue to increase rapidly as the whole world discovers this problem.

SMillerDev · 2023-08-14T06:56:09Z

Maybe Homebrew/brew#15864 will help, I'm kicking off a rebuild

AkihiroSuda · 2023-08-14T10:54:08Z

Thanks, the issue is solved in

qemu: update 8.0.4 bottle. #139492

aselvan · 2023-08-14T14:22:20Z

It looks like re-attaching the com.apple.security.hypervisor entitlement works:

$ codesign -d --entitlements - --xml /usr/local/Cellar/qemu/8.0.3/bin/qemu-system-x86_64 
Executable=/usr/local/Cellar/qemu/8.0.3/bin/qemu-system-x86_64
<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "https://www.apple.com/DTDs/PropertyList-1.0.dtd"><plist version="1.0"><dict><key>com.apple.security.hypervisor</key><true/></dict></plist>

$ codesign -d --entitlements - --xml /usr/local/Cellar/qemu/8.0.4/bin/qemu-system-x86_64 
Executable=/usr/local/Cellar/qemu/8.0.4/bin/qemu-system-x86_64
<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "https://www.apple.com/DTDs/PropertyList-1.0.dtd"><plist version="1.0"><dict><key>com.apple.security.hypervisor</key><true/></dict></plist>

$ /usr/local/Cellar/qemu/8.0.4/bin/qemu-system-x86_64 -accel hvf
qemu-system-x86_64: -accel hvf: Unknown Error
Abort trap: 6

$ cat >entitlements.xml <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>com.apple.security.hypervisor</key>
    <true/>
</dict>
</plist>
EOF

$ codesign -s - --entitlements entitlements.xml --force /usr/local/Cellar/qemu/8.0.4/bin/qemu-system-x86_64
/usr/local/Cellar/qemu/8.0.4/bin/qemu-system-x86_64: replacing existing signature

$ /usr/local/Cellar/qemu/8.0.4/bin/qemu-system-x86_64 -accel hvf
(shows iPXE screen, as expected)

A weird thing is that com.apple.security.hypervisor seems already attached to the binary, and the entitlement stuff doesn't seem changed in #139356

@AkihiroSuda Your workaround of re-attaching entitlement fixed this problem for me and thank you for the workaround. Hopefully, this will be resolved in the 8.1 version down the road. Thanks again!

AkihiroSuda · 2023-08-14T14:31:40Z

Hopefully, this will be resolved in the 8.1 version down the road.

The issue is already fixed in the "rebuild 1" of v8.0.4:

https://github.com/Homebrew/homebrew-core/pull/139492/files

AkihiroSuda · 2023-08-23T05:32:01Z

Broken again in v8.1.0 😢

qemu: invalid signature (code or signature have been modified) (regression in 8.0.4, fixed in 8.0.4-1, re-regression in 8.1.0) #140244

Revert "qemu 8.0.4"

e525dfb

github-actions bot added the autosquash Automatically squash pull request commits according to Homebrew style. label Aug 12, 2023

AkihiroSuda mentioned this pull request Aug 13, 2023

qemu 8.0.4 #139356

Merged

AkihiroSuda mentioned this pull request Aug 13, 2023

qemu: ask to sign QEMU binary when the binary is not properly signed lima-vm/lima#1743

Merged

AkihiroSuda closed this Aug 14, 2023

This was referenced Aug 18, 2023

macOS 12 (20230812) Image Update breaks qemu HVF actions/runner-images#8113

Closed

colima start no longer works on macos-12 runner 20230812.3 actions/runner-images#8104

Closed

AkihiroSuda mentioned this pull request Aug 23, 2023

qemu: invalid signature (code or signature have been modified) (regression in 8.0.4, fixed in 8.0.4-1, re-regression in 8.1.0) #140244

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Revert "qemu 8.0.4" #139409

Revert "qemu 8.0.4" #139409

AkihiroSuda commented Aug 12, 2023

chenrui333 commented Aug 12, 2023

AkihiroSuda commented Aug 12, 2023

AkihiroSuda commented Aug 12, 2023

alebcay commented Aug 12, 2023 •

edited

Loading

AkihiroSuda commented Aug 12, 2023 •

edited

Loading

alebcay commented Aug 13, 2023

Bo98 commented Aug 13, 2023

AkihiroSuda commented Aug 13, 2023

alebcay commented Aug 13, 2023

Bo98 commented Aug 13, 2023

AkihiroSuda commented Aug 13, 2023

Bo98 commented Aug 13, 2023

AkihiroSuda commented Aug 13, 2023

alebcay commented Aug 13, 2023

AkihiroSuda commented Aug 13, 2023 •

edited

Loading

AkihiroSuda commented Aug 13, 2023

rfay commented Aug 14, 2023

SMillerDev commented Aug 14, 2023

AkihiroSuda commented Aug 14, 2023

aselvan commented Aug 14, 2023

AkihiroSuda commented Aug 14, 2023

AkihiroSuda commented Aug 23, 2023

Revert "qemu 8.0.4" #139409

Revert "qemu 8.0.4" #139409

Conversation

AkihiroSuda commented Aug 12, 2023

chenrui333 commented Aug 12, 2023

AkihiroSuda commented Aug 12, 2023

AkihiroSuda commented Aug 12, 2023

alebcay commented Aug 12, 2023 • edited Loading

AkihiroSuda commented Aug 12, 2023 • edited Loading

alebcay commented Aug 13, 2023

Bo98 commented Aug 13, 2023

AkihiroSuda commented Aug 13, 2023

alebcay commented Aug 13, 2023

Bo98 commented Aug 13, 2023

AkihiroSuda commented Aug 13, 2023

Bo98 commented Aug 13, 2023

AkihiroSuda commented Aug 13, 2023

alebcay commented Aug 13, 2023

AkihiroSuda commented Aug 13, 2023 • edited Loading

AkihiroSuda commented Aug 13, 2023

rfay commented Aug 14, 2023

SMillerDev commented Aug 14, 2023

AkihiroSuda commented Aug 14, 2023

aselvan commented Aug 14, 2023

AkihiroSuda commented Aug 14, 2023

AkihiroSuda commented Aug 23, 2023

alebcay commented Aug 12, 2023 •

edited

Loading

AkihiroSuda commented Aug 12, 2023 •

edited

Loading

AkihiroSuda commented Aug 13, 2023 •

edited

Loading