Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for macOS runners #3

Open
delan opened this issue Sep 6, 2024 · 8 comments
Open

Add support for macOS runners #3

delan opened this issue Sep 6, 2024 · 8 comments

Comments

@delan
Copy link
Member

delan commented Sep 6, 2024

We may be able to run macOS on libvirt/KVM, thanks to https://github.com/kholia/OSX-KVM!

@delan
Copy link
Member Author

delan commented Sep 6, 2024

image

image

@delan
Copy link
Member Author

delan commented Sep 6, 2024

image

@delan
Copy link
Member Author

delan commented Sep 6, 2024

Installed ok with default approach in the README:

Runs fine after moving to zvol:

diff --git a/OpenCore-Boot.sh b/OpenCore-Boot.sh
index e027851..047a910 100755
--- a/OpenCore-Boot.sh
+++ b/OpenCore-Boot.sh
@@ -56,7 +56,8 @@ args=(
   -device ide-hd,bus=sata.2,drive=OpenCoreBoot
   -device ide-hd,bus=sata.3,drive=InstallMedia
   -drive id=InstallMedia,if=none,file="$REPO_PATH/BaseSystem.img",format=raw
-  -drive id=MacHDD,if=none,file="$REPO_PATH/mac_hdd_ng.img",format=qcow2
+  # -drive id=MacHDD,if=none,file="$REPO_PATH/mac_hdd_ng.img",format=qcow2
+  -drive id=MacHDD,if=none,file="/dev/zvol/cuffs/base/servo-macos13.clean",format=raw
   -device ide-hd,bus=sata.4,drive=MacHDD
   # -netdev tap,id=net0,ifname=tap0,script=no,downscript=no -device virtio-net-pci,netdev=net0,id=net0,mac=52:54:00:c9:18:27
   -netdev user,id=net0,hostfwd=tcp::2222-:22 -device virtio-net-pci,netdev=net0,id=net0,mac=52:54:00:c9:18:27

@delan
Copy link
Member Author

delan commented Sep 6, 2024

All three boot menu options crash qemu when moving to libvirt, using the changes below plus this process:

diff --git a/macOS-libvirt-Catalina.xml b/macOS-libvirt-Catalina.xml
index 05cb13f..b92bfcc 100644
--- a/macOS-libvirt-Catalina.xml
+++ b/macOS-libvirt-Catalina.xml
@@ -1,4 +1,5 @@
 <?xml version='1.0' encoding='UTF-8'?>
+<!-- Based on <https://github.com/kholia/OSX-KVM/blob/182e2dd0715175801521f6342ac7cc715044cb12/macOS-libvirt-Catalina.xml> -->
 <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
 <!--
 	macOS libvirt XML configuration.
@@ -34,17 +35,19 @@
 	give the clock a tsc timer for kvm to pass to the guest.
 	See comments on the <kvm> and <clock> attributes.
 -->
-  <name>macOS</name>
-  <uuid>2aca0dd6-cec9-4717-9ab2-0b7b13d111c3</uuid>
-  <title>macOS</title>
-  <memory unit='KiB'>4194304</memory>
-  <currentMemory unit='KiB'>4194304</currentMemory>
-  <vcpu placement='static'>4</vcpu>
+  <name>servo-macos13.init</name>
+  <uuid>281ecffa-204e-4528-9ccf-12948f4cb78f</uuid>
+  <memory unit="KiB">25165824</memory>
+  <currentMemory unit="KiB">25165824</currentMemory>
+  <memoryBacking>
+    <hugepages/>
+  </memoryBacking>
+  <vcpu placement='static'>16</vcpu>
   <os>
     <type arch='x86_64' machine='pc-q35-4.2'>hvm</type>
     <!-- We don't need patched OVMF anymore when using latest OpenCore, stock one is okay -->
-    <loader readonly='yes' type='pflash'>/home/CHANGEME/OSX-KVM/OVMF_CODE.fd</loader>
-    <nvram>/home/CHANGEME/OSX-KVM/OVMF_VARS.fd</nvram>
+    <loader readonly='yes' type='pflash'>/var/lib/libvirt/images/OSX-KVM/OVMF_CODE.fd</loader>
+    <nvram>/var/lib/libvirt/images/OSX-KVM/OVMF_VARS.fd</nvram>
   </os>
   <features>
     <acpi/>
@@ -59,24 +62,26 @@
   <on_reboot>restart</on_reboot>
   <on_crash>restart</on_crash>
   <devices>
-    <emulator>/usr/bin/qemu-system-x86_64</emulator>
+    <emulator>/run/libvirt/nix-emulators/qemu-system-x86_64</emulator>
     <disk type='file' device='disk'>
       <driver name='qemu' type='qcow2' cache='writeback' io='threads'/>
-      <source file='/home/CHANGEME/OSX-KVM/OpenCore/OpenCore.qcow2'/>
+      <source file='/var/lib/libvirt/images/OSX-KVM/OpenCore/OpenCore.qcow2'/>
       <target dev='sda' bus='sata'/>
       <boot order='2'/>
       <address type='drive' controller='0' bus='0' target='0' unit='0'/>
     </disk>
-    <disk type='file' device='disk'>
-      <driver name='qemu' type='qcow2' cache='writeback' io='threads'/>
-      <source file='/home/CHANGEME/OSX-KVM/mac_hdd_ng.img'/>
+    <disk type='block' device='disk'>
+      <!-- TODO: can we benefit from these OSX-KVM defaults? -->
+      <!-- <driver name='qemu' type='qcow2' cache='writeback' io='threads'/> -->
+      <driver name='qemu' type='raw' cache='none' io='native' discard='unmap'/>
+      <source dev='/dev/zvol/path/to/servo-macos13'/>
       <target dev='sdb' bus='sata'/>
       <boot order='1'/>
       <address type='drive' controller='0' bus='0' target='0' unit='1'/>
     </disk>
     <disk type="file" device="disk">
       <driver name="qemu" type="raw" cache="writeback"/>
-      <source file="/home/CHANGEME/OSX-KVM/BaseSystem.img"/>
+      <source file="/var/lib/libvirt/images/OSX-KVM/BaseSystem.img"/>
       <target dev="sdc" bus="sata"/>
       <boot order="3"/>
       <address type="drive" controller="0" bus="0" target="0" unit="2"/>
@@ -141,7 +146,7 @@
     <!-- Make sure you put your nic in bus 0x0 and slot 0x0y(y is numeric), this will make nic built-in and apple-store work-->
     <interface type="bridge">
       <mac address="52:54:00:e6:85:40"/>
-      <source bridge="virbr0"/>
+      <source bridge="cinet"/>
       <model type="vmxnet3"/>
       <address type="pci" domain="0x0000" bus="0x00" slot="0x00" function="0x0"/>
     </interface>
Sep 06 19:43:56 jupiter kernel: CPU 3/KVM[2060741]: segfault at 14 ip 000056257c357a24 sp 00007f81f1dfeeb0 error 4 in .qemu-system-x86_64-wrapped[56257c14c000+652000] likely on CPU 31 (core 15, socket 0)
Sep 06 19:43:56 jupiter kernel: Code: 48 83 c4 58 89 d8 5b 5d 41 5c 41 5d 41 5e 41 5f c3 0f 1f 44 00 00 89 f0 48 8b 8d 40 83 00 00 4c 8d 04 40 49 c1 e0 03 4c 01 c1 <8b> 41 14 85 c0 0f 84 11 01 00 00 83 c0 01 89 41 14 41 80 bd d1 01

Maybe there’s a discrepancy in the effective qemu configs? Top is with OpenCore-Boot.sh, bottom is with libvirt:

delan    2056484  143  2.5 5768672 3400148 pts/0 Sl+  19:40   0:41 qemu-system-x86_64 -enable-kvm -m 4096 -cpu Penryn,kvm=on,vendor=GenuineIntel,+invtsc,vmware-cpuid-freq=on,+ssse3,+sse4.2,+popcnt,+avx,+aes,+xsave,+xsaveopt,check -machine q35 -device qemu-xhci,id=xhci -device usb-kbd,bus=xhci.0 -device usb-tablet,bus=xhci.0 -smp 4,cores=2,sockets=1 -device usb-ehci,id=ehci -device isa-applesmc,osk=ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc -drive if=pflash,format=raw,readonly=on,file=././OVMF_CODE.fd -drive if=pflash,format=raw,file=././OVMF_VARS-1920x1080.fd -smbios type=2 -device ich9-intel-hda -device hda-duplex -device ich9-ahci,id=sata -drive id=OpenCoreBoot,if=none,snapshot=on,format=qcow2,file=./OpenCore/OpenCore.qcow2 -device ide-hd,bus=sata.2,drive=OpenCoreBoot -device ide-hd,bus=sata.3,drive=InstallMedia -drive id=InstallMedia,if=none,file=./BaseSystem.img,format=raw -drive id=MacHDD,if=none,file=/dev/zvol/cuffs/base/servo-macos13.clean,format=raw -device ide-hd,bus=sata.4,drive=MacHDD -netdev user,id=net0,hostfwd=tcp::2222-:22 -device virtio-net-pci,netdev=net0,id=net0,mac=52:54:00:c9:18:27 -monitor stdio -device vmware-svga
qemu-li+ 2057187  363  0.0 26995424 80788 ?      Rl   19:41   0:18 /run/libvirt/nix-emulators/qemu-system-x86_64 -name guest=servo-macos13.clean,debug-threads=on -S -object {"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-7-servo-macos13.clean/master-key.aes"} -blockdev {"driver":"file","filename":"/var/lib/libvirt/images/OSX-KVM/OVMF_CODE.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"} -blockdev {"driver":"file","filename":"/var/lib/libvirt/images/OSX-KVM/OVMF_VARS.servo-macos13.clean.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"} -machine pc-q35-4.2,usb=off,dump-guest-core=off,memory-backend=pc.ram,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,hpet=off,acpi=on -accel kvm -cpu qemu64 -m size=25165824k -object {"qom-type":"memory-backend-file","id":"pc.ram","mem-path":"/dev/hugepages/libvirt/qemu/7-servo-macos13.clean","x-use-canonical-path-for-ramblock-id":false,"prealloc":true,"size":25769803776} -overcommit mem-lock=off -smp 16,sockets=16,cores=1,threads=1 -uuid 58f5ed36-aa12-4de5-878b-767cbad803e5 -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=31,server=on,wait=off -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-shutdown -boot strict=on -device {"driver":"pcie-root-port","port":8,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x1"} -device {"driver":"pcie-root-port","port":9,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x1.0x1"} -device {"driver":"pcie-root-port","port":10,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x1.0x2"} -device {"driver":"pcie-root-port","port":11,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x1.0x3"} -device {"driver":"pcie-root-port","port":12,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x1.0x4"} -device {"driver":"pcie-root-port","port":13,"chassis":6,"id":"pci.6","bus":"pcie.0","addr":"0x1.0x5"} -device {"driver":"pcie-root-port","port":14,"chassis":7,"id":"pci.7","bus":"pcie.0","addr":"0x1.0x6"} -device {"driver":"pcie-root-port","port":15,"chassis":8,"id":"pci.8","bus":"pcie.0","addr":"0x1.0x7"} -device {"driver":"pcie-pci-bridge","id":"pci.9","bus":"pci.3","addr":"0x0"} -device {"driver":"ich9-usb-ehci1","id":"usb","bus":"pcie.0","addr":"0x7.0x7"} -device {"driver":"ich9-usb-uhci1","masterbus":"usb.0","firstport":0,"bus":"pcie.0","multifunction":true,"addr":"0x7"} -device {"driver":"ich9-usb-uhci2","masterbus":"usb.0","firstport":2,"bus":"pcie.0","addr":"0x7.0x1"} -device {"driver":"ich9-usb-uhci3","masterbus":"usb.0","firstport":4,"bus":"pcie.0","addr":"0x7.0x2"} -device {"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.2","addr":"0x0"} -blockdev {"driver":"file","filename":"/var/lib/libvirt/images/OSX-KVM/OpenCore/OpenCore.qcow2","aio":"threads","node-name":"libvirt-3-storage","auto-read-only":true,"discard":"unmap","cache":{"direct":false,"no-flush":false}} -blockdev {"node-name":"libvirt-3-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"qcow2","file":"libvirt-3-storage","backing":null} -device {"driver":"ide-hd","bus":"ide.0","drive":"libvirt-3-format","id":"sata0-0-0","bootindex":2,"write-cache":"on"} -blockdev {"driver":"host_device","filename":"/dev/zvol/cuffs/base/servo-macos13.clean","aio":"native","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap","cache":{"direct":true,"no-flush":false}} -blockdev {"node-name":"libvirt-2-format","read-only":false,"discard":"unmap","cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-2-storage"} -device {"driver":"ide-hd","bus":"ide.1","drive":"libvirt-2-format","id":"sata0-0-1","bootindex":1,"write-cache":"on"} -blockdev {"driver":"file","filename":"/var/lib/libvirt/images/OSX-KVM/BaseSystem.img","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap","cache":{"direct":false,"no-flush":false}} -blockdev {"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"} -device {"driver":"ide-hd","bus":"ide.2","drive":"libvirt-1-format","id":"sata0-0-2","bootindex":3,"write-cache":"on"} -netdev {"type":"tap","fd":"34","id":"hostnet0"} -device {"driver":"vmxnet3","netdev":"hostnet0","id":"net0","mac":"52:54:00:e7:bd:48","bus":"pci.9","addr":"0x1"} -chardev pty,id=charserial0 -device {"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":0} -chardev socket,id=charchannel0,fd=30,server=on,wait=off -device {"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"} -audiodev {"id":"audio1","driver":"spice"} -spice port=5900,addr=127.0.0.1,disable-ticketing=on,seamless-migration=on -device {"driver":"virtio-vga","id":"video0","max_outputs":1,"bus":"pci.1","addr":"0x0"} -global ICH9-LPC.noreboot=off -watchdog-action reset -device isa-applesmc,osk=ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc -smbios type=2 -usb -device usb-tablet -device usb-kbd -cpu Penryn,kvm=on,vendor=GenuineIntel,+invtsc,vmware-cpuid-freq=on,+ssse3,+sse4.2,+popcnt,+avx,+aes,+xsave,+xsaveopt,check -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on

@delan
Copy link
Member Author

delan commented Sep 9, 2024

We can’t get a core dump due to “resource limits”:

Sep 09 10:40:02 jupiter kernel: CPU 1/KVM[1545351]: segfault at 14 ip 000056473f97ca24 sp 00007f550f076eb0 error 4 in .qemu-system-x86_64-wrapped[56473f771000+652000] likely on CPU 20 (core 4, socket 0)
Sep 09 10:40:02 jupiter kernel: Code: 48 83 c4 58 89 d8 5b 5d 41 5c 41 5d 41 5e 41 5f c3 0f 1f 44 00 00 89 f0 48 8b 8d 40 83 00 00 4c 8d 04 40 49 c1 e0 03 4c 01 c1 <8b> 41 14 85 c0 0f 84 11 01 00 00 83 c0 01 89 41 14 41 80 bd d1 01
Sep 09 10:40:02 jupiter systemd-coredump[1545634]: Resource limits disable core dumping for process 1545297 (.qemu-system-x8).
Sep 09 10:40:02 jupiter systemd-coredump[1545634]: [🡕] Process 1545297 (.qemu-system-x8) of user 301 terminated abnormally without generating a coredump.

This is true even if ulimit and systemd-coredump are configured as follows:

$ ulimit -a | rg core
-c: core file size (blocks)         unlimited
systemd.coredump.extraConfig = ''
  Storage=external
  Compress=yes
  ProcessSizeMax=200G
  ExternalSizeMax=200G
  JournalSizeMax=0
  MaxUse=0
  KeepFree=0
'';

Per this blog post, this is because the qemu process had its limit set to zero:

$ rg core /proc/$(pidof qemu-system-x86_64)/limits
6:Max core file size        0                    0                    bytes
% sudo prlimit --core=-1 -p $(pidof qemu-system-x86_64)
[sudo] password for delan:
$ rg core /proc/$(pidof qemu-system-x86_64)/limits
6:Max core file size        unlimited            unlimited            bytes

We get the core dump summary below. It’s for process 1545846, but it’s not clear to me which thread crashed:

Sep 09 10:40:22 jupiter systemd-coredump[1546111]: [🡕] Process 1545846 (.qemu-system-x8) of user 301 dumped core.

                                                   Module libdigestmd5.so.3 without build-id.
                                                   Module libgs2.so.3 without build-id.
                                                   Module liblogin.so.3 without build-id.
                                                   Module libdb-5.3.so without build-id.
                                                   Module libsasldb.so.3 without build-id.
                                                   Module libcrammd5.so.3 without build-id.
                                                   Module libgssapiv2.so.3 without build-id.
                                                   Module libcrypt.so.2 without build-id.
                                                   Module libplain.so.3 without build-id.
                                                   Module libscram.so.3 without build-id.
                                                   Module libanonymous.so.3 without build-id.
                                                   Module libotp.so.3 without build-id.
                                                   Module libmp3lame.so.0 without build-id.
                                                   Module libmpg123.so.0 without build-id.
                                                   Module libogg.so.0 without build-id.
                                                   Module libvorbisenc.so.2 without build-id.
                                                   Module libvorbis.so.0 without build-id.
                                                   Module libFLAC.so.12 without build-id.
                                                   Module libgpg-error.so.0 without build-id.
                                                   Module libdatrie.so.1 without build-id.
                                                   Module libbrotlicommon.so.1 without build-id.
                                                   Module libkeyutils.so.1 without build-id.
                                                   Module libkrb5support.so.0 without build-id.
                                                   Module libcom_err.so.3 without build-id.
                                                   Module libk5crypto.so.3 without build-id.
                                                   Module libkrb5.so.3 without build-id.
                                                   Module libsndfile.so.1 without build-id.
                                                   Module libvulkan.so.1 without build-id.
                                                   Module libsharpyuv.so.0 without build-id.
                                                   Module libdeflate.so.0 without build-id.
                                                   Module libLerc.so.4 without build-id.
                                                   Module liblzma.so.5 without build-id.
                                                   Module libgcrypt.so.20 without build-id.
                                                   Module libicudata.so.73 without build-id.
                                                   Module libxml2.so.2 without build-id.
                                                   Module libjson-glib-1.0.so.0 without build-id.
                                                   Module libgraphite2.so.3 without build-id.
                                                   Module libthai.so.0 without build-id.
                                                   Module libXdmcp.so.6 without build-id.
                                                   Module libXau.so.6 without build-id.
                                                   Module libgstbase-1.0.so.0 without build-id.
                                                   Module libunwind.so.8 without build-id.
                                                   Module libbrotlidec.so.1 without build-id.
                                                   Module libgssapi_krb5.so.2 without build-id.
                                                   Module libpsl.so.5 without build-id.
                                                   Module libssh2.so.1 without build-id.
                                                   Module libnghttp2.so.14 without build-id.
                                                   Module libsamplerate.so.0 without build-id.
                                                   Module libcelt0.so.2 without build-id.
                                                   Module libpulsecommon-17.0.so without build-id.
                                                   Module libnspr4.so without build-id.
                                                   Module libplc4.so without build-id.
                                                   Module libplds4.so without build-id.
                                                   Module libnssutil3.so without build-id.
                                                   Module libnss3.so without build-id.
                                                   Module libsmime3.so without build-id.
                                                   Module libssl3.so without build-id.
                                                   Module libgfxstream_backend.so without build-id.
                                                   Module libva-drm.so.2 without build-id.
                                                   Module libva.so.2 without build-id.
                                                   Module libwebp.so.7 without build-id.
                                                   Module libwebpdemux.so.2 without build-id.
                                                   Module libtiff.so.6 without build-id.
                                                   Module libsystemd.so.0 without build-id.
                                                   Module libicuuc.so.73 without build-id.
                                                   Module libxcb-shm.so.0 without build-id.
                                                   Module libxcb-render.so.0 without build-id.
                                                   Module libxcb.so.1 without build-id.
                                                   Module libXrender.so.1 without build-id.
                                                   Module libfreetype.so.6 without build-id.
                                                   Module libXinerama.so.1 without build-id.
                                                   Module libXcomposite.so.1 without build-id.
                                                   Module libXdamage.so.1 without build-id.
                                                   Module libxkbcommon.so.0 without build-id.
                                                   Module libtracker-sparql-3.0.so.0 without build-id.
                                                   Module libfribidi.so.0 without build-id.
                                                   Module libfontconfig.so.1 without build-id.
                                                   Module libpangoft2-1.0.so.0 without build-id.
                                                   Module libharfbuzz.so.0 without build-id.
                                                   Module libpango-1.0.so.0 without build-id.
                                                   Module libpangocairo-1.0.so.0 without build-id.
                                                   Module libxcb-randr.so.0 without build-id.
                                                   Module libexpat.so.1 without build-id.
                                                   Module libdrm.so.2 without build-id.
                                                   Module libpcre2-8.so.0 without build-id.
                                                   Module libffi.so.8 without build-id.
                                                   Module libselinux.so.1 without build-id.
                                                   Module libatomic.so.1 without build-id.
                                                   Module libXss.so.1 without build-id.
                                                   Module libXrandr.so.2 without build-id.
                                                   Module libXfixes.so.3 without build-id.
                                                   Module libXi.so.6 without build-id.
                                                   Module libXcursor.so.1 without build-id.
                                                   Module libXext.so.6 without build-id.
                                                   Module libgcc_s.so.1 without build-id.
                                                   Module libcap.so.2 without build-id.
                                                   Module libgmp.so.10 without build-id.
                                                   Module libhogweed.so.6 without build-id.
                                                   Module libnettle.so.8 without build-id.
                                                   Module libtasn1.so.6 without build-id.
                                                   Module libunistring.so.5 without build-id.
                                                   Module libidn2.so.0 without build-id.
                                                   Module libp11-kit.so.0 without build-id.
                                                   Module libstdc++.so.6 without build-id.
                                                   Module libsasl2.so.3 without build-id.
                                                   Module liblz4.so.1 without build-id.
                                                   Module liborc-0.4.so.0 without build-id.
                                                   Module libgstapp-1.0.so.0 without build-id.
                                                   Module libgstreamer-1.0.so.0 without build-id.
                                                   Module libopus.so.0 without build-id.
                                                   Module libbz2.so.1 without build-id.
                                                   Module libaio.so.1 without build-id.
                                                   Module libiscsi.so.10 without build-id.
                                                   Module libpipewire-0.3.so.0 without build-id.
                                                   Module libjack.so.0 without build-id.
                                                   Module libpulse.so.0 without build-id.
                                                   Module libasound.so.2 without build-id.
                                                   Module libusbredirparser.so.1 without build-id.
                                                   Module libcacard.so.0 without build-id.
                                                   Module librutabaga_gfx_ffi.so.0 without build-id.
                                                   Module libSDL2_image-2.0.so.0 without build-id.
                                                   Module libX11.so.6 without build-id.
                                                   Module libvte-2.91.so.0 without build-id.
                                                   Module libncursesw.so.6 without build-id.
                                                   Module libvdeplug.so.3 without build-id.
                                                   Module libzstd.so.1 without build-id.
                                                   Module libnuma.so.1 without build-id.
                                                   Module libseccomp.so.2 without build-id.
                                                   Module libSDL2-2.0.so.0 without build-id.
                                                   Module liblzo2.so.2 without build-id.
                                                   Module libsnappy.so.1 without build-id.
                                                   Module libusb-1.0.so.0 without build-id.
                                                   Module libudev.so.1 without build-id.
                                                   Module libfdt.so.1 without build-id.
                                                   Module libjpeg.so.62 without build-id.
                                                   Module libz.so.1 without build-id.
                                                   Module libpng16.so.16 without build-id.
                                                   Module libgnutls.so.30 without build-id.
                                                   Module libspice-server.so.1 without build-id.
                                                   Module libcapstone.so.5 without build-id.
                                                   Module libepoxy.so.0 without build-id.
                                                   Stack trace of thread 1545895:
                                                   #0  0x0000556bf0e25a24 kvm_virtio_pci_vector_use_one (.qemu-system-x86_64-wrapped + 0x536a24)
                                                   #1  0x0000556bf10183e0 memory_region_write_accessor (.qemu-system-x86_64-wrapped + 0x7293e0)
                                                   #2  0x0000556bf1017ca6 access_with_adjusted_size (.qemu-system-x86_64-wrapped + 0x728ca6)
                                                   #3  0x0000556bf1017fd1 memory_region_dispatch_write (.qemu-system-x86_64-wrapped + 0x728fd1)
                                                   #4  0x0000556bf101fa5c flatview_write_continue (.qemu-system-x86_64-wrapped + 0x730a5c)
                                                   #5  0x0000556bf101fcd3 flatview_write (.qemu-system-x86_64-wrapped + 0x730cd3)
                                                   #6  0x0000556bf1022fe9 address_space_rw (.qemu-system-x86_64-wrapped + 0x733fe9)
                                                   #7  0x0000556bf1077498 kvm_cpu_exec (.qemu-system-x86_64-wrapped + 0x788498)
                                                   #8  0x0000556bf10786e5 kvm_vcpu_thread_fn (.qemu-system-x86_64-wrapped + 0x7896e5)
                                                   #9  0x0000556bf120af58 qemu_thread_start (.qemu-system-x86_64-wrapped + 0x91bf58)
                                                   #10 0x00007fd370a85272 start_thread (libc.so.6 + 0x8e272)
                                                   #11 0x00007fd370b00c24 __clone (libc.so.6 + 0x109c24)

                                                   Stack trace of thread 1545900:
                                                   #0  0x00007fd370b00a8d writev (libc.so.6 + 0x109a8d)
                                                   #1  0x00007fd372a752ab _ZL16stream_writev_cbP9RedStreamPK5ioveci (libspice-server.so.1 + 0x4f2ab)
                                                   #2  0x00007fd372a62320 _ZN16RedChannelClient15handle_outgoingEv (libspice-server.so.1 + 0x3c320)
                                                   #3  0x00007fd372a62497 _ZN16RedChannelClient4sendEv (libspice-server.so.1 + 0x3c497)
                                                   #4  0x00007fd372a62723 _ZN16RedChannelClient4pushEv (libspice-server.so.1 + 0x3c723)
                                                   #5  0x00007fd372a628f2 _ZN16RedChannelClient14handle_messageEtjPv (libspice-server.so.1 + 0x3c8f2)
                                                   #6  0x00007fd372a61739 _ZN16RedChannelClient15handle_incomingEv (libspice-server.so.1 + 0x3b739)
                                                   #7  0x00007fd372a6283d _ZL24red_channel_client_eventiiP16RedChannelClient (libspice-server.so.1 + 0x3c83d)
                                                   #8  0x00007fd372a4d39d spice_watch_dispatch (libspice-server.so.1 + 0x2739d)
                                                   #9  0x00007fd372054e39 g_main_dispatch (libglib-2.0.so.0 + 0x5be39)
                                                   #10 0x00007fd372057fd7 g_main_context_iterate_unlocked.isra.0 (libglib-2.0.so.0 + 0x5efd7)
                                                   #11 0x00007fd37205888f g_main_loop_run (libglib-2.0.so.0 + 0x5f88f)
                                                   #12 0x00007fd372a77239 _ZL15red_worker_mainPv (libspice-server.so.1 + 0x51239)
                                                   #13 0x00007fd370a85272 start_thread (libc.so.6 + 0x8e272)
                                                   #14 0x00007fd370b00c24 __clone (libc.so.6 + 0x109c24)

                                                   Stack trace of thread 1545894:
                                                   #0  0x00007fd370afc9cf ioctl (libc.so.6 + 0x1059cf)
                                                   #1  0x0000556bf1076d8f kvm_vcpu_ioctl (.qemu-system-x86_64-wrapped + 0x787d8f)
                                                   #2  0x0000556bf1077280 kvm_cpu_exec (.qemu-system-x86_64-wrapped + 0x788280)
                                                   #3  0x0000556bf10786e5 kvm_vcpu_thread_fn (.qemu-system-x86_64-wrapped + 0x7896e5)
                                                   #4  0x0000556bf120af58 qemu_thread_start (.qemu-system-x86_64-wrapped + 0x91bf58)
                                                   #5  0x00007fd370a85272 start_thread (libc.so.6 + 0x8e272)
                                                   #6  0x00007fd370b00c24 __clone (libc.so.6 + 0x109c24)

                                                   Stack trace of thread 1545897:
                                                   #0  0x00007fd370afc9cf ioctl (libc.so.6 + 0x1059cf)
                                                   #1  0x0000556bf1076d8f kvm_vcpu_ioctl (.qemu-system-x86_64-wrapped + 0x787d8f)
                                                   #2  0x0000556bf1077280 kvm_cpu_exec (.qemu-system-x86_64-wrapped + 0x788280)
                                                   #3  0x0000556bf10786e5 kvm_vcpu_thread_fn (.qemu-system-x86_64-wrapped + 0x7896e5)
                                                   #4  0x0000556bf120af58 qemu_thread_start (.qemu-system-x86_64-wrapped + 0x91bf58)
                                                   #5  0x00007fd370a85272 start_thread (libc.so.6 + 0x8e272)
                                                   #6  0x00007fd370b00c24 __clone (libc.so.6 + 0x109c24)

                                                   Stack trace of thread 1545896:
                                                   #0  0x00007fd370afc9cf ioctl (libc.so.6 + 0x1059cf)
                                                   #1  0x0000556bf1076d8f kvm_vcpu_ioctl (.qemu-system-x86_64-wrapped + 0x787d8f)
                                                   #2  0x0000556bf1077280 kvm_cpu_exec (.qemu-system-x86_64-wrapped + 0x788280)
                                                   #3  0x0000556bf10786e5 kvm_vcpu_thread_fn (.qemu-system-x86_64-wrapped + 0x7896e5)
                                                   #4  0x0000556bf120af58 qemu_thread_start (.qemu-system-x86_64-wrapped + 0x91bf58)
                                                   #5  0x00007fd370a85272 start_thread (libc.so.6 + 0x8e272)
                                                   #6  0x00007fd370b00c24 __clone (libc.so.6 + 0x109c24)

                                                   Stack trace of thread 1545846:
                                                   #0  0x00007fd370af3b86 ppoll (libc.so.6 + 0xfcb86)
                                                   #1  0x0000556bf1220e1e qemu_poll_ns (.qemu-system-x86_64-wrapped + 0x931e1e)
                                                   #2  0x0000556bf121e71e main_loop_wait (.qemu-system-x86_64-wrapped + 0x92f71e)
                                                   #3  0x0000556bf0e64133 qemu_main_loop (.qemu-system-x86_64-wrapped + 0x575133)
                                                   #4  0x0000556bf1081456 qemu_default_main (.qemu-system-x86_64-wrapped + 0x792456)
                                                   #5  0x00007fd370a2110e __libc_start_call_main (libc.so.6 + 0x2a10e)
                                                   #6  0x00007fd370a211c9 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2a1c9)
                                                   #7  0x0000556bf0c26625 _start (.qemu-system-x86_64-wrapped + 0x337625)

                                                   Stack trace of thread 1545893:
                                                   #0  0x00007fd370af362f __poll (libc.so.6 + 0xfc62f)
                                                   #1  0x00007fd372057f2f g_main_context_iterate_unlocked.isra.0 (libglib-2.0.so.0 + 0x5ef2f)
                                                   #2  0x00007fd37205888f g_main_loop_run (libglib-2.0.so.0 + 0x5f88f)
                                                   #3  0x0000556bf10bb909 iothread_run (.qemu-system-x86_64-wrapped + 0x7cc909)
                                                   #4  0x0000556bf120af58 qemu_thread_start (.qemu-system-x86_64-wrapped + 0x91bf58)
                                                   #5  0x00007fd370a85272 start_thread (libc.so.6 + 0x8e272)
                                                   #6  0x00007fd370b00c24 __clone (libc.so.6 + 0x109c24)

                                                   Stack trace of thread 1545869:
                                                   #0  0x00007fd370acd0c5 clock_nanosleep@GLIBC_2.2.5 (libc.so.6 + 0xd60c5)
                                                   #1  0x00007fd370ad7da7 __nanosleep (libc.so.6 + 0xe0da7)
                                                   #2  0x00007fd372086e8f g_usleep (libglib-2.0.so.0 + 0x8de8f)
                                                   #3  0x0000556bf1214eec call_rcu_thread (.qemu-system-x86_64-wrapped + 0x925eec)
                                                   #4  0x0000556bf120af58 qemu_thread_start (.qemu-system-x86_64-wrapped + 0x91bf58)
                                                   #5  0x00007fd370a85272 start_thread (libc.so.6 + 0x8e272)
                                                   #6  0x00007fd370b00dec __clone3 (libc.so.6 + 0x109dec)

                                                   Stack trace of thread 1545870:
                                                   #0  0x00007fd370a81c5e __futex_abstimed_wait_common (libc.so.6 + 0x8ac5e)
                                                   #1  0x00007fd370a847d5 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x8d7d5)
                                                   #2  0x0000556bf120b0e1 qemu_cond_timedwait_ts (.qemu-system-x86_64-wrapped + 0x91c0e1)
                                                   #3  0x0000556bf120bc80 qemu_cond_timedwait_impl (.qemu-system-x86_64-wrapped + 0x91cc80)
                                                   #4  0x0000556bf12200e4 worker_thread (.qemu-system-x86_64-wrapped + 0x9310e4)
                                                   #5  0x0000556bf120af58 qemu_thread_start (.qemu-system-x86_64-wrapped + 0x91bf58)
                                                   #6  0x00007fd370a85272 start_thread (libc.so.6 + 0x8e272)
                                                   #7  0x00007fd370b00c24 __clone (libc.so.6 + 0x109c24)
                                                   ELF object binary architecture: AMD x86-64

The segfault happened in kvm_virtio_pci_vector_use_one, on thread 1545895:

$ sudo coredumpctl debug
[...]
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000556bf0e25a24 in kvm_virtio_pci_vector_use_one ()
[Current thread is 1 (Thread 0x7fd363fff6c0 (LWP 1545895))]
(gdb) where
#0  0x0000556bf0e25a24 in kvm_virtio_pci_vector_use_one ()
#1  0x0000556bf10183e0 in memory_region_write_accessor ()
#2  0x0000556bf1017ca6 in access_with_adjusted_size ()
#3  0x0000556bf1017fd1 in memory_region_dispatch_write ()
#4  0x0000556bf101fa5c in flatview_write_continue ()
#5  0x0000556bf101fcd3 in flatview_write ()
#6  0x0000556bf1022fe9 in address_space_rw ()
#7  0x0000556bf1077498 in kvm_cpu_exec ()
#8  0x0000556bf10786e5 in kvm_vcpu_thread_fn ()
#9  0x0000556bf120af58 in qemu_thread_start ()
#10 0x00007fd370a85272 in start_thread () from /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libc.so.6
#11 0x00007fd370b00c24 in clone () from /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libc.so.6

@delan
Copy link
Member Author

delan commented Sep 9, 2024

Found the bug:

The fix commit seems to be in qemu 9.1.0 only, and we have qemu 8.2.5. We’ll need to update or patch our qemu.

@delan
Copy link
Member Author

delan commented Sep 9, 2024

Fixed in d177616:

image

@delan
Copy link
Member Author

delan commented Sep 9, 2024

Injecting files into the boot volume may be impossible:

  • Linux only has read-only support for journaled HFS+ and APFS
  • You can no longer install macOS to a non-journaled HFS+ volume
    • Maybe diskutil disableJournal would work?
  • For now, let’s partition the disk into an APFS boot volume and an ExFAT data volume
    • Disk Utility can’t partition empty disks, but we can use diskutil partitionDisk instead

Enabling SSH access:

  • System Settings > General > Sharing > Remote Login; can we automate this?

Logging in automatically:

Running stuff on boot:

  • LaunchAgents or LaunchDaemons
  • Login Items can run shell scripts in Terminal.app
    • System Settings > General > Login Items; we can automate this!
    • We need to set .sh to open in Terminal.app
      • Right click shell script > Open With > Other…
      • Enable = All Applications
      • Always Open With = on
      • /Applications/Utilities/Terminal.app
      • Can we automate this?
    • All in all, probably not worth it since LaunchAgents can run Terminal.app

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant