QubesOS 4.1 installer over Heads ISO boot: blivet doesn't find mount attribute on already mounted loopback device for install when installing packages #6792
Labels
C: installer
diagnosed
Technical diagnosis has been performed (see issue comments).
P: default
Priority: default. Default priority for new issues, to be replaced given sufficient information.
r4.1-dom0-stable
T: bug
Type: bug report. A problem or defect resulting in unintended behavior in something that exists.
Milestone
Comments
tlaurion
added
P: default
andrewdavidwong
added
C: installer
needs diagnosis
|
I'll try to have a look today. |
|
Is this is specific to your hardware? Because I don't and neither openQA reproduce the issue? If so, I cannot perform a diagnostic I don't have your hardware, only @marmarek. |
|
@fepitre : This happens when booting QubesOS 4.1 latest installer under Heads when verifying iso. The important pieces are under blivet debug trace above, where for some reason a mount property is checked and returned false The ISO is already mounted, user is created, packages are ready to be installed (as seen in full logs). If the ISO is dd'ed to a USB drive, the ISO boots and installs properly, hence not hardware related. This bug is limited to booting directly from ISO (kexec) under Heads while having the ISO verified against detached signature before offering boot options and kexec'ing into the mounted ISO. As said, behavior was not present under QubesOS 4.0 installer so something changed there in that code path to validate mount attribute on DeviceFormat, which results in the error observed here. As pointed above, the whole journalctl logtrace was uploaded here |
|
Let's go into this more for the sake of clarity based on full journalctl output
@fepitre : any insights? |
|
First: 300 seconds is 5 minutes, this is 4. Thought it could be linked to timeout having occured. |
|
From my understanding, it seems blivet forgot that the iso is actually loop0 which is mounted. Why is it constantly trying to figure out the mapping and mount option and flagging detection as incomplete? Will boot QubesOS 4.0 installer and get journalctl output to compare on same test laptop. |
|
@tlaurion we can certainly workaround this but, for the sake of completeness, what happen if you try with a standard Fedora 32 ISO? |
@fepitre : Will test https://ftp.fau.de/fedora/linux/releases/32/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-32-1.6.iso |
@fepitre : installation from verified ISO happening right now. So here: Qubesos 4.0 installer going into package install: Fedora32 installer going into package install: Let me know if you have anything for me to test/report on @fepitre! |
|
@tlaurion thank you, I'll investigate tomorrow morning as much as I can (it's past 10pm around here). |
|
@fepitre : Full bug report logs as suggested by installer, dumped under remounted RW usb drive: |
I think its unrelated, but under syslog file |
This was referenced Jul 25, 2021
|
@tlaurion have you already tried from a bare Fedora/Debian as livecd to kexec Qubes OS iso install? For example, I'm attempting without any success to kexec Xen from Fedora 33/Debian 11 livecd: In this piece of code, I've removed the fromiso and iso-scan part because I originally thought it was that making hang the kexec but no. The idea is to create a similar workflow (also for reproducing your issue) until we can plug certified hardware into openQA. |
12 tasks
I never had to do that before. Can try...
Under heads, what happens on the following lines is that the ISO (first partition) after having been verified is mounted under /boot. Then The trick applied here with fromiso/iso-scan is that the initrd and kexec'ed kernel are not aware of where the iso was launched from. This is what the step 1 2 3 here are doing to find back the medium. So here, by passing UUID of the filestystem where the ISO can be found,, the kexec'ed environment can discover that device again and search it for actual referred installation media. You need to pass this iso-scan statement to replicate this under fedora, where fromiso is for debian and where other distros differs. FYI: Kexec is being patched with the following @fepitre : let me know if I can be of any other help. |
This seems to be the missing piece. With kexec-tools modified this way, I can successfully kexec into qubes installer from fedora 34 live image. Without the patch, it hangs. |
|
@tlaurion is there any chance to get this (or likely - more generalized) patch upstream? |
|
@tlaurion you can stop holding your breathe, we have reproduce it locally! Now it's debug part. |
@marmarek Here is the thread that led to that change. More specifically: linuxboot/heads#227 (comment) I thought the patch was brought upstream into Xen: You running into it means the patch has not found its way into Xen.
Current state: We use kexec patches stil waiting for patchset to be merged under Xen |
fepitre
added a commit
to fepitre/qubes-anaconda
that referenced
this issue
Jul 28, 2021
fepitre
added a commit
to fepitre/qubes-anaconda
that referenced
this issue
Jul 28, 2021
|
Ok so here is a draft of a script to generated a Qubes OS ISO embedded into a raw disk image named In order to test similar setup to Insurgo (e.g. on openQA first or locally on KVM), one has to create a virtual machine with first disk to be the destination disk for Qubes OS installation, then attaching as a second drive, #!/usr/bin/bash
## DEPENDENCIES: grub2-mkrescue, genisoimage and xorriso
set -xe
LOCALDIR="$(readlink -f "$(dirname "$0")")"
QUBES_ISO="$(readlink -f "$1")"
QUBES_ISO_NAME="$(basename "$QUBES_ISO")"
if [[ $EUID -ne 0 ]]; then
echo "ERROR: This script must be run with root permissions."
exit 1
fi
if [ ! -e "$QUBES_ISO" ]; then
echo "ERROR: Cannot find $QUBES_ISO."
exit 1
fi
exit_clean() {
local exit_code=$?
if [ -n "$TMPMOUNTDIR" ]; then
umount "$TMPMOUNTDIR" || true
rm -rf "$TMPMOUNTDIR"
fi
if [[ $LOOP_DEVICE =~ /dev/loop[0-9] ]]; then
losetup -d "${LOOP_DEVICE}" || true
fi
exit "$exit_code"
}
qemu-img create -f raw embedded_iso.img +6G
LOOP_DEVICE="$(losetup -f --show -P embedded_iso.img)"
trap 'exit_clean' 0 1 2 3 6 15
TMPMOUNTDIR="$(mktemp -d)"
if [[ $LOOP_DEVICE =~ /dev/loop[0-9] ]] && [ -n "$TMPMOUNTDIR" ]; then
printf "o\nn\n\n\n\n\nw\n" | sudo fdisk "$LOOP_DEVICE"
if [ -e "${LOOP_DEVICE}p1" ]; then
mkfs.ext4 "${LOOP_DEVICE}p1"
mount "${LOOP_DEVICE}p1" "$TMPMOUNTDIR"
cp "$QUBES_ISO" "$TMPMOUNTDIR"
cd "$TMPDIR"
mkdir -p "$TMPMOUNTDIR/boot/grub"
VOLID="$(isoinfo -d -i "$QUBES_ISO" | grep -oP 'Volume id: \K[\w-]*')"
LOOP_UUID="$(lsblk -no uuid "${LOOP_DEVICE}p1")"
## (hd0,msdos1) is the first bootable disk which will be embedded_iso.img
cat > "$TMPMOUNTDIR/boot/grub/grub.cfg" << EOF
set timeout=3
set default=0
menuentry 'Qubes OS R4.1.0 Installation' {
set isofile='(hd0,msdos1)/$QUBES_ISO_NAME'
loopback loop \$isofile
multiboot2 (loop)/isolinux/xen.gz console=com1 no-real-mode edd=off
module2 (loop)/isolinux/vmlinuz inst.stage2=hd:LABEL=$VOLID findiso=/dev/disk/by-uuid/${LOOP_UUID}/$QUBES_ISO_NAME iso-scan/filename=/$QUBES_ISO_NAME
module2 --nounzip (loop)/isolinux/initrd.img
}
EOF
cat > loop_device.map <<EOF
(hd0) ${LOOP_DEVICE}
EOF
if [ -e /usr/sbin/grub2-install ]; then
GRUB_INSTALL=/usr/sbin/grub2-install
else
GRUB_INSTALL=/usr/sbin/grub-install
fi
$GRUB_INSTALL --no-floppy --boot-directory="$TMPMOUNTDIR/boot" --grub-mkdevicemap=loop_device.map --modules="part_msdos" "${LOOP_DEVICE}"
fi
fiIn order to this setup to work, we need to have the patch provided in QubesOS/qubes-anaconda#17. |
|
I've updated my previous comment with a all in one bootable drive. |
|
Automated announcement from builder-github The package |
|
@tlaurion today's builds on https://qubes.notset.fr/iso/ contain the fix already, can you confirm it indeed works for you too? |
andrewdavidwong
added
diagnosed
|
@marmarek Installing packages from https://qubes.notset.fr/iso/Qubes-20210731-x86_64.iso |
|
@tlaurion at least we have now a similar boot case for Insurgo until we have certified laptop plugged into openQA instance :) |
marmarek
added a commit
to QubesOS/openqa-tests-qubesos
that referenced
this issue
Aug 1, 2021
This is similar setup used in HEADs to boot the installer after verification of a detached signature. QubesOS/qubes-issues#6792
marmarek
added a commit
to marmarek/anaconda
that referenced
this issue
Aug 3, 2021
This fixes checking if given (disk image) file is mounted somewhere, especially as /run/initramfs/repo. PackagePayload._device_is_mounted_as_source() depends on it. Some more details at QubesOS/qubes-issues#6792
|
Automated announcement from builder-github The package Or update dom0 via Qubes Manager. |
fepitre
pushed a commit
to fepitre/anaconda
that referenced
this issue
Oct 16, 2021
This fixes checking if given (disk image) file is mounted somewhere, especially as /run/initramfs/repo. PackagePayload._device_is_mounted_as_source() depends on it. QubesOS/qubes-issues#6792
fepitre
pushed a commit
to fepitre/anaconda
that referenced
this issue
Dec 27, 2022
This fixes checking if given (disk image) file is mounted somewhere, especially as /run/initramfs/repo. PackagePayload._device_is_mounted_as_source() depends on it. QubesOS/qubes-issues#6792
fepitre
pushed a commit
to fepitre/anaconda
that referenced
this issue
Jan 4, 2023
This fixes checking if given (disk image) file is mounted somewhere, especially as /run/initramfs/repo. PackagePayload._device_is_mounted_as_source() depends on it. QubesOS/qubes-issues#6792
fepitre
pushed a commit
to fepitre/anaconda
that referenced
this issue
Jan 4, 2023
This fixes checking if given (disk image) file is mounted somewhere, especially as /run/initramfs/repo. PackagePayload._device_is_mounted_as_source() depends on it. QubesOS/qubes-issues#6792
fepitre
pushed a commit
to fepitre/anaconda
that referenced
this issue
Jan 10, 2023
This fixes checking if given (disk image) file is mounted somewhere, especially as /run/initramfs/repo. PackagePayload._device_is_mounted_as_source() depends on it. QubesOS/qubes-issues#6792
marmarek
added a commit
to marmarek/qubes-lorax-templates
that referenced
this issue
Feb 14, 2023
stage2 defaults to inst.repo= so stage2 is is found normally. But fallback the other way around doesn't work in all the cases - specifically if the source is a loop device (mounted via iso-scan/filename= feature). See rhinstaller/anaconda#3552 for details. QubesOS/qubes-issues#6792
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Qubes OS version
QubesOS 4.1 ISO (Qubes-R4.1.0-beta1pre1-x86_64)
Affected component(s) or functionality
QubesOS Installer over Heads (USB boot of verified detached signed ISO)
Brief summary
@marmarek @fepitre : The following happens in blivet after default partitioning of the system, right before installing packages:
Full
journalctl > fileoutput hereSearch for
AttributeErrorin the whole dumped log.How reproducible
Each install attempt fails at the same point above. Also tested https://qubes.notset.fr/iso/Qubes-20210717-x86_64.iso
Steps to reproduce
Expected behavior
I am not sure why blivet checks again for mountable property of the loop device. Expected behavior would be to continue installation of packages since the ISO is properly mounted already, partitioning has happened and software packages were discovered and are ready to install.
QubesOS 4.0 installers don't show this behavior.
Actual behavior
Installation fails just before package installation with above error.
Solutions you've tried
Heads permits to boot from USB a dd'ed copy of the ISO. This works, but is untrusted (unverified).
But the goal here would be to have fepitre-bot key added into Heads distribution signing keys to validate detached signed ISO images.
Additional context
Relevant documentation you've consulted
Related, non-duplicate issues
The text was updated successfully, but these errors were encountered: