x220, add GPG key to running Bios and reflash not working

[ghost]

After booting Heads I get the GPG Management Menu,but add GPG to running menu wont't work.I found this #675 .I can get flashrom working internally following this guide,but I am stuck at the and 6. step not knowing how to add checksums.Also not quite sure about what to do at step 4.After step 3 I get this output
https://postimg.cc/S2zKRgK3
At step 4 I ran bin/flash.sh -c chipname and the output was same like on the uploaded pic.Could some explain how I can get this thing working?

[tlaurion]

@MrChromebox seems like last version of flashrom dropped support for x220? @SebastianMcMillan ?

Current flashrom statement is https://github.com/osresearch/heads/blob/f23ced0a3bef2b65384b1ad6668f62ed2c9be633/boards/x220/x220.config#L39

looking to see if some patches were applied in tree in the past. I remember the SPI chip was specified in the past. I remember maybe falsely that past flashrom upgrade made it irrelevant to specify (since different SPI chips would need to be specified and tested in loop, but I do not own hardware)

[tlaurion]

@userongihu @MrChromebox @SebastianMcMillan :
past statement for that board was :
https://github.com/osresearch/heads/blame/83c22f3e4a154707b8cc00f1267d6591b6c24e41/boards/x220/x220.config#L36
export CONFIG_FLASHROM_OPTIONS="--force --noverify-all -p internal:laptop=force_I_want_a_brick,ich_spi_mode=hwseq --ifd --image bios"
At that time (flashrom 1.0) there was this patch required for the x220: https://github.com/osresearch/heads/blob/83c22f3e4a154707b8cc00f1267d6591b6c24e41/patches/flashrom-1.0/0101-enable-thinkpad-x220.patch

Prior to that it was :
https://github.com/osresearch/heads/blame/8e23a54f284b3847e973b70c4cecbe36acdfd19f/boards/x220/x220.config
export FLASHROM_OPTIONS='--force --noverify-all -p internal:laptop=force_I_want_a_brick,ich_spi_mode=hwseq --ifd --image bios'
At that time (flashrom 1.0) there was still this patch required for the x220: https://github.com/osresearch/heads/blob/83c22f3e4a154707b8cc00f1267d6591b6c24e41/patches/flashrom-1.0/0101-enable-thinkpad-x220.patch

Where prior of that, x220 was not officially supported and flashrom statements were included in flash.sh, where it was absent:
https://github.com/osresearch/heads/blob/bcf522cb2e40b932383bb4b06bf5564776216c93/initrd/bin/flash.sh

[MrChromebox]

@tlaurion flashrom 1.2 removed the need for laptop=force_I_want_a_brick and for most platforms ich_spi_mode=hwseq too, but I'm guessing the x220 still needs it (so -p internal:ich_spi_mode=hwseq).

[MrChromebox]

note that with hardware sequencing enabled, there is no need to specify the chip via -c xxx

[ghost]

thank you for your answers.Does it means that wherever flashing is used "-p internal:ich_spi_mode=hwseq" must be added to the flashing command?

[MrChromebox]

@userongihu it means that CONFIG_FLASHROM_OPTIONS needs to be updated for the board config to amend the parameter list.

for updating your device currently (after building with the updated params), would recommend simply flashing from the recovery shell:
mount-usb
flashrom -p internal:ich_spi_mode=hwseq -f -n --ifd -i bios -w /media/<filename to flash>
reboot

[ghost]

Thank you MrChromebox.Do I get it right that you updated the missing parameters?So all I need to do is make a new build running "make BOARD=x220" and then running the commands you mentioned above?Sorry for asking such a noob question...

[MrChromebox]

@userongihu I submitted a patch (pull request) to add the missing param, but it has not been accepted (merged) yet, so until that time you would need to make the change manually on your local copy before rebuilding

[tlaurion]

@userongihu please try
flashrom -p internal:ich_spi_mode=hwseq -f -n --ifd -i bios -w /media/<filename to flash>
If you report this as successful, I will merge #871 to fix problem in newer builds.

Please also tag me with result so that I can see it right away.

[tlaurion]

@SebastianMcMillan ?

[ghost]

@tlaurion ,here is the output of the flash https://postimg.cc/tYvk7pLS/2898bc5f
I rebooted,then tried to add GPG key to running Bios but it still won't flash.I am going to
change the flash-params in the configs,rebuild and try again.

[ghost]

When rebooting after flashing there are still no GPG keys on the keyring.Setting up TOTP works.After rebooting TOTP also persists,saying because I could get GPG to the key ring ,but after a reboot the keys were somehow gone.Also when hitting the option "update checksums and sign all files in /boot" it requires a gpg-card,but it doesn't recognize my card when plugged in,although Heads recognizes the card insofar,that it is possible to generate PGP keys with card.Maybe I have to add
the device to the configparams before building.Also I do have now a persisting "thinkpad x220 boot menu" that wasn't there at the beginning.

Update:@tlaurion,I rebuilt and ran flashrom with the updated flashromparams that you suggested.Rebooted and tried "add PGP keys to running Bios and reflash" but failed and it returned to the usual screen.But after running "add PGP keys to standalone Bios and reflash" option it returned to a screen that says "TOTP secret cannot be unsealed".I never got that screen before.But I keep on getting the message "No GPG keys in the key ring".When running bin/gpg-gui.sh it imports keys https://postimg.cc/n9ryQZ1H.The error message "No Board....." below,I am getting only with the build that has the new flashparams.

[ghost]

To be more specific the screen said "ERROR:TOTP generation failed!" not "TOTP secret cannot be unsealed".

[ghost]

https://postimg.cc/n9ryQZ1H isn't working anymore,see https://postimg.cc/YGvJ9Gmq instead.

[tlaurion]

@userongihu please try
flashrom -p internal:ich_spi_mode=hwseq -f -n --ifd -i bios -w /media/<filename to flash>
If you report this as successful, I will merge #871 to fix problem in newer builds.

Please also tag me with result so that I can see it right away.

@userongihu: Ok. I'm confused.
So following :

mount-usb
flashrom -p internal:ich_spi_mode=hwseq -f -n --ifd -i bios -w /media/<filename to flash>
reboot

You were able to flash new rom manually from recovery shell. Good. So this means that the flash parameters works.
What rom have you flashed?

On host buildsystem:

cd heads
git fetch origin
git reset --hard
make BOARD=x220 CPUS=2
(should work unless gpg2 toolstack now is too big. We do not have enough x220 boards partitipants, nor the board being under CI for checking this kind of regressions right now, so it might fail.)
sudo mount /dev/sdX /media
cp build/x220/heads-x220-.....gcd6ba01.rom /media
sudo umount /media

On your x220:

mount-usb
flashrom -p internal:ich_spi_mode=hwseq -f -n --ifd -i bios -w /media/<filename to flash>
reboot

Then on next reboot, all commands requiring to access the rom by doing read/write commands through flashrom will take the new :
https://github.com/osresearch/heads/blob/master/boards/x220/x220.config#L39
(Which is required when adding GPG public key, updating firmware internally...)

x220/t420/xx20 boards testers (@SebastianMcMillan @techge @eganonoa @shamen123 @Thrilleratplay)
cd6ba01
and
e3519f2
were merged without being tested per board owners.

[snmcmillan]

I've been having tons of issues actually building heads, which is why I haven't been able to do anything right now. Building in anything newer than Debian 10 straight-up is broken and I've got no time right now to set up a Debian 10 system. I'll get a Debian 10 system setup this weekend for me to take a look though. However, here's my understanding: GPG key is added and flashed, but is lost on reboot. This sounds like an issue we had a while back where there wasn't enough space to add the key.

…

On Tue, Oct 27, 2020, 09:53 tlaurion ***@***.***> wrote: @userongihu <https://github.com/userongihu> please try flashrom -p internal:ich_spi_mode=hwseq -f -n --ifd -i bios -w /media/<filename to flash> If you report this as successful, I will merge #871 <#871> to fix problem in newer builds. Please also tag me with result so that I can see it right away. Ok. I'm confused. So following : mount-usb flashrom -p internal:ich_spi_mode=hwseq -f -n --ifd -i bios -w /media/<filename to flash> reboot You were able to flash new rom manually from recovery shell. Good. What rom have you flashed? On host buildsystem: cd heads git fetch origin git reset --hard make BOARD=x220 CPUS=2 (should work unless gpg2 toolstack now is too big. We do not have enough x220 boards partitipants, nor the board being under CI for checking this kind of regressions right now, so it might fail.) sudo mount /dev/sdX /media cp build/x220/heads-x220-.....gcd6ba01.rom /media sudo umount /media On your x220: mount-usb flashrom -p internal:ich_spi_mode=hwseq -f -n --ifd -i bios -w /media/<filename to flash> reboot Then on next reboot, all commands requiring to access the rom by doing read/write commands through flashrom will take the new : https://github.com/osresearch/heads/blob/master/boards/x220/x220.config#L39 (Which is required when adding GPG public key, updating firmware internally...) x220/t420/xx20 boards testers ***@***.*** <https://github.com/SebastianMcMillan> @techge <https://github.com/techge> @eganonoa <https://github.com/eganonoa> @shamen123 <https://github.com/shamen123> @Thrilleratplay <https://github.com/Thrilleratplay>) cd6ba01 <cd6ba01> and e3519f2 <e3519f2> were merged without being tested per board owners <#692>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#870 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFNTUNHEFGU63A35UCLVJ33SM3NIHANCNFSM4S7NPE2A> .

[ghost]

@tlaurion,what I did is going to board/x220/x220.config and changed the CONFIG_FLASHROM_OPTIONS to the command you suggested.Then I ran "make BOARD=x220" and after quite a while the build was ready without any error notification.From shell I ran "mount-usb"(the newly build coreboot.rom was on the USBdrive of course) and then ran the flashing command,which gave me a positive
output,means no errors.Then rebooted straight into the "ERROR:TOTP generation failed" ,but I could set up the TOTP thing,means a 6digit number gets generated whenever I boot up,tried it already a couple of times.
Now whenever I boot up the first srceen says ERROR:GPG keyring empty,then when trying "add key to running Bios and reflash" flash is not working ,instead it changes
back to the ERROR:GPG keyring empty" screen.When running "add key to standalone Bios and flash" it doesn't flash but changes into the"x220 boot menu"
screen instead.

[MrChromebox]

@userongihu run gpg-gui.sh from the recovery shell, and choose the option to add the key and flash. Should give more error info when it fails and drops back to shell vs main menu

[ghost]

I flashed the corebootrom.rom that was in build/x220 after the buil was finished

[ghost]

@MrChromebox,I just ran that command and the output is the same like on here
https://postimg.cc/YGvJ9Gmq

[MrChromebox]

@userongihu you're either using an old build, or building from old sources. The output filename should be `heads-X220-[build version]-[commit hash].rom

[ghost]

From which source should I compile the right build?Should I git clone from this link https://github.com/osresearch/heads.git ?

[ghost]

I followed the instructions posted here http://osresearch.net/

[ghost]

Are these 2 commands necessary to get the latest build: "git fetch origin" and
"git reset --hard" ?

[MrChromebox]

@userongihu git fetch && git reset --hard origin/master

if you don't specify what to reset to, then it only resets any uncommitted changes to your current repo/branch, it doesn't reset to the latest remote sources

edit: git log --oneline -1 should report: boards/x220: update flashrom parameters as the latest commit (as of this comment)

[ghost]

@MrChromebox,just rebuild following the guide mentioned above but the build was as usual
build/x220/coreboot.rom.Am I missing something?git cloned into git clone https://github.com/osresearch/heads and sticked to the steps shown above.

[tlaurion]

@userongihu https://github.com/osresearch/heads-wiki/edit/master/Installing-and-Configuring/Flashing-Guides/x220.md

That guide is not yet written but insights are present under https://github.com/osresearch/heads/blob/master/blobs/x220/readme.md

Reviewing it, as a board I do not possess, shows that the line https://github.com/osresearch/heads/blob/master/blobs/x220/extract.sh#L58 is pretty incomplete for a board lacking required space, where:
$MECLEAN -O "$BLOBDIR/me.bin" -r -t "$extractdir/flashregion_2_intel_me.bin" is actually relocating binaries and trimming, which desactivate ME and actually neuters it.... But that space is not made availabe for the BIOS region since not rewritten into the IFD so BIOS region can take advantage of it. Prior of going forward, those instyrcutiosn need to be written, troubelshoot, and reviewed by the xx20 community.

Where:
http://osresearch.net/Clean-the-ME-firmware/#how-to-disabledeactive-most-of-it
python ~/me_cleaner/me_cleaner.py -r -t -d -S -O clean_flash.bin down.rom.new --extract-me extracted_me.rom is barely sufficient on x230 (12mb flash) where x220 is 8mb.

The instructions from cleaning should be based on https://github.com/corna/me_cleaner/wiki/Internal-flashing-with-coreboot#neutralize-and-shrink-intel-me per explained reasons there:
python me_cleaner.py -r -t -d -O out.bin -D ifd_shrinked.bin -M me_shrinked.bin original_dump.bin
Which is the only way to reuse liberated ME space.

You can base yourself in my personal PoC to have externally flashable full ROMs from #703.

The result would be expended BIOS region, for which CBFS region (in coreboot) should match space freed from neutered ME region.

Otherwise, by applying plain original instructions, ME is DEACTIVATED, not neutered. So the space that could be cleaned from ME SHOULD be transferred to BIOS region, where a modified IFD would permit that transfer.

I would entice any official xx20 heads user (@SebastianMcMillan @techge @eganonoa @shamen123 @Thrilleratplay) to draft an installation guide, similar to http://osresearch.net/x230-flashing/ so that this board doesn't die from absence of maintenance. And then could be transferred to other xx20 boards.

I do not own the board and do not have the time to do this myself and not test what I produce here.
Community, your turn to play.

Currently:

• I do not know how you neutered ME nor if IFD has this space, nor if xx20 coreboot config CBFS declared space matches the instructions you followed.
• I do not know if you CBFS defined by coreboot doesn't match available space.
• I do not know from which original Lenovo BIOS yo ustarted from, nor if you have total available space.

We need to have current base and have #873 to know if current commit is actually flashable when non-existent instreuctions are followed by everyone. I'm sorry @userongihu for this learning curve. But the x220 is the lesser SPI space version of the x230, where a lot less people are testing changes as they come. And... progress should continue.

Heads project could force its users to unsolder and solder a 32mb flash chip to replace the actual 8mb SPI chip. But supporting 8mb chip where I personally struggle supporting the x230 (12mb chip) where other boards have 16mb and 32mb SPI chips.... becomes quite difficult without user+dev involvement from xx20 platforms owners.

[tlaurion]

@SebastianMcMillan @techge @eganonoa @shamen123 @Thrilleratplay @userongihu :
Please test and report on #877

[ghost]

@tlaurion,I just booted a live-distro from usb and in order to make a regular install from installer.If that doesn't work I will try it the way it is shown on the link you posted.I will report on how things went.

[tlaurion]

Know that luks should be in version 1, not luks2 since not supported by Heads for the moment until opened issue is fixed.

Qubesos latest stable works

[ghost]

@tlaurion ,I just finished the Qubes Install.I chose the latest version.It ended without error messages but booting it doesn't work somehow.The regular install that I tried yesterday doesn't work either.USB booting a non-live distro does work,no problems.Easy like booting a live-distro.What do you mean that I have to consider luks1 for luks2 is currently not supported
by heads?Also,after the install that I made yesterday,after booting and hitting "default boot"
I get a message that says "ERROR:missing hash files",then when choosing "uodate checksums and sign all files in /boot" it takes me to a screen that requires to plug in and confirm a GPGcard.After plugin in and confirming the GPGcard it takes ages to get a messages that says "gpg:no default secret key ,signing failed.I got that message already before,since I noticed that the card has only limited function on heads.Some tasks were possible some not.So,/boot is now not signed anymore ,which doesn't matter much because booting from HDD wasn't possible in first place.How to proceed from here?

[ghost]

@tlaurion,I put the Qubes***.iso ans Qubes***.iso.asc and qubes***.key.asc on a usbdrive
and imported the key and ran usb-scan.It showed the Qubes***.iso,from there chose the option to install,waited until finished and rebooted,but it is not working.Heads somehow does not recognize the HDD,although the installation doesn't give out any errors.
The TOTP on the x220 and the TOTP on the authenticator-app of the smartphone keep on generating the same numbers.Does that mean that the Bios is in a "good-signatured",non-tampered state?Asking just to be on the safe side.Don't want to make wrong assumptions.
.

[tlaurion]

@tlaurion ,I just finished the Qubes Install.I chose the latest version.It ended without error messages but booting it doesn't work somehow.

@userongihu Which QubesOS version?

• Supported : LUKSv1 from default installation disk partitioning option
• Unuspported: LUKSv1 from default partitioning options because of cryptsetup under Heads being 1.75 and not supporting LUKS2

The regular install that I tried yesterday doesn't work either.USB booting a non-live distro does work,no problems.Easy like booting a live-distro.What do you mean that I have to consider luks1 for luks2 is currently not supported
by heads?Also,after the install that I made yesterday,after booting and hitting "default boot"
I get a message that says "ERROR:missing hash files",then when choosing "uodate checksums and sign all files in /boot" it takes me to a screen that requires to plug in and confirm a GPGcard.After plugin in and confirming the GPGcard it takes ages to get a messages that says "gpg:no default secret key ,signing failed.

So the public key was not injected in ROM? I'm not sure I understand the path here. Current code is detecting:
IF no OS installed + no public key in ROM = prompt user to add key/generate key in ROM. So it seems that you went to install QubesOS without having first a public key injected in ROM? please detail.

I got that message already before,since I noticed that the card has only limited function on heads.
Could you please detail what you mean by this? Required GPG functions are supported (signing/verified/encrypt).

Some tasks were possible some not.So,/boot is now not signed anymore ,which doesn't matter much because booting from HDD wasn't possible in first place.How to proceed from here?

If I underatand well, you do not have a public key in ROM. You have to go in GPG menu, and generate key. You can also go in main menu as of now (until #874 ) and generate key, inject, totp/hotp seal (you do not have HOTP support in rom as of now. Which USB security donlge are you using), and sign /boot content.

When /boot config is signed, the next step is to define a boot default under boot options, which will give you the possibility to define a default boot option and should prompt for setting a Disk Unlock Key passphrase ,which would make the TPM release the Disk Unlock Key by the TPM if TPM measurements are valid.

[ghost]

@tlaurion,I installed Qubes 4.0.3
When booting heads,going into "GPG option" and looking in "list GPG on your keyring" there is
the GPGkey which was already there,since I got heads successfully working.It is still the same GPGkey.This might answer your question wether my GPGkey was injected in the ROM or not.

Since yesterday /boot is not signed anymore and trying to sign requires a gpg-card to inserted and confirmed.I plug in my nitrokey-card and after quite some time there is a message saying
gpg:no default secret key ,signing failed",although there keys on the card,but not the GPGkey
which is on the heads-keyring.
Before yesterdays try to make a regular install inside heads my /boot was signed.I was using an OS that I installed before having installed heads.And I did try a ot of times to set up default
boot but the HDD never showed up anywhere.That's why I think that heads does not recognize the HDD,if it did recognize HDD it would show up somewhere,since USBdrives are showing uo under "USB...".
Should I make a fresh OS-installation on another computer and reflash heads internally with the "add GPGkeys to running Bios and reflash" and then start from new in order to see how things work?Asking because till yesterday the /boot was definitly signed.

[MrChromebox]

Should I make a fresh OS-installation on another computer and reflash heads internally with the "add GPGkeys to running Bios and reflash" and then start from new in order to see how things work?Asking because till yesterday the /boot was definitly signed.

no. I would simply perform the OEM reset from the options menu, and let Heads generate a fresh key and sign everything, and make sure that works. That rules out any issues from your key or it being in the correct places.

[ghost]

@tlaurion,just ran "add GPGkey to running Bios and reflash" and the internal flash worked.Made a TMP reset,which worked as well.But hitting "default boot",I get a message
that says "ERROR:missing hashfile",then selecting "yes" it aks for the GPGcard.But why?My GPGkey is already on the heads-keyring.Should I import the heads- GPGkey to the GPGcard?

[ghost]

@MrChromebox ,I am just reading your comment and I already made the reflash.Should I still make the OEM reset?

[MrChromebox]

just do the OEM factory reset, nothing else. Follow instructions upon completion and generate a new HOTP secret after rebooting. Booting the OS should work after that.

[ghost]

I did the OEM and I got the message that an TOTP secret could not be generated and I should flash the Bios.I did that and somehow I am back on where I started.

[ghost]

On the GPGkeyring is now the OEM key

[ghost]

Still getting the "ERROR:missing hashfile" message.

[MrChromebox]

I got the message that an TOTP secret could not be generated and I should flash the Bios

when did you get this message? you're making this a bit harder by not clearly communicating the steps you are performing and the errors returned.

[ghost]

@MrChromebox,followed the steps and it worked.The OS is up and running!
Is there anything else that I have to configure as far as Heads is concerned?

[ghost]

Right now Qubes is doing the initial setup.

[ghost]

Qubes finished the initial set up and after typing in the userpasswd I got into the Desktop and internet connection worked.Then I shutdown the x220.Then I powered up and hit default boot
and I got the message "The following files failed the verification process:/boot/grub2/boot.cfg This could indicate a compromise! Would you like to update your checksums now?"
I hit "yes" and it demanded to plug in and confirm the GPGcard.Thats what I did.I typed in the GPGcard passwd word and got a confirmation.Finally I have now the "select boot option" message on the screen and there are 4 different options to choose from.Which one should I choose?I am asking because Qubes OS is new to me and I am not quite sure which option is the right one.

[MrChromebox]

Then I powered up and hit default boot
and I got the message "The following files failed the verification process:/boot/grub2/boot.cfg This could indicate a compromise! Would you like to update your checksums now?"

this is expected after the first boot of Qubes, since it updates the grub config

Finally I have now the "select boot option" message on the screen and there are 4 different options to choose from.Which one should I choose?

Heads orders the boot options in the same order as they appear in the grub config. Generally speaking, the first / shortest entry is the correct one, but there are exceptions (like with some live USBs)

[tlaurion]

I consider this issue resolved where others more specific should be opened.
@userongihu : can you report under #877 if that worked for you.

Closing here.