Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

loader.efi netboot regression in 2023.11 #1970

Open
nwf-msr opened this issue Dec 15, 2023 · 10 comments
Open

loader.efi netboot regression in 2023.11 #1970

nwf-msr opened this issue Dec 15, 2023 · 10 comments

Comments

@nwf-msr
Copy link
Contributor

nwf-msr commented Dec 15, 2023

With apologies, I hadn't tested netbooting this release on the MSRC cluster. It looks like we've got a regression. Using the https://github.com/microsoft/msr-morello-automation tooling using HTTP netboot, 2022.12 boots fine, but 2023.11 (built locally) dies at the very end of loader.efi with the below wad of complaints. I have tested on both the 1.5 and 1.7 releases of Morello firmware, with no apparent difference. Fortunately, loader_lua.efi from 2022.12 continues to function, so that's pretty convenient as workarounds go.

/boot/kernel/geom_eli.ko text=0x57f6 text=0xac78 data=0x1050+0x170 TcpOnAppAbort: connection reset issued by application for TCB FC2D3E98
0x8+0x2148TcpOnAppAbort: connection reset issued by application for TCB FC2D3E98
TcpInput: Discard a packet
TcpInput: Discard a packet
TcpInput: Discard a packet
+0x
8+0x1656
TcpOnAppAbort: connection reset issued by application for TCB FC2D3E98
Using DTB provided by EFI at 0xfd7a6000.
ConvertPages: Incompatible memory types, the pages to allocate have been allocated


Synchronous Exception at 0x00000000F327C740


Synchronous Exception at 0x00000000F327C740
PC 0x0000F327C740
PC 0x0000F7D310DC
PC 0x0000F7D3A498
PC 0x0000F7D3ACE8
PC 0x0000F7D709D0
PC 0x0000F7D4D374
PC 0x0000F7D6555C
PC 0x0000F7D4D5CC
PC 0x0000F7D4C56C
PC 0x0000F7D4DB54
PC 0x0000F7D499C8
PC 0x0000F7D46770
PC 0x0000F7D3C76C
PC 0x0000F7D707C8
PC 0x0000F7D4D0A0
PC 0x0000F7D673A8
PC 0x0000F7D4D5CC
PC 0x0000F7D4C56C
PC 0x0000F7D4DB54
PC 0x0000F7D499C8
PC 0x0000F7D468A0
PC 0x0000F7D3C4C0
PC 0x0000F7D34124
PC 0x0000F7D320A4
PC 0x0000F7D3A550
PC 0x0000F7E32044 (0x0000F7E1D000+0x00015044) [ 1] UiApp.dll
PC 0x0000F7E38BD0 (0x0000F7E1D000+0x0001BBD0) [ 1] UiApp.dll
PC 0x0000FD545A10 (0x0000FD52D000+0x00018A10) [ 2] SetupBrowser.dll
PC 0x0000FD548D98 (0x0000FD52D000+0x0001BD98) [ 2] SetupBrowser.dll
PC 0x0000FD5337B4 (0x0000FD52D000+0x000067B4) [ 2] SetupBrowser.dll
PC 0x0000F7E2CEF4 (0x0000F7E1D000+0x0000FEF4) [ 3] UiApp.dll
PC 0x0000F7E2CDC4 (0x0000F7E1D000+0x0000FDC4) [ 3] UiApp.dll
PC 0x0000F7E1ED4C (0x0000F7E1D000+0x00001D4C) [ 3] UiApp.dll
PC 0x0000F7E1E5F4 (0x0000F7E1D000+0x000015F4) [ 3] UiApp.dll
PC 0x0000FD76E2CC (0x0000FD74F000+0x0001F2CC) [ 4] DxeCore.dll
PC 0x0000FD51C074 (0x0000FD50D000+0x0000F074) [ 5] BdsDxe.dll
PC 0x0000FD51FCC4 (0x0000FD50D000+0x00012CC4) [ 5] BdsDxe.dll
PC 0x0000FD516EAC (0x0000FD50D000+0x00009EAC) [ 5] BdsDxe.dll
PC 0x0000FD75AE60 (0x0000FD74F000+0x0000BE60) [ 6] DxeCore.dll
PC 0x0000FD751B80 (0x0000FD74F000+0x00002B80) [ 6] DxeCore.dll
PC 0x0000FD75181C (0x0000FD74F000+0x0000281C) [ 6] DxeCore.dll
[ 1] /builds/morello/manifest/bsp/uefi/edk2/Build/morellosoc/DEBUG_CLANGDWARF/AARCH64/MdeModulePkg/Application/UiApp/UiApp/DEBUG/UiApp.dll
[ 2] /builds/morello/manifest/bsp/uefi/edk2/Build/morellosoc/DEBUG_CLANGDWARF/AARCH64/MdeModulePkg/Universal/SetupBrowserDxe/SetupBrowserDxe/DEBUG/SetupBrowser.dll
[ 3] /builds/morello/manifest/bsp/uefi/edk2/Build/morellosoc/DEBUG_CLANGDWARF/AARCH64/MdeModulePkg/Application/UiApp/UiApp/DEBUG/UiApp.dll
[ 4] /builds/morello/manifest/bsp/uefi/edk2/Build/morellosoc/DEBUG_CLANGDWARF/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll
[ 5] /builds/morello/manifest/bsp/uefi/edk2/Build/morellosoc/DEBUG_CLANGDWARF/AARCH64/MdeModulePkg/Universal/BdsDxe/BdsDxe/DEBUG/BdsDxe.dll
[ 6] /builds/morello/manifest/bsp/uefi/edk2/Build/morellosoc/DEBUG_CLANGDWARF/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll

  X0 0x00000000F7DF0C80   X1 0x00000000F327C740   X2 0x0000000000000003   X3 0x00000000F7DF0CB0
  X4 0x0000000000000000   X5 0x0000000000000041   X6 0x00000000F7D875E0   X7 0xFF9F000000000B3F
  X8 0x00000000FFFFFFFB   X9 0x0000000000000075  X10 0x00000000F7DE09F8  X11 0xFEFEFEFEFEFEFEFF
 X12 0x0000000000000008  X13 0x0000000000000000  X14 0x00000000F327C490  X15 0x0000000000000030
 X16 0x0000000000000000  X17 0x0000000000000020  X18 0x0000000000000000  X19 0x00000000F22012B0
 X20 0x0000000000001000  X21 0x0000000000000002  X22 0x00000000F7DF0D18  X23 0x00000000F7DF0D38
 X24 0x00000000F32796B0  X25 0x0000000000000001  X26 0x00000000F3278AA0  X27 0x00000000010000C4
 X28 0x00000000F3278A80   FP 0x00000000F7DF0CA0   LR 0x00000000F7D310DC

  V0 0xAFAFAFAFAFAFAFAF AFAFAFAFAFAFAFAF   V1 0x00000000F7DF06A0 00000000F7DF0720
  V2 0x0000000000000000 0000000000000000   V3 0x0000000000000000 0000000000000000
  V4 0x0000000000000000 0000000000000000   V5 0x0000000000000000 0000000000000000
  V6 0x0000000000000000 0000000000000000   V7 0x0000000000000000 0000000000000000
  V8 0x0000000000000000 0000000000000000   V9 0x0000000000000000 0000000000000000
 V10 0x0000000000000000 0000000000000000  V11 0x0000000000000000 0000000000000000
 V12 0x0000000000000000 0000000000000000  V13 0x0000000000000000 0000000000000000
 V14 0x0000000000000000 0000000000000000  V15 0x0000000000000000 0000000000000000
 V16 0x0000000000000000 0000000000000000  V17 0x0000000000000000 0000000000000000
 V18 0x0000000000000000 0000000000000000  V19 0x0000000000000000 0000000000000000
 V20 0x0000000000000000 0000000000000000  V21 0x0000000000000000 0000000000000000
 V22 0x0000000000000000 0000000000000000  V23 0x0000000000000000 0000000000000000
 V24 0x0000000000000000 0000000000000000  V25 0x0000000000000000 0000000000000000
 V26 0x0000000000000000 0000000000000000  V27 0x0000000000000000 0000000000000000
 V28 0x0000000000000000 0000000000000000  V29 0x0000000000000000 0000000000000000
 V30 0x0000000000000000 0000000000000000  V31 0x0000000000000000 0000000000000000

  SP 0x00000000F7DF0C00  ELR 0x00000000F327C740  SPSR 0x80000209  FPSR 0x00000000
 ESR 0x02000000          FAR 0x90A84966BC426F81

 ESR : EC 0x00  IL 0x1  ISS 0x00000000

Stack dump:
  00000F7DF0B00: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
  00000F7DF0B20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
  00000F7DF0B40: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
  00000F7DF0B60: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
  00000F7DF0B80: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
  00000F7DF0BA0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
  00000F7DF0BC0: 0000000000000000 0000000000000000 00000000FD596A9C 0000000000000308
  00000F7DF0BE0: 0000000000000000 0000000002000000 90A84966BC426F81 00000000F22012B0
> 00000F7DF0C00: 0000000000000000 0000000000000000 00000000F7DF0CE0 00000000F7D88404
  00000F7DF0C20: 00000000F3278A80 00001000010000C4 00000000F3278AA0 00000000F3278AA8
  00000F7DF0C40: 00000000F7DF0C80 00000000F7D8867C 00000000F327C450 0000000000000000
  00000F7DF0C60: 00000000F3229B80 0000000000000003 0000000000000002 00000000F7DF0D00
  00000F7DF0C80: 00000000F7DF0CB0 00000000F7D86700 00000000F327C400 0000000000000000
  00000F7DF0CA0: 00000000F7DF0D20 00000000F7D3A498 00000000F3278A80 00000000010000C4
  00000F7DF0CC0: 00000000F3278AA0 00000000F3278AA8 00000000F32796B0 00000000F7D7098C
  00000F7DF0CE0: 00000000F3278AE0 0000000000000002 00000000EC600800 00000000F327C740
ASSERT [ArmCpuDxe] DefaultExceptionHandler.c(333): ((BOOLEAN)(0==1))
@jrtc27
Copy link
Member

jrtc27 commented Dec 15, 2023

If it's dying in loader that's an upstream bug, we don't turn anything CHERI on there. Can you please verify it's reproducible with 14.0-RELEASE and/or recent 15-CURRENT?

@nwf-msr
Copy link
Contributor Author

nwf-msr commented Dec 15, 2023

Yes, this is an upstream regression. Bisection (full sequence of tests below) points at 75b7d39e ("stand: efi_fmtdev can be reduced to devformat") being the culprit, though at a glance it's hard to see why.

git bisect bad 525ecfdad597980ea4cd59238e24c8530dbcd31d
git bisect good fc952ac2212b121aa6eefc273f5960ec3e0a466d
git bisect good 8824cbace389c440394bb9ea6c127d0f8f85538b
git bisect good 3fc5f6b0abf5d5e57a7f171a87b1fb7de63b096f
git bisect good 1c915de99108487f58f338ddf1287bda489cb73f
git bisect bad efbf827ae461a66402acb191c036623ce1ee608f
git bisect good 16ccda32623a98cf4a2ab3804b5ae6277c01395e
git bisect bad 03c44354fc9e82f89d6e323d5f20ab953854812d
git bisect good 09a31487c1bdcb5dc8780fdd7e78540d686a3854
git bisect good 9b2453430a8dc49d34be7560ad71ff568b973672
git bisect good 5c505b9d8166d997015d3e841846b5222375a1e7
git bisect good ec6d9023e56aebe0c3fe8fb7d15512097a20df8a
git bisect bad 26e5e3ecbd61d82b369a0ce6a83297e95526a691
git bisect good e9db6f25b5a6bb9ca76351c88720cca2ffcb6604
git bisect bad e36bd9b6cffb13f2ff50bd3551ff20188d5b8099
git bisect bad 75b7d39e116f79563ea3d8d18f9bf3141f89a712

@brooksdavis
Copy link
Member

It's a bit odd that you've hit the MFC commit to stable/13 rather than the one to main...

I will say it's not clear those functions are equivalent except in the simplest cases and the commit doesn't explain why only those cases can happen.

@bsdimp
Copy link
Contributor

bsdimp commented Dec 15, 2023

devformat should produce exactly the same results. If not, it's a bug in the dev->d_dev formatting routine (which should default to the default: case for http booting. devformat uses devsw->dv_fmtdev if it exists, and defaults to

        snprintf(name, sizeof(name), "%s%d:", d->d_dev->dv_name, d->d_unit);

if not, which is the same as the default case which was removed i 75b7d3...

-   default:
-           sprintf(buf, "%s%d:", dev->d_dev->dv_name, dev->d_unit);
-           break;

I think only DEVT_NONE type devices are different.

Since only a few devices in the devsw have fmtdev, they should be teh same. disk has a different one, and zfs has a different one as well....

And I know others have network booted.

One could revert that one change, and then unrevert the replaced calls to efi_fmtdev one at a time (there's only 3) to see which one goes south, and what the devdesc that's passed into devformat() function looks like.

@bsdimp
Copy link
Contributor

bsdimp commented Dec 15, 2023

Also, it would be nice to get a symbolic traceback on what's happening. I have some libtraceback code written, but it doesn't quite work so I've not pushed it into Upstream FreeBSD... but knowing which of these calls to efi_fmtdev dies might suffice.

@nwf-msr
Copy link
Contributor Author

nwf-msr commented Dec 15, 2023

Manually symbolizing...

Synchronous Exception at 0x00000000F327C740
PC 0x0000F327C740 
PC 0x0000F7D310DC bi_load
PC 0x0000F7D3A498 elf64_exec
PC 0x0000F7D3ACE8 command_boot
PC 0x0000F7D709D0 lua_perform
PC 0x0000F7D4D374 luaD_precall
PC 0x0000F7D6555C luaV_execute
PC 0x0000F7D4D5CC luaD_callnoyield
PC 0x0000F7D4C56C luaD_rawrunprotected
PC 0x0000F7D4DB54 luaD_pcall
PC 0x0000F7D499C8 lua_pcallk
PC 0x0000F7D46770 interp_include
PC 0x0000F7D3C76C command_include
PC 0x0000F7D707C8 lua_command
PC 0x0000F7D4D0A0 luaD_pretailcall
PC 0x0000F7D673A8 luaV_execute
PC 0x0000F7D4D5CC luaD_callnoyield
PC 0x0000F7D4C56C luaD_rawrunprotected
PC 0x0000F7D4DB54 luaD_pcall
PC 0x0000F7D499C8 lua_pcall
PC 0x0000F7D468A0 interp_run
PC 0x0000F7D3C4C0 interact
PC 0x0000F7D34124 main
PC 0x0000F7D320A4 efi_main
PC 0x0000F7D3A550 _start

In particular,

;       getrootmount(devformat(rootdev));
    10d8: 3c 7c 01 94   bl      0x601c8 <devformat>
    10dc: 06 26 00 94   bl      0xa8f4 <getrootmount>

but 0x0000F327C740 looks like it's somewhere else in EFI and so the trail runs pretty cold, though I think it's safe to say that we hit the d->d_dev->dv_fmtdev != NULL case. Oddly, looking at the rest of the EFI debug messages, I don't see anything getting loaded that far south.

@bsdimp
Copy link
Contributor

bsdimp commented Dec 16, 2023

So the exception is at 0x0000F327C740, which is well outside the range of all the other addresses on the stack. But it has survived the call to devformat...

I'm not sure what the device should be... but one test would be to set vfs.root.mountfrom to what you think it should be before issuing 'boot'. This would let us know if it was the return value (and/or the function itself) that's causing this or something else. Worst case, if you set this to something non-sensical, the kernel will just not be able to fine /, so its safe to test to see if the return value from devformat() changes in a way that runs of off the cliff we hit.

I'm not familar with cheri enough to know, but are we running in capabilities which trap on buffer overflows, or are we still in mixed mode where they might be possible?

@jrtc27
Copy link
Member

jrtc27 commented Dec 16, 2023

loader is built as a plain AArch64 binary, no capabilities allowed, since the firmware doesn't save/restore them currently on traps so they would be clobbered at arbitrary points. We only enable capability use in the kernel's locore.

@bsdimp
Copy link
Contributor

bsdimp commented Dec 16, 2023

OK, so we can't rule out an overflow or something similar....

@nwf-msr
Copy link
Contributor Author

nwf-msr commented Dec 19, 2023

So the exception is at 0x0000F327C740, which is well outside the range of all the other addresses on the stack. But it has survived the call to devformat...

I don't think that's right; the return address is 0x0000F7D310DC, which means that bi_load is still "in" its call to devformat, which has (possibly transitively) tail-called to 0x0000F327C740.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants