linux: allow building with ld.lld (for LTO) #344665
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ofborg
bot
requested review from
alyssais,
K900,
NeQuissimus,
Ma27,
thoughtpolice and
TredwellGit
|
Full build failure log is here: I'm not quite sure what is failing. Will continue looking into it. |
blitz
force-pushed
the
kernel-lld
branch
2 times, most recently
from
a1800a7
to
4a1ff2b
Compare
|
|
Looks like that should be using the binutils wrapper and not the clang wrapper. |
|
Ah, |
|
There should be a strip in bintools: (I'm assuming llvm-objcopy must be multi-call or something for it to be the target of the link.) |
Only There is also a |
|
Want to fix it then? |
Yes, will open a different PR for this given that it's not a requirement for this one. The build goes further now: We're getting closer. |
|
I've opened #345691 for the strip issue. |
alyssais
reviewed
Oct 1, 2024
|
So I've narrowed down the build issue to the compiler/bintools wrappers. If I use the unwrapped toolchain for building (the last XXX commit), the build succeeds. I'm currently debugging what the wrappers do that causes this problem. I wish there were a flag to turn the wrapper magic off for cases like these... |
|
You have ran into #321667 if you're on x86_64-linux. |
Thanks for the pointer! But why would the kernel need patching, if it builds fine with the unwrapped toolchain? It seems like we do something weird in our wrapping logic. |
Our wrapper does invoke a behavior of LLVM which GNU ld doesn't have. GNU ld has some logic to not link with dynamic libraries when necessary. LLVM ld doesn't have this behavior. I did start some work in the wrapper to implement the same GNU ld behavior but I just haven't allocated enough time into this problem. I did however open an issue in LLVM itself to see about getting this behavior implemented. |
|
I'm running a LTO kernel on my laptop without apparent issues: ❯ sudo dmesg | grep clang
[ 0.000000] Linux version 6.11.1 (nixbld@localhost) (clang version 18.1.8, LLD 18.1.8) #1-NixOS SMP Mon Sep 30 14:31:09 UTC 2024
~
❯ zgrep LTO_CLANG /proc/config.gz
CONFIG_LTO_CLANG=y
CONFIG_ARCH_SUPPORTS_LTO_CLANG=y
CONFIG_ARCH_SUPPORTS_LTO_CLANG_THIN=y
CONFIG_HAS_LTO_CLANG=y
CONFIG_LTO_CLANG_FULL=y
# CONFIG_LTO_CLANG_THIN is not set
|
This sounds very strange — where does binutils ld even come from in an lld stdenv? |
That's an excellent question. But you can check yourself: # Before this PR:
❯ cat $(nix-build -A pkgsLLVM.linux_latest.configfile) | grep LD_IS_
CONFIG_LD_IS_BFD=y# After this PR:
❯ cat $(nix-build -A pkgsLLVM.linux_latest.configfile) | grep LD_IS_
CONFIG_LD_IS_LLD=y |
|
Before this PR, we did not explicitly set LD when invoking This comes from from |
|
I think we'll have to override some stuff to produce a build platform stdenv that uses a build platform LLVM linker. It should be doable and shouldn't require changing the LLVM side of things unless we want to add some checks and make the override in there. |
|
That would be super useful but it's orthogonal to this PR. Even when we do that we need to fix passing the right LD to the Perl script so |
|
Result of 745 packages marked as broken and skipped:
1 package blacklisted:
253 packages failed to build:
2309 packages built:
|
|
Thanks for explaining! Could you please update the commit message? |
| # | ||
| # We have to pass through the target toolchain, because `make config` checks them for versions. This is | ||
| # required to get clang LTO working, among other things. | ||
| my $pid = open2(\*IN, \*OUT, "make -C $ENV{SRC} O=$buildRoot config SHELL=bash ARCH=$ENV{ARCH} CC=$ENV{CC} HOSTCC=$ENV{HOSTCC} HOSTCXX=$ENV{HOSTCXX} LD=$ENV{LD} NM=$ENV{NM} AR=$ENV{AR} OBJCOPY=$ENV{OBJCOPY} $makeFlags"); |
There was a problem hiding this comment.
We should probably split those into multiple lines at this point
There was a problem hiding this comment.
Fixed. Someone with more Perl foo might check whether it is idiomatic what I did.
|
@alyssais I updated the commit message to clarify what we are fixing here. It should be more useful now. |
|
I was able to cross compile an x86_64-linux kernel on aarch64-linux using LLVM via |
|
I'm testing this locally now, but I don't understand why we need absolute paths for anything apart from the linker on x86_64. A comment mentions PATH clobbering issues, but what path clobbering issues? Have we ever seen any? |
|
I'm also wondering whether LD etc would be better set as environment variables in stdenv. Would you be interested in exploring that as followup? It would likely fix other packages with similar problems. |
The problem it refers to seems to be that |
I can look into this. I see a couple of packages that use |
But they're not going to be autodetected if we're explicitly setting them to the prefixed versions? |
For CC this is already done by stdenv. I was just talking about e.g. LD, for consistency. |
Even when building with pkgsLLVM.stdenv where ld is ld.lld and not binutils ld, the build picks up binutils ld for linking. This prevents features from working that require ld.lld. The reason is that when the LD environment variable is not set, Linux defaults to `ld` as a linker and ld is: /nix/store/zznja5f8v3jafffyah1rk46vpfcn38dv-gcc-wrapper-13.3.0/bin/ld GNU ld (GNU Binutils) 2.42 This ld comes from pkgsLLVM.buildPackages.stdenv.cc. Fix by being more specific about which tools we want to build with. This just extends what manual-config.nix has already done for quite a while to avoid similar problems for other tools. It's important to pass LD and other toolchain environment variables to `make config` in generate-config.pl, because otherwise `make config` will also make decisions based on the wrong toolchain.
The ld wrapper leads to the following linking error on x86_64, when building pkgsLLVM.linux_latest: x86_64-unknown-linux-gnu-ld: error: ../arch/x86/boot/setup.ld:15: unable to move location counter (0x204) backward to 0x1ef for section '.bstext'
When we set CROSS_COMPILE to set the toolchain prefix, everything works out of the box without trying to drop absolute paths everywhere. This removes a lot of duplication from the kernel derivations.
I looked into this and we don't have to set them at all! We only need to set I'm currently compiling some kernels, but so far it looks good. Testing is welcome! @alyssais If this approach works out, I'll squash the commits. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Even when building the Linux kernel with a
stdenvwhere the linker isld.lldinstead of binutilsld, the build picks up binutilsldfor linking. This prevents features that requireld.lldfrom working, such as LTO.Fix by being more specific about which tools we want to build with. This extends what manual-config.nix has already done for quite a while.
I've used the following to get
ld.lddas linker:Things left to be done
make configrecognizes the right linkernix-build clang-kernel.nix -A linux_llvmcurrently fails:Things done
nix.conf? (See Nix manual)sandbox = relaxedsandbox = truenix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage./result/bin/)Add a 👍 reaction to pull requests you find important.