Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

justStaticExecutables: Forbid references to GHC #304352

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 53 additions & 8 deletions doc/languages-frameworks/haskell.section.md
Original file line number Diff line number Diff line change
Expand Up @@ -923,14 +923,59 @@ for this to work.

`justStaticExecutables drv`
: Only build and install the executables produced by `drv`, removing everything
that may refer to other Haskell packages' store paths (like libraries and
documentation). This dramatically reduces the closure size of the resulting
derivation. Note that the executables are only statically linked against their
Haskell dependencies, but will still link dynamically against libc, GMP and
other system library dependencies. If dependencies use their Cabal-generated
`Paths_*` module, this may not work as well if GHC's dead code elimination
is unable to remove the references to the dependency's store path that module
contains.
that may refer to other Haskell packages' store paths (like libraries and
documentation). This dramatically reduces the closure size of the resulting
derivation. Note that the executables are only statically linked against their
Haskell dependencies, but will still link dynamically against libc, GMP and
other system library dependencies.

If the library being built or its dependencies use their Cabal-generated
`Paths_*` module, this may not work as well if GHC's dead code elimination is
unable to remove the references to the dependency's store path that module
contains. (See [nixpkgs#164630][164630] for more information.)

Importing the `Paths_*` module may cause builds to fail with this message:

```
error: output '/nix/store/64k8iw0ryz76qpijsnl9v87fb26v28z8-my-haskell-package-1.0.0.0' is not allowed to refer to the following paths:
/nix/store/5q5s4a07gaz50h04zpfbda8xjs8wrnhg-ghc-9.6.3
```

If that happens, first disable the check for GHC references and rebuild the
derivation:

```nix
pkgs.haskell.lib.overrideCabal
(pkgs.haskell.lib.justStaticExecutables my-haskell-package)
(drv: {
disallowGhcReference = false;
})
```

Then use `strings` to determine which libraries are responsible:

```
$ nix-build ...
$ strings result/bin/my-haskell-binary | grep /nix/store/
...
/nix/store/n7ciwdlg8yyxdhbrgd6yc2d8ypnwpmgq-hs-opentelemetry-sdk-0.0.3.6/bin
...
```

Finally, use `remove-references-to` to delete those store paths from the produced output:

```nix
pkgs.haskell.lib.overrideCabal
(pkgs.haskell.lib.justStaticExecutables my-haskell-package)
(drv: {
postInstall = ''
${drv.postInstall or ""}
remove-references-to -t ${pkgs.haskellPackages.hs-opentelemetry-sdk}
'';
})
```

[164630]: https://github.com/NixOS/nixpkgs/issues/164630

`enableSeparateBinOutput drv`
: Install executables produced by `drv` to a separate `bin` output. This
Expand Down
8 changes: 8 additions & 0 deletions nixos/doc/manual/release-notes/rl-2411.section.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,14 @@
before changing the package to `pkgs.stalwart-mail` in
[`services.stalwart-mail.package`](#opt-services.stalwart-mail.package).

- `haskell.lib.compose.justStaticExecutables` now disallows references to GHC in the
output by default, to alert users to closure size issues caused by
[#164630](https://github.com/NixOS/nixpkgs/issues/164630). See ["Packaging
Helpers" in the Haskell section of the Nixpkgs
manual](https://nixos.org/manual/nixpkgs/unstable/#haskell-packaging-helpers)
for information on working around `output '...' is not allowed to refer to
the following paths` errors caused by this change.

## Other Notable Changes {#sec-release-24.11-notable-changes}

- Create the first release note entry in this section!
Expand Down
35 changes: 33 additions & 2 deletions pkgs/development/haskell-modules/generic-builder.nix
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,28 @@ in
# build products from that prior build as a starting point for accelerating
# this build
, previousIntermediates ? null
# References to these store paths are forbidden in the produced output.
, disallowedRequisites ? []
# Whether to allow the produced output to refer to `ghc`.
#
# This is used by `haskell.lib.justStaticExecutables` to help prevent static
# Haskell binaries from having erroneous dependencies on GHC.
#
# Generated `Paths_*` modules include paths for the runtime library
# directory (and similar) of the package being built. If the `Paths_*`
# module is imported, this creates a dependency from the static binary
# being built to the _library_ being built (which is dynamically linked
# and depends on the GHC used to build it).
#
# To avoid this:
# 1. Build the impacted derivation.
# 2. Run `strings` on the built binary of the impacted derivation to
# locate the store paths it depends on.
# 3. Add `remove-references-to -t ${bad-store-path-in-binary}` to the
# impacted derivation's `postInstall`.
#
# See: https://github.com/NixOS/nixpkgs/issues/164630
, disallowGhcReference ? false
, # Cabal 3.8 which is shipped by default for GHC >= 9.3 always calls
# `pkg-config --libs --static` as part of the configure step. This requires
# Requires.private dependencies of pkg-config dependencies to be present in
Expand Down Expand Up @@ -682,7 +704,7 @@ stdenv.mkDerivation ({

passthru = passthru // rec {

inherit pname version;
inherit pname version disallowGhcReference;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are you inheriting this here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So that other derivations or Nix code can introspect it. It might be useful, especially for debugging.


compiler = ghc;

Expand Down Expand Up @@ -843,10 +865,19 @@ stdenv.mkDerivation ({
// optionalAttrs (args ? dontStrip) { inherit dontStrip; }
// optionalAttrs (postPhases != []) { inherit postPhases; }
// optionalAttrs (stdenv.buildPlatform.libc == "glibc"){ LOCALE_ARCHIVE = "${glibcLocales}/lib/locale/locale-archive"; }
// optionalAttrs (disallowedRequisites != [] || disallowGhcReference) {
disallowedRequisites =
disallowedRequisites
++ (
if disallowGhcReference
then [ghc]
else []
);
}

# Implicit pointer to integer conversions are errors by default since clang 15.
# Works around https://gitlab.haskell.org/ghc/ghc/-/issues/23456.
// lib.optionalAttrs (stdenv.hasCC && stdenv.cc.isClang) {
// optionalAttrs (stdenv.hasCC && stdenv.cc.isClang) {
NIX_CFLAGS_COMPILE = "-Wno-error=int-conversion";
}
)
Expand Down
3 changes: 2 additions & 1 deletion pkgs/development/haskell-modules/lib/compose.nix
Original file line number Diff line number Diff line change
Expand Up @@ -290,7 +290,7 @@ rec {
/* link executables statically against haskell libs to reduce
closure size
*/
justStaticExecutables = overrideCabal (drv: {
justStaticExecutables = overrideCabal (drv: {
enableSharedExecutables = false;
enableLibraryProfiling = false;
isLibrary = false;
Expand All @@ -300,6 +300,7 @@ rec {
# Remove every directory which could have links to other store paths.
rm -rf $out/lib $out/nix-support $out/share/doc
'';
disallowGhcReference = true;
});

/* Build a source distribution tarball instead of using the source files
Expand Down