Skip to content

Commit

Permalink
Change format of generate gemset.nix files
Browse files Browse the repository at this point in the history 
This commit is an attempt to make Bundix more "platform-aware."

Every rubygem has a "platform" field which names the ruby-platform that
it can run on. For the vast majority of gems, this will be `"ruby"`,
indicating that it's a gem written entirely in ruby and should be able
to run on any ruby interpreter; however, there are some gems that rely
on native extensions that might only work on certain systems.

The previous `gemset.nix` format only supported one version of each
dependency. If the `Gemfile.lock` included a reference to
`nokogiri (1.14.2)` and also `nokogiri (1.14.2-x86-linux)`, Bundix would
render a `gemset.nix` that included only one of them. The trouble with
this behaviour is that, at runtime, Bundler, will see has multiple
nokogiri gems to choose from and may choose that one that was never
built. This leads to issues like [1] and [2].

The new format of `gemfile.nix` is:

```nix
{
  dependencies = ["list" "of" "top-level" "dependencies"];
  platforms = {
    ruby = {
      gem-name = {
        dependencies = ["list" "of" 'transitive" "dependencies"]
        groups = [ ... ];
        version = "...";
        source = {
          # attrs describing the git, path, or RubyGems source
        };
      };
    };
    other-platform = {
      gem-name = { ... };
    };
  };
}
```

The gemset's `dependencies` entry copies the `DEPENDENCIES` section of
the `Gemfile.lock` and names all the top-level gem dependencies from the
`Gemfile`.

The gemset's `platforms` entry is an attrset where every key is a
ruby-platform and its value is an attrset of the gems particular to that
platform. This last attrset is essentially the same as the previous
`gemset.nix` format.

This commit also introduces a new nix function: `bundixEnv` (like
`bundlerEnv`, but for Bundix). This function accepts the same arguments
as `bundlerEnv`, with the addition of a `platform`. `bundixEnv` then
converts the given gemset into a format suitable for `bundlerEnv` by
selecting the gems appropriate for the given `platform`. Finally, it
delegates to `bundlerEnv`, with the platform-specific gemset.

`bundix --init` will generate an example `flake.nix` with an example
package that demonstrates how `bundixEnv` works.

See `gem help platforms` for more info about ruby platforms. The Bundler
Guide [3] provides a few examples of rubygems that support multiple
platforms.

This commit also adds two guides, `guides/getting-started.md` and
`guides/motivations.md` that describe this format in more detail.

[1] nix-community#71
[2] https://discourse.nixos.org/t/issues-with-nix-reproducibility-on-macos-trying-to-build-nokogiri-ruby-error-unknown-warning-option/22019
[3] https://guides.rubygems.org/gems-with-extensions/
  • Loading branch information
@sangster
sangster committed Mar 18, 2023
1 parent a09f701 commit dabe2d3
Show file tree
Hide file tree
Showing 78 changed files with 3,302 additions and 1,889 deletions.
11 changes: 7 additions & 4 deletions .gitignore
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
.bundle/
*.gem
tmp
result
/*.gem
/.bundle/
/.yardoc/
/coverage/
/doc/
/result
/tmp/
/vendor/
22 changes: 13 additions & 9 deletions Gemfile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,16 @@
source 'https://rubygems.org'
gemspec

# Development dependencies
gem 'guard-rspec', '~> 4.7'
gem 'guard-rubocop', '~> 1.5'
gem 'pry-byebug', '~> 3.10'
gem 'rake', '~> 13.0'
gem 'rspec', '~> 3.12'
gem 'rubocop', '~> 1.45'
gem 'rubocop-rake', '~> 0.6'
gem 'rubocop-rspec', '~> 2.18'
group :development do
gem 'guard-rspec', '~> 4.7'
gem 'guard-rubocop', '~> 1.5'
gem 'pry-byebug', '~> 3.10'
gem 'rake', '~> 13.0'
gem 'rspec', '~> 3.12'
gem 'rubocop', '~> 1.45'
gem 'rubocop-rake', '~> 0.6'
gem 'rubocop-rspec', '~> 2.18'
gem 'simplecov', '~> 0.22'
gem 'webmock', '~> 3.18'
gem 'yard', '~> 0.9'
end
23 changes: 23 additions & 0 deletions Gemfile.lock
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,15 @@ PATH
GEM
remote: https://rubygems.org/
specs:
addressable (2.8.1)
public_suffix (>= 2.0.2, < 6.0)
ast (2.4.2)
byebug (11.1.3)
coderay (1.1.3)
crack (0.4.5)
rexml
diff-lcs (1.5.0)
docile (1.4.0)
ffi (1.15.5)
formatador (1.1.0)
guard (2.18.0)
Expand All @@ -31,6 +36,7 @@ GEM
guard-rubocop (1.5.0)
guard (~> 2.0)
rubocop (< 2.0)
hashdiff (1.0.1)
json (2.6.3)
listen (3.8.0)
rb-fsevent (~> 0.10, >= 0.10.3)
Expand All @@ -50,6 +56,7 @@ GEM
pry-byebug (3.10.1)
byebug (~> 11.0)
pry (>= 0.13, < 0.15)
public_suffix (5.0.1)
rainbow (3.1.1)
rake (13.0.6)
rb-fsevent (0.11.2)
Expand Down Expand Up @@ -91,8 +98,21 @@ GEM
rubocop-capybara (~> 2.17)
ruby-progressbar (1.12.0)
shellany (0.0.1)
simplecov (0.22.0)
docile (~> 1.1)
simplecov-html (~> 0.11)
simplecov_json_formatter (~> 0.1)
simplecov-html (0.12.3)
simplecov_json_formatter (0.1.4)
thor (1.2.1)
unicode-display_width (2.4.2)
webmock (3.18.1)
addressable (>= 2.8.0)
crack (>= 0.3.2)
hashdiff (>= 0.4.0, < 2.0.0)
webrick (1.7.0)
yard (0.9.28)
webrick (~> 1.7.0)
zeitwerk (2.6.7)

PLATFORMS
Expand All @@ -108,6 +128,9 @@ DEPENDENCIES
rubocop (~> 1.45)
rubocop-rake (~> 0.6)
rubocop-rspec (~> 2.18)
simplecov (~> 0.22)
webmock (~> 3.18)
yard (~> 0.9)

BUNDLED WITH
2.4.6
230 changes: 129 additions & 101 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,121 +1,140 @@
# About

Bundix makes it easy to package your [Bundler](http://bundler.io/)-enabled Ruby
applications with the [Nix](http://nixos.org/nix/) package manager.

## Installation
applications with the [Nix](https://nixos.org/download.html) package manager.

Installing from this repo:
# Basic Usage

nix-env -iA bundix
**Note**: See [Getting Started](./guides/getting-started.md) for a more detailed
description of setting up a new ruby project.

Please note that in order to actually use this gem you must have Nix installed.
> Please note that in order to actually use this gem you must have Nix installed.
>
> I recommend first reading the [nixpkgs manual entry for
> Ruby](http://nixos.org/nixpkgs/manual/#sec-language-ruby) as this README might
> become outdated, it's a short read right now, so you won't regret it.
## Basic Usage
To use Bundix, all your project needs is a `Gemfile` describing your project's
ruby dependencies. If you already have a `Gemfile.lock`, Bundix will use it, but
it will generate one if you don't.

I recommend first reading the
[nixpkgs manual entry for Ruby](http://nixos.org/nixpkgs/manual/#sec-language-ruby)
as this README might become outdated, it's a short read right now, so you won't
regret it.
```sh
$ nix run github:sangster/bundix
$ nix run nixpkgs#git -- add gemset.nix
```

1. Making a gemset.nix
## Adding Bundix to your `flake.nix`

Change to your project's directory and run this:
To integrate Bundix into your Nix package, you'll need to make 3 changes:

bundix -l
### 1. Import Bundix's overlay

This will generate a `gemset.nix` file that you then can use in your
`bundlerEnv` expression like this:
For example, if you have a `import nixpkgs` line in your flake, add a `overlays
= [ ...];` attribute to it. For example:

2. Using `nix-shell`
```nix
{
inputs.bundix.url = github:sangster/bundix;
To try your package in `nix-shell`, create a `default.nix` like this:
outputs = { bundix, ... }:
let
pkgs = import nixpkgs {
system = "x86_64-linux";
overlays = [bundix.overlays.default];
};
in { ... };
}
```

```nix
with (import <nixpkgs> {});
let
gems = bundlerEnv {
name = "your-package";
inherit ruby;
gemdir = ./.;
};
in stdenv.mkDerivation {
name = "your-package";
buildInputs = [gems ruby];
}
```
### 2. Create the gem bundle with Bundix

Now we use the `pkgs.bundixEnv` nix function to convert your project's
`gemset.nix` into a nix derivation that will be used as a runtime dependency for
your own ruby package. Here is an example usage:

```nix
{
gems = pkgs.bundixEnv {
name = "bundix-project-gems";
ruby = pkgs.ruby;
gemdir = ./.;
platform = "x86_64-linux";
};
}
```

and then simply run `nix-shell`.
`bundixEnv` accepts the same attribute arguments as
[bundlerEnv](https://github.com/NixOS/nixpkgs/blob/48e4e2a1/pkgs/development/ruby-modules/bundler-env/default.nix),
with the addition of two:

- `platform`: Specifies the gem platform we want to build this package for.
- `system`: As an alternative to `platform`, you can provide your nix `system`
and `bundixEnv` will attempt to figure out the correct `platform` from that.

### 3. Use the gem bundle in your app

Finally, you need to integrate your new gem bundle into your package. An easy
method is to use its `wrappedRuby` package as the `ruby` used to execute your
ruby code.

```nix
pkgs.stdenv.mkDerivation {
phases = "installPhase";
installPhase = ''
mkdir -p $out/bin
cat << EOF > "$out/bin/my-app"
#!/bin/sh
exec ${gems.wrappedRuby}/bin/ruby ${./my-ruby-script.rb}
EOF
chmod +x "$out/bin/my-app"
'';
}
```

3. Proper packages
### Generate an example `flake.nix`

To make a package for nixpkgs, you can try something like this:
If your project doesn't have a +flake.nix+ yet, Bundix can make an example one
for you:

```nix
{ stdenv, bundlerEnv, ruby }:
let
gems = bundlerEnv {
name = "your-package";
inherit ruby;
gemdir = ./.;
};
in stdenv.mkDerivation {
name = "your-package";
src = ./.;
buildInputs = [gems ruby];
installPhase = ''
mkdir -p $out
cp -r $src $out
'';
}
```
```sh
$ nix run github:sangster/bundix -- --init
```

### Command-line Flags
## Command-line Flags

```
$ nix run github:sangster/bundix -- --help
Usage: bundix [options]
-i, --init[=RUBY_DERIVATION] initialize a new flake.nix for 'nix develop' (won't overwrite old ones)
-t, --init-template=TEMPLATE the flake.nix template to use. may be 'default', 'flake-utils', or a filename (default: default)
-p, --init-project=NAME project name to use with --init (default: moo)
--gemset=PATH path to the gemset.nix (default: ./gemset.nix)
--lockfile=PATH path to the Gemfile.lock (default: ./Gemfile.lock)
--gemfile=PATH path to the Gemfile (default: ./Gemfile)
--skip-gemset do not generate gemset
-q, --quiet only output errors
-l, --bundle-lock generate Gemfile.lock first
-u, --bundle-update[=GEMS] ignores the existing lockfile. Resolve then updates lockfile. Taking a list of gems or updating all gems if no list is given (implies --bundle-lock)
-c, --bundle-cache[=DIRECTORY] package .gem files into directory (default: ./vendor/bundle)
-v, --version show the version of bundix
--env show the environment in bundix
```

## How & Why

I'd usually just tell you to read the code yourself, but the big picture is
that bundix tries to fetch a hash for each of your bundle dependencies and
store them all together in a format that Nix can understand and is then used by
`bundlerEnv`.
I wrote this new version of bundix because I became frustrated with the poor
performance of the old bundix, and wanted to save both time and bandwidth, as
well as learn more about Nix.
File options:
--gemfile=PATH path to the existing Gemfile (default: ./Gemfile)
--lockfile=PATH path to the Gemfile.lock (default: ./Gemfile.lock)
For each gem, it first tries to look for an existing gem in the bundler cache
(usually generated via `bundle package`), and if that fails it goes through
each remote and tries to fetch the gem from there. If the remote happens to be
[rubygems.org](http://rubygems.org/) we ask the API first for a hash of the
gem, and then ask the Nix store whether we have this version already. Only if
that also fails do we download the gem.
Output options:
--gemset=PATH destination path of the gemset.nix (default: ./gemset.nix)
-g, --groups=GROUPS bundler groups to include in the gemset.nix (default: all groups)
--bundler-env[=PLATFORM] export a nixpkgs#bundlerEnv compatiblegemset (default: ruby)
--skip-gemset do not generate gemset
As an added bonus I also implemented parsing the `gemset.nix` if it already
exists, and get hashes from there directly, that way updating an existing
`gemset.nix` only takes a few seconds.
Bundler options:
-l, --lock lock the gemfile gems into the lockfile
-u, --update[=GEMS] update the lockfile with new versions of the specified gems, or each one, if none given (implies --lock)
-a, --add-platforms=PLATFORMS add platforms to the lockfile (implies --lock)
-r, --remove-platforms=PLATFORMS remove platforms from the lockfile (implies --lock)
-p, --platforms=PLATFORMS replace all platforms in the lockfile (implies --lock)
-c, --bundle-cache[=DIR] package .gem files into directory (default: ./vendor/bundle)
--ignore-bundler-configs ignores Bundler config files
The output from bundix should be as stable as possible, to make auditing diffs
easier, that's why I also implemented a pretty printer for the `gemset.nix`.
flake.nix options:
-i, --init[=RUBY_DERIVATION] initialize a new flake.nix for 'nix develop' (won't overwrite old ones)
-t, --init-template=TEMPLATE the flake.nix template to use. may be 'default', 'flake-utils', or a filename (default: default)
-n, --project-name=NAME project name to use with --init (default: bundix)
I hope you enjoy using bundix as much as I do, and if you don't, let me know.
Environment options:
-v, --version show the version of bundix
--env show the environment in Bundix
--platform show the gem platform of this host
```

## Development

Expand All @@ -124,11 +143,12 @@ some utilities which may help you. Furthermore, running `nix develop` will start
a new shell where `rake`, and other development dependencies are available. Some
example `rake` commands (via `nix develop` in these examples):

``` sh
$ nix develop -c rake -T # List available rake commands
$ nix develop -c rake # Default rake command: all tests and linters
$ nix develop -c rake dev:console # Open a ruby REPL shell
$ nix develop -c rake dev:guard # Begin automated test-runner
```sh
$ nix develop -c rake -T # List available rake commands
$ nix develop -c rake # Default rake command: all tests and linters
$ nix develop -c rake dev:console # Open a ruby REPL shell
$ nix develop -c rake dev:guard # Begin automated test-runner
$ nix develop -c rake docs:generate # Generate ruby documentation
```

### Building
Expand All @@ -143,11 +163,19 @@ command-line arguments must be preceeded with `--`; for example:

## Closing words

For any questions or suggestions, please file an issue on Github or ask in
`#nixos` on [Freenode](http://freenode.net/).
For any questions or suggestions, please file an issue on Github.

If you're curious about the rationale behind the different versions of Bundix,
see the [Motivations Guide](./guides/motivations.md).

A huge shoutout to Michael 'manveru' Fellinger! As someone who writes ruby
professionally, without his work on Bundix 2, I never would have had the
opportunity to discover how great nix is.

From Bundix 2:

Big thanks go out to
[Charles Strahan](http://www.cstrahan.com/) for his awesome work bringing Ruby to Nix,
[zimbatm](https://zimbatm.com/) for being a good rubber duck and tester, and
[Alexander Flatter](https://github.com/aflatter) for the original bundix. I
couldn't have done this without you guys.
> Big thanks go out to [Charles Strahan](http://www.cstrahan.com/) for his
> awesome work bringing Ruby to Nix, [zimbatm](https://zimbatm.com/) for being a
> good rubber duck and tester, and [Alexander
> Flatter](https://github.com/aflatter) for the original bundix. I couldn't have
> done this without you guys.
8 changes: 8 additions & 0 deletions Rakefile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
require 'rake/testtask'
require 'rspec/core/rake_task'
require 'rubocop/rake_task'
require 'yard'

namespace :dev do
desc 'Start a ruby REPL'
Expand All @@ -18,6 +19,13 @@ namespace :dev do
end
end

namespace :docs do
YARD::Rake::YardocTask.new(:generate)

desc 'Serve YARD Documentation with web server'
task(:serve) { YARD::CLI::Server.new.run }
end

namespace :test do
RSpec::Core::RakeTask.new(:specs)
end
Expand Down
Loading

0 comments on commit dabe2d3

Please sign in to comment.