Unable to resolve dependency with pre-release version

[Blacksmoke16]

I'm still trying to reduce the issue, but for now to reproduce:

1. Download Athena monorepo at: https://github.com/athena-framework/athena/tree/bump-components
2. run SHARDS_OVERRIDE=shard.dev.yml shards install
3. Notice you get:

Resolving dependencies
Fetching https://github.com/crystal-ameba/ameba.git
Unable to satisfy the following requirements:

- `athena-console (*)` required by `shard.yml`
- `athena-console (*)` required by `athena 0.17.1`
Failed to resolve dependencies

Currently also unsure if I'm just missing something or there is a bug here.

Debug Notes

• Tested against shards 0.17.2
• The root shard.yml doesn't have any version constraints and sources everything from latest GH release
• shard.dev.yml doesn't have any version constraints either and sources everything from the related path within src/components
• The console component is versioned as 0.4.0-dev
• The framework component's version constraint for the console component is >=0.3.0, <0.4.0, which 0.4.0-dev should be valid against
  • Confirmed via matches?("0.4.0-dev", ">=0.3.0, <0.4.0").should be_true in shards repo
• Changing the version constraint to >=0.3.0, <=0.4.0-dev produces same error
• Changing the version constraint to 0.4.0-dev also produces the same error
• Moving the path sources out of shard.dev.yml into shard.yml and trying again produces:

Resolving dependencies
Fetching https://github.com/crystal-ameba/ameba.git
Fetching https://github.com/athena-framework/spec.git
Unable to satisfy the following requirements:

- `athena-console (*)` required by `shard.yml`
- `athena-console (>=0.3.0, <0.4.0)` required by `athena 0.17.1`
Failed to resolve dependencies

• So far unsure if the path part of it is the problem, or the fact the component versions have pre-releases
• Could it have something to do with the current latest release of the framework component not having an athena-console dep? My assumption was that the override file would make it just use whatever is in src/components/framework/shard.yml.

[Blacksmoke16]

Reduced it to a test case in molinillo. Definitely seems related to the pre-release shard version.

Steps to reproduce:

1. https://github.com/crystal-lang/crystal-molinillo#development
2. Create spec/fixture/index/pre_release.json with:

{
  "console": [
    {
      "name": "console",
      "version": "0.4.0-dev",
      "dependencies": {
      }
    }
  ],
  "framework": [
    {
      "name": "framework",
      "version": "0.17.1",
      "dependencies": {
        "console": ">=0.3.0, <0.4.0"
      }
    }
  ],
  "athena": [
    {
      "name": "athena",
      "version": "0.0.0",
      "dependencies": {
        "framework": "*",
        "console": "*"
      }
    }
  ]
}

3. Create spec/fixture/case/pre_release.json with:

{
  "name": "resolves an index with a pre-release version",
  "index": "pre_release",
  "requested": {
    "athena": ""
  },
  "base": [],
  "resolved": [
    {
      "name": "athena",
      "version": "0.0.0",
      "dependencies": [
        {
          "name": "framework",
          "version": "0.17.1",
          "dependencies": [
            {
              "name": "console",
              "version": "0.4.0-dev",
              "dependencies": []
            }
          ]
        },
        {
          "name": "console",
          "version": "0.4.0-dev",
          "dependencies": []
        }
      ]
    }
  ],
  "conflicts": []
}

4. Run crystal spec
5. Notice it fails:

  1) Molinillo::Resolver(R, S) dependency resolution with the TestIndex index resolves an index with a pre-release version

       Unable to satisfy the following requirements:
       
       - `console` required by `athena (0.0.0)`
       - `console` required by `framework (0.17.1)` (Molinillo::VersionConflict(Gem::Dependency | Molinillo::TestSpecification, Molinillo::TestSpecification))
         from src/molinillo/resolution.cr:333:9 in 'raise_error_unless_state'
         from src/molinillo/resolution.cr:316:11 in 'unwind_for_conflict'
         from src/molinillo/resolution.cr:698:13 in 'attempt_to_activate'
         from src/molinillo/resolution.cr:266:11 in 'process_topmost_state'
         from src/molinillo/resolution.cr:200:11 in 'resolve'
         from src/molinillo/resolver.cr:33:7 in 'resolve'
         from spec/resolver_spec.cr:78:7 in 'resolve'
         from spec/resolver_spec.cr:96:20 in '->'
         from /usr/lib/crystal/spec/example.cr:45:13 in 'internal_run'
         from /usr/lib/crystal/spec/example.cr:33:16 in 'run'
         from /usr/lib/crystal/spec/context.cr:18:23 in 'internal_run'
         from /usr/lib/crystal/spec/context.cr:339:7 in 'run'
         from /usr/lib/crystal/spec/context.cr:18:23 in 'internal_run'
         from /usr/lib/crystal/spec/context.cr:339:7 in 'run'
         from /usr/lib/crystal/spec/context.cr:18:23 in 'internal_run'
         from /usr/lib/crystal/spec/context.cr:339:7 in 'run'
         from /usr/lib/crystal/spec/context.cr:18:23 in 'internal_run'
         from /usr/lib/crystal/spec/context.cr:156:7 in 'run'
         from /usr/lib/crystal/spec/dsl.cr:220:7 in '->'
         from /usr/lib/crystal/crystal/at_exit_handlers.cr:14:19 in 'run'
         from /usr/lib/crystal/crystal/main.cr:50:14 in 'exit'
         from /usr/lib/crystal/crystal/main.cr:45:5 in 'main'
         from /usr/lib/crystal/crystal/main.cr:127:3 in 'main'
         from /usr/lib/libc.so.6 in '??'
         from /usr/lib/libc.so.6 in '__libc_start_main'
         from ../sysdeps/x86_64/start.S:117 in '_start'
         from ??

Pretty sure it's a bug at this point given replacing the three instances of 0.4.0-dev with like 0.3.5 in both files allows it then pass, even tho both versions are valid based on the requirements.

[straight-shoota]

@jwoertink What you describe is pretty certainly unrelated, so I created a new issue for it: #581

[straight-shoota]

The exclusion of dependencies with pre-release versions is deliberate and identical to bundler (rubygems/bundler#2938, rubygems/bundler#4340).

There is a specific case for a pre-release version matching against a dependency restriction that is not a pre-release and defines that as not satisfying the requirement:

shards/src/molinillo_solver.cr

Lines 225 to 234 in e56c44a

	def requirement_satisfied_by?(dependency, activated, spec)
	unless @prereleases
	if !spec.version.has_metadata? && spec.version.prerelease? && !dependency.prerelease?
	vertex = activated.vertex_named(spec.name)
	return false if !vertex || vertex.requirements.none?(&.prerelease?)
	end
	end
	dependency.matches?(spec.version)
	end

Note: @prereleases is only true when --pre option is passed to shards outdated. In any other use case (including shards install), @prereleases is false.

So I think this should be considered not as a bug report but a discussion about this pre-release behaviour. But considering that bundler behaves the same, I would expect good reasons for it.

@Blacksmoke16 Your reduction into a Molinillo test case is great, but it's actually coincidental: Molinillo has no understanding of versions and pre-releases. That's entirely delegated to the user of the Molinillo API, in form of implementing SpecificationProvider#requirement_satisfied_by?. The reason you see the same behaviour in the Molinillo specs is that the spec helper used to execute these test cases implements the same pre-release logic as shards and bundler.

[straight-shoota]

Interestingly, the most recent bundler release has added support for the --pre option to the update and lock commands: https://bundler.io/blog/2023/01/31/bundler-v2-4.html#new-cli-features

We added a few small CLI features, such as a new --pre flag to bundle update and bundle lock to explicitly opt-in to prereleases of selected (of all) gems without having to explictly change your Gemfile with pre-release requirements such as >= 7.1.0.beta.

(The same release also replaced Molinnillo by PubGrub, but I don't think the option is related to the resolver engine)