-
Notifications
You must be signed in to change notification settings - Fork 33
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Disable precompilation of BijectorsEnzymeExt on Julia 1.11+ #333
Conversation
d683d75
to
a3cb9b2
Compare
5086c31
to
95f8888
Compare
6a42054
to
a962770
Compare
Note that to get CI to pass, we probably need to update Tapir to Mooncake. I started some work on this here https://github.com/TuringLang/Bijectors.jl/tree/py/tapir-to-mooncake but would prefer to keep that to a separate PR. |
This doesn’t really fix things so much as cause more breakage for users thereof you can specify a package to have multiple dependencies to ensure loading. If you mark the enzyme ext as requiring both Enzyme and CR, does that resolve. And/or actually loading CR in the extension |
Can you explain?
Neither of those work, unfortunately! |
yeah so for enzyme users here, now its much more broken =/ What does CI say when you do the corresponding fixes above? |
I'm very unfamiliar with Enzyme and the AD ecosystem, so I'm afraid you might have to spell this out more clearly for me. How is it broken? Or to be more precise, it is definitely still somewhat broken (I wrote as much in my original comment), but how is it much more broken than the current version?
Precompilation fails in all cases: #334 #335 #336 and it's reproducible locally too. |
Disabled fast failing on CI so that we can see exactly which tests error. They're fairly cheap to run anyway. |
Depending on import order in one's code I think is pretty horrendous, and we should make a proper fix a priority. However, this seems like an improvement to me, at least allowing environments with both packages to precompile. Would like to hear why @wsmoses considers the partial fix worse than no fix at all though, before approving. |
3f94fbc
to
adc412d
Compare
Closing in favour of #337, which imo is a better fix |
In the following,
B = Bijectors
,E = Enzyme
, andC = ChainRulesCore
.We have
BEExt
in this repository, and Enzyme.jl definesECExt
, andB
depends onC
. The problem is thatBEExt
depends on a method that is defined inECExt
, and therefore requiresECExt
to have been loaded beforeBEExt
.In Julia 1.10.5 and 1.11.0, this requirement was somehow always satisfied. (I don't know why.)
However, JuliaLang/julia#55589 shook this up and now, there's no guarantee that
ECExt
will be loaded first. Indeed, it turns out that in Julia 1.11.1,BEExt
is always loaded first.The PR is slated to be backported to 1.10.6 as well.(It hasn't been backported.) I suppose that we cannot really complain about the PR because the dependency graph in itself doesn't specify that one should be loaded before the other (the order of loading depends only on how the dependency graph is traversed), and we probably should not be relying on that implicit behaviour. Although the discussion in JuliaLang/julia#55589 suggests that we might one day be able to explicitly specify dependencies between extensions (i.e. adding an extra edge to the dependency graph), this isn't in place yet.Anyway, the result of this is that precompilation fails when adding Bijectors and Enzyme to an environment, as reported in #332.
This PR uses a workaround described in JuliaLang/julia#56204 to disable precompilation for
BEExt
, so that we can at least add Bijectors and Enzyme to the same environment.It doesn't fully solve the issue at runtime, though. At runtime, we can – to some extent – control the order of loading. If we run
using E
beforeusing B
, thenECExt
gets loaded first, and all is good. But if we runusing B
first, thenBEExt
is loaded, and we get an error.I don't know how to fully solve the runtime issue, but this PR will at least solve the precomp issue.
Closes #332 (in the sense that it fixes the immediate problem of precompilation failing; the root cause of this issue is really the underspecified dependency graph, but there isn't a fix for that)