Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Inactivated PyCall Calling Functions Block Enzyme? #1481

Closed
ChrisRackauckas opened this issue May 29, 2024 · 3 comments
Closed

Inactivated PyCall Calling Functions Block Enzyme? #1481

ChrisRackauckas opened this issue May 29, 2024 · 3 comments

Comments

@ChrisRackauckas
Copy link
Contributor

This seems unintentional. Found this as part of a bigger push ODINN-SciML/ODINN.jl#151. If I have these two lines:

            MB_timestep!(model, glacier, params.solver.step, integrator.t; batch_id = batch_id)
            apply_MB_mask!(integrator.u, glacier, model.iceflow[batch_id])

https://github.com/ODINN-SciML/ODINN.jl/pull/151/files#diff-71e70aa5cc3c0507c49d0979eb88a9ac88cf5771a6bfe621f134fdbe146aba34R135-R136

uncommented, I get an error due to PyCall:

ERROR: Enzyme execution failed.
Enzyme: jl_call calling convention not implemented in aug_forward for   %9 = call nonnull "enzyme_type"="{[-1]:Pointer}" {} addrspace(10)* ({} addrspace(10)* ({} addrspace(10)*, {} addrspace(10)**, i32)*, {} addrspace(10)*, ...) @julia.call({} addrspace(10)* ({} addrspace(10)*, {} addrspace(10)**, i32)* noundef nonnull @jl_f_finalizer, {} addrspace(10)* noundef null, {} addrspace(10)* addrspacecast ({}* inttoptr (i64 5554339744 to {}*) to {} addrspace(10)*), {} addrspace(10)* nofree nonnull align 8 dereferenceable(8) %newstruct) #65, !dbg !81
Stacktrace:
 [1] finalizer
   @ ./gcutils.jl:87
 [2] PyObject
   @ ~/.julia/packages/PyCall/1gn3u/src/PyCall.jl:79
 [3] __getproperty
   @ ~/.julia/packages/PyCall/1gn3u/src/PyCall.jl:314
 [4] getproperty
   @ ~/.julia/packages/PyCall/1gn3u/src/PyCall.jl:318
 [5] get_cumulative_climate!
   @ ~/.julia/dev/Sleipnir/src/glaciers/climate/climate2D_utils.jl:62

Stacktrace:
  [1] throwerr(cstr::Cstring)
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/F71IJ/src/compiler.jl:1338
  [2] finalizer
    @ ./gcutils.jl:87 [inlined]
  [3] PyObject
    @ ~/.julia/packages/PyCall/1gn3u/src/PyCall.jl:79 [inlined]
  [4] __getproperty
    @ ~/.julia/packages/PyCall/1gn3u/src/PyCall.jl:314 [inlined]
  [5] getproperty
    @ ~/.julia/packages/PyCall/1gn3u/src/PyCall.jl:318 [inlined]
  [6] get_cumulative_climate! (repeats 2 times)
    @ ~/.julia/dev/Sleipnir/src/glaciers/climate/climate2D_utils.jl:62
  [7] get_cumulative_climate!
    @ ~/.julia/dev/Sleipnir/src/glaciers/climate/climate2D_utils.jl:0 [inlined]
  [8] augmented_julia_get_cumulative_climate__24808_inner_1wrap
    @ ~/.julia/dev/Sleipnir/src/glaciers/climate/climate2D_utils.jl:0
  [9] macro expansion
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/F71IJ/src/compiler.jl:5916 [inlined]
 [10] enzyme_call
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/F71IJ/src/compiler.jl:5566 [inlined]
 [11] AugmentedForwardThunk
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/F71IJ/src/compiler.jl:5454 [inlined]
 [12] runtime_generic_augfwd(activity::Type{…}, width::Val{…}, ModifiedBetween::Val{…}, RT::Val{…}, f::typeof(get_cumulative_climate!), df::Nothing, primal_1::Climate2D{…}, shadow_1_1::Climate2D{…}, primal_2::StepRange{…}, shadow_2_1::Nothing)
    @ Enzyme.Compiler ~/.julia/packages/Enzyme/F71IJ/src/rules/jitrules.jl:179
 [13] #MB_timestep!#3
    @ ~/.julia/dev/Muninn/src/models/mass_balance/mass_balance_utils.jl:23
 [14] MB_timestep!
    @ ~/.julia/dev/Muninn/src/models/mass_balance/mass_balance_utils.jl:19
 [15] action!
    @ ~/.julia/dev/ODINN/src/simulations/functional_inversions/functional_inversion_utils.jl:135
 [16] #273
    @ ~/.julia/dev/SciMLSensitivity/src/callback_tracking.jl:388
 [17] #136
    @ ~/.julia/dev/SciMLSensitivity/src/adjoint_common.jl:439 [inlined]
 [18] diffejulia__136_19143_inner_1wrap
    @ ~/.julia/dev/SciMLSensitivity/src/adjoint_common.jl:0
 [19] macro expansion
    @ ~/.julia/packages/Enzyme/F71IJ/src/compiler.jl:5916 [inlined]
 [20] enzyme_call
    @ ~/.julia/packages/Enzyme/F71IJ/src/compiler.jl:5566 [inlined]
 [21] CombinedAdjointThunk
    @ ~/.julia/packages/Enzyme/F71IJ/src/compiler.jl:5443 [inlined]
 [22] autodiff
    @ ~/.julia/packages/Enzyme/F71IJ/src/Enzyme.jl:291 [inlined]
 [23] _vecjacobian!(dλ::Vector{…}, y::Matrix{…}, λ::Vector{…}, p::ComponentArrays.ComponentVector{…}, t::Float64, S::SciMLSensitivity.CallbackSensitivityFunction{…}, isautojacvec::SciMLSensitivity.EnzymeVJP, dgrad::Nothing, dy::Nothing, W::Nothing)
    @ SciMLSensitivity ~/.julia/dev/SciMLSensitivity/src/derivative_wrappers.jl:696
 [24] #vecjacobian!#18
    @ ~/.julia/dev/SciMLSensitivity/src/derivative_wrappers.jl:231 [inlined]
 [25] (::SciMLSensitivity.var"#affect!#272"{…})(integrator::OrdinaryDiffEq.ODEIntegrator{…})
    @ SciMLSensitivity ~/.julia/dev/SciMLSensitivity/src/callback_tracking.jl:338
 [26] (::DiffEqCallbacks.var"#112#116"{…})(c::SciMLBase.DiscreteCallback{…}, u::Vector{…}, t::Float64, integrator::OrdinaryDiffEq.ODEIntegrator{…})
    @ DiffEqCallbacks ~/.julia/packages/DiffEqCallbacks/9fKPq/src/preset_time.jl:75
 [27] initialize!
    @ ~/.julia/packages/DiffEqBase/PBhFc/src/callbacks.jl:13 [inlined]
 [28] initialize! (repeats 2 times)
    @ ~/.julia/packages/DiffEqBase/PBhFc/src/callbacks.jl:14 [inlined]
 [29] initialize!
    @ ~/.julia/packages/DiffEqBase/PBhFc/src/callbacks.jl:7 [inlined]
 [30] initialize_callbacks!(integrator::OrdinaryDiffEq.ODEIntegrator{…}, initialize_save::Bool)
    @ OrdinaryDiffEq ~/.julia/packages/OrdinaryDiffEq/tAI61/src/solve.jl:663
 [31] __init(prob::SciMLBase.ODEProblem{…}, alg::OrdinaryDiffEq.RDPK3Sp35{…}, timeseries_init::Tuple{}, ts_init::Tuple{}, ks_init::Tuple{}, recompile::Type{…}; saveat::Vector{…}, tstops::Vector{…}, d_discontinuities::Tuple{}, save_idxs::Nothing, save_everystep::Bool, save_on::Bool, save_start::Bool, save_end::Bool, callback::SciMLBase.CallbackSet{…}, dense::Bool, calck::Bool, dt::Float64, dtmin::Float64, dtmax::Float64, force_dtmin::Bool, adaptive::Bool, gamma::Rational{…}, abstol::Float64, reltol::Float64, qmin::Rational{…}, qmax::Int64, qsteady_min::Int64, qsteady_max::Int64, beta1::Nothing, beta2::Nothing, qoldinit::Rational{…}, controller::Nothing, fullnormalize::Bool, failfactor::Int64, maxiters::Int64, internalnorm::typeof(DiffEqBase.ODE_DEFAULT_NORM), internalopnorm::typeof(LinearAlgebra.opnorm), isoutofdomain::typeof(DiffEqBase.ODE_DEFAULT_ISOUTOFDOMAIN), unstable_check::typeof(DiffEqBase.ODE_DEFAULT_UNSTABLE_CHECK), verbose::Bool, timeseries_errors::Bool, dense_errors::Bool, advance_to_tstop::Bool, stop_at_next_tstop::Bool, initialize_save::Bool, progress::Bool, progress_steps::Int64, progress_name::String, progress_message::typeof(DiffEqBase.ODE_DEFAULT_PROG_MESSAGE), progress_id::Symbol, userdata::Nothing, allow_extrapolation::Bool, initialize_integrator::Bool, alias_u0::Bool, alias_du0::Bool, initializealg::OrdinaryDiffEq.DefaultInit, kwargs::@Kwargs{})
    @ OrdinaryDiffEq ~/.julia/packages/OrdinaryDiffEq/tAI61/src/solve.jl:517
 [32] __init(prob::Union{…}, alg::Union{…}, timeseries_init::Any, ts_init::Any, ks_init::Any, recompile::Type{…}) where recompile_flag (repeats 5 times)
    @ OrdinaryDiffEq ~/.julia/packages/OrdinaryDiffEq/tAI61/src/solve.jl:11 [inlined]
 [33] #__solve#805
    @ ~/.julia/packages/OrdinaryDiffEq/tAI61/src/solve.jl:6 [inlined]
 [34] __solve
    @ ~/.julia/packages/OrdinaryDiffEq/tAI61/src/solve.jl:1 [inlined]
 [35] solve_call(_prob::SciMLBase.ODEProblem{…}, args::OrdinaryDiffEq.RDPK3Sp35{…}; merge_callbacks::Bool, kwargshandle::Nothing, kwargs::@Kwargs{…})
    @ DiffEqBase ~/.julia/packages/DiffEqBase/PBhFc/src/solve.jl:612
 [36] solve_call
    @ DiffEqBase ~/.julia/packages/DiffEqBase/PBhFc/src/solve.jl:569 [inlined]
 [37] #solve_up#53
    @ DiffEqBase ~/.julia/packages/DiffEqBase/PBhFc/src/solve.jl:1080 [inlined]
 [38] solve_up
    @ DiffEqBase ~/.julia/packages/DiffEqBase/PBhFc/src/solve.jl:1066 [inlined]
 [39] #solve#51
    @ DiffEqBase ~/.julia/packages/DiffEqBase/PBhFc/src/solve.jl:1003 [inlined]
 [40] _adjoint_sensitivities(sol::SciMLBase.ODESolution{…}, sensealg::SciMLSensitivity.GaussAdjoint{…}, alg::OrdinaryDiffEq.RDPK3Sp35{…}; t::Vector{…}, dgdu_discrete::Function, dgdp_discrete::Nothing, dgdu_continuous::Nothing, dgdp_continuous::Nothing, g::Nothing, abstol::Float64, reltol::Float64, checkpoints::Vector{…}, corfunc_analytical::Bool, callback::SciMLBase.CallbackSet{…}, kwargs::@Kwargs{…})
    @ SciMLSensitivity ~/.julia/dev/SciMLSensitivity/src/gauss_adjoint.jl:540
 [41] _adjoint_sensitivities
    @ SciMLSensitivity ~/.julia/dev/SciMLSensitivity/src/gauss_adjoint.jl:507 [inlined]
 [42] #adjoint_sensitivities#63
    @ SciMLSensitivity ~/.julia/dev/SciMLSensitivity/src/sensitivity_interface.jl:386 [inlined]
 [43] (::SciMLSensitivity.var"#adjoint_sensitivity_backpass#314"{…})(Δ::SciMLBase.ODESolution{…})
    @ SciMLSensitivity ~/.julia/dev/SciMLSensitivity/src/concrete_solve.jl:556
 [44] ZBack
    @ ~/.julia/packages/Zygote/nsBv0/src/compiler/chainrules.jl:211 [inlined]
 [45] (::Zygote.var"#kw_zpullback#53"{…})(dy::SciMLBase.ODESolution{…})
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/compiler/chainrules.jl:237
 [46] #291
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/lib/lib.jl:206 [inlined]
 [47] (::Zygote.var"#2169#back#293"{…})(Δ::SciMLBase.ODESolution{…})
    @ Zygote ~/.julia/packages/ZygoteRules/M4xmc/src/adjoint.jl:72
 [48] #solve#51
    @ ~/.julia/packages/DiffEqBase/PBhFc/src/solve.jl:1003 [inlined]
 [49] (::Zygote.Pullback{…})(Δ::SciMLBase.ODESolution{…})
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/compiler/interface2.jl:0
 [50] #291
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/lib/lib.jl:206 [inlined]
 [51] (::Zygote.var"#2169#back#293"{…})(Δ::SciMLBase.ODESolution{…})
    @ Zygote ~/.julia/packages/ZygoteRules/M4xmc/src/adjoint.jl:72
 [52] solve
    @ ~/.julia/packages/DiffEqBase/PBhFc/src/solve.jl:993 [inlined]
 [53] (::Zygote.Pullback{…})(Δ::SciMLBase.ODESolution{…})
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/compiler/interface2.jl:0
 [54] #simulate_iceflow_UDE!#29
    @ ~/.julia/dev/ODINN/src/simulations/functional_inversions/functional_inversion_utils.jl:187 [inlined]
 [55] (::Zygote.Pullback{Tuple{…}, Tuple{…}})(Δ::Nothing)
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/compiler/interface2.jl:0
 [56] simulate_iceflow_UDE!
    @ ~/.julia/dev/ODINN/src/simulations/functional_inversions/functional_inversion_utils.jl:171 [inlined]
 [57] (::Zygote.Pullback{Tuple{…}, Any})(Δ::Nothing)
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/compiler/interface2.jl:0
 [58] batch_iceflow_UDE
    @ ~/.julia/dev/ODINN/src/simulations/functional_inversions/functional_inversion_utils.jl:146 [inlined]
 [59] (::Zygote.Pullback{…})(Δ::@NamedTuple{…})
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/compiler/interface2.jl:0
 [60] #25
    @ ~/.julia/dev/ODINN/src/simulations/functional_inversions/functional_inversion_utils.jl:114 [inlined]
 [61] (::Zygote.Pullback{…})(Δ::@NamedTuple{…})
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/compiler/interface2.jl:0
 [62] #680
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/lib/array.jl:201 [inlined]
 [63] #235
    @ Distributed ~/.julia/juliaup/julia-1.10.0+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Distributed/src/pmap.jl:157 [inlined]
 [64] (::Base.var"#1023#1028"{Distributed.var"#235#236"{…}})(r::Base.RefValue{Any}, args::Tuple{Tuple{…}})
    @ Base ./asyncmap.jl:94
Stacktrace:
  [1] (::Base.var"#1033#1035")(x::Task)
    @ Base ./asyncmap.jl:171
  [2] foreach(f::Base.var"#1033#1035", itr::Vector{Any})
    @ Base ./abstractarray.jl:3094
  [3] maptwice(wrapped_f::Function, chnl::Channel{Any}, worker_tasks::Vector{Any}, c::Base.Iterators.Zip{Tuple{…}})
    @ Base ./asyncmap.jl:171
  [4] wrap_n_exec_twice
    @ ./asyncmap.jl:147 [inlined]
  [5] #async_usemap#1018
    @ ./asyncmap.jl:97 [inlined]
  [6] kwcall(::NamedTuple, ::typeof(Base.async_usemap), f::Any, c::Vararg{Any})
    @ Base ./asyncmap.jl:78 [inlined]
  [7] #asyncmap#1017
    @ ./asyncmap.jl:75 [inlined]
  [8] asyncmap
    @ ./asyncmap.jl:74 [inlined]
  [9] pmap(f::Function, p::Distributed.WorkerPool, c::Base.Iterators.Zip{…}; distributed::Bool, batch_size::Int64, on_error::Nothing, retry_delays::Vector{…}, retry_check::Nothing)
    @ Distributed ~/.julia/juliaup/julia-1.10.0+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Distributed/src/pmap.jl:126
 [10] pmap(f::Function, p::Distributed.WorkerPool, c::Base.Iterators.Zip{Tuple{Vector{Tuple{…}}, Vector{@NamedTuple{…}}}})
    @ Distributed ~/.julia/juliaup/julia-1.10.0+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Distributed/src/pmap.jl:99
 [11] pmap(f::Function, c::Base.Iterators.Zip{Tuple{Vector{Tuple{…}}, Vector{@NamedTuple{…}}}}; kwargs::@Kwargs{})
    @ Distributed ~/.julia/juliaup/julia-1.10.0+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Distributed/src/pmap.jl:156
 [12] pmap
    @ Distributed ~/.julia/juliaup/julia-1.10.0+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Distributed/src/pmap.jl:156 [inlined]
 [13] pmap(f::Function, c1::Vector{Tuple{…}}, c::Vector{@NamedTuple{…}})
    @ Distributed ~/.julia/juliaup/julia-1.10.0+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Distributed/src/pmap.jl:157
 [14] (::Zygote.var"#map_back#682"{ODINN.var"#25#26"{…}, 1, Tuple{…}, Tuple{…}, Vector{…}})(Δ::Vector{@NamedTuple{…}})
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/lib/array.jl:201
 [15] (::Zygote.var"#2861#back#688"{Zygote.var"#map_back#682"{…}})(Δ::Vector{@NamedTuple{…}})
    @ Zygote ~/.julia/packages/ZygoteRules/M4xmc/src/adjoint.jl:72
 [16] predict_iceflow!
    @ ~/.julia/dev/ODINN/src/simulations/functional_inversions/functional_inversion_utils.jl:114 [inlined]
 [17] (::Zygote.Pullback{Tuple{…}, Tuple{…}})(Δ::Nothing)
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/compiler/interface2.jl:0
 [18] loss_iceflow
    @ ~/.julia/dev/ODINN/src/simulations/functional_inversions/functional_inversion_utils.jl:58 [inlined]
 [19] (::Zygote.Pullback{Tuple{…}, Any})(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/compiler/interface2.jl:0
 [20] #22
    @ ~/.julia/dev/ODINN/src/simulations/functional_inversions/functional_inversion_utils.jl:31 [inlined]
 [21] (::Zygote.Pullback{Tuple{…}, Tuple{…}})(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/compiler/interface2.jl:0
 [22] #291
    @ ~/.julia/packages/Zygote/nsBv0/src/lib/lib.jl:206 [inlined]
 [23] #2169#back
    @ ~/.julia/packages/ZygoteRules/M4xmc/src/adjoint.jl:72 [inlined]
 [24] OptimizationFunction
    @ ~/.julia/packages/SciMLBase/JUp1I/src/scimlfunctions.jl:3762 [inlined]
 [25] (::Zygote.Pullback{Tuple{…}, Tuple{…}})(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/compiler/interface2.jl:0
 [26] #291
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/lib/lib.jl:206 [inlined]
 [27] (::Zygote.var"#2169#back#293"{Zygote.var"#291#292"{Tuple{…}, Zygote.Pullback{…}}})(Δ::Float64)
    @ Zygote ~/.julia/packages/ZygoteRules/M4xmc/src/adjoint.jl:72
 [28] #37
    @ ~/.julia/dev/OptimizationBase/ext/OptimizationZygoteExt.jl:90 [inlined]
 [29] (::Zygote.Pullback{Tuple{…}, Tuple{…}})(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/compiler/interface2.jl:0
 [30] #291
    @ ~/.julia/packages/Zygote/nsBv0/src/lib/lib.jl:206 [inlined]
 [31] #2169#back
    @ ~/.julia/packages/ZygoteRules/M4xmc/src/adjoint.jl:72 [inlined]
 [32] #39
    @ ~/.julia/dev/OptimizationBase/ext/OptimizationZygoteExt.jl:93 [inlined]
 [33] (::Zygote.Pullback{Tuple{…}, Tuple{…}})(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/compiler/interface2.jl:0
 [34] (::Zygote.var"#75#76"{Zygote.Pullback{Tuple{…}, Tuple{…}}})(Δ::Float64)
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/compiler/interface.jl:91
 [35] gradient(f::Function, args::ComponentArrays.ComponentVector{Float64, Vector{Float64}, Tuple{ComponentArrays.Axis{…}}})
    @ Zygote ~/.julia/packages/Zygote/nsBv0/src/compiler/interface.jl:148
 [36] (::OptimizationZygoteExt.var"#38#56"{…})(::ComponentArrays.ComponentVector{…}, ::ComponentArrays.ComponentVector{…}, ::Vector{…}, ::Vararg{…})
    @ OptimizationZygoteExt ~/.julia/dev/OptimizationBase/ext/OptimizationZygoteExt.jl:93
 [37] macro expansion
    @ ~/.julia/packages/OptimizationOptimisers/AOkbT/src/OptimizationOptimisers.jl:68 [inlined]
 [38] macro expansion
    @ ~/.julia/packages/ProgressLogging/6KXlp/src/ProgressLogging.jl:328 [inlined]
 [39] (::OptimizationOptimisers.var"#12#13"{OptimizationBase.OptimizationCache{…}, ComponentArrays.ComponentVector{…}})()
    @ OptimizationOptimisers ~/.julia/packages/Optimization/jWtfU/src/utils.jl:29
 [40] with_logstate(f::Function, logstate::Any)
    @ Base.CoreLogging ./logging.jl:515
 [41] with_logger
    @ Base.CoreLogging ./logging.jl:627 [inlined]
 [42] maybe_with_logger(f::OptimizationOptimisers.var"#12#13"{…}, logger::LoggingExtras.TeeLogger{…})
    @ Optimization ~/.julia/packages/Optimization/jWtfU/src/utils.jl:7
 [43] macro expansion
    @ ~/.julia/packages/Optimization/jWtfU/src/utils.jl:28 [inlined]
 [44] __solve(cache::OptimizationBase.OptimizationCache{…})
    @ OptimizationOptimisers ~/.julia/packages/OptimizationOptimisers/AOkbT/src/OptimizationOptimisers.jl:66
 [45] solve!(cache::OptimizationBase.OptimizationCache{…})
    @ SciMLBase ~/.julia/packages/SciMLBase/JUp1I/src/solve.jl:188
 [46] solve(prob::SciMLBase.OptimizationProblem{…}, alg::Optimisers.Adam, args::IterTools.NCycle{…}; kwargs::@Kwargs{…})
    @ SciMLBase ~/.julia/packages/SciMLBase/JUp1I/src/solve.jl:96
 [47] train_UDE!(simulation::FunctionalInversion)
    @ ODINN ~/.julia/dev/ODINN/src/simulations/functional_inversions/functional_inversion_utils.jl:43
 [48] run!(simulation::FunctionalInversion)
    @ ODINN ~/.julia/dev/ODINN/src/simulations/functional_inversions/functional_inversion_utils.jl:11
 [49] top-level scope
    @ ./timing.jl:279
 [50] top-level scope
    @ none:1
Some type information was truncated. Use `show(err)` to see complete types.

However, both functions are inactivated at the top of the file:

https://github.com/ODINN-SciML/ODINN.jl/pull/151/files#diff-d3c9b3bf5488114c648831dcc69c1e95f51dd3e64362e32ed25631401ff945afR37-R39

I can "fix" this by commenting out those lines, and those functions have PyCall interior to them, so that makes it clear that these are blocking Enzyme from building the function. However, because those functions are inactivated, my understanding is that Enzyme shouldn't care what's inside of there for the purpose of differentiation. It seems that something in the compiler pass is getting stuck at what's going on with PyCalls when in this case it should just not care / skip over it.
@ChrisRackauckas ChrisRackauckas mentioned this issue May 29, 2024
Open
@wsmoses
Copy link
Member

wsmoses commented May 30, 2024

I think the answer here is to add a cutom rule to the pyobject constructor which ensures that all the cross language GC is handled properly. Specifically: https://github.com/JuliaPy/PyCall.jl/blob/2f600fbebee50ab0672e153455e3c0fda1694fba/src/PyCall.jl#L53

Ironically we can almost certainly just copy the cuarray rule https://github.com/JuliaGPU/CUDA.jl/blob/bbfd1ad2c3c3ace2502d61c254a6a7f3c48f2197/ext/EnzymeCoreExt.jl#L107 and https://github.com/JuliaGPU/CUDA.jl/blob/bbfd1ad2c3c3ace2502d61c254a6a7f3c48f2197/ext/EnzymeCoreExt.jl#L304 and https://github.com/JuliaGPU/CUDA.jl/blob/bbfd1ad2c3c3ace2502d61c254a6a7f3c48f2197/ext/EnzymeCoreExt.jl#L331

@wsmoses
Copy link
Member

wsmoses commented Jun 16, 2024

Further investigation, I think the issue here is inactive wasn't being checked in the context of kwargs (and also there may have been kwargs here on the call)

Regardless, try #1539 and also possibly doing args...; kwargs... on your inactive marker

@wsmoses
Copy link
Member

wsmoses commented Jun 17, 2024

@ChrisRackauckas going to go ahead and close this. please reopen if it persists

@wsmoses wsmoses closed this as completed Jun 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@wsmoses @ChrisRackauckas