BAT dependencies, package load time(s) and modularization #351
Comments
oschulz
changed the title
Even with v3 splitting out the solvers? |
Interesting, how expensive, and due to what? There's not much in there 😅 |
I was very surprised as well, I had always assumed SciMLBase to be very lightweight, I just never timed it before.
Hm, maybe I had GalacticOptim v2 due to some dep in my tests - though even with v3 it has a not insignificant load time: Session 1: Session 2: Session 3: Also, in comparison (though Optim and GalacticOptim are not actually replacements for each other of course) - loading Optim: julia> using InverseFunctions
julia> @time using Distributions, StaticArrays, ArrayInterface # Pretty much unavoidable for BAT
1.918963 seconds (4.79 M allocations: 333.435 MiB, 3.12% gc time, 37.83% compilation time)
julia> @time using Optim
0.181226 seconds (475.28 k allocations: 30.906 MiB, 14.12% gc time, 13.72% compilation time)Loading GalacticOptim: julia> using InverseFunctions
julia> @time using Distributions, StaticArrays, ArrayInterface # Unavoidable deps for BAT
1.921450 seconds (4.79 M allocations: 333.419 MiB, 3.04% gc time, 37.15% compilation time)
julia> @time using GalacticOptim
0.936281 seconds (2.32 M allocations: 181.437 MiB, 3.88% gc time, 59.31% compilation time)
julia> @time using SciMLBase
0.595445 seconds (861.23 k allocations: 45.663 MiB, 6.73% gc time, 99.98% compilation time)Why does loading SciMLBase after GalacticOptim take any time at all? This is really weird: julia> using InverseFunctions
julia> @time using Distributions, StaticArrays, ArrayInterface # Unavoidable deps for BAT
1.913109 seconds (4.79 M allocations: 333.435 MiB, 3.06% gc time, 37.17% compilation time)
julia> @time using SciMLBase, GalacticOptim
1.513281 seconds (3.17 M allocations: 227.054 MiB, 4.97% gc time, 75.43% compilation time)Why is loading SciMLBase and GalacticOptim more expensive than just loading GalacticOptim, which depends on SciMLBase? Some strange Requires effect? |
|
SciMLBase seems to have some strange load time effects depending on order of package loading in general. I don't get why ... When timing it all in one go it's not so extreme, but around 300 ms still seems to be very high for the actual code of a "...Base" package: julia> using InverseFunctions
julia> @time_imports using StaticArrays, ArrayInterface, RecursiveArrayTools, SciMLBase
527.0 ms StaticArrays
3.3 ms ┌ Compat
0.4 ms ┌ Requires
0.1 ms ┌ IfElse
36.9 ms ┌ Static
721.1 ms ArrayInterface
10.4 ms ┌ MacroTools
11.0 ms ┌ ZygoteRules
149.6 ms ┌ FillArrays
3.3 ms ┌ DocStringExtensions
19.0 ms ┌ RecipesBase
0.6 ms ┌ Adapt
59.7 ms ┌ ChainRulesCore
309.3 ms RecursiveArrayTools
0.2 ms ┌ IteratorInterfaceExtensions
0.6 ms ┌ TableTraits
0.2 ms ┌ DataValueInterfaces
1.3 ms ┌ DataAPI
14.5 ms ┌ Tables
0.2 ms ┌ CommonSolve
0.8 ms ┌ ConstructionBase
0.2 ms ┌ TreeViews
307.5 ms SciMLBaseStaticArrays and especially ArrayInterface make up more of the total load time, of course, together with RecursiveArrayTools which is also not exactly lightweight. |
|
I wonder how requires is measured. My guess is that it's triggering requires in ArrayInterface and that's measured as part of the SciMLBase time. |
|
I would think so ... probably have to ask Tim or so. :-) |
Oh wow, ArrayInterface has a lot of requires! |
|
Hence the idea to make it like GalacticOptim in terms of subpackages. |
|
Update: We have a very lightweight ArrayInterfaceCore now. |
Indeed, that's the dream. |
This issue is intended to keep track of modularizing (splitting up) BAT to reduce package load time(s) and increase flexibility.
The current high load time of BAT is mainly due to it's dependencies. Dependency optiions and load-time cost, preliminary analysis:
Unavoidable expensive core deps:
Unavoidable non-negligible deps:
Hard-to-avoid non-negligible direct/indirect deps
Cheap deps:
Autodiff choices:
Math deps:
Statistics deps:
Optimizer deps:
BAT would need only Nelder-Mead and L-BFGS for core functionality though
Sampler deps:
Integator deps:
Deps to avoid:
Expensive deps to get rid of:
Non-negligible deps to get rid of:
The text was updated successfully, but these errors were encountered: