Error precompiling SymbolicsPreallocationToolsExt
#53081
Comments
|
ForwardDiff is an indirect dep of Symbolics exts: |
This was referenced Jan 30, 2024
|
The issue is still present. |
KristofferC
added a commit
that referenced
this issue
May 9, 2024
… package precompilation (#53972) In the parallel package precompilation code we are mapping what packages depend on what other packages so that we precompile things in the correct order ("bottom up") and so what we can also detect cycles and avoid precompiling packages in those cycles. However, we fail to detect one type of dependency which is that an extension can "indirectly" depend on another extension. This happens when the transitive dependencies of the extension (it's parent + triggers) are a superset of the dependencies of another extension. In other words, this means that the other extension will get loaded into the first extension once it gets loaded, effectively being a dependency. The failure to model this leads to some issues, for example using one of the examples in our own tests: ```julia julia> Base.active_project() "/home/kc/julia/test/project/Extensions/HasDepWithExtensions.jl/Project.toml" (HasDepWithExtensions) pkg> status --extensions Project HasDepWithExtensions v0.1.0 Status `~/julia/test/project/Extensions/HasDepWithExtensions.jl/Project.toml` [4d3288b3] HasExtensions v0.1.0 `../HasExtensions.jl` ├─ ExtensionFolder [ExtDep, ExtDep2] ├─ Extension [ExtDep] └─ LinearAlgebraExt [LinearAlgebra] julia> Base.Precompilation.precompilepkgs(; timing=true) Precompiling all packages... 196.1 ms ✓ HasExtensions 244.4 ms ✓ ExtDep2 207.9 ms ✓ SomePackage 201.6 ms ✓ ExtDep 462.5 ms ✓ HasExtensions → ExtensionFolder 200.1 ms ✓ HasExtensions → Extension 173.1 ms ✓ HasExtensions → LinearAlgebraExt 222.2 ms ✓ HasDepWithExtensions 8 dependencies successfully precompiled in 2 seconds julia> Base.Precompilation.precompilepkgs(; timing=true) Precompiling all packages... 213.4 ms ✓ HasExtensions → ExtensionFolder 1 dependency successfully precompiled in 0 seconds. 7 already precompiled. julia> Base.Precompilation.precompilepkgs(; timing=true) julia> ``` We can see here that `ExtensionFolder` gets precompiled twice which is due to `Extension` actually being an "indirect dependency" of `ExtensionFolder` and therefore should be precompiled before it. With this PR we instead get: ```julia julia> Precompilation.precompilepkgs(; timing=true) Precompiling all packages... 347.5 ms ✓ ExtDep2 294.0 ms ✓ SomePackage 293.2 ms ✓ HasExtensions 216.5 ms ✓ HasExtensions → LinearAlgebraExt 554.9 ms ✓ ExtDep 580.9 ms ✓ HasExtensions → Extension 593.8 ms ✓ HasExtensions → ExtensionFolder 261.3 ms ✓ HasDepWithExtensions 8 dependencies successfully precompiled in 2 seconds julia> Precompilation.precompilepkgs(; timing=true) julia> ``` `Extension` is precompiled after `ExtensionFolder` and nothing happens on the second call. Also, with this PR we get for the issue in #53081 (comment): ```julia (jl_zuuRGt) pkg> st Status `/private/var/folders/tp/2p4x9ygx48sgsdl1ccg1mp_40000gn/T/jl_zuuRGt/Project.toml` ⌃ [d236fae5] PreallocationTools v0.4.17 ⌃ [0c5d862f] Symbolics v5.16.1 Info Packages marked with ⌃ have new versions available and may be upgradable. julia> Precompilation.precompilepkgs(; timing=true) ┌ Warning: Circular dependency detected. Precompilation will be skipped for: │ SymbolicsPreallocationToolsExt │ SymbolicsForwardDiffExt ``` and we avoid precompiling the problematic extensions. This should also allow extensions to precompile in parallel which I think was prevented before (from the code that is removed in the beginning of the diff).
|
Fixed by #53972 |
KristofferC
added a commit
that referenced
this issue
May 27, 2024
… package precompilation (#53972) In the parallel package precompilation code we are mapping what packages depend on what other packages so that we precompile things in the correct order ("bottom up") and so what we can also detect cycles and avoid precompiling packages in those cycles. However, we fail to detect one type of dependency which is that an extension can "indirectly" depend on another extension. This happens when the transitive dependencies of the extension (it's parent + triggers) are a superset of the dependencies of another extension. In other words, this means that the other extension will get loaded into the first extension once it gets loaded, effectively being a dependency. The failure to model this leads to some issues, for example using one of the examples in our own tests: ```julia julia> Base.active_project() "/home/kc/julia/test/project/Extensions/HasDepWithExtensions.jl/Project.toml" (HasDepWithExtensions) pkg> status --extensions Project HasDepWithExtensions v0.1.0 Status `~/julia/test/project/Extensions/HasDepWithExtensions.jl/Project.toml` [4d3288b3] HasExtensions v0.1.0 `../HasExtensions.jl` ├─ ExtensionFolder [ExtDep, ExtDep2] ├─ Extension [ExtDep] └─ LinearAlgebraExt [LinearAlgebra] julia> Base.Precompilation.precompilepkgs(; timing=true) Precompiling all packages... 196.1 ms ✓ HasExtensions 244.4 ms ✓ ExtDep2 207.9 ms ✓ SomePackage 201.6 ms ✓ ExtDep 462.5 ms ✓ HasExtensions → ExtensionFolder 200.1 ms ✓ HasExtensions → Extension 173.1 ms ✓ HasExtensions → LinearAlgebraExt 222.2 ms ✓ HasDepWithExtensions 8 dependencies successfully precompiled in 2 seconds julia> Base.Precompilation.precompilepkgs(; timing=true) Precompiling all packages... 213.4 ms ✓ HasExtensions → ExtensionFolder 1 dependency successfully precompiled in 0 seconds. 7 already precompiled. julia> Base.Precompilation.precompilepkgs(; timing=true) julia> ``` We can see here that `ExtensionFolder` gets precompiled twice which is due to `Extension` actually being an "indirect dependency" of `ExtensionFolder` and therefore should be precompiled before it. With this PR we instead get: ```julia julia> Precompilation.precompilepkgs(; timing=true) Precompiling all packages... 347.5 ms ✓ ExtDep2 294.0 ms ✓ SomePackage 293.2 ms ✓ HasExtensions 216.5 ms ✓ HasExtensions → LinearAlgebraExt 554.9 ms ✓ ExtDep 580.9 ms ✓ HasExtensions → Extension 593.8 ms ✓ HasExtensions → ExtensionFolder 261.3 ms ✓ HasDepWithExtensions 8 dependencies successfully precompiled in 2 seconds julia> Precompilation.precompilepkgs(; timing=true) julia> ``` `Extension` is precompiled after `ExtensionFolder` and nothing happens on the second call. Also, with this PR we get for the issue in #53081 (comment): ```julia (jl_zuuRGt) pkg> st Status `/private/var/folders/tp/2p4x9ygx48sgsdl1ccg1mp_40000gn/T/jl_zuuRGt/Project.toml` ⌃ [d236fae5] PreallocationTools v0.4.17 ⌃ [0c5d862f] Symbolics v5.16.1 Info Packages marked with ⌃ have new versions available and may be upgradable. julia> Precompilation.precompilepkgs(; timing=true) ┌ Warning: Circular dependency detected. Precompilation will be skipped for: │ SymbolicsPreallocationToolsExt │ SymbolicsForwardDiffExt ``` and we avoid precompiling the problematic extensions. This should also allow extensions to precompile in parallel which I think was prevented before (from the code that is removed in the beginning of the diff). (cherry picked from commit df89440)
lazarusA
pushed a commit
to lazarusA/julia
that referenced
this issue
Jul 12, 2024
… package precompilation (JuliaLang#53972) In the parallel package precompilation code we are mapping what packages depend on what other packages so that we precompile things in the correct order ("bottom up") and so what we can also detect cycles and avoid precompiling packages in those cycles. However, we fail to detect one type of dependency which is that an extension can "indirectly" depend on another extension. This happens when the transitive dependencies of the extension (it's parent + triggers) are a superset of the dependencies of another extension. In other words, this means that the other extension will get loaded into the first extension once it gets loaded, effectively being a dependency. The failure to model this leads to some issues, for example using one of the examples in our own tests: ```julia julia> Base.active_project() "/home/kc/julia/test/project/Extensions/HasDepWithExtensions.jl/Project.toml" (HasDepWithExtensions) pkg> status --extensions Project HasDepWithExtensions v0.1.0 Status `~/julia/test/project/Extensions/HasDepWithExtensions.jl/Project.toml` [4d3288b3] HasExtensions v0.1.0 `../HasExtensions.jl` ├─ ExtensionFolder [ExtDep, ExtDep2] ├─ Extension [ExtDep] └─ LinearAlgebraExt [LinearAlgebra] julia> Base.Precompilation.precompilepkgs(; timing=true) Precompiling all packages... 196.1 ms ✓ HasExtensions 244.4 ms ✓ ExtDep2 207.9 ms ✓ SomePackage 201.6 ms ✓ ExtDep 462.5 ms ✓ HasExtensions → ExtensionFolder 200.1 ms ✓ HasExtensions → Extension 173.1 ms ✓ HasExtensions → LinearAlgebraExt 222.2 ms ✓ HasDepWithExtensions 8 dependencies successfully precompiled in 2 seconds julia> Base.Precompilation.precompilepkgs(; timing=true) Precompiling all packages... 213.4 ms ✓ HasExtensions → ExtensionFolder 1 dependency successfully precompiled in 0 seconds. 7 already precompiled. julia> Base.Precompilation.precompilepkgs(; timing=true) julia> ``` We can see here that `ExtensionFolder` gets precompiled twice which is due to `Extension` actually being an "indirect dependency" of `ExtensionFolder` and therefore should be precompiled before it. With this PR we instead get: ```julia julia> Precompilation.precompilepkgs(; timing=true) Precompiling all packages... 347.5 ms ✓ ExtDep2 294.0 ms ✓ SomePackage 293.2 ms ✓ HasExtensions 216.5 ms ✓ HasExtensions → LinearAlgebraExt 554.9 ms ✓ ExtDep 580.9 ms ✓ HasExtensions → Extension 593.8 ms ✓ HasExtensions → ExtensionFolder 261.3 ms ✓ HasDepWithExtensions 8 dependencies successfully precompiled in 2 seconds julia> Precompilation.precompilepkgs(; timing=true) julia> ``` `Extension` is precompiled after `ExtensionFolder` and nothing happens on the second call. Also, with this PR we get for the issue in JuliaLang#53081 (comment): ```julia (jl_zuuRGt) pkg> st Status `/private/var/folders/tp/2p4x9ygx48sgsdl1ccg1mp_40000gn/T/jl_zuuRGt/Project.toml` ⌃ [d236fae5] PreallocationTools v0.4.17 ⌃ [0c5d862f] Symbolics v5.16.1 Info Packages marked with ⌃ have new versions available and may be upgradable. julia> Precompilation.precompilepkgs(; timing=true) ┌ Warning: Circular dependency detected. Precompilation will be skipped for: │ SymbolicsPreallocationToolsExt │ SymbolicsForwardDiffExt ``` and we avoid precompiling the problematic extensions. This should also allow extensions to precompile in parallel which I think was prevented before (from the code that is removed in the beginning of the diff).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This may have overlap with other open extensions issues, but here's a tracking issue to debug it.
A couple of things of note:
SymbolicsPreallocationToolsExtdoesn't actually fail its precompile job, just@error'sSymbolicsPreallocationToolsExtis being loaded in the precompile worker via a trigger before it specifically tries to precompileSymbolicsPreallocationToolsExtand I assume succeeds because the worker doesn't failcc. @ChrisRackauckas
The text was updated successfully, but these errors were encountered: