Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

create_app: load "deps" packages besides just the app package #463

Closed
wants to merge 8 commits into from
Closed

create_app: load "deps" packages besides just the app package #463

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
6aa2fb2
Log precompile _failures_ as well as errors. Pass log level through t…
NHDaly Oct 16, 2020
5dc40a1
Log the precompile statements file so users can inspect it
NHDaly Oct 19, 2020
2e50a35
Log a total count of all precompile statements successes, failures, a…
NHDaly Oct 19, 2020
5409fb3
create_app: load all packages in Project.toml
NHDaly Oct 23, 2020
0155b7c
Merge remote-tracking branch 'origin/master' into nhd-Log-precompile-…
NHDaly Oct 23, 2020
2d9620e
Merge branch 'nhd-Log-precompile-failures' into nhd-app-all-packages
NHDaly Oct 23, 2020
bbe20da
Apply suggestions from code review
NHDaly Oct 23, 2020
af09bd8
Fix suggestion from code-review: `packages = Symbol.(keys(ctx.env.pro…
NHDaly Oct 23, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Project.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ version = "1.2.3"

[deps]
Libdl = "8f399da3-3557-5675-b5ff-fb832c97cbdb"
Logging = "56ddb016-857b-54e1-b83d-db4d58db5568"
Pkg = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"
UUIDs = "cf7118a7-6976-5b1a-9a39-7adc72f591a4"

Expand Down
62 changes: 47 additions & 15 deletions src/PackageCompiler.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ using Base: active_project
using Libdl: Libdl
using Pkg: Pkg
using UUIDs: UUID, uuid1
using Logging

export create_sysimage, create_app, audit_app, restore_default_sysimage

Expand Down Expand Up @@ -201,7 +202,11 @@ function create_sysimg_object_file(object_file::String, packages::Vector{String}
@debug "running precompilation execution script..."
tracefiles = String[]
for file in (isempty(precompile_execution_file) ? (nothing,) : precompile_execution_file)
# TODO(PR): Even better, can we compute the loaded_modules at the end of running
# each precompilation_script, and then add those into the supplied `packages`?
# Maybe only if `isapp == true`?
tracefile = run_precompilation_script(project, base_sysimage, file)
@debug "precompile statements written to: $tracefile"
precompile_statements *= " append!(precompile_statements, readlines($(repr(tracefile))))\n"
end
for file in precompile_statements_file
Expand All @@ -212,25 +217,41 @@ function create_sysimg_object_file(object_file::String, packages::Vector{String}
precompile_code = """
# This @eval prevents symbols from being put into Main
@eval Module() begin

using Logging
# Pass through the user's logging level to the spawned process.
logger = ConsoleLogger(stderr, Logging.$(Logging.min_enabled_level(global_logger())))
global_logger(logger)

PrecompileStagingArea = Module()
@show Base.loaded_modules
for (_pkgid, _mod) in Base.loaded_modules
if !(_pkgid.name in ("Main", "Core", "Base"))
eval(PrecompileStagingArea, :(const \$(Symbol(_mod)) = \$_mod))
end
end
precompile_statements = String[]
$precompile_statements
for statement in sort(precompile_statements)
# println(statement)
# The compiler has problem caching signatures with `Vararg{?, N}`. Replacing
# N with a large number seems to work around it.
statement = replace(statement, r"Vararg{(.*?), N} where N" => s"Vararg{\\1, 100}")
try
Base.include_string(PrecompileStagingArea, statement)
catch
# See julia issue #28808
@debug "failed to execute \$statement"
let (failures, errors) = (0, 0)
precompile_statements = String[]
$precompile_statements
for statement in sort(precompile_statements)
# println(statement)
# The compiler has problem caching signatures with `Vararg{?, N}`. Replacing
# N with a large number seems to work around it.
statement = replace(statement, r"Vararg{(.*?), N} where N" => s"Vararg{\\1, 100}")
try
success = Base.include_string(PrecompileStagingArea, statement)
if !success
failures += 1
@debug "Precompilation failed: \$statement"
end
catch e
# See julia issue #28808
errors += 1
@debug "Error executing \$statement:\n\$e"
end
end
total = length(precompile_statements)
@info "PackageCompiler: completed \$total precompile statements with \$failures failures, \$errors errors."
end
end # module
"""
Expand Down Expand Up @@ -268,7 +289,7 @@ function create_sysimg_object_file(object_file::String, packages::Vector{String}

if isapp
# If it is an app, there is only one packages
@assert length(packages) == 1
#@assert length(packages) == 1
NHDaly marked this conversation as resolved.
Show resolved Hide resolved
packages[1]
app_start_code = """
Base.@ccallable function julia_main()::Cint
Expand Down Expand Up @@ -622,6 +643,14 @@ function create_app(package_dir::String,
bundle_julia_libraries(app_dir)
bundle_artifacts(ctx, app_dir)

# Load all packages in the Project into the sysimg for Apps. Without specifying this,
# we cannot serialize any methods from those packages, because the packages won't be
# loaded in the final julia process. Since we cannot know which packages will be needed
# we can load _all_ of the packages (and their recursive packages) from the project.
# This may be more than the user will actually need, but that is up to users to prune
# their Project files to only those `deps` actually in use.
packages = Symbol.(keys(Pkg.TOML.parsefile(ctx.env.project_file)["deps"]))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

julia> ctx.env.project.deps
Dict{String,Base.UUID} with 37 entries:
  "BinaryBuilder"           => UUID("12aac903-9f7c-5d81-afc2-d9565ea332ae")
  "Documenter"              => UUID("e30172f5-a6a5-5a46-863b-614d45cd2de4")
  "ForwardDiff"             => UUID("f6369f11-7733-5829-9624-2563aa707210")
...

is probably better

Copy link
Member Author

@NHDaly NHDaly Oct 23, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
packages = Symbol.(keys(Pkg.TOML.parsefile(ctx.env.project_file)["deps"]))
packages = keys(ctx.env.project.deps)

Awesome thanks. Does this look good? ⬆️

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, I guess that should be an array of symbols:

    packages = Symbol.(keys(ctx.env.project.deps))

Fixed in af09bd8


# TODO: Create in a temp dir and then move it into place?
binpath = joinpath(app_dir, "bin")
mkpath(binpath)
Expand All @@ -636,15 +665,18 @@ function create_app(package_dir::String,
incremental=false, filter_stdlibs=filter_stdlibs,
cpu_target=cpu_target)

create_sysimage(Symbol(sysimg_name); sysimage_path=sysimg_file, project=package_dir,
create_sysimage([Symbol(sysimg_name); packages...];
sysimage_path=sysimg_file,
project=package_dir,
incremental=true,
precompile_execution_file=precompile_execution_file,
precompile_statements_file=precompile_statements_file,
cpu_target=cpu_target,
base_sysimage=tmp_base_sysimage,
isapp=true)
else
create_sysimage(Symbol(sysimg_name); sysimage_path=sysimg_file, project=package_dir,
create_sysimage([Symbol(sysimg_name); packages...];
sysimage_path=sysimg_file, project=package_dir,
incremental=incremental, filter_stdlibs=filter_stdlibs,
precompile_execution_file=precompile_execution_file,
precompile_statements_file=precompile_statements_file,
Expand Down