Replace Distributed with Malt soon! #2240

savq · 2022-08-05T16:59:42Z

After this PR, we will no longer use the Distributed stdlib to start and control the notebook Julia process (where your code runs) but we use our home-brewed package Malt.jl. The advantages are:

It is not Distributed, which means that your code does not run as a Distributed worker. This fixes Allow using Distributed #300.
The process launches faster, about 1 second with Malt instead of 4 seconds with Distributed.
Slightly improved developer experience: the API of Malt is more intuitive for our use-case and we need fewer workarounds.
Room for future improvement: splitting Malt into its own package makes it much easier to improve our multiprocessing functionality (because the codebase and test/benchmark suite is much smaller). This might include:
- Supporting interrupt on Windows
- Faster data transfer
- Crazy stuff like criu based worker JuliaPluto/Malt.jl#46

This PR replaces Distributed with Malt.jl to add support for Distributed itself. Fix #300

Test release

After this PR is merged, we have a testing period where Pluto still uses Distributed, but you can enable Malt with:

Pluto.run(workspace_use_distributed_stdlib=false)

This PR is part of GSoC 2022. Some info on that here.

github-actions · 2022-08-05T16:59:54Z

Try this Pull Request!

Open Julia and type:

julia> import Pkg
julia> Pkg.activate(temp=true)
julia> Pkg.add(url="https://github.com/savq/Pluto.jl", rev="malt")
julia> using Pluto

Project.toml

src/evaluation/WorkspaceManager.jl

src/Configuration.jl

disberd · 2022-10-11T08:38:28Z

Would this change allow to interrupt cells on windows as well, or is that broken anyhow?

- Prefer quote blocks to colon quote expressions - Prefer if blocks to ternary operators - Remove single pipes - Minor changes to docstrings and comments

- Remove `workspace_use_distributed` flag. Pluto will now always use separate processes. - Replace calls to `remotecall_eval` with the appropiate Malt calls. - Replace Distributed with Malt in tests to remove it from dependencies. - Add `unmake_workspace` calls in `MacroAnalysis` to prevent creating too many processes at the same time. - Replace Distributed exceptions with Malt exceptions. - Replace `pid::Integer` with `worker::Worker` in WorkspaceManager. - Rename `workspaces` to `active_workspaces`. - Rename `distributed_exception_result` to `workspace_exception_result`. - Rename `make_distributed_serializable` to `make_serializable`.

fonsp · 2023-09-13T13:34:16Z

Wowww well found @Pangoraw !
This is currently set on all distributed procs for every OS, right? Let's do the same by default on Malt directly?

fonsp · 2023-09-13T13:34:34Z

Not sure about the nightly test, and on 1.6 the test never seems to start...

Pangoraw · 2023-09-13T13:47:00Z

This is currently set on all distributed procs for every OS, right? Let's do the same by default on Malt directly?

yes, this is probably the right move.

For nightly, we could test 1.10.0-beta2 instead? and for 1.6, I don't know...

fonsp · 2023-09-13T14:05:42Z

Which 1.10 are we testing right now? Testing the beta sounds good!

I also made #2643 to avoid the interrupt call. This means avoiding the problem instead of fixing it, but I don't think that supporting schedule(pluto_server_task, InterruptException()) on Windows is something we need to test. It's important that Ctrl+C in the terminal shuts down the server, but I think that we are simulating this wrong with schedule, so I don't really know how to test for that.

Pangoraw · 2023-09-13T14:10:39Z

nightly is actually julia master (future 1.11), it would be nice if a docker tag for 1.10 existed so that we can get the alphas/betas & stable release for 1.10 automatically.

fonsp · 2023-09-13T17:18:01Z

Still failing with out-of-memory on Windows Julia 1.6 :(

fonsp · 2023-09-16T20:44:58Z

Maybe we just merge for testing (disabled by default), and eventually only enable Malt as default on Julia 1.8 and up?

@Pangoraw wdyt, any other ideas?

fonsp · 2023-09-16T20:47:13Z

Or @savq any ideas why we have this last failure with Malt julia 1.6 windows, but not with Distributed backend or another julia distro?

Pangoraw · 2023-09-17T18:37:33Z

@Pangoraw wdyt, any other ideas?

I don't have more ideas, we might have to test in real windows and see if we can spot the problem here.

fonsp · 2023-09-18T08:15:38Z

(btw some improvement to Crrl+C on windows in JuliaLang/julia#51307)

This reverts commit 82de248.

This reverts commit 45e6e1b.

pankgeorg · 2023-09-18T10:36:49Z

(btw some improvement to Crrl+C on windows in JuliaLang/julia#51307)

I do not see an improvement, see https://julialang.slack.com/archives/C6SMTHQ3T/p1694888153763359?thread_ts=1694639864.397389&cid=C6SMTHQ3T (which contains that PR)

fonsp · 2023-09-18T13:30:26Z

Heyy so I think I know what happened!

This initial PR removed the workspace_use_distributed option altogether, then later we added it to Malt (JuliaPluto/Malt.jl#39) but some of the tests in the PR still had the workspace_use_distributed=false setting removed.

I fixed this in two recent commits: 7b09e1d (#2240) and a37fd31 (#2240) and then suddenly the tests passed with Malt enabled! fd8c68a (#2240)

I noticed that we were also getting OOM errors with the Distributed backend as default (27307e3 (#2240)) which means that it's not caused by Malt :) It's just some freaky combination of starting/closing an HTTP server while starting a Julia process that does not work on Windows.

Yayy so I will merge this with Malt still disabled by default, for the testing period, and then we can enable it by default soon (for all Julia versions)!

Pangoraw · 2023-09-18T13:42:57Z

Awesome!! 🎉 🎉

savq force-pushed the malt branch from 04cecba to eabf4fc Compare August 28, 2022 04:46

rikhuijzer reviewed Aug 28, 2022

View reviewed changes

Project.toml Show resolved Hide resolved

rikhuijzer reviewed Aug 28, 2022

View reviewed changes

src/evaluation/WorkspaceManager.jl Outdated Show resolved Hide resolved

savq force-pushed the malt branch 3 times, most recently from afd0fa0 to 9c6a566 Compare September 4, 2022 20:50

fonsp reviewed Sep 5, 2022

View reviewed changes

src/Configuration.jl Show resolved Hide resolved

savq force-pushed the malt branch 4 times, most recently from 344ac94 to 8beea21 Compare September 24, 2022 23:53

savq force-pushed the malt branch 2 times, most recently from e528ace to e948e7b Compare October 1, 2022 07:00

Format WorkspaceManager

906838c

- Prefer quote blocks to colon quote expressions - Prefer if blocks to ternary operators - Remove single pipes - Minor changes to docstrings and comments

savq force-pushed the malt branch from e948e7b to 82ad7fe Compare October 15, 2022 07:10

savq force-pushed the malt branch from 82ad7fe to b1463dd Compare October 15, 2022 07:11

savq mentioned this pull request Oct 16, 2022

Refactor WorkspaceManager #2318

Merged

fonsp assigned fonsp, Pangoraw and savq Nov 16, 2022

VarLad mentioned this pull request Feb 4, 2023

Doesn't work on Pluto.jl JuliaConstraints/PerfChecker.jl#14

Closed

fonsp mentioned this pull request Mar 7, 2023

Reuse the TCP socket and buffer IO to improve performance JuliaPluto/Malt.jl#32

Merged

17 tasks

fonsp added backend Concerning the julia server and runtime wide audience This affects a wide audience of Pluto users and future Pluto users performance labels Apr 26, 2023

fonsp added 2 commits July 9, 2023 09:58

Discard changes to test/Bonds.jl

ec8d989

Discard changes to test/Logging.jl

a3ca342

fonsp mentioned this pull request Sep 13, 2023

Non-blocking API Pluto.run! that returns a reference to the running server #2643

Merged

Merge branch 'main' into malt

97deb81

fonsp added 2 commits September 13, 2023 21:41

Merge branch 'main' into malt

ed12abe

trying JULIA_NUM_PRECOMPILE_TASKS=1

82de248

Revert "trying JULIA_NUM_PRECOMPILE_TASKS=1"

d542e4c

This reverts commit 82de248.

fonsp mentioned this pull request Sep 18, 2023

OPENBLAS_NUM_THREADS=1 by default JuliaPluto/Malt.jl#59

Merged

fonsp added 3 commits September 18, 2023 11:48

Revert "Test setting OPENBLAS_NUM_THREADS=1 on windows for Malt.Worker"

07101de

This reverts commit 45e6e1b.

Disable Malt by default

27307e3

Put back some workspace_use_distributed=false in tests

7b09e1d

fonsp added 3 commits September 18, 2023 13:59

one more distributed test disable

a37fd31

test Malt by default again...

fd8c68a

Disable Malt by default again

0106663

fonsp changed the title ~~Replace Distributed with Malt~~ Replace Distributed with Malt soon! Sep 18, 2023

fonsp merged commit a7a424c into fonsp:main Sep 18, 2023
14 of 15 checks passed

fonsp mentioned this pull request Sep 18, 2023

Allow using Distributed #300

Closed

fonsp mentioned this pull request Sep 27, 2023

Support interrupt on Windows soon! #2659

Merged

fonsp mentioned this pull request Jan 3, 2024

Enable Malt.jl by default on all systems #2767

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace Distributed with Malt soon! #2240

Replace Distributed with Malt soon! #2240

savq commented Aug 5, 2022 •

edited by fonsp

Loading

github-actions bot commented Aug 5, 2022

disberd commented Oct 11, 2022

fonsp commented Sep 13, 2023

fonsp commented Sep 13, 2023

Pangoraw commented Sep 13, 2023

fonsp commented Sep 13, 2023

Pangoraw commented Sep 13, 2023

fonsp commented Sep 13, 2023

fonsp commented Sep 16, 2023 •

edited

Loading

fonsp commented Sep 16, 2023

Pangoraw commented Sep 17, 2023

fonsp commented Sep 18, 2023

pankgeorg commented Sep 18, 2023

fonsp commented Sep 18, 2023 •

edited

Loading

Pangoraw commented Sep 18, 2023

Replace Distributed with Malt soon! #2240

Replace Distributed with Malt soon! #2240

Conversation

savq commented Aug 5, 2022 • edited by fonsp Loading

Test release

github-actions bot commented Aug 5, 2022

Try this Pull Request!

disberd commented Oct 11, 2022

fonsp commented Sep 13, 2023

fonsp commented Sep 13, 2023

Pangoraw commented Sep 13, 2023

fonsp commented Sep 13, 2023

Pangoraw commented Sep 13, 2023

fonsp commented Sep 13, 2023

fonsp commented Sep 16, 2023 • edited Loading

fonsp commented Sep 16, 2023

Pangoraw commented Sep 17, 2023

fonsp commented Sep 18, 2023

pankgeorg commented Sep 18, 2023

fonsp commented Sep 18, 2023 • edited Loading

Pangoraw commented Sep 18, 2023

savq commented Aug 5, 2022 •

edited by fonsp

Loading

fonsp commented Sep 16, 2023 •

edited

Loading

fonsp commented Sep 18, 2023 •

edited

Loading