Fix: asyncio.run was sometimes used within a coroutine

[hoh]

asyncio.run was called when initializing a pool object using VmPool.__init__(...).

This caused two issues:

1. The pool was sometimes created from within a coroutine in the context of tests, and this would raise an error.
2. Having side effects inside the __init__ method makes objects more difficult to manipulate and test.
3. Tests should not load persistent executions automatically.
4. The network was configured after loading persistent executions, which could cause networking issues.

A related issue is the snapshot manager being started when initializing the VmPool, while this is not always desirable.

Solution proposed:

1. Explicitly load the persistent executions using pool.load_persistent_executions() from the supervisor.run() function. This is now called after VmPool.setup() and therefore after the networking of the host has been configured.
2. The snapshot manager is now started by VmPool.setup() instead of VmPool.__init__. This function is almost always called just after initializing the pool.
3. Configuring settings.SNAPSHOT_FREQUENCY to zero now disables the snapshot manager.
4. SnapshotManager.run_snapshots is renamed SnapshotManager.run_in_thread to make its behaviour more explicit.

RuntimeError: asyncio.run() cannot be called from a running event loop
(5 additional frame(s) were not displayed)
...
  File "<frozen runpy>", line 88, in _run_code
  File "__main__.py", line 4, in <module>
    main()
  File "aleph/vm/orchestrator/cli.py", line 353, in main
    asyncio.run(benchmark(runs=args.benchmark), debug=args.debug_asyncio)
  File "aleph/vm/orchestrator/cli.py", line 197, in benchmark
    pool = VmPool()
  File "aleph/vm/pool.py", line 73, in __init__
    asyncio.run(self._load_persistent_executions())

[codecov]

Codecov Report

Attention: Patch coverage is 21.42857% with 11 lines in your changes are missing coverage. Please review.

Project coverage is 34.61%. Comparing base (d6025f5) to head (3c384e0).

Files	Patch %	Lines
src/aleph/vm/pool.py	20.00%	8 Missing ⚠️
src/aleph/vm/orchestrator/supervisor.py	0.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #560      +/-   ##
==========================================
- Coverage   34.65%   34.61%   -0.04%     
==========================================
  Files          52       52              
  Lines        4776     4781       +5     
  Branches      558      561       +3     
==========================================
  Hits         1655     1655              
- Misses       3103     3108       +5     
  Partials       18       18              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[nesitor]

Putting the load in this step, will cause some side effects to fail, like the payment tasks, that depends on running executions.
Maybe putting it in the setup step will be better. Also, I noticed that the loading of executions don't start again the snapshots for the running executions. I will create a PR to fix that issue.

[hoh]

Should we call await self.snapshot_manager.start_for(vm=execution.vm) inside pool.load_persistent_executions regarding the snasphot issue ?

I looked at monitor_payments and I don't see the issue to only load persistent executions after starting it.

[nesitor]

Yes, I think so, is the snapshot_manager is set, we need to call await self.snapshot_manager.start_for(vm=execution.vm) for every execution already running.

And about the monitor_payment method, the issue can appear at the moment that the executions aren't loaded yet. Yes, will not raise any error, but we will need to wait to the next loop to check the payments for running executions. In this case isn't a big deal, but there will be another logic involved apart from payments.

[hoh]

Snapshot manager started in fefede9