Measure actual spilled bytes, not output of sizeof()

[crusaderky]

Closes #5364

#5543 changed the measure of spilled memory from the output of sizeof() to the actual number of bytes written to disk. This introduced a substantial difference in case of

1. highly compressible data*
2. data where sizeof() returns an inaccurate output
3. numpy views, which become real arrays upon serialization.

*Compression upon spill requires lz4 or snappy to be installed. Notably, in our CI they're exclusively installed on Python 3.9. See #5807.

The same PR introduced a regression, where this new difference would add or detract to the managed_in_memory measure, which in turn would detract or add to the unmanaged memory.

Example

"x" * (50 * 2**20) occupies 50 MiB in RAM, but only 200 kiB on disk.

Before #5543, you would see 50 MiB on the dashboard both if the value was in RAM or spilled to disk.

After #5543, when the value was spilled to disk you would read on the dashboard:

• 200 kiB spilled memory (grey)
• 49.8 MiB more managed_in_memory (solid color) than you actually have
• 49.8 MiB less unmanaged memory (semi transparent) than you actually have

This PR removes the above artifact.

Demo

import time
from distributed import Client

c = Client()

futures = []
while True:
    futures += [
        c.submit(lambda: "x" * (50 * 2**20), pure=False)
        for _ in range(64)
    ]
    time.sleep(0.5)

Before #5543:

After both PRs:

[github-actions]

Unit Test Results

       18 files         18 suites   10h 15m 58s ⏱️
  2 603 tests   2 523 ✔️      78 💤 2 ❌
23 293 runs  21 880 ✔️ 1 411 💤 2 ❌

For more details on these failures, see this check.

Results for commit adf1b7c.

♻️ This comment has been updated with latest results.

[ncclementi]

@crusaderky it looks like we are having a related failure in a test_scheduler.py::test_memory see https://github.com/dask/distributed/runs/5186888061?check_suite_focus=true#step:12:1260

[ncclementi]

@crusaderky With the changes on this PR, Does this documentation need extra information?
https://github.com/dask/distributed/blob/main/docs/source/worker.rst#using-the-dashboard-to-monitor-memory-usage

[crusaderky]

@crusaderky With the changes on this PR, Does this documentation need extra information? https://github.com/dask/distributed/blob/main/docs/source/worker.rst#using-the-dashboard-to-monitor-memory-usage

updated

[crusaderky]

@crusaderky it looks like we are having a related failure in a test_scheduler.py::test_memory see https://github.com/dask/distributed/runs/5186888061?check_suite_focus=true#step:12:1260

This is... a major pain. The problem is that Python 3.7/3.8 CI installs neither lz4 nor snappy, so distributed.comm.compression falls back to no compression. However, I can't force it to zlib either (which is always available), because of a design flaw in distributed.protocol.compression. Which is not trivial to fix. I'll look into it more but this looks like yet another blocker issue.

Edit: worked around for now. Tracked at #5807

[crusaderky]

All green except issues already spotted in parent PR. Ready for review (but not merge!)

[ncclementi]

I think we are missing renaming some things in the dashboard to be consistent with the changes done in this PR, unless we think it is clear enough that "managed" is "managed_in_memory".

• The WorkerTable it uses the word "memory_managed" and there is an assignment of "memory_managed": "managed" if I'm not wrong this should be "memory_managed": "managed in memory",

• The ClusterMemory and WorkersMemory use also as name "managed",

[crusaderky]

• "memory_managed": "managed"

I think we are missing renaming some things in the dashboard to be consistent with the changes done in this PR, unless we think it is clear enough that "managed" is "managed_in_memory".

• The WorkerTable it uses the word "memory_managed" and there is an assignment of "memory_managed": "managed" if I'm not wrong this should be "memory_managed": "managed in memory",
• The ClusterMemory and WorkersMemory use also as name "managed",

I've renamed the variables for clarity. Note that the measure being shown was already managed_in_memory. The choice to just write "managed" in the UI is a conscious one to avoid confusing users.

[quasiben]

@shwina given you recent work with visualizing RMM data, this PR may be of interest as it illuminates issues with how spilled data is visualized

[ncclementi]

All the comments/requests were addressed in 905defa

[crusaderky]

IMPORTANT NOTE

🚨 🚨 🚨 🚨 🚨 🚨
All open PRs in dask/distributed will start failing on CI on Python 3.9 due to (deliberate) forward incompatibility with zict git tip. The solution is to merge from main.
🚨 🚨 🚨 🚨 🚨 🚨