Add HTML reprs for Client.who_has and Client.has_what

[jacobtomlinson]

@quasiben mentioned folks are regularly asking about how to better understand what workers have what keys.

Threw together some quick HTML reprs for Client.who_has and Client.has_what to try and make things more understandable.

I followed the same approach as distributed.utils.Logs by subclassing a dict and adding an HTML repr to it. I wasn't sure if distributed.utils was the right place for this though, so I started distributed.objects for a place to put custom objects with fancy reprs. I expect this to grow and contain more variations on dict, str, etc with fancy reprs.

Happy to move Log and Logs there if that makes sense.

has_what

who_has

[jakirkham]

That's really neat Jacob! Thanks for putting this together 😄

[mrocklin]

If you wanted to go big here we could also improve the HTML reprs for WorkerState and TaskState, and then start sending/using those objects on the client side. This would be an API change though, and so possibly not worth it for this method.

…

On Wed, May 26, 2021 at 9:41 AM Jacob Tomlinson ***@***.***> wrote: @quasiben <https://github.com/quasiben> mentioned folks are regularly asking about how to better understand what workers have what keys. Threw together some quick HTML reprs for Client.who_has and Client.has_what to try and make things more understandable. I followed the same approach as distributed.utils.Logs by subclassing a dict and adding an HTML repr to it. I wasn't sure if distributed.utils was the right place for this though, so I started distributed.objects for a place to put custom objects with fancy reprs. I expect this to grow and container more variations on dict, str, etc with fancy reprs. Happy to move Log and Logs there if that makes sense. *has_what* [image: image] <https://user-images.githubusercontent.com/1610850/119679859-9c9e5400-be38-11eb-8e23-6186469672c2.png> *who_has* [image: image] <https://user-images.githubusercontent.com/1610850/119679971-b475d800-be38-11eb-805a-7137fdcfce34.png> ------------------------------ You can view, comment on, or merge this pull request online at: #4853 Commit Summary - Add HTML reprs for Client.who_has and Client.has_what File Changes - *M* distributed/client.py <https://github.com/dask/distributed/pull/4853/files#diff-fa07d391e7f82923d61dc0643961fdebcb9c4582e8f826bbc33710c5ae45a380> (5) - *A* distributed/objects.py <https://github.com/dask/distributed/pull/4853/files#diff-b89802a660a3d13b2880547477b2305969318e7bc30a5deb3caf5e7a38ff37b4> (67) Patch Links: - https://github.com/dask/distributed/pull/4853.patch - https://github.com/dask/distributed/pull/4853.diff — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#4853>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTDOVKHSYYWCP2MVDBTTPUCA7ANCNFSM45SGPUCA> .

[jacobtomlinson]

I'm keen to improve HTML reprs on ALL THE THINGS!

This seemed like low hanging fruit.

[jacobtomlinson]

Test failure seems flaky and unrelated.

Given this is a small change to reprs only I'm going to hit merge as I want to build upon it. We can iterate on this later.

[crusaderky]

This breaks has_what and who_has for asynchronous clients.
CC @jrbourbeau I understand there should be a release this friday? This looks like a showstopper.

[jrbourbeau]

Thanks for flagging @crusaderky! Do you have a link to a failing CI build you can point other to?

[crusaderky]

https://github.com/dask/distributed/runs/2684366276

>       return HasWhat(self.sync(self.scheduler.has_what, workers=workers, **kwargs))
E       TypeError: 'coroutine' object is not iterable

The issue is that there are no tests in master that run who_has or has_what with an asynchronous Client. I just happened to add two in my PR.

[jrbourbeau]

Thanks @crusaderky -- linking to the referenced PR for others #4774

[quasiben]

We are poking at this now. Thanks @crusaderky for flagging

[jacobtomlinson]

Thanks for flagging this folks.

I expect this is because self.sync returns an asyncio future when in async mode. Therefore the type conversion fails.

The fix will likely be to move the call further down into the scheduler code, as long as it plays nicely with the comm.

I'm dealing with a family emergency this afternoon, so if you want to revert this in the meantime I'm happy with that.

[quasiben]

Sorry you are having troubles. We'll take care of it.

[jrbourbeau]

Just cross-referencing #4860 which fixes the TypeError: 'coroutine' object is not iterable we were running into