Skip to content

Commit

Permalink
Use memoryview in unpack_frames (#3980)
Browse files Browse the repository at this point in the history
* Minor whitespace adjustment

* Coerce input to `memoryview` in `unpack_frames`

Selecting out each frame from the input causes a copy, which increases
memory usage and slows down `unpack_frames`. To fix this, coerce the
input to a `memoryview`. This way slices into the `memoryview` only take
a view onto the underlying data, which is quite fast and doesn't result
in additional memory usage.
  • Loading branch information
jakirkham authored Jul 22, 2020
1 parent 6eab3ce commit eb10a53
Showing 1 changed file with 3 additions and 0 deletions.
3 changes: 3 additions & 0 deletions distributed/protocol/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -120,8 +120,11 @@ def unpack_frames(b):
--------
pack_frames
"""
b = memoryview(b)

fmt = "Q"
fmt_size = struct.calcsize(fmt)

(n_frames,) = struct.unpack_from(fmt, b)
lengths = struct.unpack_from(f"{n_frames}{fmt}", b, fmt_size)

Expand Down

0 comments on commit eb10a53

Please sign in to comment.