Skip to content

Commit

Permalink
Streamline pack_frames/unpack_frames frames (#3973)
Browse files Browse the repository at this point in the history
* Use `.extend` to collect and append `frames`

Instead of copying `frames` into a new `list`, just use `.extend` to
add them into the `list` already containing the prelude. This is
flexible and handles other non-`list` objects just as well.

* Assign format to a variable for easier reference

* Refactor itemsize from number of elements read

* Compute the size of `"Q"` and use throughout

* Use `struct.unpack_from` to ignore extra bytes

* Unpack frame lengths all at once

* Compute frame `end` once

* Run `black`
  • Loading branch information
jakirkham authored Jul 21, 2020
1 parent 8ebd396 commit 1dba162
Showing 1 changed file with 12 additions and 12 deletions.
24 changes: 12 additions & 12 deletions distributed/protocol/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -105,12 +105,9 @@ def pack_frames(frames):
--------
unpack_frames
"""
prelude = [pack_frames_prelude(frames)]

if not isinstance(frames, list):
frames = list(frames)

return b"".join(prelude + frames)
data = [pack_frames_prelude(frames)]
data.extend(frames)
return b"".join(data)


def unpack_frames(b):
Expand All @@ -123,14 +120,17 @@ def unpack_frames(b):
--------
pack_frames
"""
(n_frames,) = struct.unpack("Q", b[:8])
fmt = "Q"
fmt_size = struct.calcsize(fmt)
(n_frames,) = struct.unpack_from(fmt, b)
lengths = struct.unpack_from(f"{n_frames}{fmt}", b, fmt_size)

frames = []
start = 8 + n_frames * 8
for i in range(n_frames):
(length,) = struct.unpack("Q", b[(i + 1) * 8 : (i + 2) * 8])
frame = b[start : start + length]
start = fmt_size * (1 + n_frames)
for length in lengths:
end = start + length
frame = b[start:end]
frames.append(frame)
start += length
start = end

return frames

0 comments on commit 1dba162

Please sign in to comment.