Skip to content

Commit

Permalink
Skip coercing to bytes in merge_frames (#3960)
Browse files Browse the repository at this point in the history
As the frames we receive are typically mutable, non-`bytes` objects like
`bytearray`s or NumPy `ndarray`s, coercing to `bytes` at this stage
triggers a copy of all frames. As we are going to toss those copied
versions anyways when joining them into a larger `bytes` object, this
ends up being wasteful with memory. Fortunately `bytes.join(...)`
accepts any and all `bytes`-like objects. So instead just pass them all
through as-is to `bytes.join(...)`, which is free and doesn't require a
copy.  Should cutdown on the memory usage in this part of the code.
  • Loading branch information
jakirkham authored Jul 14, 2020
1 parent d9da4d7 commit 1d92125
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions distributed/protocol/utils.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import struct
import msgpack

from ..utils import ensure_bytes, nbytes
from ..utils import nbytes

BIG_BYTES_SHARD_SIZE = 2 ** 26

Expand Down Expand Up @@ -84,7 +84,7 @@ def merge_frames(header, frames):
if len(L) == 1: # no work necessary
out.extend(L)
else:
out.append(b"".join(map(ensure_bytes, L)))
out.append(b"".join(L))
return out


Expand Down

0 comments on commit 1d92125

Please sign in to comment.