Skip to content

Commit

Permalink
apacheGH-40153: [Python] Make Tensor.__getbuffer__ work on 32-bit p…
Browse files Browse the repository at this point in the history
…latforms (apache#40294)

### Rationale for this change

`Tensor.__getbuffer__` would silently assume that `Py_ssize_t` is the same width as `int64_t`, which is true only on 64-bit platforms.

### What changes are included in this PR?

Create an internal buffer of `Py_ssize_t` values mirroring a Tensor's shape and strides, to avoid relying on the aforementioned assumption.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

No.

* GitHub Issue: apache#40153

Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
  • Loading branch information
pitrou authored and mapleFU committed Mar 7, 2024
1 parent 4ae114d commit 9fd85eb
Show file tree
Hide file tree
Showing 2 changed files with 15 additions and 4 deletions.
2 changes: 2 additions & 0 deletions python/pyarrow/lib.pxd
Original file line number Diff line number Diff line change
Expand Up @@ -295,6 +295,8 @@ cdef class Tensor(_Weakrefable):

cdef readonly:
DataType type
bytes _ssize_t_shape
bytes _ssize_t_strides

cdef void init(self, const shared_ptr[CTensor]& sp_tensor)

Expand Down
17 changes: 13 additions & 4 deletions python/pyarrow/tensor.pxi
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,9 @@
# specific language governing permissions and limitations
# under the License.

# Avoid name clash with `pa.struct` function
import struct as _struct


cdef class Tensor(_Weakrefable):
"""
Expand All @@ -40,6 +43,14 @@ cdef class Tensor(_Weakrefable):
self.sp_tensor = sp_tensor
self.tp = sp_tensor.get()
self.type = pyarrow_wrap_data_type(self.tp.type())
self._ssize_t_shape = self._make_shape_or_strides_buffer(self.shape)
self._ssize_t_strides = self._make_shape_or_strides_buffer(self.strides)

def _make_shape_or_strides_buffer(self, values):
"""
Make a bytes object holding an array of `values` cast to `Py_ssize_t`.
"""
return _struct.pack(f"{len(values)}n", *values)

def __repr__(self):
return """<pyarrow.Tensor>
Expand Down Expand Up @@ -282,10 +293,8 @@ strides: {0.strides}""".format(self)
buffer.readonly = 0
else:
buffer.readonly = 1
# NOTE: This assumes Py_ssize_t == int64_t, and that the shape
# and strides arrays lifetime is tied to the tensor's
buffer.shape = <Py_ssize_t *> &self.tp.shape()[0]
buffer.strides = <Py_ssize_t *> &self.tp.strides()[0]
buffer.shape = <Py_ssize_t *> cp.PyBytes_AsString(self._ssize_t_shape)
buffer.strides = <Py_ssize_t *> cp.PyBytes_AsString(self._ssize_t_strides)
buffer.suboffsets = NULL


Expand Down

0 comments on commit 9fd85eb

Please sign in to comment.