[Java] Handle offset field from ArrowArray when BufferImportTypeVisitor imports offset buffer

[viirya]

Describe the bug, including details regarding any error messages, version, and platform.

This bug is found during debugging the issue apache/datafusion-comet#540.

We found that some string arrays' offsets are out of the range of their value buffer. I.e., a string array's value buffer is only 147456 bytes, but the offsets of the last string is (294894, 294912).

The string is output from DataFusion aggregation operator AggregateExec. When producing the output batch, the operator will possibly slice the output batch if it is larger than a configured size. The slice of a string array in arrow-rs, keeps original value buffer and moves the pointer of offset buffer so it is a zero-copy slice.

During importOffsets call in BufferImportTypeVisitor, the slice of offset can be imported correctly as it uses the moved pointer and calculates the offset buffer correctly.

But when it goes to import value buffer, it calculates the capacity of it by using the difference between imported last offset and first offset. Because the imported offsets are from the slice, the calculated capacity is only for the certain slice of the value buffer.

For example, the original string array's value buffer is 346536 bytes, last offset is 346536. We take a slice of 8192 strings from it. The slice array's last offset is 294912 but the value buffer is the same (346536 bytes).

When BufferImportTypeVisitor imports the slice, the imported offsets are [147456, ..., 294912]. It calculates the length of value buffer is 294912 - 147456 = 147456. But actually the length of value buffer is 346536.

Obviously the offsets are now out of range of the incorrect value buffer size 147456.

To be clear, the source of the issue comes from apache/arrow-rs#5896 where it exports moved pointer of offer buffer of a slice of string array. Instead, we should use offset field in ArrowArray for it. We are going to fix it in arrow-rs for that.

But actually seems either ArrayImporter or BufferImportTypeVisitor also doesn't consider offset in ArrowArray.

Component(s)

Java

[lidavidm]

Ah, thanks for the explanation.

In general Arrow Java does not support offset in the same way that C++ and Rust do (this is purely a fiction of their implementations and not part of IPC, though now C Data forces Java to deal with it). But we can correct the problem here at least

CC @vibhatha

[vibhatha]

I will take a look.

[viirya]

To be clear, the source of the issue comes from apache/arrow-rs#5896 where it exports moved pointer of offer buffer of a slice of string array. Instead, we should use offset field in ArrowArray for it. We are going to fix it in arrow-rs for that.

But actually seems either ArrayImporter or BufferImportTypeVisitor also doesn't consider offset in ArrowArray. So we still need to make BufferImportTypeVisitor handle offset correctly.

[lidavidm]

Ah, thanks. Yes, either way, we need to consider offset in Java and figure out how best to deal with it.

[viirya]

Ah, thanks. Yes, either way, we need to consider offset in Java and figure out how best to deal with it.

Yes. I'm going to fix arrow-rs to have correct offset field and we also need to handle offset in Java to make it work.

[vibhatha]

@lidavidm @bkietz I have been working on a solution for this issue in the Java side, but I later realized that we don't have any C data interface tests to evaluate something like slicing.

[bkietz]

One easy method to support importing sliced arrays in Java would be to import all arrays as if they had offset 0, then use splitAndTransfer() to effect the slicing after import is complete. IIUC, since importing is zero copy and splitAndTransfer() is too (when possible) this should be about as efficient as handling the slice offset directly

[joellubi]

If implementations can handle offset buffers that do not start at zero, then C data arrays with offsets should be possible to import following one of these two cases:

• For fixed-width types, slice the value buffer starting at offset*width bytes.
• For binary or list types, instead slice the offset buffer at offset bytes.

The latter means that offset buffers won't necessarily start at zero, which not all implementations support right now. This is something we fixed recently in Go, also related to datafusion's shared buffer usage: #41993.

If implementations can support this case, then implementing C data offsets should become simpler.