Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for writing other primitive arrays to Buffers #357

Open
DRSchlaubi opened this issue Jul 18, 2024 · 2 comments
Open

Add support for writing other primitive arrays to Buffers #357

DRSchlaubi opened this issue Jul 18, 2024 · 2 comments
Labels

Comments

@DRSchlaubi
Copy link

DRSchlaubi commented Jul 18, 2024

Due to the deprecation of the Ktor I/O APIs we need to migrate this code, however with kotlinx-io there is currently no easy way to replace this

internal actual fun formatIntegerFromLittleEndianLongArray(data: LongArray) =
    withBuffer(data.size * Long.SIZE_BYTES) {
        // need to convert from little-endian data to big-endian expected by BigInteger
        writeFully(data.reversedArray())
        BigInteger.fromByteArray(readBytes(), Sign.POSITIVE).toString()
    }
@JakeWharton
Copy link
Contributor

If performance is a concern here, I would recommend that you create the byte array directly and then write the longs by iterating in reverse and writing their bytes directly with shifts. This will produce zero extra allocations or copies. The current code has a lot of wasted allocations and copies.

If you just want to do this with kotlinx-io you can do something like

val buffer = Buffer()
data.reversedArray().forEach(buffer::writeLong)
BigInteger.fromByteArray(data.readByteArray(), Sign.POSITIVE).toString()

This wastes approximately the same amount of allocations as the original code.

A very easy improvement would be to avoid the LongArray copy

val buffer = Buffer()
data.indices.reversed().forEach { buffer.writeLong(data[it]) }
BigInteger.fromByteArray(buffer.readByteArray(), Sign.POSITIVE).toString()

I'll also note that you don't appear to actually be doing an endian-ness switch. The longs are written in reverse, but their byte order is maintained. If you need an actual endian switch you can use writeLongLe for the function reference.

@fzhinkin fzhinkin changed the title Add support for writing other primitives to Buffers Add support for writing other primitive arrays to Buffers Aug 12, 2024
@fzhinkin
Copy link
Collaborator

One of the concerns regarding API to read/write arbitrary primitive array types is the API surface bloating.
By following what we have for primitive types and for bytes array, following could be added:

  • functions to write an array for every primitive type (Short, Int, Long, Float, Double);
  • functions to write an array of unsigned integer types (UShort, UInt, ULong);
  • functions that do the same as two previous categories, but using different endianness;
  • for all mentioned above, we need read-functions (and for ByteArray we have two: one filling existing array and one allocating a new one).

Something around 42 functions, that's a lot! Especially, given that most of them may remain unused.

However, we can provide optimized versions of these operations (using the Unsafe API and, for example, var handles or byte buffer views on JVM).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants