Consider adding offset in the Deno.write interface

[ThePrimeagen]

When writing a large file you have 2 options. 1 use writeAll, or two use write which will return bytesWritten. If bytesWritten does not equal the length, you need to continue to write to the file. The problem becomes there is no offset, therefore my code looked like the following.

let idx = 0;
do {
    // Not a fan of having to slice the uint8array over and over again
    const bytesWritten = await Deno.write(file.rid, data.slice(idx));
    idx += bytesWritten;

} while (idx < data.length);

Instead I would like to see the following

let idx = 0;
do {
    // I am happy
    idx += await Deno.write(file.rid, data, idx);
} while (idx < data.length);

This is only to avoid needless heavy copying when it comes to large Uint8Arrays. ;)

[vegerot]

I don't know if we should really take his suggestions into consideration. Look at this profile, he doesn't even use Rust

[ThePrimeagen]

Ok tough guy

[bartlomieju]

This is only to avoid needless heavy copying when it comes to large Uint8Arrays. ;)

@ThePrimeagen you can use data.subarray(idx) instead - it won't copy contents of Uint8Array but rather create a new view on that data using same buffer. That's what's used internally in writeAll():

export async function writeAll(w: Writer, arr: Uint8Array): Promise<void> {
  let nwritten = 0;
  while (nwritten < arr.length) {
    nwritten += await w.write(arr.subarray(nwritten));
  }
}

I don't know if we should really take his suggestions into consideration. Look at this profile, he doesn't even use Rust

@vegerot please refrain from such comments; everyone is welcome to open an issue and make suggestions.

[ThePrimeagen]

The only thing I don't like, even with a view adjustment, is I am still creating new objects to update a pointer. I know these are trivial objects but if I wanted to create a zero allocation scheme sending very large data payloads around would cause lots of views to be created. To send a MiB (I was seeing 2^14 bytes per call to write) would be 64 calls to do that, which means I had to create 64 views of the data. This will add up over time especially if there is a similar thing going on with http/2/ws.

On a side note, I am writing a WS/H2/H3 client in typescript for netflix, perhaps at somepoint i'll see the performance of it in deno.

Anywho, just a thought, thanks for Deno, its been fun dev'ing on stream on twitch. @vegerot was just trolling me (he was watching on twitch).

[bartlomieju]

The only thing I don't like, even with a view adjustment, is I am still creating new objects to update a pointer. I know these are trivial objects but if I wanted to create a zero allocation scheme sending very large data payloads around would cause lots of views to be created. To send a MiB (I was seeing 2^14 bytes per call to write) would be 64 calls to do that, which means I had to create 64 views of the data. This will add up over time especially if there is a similar thing going on with http/2/ws.

On a side note, I am writing a WS/H2/H3 client in typescript for netflix, perhaps at somepoint i'll see the performance of it in deno.

@ThePrimeagen this is the most fundamental way to perform IO in Deno and is still open to optimizations. We keep track of performance in the benchmarks: https://deno.land/benchmarks.html

Anywho, just a thought, thanks for Deno, its been fun dev'ing on stream on twitch. @vegerot was just trolling me (he was watching on twitch).

I see, just try to be professional :)

[vegerot]

for
context

[ry]

We would only add this complication if there was some measurable performance win. And even then it would not be done in v1.x. It would have to wait for v2.

I think we've seen that subarray can be a bottleneck, but I'm not sure about slice.

Until we have a clear demonstration of the performance win, I would say this change is off the table.

[ThePrimeagen]

slice would definitely be a huge perf loss as it creates a deep copy (doesn't adjust byteOffset / length).

Thats fine. It feels more like other c libs where offset into a void* is provide, which can be nice. Either way, just a suggestion, thanks for the quick replies.

One last thing. As far as complication, it does feel as the same level of complication. Both times I am using a loop with an await. Both times I am keeping track on an index. With the Writer this could be done under the hood and writeAll could avoid unnecessary object creation. As an embedded js dev I have found that most javascript problems are not singular but the sum of the whole. So avoiding these small allocations can help the slow death that is JS :) (ps I love JS).

[crowlKats]

Since this issue was opened, the op layer has undergone drastic performance improvements, so there shouldnt as much as a concern as back then regarding performance.

[vegerot]

Since this issue was opened, the op layer has undergone drastic performance improvements, so there shouldnt as much as a concern as back then regarding performance.

shouldn't be as much of a concern

Agreed. But still, having an offset is pretty standard for these kinds of interfaces. It'll always be faster to manually specify where you want to start writing than having to write the whole thing

[crowlKats]

in regards to

manually specify where you want to start writing

That is what we have the Seeker interface for.

[crowlKats]

It seems to me that this would be a duplicate of #3508

[mmastrac]

We've gone through a number of performance improvements in the op layer, which means that subarray should be extremely cheap at this point as we get the fastcall version of the underlying data which is basically the pointer to the bytes + whatever view offset you've added.

[kt3k]

I think this issue is no longer relevant as we removed Deno.write at Deno 2 #22079

Closing