feat(ext/canvas): enhance createImageBitmap specification compliance

[Hajime-san]

Feature

resolves #26032

basic support for decoding image format

• jpeg, png, bmp, ico

  • animated image
  • 16-bit image
  • Grayscale image

• webp, gif
  Temporarily, these formats are not supported due to increases of build size.

issues for supporting AVIF

The AVIF image format is supported by all browsers today.
However, the standardization of sniffing mime type seems to have hard going.

• Missing image codecs whatwg/mimesniff#143

support colorSpace for ImageData

There can convert color space from srgb to display-p3 with createImageBitmap.

This was a misunderstanding on my part, so I reverted it.
The conversion using the ImageData colorSpace option was probably the responsibility of CanvasRenderingContext2D.
I've left the implementation commented out as it may be useful as a reference for implementing CanvasRenderingContext2D or OffscreenCanvasRenderingContext2D.

support colorSpaceConversion option

The specifications are not particularly clear about the implementation details, and I had a hard time understanding its behavior, but I eventually do reverse engineering the results of wpt and implemented it to be consistent with those results.

support receives ImageBitmap

• createImageBitmap: Clarify treatment of colorSpaceConversion:"none" ImageBitmaps whatwg/html#10578

port to Rust

To simplify code, almost logic moved to Rust side.

added a simple architecture document for README

It is basically based on the image implementation, and I wrote roughly how you can handle data effectively.

performance impact

I was curious to see if the increased amount of Rust code, especially static dispatch, was affecting the bootstrap time of the runtime.
I'm not very confident about the suitability of this benchmark.

bootstrap

console.log('Hello, world!');

• this pr fb6b87d

% hyperfine --warmup 5 'target/release/deno run bootstrap.ts'
Benchmark 1: target/release/deno run bootstrap.ts
  Time (mean ± σ):      21.5 ms ±   3.6 ms    [User: 14.2 ms, System: 5.5 ms]
  Range (min … max):    18.7 ms …  54.5 ms    102 runs
 
  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

• main b1b72a8

% hyperfine --warmup 5 'target/release/deno run bootstrap.ts'
Benchmark 1: target/release/deno run bootstrap.ts
  Time (mean ± σ):      22.2 ms ±   4.7 ms    [User: 14.8 ms, System: 5.4 ms]
  Range (min … max):    18.8 ms …  66.8 ms    97 runs
 
  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

API benchmark

const imageData = new ImageData(new Uint8ClampedArray([255, 0, 0, 255]), 1, 1);

Deno.bench("createImageBitmap", async () => {
  await createImageBitmap(imageData);
})

• this pr acc4119

% target/release/deno bench bench.ts
    CPU | Apple M1
Runtime | Deno 2.0.0-rc.2 (aarch64-apple-darwin)

file:///path/to/deno/bench.ts

benchmark           time/iter (avg)        iter/s      (min … max)           p75      p99     p995
------------------- ----------------------------- --------------------- --------------------------
createImageBitmap            1.6 µs       615,000 (  1.6 µs …   1.7 µs)   1.6 µs   1.7 µs   1.7 µs

• main b1b72a8

% target/release/deno bench bench.ts
    CPU | Apple M1
Runtime | Deno 1.46.1 (aarch64-apple-darwin)

file:///path/to/deno/bench.ts

benchmark           time/iter (avg)        iter/s      (min … max)           p75      p99     p995
------------------- ----------------------------- --------------------- --------------------------
createImageBitmap            3.5 µs       287,200 (  3.3 µs …   3.6 µs)   3.5 µs   3.6 µs   3.6 µs

extra

The benchmarks below are for when the argument to op_create_image_bitmap is a struct except for the first argument of buf.
It seems that serializing the argument to op as a struct has a fairly large impact on performance.
This may apply to things other than op_create_image_bitmap as well.

% target/release/deno bench bench.ts
    CPU | Apple M1
Runtime | Deno 2.0.0-rc.2 (aarch64-apple-darwin)

file:///path/to/deno/bench.ts

benchmark           time/iter (avg)        iter/s      (min … max)           p75      p99     p995
------------------- ----------------------------- --------------------- --------------------------
createImageBitmap            4.2 µs       235,900 (  4.1 µs …   4.5 µs)   4.3 µs   4.5 µs   4.5 µs

plans for enhancement in another PRs

• improve performance
  There is known to that processing image is a cpu-bound task. So, op_create_image_bitmap consider to be data-parallelism with using rayon.

• align error message

  • Normalize error message grammar (TS/JS) #25269
  • refactor(ext): align error messages #25310

[crowlKats]

Overall looking very good, though one nitpick: in the newly added error messages, could you remove the trailing .? This is just a design decision that we have made to have a unified style for error messages

[petamoriken]

@Hajime-san Happy new year!

I updated copyright notice by #27509. Please merge main into this branch and update the same 🙏

[crowlKats]

@Hajime-san I know this took a while, but it had its reasons. The merge window for 2.2 is open now, so i want to get this landed this week; could you rebase it? and i will review it once more to be sure and then we can land it!

[crowlKats]

@Hajime-san so i just compiled the PR locally, and noticed that the binary increased 604kb; do you have any insight into the cause behind this? probably one of the dependencies being really big? regardless, it is a bit too much of an increase. that aside this PR looks good to land, so thats the only blocker.

[Hajime-san]

@crowlKats
I didn't care about the increased build size. Yes, it's huge.
I will take a look soon.

MEMO:

% uname -a
Darwin user.local 23.5.0 Darwin Kernel Version 23.5.0: Wed May  1 20:16:51 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T8103 arm64

main on d9db0b3

% cargo b -r --target-dir target/d9db0b35e && ls -l target/d9db0b35e/release/deno
-rwxr-xr-x  1 user  46682944  110354728 Jan 31 11:39 target/d9db0b35e/release/deno

this pr ca52fe8

% cargo b -r --target-dir target/ca52fe82f && ls -l target/ca52fe82f/release/deno         
-rwxr-xr-x  1 user  46682944  110646296 Jan 31 12:46 target/ca52fe82f/release/deno

approximately 291kb larger

[crowlKats]

if its one of the image formats that is causing massive increase, i think its fine to drop support for it and we can think about adding it in the future

[Hajime-san]

@crowlKats
Thanks!

From my research, of the newly added image formats, jpeg and webp are the largest, followed by gif, while bmp and ico do not seem to be affected as they have no external dependencies.
Also, although the color space conversion library lcms2 requires static linking on aarch64-unknown-linux-gnu, which may account as much as jpeg and webp.

In addition, I think that in terms of actual demand, png and jpeg(#26032) are the most popular, followed by webp and gif.

I can follow your instructions and make additional commits to not include certain image formats in this PR.
However, if the 2.2 release is imminent, feel free to make commits at your decision.

---

Here are simplified steps to reproduce for comparison of build sizes:

1. to remove some specific format from dependencies.image.features in ext/canvas/Cargo.toml
2. to comment out use image::codecs::foo::FooDecoder in ext/canvas/op_create_image_bitmap.rs that follows ext/canvas/Cargo.toml and insert unreachable!() macro to the MimeType enum match arm
3. cargo build --release

build result

my pc

% uname -a
Darwin user.local 23.5.0 Darwin Kernel Version 23.5.0: Wed May  1 20:16:51 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T8103 arm64

main on d9db0b3

% ls -l target/d9db0b35e/release/deno
-rwxr-xr-x  1 user  46682944  110354728 Jan 31 11:39 target/d9db0b35e/release/deno

based on this pr ca52fe8

original

% ls -l target/ca52fe82f/release/deno         
-rwxr-xr-x  1 user  46682944  110646296 Jan 31 12:46 target/ca52fe82f/release/deno

291kb larger

dynamic link lcms2

% ls -l target/ca52fe82f_lcms2_dynamic_link/release/deno
-rwxr-xr-x  1 user  46682944  110447720 Jan 31 22:35 target/ca52fe82f_lcms2_dynamic_link/release/deno

92kb larger

remove gif

% ls -l target/ca52fe82f_png_jpeg_webp_bmp_ico/release/deno      
-rwxr-xr-x  1 user  46682944  110629416 Jan 31 15:33 target/ca52fe82f_png_jpeg_webp_bmp_ico/release/deno

274kb larger

remove gif,webp

% ls -l target/ca52fe82f_png_jpeg_bmp_ico/release/deno
-rwxr-xr-x  1 user  46682944  110499304 Jan 31 15:59 target/ca52fe82f_png_jpeg_bmp_ico/release/deno

144kb larger

remove gif,webp,jpeg

% ls -l target/ca52fe82f_png_bmp_ico/release/deno     
-rwxr-xr-x  1 user  46682944  110408040 Jan 31 16:23 target/ca52fe82f_png_bmp_ico/release/deno

53kb larger

[crowlKats]

@Hajime-san lets disable gif and webp for now then

[Hajime-san]

I commented out the relevant lines with an explanation of the implementation, changed them to throw an error for each MIME type gif and webp, and modified the unit test to verify that an error is thrown.
Cloud you tell me if it is more appropriate to delete the source code itself rather than commenting it out?

[crowlKats]

commenting out sounds better so we can re-enable it when we want to

[crowlKats]

LGTM!

[crowlKats]

As a follow-up and to enable #23783 given the changes in #27665, it would be nice if the ImageBitmap class would be changed to be defined in rust instead of JS, and be returned by the op. not sure if this is something you'd like to do @Hajime-san

[Hajime-san]

@crowlKats @petamoriken
Thanks a lot!
I was thinking of adding avif support next, but since an increase in build size would be unavoidable,
so I will propose a way to reduce the size on the image crate side.
Indeed, since we ported a lot of the core logic to Rust with this PR, object wrap shouldn't be so hard.

BTW, it would be nice if you update the doc of the colorSpaceConversion option in MDN when you commit to the repo.
https://github.com/mdn/browser-compat-data/blob/c04c5b51405486a780d78fa1c43b473f5214c6da/api/_globals/createImageBitmap.json#L55-L57