Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf(): optimize perPixelTargetFind #8770

Merged
merged 34 commits into from
Mar 19, 2023
Merged

perf(): optimize perPixelTargetFind #8770

merged 34 commits into from
Mar 19, 2023

Conversation

ShaMan123
Copy link
Contributor

@ShaMan123 ShaMan123 commented Mar 9, 2023

Motivation

Making perPixelTargetFind faster

Description

limited the canvas size
view the diff w/o ws

Changes

renamed cacheCanvasEl => pixelFindCanvasEl because it is no longer a generic cache canvas

Gist

I don't really know how to benchmark this.

Before

image

After

image

In Action

@ShaMan123 ShaMan123 marked this pull request as ready for review March 9, 2023 12:38
@github-actions
Copy link
Contributor

github-actions bot commented Mar 9, 2023

Build Stats

file / KB (diff) bundled minified
fabric 895.394 (-0.275) 304.351 (+0.027)

@github-actions
Copy link
Contributor

github-actions bot commented Mar 9, 2023

Coverage after merging perf-pixel-detection into master will be

83.71%

Coverage Report
FileStmtsBranchesFuncsLinesUncovered Lines
index.node.ts7.69%100%0%14.29%17, 20, 23, 29, 32, 35
src
   ClassRegistry.ts100%100%100%100%
   Collection.ts94.71%94.64%86.67%97.09%100, 103, 206–207, 232–233
   CommonMethods.ts96.55%87.50%100%100%10
   Intersection.ts100%100%100%100%
   Observable.ts87.23%85.29%84.62%89.36%144–145, 170–171, 39–40, 48, 57, 91, 99
   Pattern.ts92.21%91.89%90%93.33%114, 138, 149, 158, 51
   Point.ts100%100%100%100%
   Shadow.ts98.51%96%100%100%168
   cache.ts97.06%90%100%100%56
   config.ts75%66.67%66.67%82.76%130, 138, 138, 138, 138, 138–140, 151–153
   constants.ts100%100%100%100%
   typedefs.ts100%100%100%100%
src/brushes
   BaseBrush.ts100%100%100%100%
   CircleBrush.ts0%0%0%0%100, 102–104, 113, 113, 113, 115, 117, 119–121, 123–126, 133–134, 141, 143, 28–29, 37–41, 45–49, 56–59, 67–71, 73, 81, 81, 81, 81, 81–82, 84, 84, 84–87, 89, 97–98
   PatternBrush.ts97.06%87.50%100%100%21
   PencilBrush.ts91.01%82.35%100%93.75%122–123, 152, 152–154, 176, 176, 276, 280, 285–286, 68–69, 84–85
   SprayBrush.ts0%0%0%0%100–101, 103–104, 112, 112, 112, 112, 112–113, 115–116, 123–124, 126, 128–132, 141, 145–146, 146, 154, 154, 154–157, 159–162, 166–167, 169, 171–174, 177, 184–185, 187, 189–190, 192, 199–200, 202–203, 206, 206, 213, 213, 217, 22–23, 25–27, 27, 27–29, 33, 42, 49, 56, 63, 70, 89–91, 99
src/canvas
   Canvas.ts78.98%77.54%81.67%79.60%1000, 1000, 1000–1002, 1004–1005, 1005, 1005, 1007, 1015, 1015, 1015–1017, 1017, 1017, 1023–1024, 1032–1033, 1033, 1033–1034, 1039, 1041, 1072–1074, 1077–1078, 1082–1083, 1197–1199, 1202–1203, 1276, 1396, 1519, 1589, 161, 186, 295–296, 299–303, 308, 331–332, 337–342, 36, 362, 362, 362–363, 363, 363–364, 372, 377–378, 378, 378–379, 381, 390, 396–397, 397, 397, 40, 440, 448, 452, 452, 452–453, 455, 537–538, 538, 538–539, 545, 545, 545–547, 567, 569, 569, 569–570, 570, 570, 573, 573, 573–574, 577, 586–587, 589–590, 592, 592–593, 595–596, 608–609, 609, 609–610, 612–616, 622, 629, 669, 669, 669, 671, 673–677, 683, 689, 689, 689–690, 692, 695, 700, 713, 740, 740, 798–799, 799, 799–800, 802, 805–806, 806, 806–807, 809–810, 813, 813–815, 818–819, 889, 901, 908, 929, 961, 982–983, 999
   SelectableCanvas.ts94.39%91.16%94.55%96.63%1127, 1127–1128, 1131, 1151, 1151, 1209, 1262–1263, 1284, 1292, 1417, 1419, 1421–1422, 526, 706–707, 709–710, 710, 710, 759–760, 821–822, 875–877, 909, 914–915, 942–943
   StaticCanvas.ts96.01%91.48%100%97.93%1112–1113, 1113, 1113–1114, 1234, 1244, 1298–1299, 1302, 1337–1338, 1415, 1424, 1424, 1428, 1428, 1475–1476, 1690–1691, 311–312, 329, 409–410, 412–413, 774, 786–787, 872
   TextEditingManager.ts100%100%100%100%
src/color
   Color.ts92.16%86.49%100%94.29%330–331, 335–336, 339–340, 58, 88–89, 89, 91, 91, 91–92, 94–95
   color_map.ts100%100%100%100%
   constants.ts100%100%100%100%
   util.ts100%100%100%100%
src/controls
   Control.ts93.90%88.89%90.91%97.73%235, 319, 319, 354
   changeWidth.ts100%100%100%100%
   commonControls.ts100%100%100%100%
   controlRendering.ts81.63%78%100%84.78%106, 111, 121, 121, 45, 50, 61, 61, 65–72, 81–82
   drag.ts100%100%100%100%
   polyControl.ts5.97%0%0%11.11%100, 105, 119, 119, 119, 119, 119, 121–124, 124, 127, 134, 17, 25–29, 29, 29, 29, 29, 29, 29, 29, 50–56, 56, 56, 56, 56, 58, 63–64, 66, 76, 82–83, 83, 83–84, 88–90, 90, 90, 90, 90, 92
   rotate.ts19.57%12.50%50%21.43%41, 45, 51, 51, 51–52, 55–57, 59, 59, 59, 59, 59–61, 61, 61–63, 65, 65, 65–67, 67, 67–68, 73, 73, 73–74, 76, 78, 80–81
   scale.ts93.57%92.94%100%93.67%129–130, 132–134, 148–149, 181–183, 42
   scaleSkew.ts78.79%64.29%100%85.71%27, 29, 29, 29, 31, 33, 35
   skew.ts91.03%79.31%100%97.67%130–131, 162–163, 170, 176, 178
   util.ts100%100%100%100%
   wrapWithFireEvent.ts100%100%100%100%
   wrapWithFixedAnchor.ts100%100%100%100%
src/env
   browser.ts84.21%77.78%50%100%14, 17
   index.ts91.67%50%100%100%20
   node.ts76.92%50%66.67%88.24%27, 31–32, 32, 32
src/filters
   BaseFilter.ts21.93%22.81%32%19.05%106–109, 109, 109–110, 116, 116, 116–119, 137, 155, 169–174, 178–179, 179, 179–182,

@ShaMan123 ShaMan123 changed the title perf(): perPixelTargetFind perf(): optimize perPixelTargetFind Mar 9, 2023
Copy link
Contributor Author

@ShaMan123 ShaMan123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems ready
Could you help me benchmark the change?

@ShaMan123 ShaMan123 requested a review from asturur March 9, 2023 21:22
@jiayihu
Copy link
Contributor

jiayihu commented Mar 10, 2023

@ShaMan123 I'll gladly have a look personally in the week-end. Do you have any documentation about how to setup the fabric repo locally? What are the steps to open the project that you have used in your screenshots in localhost?

Regarding how to analyse the performance, you typically need to record actions that trigger the method a lot isTransparent in this case, e.g. several transparent objects all above each other and then move the mouse above them.
Then you need to look for the method in the flame graph and see how it behaves. The most basic evaluation is just observing the time and compare it to a new recording with the latest PR changes.

Once you select a method in the flame graph, you can also click on the bottom to jump to its file and see the timings of each line. There you can also have a quick evaluation of where most of the time is spent and see if there are significant improvements. This is an example of what I did last week, where I replaced the dynamically created styles declaration with a static one which is significantly more performant for V8 because it has a static shape (we can talk about stuff like this that I noticed in fabric in another time)

Before

Screenshot 2023-03-06 at 12 06 07

After

Screenshot 2023-03-06 at 12 34 00

@ShaMan123
Copy link
Contributor Author

ShaMan123 commented Mar 10, 2023

I use the next app from #8135

  • clone .codesandbox/templates/next
  • install deps
  • link fabric to your local repo or install this branch

OR you can squash #8315 into this locally and run npm run sandbox start next

@danielcrisp
Copy link

Interesting! Will take a look later. Thanks @ShaMan123

// in case the target is the activeObject, we cannot execute this optimization
// because we need to draw controls too.
if (isFabricObjectCached(target) && target !== this._activeObject) {
// optimizatio: we can reuse the cache
if (
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it happened that i thought a couple of times of this:

// in case the target is the activeObject, we cannot execute this optimization
    // because we need to draw controls too.

And then i asked myself, why do i have to draw the controls?
The controls can be found with the coordinates.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

absolutely
I had no idea what was that.
And I think active selection never caches, or at least it didn't in v5. Now I don't remember.

@@ -667,10 +667,23 @@ export class SelectableCanvas<
* @return {Boolean}
*/
isTargetTransparent(target: FabricObject, x: number, y: number): boolean {
const ctx = this.contextCache;
const retina = this.getRetinaScaling();
const size = Math.max(2 * this.targetFindTolerance * retina, 10);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why would you max it out?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i would rather have a method setTargetFindTolerance that creates this mini canvas, i m not even sure retina scaling is a requirement since the mouse pixels ( the so called css pixels ) are always without retina.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i would rather have a method setTargetFindTolerance that creates this mini canvas,

will do

I wasn't sure about retina. What will happen in case of ay scaled and blurry image? Will the hit test return a false positive?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought I removed the max value of 10

@ShaMan123
Copy link
Contributor Author

ShaMan123 commented Mar 11, 2023

Truth be told IMO this method doesn't belong to canvas. It is a standalone util. The canvas element can be a const and we could even make it offscreen I believe.

target.render(ctx);
target.selectionBackgroundColor = selectionBgc;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to be honest i think selectionBackgroundColor is just a terrible feature and that it could be deleted, but we shouldn't do it here.
Is a property that pertain the control bounding box and was added to the objects because rush/inexperience

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I met it redoing controls. Managed to get it working there but I agree it is weird because it seems not part of object rendering scope, same as controls, but because it needs to be behind and object stacking I understand why you did it.

Comment on lines 18 to 29
tolerance = Math.ceil(Math.max(tolerance, 0));
const point = new Point(x, y);
const start = point.scalarSubtract(tolerance).floor();
const end = point
.scalarAdd(Math.max(tolerance, 1))
.ceil()
.min(new Point(ctx.canvas.width - 1, ctx.canvas.height - 1));
const boundStart = start.max(new Point());
const size = end.subtract(boundStart);
if (size.x <= 0 || size.y <= 0) {
// out of bounds
return true;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i notice now this change, i don't think we should do it. the old loop is so much clear and smaller.
There is something weird in the old code and is that if we set x or y to 0 we do not adjust the tolerance to avoid considering extra pixels, but this change here is definitely extra complication.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will try to make simple but seems to me there are edge cases in the logic.
I can't understand the logic

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please try to explain it

under group will need more handling
ad is such a headache and probably not giving anything because `render` is optimized to use the cache so no reason for that
@asturur
Copy link
Member

asturur commented Mar 12, 2023

@asturur you were right about the tolerance, it is in canvas element space, so not affected by transformations. IMO we should drop the so called optimization using the cache because render is optimized to use the cache and it is a head to make correct and extremely hard

I think the cache canvas optimization ( reusing the cache ) is still ok and usable for top level obects at least.
render will reuse the cache correct, so is probalby doing the exact same thing?
Right now my code has drifet apart i have a fork ( it drifted becuase i was trying to recover this KB added ) but let's focus on the fact that this PR is introducing the optiization of the small canvas, if we change too much it means we got confused along the road. It shouldn't be hard to just make the operational canvas small.

@asturur
Copy link
Member

asturur commented Mar 12, 2023

It would be also ok to remove the cacheCanvas optimization if we can see that for already cached object the change in performance is minimal ( maybe is 0 ) since with interactive groups lot of things could be different.
But there are also other considerations to do here and there. I have to stop now, but i was enjoying this exploration so i ll do some more later.

@asturur
Copy link
Member

asturur commented Mar 12, 2023

my code is here fork-pixel-detection and is probably broken right now because i left in half

@asturur
Copy link
Member

asturur commented Mar 16, 2023

Let's try tomorrow to finish this.
I ll show what i have and i ll see if i can pull a benchmark.
The only reason why i want to pull a benchmark out is because i m curious about the effects of willReadFrequently when we want to paint over the willReadFrequently canvas some canvases that are in the GPU.

@asturur
Copy link
Member

asturur commented Mar 18, 2023

Collecting some data from the fork at: fork-pixel-detection

cached object
image

normal active selection uncached
image

image

Now i ll compare to master asap just for curiosity and compare the data extraction logic

Comment on lines 687 to 689
this.targetFindTolerance,
this.targetFindTolerance,
this.targetFindTolerance
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looking at this again shouldn't it be multiplied by retina?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in my branch i completely removed retina from the logic of target transparent.
I don't think it should be involved at all.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought it is needed for precision in case of blur

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

considering retina makes the cached/noncached case very similar and cut differences there.
But force us to continuosly round numbers

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That isn't too bad, is it?
I am thinking that this feature is used with text and text is heavily impacted by retina

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes i think it will win the smaller code at the end.
traget transparency isn't meant to be a pixel sniper effect. It has just to guess if is completely transparent or not.

@asturur
Copy link
Member

asturur commented Mar 19, 2023

Is not very easy to benchmark this, but i see like improvents here.
Sometimes my laptop goes in slow mode and the results all over the place.

image

image

Master is so higher up right now, sometimes it get the full 16ms to get back a result, and i also notice we run findTarget twice when not needed ( i ll open a PR for that )

image

Comment on lines 652 to 661
setTargetFindTolerance(value: number) {
this.targetFindTolerance = value;
const size = Math.ceil(
2 * Math.max(this.targetFindTolerance, 1) * this.getRetinaScaling()
);
if (this.pixelFindCanvasEl.width < size) {
this.pixelFindCanvasEl.width = this.pixelFindCanvasEl.height = size;
const retina = this.getRetinaScaling();
this.pixelFindContext.scale(retina, retina);
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

after playing around with this branch a bit and the feature, i think this should be slightly different:

Suggested change
setTargetFindTolerance(value: number) {
this.targetFindTolerance = value;
const size = Math.ceil(
2 * Math.max(this.targetFindTolerance, 1) * this.getRetinaScaling()
);
if (this.pixelFindCanvasEl.width < size) {
this.pixelFindCanvasEl.width = this.pixelFindCanvasEl.height = size;
const retina = this.getRetinaScaling();
this.pixelFindContext.scale(retina, retina);
}
setTargetFindTolerance(value: number) {
value = Math.floor(value); // floats aren't supported. but just in case get rid of them
this.targetFindTolerance = value;
const size = 2 * value + 1;
this.pixelFindCanvasEl.width = this.pixelFindCanvasEl.height = size;
}

notice that size is 2 * tolerance + 1. 1 is the pixel that we are hoverin and tolerance 3 for example, would mean we always take 3 pixels each side in addition to the original pixel.
So the pixelTargetFind canvas is always an odd number of pixels ( 1,3,5,7... )

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sounds correct

Comment on lines 32 to 37
const { data } = ctx.getImageData(
x,
y,
tolerance * 2 || 1,
tolerance * 2 || 1
);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
const { data } = ctx.getImageData(
x,
y,
tolerance * 2 || 1,
tolerance * 2 || 1
);
const { data } = ctx.getImageData(
x - tolerance,
y - tolerance,
tolerance * 2 + 1,
tolerance * 2 + 1,
);

If we write this in this way we can get rid of the if condition above.
Reading getImageData out of bounds seems in spec

The getImageData(sx, sy, sw, sh, settings) method steps are:

If either the sw or sh arguments are zero, then throw an ["IndexSizeError"](https://webidl.spec.whatwg.org/#indexsizeerror) [DOMException](https://webidl.spec.whatwg.org/#dfn-DOMException).

If the [CanvasRenderingContext2D](https://html.spec.whatwg.org/multipage/canvas.html#canvasrenderingcontext2d)'s [origin-clean](https://html.spec.whatwg.org/multipage/canvas.html#concept-canvas-origin-clean) flag is set to false, then throw a ["SecurityError"](https://webidl.spec.whatwg.org/#securityerror) [DOMException](https://webidl.spec.whatwg.org/#dfn-DOMException).

Let imageData be a [new](https://webidl.spec.whatwg.org/#new) [ImageData](https://html.spec.whatwg.org/multipage/canvas.html#imagedata) object.

[Initialize](https://html.spec.whatwg.org/multipage/canvas.html#initialize-an-imagedata-object) imageData given sw, sh, [settings](https://html.spec.whatwg.org/multipage/canvas.html#initialize-imagedata-settings) set to settings, and [defaultColorSpace](https://html.spec.whatwg.org/multipage/canvas.html#initialize-imagedata-defaultcolorspace) set to [this](https://webidl.spec.whatwg.org/#this)'s [color space](https://html.spec.whatwg.org/multipage/canvas.html#concept-canvas-color-space).

Let the source rectangle be the rectangle whose corners are the four points (sx, sy), (sx+sw, sy), (sx+sw, sy+sh), (sx, sy+sh).

Set the pixel values of imageData to be the pixels of [this](https://webidl.spec.whatwg.org/#this)'s [output bitmap](https://html.spec.whatwg.org/multipage/canvas.html#output-bitmap) in the area specified by the source rectangle in the bitmap's coordinate space units, converted from [this](https://webidl.spec.whatwg.org/#this)'s [color space](https://html.spec.whatwg.org/multipage/canvas.html#concept-canvas-color-space) to imageData's [colorSpace](https://html.spec.whatwg.org/multipage/canvas.html#dom-imagedata-colorspace) using ['relative-colorimetric'](https://drafts.csswg.org/css-color/#valdef-color-profile-rendering-intent-relative-colorimetric) rendering intent.

Set the pixels values of imageData for areas of the source rectangle that are outside of the [output bitmap](https://html.spec.whatwg.org/multipage/canvas.html#output-bitmap) to [transparent black](https://drafts.csswg.org/css-color/#transparent-black).

Return imageData.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • document.createElement('canvas').getContext('2d').getImageData(0,0,1,1) returns [0,0,0,0] though IMO it should return null

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so that is fine as well

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Set the pixels values of imageData for areas of the source rectangle that are outside of the

OK I see

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

getImageData(0,0,1,1) should give you a pixel correct? why would you have null?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the spec is confusing. I am asking for a non existing pixel and I get one. But for our sake it is fine since it is transparent

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh i see what you mean.
I think is a point of view. Having always pixels is definitely more ergonomic.
Of course if you are hunting for transparent pixels and you go accidentally out of bounds, you are confused.

@asturur
Copy link
Member

asturur commented Mar 19, 2023

@ShaMan123 keep this branch steady one moment i m going to fork again and open a PR on your PR. with some changes ( don't want to commit directly or discussing changes is a mess )

@ShaMan123
Copy link
Contributor Author

As far as I am concerned you have taken over this PR and are in charge

Copy link
Contributor Author

@ShaMan123 ShaMan123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, unit tests seem to prove accuracy has been reached

Copy link
Contributor Author

@ShaMan123 ShaMan123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we should a test under a transformed group

@asturur
Copy link
Member

asturur commented Mar 19, 2023

Ok i ll merge now.
I m half sad that code readibility affect code size so much.
I wonder if there is some terser option that can help.
Like protected methods do not need to be maintained in full size in naming for example.
We removed a lot in this PR yet the final bundle size growns.

@asturur
Copy link
Member

asturur commented Mar 19, 2023

Kind of we forgot that now you can't read how much tolerance you set.

@asturur asturur merged commit f48f02e into master Mar 19, 2023
@ShaMan123
Copy link
Contributor Author

Kind of we forgot that now you can't read how much tolerance you set.

You mean the enhanced value?

I wonder about this bundle size. Is it really accurate? We dropped 50 lines of code and didn't add anything that is compiled down. So what is this data? Whatever @asturur we made it better.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants