Is Select Row too variable again?

[ryansolid]

I don't know if it is just speed improvements or was slowdown turned off (visually doesn't appear to be). But when I run locally (given the official results I assume this is the case here as well), I get a bunch of either 2ms or 20ms. And the score becomes a tally of how many fall which way. Making it way too variable. More so it has a larger impact of relative score since being a few ms off a score that averages around 16ms is seen more significant. Are we back at #424?

[krausest]

I shouldn't have bought a faster PC. Or intel should have more CPU bugs and I should care more about them...

Seriously, select row is too fast for the real fast frameworks. Solid passes vanillajs due to that benchmark.

Still I see some value in this benchmark. It definitely shows that vue.js has an opportunity to improve:

What are the options?

1. Dropping select row benchmark. I would be a bit unhappy about that since there are many framework in the range of 60 msecs and a few doing even worse.
2. Soften the results, something like take max(measured value, arbitrary value e.g. 32 msecs). For the fast frameworks this would result in a factor of 1 and only real slow frameworks would be punished.
3. Increase CPU slowdown. The benchmark runs already with 16x CPU thorttlong. I'm not sure if higher values are meaningful.
4. Changing the benchmark task, but I don't think we can update all 137 implementations.

I think I'm for #2, but we've had something like that and we've had discussions about it.

[ryansolid]

Yeah doesn't sound like there are any really good options. This benchmark is a differentiator even on the top end (back when it was in the 30-40ms range). You can see on the graph even that say DomC/Stage0 are in a different range than the others. I'd love to not lose that difference but I also don't see meaningful ways to make this test any more heavy. And I imagine clamping at 20ms won't do enough.

As you said updating the tests would be hard. Making the list longer probably doesn't do much (and I believe you tried that before). Multi-select actually would hide the cost here (it's that you need to search the whole list to find a match). It would push the cost on to the DOM update and not on to the logic to determine that update. This test tests something very unique.

On the other hand, this test is also the most controversial one. It is one of two tests that rewards event delegation. The one that encourages people to do direct DOM manipulation, or dirtying the data model by adding isSelected to each row. No library that is near the top is not using one of these techniques. I'd like to think that there is a generalizable declarative solution here that one could legitimately use to achieve the same. (I've played with binding syntaxes to do the same previously in Solid but they all proved too verbose).

I will say this. The results are less variable than I was expecting. They were on my machine but the box plot here doesn't look as bad as I was expecting. Maybe do nothing for now is an option. I mean VanillaJS is bad but it seems to be more of an exception. Maybe it's time to look at the VanillaJS implementations again. I wasn't expecting Solid's recent results, but I have an idea of what may have caused them(beyond an absurdly good run). I think I had a couple misconceptions in VanillaJS-1, and if I employed a technique that is a hybrid of both approaches we could make it even better.

[benmccann]

I was thinking something really weird happened between benchmark 8 and the snapshot, but I just noticed that the row label changed. So I guess the test aws changed to add CPU slowdown? I'm curious why that would change the relative results so much though. E.g. Vue dropped way down the list. Maybe Vue's implementation is more CPU intensive than the others?

[ryansolid]

Chrome 83 particularly has some issues which could be exaggerating it. That being said I have a bunch of snapshots from articles I've written and Vue has been particularly bad here since the slowdown was introduced. It would be interesting to understand what particularly is expensive here since it is fairly unique to Vue and Vue 3 still has this even with its improved performance.

[krausest]

Yes, there was indeed a change. So far select rows used to select 6 rows as a "warm up" and the seventh click was measured. It turned out that 6 isn't sufficient for warmup (see #741 (comment)). So I decided to remove the warmup.
I'll still have to check whether it's okay to reuse the chomedriver instance for select rows (it might be interesting to see whether there's a difference between just reloading the page and recreating the chromedriver instance for each run).

[benmccann]

Evan, the maintainer of Vue, suggested the reason that Vue is doing worse than the other frameworks is because they're doing different things. I'm not familiar enough with Vue to know whether it could be similarly optimized the way the other frameworks are. Or are the other frameworks taking shortcuts we don't want to encourage in the benchmarking?

vuejs/core#1331 (comment)

[leeoniya]

@benmccann take all metrics that run with 16x CPU slowdown with a huge grain of salt.

[ryansolid]

@benmccann I see.. yes. Yeah it's a tradeoff. See being a VDOM implementation splitting into components probably would improve performance since that is how it controls change propagation. But being reactive I imagine those components having a steeper creation cost. I've actually done tests into breaking down into more Components with other libraries and having less definitely overall favors libraries like Svelte and LitHTML. Still Vue is quite a bit slower here.

So 3 things here:

1. As @leeoniya said this is a bit harsh test. Doing a simulated 16x slow down, is a bit like zooming in with your camera on your phone to measure distance.
2. Are libraries taking shortcuts? Definitely. This is the test most prone to shortcuts. That being said only really being exploited by a small number of libraries. Projecting selected to isSelected makes a difference and many libraries have creative ways of doing that.
3. That being said Vue is still relatively slow here even with all that in consideration. Most of the time their hybrid Reactive/VDOM solution brings a lot of benefits. But this is a place where there is no best solution since you are trading downsides of one half of the approach for the downsides of the other. Here it is the worst of both worlds.

I mean in perspective we are talking about an absurd scenario of 1000 rows at 16x throttling. You lessen any part of this scenario and the performance is nothing to worry about.

[krausest]

I'm closing older issues that won't get fixed.