Example browser benchmark

[Beanow]

It isn't the prettiest framework, but it produced some insights.
https://github.com/Beanow/pull-block-browserbench

To test the #2 theories. There's a "dev" version of pull-block there which has the implementation without any concat or slices, using only alloc and copy. PR #4

var tests = {
	// Suspected worst-case.
	manyPairs: test({
		inSize: 16*KB,
		inCount: 10000,
		blockSize: 30*KB
	}),

	// Suspected worst-case.
	manyTriplets: test({
		inSize: 11*KB,
		inCount: 14000,
		blockSize: 30*KB
	}),

	// Large incoming buffers scenario.
	bigBuffers: test({
		inSize: 20*MB,
		inCount: 1,
		blockSize: 50*KB
	})
}

Firefox:
Test manyPairs with pull-block dev ran an average of 107ms
Test manyPairs with pull-block master ran an average of 257ms
Test manyPairs with pull-block v1.2.0 ran an average of 178ms
Test manyTriplets with pull-block dev ran an average of 105ms
Test manyTriplets with pull-block master ran an average of 318ms
Test manyTriplets with pull-block v1.2.0 ran an average of 186ms
Test bigBuffers with pull-block dev ran an average of 39ms
Test bigBuffers with pull-block master ran an average of 23ms
Test bigBuffers with pull-block v1.2.0 ran an average of 2951ms

Chromium:
Test manyPairs with pull-block dev ran an average of 63ms
Test manyPairs with pull-block master ran an average of 99ms
Test manyPairs with pull-block v1.2.0 ran an average of 57ms
Test manyTriplets with pull-block dev ran an average of 61ms
Test manyTriplets with pull-block master ran an average of 101ms
Test manyTriplets with pull-block v1.2.0 ran an average of 66ms
Test bigBuffers with pull-block dev ran an average of 25ms
Test bigBuffers with pull-block master ran an average of 16ms
Test bigBuffers with pull-block v1.2.0 failed: RangeError: Array buffer allocation failed

So it's a gain, but certainly not as big of a difference as the previous patch. And interestingly it seems my previous patch slightly regressed performance for the high volume scenario.

I do suppose that these tests don't cover enough cases to decide which is better. Hence I'm putting it up as a separate issue. Does this make sense as a test method and what more cases does it need?

[Beanow]

I think after a few rounds of perf improvements this testing method is starting to show it's weaknesses. Particularly how often a test is repeated and how outliers are handled.

Here's a test on Firefox with 22 repeats, removing the min and max value before averaging, listing them separately. (There should be no changes in the code paths for these tests between master/dev).

Test	Version	Min	AVG	Max
almostHalves	master	21ms	84ms	150ms
almostHalves	dev	19ms	63ms	156ms
almost10ths	master	11ms	42ms	144ms
almost10ths	dev	11ms	35ms	156ms
tinyBufs	master	177ms	204ms	308ms
tinyBufs	dev	169ms	208ms	307ms

Note that almostHalves has 21ms difference in AVG, even though they run identical code, across the 20 results used for each value. My guess is that these large deviations have to do with the browser's scheduling and garbage collection. As the same test on Chromium looks like:

Test	Version	Min	AVG	Max
almostHalves	master	22ms	45ms	63ms
almostHalves	dev	21ms	43ms	63ms
almost10ths	master	7ms	13ms	41ms
almost10ths	dev	8ms	13ms	20ms
tinyBufs	master	56ms	91ms	100ms
tinyBufs	dev	83ms	92ms	98ms

Both browsers show though, the minimums do not look like the averages, which do not look like the maximums. Significant deviation (2x-15x max vs min) is to be expected.

Concluding

I would say, this method can't currently show small improvements like 10-20% better than before due to it being so jittery. Specific test cases should be made to magnify performance gains so they show around 2x improvement or more for that case.

Also, the results between browsers should be compared to estimate the influence of scheduling and GC.

Future

For the time being I think pull-block has gained a lot of speed boost and is unlikely to be the performance bottleneck for it's current use-cases. Now is probably a good time to look at other components for this type of perf attention rather than diving into the 10-20% improvements domain.

So here are some options:

• Current test cases can be used as regression checks for new features.
• Adopt a testing framework that has less naive result aggregation.
• Look into measuring the amount of memory being allocated and GC calls being made, as indicators of efficiency.

[dignifiedquire]

I have been using Benchmark quite successfully in other projects, and it's pretty easy to use in both browser and nodejs if you make sure to assign window.Benchmark = require('benchmark') in the browser. so might be good to migrate the benchmarks to use that. there is a nice setup for this which compares current checked out version with the npm version here for example: https://github.com/crypto-browserify/browserify-aes/blob/master/bench/index.js

[Beanow]

I spoke too soon about the inconsistent results in Firefox. Turns out the DOM changes I did for displaying status triggered uBlock and that ruined most of the performance. Disabling all addons and doing less DOM operations speeds up and smoothes out the results a lot. Almost on par with Chromium.

Test	Version	Min	AVG	Max
almostHalves	master	20ms	48ms	70ms
almostHalves	dev	24ms	59ms	73ms
almost10ths	master	10ms	26ms	56ms
almost10ths	dev	6ms	22ms	54ms
tinyBufs	master	87ms	108ms	125ms
tinyBufs	dev	88ms	102ms	124ms

[dignifiedquire]

Published the latest version as 1.2.1