perf(gatsby): use fastq instead of better-queue + refactor

[vladar]

Description

This PR is a polished version of this commit: 78d6986

Original comment from @KyleAMathews:

Various optimizations to query running that make QPS 20% faster. I moved all work into the queue so the event loop wouldn't get overwhelmed for large sites + moved to a more efficient queue https://www.npmjs.com/package/fastq — our previous queue, better-queue was ~7-10% of the CPU by itself and fastq is maybe 0.1%.

One major benefit of better-queue was support for assigning priority to tasks. We used it in develop to run the currently active query first. But after "query on demand" this is no longer needed as we run only active queries anyway.

Benchmarks

Benchmarks with query-filter-sort site:

cross-env NUM_NODES=10000 yarn bench

Before:

success run page queries - 1.962s - 1001/1001 510.13/s
success run page queries - 2.088s - 1001/1001 479.45/s
success run page queries - 1.735s - 1001/1001 576.95/s
success run page queries - 2.056s - 1001/1001 486.87/s
success run page queries - 1.780s - 1001/1001 562.41/s
success run page queries - 1.853s - 1001/1001 540.20/s
success run page queries - 1.676s - 1001/1001 597.24/s
success run page queries - 1.740s - 1001/1001 575.15/s

After:

success run page queries - 1.573s - 1001/1001 636.54/s
success run page queries - 1.566s - 1001/1001 639.17/s
success run page queries - 1.637s - 1001/1001 611.55/s
success run page queries - 1.627s - 1001/1001 615.29/s
success run page queries - 1.665s - 1001/1001 601.12/s
success run page queries - 1.763s - 1001/1001 567.75/s
success run page queries - 1.566s - 1001/1001 639.24/s
success run page queries - 1.575s - 1001/1001 635.54/s
success run page queries - 1.553s - 1001/1001 644.61/s

Interestingly version of fastq with promises is slower (has basically the same numbers as current version)

[KyleAMathews]

Good deleted/added code ratio 👍

Looks like a ~12% speedup for 1000 queries (using my handy one-liner — split(\n).map(l => parseFloat(l.split( ).slice(-1)[0].split(/)[0])).reduce((a,b) => a + b) / 9)

I got 20% but that was with 100k queries — the more queries the bigger the speedup I'd assume given better-queue's sorting logic would get slower on bigger datasets.

[vladar]

Yeah, also benchmark setup is probably different. I believe my benchmark still does a pass of fast filters (since I am not filtering by the id field). Filtering by id has less overhead and so the perf benefit is more noticeable.

[pieh]

Awesome results. LGTM