Improve performance

[gwer]

Hi!

I am using ts-proto and have noticed that the generation takes a very long time. With 2000 input lines and 25000 output lines, the generation takes 20 seconds.

Plugin profiling showed that the issue is with ts-poet.

I got rid of unnecessary tree traversals, and also optimized working with arrays. My .proto is now processed in 6 seconds.

I also made a simple benchmark to show the difference. The result is clearly visible on large files.

const Benchmark = require("benchmark");
const { code, joinCode } = require("./build");
const { code: codeOld, joinCode: joinCodeOld } = require("./build_old");

const codes = new Array(10000).fill(null).map(
  (_, i) => code`
    console.log('Hello world ${i});
  `
);

const codesOld = new Array(10000).fill(null).map(
  (_, i) => codeOld`
    console.log('Hello world ${i});
  `
);

const suite = new Benchmark.Suite();

suite
  .add("Old: 1 lines", function () {
    joinCodeOld(codesOld.slice(0, 1)).toString();
  })
  .add("New: 1 lines", function () {
    joinCode(codes.slice(0, 1)).toString();
  })
  .add("Old: 10 lines", function () {
    joinCodeOld(codesOld.slice(0, 10)).toString();
  })
  .add("New: 10 lines", function () {
    joinCode(codes.slice(0, 10)).toString();
  })
  .add("Old: 100 lines", function () {
    joinCodeOld(codesOld.slice(0, 100)).toString();
  })
  .add("New: 100 lines", function () {
    joinCode(codes.slice(0, 100)).toString();
  })
  .add("Old: 1000 lines", function () {
    joinCodeOld(codesOld.slice(0, 1000)).toString();
  })
  .add("New: 1000 lines", function () {
    joinCode(codes.slice(0, 1000)).toString();
  })
  .add("Old: 5000 lines", function () {
    joinCodeOld(codesOld.slice(0, 5000)).toString();
  })
  .add("New: 5000 lines", function () {
    joinCode(codes.slice(0, 5000)).toString();
  })
  .add("Old: 10000 lines", function () {
    joinCodeOld(codesOld.slice(0, 10000)).toString();
  })
  .add("New: 10000 lines", function () {
    joinCode(codes.slice(0, 10000)).toString();
  })
  .on("cycle", function (event) {
    console.log(String(event.target));
  })
  .run();

Benchmark results:

Old: 1 lines x 18,456 ops/sec ±2.23% (87 runs sampled)
New: 1 lines x 18,989 ops/sec ±0.40% (95 runs sampled)
Old: 10 lines x 17,859 ops/sec ±0.51% (95 runs sampled)
New: 10 lines x 18,386 ops/sec ±0.55% (95 runs sampled)
Old: 100 lines x 5,056 ops/sec ±1.69% (91 runs sampled)
New: 100 lines x 10,970 ops/sec ±0.44% (94 runs sampled)
Old: 1000 lines x 162 ops/sec ±2.19% (82 runs sampled)
New: 1000 lines x 2,603 ops/sec ±9.35% (88 runs sampled)
Old: 5000 lines x 7.54 ops/sec ±7.95% (22 runs sampled)
New: 5000 lines x 575 ops/sec ±0.54% (93 runs sampled)
Old: 10000 lines x 2.01 ops/sec ±3.51% (10 runs sampled)
New: 10000 lines x 291 ops/sec ±0.28% (92 runs sampled)

Tests are green. The outputs is the same for my inputs. But I'm not sure about the conditions in deepFindAll method. I may have missed some corner cases. So please double check this after me.

[stephenh]

Hey @gwer , this is great!

I must admit I went too long without paying attention to performance, and then recently got lucky that just switching from prettier to dprint gave a huge performance gain, and then I had stopped there.

I'll release a new version of this and then we can bump ts-poet in ts-proto and see how it works over there (i.e. the ts-proto integration tests can serve as another regression test of this change). But so far everything looks good to me!

[stephenh]

I manually published ts-poet 6.2.0 with this change. Thanks again!