Synthrace workers and refactoring

[Mpdreamz]

Refactor DSL

• Naming change from being span based to entity based. E.g SpanIterable => EntityIterable
• concat two streams together is now done through merge the resulting iterable round robins over all the streams it holds weighted by rate per minute (more on that later)
• randomize() on Interval allows you to quickly create intervals with variable yields.
• Included unit tests for the DSL behavior, ratePerMinute as well as weighted merging of entity generators.

Rate per minute estimates.

Both Interval and EntityIterable implementations can now report an estimated event rate per minute. This is used in various places but most notably to estimate the range needed to satisfy --maxDocs. It's also used to ensure the segmentation over workers is sufficient.

CLI improvements

• The return of the workers, this reintroduces workers to the CLI. Thanks to the ratePerMinute() estimates you can now target any scenario and the tool will automatically select an appropriate overal range, worker ranges + concurrency settings.
• Improves reporting e.g it now provides visual progress updates every Math.min(2000000, maxDocs /20) documents.
  • A summary for each worker is now printed at the end of the run as well.
• Introduces --local flag
  • For local development purposes this discovers both Elasticsearch and Kibana (with current basePath).
• An explicit --kibana flag also exists to accompay --target
• Will automatically install the appropriate APM integration package if either --kibana or --local are used.
  • When using --cloudId we still use our own migrate API to move over.
• Adds a --gcpRepository flag this is to aid with restoring data or snapshotting a large local dataset to share with others. See also Extend yarn es: plugin support + adding secure files to es keystore elastic/kibana#126938
• Added a --maxDocConfidence parameter, ratePerMinute works by peeking the first document of a stream. If your traces are more variable then that you can amplify/dampen the estimates using this parameter.
• --dryRun will simply iterate over the StreamProcessor, useful to debug processor performance.
• The tool now assumes datastreams, you have to set --forceLegacyIndices to revert to old writeTargets behavior.
• The CLI now also discovers the running Elasticsearch version and makes sure the documents StreamProcessor yields have an equivalent observer.version so that APM's pipeline processors accept the source documents.