Use PooledArrayBuilder<SyntaxToken> throughout parsers and tokenizer

[DustinCampbell]

@ToddGrun shared a couple PerfView stacks with me where the Razor compiler was allocating List<SyntaxToken> instances in the tokenizer. To remove those allocation, this change replaces List<SyntaxToken>s throughout the parsers and tokenizer, and uses PooledArrayBuilder<SyntaxToken> instead. In addition, I removed a couple of Concat calls and replaced them with pooled StringBuilders.

[ryzngard]

@ToddGrun shared a couple PerfView stacks with me where the Razor compiler was allocating List<SyntaxToken> instances in the tokenizer. To remove those allocation, this change replaces List<SyntaxToken>s throughout the parsers and tokenizer, and uses PooledArrayBuilder<SyntaxToken> instead. In addition, I removed a couple of Concat calls and replaced them with pooled StringBuilders.

❓is yardstick still a thing? does it capture these?

[ToddGrun]

Found in the completion speedometer test

[DustinCampbell]

❓is yardstick still a thing? does it capture these?

Yardstick is still a thing. What it really does is run PerfView with a set of good defaults for Visual Studio, does a bunch of analysis over the results, and produces a text report. In particular, I've found the CaptureMeasurements.bat included with it to be particularly useful. It launches PerfView with a bunch of event trace sources enabled for VS, like StreamJsonRpc and MSBuild.

[ryzngard]

❓is yardstick still a thing? does it capture these?

Yardstick is still a thing. What it really does is run PerfView with a set of good defaults for Visual Studio, does a bunch of analysis over the results, and produces a text report. In particular, I've found the CaptureMeasurements.bat included with it to be particularly useful. It launches PerfView with a bunch of event trace sources enabled for VS, like StreamJsonRpc and MSBuild.

Oh yea, I just meant I generally enjoy the amount of warnings/errors it gave on finding things that are probably bad. I was hoping if you ran it on this speedometer etl it would just be like "Hey, don't do that please"

[DustinCampbell]

Oh yea, I just meant I generally enjoy the amount of warnings/errors it gave on finding things that are probably bad. I was hoping if you ran it on this speedometer etl it would just be like "Hey, don't do that please"

@ToddGrun is my yard stick. 😄

[333fred]

Generally seems fine to me, but it does seem like ref readonly would be better in a few places than in here.