Faster AST Construction

[RetroDev256]

• Faster on majority of source files
• Doesn't make the source code too obscure

With alternate changes, I could consistently get ~10% speed improvements on large files, but it had a big downside. Firstly, smaller files took a lot longer to parse, and secondly, it made the Parse.zig source code really hard to maintain.

With these sets of tweaks, a typical source file will be parsed ~3-5% faster, but very small files seem to be parsed slightly slower. Do note that by "slightly slower", it is a percentage difference. Smaller files are inherently less complicated, so less time is gained/wasted by the same percentage difference of parsing a small file, than a large file, and if you look at the benchmarks, the standard deviation makes telling if something was "slow" or "fast" pretty difficult.

This does constitute a breaking change on std.zig.Tokenizer.Token - my reasoning for the change is that the language itself limits itself to 2^32 bytes per file at most, so the standard library should not need to bother allowing us to tokenize up to 2^64 bytes of source, while not being able to parse it all. With the breaking change, I got rid of that annoying 'loc', which personally I don't think was too necessary.

Optimizations in the Zig compiler are of course really enjoyable to try out, but do note that the AST construction takes a very insignificant time in the grand scheme of things.

Benchmarking ast-check on Sema.zig:

Benchmarking ast-check on print_air.zig:

Benchmarking ast-check on the zig init main.zig:

[Rexicon226]

Just a quick glance.

[RetroDev256]

For the recent changes, perf on the zig init main.zig and print_air.zig remain very slightly improved, but now I am consistently seeing this benchmark for Sema.zig:

[mlugg]

Why has this changed? Storing only the start offset of the token is a very intentional design choice.

[RetroDev256]

I found that the tokenSlice function in Ast.zig could avoid running the tokenizer if I stored the end index. Would you like me to revert that change?

That change alone is responsible for the higher memory usage, but also cut off a few percent on the wall time.

[mlugg]

Yes, please. This tradeoff was made intentionally to save memory at the potential expense of a small amount of performance. I say "potential" because tokenization is fast enough that it could be faster to re-tokenize this one then than to get more cache misses from storing this data in memory. See Andrew's talk on DOD for details.

If the performance impact is more significant than expected perhaps we can revisit this. If you'd like to quantify the performance difference this gives on ast-check, that'd be appreciated.

[RetroDev256]

Here is the benchmark on Sema.zig with the changes to Ast.zig:

Here is the benchmark on Sema.zig without the changes to Ast.zig:

The results honestly make me wonder if anything else I did was just pure noise.

[RetroDev256]

And the difference between these two runs is purely the pub const TokenList = std.MultiArrayList(Token); change (alongside the tokenSlice change), without any of the other changes in this PR thrown in.

[RetroDev256]

I feel that based on these findings, I should close this PR if we want the memory savings.

[RetroDev256]

In any case, the changes aren't playing well with translate-c/ast.zig, I don't think this change is significant enough to warrant a ton of code shuffling.