measure NLL performance on perf

[nikomatsakis]

It would be great to be tracking NLL performance on perf.rust-lang.org. I'm not entirely sure the best way to go about this. I've been tracking NLL perf on the syn benchmark specifically, but I'd obviously like to also test a broader set.

Three alternatives:

• Add patches to select benchmarks on perf that add #![feature(nll)].
  • Incremental ought to ensure that we rebuild what needs to be rebuilt in this case; I'm a bit wary in that I'm not sure exactly what that set will be, but it ought to include at least MIR borrowck.
• Clone the benchmarks and add #![feature(nll)] into the lib.rs
  • Thus we would have a distinct syn-nll benchmark, for example
• Just add a new NLL mode
  • Similar to -opt, consider running all benchmarks with NLL enabled using RUSTFLAGS
  • This might fail for some benchmarks, not sure, but hopefully not

I'm sort of leaning towards the last one, but I'd love to get feedback, as well as advice from @Mark-Simulacrum on how to implement whichever route we choose.

cc @rust-lang/wg-compiler-performance @rust-lang/wg-compiler-nll @Mark-Simulacrum

[Mark-Simulacrum]

Since we probably want to compare normal vs. NLL performance I think a patch would work best. If we consistently name it nll.patch or something like that we can fairly easily schedule it to run in a clean environment that won't interfere with other incremental diffs.

Cloning benchmarks or adding a new NLL mode seem like not the best idea to me since comparing will be somewhat difficult. However, it's arguably more consistent with the future stabilized NLL, so perhaps might be better. I feel like it doesn't provide the same benefits as the patch, though.

It should be relatively easy to add the ability to "enable" NLL as a patch for a particular benchmark that would act somewhat like the suggested mode (i.e., enable via RUSTFLAGS) or via cargo rustc -- -Znll). Either shouldn't take too long to implement and would seemingly work rather well for the intended use case, I think.

Happy to mentor/implement any of these (or, if we go the patch route, just deploy).

[nikomatsakis]

I guess we would only want the NLL to apply to the 'leaf crate' (versus all dependencies), just to keep things more controlled.

[nikomatsakis]

@Mark-Simulacrum would you rather just make the patches yourself, or mentor? Are there instructions I can point people at?

I say let's do these tests:

• syn-0.12.0
• clap-rs
• crates-io
• hyper
• html5ever
• inflate
• parser
• regex

How did I choose these? Semi-arbitrarily, but they seem like "real rust code in the wild". I guess we don't measure serde? Based on #47597 it sounds like that was a perf problem at some point:

[These measurements] add up to roughly all of the time difference. They are for the serde, syn, serde_derive_internals, and serde_derive crates respectively.

[Zoxc]

Could we parse the output of -Z time-passes to get some more details of things?

[nikomatsakis]

@Zoxc please no. -Ztime-passes does not yield useful information.

[nikomatsakis]

rust-lang/rustc-perf#182 would be awesome though

[Mark-Simulacrum]

I've added a sample PR of what I think we'll want this to look like here with some steps rust-lang/rustc-perf#194, I'm going to merge and deploy that and if we like what we see then we can replicate it over the other benchmarks or change the approach.

[nikomatsakis]

Hmm, I'm finding this a bit harder to use than anticipated. It does give us some idea of the relative motion of NLL over time, but it's hard to get an "absolute" sense of how good/bad it is. For examine, I don't think that patch can be directly compared to any of the other lines -- if we did a make clean or something before, then we could compare to the "clean incremental" build though.

[Mark-Simulacrum]

I've quickly implemented an alternative which makes all benchmarks (we might limit this in the future, but for now this was easier) have an "nll" mode which should look just like clean on the UI and will run after clean (with just a touch on all files, but that should be enough as Cargo will rebuild anyway).

Will deploy shortly and we'll see how this looks.

[nikomatsakis]

@Mark-Simulacrum does it run with incremental enabled or disabled?

[Mark-Simulacrum]

Disabled, but we can change that without too much difficulty. Or do both. Incremental shouldn't really matter too much here I'd hope since it's a clean environment anyway (so it might slow it down)...

[nikomatsakis]

@Mark-Simulacrum I prefer disabled for now at least, makes it easier for us to measure the delta.

[nikomatsakis]

@Mark-Simulacrum I didn't see anything yet on perf.rust-lang.org though (not sure if I should expect to, or am looking at the wrong things)

[pnkfelix]

(for those following along: There is now data available on perf.rust-lang.org, and it tends to show that NLL is indeed currently injecting a huge performance hit.)

[Mark-Simulacrum]

@nikomatsakis I'm going to close this as I believe we've implemented what we desired here -- but please let me know if that's incorrect.