(Big performance change) Do not run lints that cannot emit #125116

blyxyas · 2024-05-14T11:20:44Z

Before this change, adding a lint was a difficult matter because it always had some overhead involved. This was because all lints would run, no matter their default level, or if the user had #![allow]ed them. This PR changes that. This change would improve both the Rust lint infrastructure and Clippy, but Clippy will see the most benefit, as it has about 900 registered lints (and growing!)

So yeah, with this little patch we filter all lints pre-linting, and remove any lint that is either:

Manually #![allow]ed in the whole crate,
Allowed in the command line, or
Not manually enabled with #[warn] or similar, and its default level is Allow

As some lints need to run, this PR also adds loadbearing lints. On a lint declaration, you can use the @eval_always = true marker to label it as loadbearing. A loadbearing lint will never be filtered (it will always run)

Fixes #106983

rustbot · 2024-05-14T11:20:52Z

r? @michaelwoerister

rustbot has assigned @michaelwoerister.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

rustbot · 2024-05-14T11:20:54Z

These commits modify the Cargo.lock file. Unintentional changes to Cargo.lock can be introduced when switching branches and rebasing PRs.

If this was unintentional then you should revert the changes before this PR is merged.
Otherwise, you can ignore this comment.

Some changes occurred in src/tools/clippy

cc @rust-lang/clippy

blyxyas · 2024-05-14T11:22:05Z

cc @nnethercote @Kobzol, the perf wizards. Could you please give this PR a look and tell me if there are any obvious performance issues on the filtering?

matthiaskrgr · 2024-05-14T11:26:37Z

@bors try @rust-timer queue

…r=<try> (Big performance change) Do not run lints that cannot emit Before this lint, adding a lint was a difficult matter because it always had some overhead involved. This was because all lints would run, no matter their default level, or if the user had `#![allow]`ed them. This PR changes that. This change would improve both the Rust lint infrastructure and Clippy, but Clippy will see the most benefit, as it has about 900 registered lints (and growing!) So yeah, with this little patch we filter all lints pre-linting, and remove any lint that is either: - Manually `#![allow]`ed in the whole crate, - Allowed in the command line, or - Not manually enabled with `#[warn]` or similar, and its default level is `Allow` I open this PR to receive some feedback, mainly related to performance. We have lots of `Lock`s, `with_lock` and similar functions (also lots of cloning), so the filtering performance is not the best. In an older iteration, instead of doing this in the parsing phase, we developed a visitor with the same function but without so many locks, would reverting to that change help? I'm not sure tbh.

bors · 2024-05-14T11:27:49Z

⌛ Trying commit 7606f89 with merge cc1d40f...

Kobzol · 2024-05-14T11:54:10Z

@lqd haven't you tried something like this before? 🤔

bors · 2024-05-14T13:04:11Z

☀️ Try build successful - checks-actions
Build commit: cc1d40f (cc1d40f134ee8336cbb7c7561deaed4aa5906e0e)

lqd · 2024-05-14T13:14:38Z

@lqd haven't you tried something like this before? 🤔

We've tried a few different things yes, and so has blyxyas -- it maybe wasn't exactly like this, but I encountered annoyances like: some slow const eval loadbearing lint that shouldn't be ignored, lints that would be allowed unexpectedly because cargo allows lints unconditionally on dependencies (arguably the most common usage, and where perf gains would show up AFAICT) but some may trigger FCWs or are required to lint on dependencies despite being allowed, et cetera.

Refactoring and fixing all these were too costly compared to the gains at the time, as rustc's lints were fast enough on dependencies, also a "rarer" use-case. That being said, we've added and uplifted more lints since then, including possibly costly ones like the non local impls one, and the situation may also be different for clippy itself (but we won't see that in the perf.rlo results, only locally with the clippy dedicated commands IIUC)

rust-timer · 2024-05-14T14:45:34Z

Finished benchmarking commit (cc1d40f): comparison URL.

Overall result: ❌ regressions - ACTION NEEDED

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

Next Steps: If you can justify the regressions found in this try perf run, please indicate this with @rustbot label: +perf-regression-triaged along with sufficient written justification. If you cannot justify the regressions please fix the regressions and do another perf run. If the next run shows neutral or positive results, the label will be automatically removed.

@bors rollup=never
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	0.3%	[0.2%, 0.5%]	25
Regressions ❌ (secondary)	0.4%	[0.1%, 1.6%]	11
Improvements ✅ (primary)	-0.4%	[-0.4%, -0.4%]	1
Improvements ✅ (secondary)	-0.1%	[-0.1%, -0.1%]	1
All ❌✅ (primary)	0.3%	[-0.4%, 0.5%]	26

Max RSS (memory usage)

Results (primary 2.3%, secondary -0.5%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	2.3%	[2.3%, 2.3%]	1
Regressions ❌ (secondary)	3.6%	[2.0%, 5.3%]	2
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-4.7%	[-5.2%, -4.1%]	2
All ❌✅ (primary)	2.3%	[2.3%, 2.3%]	1

Cycles

Results (secondary 2.4%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	2.4%	[1.5%, 3.1%]	6
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 676.788s -> 676.098s (-0.10%)
Artifact size: 316.11 MiB -> 316.15 MiB (0.01%)

Centri3 · 2024-05-14T14:54:05Z

The benchmark doesn't check clippy, right? As lqd hinted at as well? And without splitting allow-by-default rustc lints it does nothing without clippy, so I think this just shows how much time it takes to filter them (can someone else confirm this :3c)

Thus, basically nothing it seems :3 (So @blyxyas maybe the cloning is ok?)

blyxyas · 2024-05-15T08:51:10Z

The benchmark doesn't check clippy, right?

Yeah, the benchmarks currently doesn't check Clippy, that's why I'm currently benchmarking on a different server via SSH (A server that we got explicitly to benchmark Clippy). I'll post the results here when they arrive :)

Also, it currently doesn't check builtin lints because I'm having some issues checking that. That's also part of why I decided to open the PR, maybe someone has some idea (I'll see if I can read the previous attempts by lqd, maybe I can learn something from them)

EDIT: Seems like lqd hasn't pushed their attempts, I'll have to keep trying new approaches by myself.

blyxyas · 2024-05-15T11:04:14Z

Okis, here are the results (Wall time, Clippy)

Wall time

❌
[ +0.45%, +190.50%]
+9.60% 81 (36)
✅
[-73.80%, -0.44%]
-7.00% 77 (36)
❌,✅
[-73.80%, +190.50%]
+1.51% 158 (44)

Max RSS

❌
[ +0.40%, +2.80%]
+1.17% 21 (16)
✅
[ -5.15%, -0.44%]
-1.60% 39 (24)
❌,✅
[ -5.15%, +2.80%]
-0.63% 60 (31)

Instructions

❌
[ +0.42%, +0.62%]
+0.52% 6 (4)
✅
[ -1.82%, -0.32%]
-0.74% 13 (7)
❌,✅
[ -1.82%, +0.62%]
-0.34% 19 (11)

Cycles

[ +0.42%, +25.48%]
+4.33% 91 (38)
✅
[-14.52%, -0.40%]
-3.33% 59 (32)
❌,✅
[-14.52%, +25.48%]
+1.31% 150 (43)

blyxyas · 2024-05-15T11:06:43Z

Those wall times are proof that this optimization has a lot of potential, the main drawback is that the filtering / parsing code is not fast enough, so in some scenarios that I'm not really able to determine exactly what do they have in common, the optimization goes backwards.

But a ~70% in Wall time, that's great and we should look more into it.

Kobzol · 2024-05-15T11:09:50Z

I wouldn't draw too many conclusions from these results, they seem to be quite unstable (there is also a 190% walltime regression). Note that even for PRs that don't have large perf. impacts, we can see ~30% walltime swings even on the stable benchmarking server (https://perf.rust-lang.org/compare.html?start=9e7aff794539aa040362f4424eb29207449ffce0&end=44fa5fd39a1d2af41bd7f43bc246a5e4f6d94696&stat=wall-time&nonRelevant=true).

blyxyas · 2024-05-15T17:13:50Z

I've changed the system, we're back to using visitors (I've benchmarked this new commit, it should have 0 regressions and about -0.66% improvement)

@bors try @rust-timer queue

bors · 2024-05-15T17:15:02Z

⌛ Trying commit 828cd60 with merge 68a9e31...

bors · 2024-10-07T11:49:43Z

☔ The latest upstream changes (presumably #131235) made this pull request unmergeable. Please resolve the merge conflicts.

Before this change, adding a lint was a difficult matter because it always had some overhead involved. This was because all lints would run, no matter their default level, or if the user had #![allow]ed them. This PR changes that

cjgillot · 2024-10-26T14:02:40Z

Do you intend to investigate https://github.com/rust-lang/rust/pull/125116/files#r1770598991 as a follow-up?

Thanks for the work anyway!
@bors r+

bors · 2024-10-26T14:02:42Z

📌 Commit 1dcfa27 has been approved by cjgillot

It is now in the queue for this repository.

bors · 2024-10-26T16:37:47Z

⌛ Testing commit 1dcfa27 with merge 4d88de2...

bors · 2024-10-26T19:00:10Z

☀️ Test successful - checks-actions
Approved by: cjgillot
Pushing 4d88de2 to master...

rust-timer · 2024-10-26T20:18:26Z

Finished benchmarking commit (4d88de2): comparison URL.

Overall result: no relevant changes - no action needed

@rustbot label: -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

Results (secondary -4.7%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-4.7%	[-4.7%, -4.7%]	1
All ❌✅ (primary)	-	-	0

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 781.782s -> 783.187s (0.18%)
Artifact size: 333.71 MiB -> 333.73 MiB (0.01%)

rustbot assigned michaelwoerister May 14, 2024

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. T-rustdoc Relevant to the rustdoc team, which will review and decide on the PR/issue. labels May 14, 2024

blyxyas marked this pull request as draft May 14, 2024 11:23

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label May 14, 2024

This comment has been minimized.

Sign in to view

rustbot added perf-regression Performance regression. and removed S-waiting-on-perf Status: Waiting on a perf run to be completed. labels May 14, 2024

This comment has been minimized.

Sign in to view

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label May 15, 2024

blyxyas force-pushed the ignore-allowed-lints-final branch from cf3326e to 187ff0c Compare September 26, 2024 00:06

This comment has been minimized.

Sign in to view

blyxyas force-pushed the ignore-allowed-lints-final branch from 187ff0c to 4cb4c1a Compare September 26, 2024 13:27

jieyouxu removed their assignment Sep 26, 2024

michaelwoerister removed their assignment Oct 7, 2024

blyxyas force-pushed the ignore-allowed-lints-final branch from 4cb4c1a to 70e9bc2 Compare October 10, 2024 22:45

blyxyas added 6 commits October 19, 2024 16:19

Do not run lints that cannot emit

b4da058

Before this change, adding a lint was a difficult matter because it always had some overhead involved. This was because all lints would run, no matter their default level, or if the user had #![allow]ed them. This PR changes that

Change lints_to_emit to lints_that_actually_run

edc6577

Follow review comments (optimize the filtering)

71b4d10

Remove module passes filtering

637d5cc

Unify syntax (all to @eval_always)

8a40884

Apply review comments + use shallow_lint_levels_on

ddad55f

blyxyas force-pushed the ignore-allowed-lints-final branch from 70e9bc2 to ddad55f Compare October 19, 2024 14:48

Move COGNITIVE_COMPLEXITY to use macro again

1dcfa27

blyxyas force-pushed the ignore-allowed-lints-final branch from 64c914b to 1dcfa27 Compare October 21, 2024 17:27

Dylan-DPC added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Oct 25, 2024

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Oct 26, 2024

bors added the merged-by-bors This PR was explicitly merged by bors. label Oct 26, 2024

bors merged commit 4d88de2 into rust-lang:master Oct 26, 2024
7 checks passed

rustbot added this to the 1.84.0 milestone Oct 26, 2024

This was referenced Oct 26, 2024

Add lint against function pointer comparisons #118833

Open

Add lint against (some) interior mutable consts #132146

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

(Big performance change) Do not run lints that cannot emit #125116

(Big performance change) Do not run lints that cannot emit #125116

blyxyas commented May 14, 2024 •

edited

Loading

rustbot commented May 14, 2024

rustbot commented May 14, 2024

blyxyas commented May 14, 2024

matthiaskrgr commented May 14, 2024

This comment has been minimized.

bors commented May 14, 2024

This comment has been minimized.

Kobzol commented May 14, 2024

bors commented May 14, 2024

This comment has been minimized.

lqd commented May 14, 2024 •

edited

Loading

rust-timer commented May 14, 2024

Centri3 commented May 14, 2024 •

edited

Loading

blyxyas commented May 15, 2024 •

edited

Loading

blyxyas commented May 15, 2024

blyxyas commented May 15, 2024

Kobzol commented May 15, 2024

blyxyas commented May 15, 2024

This comment has been minimized.

bors commented May 15, 2024

This comment has been minimized.

bors commented Oct 7, 2024

cjgillot commented Oct 26, 2024

bors commented Oct 26, 2024

bors commented Oct 26, 2024

bors commented Oct 26, 2024

rust-timer commented Oct 26, 2024

(Big performance change) Do not run lints that cannot emit #125116

(Big performance change) Do not run lints that cannot emit #125116

Conversation

blyxyas commented May 14, 2024 • edited Loading

rustbot commented May 14, 2024

rustbot commented May 14, 2024

blyxyas commented May 14, 2024

matthiaskrgr commented May 14, 2024

This comment has been minimized.

bors commented May 14, 2024

This comment has been minimized.

Kobzol commented May 14, 2024

bors commented May 14, 2024

This comment has been minimized.

lqd commented May 14, 2024 • edited Loading

rust-timer commented May 14, 2024

Overall result: ❌ regressions - ACTION NEEDED

Centri3 commented May 14, 2024 • edited Loading

blyxyas commented May 15, 2024 • edited Loading

blyxyas commented May 15, 2024

Wall time

Max RSS

Instructions

Cycles

blyxyas commented May 15, 2024

Kobzol commented May 15, 2024

blyxyas commented May 15, 2024

This comment has been minimized.

bors commented May 15, 2024

This comment has been minimized.

bors commented Oct 7, 2024

cjgillot commented Oct 26, 2024

bors commented Oct 26, 2024

bors commented Oct 26, 2024

bors commented Oct 26, 2024

rust-timer commented Oct 26, 2024

Overall result: no relevant changes - no action needed

blyxyas commented May 14, 2024 •

edited

Loading

lqd commented May 14, 2024 •

edited

Loading

Centri3 commented May 14, 2024 •

edited

Loading

blyxyas commented May 15, 2024 •

edited

Loading