Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Foundations of location-sensitive polonius #134268

Merged
merged 6 commits into from
Dec 22, 2024
Merged

Conversation

lqd
Copy link
Member

@lqd lqd commented Dec 13, 2024

I'd like to land the prototype I'm describing in the polonius project goal. It still is incomplete and naive and terrible but it's working "well enough" to consider landing.

I'd also like to make review easier by not opening a huge PR, but have a couple small-ish ones (the +/- line change summary of this PR looks big, but >80% is moving datalog to a single place).

This PR starts laying the foundation for that work:

  • it refactors and collects 99% of the old datalog fact gen, which was spread around everywhere, into a single dedicated module. It's still present at 3 small places (one of which we should revert anyways) that are kinda deep within localized components and are not as easily extractable into the rest of fact gen, so it's fine for now.
  • starts introducing the localized constraints, the building blocks of the naive way of implementing the location-sensitive analysis in-tree, which is roughly sketched out in https://smallcultfollowing.com/babysteps/blog/2023/09/22/polonius-part-1/ and https://smallcultfollowing.com/babysteps/blog/2023/09/29/polonius-part-2/ but with a different vibe than per-point environments described in these posts, just r1@p: r2@q constraints.
  • sets up the skeleton of generating these localized constraints: converting NLL typeck constraints, and creating liveness constraints
  • introduces the polonius dual to NLL MIR to help development and debugging. It doesn't do much currently but is a way to see these localized constraints: it's an NLL MIR dump + a dumb listing of the constraints, that can be dumped with -Zdump-mir=polonius -Zpolonius=next. Its current state is not intended to be a long-term thing, just for testing purposes -- I will replace its contents in the future with a different approach (an HTML+js file where we can more easily explore/filter/trace these constraints and loan reachability, have mermaid graphs of the usual graphviz dumps, etc).

I've started documenting the approach in this PR, I'll add more in the future. It's quite simple, and should be very clear when more constraints are introduced anyways.

r? @matthewjasper

Best reviewed per commit so that the datalog move is less bothersome to read, but if you'd prefer we separate that into a different PR, I can do that (and michael has offered to review these more mechanical changes if it'd help).

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Dec 13, 2024
@lqd
Copy link
Member Author

lqd commented Dec 13, 2024

This should all be gated away and deactivatd by default, but let's check: @bors try @rust-timer queue

@rust-timer

This comment was marked as resolved.

@rust-timer

This comment has been minimized.

@rustbot rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Dec 13, 2024
bors added a commit to rust-lang-ci/rust that referenced this pull request Dec 13, 2024
Foundations of location-sensitive polonius

I'd to land the prototype I'm describing in the [polonius project goal](rust-lang/rust-project-goals#118). It still is incomplete and naive and terrible but it's working "well enough" to consider landing.

I'd also like to make review easier by not opening a huge PR, but have a couple small-ish ones (the +/- line change summary of this PR looks big, but >80% is moving datalog to a single place).

This PR starts laying the foundation for that work:
- it refactors and collects 99% of the old datalog fact gen, which was spread around everywhere, into a single dedicated module. It's still present at 3 small places (one of which we should revert anyways) that are kinda deep within localized components that are not as easily extractable into the rest of fact gen, so it's fine for now.
- starts introducing the localized constraints, the building blocks of the naive way of implementing the location-sensitive analysis in-tree, which is roughly sketched out in https://smallcultfollowing.com/babysteps/blog/2023/09/22/polonius-part-1/ and https://smallcultfollowing.com/babysteps/blog/2023/09/29/polonius-part-2/ but with a different vibe than per-point environments, just `r1@p: r2@q` constraints.
- sets up the skeleton of generating these localized constraints: converting NLL typeck constraints, and creating liveness constraints
- introduces the polonius dual to NLL MIR to help development and debugging. It doesn't do much original currently but is a way to see these localized constraints: its an NLL MIR dump + a dumb listing of the constraints. It's not intended to be a long-term thing, it's for testing purposes, and I will replace its contents in the future with a different approach (an HTML where we can more easily explore these constraints, have mermaid graphs of the usual graphviz dumps, etc)

I've started documenting the approach in this PR, I'll add more in the future. It's quite simple, and should be very clear when more constraints are introduced anyways.

r? `@matthewjasper`

Best reviewed per commit so that the datalog move is less bothersome to read, but if you'd prefer we separate that into a different PR, I can do that.
@bors
Copy link
Contributor

bors commented Dec 13, 2024

⌛ Trying commit 1331ccb with merge 0b8e535...

@bors
Copy link
Contributor

bors commented Dec 13, 2024

☀️ Try build successful - checks-actions
Build commit: 0b8e535 (0b8e535a30719ca2ebdd3c9ebb1ea3f99530bd26)

@rust-timer

This comment has been minimized.

@rust-timer
Copy link
Collaborator

Finished benchmarking commit (0b8e535): comparison URL.

Overall result: no relevant changes - no action needed

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

@bors rollup=never
@rustbot label: -S-waiting-on-perf -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

Results (primary 4.6%, secondary 2.7%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

mean range count
Regressions ❌
(primary)
4.6% [4.6%, 4.6%] 1
Regressions ❌
(secondary)
2.7% [2.7%, 2.7%] 1
Improvements ✅
(primary)
- - 0
Improvements ✅
(secondary)
- - 0
All ❌✅ (primary) 4.6% [4.6%, 4.6%] 1

Cycles

This benchmark run did not return any relevant results for this metric.

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 770.238s -> 768.851s (-0.18%)
Artifact size: 330.40 MiB -> 330.38 MiB (-0.00%)

@rustbot rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Dec 13, 2024
@jackh726
Copy link
Member

Started the review this, but need to step away. Thoughts:

First, do you have a branch with all the changes you've made so far? Could be good to link it if so for context.

Second, this is still a "big" PR, definitely worth splitting into a couple, I think.

That being said, reviewed the first 2 commits and r+ for me on them (if you want to split them out). I'll keep going through the commits as I get time.

@lqd
Copy link
Member Author

lqd commented Dec 14, 2024

First, do you have a branch with all the changes you've made so far? Could be good to link it if so for context.

Yes and no, I have a messy older incomplete one and a bunch of local changes because I'm still working on it, so I'm basically cleaning them up as I go to open PRs.

Second, this is still a "big" PR, definitely worth splitting into a couple, I think.

Ok. I'll split the first 2 commits out now, it should simplify this PR a bit. I can split the rest of the old polonius' fact gen later if you or michael want to look at it.

(I also rebased to fix conflicts)

Zalathar added a commit to Zalathar/rust that referenced this pull request Dec 15, 2024
…kh726

A couple of polonius fact generation cleanups

This PR is extracted from rust-lang#134268 for easier review and contains its first two commits. They have already been reviewed by `@jackh726.`

r? `@jackh726`
rust-timer added a commit to rust-lang-ci/rust that referenced this pull request Dec 15, 2024
Rollup merge of rust-lang#134315 - lqd:polonius-next-episode-1, r=jackh726

A couple of polonius fact generation cleanups

This PR is extracted from rust-lang#134268 for easier review and contains its first two commits. They have already been reviewed by `@jackh726.`

r? `@jackh726`
Copy link
Member

@jackh726 jackh726 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have reviewed first 8 commits (through "improve consistency..."). A small nit, but otherwise r=me if you want to split them out.

compiler/rustc_borrowck/src/polonius/legacy/mod.rs Outdated Show resolved Hide resolved
Copy link
Member

@jackh726 jackh726 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursory review of commits 9 onward.

Overall they look fine, but didn't think too deeply so not a full approval.

compiler/rustc_borrowck/src/polonius/mod.rs Show resolved Hide resolved
compiler/rustc_borrowck/src/nll.rs Outdated Show resolved Hide resolved
compiler/rustc_borrowck/src/polonius/mod.rs Outdated Show resolved Hide resolved
compiler/rustc_borrowck/src/polonius/mod.rs Outdated Show resolved Hide resolved
compiler/rustc_borrowck/src/polonius/mod.rs Outdated Show resolved Hide resolved
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request Dec 16, 2024
…kh726

An octuple of polonius fact generation cleanups

This PR is extracted from rust-lang#134268 for easier review and contains its first 8 commits. They have already been reviewed by `@jackh726` over there.

r? `@jackh726`
@jackh726
Copy link
Member

I will come back to this for approval of the last few commits once the PR with the first 8 commits lands.

jieyouxu added a commit to jieyouxu/rust that referenced this pull request Dec 17, 2024
…kh726

An octuple of polonius fact generation cleanups

This PR is extracted from rust-lang#134268 for easier review and contains its first 8 commits. They have already been reviewed by ``@jackh726`` over there.

r? ``@jackh726``
jieyouxu added a commit to jieyouxu/rust that referenced this pull request Dec 17, 2024
…kh726

An octuple of polonius fact generation cleanups

This PR is extracted from rust-lang#134268 for easier review and contains its first 8 commits. They have already been reviewed by ```@jackh726``` over there.

r? ```@jackh726```
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request Dec 17, 2024
…kh726

An octuple of polonius fact generation cleanups

This PR is extracted from rust-lang#134268 for easier review and contains its first 8 commits. They have already been reviewed by ````@jackh726```` over there.

r? ````@jackh726````
lqd added 2 commits December 18, 2024 07:33
these are the basic blocks of the naive polonius constraint graph
implementation.
this describes the rough algorithm using the localized constraint graph
lqd added 4 commits December 18, 2024 07:33
this will allow calling from polonius MIR
This is mostly for test purposes to show the localized constraints until
the MIR debugger is set up.
- move constraints to an Option
- check `-Zpolonius=next` only once
- rewrite fixme comments to make the actionable part clear
@lqd
Copy link
Member Author

lqd commented Dec 20, 2024

I rebased after #134378 landed.

@jackh726
Copy link
Member

r? jackh726

@bors r+

@bors
Copy link
Contributor

bors commented Dec 21, 2024

📌 Commit aeb3d10 has been approved by jackh726

It is now in the queue for this repository.

@rustbot rustbot assigned jackh726 and unassigned matthewjasper Dec 21, 2024
@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Dec 21, 2024
@bors
Copy link
Contributor

bors commented Dec 21, 2024

⌛ Testing commit aeb3d10 with merge 426d173...

@bors
Copy link
Contributor

bors commented Dec 22, 2024

☀️ Test successful - checks-actions
Approved by: jackh726
Pushing 426d173 to master...

@bors bors added the merged-by-bors This PR was explicitly merged by bors. label Dec 22, 2024
@bors bors merged commit 426d173 into rust-lang:master Dec 22, 2024
7 checks passed
@rustbot rustbot added this to the 1.85.0 milestone Dec 22, 2024
@rust-timer
Copy link
Collaborator

Finished benchmarking commit (426d173): comparison URL.

Overall result: ❌ regressions - no action needed

@rustbot label: -perf-regression

Instruction count

This is the most reliable metric that we have; it was used to determine the overall result at the top of this comment. However, even this metric can sometimes exhibit noise.

mean range count
Regressions ❌
(primary)
- - 0
Regressions ❌
(secondary)
0.2% [0.2%, 0.2%] 1
Improvements ✅
(primary)
- - 0
Improvements ✅
(secondary)
- - 0
All ❌✅ (primary) - - 0

Max RSS (memory usage)

Results (primary 2.2%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

mean range count
Regressions ❌
(primary)
2.2% [2.2%, 2.2%] 1
Regressions ❌
(secondary)
- - 0
Improvements ✅
(primary)
- - 0
Improvements ✅
(secondary)
- - 0
All ❌✅ (primary) 2.2% [2.2%, 2.2%] 1

Cycles

Results (primary -4.4%)

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

mean range count
Regressions ❌
(primary)
- - 0
Regressions ❌
(secondary)
- - 0
Improvements ✅
(primary)
-4.4% [-6.7%, -2.0%] 2
Improvements ✅
(secondary)
- - 0
All ❌✅ (primary) -4.4% [-6.7%, -2.0%] 2

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 760.428s -> 762.028s (0.21%)
Artifact size: 330.62 MiB -> 330.62 MiB (0.00%)

@lqd lqd deleted the polonius-next branch December 22, 2024 05:50
bors added a commit to rust-lang-ci/rust that referenced this pull request Jan 18, 2025
Encode constraints that hold at all points as logical edges in location-sensitive polonius

Currently, with the full setup in rust-lang#134980 (but is from rust-lang#134268), the polonius location-sensitive analysis converts `Locations::All` typeck constraints as edges at all points in the CFG. This was temporary.

There's a FIXME about that already, and this PR implements it: we now use the constraints that hold at all points during traversal instead of eagerly materializing them as physical edges.

Another easy one `@jackh726.`

This fixes the slowness that was happening on the big CFG from the `saturating-float-casts` test (because of its 12M materialized edges) without, AFAICT, simply moving this overhead to traversal: materializing the logical edges is done on-demand.

r? `@jackh726` (no rush either)
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request Jan 18, 2025
…kh726

Encode constraints that hold at all points as logical edges in location-sensitive polonius

Currently, with the full setup in rust-lang#134980 (but is from rust-lang#134268), the polonius location-sensitive analysis converts `Locations::All` typeck constraints as edges at all points in the CFG. This was temporary.

There's a FIXME about that already, and this PR implements it: we now use the constraints that hold at all points during traversal instead of eagerly materializing them as physical edges.

Another easy one `@jackh726.`

This fixes the slowness that was happening on the big CFG from the `saturating-float-casts` test (because of its 12M materialized edges) without, AFAICT, simply moving this overhead to traversal: materializing the logical edges is done on-demand.

r? `@jackh726` (no rush either)
rust-timer added a commit to rust-lang-ci/rust that referenced this pull request Jan 18, 2025
Rollup merge of rust-lang#135290 - lqd:polonius-next-episode-8, r=jackh726

Encode constraints that hold at all points as logical edges in location-sensitive polonius

Currently, with the full setup in rust-lang#134980 (but is from rust-lang#134268), the polonius location-sensitive analysis converts `Locations::All` typeck constraints as edges at all points in the CFG. This was temporary.

There's a FIXME about that already, and this PR implements it: we now use the constraints that hold at all points during traversal instead of eagerly materializing them as physical edges.

Another easy one `@jackh726.`

This fixes the slowness that was happening on the big CFG from the `saturating-float-casts` test (because of its 12M materialized edges) without, AFAICT, simply moving this overhead to traversal: materializing the logical edges is done on-demand.

r? `@jackh726` (no rush either)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
merged-by-bors This PR was explicitly merged by bors. S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants