Proper handling $crate Take 2

[edwin0cheng]

Similar to previous PR (#7133) , but improved the following things :

1. Instead of storing the whole ExpansionInfo, we store a similar but stripped version HygieneInfo.
2. Instread of storing the SyntaxNode (because every token we are interested are IDENT), we store the TextRange only.
3. Because of 2, we now can put it in Salsa.
4. And most important improvement: Instead of computing the whole frames every single time, we compute it recursively through salsa: (Such that in the best scenario, we only need to compute the first layer of frame)

        let def_site = db.hygiene_frame(info.def.file_id);
        let call_site = db.hygiene_frame(info.arg.file_id);

        HygieneFrame { expansion: Some(info), local_inner, krate, call_site, def_site }

The overall speed compared to previous PR is much faster (65s vs 45s) :

[WITH old PR]
Database loaded 644.86ms, 284mi
Crates in this dir: 36
Total modules found: 576
Total declarations: 11153
Total functions: 8715
Item Collection: 15.78s, 91562mi
Total expressions: 240721
Expressions of unknown type: 2635 (1%)
Expressions of partially unknown type: 2064 (0%)
Type mismatches: 865
Inference: 49.84s, 250747mi
Total: 65.62s, 342310mi
rust-analyzer -q analysis-stats .  66.72s user 0.57s system 99% cpu 1:07.40 total

[WITH this PR]
Database loaded 665.83ms, 284mi
Crates in this dir: 36
Total modules found: 577
Total declarations: 11188
Total functions: 8743
Item Collection: 15.28s, 84919mi
Total expressions: 241229
Expressions of unknown type: 2637 (1%)
Expressions of partially unknown type: 2064 (0%)
Type mismatches: 868
Inference: 30.15s, 135293mi
Total: 45.43s, 220213mi   
rust-analyzer -q analysis-stats .  46.26s user 0.74s system 99% cpu 47.294 total

HOWEVER, it is still a perf regression (35s vs 45s):

[WITHOUT this PR]
Database loaded 657.42ms, 284mi
Crates in this dir: 36
Total modules found: 577
Total declarations: 11177
Total functions: 8735
Item Collection: 12.87s, 72407mi
Total expressions: 239380
Expressions of unknown type: 2643 (1%)
Expressions of partially unknown type: 2064 (0%)
Type mismatches: 868
Inference: 22.88s, 97889mi
Total: 35.74s, 170297mi
rust-analyzer -q analysis-stats .  36.71s user 0.63s system 99% cpu 37.498 total

[matklad]

Couple of quick observations before review:

• during expansion, there are a lot of contiguous token ranges. We should take advantage of that, and store ranges, and not individual tokens in TokenMap
• I am unsure it we should store TextRanges in frames. It seems that, after origenal conversion of user-written syntax to mbe, we only need mappings between token ids

[matklad]

Looks reasonable!

Whats the impact on memory use? per_query_memory_usage needs to account for new query.

cc @jonas-schievink

[edwin0cheng]

per_query_memory_usage needs to account for new query.

I added HygieneFrameQuery in per_query_memory_usage .

during expansion, there are a lot of contiguous token ranges. We should take advantage of that, and store ranges, and not individual tokens in TokenMap

IIUC, we only store token range in TokenMap right now.

I am unsure it we should store TextRanges in frames. It seems that, after origenal conversion of user-written syntax to mbe, we only need mappings between token ids

We store the TextRange for the def site and call site syntax node, it is needed because at the current frame, we can't determine the upper level token id without knoning the text range of upper level syntax text starting position.

However, I changed it to just store the starting position (which is TextSize), it should save some memory.

[edwin0cheng]

Memory used : (rust-analyzer analysis-stats -q --memory-usage .)

Database loaded 643.90ms, 284mi, 81mb
Crates in this dir: 36
Total modules found: 577
Total declarations: 11187
Total functions: 8742
Item Collection: 14.57s, 84873mi, 458mb
Total expressions: 241372
Expressions of unknown type: 2643 (1%)
Expressions of partially unknown type: 2076 (0%)
Type mismatches: 871
Inference: 28.91s, 135567mi, 633mb
Total: 43.48s, 220440mi, 1091mb
   243mb ItemTreeQuery
    76mb BodyQuery
    71mb HygieneFrameQuery  <-------------------------------
    57mb InferQueryQuery
    48mb TraitImplsInDepsQuery
    46mb TraitSolveQuery (purge)
    41mb CrateDefMapQueryQuery
    38mb ImplDatumQuery
    38mb FileTextQuery (purge)
    33mb MacroArgTextQuery
    28mb BodyWithSourceMapQuery
    26mb ImplDataQuery
...

[matklad]

yeah, that's quite high there. But we can't really not do this, so I think it's better to merge and optimize

[edwin0cheng]

Then bite the bullet now :)

bors r+

[edwin0cheng]

cc @jonas-schievink

[bors]

Build succeeded:

• Rust (macos-latest)
• Rust (ubuntu-latest)
• Rust (windows-latest)
• TypeScript (ubuntu-latest)
• TypeScript (windows-latest)