Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make FileMap::{lines, multibyte_chars, non_narrow_chars} non-mutable. #50997

Merged
merged 8 commits into from
Jun 28, 2018

Conversation

michaelwoerister
Copy link
Member

This PR removes most of the interior mutability from FileMap, which should be beneficial, especially in a multithreaded setting. This is achieved by initializing the state in question when the filemap is constructed instead of during lexing. Hopefully this doesn't degrade performance.

cc @wesleywiser

@rust-highfive
Copy link
Collaborator

r? @aturon

(rust_highfive has picked a reviewer for you, use r? to override)

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label May 23, 2018
@michaelwoerister
Copy link
Member Author

@bors try

@bors
Copy link
Contributor

bors commented May 23, 2018

⌛ Trying commit fbd1750 with merge 28e2ad9...

bors added a commit that referenced this pull request May 23, 2018
 Make FileMap::{lines, multibyte_chars, non_narrow_chars} non-mutable.

This PR removes most of the interior mutability from `FileMap`, which should be beneficial, especially in a multithreaded setting. This is achieved by initializing the state in question when the filemap is constructed instead of during lexing. Hopefully this doesn't degrade performance.

cc @wesleywiser
@wesleywiser
Copy link
Member

Nice job @michaelwoerister! It didn't even occur to me to try this kind of approach. IMO, this code is much easier to follow than the previous version since all of the special character handling is consolidated into one place. 👍

@michaelwoerister
Copy link
Member Author

Yes, let's hope performance doesn't throw a wrench into the works :) But it might even be faster, since less locking is going on. In the parallel case it should definitely be faster.

@michaelwoerister
Copy link
Member Author

Hm, UI tests also seem to have some trouble...

@bors
Copy link
Contributor

bors commented May 23, 2018

☀️ Test successful - status-travis
State: approved= try=True

@michaelwoerister
Copy link
Member Author

@Mark-Simulacrum, could I get a perf-run, please?

@michaelwoerister michaelwoerister added S-waiting-on-perf Status: Waiting on a perf run to be completed. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels May 23, 2018
@@ -460,18 +454,6 @@ impl<'a> StringReader<'a> {
let next_ch = char_at(&self.src, next_src_index);
let next_ch_len = next_ch.len_utf8();

if self.ch.unwrap() == '\n' {
if self.save_new_lines_and_multibyte {
self.filemap.next_line(self.next_pos);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This checks if the current character is a newline, but then it passes next_pos into next_line

if byte.is_ascii() {
match byte {
b'\n' => {
lines.push(byte_pos);
Copy link
Member

@wesleywiser wesleywiser May 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This checks if the current character is a newline and then records the current position. Is this where the off-by one on the failing UI tests is coming from?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think that's the problem. Good catch!

@michaelwoerister
Copy link
Member Author

Should be fixed now. The perf run should still be valid, despite the change.

@Mark-Simulacrum
Copy link
Member

Perf queued!

@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-3.9 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
[01:23:58] travis_fold:start:test_stage1-syntax
travis_time:start:test_stage1-syntax
Testing syntax stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:23:58]    Compiling syntax v0.0.0 (file:///checkout/src/libsyntax)
[01:24:04] error[E0599]: no method named `next_line` found for type `std::rc::Rc<syntax_pos::FileMap>` in the current scope
[01:24:04]    --> libsyntax/codemap.rs:951:12
[01:24:04]     |
[01:24:04] 951 |         fm.next_line(BytePos(0));
[01:24:04]     |
[01:24:04]     = help: did you mean `get_line`?
[01:24:04] 
[01:24:04] 
[01:24:04] error[E0599]: no method named `next_line` found for type `std::rc::Rc<syntax_pos::FileMap>` in the current scope
[01:24:04]    --> libsyntax/codemap.rs:955:12
[01:24:04]     |
[01:24:04] 955 |         fm.next_line(BytePos(10));
[01:24:04]     |
[01:24:04]     = help: did you mean `get_line`?
[01:24:04] 
[01:24:04] 
[01:24:04] error[E0599]: no method named `next_line` found for type `std::rc::Rc<syntax_pos::FileMap>` in the current scope
[01:24:04]    --> libsyntax/codemap.rs:957:12
[01:24:04]     |
[01:24:04] 957 |         fm.next_line(BytePos(12));
[01:24:04]     |
[01:24:04]     = help: did you mean `get_line`?
[01:24:04] 
[01:24:04] 
[01:24:04] error[E0599]: no method named `next_line` found for type `std::rc::Rc<syntax_pos::FileMap>` in the current scope
[01:24:04]    --> libsyntax/codemap.rs:968:12
[01:24:04]     |
[01:24:04] 968 |         fm.next_line(BytePos(0));
[01:24:04]     |
[01:24:04]     = help: did you mean `get_line`?
[01:24:04] 
[01:24:04] 
[01:24:04] error[E0599]: no method named `next_line` found for type `std::rc::Rc<syntax_pos::FileMap>` in the current scope
[01:24:04]    --> libsyntax/codemap.rs:969:12
[01:24:04]     |
[01:24:04] 969 |         fm.next_line(BytePos(10));
[01:24:04]     |
[01:24:04]     = help: did you mean `get_line`?
[01:24:04] 
[01:24:04] 
[01:24:04] error[E0599]: no method named `next_line` found for type `std::rc::Rc<syntax_pos::FileMap>` in the current scope
[01:24:04]    --> libsyntax/codemap.rs:970:12
[01:24:04]     |
[01:24:04] 970 |         fm.next_line(BytePos(2));
[01:24:04]     |
[01:24:04]     = help: did you mean `get_line`?
[01:24:04] 
[01:24:04] 
[01:24:04] error[E0599]: no method named `next_line` found for type `std::rc::Rc<syntax_pos::FileMap>` in the current scope
[01:24:04]    --> libsyntax/codemap.rs:982:13
[01:24:04]     |
[01:24:04] 982 |         fm1.next_line(BytePos(0));
[01:24:04]     |
[01:24:04]     = help: did you mean `get_line`?
[01:24:04] 
[01:24:04] 
[01:24:04] error[E0599]: no method named `next_line` found for type `std::rc::Rc<syntax_pos::FileMap>` in the current scope
[01:24:04]    --> libsyntax/codemap.rs:983:13
[01:24:04]     |
[01:24:04] 983 |         fm1.next_line(BytePos(12));
[01:24:04]     |
[01:24:04]     = help: did you mean `get_line`?
[01:24:04] 
[01:24:04] 
[01:24:04] error[E0599]: no method named `next_line` found for type `std::rc::Rc<syntax_pos::FileMap>` in the current scope
[01:24:04]    --> libsyntax/codemap.rs:984:13
[01:24:04]     |
[01:24:04] 984 |         fm2.next_line(fm2.start_pos);
[01:24:04]     |
[01:24:04]     = help: did you mean `get_line`?
[01:24:04] 
[01:24:04] 
[01:24:04] error[E0599]: no method named `next_line` found for type `std::rc::Rc<syntax_pos::FileMap>` in the current scope
[01:24:04]    --> libsyntax/codemap.rs:985:13
[01:24:04]     |
[01:24:04] 985 |         fm3.next_line(fm3.start_pos);
[01:24:04]     |
[01:24:04]     = help: did you mean `get_line`?
[01:24:04] 
[01:24:04] 
[01:24:04] error[E0599]: no method named `next_line` found for type `std::rc::Rc<syntax_pos::FileMap>` in the current scope
[01:24:04]    --> libsyntax/codemap.rs:986:13
[01:24:04]     |
[01:24:04] 986 |         fm3.next_line(fm3.start_pos + BytePos(12));
[01:24:04]     |
[01:24:04]     = help: did you mean `get_line`?
[01:24:04] 
[01:24:04] 
[01:24:04] error[E0599]: no method named `next_line` found for type `std::rc::Rc<syntax_pos::FileMap>` in the current scope
[01:24:04]     --> libsyntax/codemap.rs:1046:13
[01:24:04]      |
[01:24:04] 1046 |         fm1.next_line(BytePos(0));
[01:24:04]      |
[01:24:04]      = help: did you mean `get_line`?
[01:24:04] 
[01:24:04] 
[01:24:04] error[E0599]: no method named `next_line` found for type `std::rc::Rc<syntax_pos::FileMap>` in the current scope
[01:24:04]     --> libsyntax/codemap.rs:1047:13
[01:24:04]      |
[01:24:04] 1047 |         fm1.next_line(BytePos(28));
[01:24:04]      |
[01:24:04]      = help: did you mean `get_line`?
[01:24:04] 
[01:24:04] 
[01:24:04] error[E0599]: no method named `next_line` found for type `std::rc::Rc<syntax_pos::FileMap>` in the current scope
[01:24:04]     --> libsyntax/codemap.rs:1048:13
[01:24:04]      |
[01:24:04] 1048 |         fm2.next_line(fm2.start_pos);
[01:24:04]      |
[01:24:04]      = help: did you mean `get_line`?
[01:24:04] 
[01:24:04] 
[01:24:04] error[E0599]: no method named `next_line` found for type `std::rc::Rc<syntax_pos::FileMap>` in the current scope
[01:24:04]     --> libsyntax/codemap.rs:1049:13
[01:24:04]      |
[01:24:04] 1049 |         fm2.next_line(fm2.start_pos + BytePos(20));
[01:24:04]      |
[01:24:04]      = help: did you mean `get_line`?
[01:24:04] 
[01:24:04] 
[01:24:04] error[E0599]: no method named `record_multibyte_char` found for type `std::rc::Rc<syntax_pos::FileMap>` in the current scope
[01:24:04]     --> libsyntax/codemap.rs:1051:13
[01:24:04]      |
[01:24:04] 1051 |         fm1.record_multibyte_char(BytePos(3), 3);
[01:24:04] 
[01:24:04] 
[01:24:04] error[E0599]: no method named `record_multibyte_char` found for type `std::rc::Rc<syntax_pos::FileMap>` in the current scope
[01:24:04]     --> libsyntax/codemap.rs:1052:13
[01:24:04]      |
[01:24:04] 1052 |         fm1.record_multibyte_char(BytePos(9), 3);
[01:24:04] 
[01:24:04] 
[01:24:04] error[E0599]: no method named `record_multibyte_char` found for type `std::rc::Rc<syntax_pos::FileMap>` in the current scope
[01:24:04]     --> libsyntax/codemap.rs:1053:13
[01:24:04]      |
[01:24:04] 1053 |         fm1.record_multibyte_char(BytePos(12), 3);
[01:24:04] 
[01:24:04] 
[01:24:04] error[E0599]: no method named `record_multibyte_char` found for type `std::rc::Rc<syntax_pos::FileMap>` in the current scope
[01:24:04]     --> libsyntax/codemap.rs:1054:13
[01:24:04]      |
[01:24:04] 1054 |         fm1.record_multibyte_char(BytePos(15), 3);
[01:24:04] 
[01:24:04] 
[01:24:04] error[E0599]: no method named `record_multibyte_char` found for type `std::rc::Rc<syntax_pos::FileMap>` in the current scope
[01:24:04]     --> libsyntax/codemap.rs:1055:13
[01:24:04]      |
[01:24:04] 1055 |         fm1.record_multibyte_char(BytePos(18), 3);
[01:24:04] 
[01:24:04] 
[01:24:04] error[E0599]: no method named `record_multibyte_char` found for type `std::rc::Rc<syntax_pos::FileMap>` in the current scope
[01:24:04]     --> libsyntax/codemap.rs:1056:13
[01:24:04]      |
[01:24:04] 1056 |         fm2.record_multibyte_char(fm2.start_pos + BytePos(10), 3);
[01:24:04] 
[01:24:04] 
[01:24:04] error[E0599]: no method named `record_multibyte_char` found for type `std::rc::Rc<syntax_pos::FileMap>` in the current scope
[01:24:04]     --> libsyntax/codemap.rs:1057:13
[01:24:04]      |
[01:24:04] 1057 |         fm2.record_multibyte_char(fm2.start_pos + BytePos(13), 3);
[01:24:04] 
[01:24:04] 
[01:24:04] error[E0599]: no method named `record_multibyte_char` found for type `std::rc::Rc<syntax_pos::FileMap>` in the current scope
[01:24:04]     --> libsyntax/codemap.rs:1058:13
[01:24:04]      |
[01:24:04] 1058 |         fm2.record_multibyte_char(fm2.start_pos + BytePos(18), 3);
[01:24:04] 
[01:24:08] error: aborting due to 23 previous errors
[01:24:08] 
[01:24:08] For more information about this error, try `rustc --explain E0599`.
[01:24:08] For more information about this error, try `rustc --explain E0599`.
[01:24:08] error: Could not compile `syntax`.
[01:24:08] 
[01:24:08] To learn more, run the command again with --verbose.
[01:24:08] 
[01:24:08] 
[01:24:08] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "test" "--target" "x86_64-unknown-linux-gnu" "-j" "4" "--release" "--locked" "--color" "always" "--features" " jemalloc" "--manifest-path" "/checkout/src/rustc/Cargo.toml" "-p" "syntax" "--" "--quiet"
[01:24:08] 
[01:24:08] 
[01:24:08] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
[01:24:08] Build completed unsuccessfully in 0:39:18
[01:24:08] Build completed unsuccessfully in 0:39:18
[01:24:08] make: *** [check] Error 1
[01:24:08] Makefile:58: recipe for target 'check' failed

The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:1d70ef0a
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@michaelwoerister
Copy link
Member Author

Thanks, @Mark-Simulacrum!

@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-3.9 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
[00:42:57] .....................................................i..............................................
[00:43:01] .........................................................................ii.........................
[00:43:07] ....................................................................................................
[00:43:13] ...................................................................................i................
[00:43:15] .iiiiiiiii...................................................
[00:43:15] 
[00:43:15] travis_fold:start:test_ui_nll
travis_time:start:test_ui_nll
Check compiletest suite=ui mode=ui compare_mode=nll (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
---
[00:44:01] .....................................................i..............................................
[00:44:05] .........................................................................ii.........................
[00:44:10] ....................................................................................................
[00:44:16] ...................................................................................i................
[00:44:18] .iiiiiiiii...................................................
[00:44:18] 
[00:44:18]  finished in 62.980
[00:44:18] travis_fold:end:test_ui_nll

---
[01:17:20] 
[01:17:20] To learn more, run the command again with --verbose.
[01:17:20] 
[01:17:20] 
[01:17:20] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "test" "--target" "x86_64-unknown-linux-gnu" "-j" "4" "--release" "--locked" "--color" "always" "--features" " jemalloc" "--manifest-path" "/checkout/src/rustc/Cargo.toml" "-p" "syntax" "--" "--quiet"
[01:17:20] 
[01:17:20] 
[01:17:20] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
[01:17:20] Build completed unsuccessfully in 0:36:30
[01:17:20] Build completed unsuccessfully in 0:36:30
[01:17:20] Makefile:58: recipe for target 'check' failed
[01:17:20] make: *** [check] Error 1

The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:072c2d5a
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-3.9 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
[00:59:47] .....................................................i..............................................
[00:59:52] .........................................................................ii.........................
[01:00:00] ....................................................................................................
[01:00:07] ...................................................................................i................
[01:00:10] .iiiiiiiii...................................................
[01:00:10] 
[01:00:10] travis_fold:start:test_ui_nll
travis_time:start:test_ui_nll
Check compiletest suite=ui mode=ui compare_mode=nll (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
---
[01:01:07] .....................................................i..............................................
[01:01:13] .........................................................................ii.........................
[01:01:19] ....................................................................................................
[01:01:27] ...................................................................................i................
[01:01:29] .iiiiiiiii...................................................
[01:01:29] 
[01:01:29]  finished in 79.239
[01:01:29] travis_fold:end:test_ui_nll

---
[01:49:21] travis_fold:start:test_stage1-syntax
travis_time:start:test_stage1-syntax
Testing syntax stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:49:21]    Compiling syntax v0.0.0 (file:///checkout/src/libsyntax)
0m^^^ help: consider using `_fm3` instead
[01:49:37] error: unused variable: `fm1`
[01:49:37]     --> libsyntax/codemap.rs:1077:13
[01:49:37]      |
[01:49:37] 1077 |         let fm1 =
[01:49:37] 1077 |         let fm1 =
[01:49:37]      |             ^^^ help: consider using `_fm1` instead
[01:49:37] error: unused variable: `fm2`
[01:49:37]     --> libsyntax/codemap.rs:1080:13
[01:49:37]      |
[01:49:37]      |
[01:49:37] 1080 |         let fm2 = cm.new_filemap(PathBuf::from("blork2.rs").into(),
[01:49:37]      |             ^^^ help: consider using `_fm2` instead
[01:49:38] error: aborting due to 5 previous errors
[01:49:38] 
[01:49:38] error: Could not compile `syntax`.
[01:49:38] 
[01:49:38] 
[01:49:38] To learn more, run the command again with --verbose.
[01:49:38] 
[01:49:38] 
[01:49:38] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "test" "--target" "x86_64-unknown-linux-gnu" "-j" "4" "--release" "--locked" "--color" "always" "--features" " jemalloc" "--manifest-path" "/checkout/src/rustc/Cargo.toml" "-p" "syntax" "--" "--quiet"
[01:49:38] 
[01:49:38] 
[01:49:38] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
[01:49:38] Build completed unsuccessfully in 0:52:42
[01:49:38] Build completed unsuccessfully in 0:52:42
[01:49:38] make: *** [check] Error 1
[01:49:38] Makefile:58: recipe for target 'check' failed

The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:2fdadc87
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@michaelwoerister
Copy link
Member Author

I think something might have gone wrong with that previous perf run. Let's do another one.

@bors try

@bors
Copy link
Contributor

bors commented May 25, 2018

⌛ Trying commit 2874a87 with merge 23299ed...

bors added a commit that referenced this pull request May 25, 2018
 Make FileMap::{lines, multibyte_chars, non_narrow_chars} non-mutable.

This PR removes most of the interior mutability from `FileMap`, which should be beneficial, especially in a multithreaded setting. This is achieved by initializing the state in question when the filemap is constructed instead of during lexing. Hopefully this doesn't degrade performance.

cc @wesleywiser
@bors
Copy link
Contributor

bors commented May 25, 2018

☀️ Test successful - status-travis
State: approved= try=True

@michaelwoerister
Copy link
Member Author

@Mark-Simulacrum, can we give this another try?

@michaelwoerister
Copy link
Member Author

Rebased.

@bors r=Mark-Simulacrum

@bors
Copy link
Contributor

bors commented Jun 27, 2018

📌 Commit ba30c1d has been approved by Mark-Simulacrum

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jun 27, 2018
@bors
Copy link
Contributor

bors commented Jun 27, 2018

⌛ Testing commit ba30c1d with merge a9880eb97bcb8fac55e8b0b34aac9f42ee11a0f7...

@bors
Copy link
Contributor

bors commented Jun 27, 2018

💔 Test failed - status-appveyor

@bors bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Jun 27, 2018
@michaelwoerister
Copy link
Member Author

I can reproduce locally, will investigate.

@kennytm kennytm added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 28, 2018
The method relied on the FileMap still being under construction in
order for it to do what the name promises. It's now independent of
the current state.
@michaelwoerister
Copy link
Member Author

Fixed FileMap::line_begin_pos() which made the incorrect assumption that the "current" line is that last one.

@bors r=Mark-Simulacrum

@bors
Copy link
Contributor

bors commented Jun 28, 2018

📌 Commit a1f8a6c has been approved by Mark-Simulacrum

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jun 28, 2018
@bors
Copy link
Contributor

bors commented Jun 28, 2018

⌛ Testing commit a1f8a6c with merge 9f79d2f...

bors added a commit that referenced this pull request Jun 28, 2018
…Simulacrum

 Make FileMap::{lines, multibyte_chars, non_narrow_chars} non-mutable.

This PR removes most of the interior mutability from `FileMap`, which should be beneficial, especially in a multithreaded setting. This is achieved by initializing the state in question when the filemap is constructed instead of during lexing. Hopefully this doesn't degrade performance.

cc @wesleywiser
@bors
Copy link
Contributor

bors commented Jun 28, 2018

☀️ Test successful - status-appveyor, status-travis
Approved by: Mark-Simulacrum
Pushing 9f79d2f to master...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
Projects
None yet
Development

Successfully merging this pull request may close these issues.