ICE: "no saved object file in work product" - potential incremental issue #123376
Comments
jamesmunns
added
C-bug
rustbot
added
the
needs-triage
|
Maybe a dupe of #109879 - tho this is on Linux instead of Windows, and at a different worker stage (not LTO). |
jieyouxu
added
the
S-needs-repro
|
I did have this happen again, though it is very rare. One potential note is that it seems to occur when I am using It's possible what is happening is:
Does incremental compilation have any notion of "transactions"? e.g. what if the compilation is killed? |
|
In the run BEFORE the error occurred, I did get this in the subsequent error: |
This is great info. Maybe I can craft another test harness that provokes this crash. |
To be clear: There is supposed to be a mechanism to prevent the scenario you're reporting here. We write all our output to a staging directory then |
|
Also potentially worth mentioning: I'm using a VM with a shared folder for this, with:
IMO it shouldn't be THAT different than having an IDE with R-A and running cargo-watch at the same time, but just wanted to note. |
Hmmmmmmmmmmmmmmmmmmmmmmm rust/compiler/rustc_incremental/src/persist/fs.rs Lines 94 to 104 in 4fd4797 |
saethlin
added
A-incr-comp
|
HMMMM indeed. For MORE detail on my setup:
I will try two things today (tho it might not proc/repro today) to see if I can retrigger the issue (it only happens once every day or two, but maybe I can try to trigger it intentionally):
In both of those cases I eliminate the "shared" folder, if I can get it to trigger in either of those cases it's probably not virtiofs' fault, if it is then it is. I'll also peek at some of the other issues to see if they are also potentially using virtualization or Weird Filesystems |
|
So, I wrote this crude stress testing script, sharing in case it is useful: #!/bin/bash
set -euxo pipefail
for i in {1..1000}
do
stime=`bc <<< "scale=2; $i/100"`
echo "// hehe" >> ./toml.rs
sleep $stime
doneI was able to get it to repro once (out of a couple of runs) using the VM + Host setup where it repro'd before, and it acted weird one other time (errors about "can't copy, folder doesn't exist", but no ICE). I wasn't able to get it to repro at all in the other setups (only linux on btrfs, only macos on apfs). I'll try it again a couple times, it doesn't proc reliably, so there's still some chance the timing is just different in the other configs. Still definitely could be virtiofs not handling file locking or hard linking correctly. Let me know if there's anything I can do to help with my occasionally reproing setup. Edit, here's the "weird" errors I mention in the not-quite-repro case above: |
|
I slapped together a little flock stress tester using the Can you run it in your virtiofs? |
|
@saethlin is there a specific way you want me to run it?
Trying either of these it doesn't seem to crash |
|
You did everything I would have suggested. I suppose exposing a bug with such a simple test would have been too easy. I have access to a Mac, I'll see if I can recreate your setup. |
Add a footer in FileEncoder and check for it in MemDecoder We have a few reports of ICEs due to decoding failures, where the fault does not lie with the compiler. The goal of this PR is to add some very lightweight and on-by-default validation to the compiler's outputs. If validation fails, we emit a fatal error for rmeta files in general that mentions the path that didn't load, and for incremental compilation artifacts we emit a verbose warning that tries to explain the situation and treat the artifacts as outdated. The validation currently implemented here is very crude, and yet I think we have 11 ICE reports currently open (you can find them by searching issues for `1002111927320821928687967599834759150` which this simple validation would have detected. The structure of the code changes here should permit the addition of further validation code, such as a checksum, if it is merited. I would like to have code to detect corruption such as reported in rust-lang#124719, but I'm not yet sure how to do that efficiently, and this PR is already a good size. The ICE reports I have in mind that this PR would have smoothed over are: rust-lang#124469 rust-lang#123352 rust-lang#123376 [^1] rust-lang#99763 rust-lang#93900. --- [^1]: This one might be a compiler bug, but even if it is I think the workflow described is pushing the envelope of what we can support
Add a footer in FileEncoder and check for it in MemDecoder We have a few reports of ICEs due to decoding failures, where the fault does not lie with the compiler. The goal of this PR is to add some very lightweight and on-by-default validation to the compiler's outputs. If validation fails, we emit a fatal error for rmeta files in general that mentions the path that didn't load, and for incremental compilation artifacts we emit a verbose warning that tries to explain the situation and treat the artifacts as outdated. The validation currently implemented here is very crude, and yet I think we have 11 ICE reports currently open (you can find them by searching issues for `1002111927320821928687967599834759150`) which this simple validation would have detected. The structure of the code changes here should permit the addition of further validation code, such as a checksum, if it is merited. I would like to have code to detect corruption such as reported in rust-lang#124719, but I'm not yet sure how to do that efficiently, and this PR is already a good size. The ICE reports I have in mind that this PR would have smoothed over are: rust-lang#124469 rust-lang#123352 rust-lang#123376 [^1] rust-lang#99763 rust-lang#93900. --- [^1]: This one might be a compiler bug, but even if it is I think the workflow described is pushing the envelope of what we can support. This issue is one of the reasons this warning still asks people to file an issue.
Add a footer in FileEncoder and check for it in MemDecoder We have a few reports of ICEs due to decoding failures, where the fault does not lie with the compiler. The goal of this PR is to add some very lightweight and on-by-default validation to the compiler's outputs. If validation fails, we emit a fatal error for rmeta files in general that mentions the path that didn't load, and for incremental compilation artifacts we emit a verbose warning that tries to explain the situation and treat the artifacts as outdated. The validation currently implemented here is very crude, and yet I think we have 11 ICE reports currently open (you can find them by searching issues for `1002111927320821928687967599834759150`) which this simple validation would have detected. The structure of the code changes here should permit the addition of further validation code, such as a checksum, if it is merited. I would like to have code to detect corruption such as reported in rust-lang/rust#124719, but I'm not yet sure how to do that efficiently, and this PR is already a good size. The ICE reports I have in mind that this PR would have smoothed over are: rust-lang/rust#124469 rust-lang/rust#123352 rust-lang/rust#123376 [^1] rust-lang/rust#99763 rust-lang/rust#93900. --- [^1]: This one might be a compiler bug, but even if it is I think the workflow described is pushing the envelope of what we can support. This issue is one of the reasons this warning still asks people to file an issue.
Add a footer in FileEncoder and check for it in MemDecoder We have a few reports of ICEs due to decoding failures, where the fault does not lie with the compiler. The goal of this PR is to add some very lightweight and on-by-default validation to the compiler's outputs. If validation fails, we emit a fatal error for rmeta files in general that mentions the path that didn't load, and for incremental compilation artifacts we emit a verbose warning that tries to explain the situation and treat the artifacts as outdated. The validation currently implemented here is very crude, and yet I think we have 11 ICE reports currently open (you can find them by searching issues for `1002111927320821928687967599834759150`) which this simple validation would have detected. The structure of the code changes here should permit the addition of further validation code, such as a checksum, if it is merited. I would like to have code to detect corruption such as reported in rust-lang/rust#124719, but I'm not yet sure how to do that efficiently, and this PR is already a good size. The ICE reports I have in mind that this PR would have smoothed over are: rust-lang/rust#124469 rust-lang/rust#123352 rust-lang/rust#123376 [^1] rust-lang/rust#99763 rust-lang/rust#93900. --- [^1]: This one might be a compiler bug, but even if it is I think the workflow described is pushing the envelope of what we can support. This issue is one of the reasons this warning still asks people to file an issue.
Code
This commit: memorysafety/river@5ff346d
Meta
This didn't seem to reproduce after a
cargo clean, so I can't provide too much more wrt what caused this to happen.So far I've only seen this once.
This was running in an aarch64 fedora VM running on an M2 Mac inside of UTM.
Error output
The text was updated successfully, but these errors were encountered: