Bring back read_dir and remove_dir_all support

[RalfJung]

The tests for read_dir were temporarily disabled since the implementation was switched to readdir64, which Miri does not implement. Implementing it will be a bit of work due to having to manage the memory where the dirent is stored.

Also, we currently do not have a test that runs remove_dir_all on an existing directory; we should add such a test -- this used to work but now also requires openat.

[RalfJung]

read_dir works again. :) Thanks to @tavianator!

However, remove_dir_all still fails. Test case diff:

diff --git a/tests/run-pass/fs.rs b/tests/run-pass/fs.rs
index 7f5553e2..67817e3e 100644
--- a/tests/run-pass/fs.rs
+++ b/tests/run-pass/fs.rs
@@ -395,6 +395,12 @@ fn test_directory() {
     remove_dir(&dir_path).unwrap();
     // Reading the metadata of a non-existent directory should fail with a "not found" error.
     assert_eq!(ErrorKind::NotFound, check_metadata(&[], &dir_path).unwrap_err().kind());
+
+    // To test remove_dir_all, re-create the directory with a file and a directory in it.
+    create_dir(&dir_path).unwrap();
+    drop(File::create(&path_1).unwrap());
+    create_dir(&path_2).unwrap();
+    remove_dir_all(&dir_path).unwrap();
 }
 
 fn test_dup_stdout_stderr() {

Miri error:

error: unsupported operation: can't call foreign function: openat
    --> /home/r/.rustup/toolchains/miri/lib/rustlib/src/rust/library/std/src/sys/unix/fs.rs:1546:13
     |
1546 | /             openat(
1547 | |                 parent_fd.unwrap_or(libc::AT_FDCWD),
1548 | |                 p.as_ptr(),
1549 | |                 libc::O_CLOEXEC | libc::O_RDONLY | libc::O_NOFOLLOW | libc::O_DIRECTORY,
1550 | |             )
     | |_____________^ can't call foreign function: openat

[RalfJung]

Is it even possible to implement this in a cross-platform way? std does not expose a way to call openat, I think. We would require more methods on ReadDir to do that. (And we will also need unlinkat and fdopendir.)

So we could only implement this on Linux (and possibly macOS) hosts. Or we could patch libstd to use the fallback implementation -- that's what already happens on macOS targets (since dynamically resolving openat returns null in Miri), we could make it use that codepath for all unixes in Miri.

[tavianator]

std should expose an openat() style method IMO. It's slightly complicated by the lack of a Win32 API for it, but there is an NT kernel API that would work.

Adding the fallback might be okay, but I worry that it might negate the race protection that was the point of all those changes. (I'm not sure about this, I'd have to study the implementation in depth.) Then again the race probably doesn't matter in miri, and openat() exists on all real Linux targets rust supports.

[RalfJung]

The fallback would be gated under cfg(miri). So yes code running under Miri would suffer from the race, but it would not affect anyone else. I surely hope nobody runs anything critical under Miri, let alone running Miri as root...

[bjorn3]

You could use cap-std I think for openat. This is what wasmtime internally uses to implement wasi.

[RalfJung]

Oh, nice! I am inclined to go with cfg(miri) for now but track that as a future option in case someone wants to adjust our existing FS support. cap-std also pulls in quite a few dependencies that would be added to the rustc workspace.

[tavianator]

cap-std doesn't support things like openat(fd, "..") (by design). There is the openat crate but it doesn't seem to support Windows.

[bjorn3]

You can do std::fs::File::from_raw_fd and then pass the resulting File to cap-std (or cap-primitives which is a dependency of cap-std), right?

[tavianator]

Right, it's the .. part that doesn't work. cap-std enforces that the opened paths are beneath the initial path.