introduce file system watching features to the zig build system #20580
Commits on Jul 12, 2024
-
-
-
build system: add --watch flag and report source file in InstallFile
This direction is not quite right because it mutates shared state in a threaded context, so the next commit will need to fix this.
-
std.os.linux: fanotify_init, fanotify_mark, name_to_handle_at
* Delete existing `FAN` struct in favor of a `fanotify` struct which has type-safe bindings (breaking). * Add name_to_handle_at syscall wrapper. * Add file_handle * Add kernel_fsid_t * Add fsid_t * Add and update std.posix wrappers.
-
std.Build.Cache.Path: add
subPathOptandTableAdapterHelpful methods when using one of these structs as a hash table key.
-
build runner: implement --watch (work-in-progress)
I'm still learning how the fanotify API works but I think after playing with it in this commit, I finally know how to implement it, at least on Linux. This commit does not accomplish the goal but I want to take the code in a different direction and still be able to reference this point in time by viewing a source control diff. I think the move is going to be saving the file_handle for the parent directory, which combined with the dirent names is how we can correlate the events back to the Step instances that have registered file system inputs. I predict this to be similar to implementations on other operating systems.
-
-
proof-of-concept --watch implementation based on fanotify
So far, only implemented for InstallFile steps. Default debounce interval bumped to 50ms. I think it should be configurable. Next I have an idea to simplify the fanotify implementation, but other OS implementations might want to refer back to this commit before I make those changes.
-
-
std.Build.Watch: make dirty steps invalidate each other
and make failed steps always be invalidated and make steps that don't need to be reevaluated marked as cached
-
build runner: fix build summary painting over CLI progress
by obtaining the stderr lock when printing the build summary
-
-
-
std.Build.Step.WriteFile: extract UpdateSourceFiles
This has been planned for quite some time; this commit finally does it. Also implements file system watching integration in the make() implementation for UpdateSourceFiles and fixes the reporting of step caching for both. WriteFile does not yet have file system watching integration.
-
std.Build.LazyPath: add getPath3; deprecate getPath2 and getPath
The goal is to move towards using `std.Build.Cache.Path` instead of absolute path names. This was helpful for implementing file watching integration to the InstallDir Step
-
std.Build.Watch: introduce special file "." to watch entire dir
And use it to implement InstallDir Step watch integration. I'm not seeing any events triggered when I run `mkdir` in the watched directory, however, and I have not yet figured out why.
-
build runner: don't pass a dirfd + null to fanotify_mark
Otherwise it reports EBADF.
-
build runner: ignore ENOENT of fanotify_mark REMOVE
This happens when deleting watched directories and is harmless.
-
std.Build.Watch: add ONDIR to fanotify event mask
This makes mkdir/rmdir events show up.
-
td.Build.Step.InstallDir: leave hint for wrong cached status
Since I spent a couple minutes debugging this, hopefully this saves someone some future trouble doing the same.
-
introduce std.Build.Cache.Manifest.addFilePath
and deprecate `addFile`. Part of an effort to move towards using `std.Build.Cache.Path` abstraction in more places, which makes it easier to avoid absolute paths and path resolution.
-
std.Build.Step.WriteFile: fix handling of directories
and add file system watching integration. `addDirectoryWatchInput` now returns a `bool` which helps remind the caller to 1. call addDirectoryWatchInputFromPath on any derived paths 2. but only if the dependency is not already captured by a step dependency edge. The make function now recursively walks all directories and adds the found files to the cache hash rather than incorrectly only adding the directory name to the cache hash. closes #20571
-
std.Build.Step.WriteFile: remove random bytes from cache hash
The cache hash already has the zig version in there, so it's not really needed.
-
-
-
-
std.Build.Step.ObjCopy: remove random bytes from cache hash
The cache hash already has the zig version in there, so it's not really needed.
-
std.Build.Cache.Path: fix the format method
This function previously wrote a trailing directory separator, but that's not correct if the path refers to a file.
-
integrate Compile steps with file watching
Updates the build runner to unconditionally require a zig lib directory parameter. This parameter is needed in order to correctly understand file system inputs from zig compiler subprocesses, since they will refer to "the zig lib directory", and the build runner needs to place file system watches on directories in there. The build runner's fanotify file watching implementation now accounts for when two or more Cache.Path instances compare unequal but ultimately refer to the same directory in the file system. Breaking change: std.Build no longer has a zig_lib_dir field. Instead, there is the Graph zig_lib_directory field, and individual Compile steps can still have their zig lib directories overridden. I think this is unlikely to break anyone's build in practice. The compiler now sends a "file_system_inputs" message to the build runner which shares the full set of files that were added to the cache system with the build system, so that the build runner can watch properly and redo the Compile step. This is implemented for whole cache mode but not yet for incremental cache mode.
-
frontend: add file system inputs for incremental cache mode
These are also used for whole cache mode in the case that any compile errors are emitted.
-
-
need to add another field to initialize now
-
zig build: add a --debug-target CLI flag
it's not advertised in the usage and only available in debug builds of the compiler. Makes it easier to test changes to the build runner that might affect targets differently.
-
build runner: refactor fs watch logic for OS abstraction
Makes the build runner compile successfully for non-linux targets; printing an error if you ask for --watch rather than making build scripts fail to compile.
