Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

introduce file system watching features to the zig build system #20580

Merged
merged 38 commits into from
Jul 12, 2024
Merged

Conversation

andrewrk
Copy link
Member

@andrewrk andrewrk commented Jul 10, 2024

Feature Explanation

  --watch                      Continuously rebuild when source files are modified
  --debounce <ms>              Delay before rebuilding after changed file detected

Uses the build system's perfect knowledge of all file system inputs to the pipeline to keep the build runner alive after completion, watching the minimal number of directories in order to trigger re-running only the dirty steps from the graph.

Default debounce time is 50ms but this is configurable. It helps prevent wasted rebuilds when source files are changed in rapid succession, for example when saving with vim it does not do an atomic rename into place but actually deletes the destination file before writing it again, causing a brief period of invalid state, which would cause a build failure without debouncing (it would be followed by a successful build, but it's annoying to experience the temporary build failure regardless).

The purpose of this feature is to reduce latency between editing and debugging in the development cycle. In large projects, the cache system must call fstat on a large number of files even when it is a cache hit. File system watching allows more efficient detection of stale pipeline steps.

Mainly this is motivated by incremental compilation landing soon, so that we can keep the compiler running and responding to source code changes as fast as possible. In this case, also keeping the rest of the build pipeline up-to-date is table stakes.

This also paves the road towards #68. A Run step combined with --watch connects file system updates directly to new code inside an already-running executable. It takes steps closer to more advanced use cases as well:

  • IDE plugin running zig build as a child process, speaking a compiler protocol (Implement a server in the compiler that serves information about the compilation #615) to learn about type information, perform refactors, request rebuilds, receive errors, etc. The protocol will multiplex between an arbitrary number of Compile steps, each with a running instance of the compiler.
  • An application, compiled in debug mode, speaking the build runner protocol, able to poll on a file descriptor and learn when a hot swap is available, requesting it when it is convenient, or learning about new installed asset updates occurring.

Run step asciinema demo This demo only shows 1/2 terminals used, but in the other window I'm editing assembly files with vim and saving them.

Compile step asciinema demo - getting quick compile error feedback. Incremental compilation is not done yet, this is full compiler rebuilds, but skipping codegen.

unit test workflow asciinema demo

Migration Guide

If you were using WriteFile for its ability to update source files, that functionality has been extracted into a separate step called UpdateSourceFiles. Everything else is the same, so migration looks like this:

-    const copy_zig_h = b.addWriteFiles();
+    const copy_zig_h = b.addUpdateSourceFiles();

If you were using a RemoveDir step, it now takes a LazyPath instead of []const u8. Probably your migration looks like this:

-        const cleanup = b.addRemoveDirTree(tmp_path);
+        const cleanup = b.addRemoveDirTree(.{ .cwd_relative = tmp_path });

However, please consider not choosing a temporary path at configure time as this is somewhat brittle when it comes to running your build pipeline concurrently.

Follow-Up Work

@andrewrk andrewrk added zig build system std.Build, the build runner, `zig build` subcommand, package management release notes This PR should be mentioned in the release notes. breaking Implementing this issue could cause existing code to no longer compile or have different behavior. labels Jul 10, 2024
This direction is not quite right because it mutates shared state in a
threaded context, so the next commit will need to fix this.
* Delete existing `FAN` struct in favor of a `fanotify` struct which has
  type-safe bindings (breaking).
* Add name_to_handle_at syscall wrapper.
* Add file_handle
* Add kernel_fsid_t
* Add fsid_t
* Add and update std.posix wrappers.
Helpful methods when using one of these structs as a hash table key.
I'm still learning how the fanotify API works but I think after playing
with it in this commit, I finally know how to implement it, at least on
Linux. This commit does not accomplish the goal but I want to take the
code in a different direction and still be able to reference this point
in time by viewing a source control diff.

I think the move is going to be saving the file_handle for the parent
directory, which combined with the dirent names is how we can correlate
the events back to the Step instances that have registered file system
inputs. I predict this to be similar to implementations on other
operating systems.
So far, only implemented for InstallFile steps.

Default debounce interval bumped to 50ms. I think it should be
configurable.

Next I have an idea to simplify the fanotify implementation, but other
OS implementations might want to refer back to this commit before I make
those changes.
and make failed steps always be invalidated
and make steps that don't need to be reevaluated marked as cached
by obtaining the stderr lock when printing the build summary
This has been planned for quite some time; this commit finally does it.

Also implements file system watching integration in the make()
implementation for UpdateSourceFiles and fixes the reporting of step
caching for both.

WriteFile does not yet have file system watching integration.
The goal is to move towards using `std.Build.Cache.Path` instead of
absolute path names.

This was helpful for implementing file watching integration to
the InstallDir Step
And use it to implement InstallDir Step watch integration.

I'm not seeing any events triggered when I run `mkdir` in the watched
directory, however, and I have not yet figured out why.
This happens when deleting watched directories and is harmless.
This makes mkdir/rmdir events show up.
Since I spent a couple minutes debugging this, hopefully this saves
someone some future trouble doing the same.
and deprecate `addFile`. Part of an effort to move towards using
`std.Build.Cache.Path` abstraction in more places, which makes it easier
to avoid absolute paths and path resolution.
and add file system watching integration.

`addDirectoryWatchInput` now returns a `bool` which helps remind the
caller to
1. call addDirectoryWatchInputFromPath on any derived paths
2. but only if the dependency is not already captured by a step
   dependency edge.

The make function now recursively walks all directories and adds the
found files to the cache hash rather than incorrectly only adding the
directory name to the cache hash.

closes #20571
The cache hash already has the zig version in there, so it's not really
needed.
The cache hash already has the zig version in there, so it's not really
needed.
This function previously wrote a trailing directory separator, but
that's not correct if the path refers to a file.
Updates the build runner to unconditionally require a zig lib directory
parameter. This parameter is needed in order to correctly understand
file system inputs from zig compiler subprocesses, since they will refer
to "the zig lib directory", and the build runner needs to place file
system watches on directories in there.

The build runner's fanotify file watching implementation now accounts
for when two or more Cache.Path instances compare unequal but ultimately
refer to the same directory in the file system.

Breaking change: std.Build no longer has a zig_lib_dir field. Instead,
there is the Graph zig_lib_directory field, and individual Compile steps
can still have their zig lib directories overridden. I think this is
unlikely to break anyone's build in practice.

The compiler now sends a "file_system_inputs" message to the build
runner which shares the full set of files that were added to the cache
system with the build system, so that the build runner can watch
properly and redo the Compile step. This is implemented for whole cache
mode but not yet for incremental cache mode.
These are also used for whole cache mode in the case that any compile
errors are emitted.
need to add another field to initialize now
it's not advertised in the usage and only available in debug builds of
the compiler. Makes it easier to test changes to the build runner that
might affect targets differently.
Makes the build runner compile successfully for non-linux targets;
printing an error if you ask for --watch rather than making build
scripts fail to compile.
@andrewrk andrewrk enabled auto-merge July 12, 2024 23:42
@andrewrk andrewrk merged commit 1d20ff1 into master Jul 12, 2024
10 checks passed
@andrewrk andrewrk deleted the watch branch July 12, 2024 23:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
breaking Implementing this issue could cause existing code to no longer compile or have different behavior. release notes This PR should be mentioned in the release notes. zig build system std.Build, the build runner, `zig build` subcommand, package management
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant