separate the build runner process from the configure process #20981
Labels
breaking
Implementing this issue could cause existing code to no longer compile or have different behavior.
enhancement
Solving this issue will likely involve adding new logic or components to the codebase.
zig build system
std.Build, the build runner, `zig build` subcommand, package management
Milestone
Comments
andrewrk
added
enhancement
|
This breaking change may also be an opportune time to rename |
|
As you've already stated on discord, this will break certain use cases of custom build steps. I'd consider #20935 a blocker for this as currently we need to fall back to custom build steps to achieve re-running the step when something in the directory changes. Uses cases here are asset bundling for games/websites/... Also projects like https://github.com/zig-osdev/disk-image-step will suffer from this because of two reasons:
|
matklad
added a commit
to tigerbeetle/tigerbeetle
that referenced
this issue
Aug 23, 2024
This moves multiversion from release.zig to build.zig, so that you can
just build a multiversion binary normally, eg:
```
$ ./zig/zig build -Dmultiversion=0.15.6 -Dconfig-release=0.15.7 -Dconfig-release-client-min=0.15.6 -Dtarget=x86_64-linux
$ ./tigerbeetle multiversion ./tigerbeetle
multiversioning.exe_path=./tigerbeetle
multiversioning.absolute_exe_path=/home/matklad/p/tb/work/tigerbeetle
multiversioning.releases_bundled={ 0.15.3, 0.15.4, 0.15.5, 0.15.6, 0.15.7 }
...
```
You can pass `-Dmultiversion=latest` to fetch the latest release from
GitHub.
Note that `-Dtarget=aarch64-macos` will build a _universal_ macos image
(this _is_ confusing flag name, but I don't see a simple non-ambigious
solution here).
As a bonus, multiversion build now should work on any host, and not on
x86_64 linux only.
Implementation wise, the bulk of work happens in
`build_multiversion.zig` program, which is invoked from build.zig as a
custom Run step. Why not encode all that logic direcctly into build.zig?
The main reason is adrewrk, the most famous Zig troll. His favorite way
of trolling Zig developers is by regularly breaking the API of build.zig:
ziglang/zig#14498
The next planned trolling is to separate "configure" and "make" phases
of `build.zig` into physically separate processes, and remove support
for custom steps:
ziglang/zig#20981
In this PR, we are going to troll adrewrk _back_ by fixing our code even
before it gets broken, and by making our custom step a Run step instead.
That being said, it I have split `gh release download` into a separate
step, so that is properly cached.
matklad
added a commit
to tigerbeetle/tigerbeetle
that referenced
this issue
Aug 23, 2024
This moves multiversion from release.zig to build.zig, so that you can
just build a multiversion binary normally, eg:
```
$ ./zig/zig build -Dmultiversion=0.15.6 -Dconfig-release=0.15.7 -Dconfig-release-client-min=0.15.6 -Dtarget=x86_64-linux
$ ./tigerbeetle multiversion ./tigerbeetle
multiversioning.exe_path=./tigerbeetle
multiversioning.absolute_exe_path=/home/matklad/p/tb/work/tigerbeetle
multiversioning.releases_bundled={ 0.15.3, 0.15.4, 0.15.5, 0.15.6, 0.15.7 }
...
```
You can pass `-Dmultiversion=latest` to fetch the latest release from
GitHub.
Note that `-Dtarget=aarch64-macos` will build a _universal_ macos image
(this _is_ confusing flag name, but I don't see a simple non-ambigious
solution here).
As a bonus, multiversion build now should work on any host, and not on
x86_64 linux only.
Implementation wise, the bulk of work happens in
`build_multiversion.zig` program, which is invoked from build.zig as a
custom Run step. Why not encode all that logic direcctly into build.zig?
The main reason is adrewrk, the most famous Zig troll. His favorite way
of trolling Zig developers is by regularly breaking the API of build.zig:
ziglang/zig#14498
The next planned trolling is to separate "configure" and "make" phases
of `build.zig` into physically separate processes, and remove support
for custom steps:
ziglang/zig#20981
In this PR, we are going to troll adrewrk _back_ by fixing our code even
before it gets broken, and by making our custom step a Run step instead.
That being said, it I have split `gh release download` into a separate
step, so that is properly cached.
matklad
added a commit
to tigerbeetle/tigerbeetle
that referenced
this issue
Aug 23, 2024
This moves multiversion from release.zig to build.zig, so that you can
just build a multiversion binary normally, eg:
```
$ ./zig/zig build -Dmultiversion=0.15.6 -Dconfig-release=0.15.7 -Dconfig-release-client-min=0.15.6 -Dtarget=x86_64-linux
$ ./tigerbeetle multiversion ./tigerbeetle
multiversioning.exe_path=./tigerbeetle
multiversioning.absolute_exe_path=/home/matklad/p/tb/work/tigerbeetle
multiversioning.releases_bundled={ 0.15.3, 0.15.4, 0.15.5, 0.15.6, 0.15.7 }
...
```
You can pass `-Dmultiversion=latest` to fetch the latest release from
GitHub.
Note that `-Dtarget=aarch64-macos` will build a _universal_ macos image
(this _is_ confusing flag name, but I don't see a simple non-ambigious
solution here).
As a bonus, multiversion build now should work on any host, and not on
x86_64 linux only.
Implementation wise, the bulk of work happens in
`build_multiversion.zig` program, which is invoked from build.zig as a
custom Run step. Why not encode all that logic direcctly into build.zig?
The main reason is adrewrk, the most famous Zig troll. His favorite way
of trolling Zig developers is by regularly breaking the API of build.zig:
ziglang/zig#14498
The next planned trolling is to separate "configure" and "make" phases
of `build.zig` into physically separate processes, and remove support
for custom steps:
ziglang/zig#20981
In this PR, we are going to troll adrewrk _back_ by fixing our code even
before it gets broken, and by making our custom step a Run step instead.
That being said, it I have split `gh release download` into a separate
step, so that is properly cached.
matklad
added a commit
to tigerbeetle/tigerbeetle
that referenced
this issue
Aug 23, 2024
This moves multiversion from release.zig to build.zig, so that you can
just build a multiversion binary normally, eg:
```
$ ./zig/zig build -Dmultiversion=0.15.6 -Dconfig-release=0.15.7 -Dconfig-release-client-min=0.15.6 -Dtarget=x86_64-linux
$ ./tigerbeetle multiversion ./tigerbeetle
multiversioning.exe_path=./tigerbeetle
multiversioning.absolute_exe_path=/home/matklad/p/tb/work/tigerbeetle
multiversioning.releases_bundled={ 0.15.3, 0.15.4, 0.15.5, 0.15.6, 0.15.7 }
...
```
You can pass `-Dmultiversion=latest` to fetch the latest release from
GitHub.
Note that `-Dtarget=aarch64-macos` will build a _universal_ macos image
(this _is_ confusing flag name, but I don't see a simple non-ambigious
solution here).
As a bonus, multiversion build now should work on any host, and not on
x86_64 linux only.
Implementation wise, the bulk of work happens in
`build_multiversion.zig` program, which is invoked from build.zig as a
custom Run step. Why not encode all that logic direcctly into build.zig?
The main reason is adrewrk, the most famous Zig troll. His favorite way
of trolling Zig developers is by regularly breaking the API of build.zig:
ziglang/zig#14498
The next planned trolling is to separate "configure" and "make" phases
of `build.zig` into physically separate processes, and remove support
for custom steps:
ziglang/zig#20981
In this PR, we are going to troll adrewrk _back_ by fixing our code even
before it gets broken, and by making our custom step a Run step instead.
That being said, it I have split `gh release download` into a separate
step, so that is properly cached.
matklad
added a commit
to tigerbeetle/tigerbeetle
that referenced
this issue
Aug 23, 2024
This moves multiversion from release.zig to build.zig, so that you can
just build a multiversion binary normally, eg:
```
$ ./zig/zig build -Dmultiversion=0.15.6 -Dconfig-release=0.15.7 -Dconfig-release-client-min=0.15.6 -Dtarget=x86_64-linux
$ ./tigerbeetle multiversion ./tigerbeetle
multiversioning.exe_path=./tigerbeetle
multiversioning.absolute_exe_path=/home/matklad/p/tb/work/tigerbeetle
multiversioning.releases_bundled={ 0.15.3, 0.15.4, 0.15.5, 0.15.6, 0.15.7 }
...
```
You can pass `-Dmultiversion=latest` to fetch the latest release from
GitHub.
Note that `-Dtarget=aarch64-macos` will build a _universal_ macos image
(this _is_ confusing flag name, but I don't see a simple non-ambigious
solution here).
As a bonus, multiversion build now should work on any host, and not on
x86_64 linux only.
Implementation wise, the bulk of work happens in
`build_multiversion.zig` program, which is invoked from build.zig as a
custom Run step. Why not encode all that logic direcctly into build.zig?
The main reason is adrewrk, the most famous Zig troll. His favorite way
of trolling Zig developers is by regularly breaking the API of build.zig:
ziglang/zig#14498
The next planned trolling is to separate "configure" and "make" phases
of `build.zig` into physically separate processes, and remove support
for custom steps:
ziglang/zig#20981
In this PR, we are going to troll adrewrk _back_ by fixing our code even
before it gets broken, and by making our custom step a Run step instead.
That being said, it I have split `gh release download` into a separate
step, so that is properly cached.
matklad
added a commit
to tigerbeetle/tigerbeetle
that referenced
this issue
Aug 23, 2024
This moves multiversion from release.zig to build.zig, so that you can
just build a multiversion binary normally, eg:
```
$ ./zig/zig build -Dmultiversion=0.15.6 -Dconfig-release=0.15.7 -Dconfig-release-client-min=0.15.6 -Dtarget=x86_64-linux
$ ./tigerbeetle multiversion ./tigerbeetle
multiversioning.exe_path=./tigerbeetle
multiversioning.absolute_exe_path=/home/matklad/p/tb/work/tigerbeetle
multiversioning.releases_bundled={ 0.15.3, 0.15.4, 0.15.5, 0.15.6, 0.15.7 }
...
```
You can pass `-Dmultiversion=latest` to fetch the latest release from
GitHub.
Note that `-Dtarget=aarch64-macos` will build a _universal_ macos image
(this _is_ confusing flag name, but I don't see a simple non-ambigious
solution here).
As a bonus, multiversion build now should work on any host, and not on
x86_64 linux only.
Implementation wise, the bulk of work happens in
`build_multiversion.zig` program, which is invoked from build.zig as a
custom Run step. Why not encode all that logic direcctly into build.zig?
The main reason is adrewrk, the most famous Zig troll. His favorite way
of trolling Zig developers is by regularly breaking the API of build.zig:
ziglang/zig#14498
The next planned trolling is to separate "configure" and "make" phases
of `build.zig` into physically separate processes, and remove support
for custom steps:
ziglang/zig#20981
In this PR, we are going to troll adrewrk _back_ by fixing our code even
before it gets broken, and by making our custom step a Run step instead.
That being said, it I have split `gh release download` into a separate
step, so that is properly cached.
matklad
added a commit
to tigerbeetle/tigerbeetle
that referenced
this issue
Aug 23, 2024
This moves multiversion from release.zig to build.zig, so that you can
just build a multiversion binary normally, eg:
```
$ ./zig/zig build -Dmultiversion=0.15.6 -Dconfig-release=0.15.7 -Dconfig-release-client-min=0.15.6 -Dtarget=x86_64-linux
$ ./tigerbeetle multiversion ./tigerbeetle
multiversioning.exe_path=./tigerbeetle
multiversioning.absolute_exe_path=/home/matklad/p/tb/work/tigerbeetle
multiversioning.releases_bundled={ 0.15.3, 0.15.4, 0.15.5, 0.15.6, 0.15.7 }
...
```
You can pass `-Dmultiversion=latest` to fetch the latest release from
GitHub.
Note that `-Dtarget=aarch64-macos` will build a _universal_ macos image
(this _is_ confusing flag name, but I don't see a simple non-ambigious
solution here).
As a bonus, multiversion build now should work on any host, and not on
x86_64 linux only.
Implementation wise, the bulk of work happens in
`build_multiversion.zig` program, which is invoked from build.zig as a
custom Run step. Why not encode all that logic direcctly into build.zig?
The main reason is adrewrk, the most famous Zig troll. His favorite way
of trolling Zig developers is by regularly breaking the API of build.zig:
ziglang/zig#14498
The next planned trolling is to separate "configure" and "make" phases
of `build.zig` into physically separate processes, and remove support
for custom steps:
ziglang/zig#20981
In this PR, we are going to troll adrewrk _back_ by fixing our code even
before it gets broken, and by making our custom step a Run step instead.
That being said, it I have split `gh release download` into a separate
step, so that is properly cached.
matklad
added a commit
to tigerbeetle/tigerbeetle
that referenced
this issue
Aug 23, 2024
This moves multiversion from release.zig to build.zig, so that you can
just build a multiversion binary normally, eg:
```
$ ./zig/zig build -Dmultiversion=0.15.6 -Dconfig-release=0.15.7 -Dconfig-release-client-min=0.15.6 -Dtarget=x86_64-linux
$ ./tigerbeetle multiversion ./tigerbeetle
multiversioning.exe_path=./tigerbeetle
multiversioning.absolute_exe_path=/home/matklad/p/tb/work/tigerbeetle
multiversioning.releases_bundled={ 0.15.3, 0.15.4, 0.15.5, 0.15.6, 0.15.7 }
...
```
You can pass `-Dmultiversion=latest` to fetch the latest release from
GitHub.
Note that `-Dtarget=aarch64-macos` will build a _universal_ macos image
(this _is_ confusing flag name, but I don't see a simple non-ambigious
solution here).
As a bonus, multiversion build now should work on any host, and not on
x86_64 linux only.
Implementation wise, the bulk of work happens in
`build_multiversion.zig` program, which is invoked from build.zig as a
custom Run step. Why not encode all that logic direcctly into build.zig?
The main reason is adrewrk, the most famous Zig troll. His favorite way
of trolling Zig developers is by regularly breaking the API of build.zig:
ziglang/zig#14498
The next planned trolling is to separate "configure" and "make" phases
of `build.zig` into physically separate processes, and remove support
for custom steps:
ziglang/zig#20981
In this PR, we are going to troll adrewrk _back_ by fixing our code even
before it gets broken, and by making our custom step a Run step instead.
That being said, it I have split `gh release download` into a separate
step, so that is properly cached.
matklad
added a commit
to tigerbeetle/tigerbeetle
that referenced
this issue
Aug 23, 2024
This moves multiversion from release.zig to build.zig, so that you can
just build a multiversion binary normally, eg:
```
$ ./zig/zig build -Dmultiversion=0.15.6 -Dconfig-release=0.15.7 -Dconfig-release-client-min=0.15.6 -Dtarget=x86_64-linux
$ ./tigerbeetle multiversion ./tigerbeetle
multiversioning.exe_path=./tigerbeetle
multiversioning.absolute_exe_path=/home/matklad/p/tb/work/tigerbeetle
multiversioning.releases_bundled={ 0.15.3, 0.15.4, 0.15.5, 0.15.6, 0.15.7 }
...
```
You can pass `-Dmultiversion=latest` to fetch the latest release from
GitHub.
Note that `-Dtarget=aarch64-macos` will build a _universal_ macos image
(this _is_ confusing flag name, but I don't see a simple non-ambigious
solution here).
As a bonus, multiversion build now should work on any host, and not on
x86_64 linux only.
Implementation wise, the bulk of work happens in
`build_multiversion.zig` program, which is invoked from build.zig as a
custom Run step. Why not encode all that logic direcctly into build.zig?
The main reason is adrewrk, the most famous Zig troll. His favorite way
of trolling Zig developers is by regularly breaking the API of build.zig:
ziglang/zig#14498
The next planned trolling is to separate "configure" and "make" phases
of `build.zig` into physically separate processes, and remove support
for custom steps:
ziglang/zig#20981
In this PR, we are going to troll adrewrk _back_ by fixing our code even
before it gets broken, and by making our custom step a Run step instead.
That being said, it I have split `gh release download` into a separate
step, so that is properly cached.
matklad
added a commit
to tigerbeetle/tigerbeetle
that referenced
this issue
Aug 26, 2024
This moves multiversion from release.zig to build.zig, so that you can
just build a multiversion binary normally, eg:
```
$ ./zig/zig build -Dmultiversion=0.15.6 -Dconfig-release=0.15.7 -Dconfig-release-client-min=0.15.6 -Dtarget=x86_64-linux
$ ./tigerbeetle multiversion ./tigerbeetle
multiversioning.exe_path=./tigerbeetle
multiversioning.absolute_exe_path=/home/matklad/p/tb/work/tigerbeetle
multiversioning.releases_bundled={ 0.15.3, 0.15.4, 0.15.5, 0.15.6, 0.15.7 }
...
```
You can pass `-Dmultiversion=latest` to fetch the latest release from
GitHub.
Note that `-Dtarget=aarch64-macos` will build a _universal_ macos image
(this _is_ confusing flag name, but I don't see a simple non-ambigious
solution here).
As a bonus, multiversion build now should work on any host, and not on
x86_64 linux only.
Implementation wise, the bulk of work happens in
`build_multiversion.zig` program, which is invoked from build.zig as a
custom Run step. Why not encode all that logic direcctly into build.zig?
The main reason is adrewrk, the most famous Zig troll. His favorite way
of trolling Zig developers is by regularly breaking the API of build.zig:
ziglang/zig#14498
The next planned trolling is to separate "configure" and "make" phases
of `build.zig` into physically separate processes, and remove support
for custom steps:
ziglang/zig#20981
In this PR, we are going to troll adrewrk _back_ by fixing our code even
before it gets broken, and by making our custom step a Run step instead.
That being said, it I have split `gh release download` into a separate
step, so that is properly cached.
matklad
added a commit
to tigerbeetle/tigerbeetle
that referenced
this issue
Aug 26, 2024
This moves multiversion from release.zig to build.zig, so that you can
just build a multiversion binary normally, eg:
```
$ ./zig/zig build -Dmultiversion=0.15.6 -Dconfig-release=0.15.7 -Dconfig-release-client-min=0.15.6 -Dtarget=x86_64-linux
$ ./tigerbeetle multiversion ./tigerbeetle
multiversioning.exe_path=./tigerbeetle
multiversioning.absolute_exe_path=/home/matklad/p/tb/work/tigerbeetle
multiversioning.releases_bundled={ 0.15.3, 0.15.4, 0.15.5, 0.15.6, 0.15.7 }
...
```
You can pass `-Dmultiversion=latest` to fetch the latest release from
GitHub.
Note that `-Dtarget=aarch64-macos` will build a _universal_ macos image
(this _is_ confusing flag name, but I don't see a simple non-ambigious
solution here).
As a bonus, multiversion build now should work on any host, and not on
x86_64 linux only.
Implementation wise, the bulk of work happens in
`build_multiversion.zig` program, which is invoked from build.zig as a
custom Run step. Why not encode all that logic direcctly into build.zig?
The main reason is adrewrk, the most famous Zig troll. His favorite way
of trolling Zig developers is by regularly breaking the API of build.zig:
ziglang/zig#14498
The next planned trolling is to separate "configure" and "make" phases
of `build.zig` into physically separate processes, and remove support
for custom steps:
ziglang/zig#20981
In this PR, we are going to troll adrewrk _back_ by fixing our code even
before it gets broken, and by making our custom step a Run step instead.
That being said, it I have split `gh release download` into a separate
step, so that is properly cached.
matklad
added a commit
to tigerbeetle/tigerbeetle
that referenced
this issue
Aug 27, 2024
This moves multiversion from release.zig to build.zig, so that you can
just build a multiversion binary normally, eg:
```
$ ./zig/zig build -Dmultiversion=0.15.6 -Dconfig-release=0.15.7 -Dconfig-release-client-min=0.15.6 -Dtarget=x86_64-linux
$ ./tigerbeetle multiversion ./tigerbeetle
multiversioning.exe_path=./tigerbeetle
multiversioning.absolute_exe_path=/home/matklad/p/tb/work/tigerbeetle
multiversioning.releases_bundled={ 0.15.3, 0.15.4, 0.15.5, 0.15.6, 0.15.7 }
...
```
You can pass `-Dmultiversion=latest` to fetch the latest release from
GitHub.
Note that `-Dtarget=aarch64-macos` will build a _universal_ macos image
(this _is_ confusing flag name, but I don't see a simple non-ambigious
solution here).
As a bonus, multiversion build now should work on any host, and not on
x86_64 linux only.
Implementation wise, the bulk of work happens in
`build_multiversion.zig` program, which is invoked from build.zig as a
custom Run step. Why not encode all that logic direcctly into build.zig?
The main reason is adrewrk, the most famous Zig troll. His favorite way
of trolling Zig developers is by regularly breaking the API of build.zig:
ziglang/zig#14498
The next planned trolling is to separate "configure" and "make" phases
of `build.zig` into physically separate processes, and remove support
for custom steps:
ziglang/zig#20981
In this PR, we are going to troll adrewrk _back_ by fixing our code even
before it gets broken, and by making our custom step a Run step instead.
That being said, it I have split `gh release download` into a separate
step, so that is properly cached.
matklad
added a commit
to tigerbeetle/tigerbeetle
that referenced
this issue
Aug 27, 2024
This moves multiversion from release.zig to build.zig, so that you can
just build a multiversion binary normally, eg:
```
$ ./zig/zig build -Dmultiversion=0.15.6 -Dconfig-release=0.15.7 -Dconfig-release-client-min=0.15.6 -Dtarget=x86_64-linux
$ ./tigerbeetle multiversion ./tigerbeetle
multiversioning.exe_path=./tigerbeetle
multiversioning.absolute_exe_path=/home/matklad/p/tb/work/tigerbeetle
multiversioning.releases_bundled={ 0.15.3, 0.15.4, 0.15.5, 0.15.6, 0.15.7 }
...
```
You can pass `-Dmultiversion=latest` to fetch the latest release from
GitHub.
Note that `-Dtarget=aarch64-macos` will build a _universal_ macos image
(this _is_ confusing flag name, but I don't see a simple non-ambigious
solution here).
As a bonus, multiversion build now should work on any host, and not on
x86_64 linux only.
Implementation wise, the bulk of work happens in
`build_multiversion.zig` program, which is invoked from build.zig as a
custom Run step. Why not encode all that logic direcctly into build.zig?
The main reason is adrewrk, the most famous Zig troll. His favorite way
of trolling Zig developers is by regularly breaking the API of build.zig:
ziglang/zig#14498
The next planned trolling is to separate "configure" and "make" phases
of `build.zig` into physically separate processes, and remove support
for custom steps:
ziglang/zig#20981
In this PR, we are going to troll adrewrk _back_ by fixing our code even
before it gets broken, and by making our custom step a Run step instead.
That being said, it I have split `gh release download` into a separate
step, so that is properly cached.
|
This might be a silly question based on my very limited understanding of the build runner, so apologies in advance if that's the case. Would it be possible to achieve the goals of
by just doing #14941 and then leaning on linking? That is, compile the build runner and the build script separately so that the build runner object is almost always a cache hit, and then link the two together to create the final build binary. Any communication between the two would then happen via Again, I'm not familiar enough with the build runner internals to know if this even makes sense. If it does, though, it seems worth considering since we wouldn't have to open the can of worms that is serialization of build state, and what that means for custom steps. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This is a prerequisite to #14286.
Currently, the build runner process starts from the main() function inside
lib/compiler/build_runner.zigand is compiled alongside the user'sbuild.ziglogic. The build runner invokes the user'sbuild.ziglogic, which is the "configure phase". After thebuildfunction returns, the build runner proceeds to execute the build graph, which is the "make" phase.This issue is to separate the one compilation into two:
build.ziglogic and thebuild.ziglogic of its dependencies.The primary motivation for this is to make
zig buildfaster, both in the sense that it will take less time to compile, because only the user'sbuild.ziglogic is being recompiled when it changes, and in the sense that the build runner could be built with optimizations enabled, which is starting to become more valuable now that we have introduced--watchand--fuzz.--debug-runner-optimize=[mode]should be introduced to allow switching the build runner to be compiled in a different optimization mode; this will be helpful when working on a patch to the build runner itself.The primary challenge for implementing this will be separating out state from
std.Build.Stepthat corresponds to configure phase and that corresponds to make phase. It will be a welcome change, because it will make the API a lot smaller and prevent users from having access to state that is only meant for use by the build runner. The build runner will probably have its own version of eachstd.Build.Stepwith different fields and logic.The other major thing that needs to be done to make this work is serialization of the build graph. The output of the configure process needs to be communicated to the runner process, ideally by using stdout alone.
This constitutes a breaking change. In particular, custom build steps will stop working. Users should switch to using Run steps instead.
If we can't get this fully done in this release cycle, #14941 will be a good first step along the way.
The text was updated successfully, but these errors were encountered: