Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

make self-hosted the default compiler #12368

Merged
merged 24 commits into from
Aug 20, 2022
Merged

make self-hosted the default compiler #12368

merged 24 commits into from
Aug 20, 2022

Conversation

andrewrk
Copy link
Member

@andrewrk andrewrk commented Aug 8, 2022

stage1 is available behind the -fstage1 flag.

closes #89

Merge Blockers

General

x86_64-freebsd

  • LavaTech outage
  • ZIG_EXECUTABLE set to bad value in ci script
  • stale CI tarball
  • ld: error: undefined symbol, compiler rt functions
  • not picking up libc headers, sys/endian.h

aarch64-linux:

  • stage3 crashing building behavior tests
  • stage3 crashing building one of the test-compare-output cases

x86_64-linux:

  • test-cases hanging
  • run-translated-c remainder test failing

x86_64-macos:

x86_64-windows:

@ghost
Copy link

ghost commented Aug 8, 2022

YOOOOOOOOOOOOOOOOO

@ghost
Copy link

ghost commented Aug 8, 2022

Hang on a minute. #89 now says

async/await/suspend/resume will not be available in 0.10.0

So what's the plan there? Do we fall back to stage1 if we see it? Do we just break all code? I feel like I must be out of the loop here.

@Vexu
Copy link
Member

Vexu commented Aug 8, 2022

See #12345

@kubkon
Copy link
Member

kubkon commented Aug 8, 2022

Re failures on macOS, I haven't been able to repro locally on arm64, however, I did notice that for folks linking against a system provided LLVM (e.g., installed via Homebrew, etc.), they will now be forced to manually specify system include and search dirs as well as sysroot path. In order to avoid this silliness, here's my patch suggestion:

diff --git a/src/main.zig b/src/main.zig
index d041036b9..758d27084 100644
--- a/src/main.zig
+++ b/src/main.zig
@@ -2375,14 +2375,9 @@ fn buildOutputType(
         }
     }
 
-    if (comptime builtin.target.isDarwin()) {
-        // If we want to link against frameworks, we need system headers.
-        if (framework_dirs.items.len > 0 or frameworks.count() > 0)
-            want_native_include_dirs = true;
-    }
-
     if (sysroot == null and cross_target.isNativeOs() and
-        (system_libs.count() != 0 or want_native_include_dirs))
+        (system_libs.count() != 0 or want_native_include_dirs or
+        comptime builtin.target.isDarwin()))
     {
         const paths = std.zig.system.NativePaths.detect(arena, target_info) catch |err| {
             fatal("unable to detect native system paths: {s}", .{@errorName(err)});

In other words, if we are native and targeting Darwin, add system include and search dirs by default, and additionally, try locating the sysroot automatically too.

@kubkon
Copy link
Member

kubkon commented Aug 8, 2022

I did hit this when running a couple of test suites on arm64 macOS:

zig/build on  stage3-default [$!?] via C v13.1.6-clang via △ v3.22.3 via ↯ v0.10.0-dev.3433+cbac7a019                                                                                                                                
❯ ./stage3/bin/zig build test-link -Denable-macos-sdk                                                                                                                                                                                 
info: PASS build test/link/bss/build.zig (Debug)                                                                                                                                                                                      
All 2 tests passed.                                                                                                                                                                                                                   
info: PASS build test/link/common_symbols/build.zig (Debug)                                                                                                                                                                           
All 2 tests passed.                                                                                                                                                                                                                   
info: PASS build test/link/common_symbols/build.zig (ReleaseSafe)                                                                                                                                                                     
All 2 tests passed.                                                                                                                                                                                                                   
info: PASS build test/link/common_symbols/build.zig (ReleaseFast)                                                                                                                                                                     
All 2 tests passed.                                                                                                                                                                                                                   
info: PASS build test/link/common_symbols/build.zig (ReleaseSmall)                                                                                                                                                                    
All 1 tests passed.                                                                                                                                                                                                                   
info: PASS build test/link/common_symbols_alignment/build.zig (Debug)                                                                                                                                                                 
All 1 tests passed.                                                                                                                                                                                                                   
info: PASS build test/link/common_symbols_alignment/build.zig (ReleaseSafe)                                                                                                                                                           
All 1 tests passed.                                                                                                                                                                                                                   
info: PASS build test/link/common_symbols_alignment/build.zig (ReleaseFast)                                                                                                                                                           
All 1 tests passed.                                                                                                                                                                                                                   
info: PASS build test/link/common_symbols_alignment/build.zig (ReleaseSmall)                                                                                                                                                          
Segmentation fault at address 0x70                                                                                                                                                                                                    
???:?:?: 0x1025dcc24 in __ZN4llvm16cast_convert_valIN5clang16EnumConstantDeclEPNS1_4DeclES4_E4doitERKS4_ (???)                                                                                                                        
???:?:?: 0x10243ac67 in _ZigClangPreprocessingRecord_iterator_deref (???)                                                                                                                                                             
/Users/kubkon/dev/zig/src/translate_c.zig:452:32: 0x1010f9e7f in prepopulateGlobalNameTable (zig)                                                                                                                                     
        const entity = it.deref();                                                                                                                                                                                                    
                               ^                                                                                                                                                                                                      
/Users/kubkon/dev/zig/src/translate_c.zig:423:35: 0x101107563 in translate (zig)                                                                                                                                                      
    try prepopulateGlobalNameTable(ast_unit, &context);                                                                                                                                                                               
                                  ^                                                                                                                                                                                                   
/Users/kubkon/dev/zig/src/Compilation.zig:3446:41: 0x10111fd0f in cImport (zig)                                                                                                                                                       
        var tree = translate_c.translate(                                                                                                                                                                                             
                                        ^                                                                                                                                                                                             
/Users/kubkon/dev/zig/src/Sema.zig:4741:42: 0x1017ce66f in zirCImport (zig)                                                                                                                                                           
    const c_import_res = mod.comp.cImport(c_import_buf.items) catch |err|                                                                                                                                                             
                                         ^                                                                                                                                                                                            
/Users/kubkon/dev/zig/src/Sema.zig:717:65: 0x10166f8af in analyzeBodyInner (zig)                                                                                                                                                      
            .c_import                     => try sema.zirCImport(block, inst),                                                                                                                                                        
                                                                ^                                                                                                                                                                     
/Users/kubkon/dev/zig/src/Sema.zig:620:45: 0x10153d1ab in analyzeBodyBreak (zig)                                                                                                                                                      
    const break_inst = sema.analyzeBodyInner(block, body) catch |err| switch (err) {                                                                                                                                                  
                                            ^                                                                                                                                                                                         
/Users/kubkon/dev/zig/src/Module.zig:4500:50: 0x10153a8bb in semaDecl (zig)                                                                                                                                                           
    const result_ref = (try sema.analyzeBodyBreak(&block_scope, body)).?.operand;                                                                                                                                                     
                                                 ^                                                                                                                                                                                    
/Users/kubkon/dev/zig/src/Module.zig:4141:38: 0x1013c01ab in ensureDeclAnalyzed (zig)                                                                                                                                                 
    const type_changed = mod.semaDecl(decl_index) catch |err| switch (err) {                                                                                                                                                          
                                     ^                                                                                                                                                                                                
/Users/kubkon/dev/zig/src/Sema.zig:25231:32: 0x1019e3a47 in ensureDeclAnalyzed (zig)                                                                                                                                                  
    sema.mod.ensureDeclAnalyzed(decl_index) catch |err| {                                                                                                                                                                             
                               ^                                                                                                                                                                                                      
/Users/kubkon/dev/zig/src/Sema.zig:25273:32: 0x101846e07 in analyzeDeclRef (zig)                                                                                                                                                      
    try sema.ensureDeclAnalyzed(decl_index);                                                                                                                                                                                          
                               ^                                                                                                                                                                                                      
/Users/kubkon/dev/zig/src/Sema.zig:5326:31: 0x1017d19af in zirDeclRef (zig)                                                                                                                                                           
    return sema.analyzeDeclRef(decl_index) catch |err| switch (err) {                                                                                                                                                                 
                              ^                                                                                                                                                                                                       
/Users/kubkon/dev/zig/src/Sema.zig:727:65: 0x10166ff7f in analyzeBodyInner (zig)                                                                                                                                                      
            .decl_ref                     => try sema.zirDeclRef(block, inst),                                                                                                                                                        
                                                                ^                                                                                                                                                                     
/Users/kubkon/dev/zig/src/Sema.zig:603:30: 0x10166e66f in analyzeBody (zig)                                                                                                                                                           
    _ = sema.analyzeBodyInner(block, body) catch |err| switch (err) {                                                                                                                                                                 
                             ^                                                                                                                                                                                                        
/Users/kubkon/dev/zig/src/Module.zig:5517:21: 0x10152fc4f in analyzeFnBody (zig)                                                                                                                                                      
    sema.analyzeBody(&inner_block, fn_info.body) catch |err| switch (err) {                                                                                                                                                           
                    ^                                                                                                                                                                                                                 
/Users/kubkon/dev/zig/src/Module.zig:4227:40: 0x1013bf53f in ensureFuncBodyAnalyzed (zig)                                                                                                                                             
            var air = mod.analyzeFnBody(func, sema_arena) catch |err| switch (err) {                                                                                                                                                  
                                       ^                                                                                                                                                                                              
/Users/kubkon/dev/zig/src/Compilation.zig:2941:42: 0x1013bdf93 in processOneJob (zig)                                                                                                                                                 
            module.ensureFuncBodyAnalyzed(func) catch |err| switch (err) {              

/Users/kubkon/dev/zig/src/Compilation.zig:2879:30: 0x1012f8867 in performAllTheWork (zig)                                                                                                                                             
            try processOneJob(comp, work_item);
                             ^
/Users/kubkon/dev/zig/src/Compilation.zig:2220:31: 0x1012f5f43 in update (zig)
    try comp.performAllTheWork(main_progress_node);
                              ^
/Users/kubkon/dev/zig/src/main.zig:3327:20: 0x101317d67 in updateModule (zig)
    try comp.update();
                   ^
/Users/kubkon/dev/zig/src/main.zig:3016:17: 0x1011456ab in buildOutputType (zig)
    updateModule(gpa, comp, hook) catch |err| switch (err) {
                ^
/Users/kubkon/dev/zig/src/main.zig:236:31: 0x1010e519b in mainArgs (zig)
        return buildOutputType(gpa, arena, args, .zig_test);
                              ^
/Users/kubkon/dev/zig/src/stage1.zig:48:24: 0x1010e49ef in main (zig)
        stage2.mainArgs(gpa, arena, args) catch unreachable;
                       ^
???:?:?: 0x10d8f108b in ??? (???)
???:?:?: 0xaf58ffffffffffff in ??? (???)
error: test...
error: The following command terminated unexpectedly:
/Users/kubkon/dev/zig/build/stage3/bin/zig test /Users/kubkon/dev/zig/test/link/interdependent_static_c_libs/main.zig /Users/kubkon/dev/zig/test/link/interdependent_static_c_libs/zig-cache/o/3dca64fe37149040acf5e5f1f76b3493/liba.a
 /Users/kubkon/dev/zig/test/link/interdependent_static_c_libs/zig-cache/o/42a1147a229d50846f350c2c0f4d2eb4/libb.a --cache-dir /Users/kubkon/dev/zig/test/link/interdependent_static_c_libs/zig-cache --global-cache-dir /Users/kubkon/
.cache/zig --name test -I /Users/kubkon/dev/zig/test/link/interdependent_static_c_libs --enable-cache 
error: the following build command failed with exit code 6:
/Users/kubkon/dev/zig/test/link/interdependent_static_c_libs/zig-cache/o/1d2b0f7660520185ebb1ada8ad4c2034/build /Users/kubkon/dev/zig/build/stage3/bin/zig /Users/kubkon/dev/zig/test/link/interdependent_static_c_libs /Users/kubkon/
dev/zig/test/link/interdependent_static_c_libs/zig-cache /Users/kubkon/.cache/zig test

@Jarred-Sumner
Copy link
Contributor

Instead of breaking existing projects with the upgrade, could sema detect usage of async (or other unsupported feature) and automatically switch to stage1?

@andrewrk
Copy link
Member Author

andrewrk commented Aug 8, 2022

Instead of breaking existing projects with the upgrade, could sema detect usage of async (or other unsupported feature) and automatically switch to stage1?

I thought about this. I think that the convenience it brings is outweighed by the confusion it might cause, of people not knowing which compiler implementation they are using. I think it should be an intentional choice, and the first question I'm going to ask someone when they report a bug is, "what CLI invocation are you using?" which will tell me which compiler they are using. If it depended on the code, bugs would be harder to diagnose.

Also keep in mind this is transitional period. At some point between now and the next release, this option just disappears entirely.

Also note that for your project you can set stage1 = true in your build script, preventing any users from having to follow any extra directions.

@andrewrk
Copy link
Member Author

andrewrk commented Aug 8, 2022

The aarch64-linux CI failure is #12360

@kamidev
Copy link

kamidev commented Aug 9, 2022

I did hit this when running a couple of test suites on arm64 macOS:

zig/build on  stage3-default [$!?] via C v13.1.6-clang via △ v3.22.3 via ↯ v0.10.0-dev.3433+cbac7a019                                                                                                                                

I fail at the same test

➜  build git:(master) ✗ sw_vers;clang --version; zig version
ProductName:	macOS
ProductVersion:	12.5
BuildVersion:	21G72
Homebrew clang version 14.0.6
Target: arm64-apple-darwin21.6.0
Thread model: posix
InstalledDir: /opt/homebrew/opt/llvm/bin
0.10.0-dev.3498+d769fd010
➜  build git:(master) ✗ ./stage3/bin/zig build test-link -Denable-macos-sdk
info: PASS build test/link/bss/build.zig (Debug)
All 2 tests passed.
info: PASS build test/link/common_symbols/build.zig (Debug)
All 2 tests passed.
info: PASS build test/link/common_symbols/build.zig (ReleaseSafe)
All 2 tests passed.
info: PASS build test/link/common_symbols/build.zig (ReleaseFast)
All 2 tests passed.
info: PASS build test/link/common_symbols/build.zig (ReleaseSmall)
All 1 tests passed.
info: PASS build test/link/common_symbols_alignment/build.zig (Debug)
All 1 tests passed.
info: PASS build test/link/common_symbols_alignment/build.zig (ReleaseSafe)
All 1 tests passed.
info: PASS build test/link/common_symbols_alignment/build.zig (ReleaseFast)
All 1 tests passed.
info: PASS build test/link/common_symbols_alignment/build.zig (ReleaseSmall)
thread 2794218 panic: Segmentation fault at address 0x70
Analyzing /Users/jonas/src/zig/zig/test/link/interdependent_static_c_libs/main.zig: main.zig:c
    > %10 = c_import({
        %11 = str("b.h")
        %12 = as_node(@Zir.Inst.Ref.const_slice_u8_type, %11)
        %13 = extended(c_include(%12))
        %14 = ensure_result_used(@Zir.Inst.Ref.void_value)
        %15 = break_inline(%10, @Zir.Inst.Ref.void_value)
      })
      %16 = break_inline(%9, %10)
    For full context, use the command
      zig ast-check -t /Users/jonas/src/zig/zig/test/link/interdependent_static_c_libs/main.zig

  in /Users/jonas/src/zig/zig/test/link/interdependent_static_c_libs/main.zig: main.zig:test.import C sub
    > %21 = decl_ref("c")

???:?:?: 0x106b2cbc8 in __ZN4llvm16cast_convert_valIN5clang16EnumConstantDeclEPNS1_4DeclES4_E4doitERKS4_ (???)
???:?:?: 0x1063caa1f in _ZigClangPreprocessingRecord_iterator_deref (???)
/Users/jonas/src/zig/zig/src/translate_c.zig:452:32: 0x102415aff in prepopulateGlobalNameTable (zig)
        const entity = it.deref();
                               ^
/Users/jonas/src/zig/zig/src/translate_c.zig:423:35: 0x1024231e3 in translate (zig)
    try prepopulateGlobalNameTable(ast_unit, &context);
                                  ^
/Users/jonas/src/zig/zig/src/Compilation.zig:3482:41: 0x102a75173 in cImport (zig)
        var tree = translate_c.translate(
                                        ^
/Users/jonas/src/zig/zig/src/Sema.zig:4741:42: 0x102895657 in zirCImport (zig)
    const c_import_res = mod.comp.cImport(c_import_buf.items) catch |err|
                                         ^
/Users/jonas/src/zig/zig/src/Sema.zig:717:65: 0x102740cd3 in analyzeBodyInner (zig)
            .c_import                     => try sema.zirCImport(block, inst),
                                                                ^
/Users/jonas/src/zig/zig/src/Sema.zig:620:45: 0x1025ffbf3 in analyzeBodyBreak (zig)
    const break_inst = sema.analyzeBodyInner(block, body) catch |err| switch (err) {
                                            ^
/Users/jonas/src/zig/zig/src/Module.zig:4500:50: 0x1025fd303 in semaDecl (zig)
    const result_ref = (try sema.analyzeBodyBreak(&block_scope, body)).?.operand;
                                                 ^
/Users/jonas/src/zig/zig/src/Module.zig:4141:38: 0x102468777 in ensureDeclAnalyzed (zig)
    const type_changed = mod.semaDecl(decl_index) catch |err| switch (err) {
                                     ^
/Users/jonas/src/zig/zig/src/Sema.zig:25233:32: 0x102a9fa2f in ensureDeclAnalyzed (zig)
    sema.mod.ensureDeclAnalyzed(decl_index) catch |err| {
                               ^
/Users/jonas/src/zig/zig/src/Sema.zig:25275:32: 0x10290ddf7 in analyzeDeclRef (zig)
    try sema.ensureDeclAnalyzed(decl_index);
                               ^
/Users/jonas/src/zig/zig/src/Sema.zig:5326:31: 0x102898997 in zirDeclRef (zig)
    return sema.analyzeDeclRef(decl_index) catch |err| switch (err) {
                              ^
/Users/jonas/src/zig/zig/src/Sema.zig:727:65: 0x1027413a3 in analyzeBodyInner (zig)
            .decl_ref                     => try sema.zirDeclRef(block, inst),
                                                                ^
/Users/jonas/src/zig/zig/src/Sema.zig:603:30: 0x10273fa8b in analyzeBody (zig)
    _ = sema.analyzeBodyInner(block, body) catch |err| switch (err) {
                             ^
/Users/jonas/src/zig/zig/src/Module.zig:5517:21: 0x1025f2697 in analyzeFnBody (zig)
    sema.analyzeBody(&inner_block, fn_info.body) catch |err| switch (err) {
                    ^
/Users/jonas/src/zig/zig/src/Module.zig:4227:40: 0x102467b0b in ensureFuncBodyAnalyzed (zig)
            var air = mod.analyzeFnBody(func, sema_arena) catch |err| switch (err) {
                                       ^
/Users/jonas/src/zig/zig/src/Compilation.zig:2962:42: 0x10246655f in processOneJob (zig)
            module.ensureFuncBodyAnalyzed(func) catch |err| switch (err) {
                                         ^
/Users/jonas/src/zig/zig/src/Compilation.zig:2894:30: 0x10239643f in performAllTheWork (zig)
            try processOneJob(comp, work_item);
                             ^
/Users/jonas/src/zig/zig/src/Compilation.zig:2234:31: 0x102393a3b in update (zig)
    try comp.performAllTheWork(main_progress_node);
                              ^
/Users/jonas/src/zig/zig/src/main.zig:3327:20: 0x1023b593f in updateModule (zig)
    try comp.update();
                   ^
/Users/jonas/src/zig/zig/src/main.zig:3016:17: 0x1021e5297 in buildOutputType (zig)
    updateModule(gpa, comp, hook) catch |err| switch (err) {
                ^
/Users/jonas/src/zig/zig/src/main.zig:236:31: 0x1021c2c1f in mainArgs (zig)
        return buildOutputType(gpa, arena, args, .zig_test);
                              ^
/Users/jonas/src/zig/zig/src/main.zig:174:20: 0x1021c24f7 in main (zig)
    return mainArgs(gpa, arena, args);
                   ^
/Users/jonas/src/zig/zig/build/stage2/lib/zig/std/start.zig:576:37: 0x1021c39c7 in main (zig)
            const result = root.main() catch |err| {
                                    ^
???:?:?: 0x10e9cd08b in ??? (???)
???:?:?: 0xbf037fffffffffff in ??? (???)
error: test...
error: The following command terminated unexpectedly:
/Users/jonas/src/zig/zig/build/stage3/bin/zig test /Users/jonas/src/zig/zig/test/link/interdependent_static_c_libs/main.zig /Users/jonas/src/zig/zig/test/link/interdependent_static_c_libs/zig-cache/o/53b36e50ce70877cd479c2434a2457f3/liba.a /Users/jonas/src/zig/zig/test/link/interdependent_static_c_libs/zig-cache/o/8bad882ec5f57577e522c8c74924c9ee/libb.a --cache-dir /Users/jonas/src/zig/zig/test/link/interdependent_static_c_libs/zig-cache --global-cache-dir /Users/jonas/.cache/zig --name test -I /Users/jonas/src/zig/zig/test/link/interdependent_static_c_libs --enable-cache
error: the following build command failed with exit code 6:
/Users/jonas/src/zig/zig/test/link/interdependent_static_c_libs/zig-cache/o/88be37ed130f9e8fabcccd4a0685ea45/build /Users/jonas/src/zig/zig/build/stage3/bin/zig /Users/jonas/src/zig/zig/test/link/interdependent_static_c_libs /Users/jonas/src/zig/zig/test/link/interdependent_static_c_libs/zig-cache /Users/jonas/.cache/zig test
The following command exited with error code 1 (expected 0):
cd /Users/jonas/src/zig/zig && build/stage3/bin/zig build --build-file /Users/jonas/src/zig/zig/test/link/interdependent_static_c_libs/build.zig test
error: UnexpectedExitCode

andrewrk and others added 20 commits August 19, 2022 16:45
This updates to a stage3 freebsd tarball.
Instead of a switch in a bash file we just properly name
the pipeline steps.

Also use zig build to produce the release artifact.
it was returning a pointer to a parameter.
This includes a few recent bug fixes which may solve the Windows CI
failure.
Simplify the script by not creating an unnecessary build directory and
then changing directories to and from it.
This was present in stage1 but missing from self-hosted.
 * accept forward and backward slashes in file paths
 * strip multiple extensions; e.g. if the basename is "test.exe.obj"
   then it strips to "test".
empirically this avoids a segfault on the CI.
@andrewrk andrewrk merged commit e5e6eb9 into master Aug 20, 2022
@andrewrk andrewrk deleted the stage3-default branch August 20, 2022 00:26
@itsmontoya
Copy link

Congratulations @andrewrk ! This is huge!

@IgorKorobeynikov
Copy link

@andrewrk #1481

Vexu pushed a commit that referenced this pull request Aug 31, 2022
Some recent change makes slice concatenation runtime (merge #12368), so the example needs to be explicitly made comptime.
qcodeadmin pushed a commit to nakamochi/ndg that referenced this pull request Feb 8, 2023
stage2 has been merged and set as default around end of aug 2022 in
ziglang/zig#12368. since then the main focus has
been on this new stage2 compiler.

with the release of 0.10.1 the stage2 is fairly useable now and it is
the way forward, towards zig 0.11 or 1.0. read more about difference in
https://github.com/ziglang/zig/wiki/Self-Hosted-Compiler-Upgrade-Guide

another good read on the relevant topic is
https://ziglang.org/news/goodbye-cpp/
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

self-hosted compiler: ship it!
7 participants