Incremental compilation

[mlugg]

Incremental compilation is beginning to work on master under -fno-emit-bin or -ofmt=c. Bug reports for crashes or incorrect behavior when using incremental compilation with these compiler flags are encouraged.

Rather than bugs in existing parts of the incremental compilation logic, this issue tracks all logic which is explicitly WIP or incomplete, such as most linkers.

Codegen and linking

• self-hosted ELF linker
• self-hosted COFF linker
• self-hosted MachO linker
  • this is slightly lower priority as we don't have a working self-hosted aarch64 backend yet so this would only be usable on x86_64 macOS
• LLVM backend
  • this is low priority as there isn't much to gain from it; we still have to wait for the full LLVM emit every update

Other

• incremental updates for std.builtin.panic and other cached values
  • some values are cached in Zcu, e.g. Zcu.panic_func_index
  • so, updates to panic don't register the correct dependencies and can crash the compiler or worse
• incremental updates for std.builtin types
  • we currently don't register dependencies when doing this; Zcu.PerThread.getBuiltin etc should live on Sema so as to register appropriate dependencies
• enable comptime call memoization
  • first, do some experiments to see how valuable memoization actually is
  • assuming it's worthwhile, associated each memoized call with a Cau to track dependencies properly
• figure out how to handle test references
  • determining which tests are referenced at the end of an update would require a resolveReferences
  • if we're doing that, let's just memoize its result in a field on Zcu
  • but, changing the test discovery rules could be an alternative solution, since it's an idea that's been floated anyway

Tooling

• fuzzing infrastructure to find crashes and incorrect behavior in incremental compilation
• debug tooling to dump dependency information into a graphviz/similar file
• a compact binary format giving information about invalidated dependencies
  • if the compiler generates this under -fincremental and has a way to give it to the user, this could be useful for incremental bug reports

[andrewrk]

Regarding fuzzing, here are some properties we can test in addition to the obvious ones (i.e. not panicking or crashing)

• path independence: regardless of the incremental steps taken along the way, two compilations based on the same set of input files should result in the same output binary and/or compilation errors.
• Detection of triggering too much invalidation: test cases can be generated in an environment that includes the standard library; no matter what test case is generated, it should not cause, for example, std.process.exit to be reanalyzed. This applies to everything not generated by the test case, assuming the test case cannot set std options.