Hxb

[Simn]

This PR adds the writer and reader for hxb, the new Haxe binary format.

What is a binary format?

When Haxe compiles a program, it first parses syntax into an abstract syntax tree (AST) and then types that into a typed expression representation. This representation is handed of to the generators, who then produce the desired output. With a binary format, we can export that representation and reuse it later. With a Main.hx like this:

/**
	Can you spot this documentation in the output?
**/
function main() {
	trace("Hello hxb");
}

haxe -cp src -main Main --js export/hello-hxb.js --hxb export/hello-hxb.zip

This will generate export/hello-hxb.zip, which contains, among others, a file js/Main.hxb that looks like this in a hex editor:

The good news is that you don't have to understand this in order to use it! The generated data can now be consumed with the --hxb-lib argument:

haxe --hxb-lib export/hello-hxb.zip -main Main --js export/hello-hxb-2.js

Note how we are not adding -cp src here. It's not necessary because the compiler doesn't need any source files and can instead completely rely on the binary data provided via --hxb-lib export/hello-hxb.zip. We can also easily confirm that the generated JavaScript output is equal.

How do I use this?

At the moment, there are exactly two command line arguments for hxb:

1. --hxb path.zip generates the binary data for all compiled types in the specified .zip-file. This is auxiliary output, like --xml and --json.
2. --hxb-lib path.zip reads the binary data from the specified .zip-file. It can be understood as an equivalent to -cp but with binary data instead of .hx source files.

Additionally, the Haxe compilation cache uses hxb as its internal representation - exclusively for now. While we ultimately want to implement a hybrid memory/hxb cache for optimal performance, we need proper testing of the hxb part before we can look into that.

Limitations

• At the moment, --hxb generates all known types to the .zip file. We will add the option to specify what exactly should be generated once we've designed how to exactly do this. Track [hxb] Filter for --hxb output #11490 for further information.
• For the time being, --hxb-lib takes priority over -cp, -lib and any native library include like --java-lib. There are plans to make it respect the order of appearance in the command line arguments, but this is going to require some additional work. See [hxb] --hxb-lib vs. -cp vs. --java-lib #11494.
• The information specified by -main is not stored in the generated .zip-file yet, which means we don't have runnable archives. See [hxb] Add -main information #11502.
• All hxb output is target-specific and, in the general case, cannot be used across multiple targets. In fact, the output is bucketed by target name in the generated .zip file and won't even be found when using a different target. This limitation is likely to be permanent due to the way Haxe's typing works.
• The format itself isn't quite stable yet, and probably won't be entirely stable until the release of Haxe 5. This affects users only insofar as an archive generated on Haxe commit N might not be compatible when being used on Haxe commit N + 1.
• Even after the format is stable, we won't be able to guarantee binary compatibility between minor Haxe versions. Any internal change we make has to be reflected in the binary format. We will make an effort to handle changes in a way that the output from the newest Haxe version is compatible with older Haxe versions, too.
• There's no other way to put this: the hxb-powered server cache will never be as fast as the current in-memory cache. This is simply because we're now doing something (reading hxb) instead of nothing, and there will always be a cost associated with that. Whether or not that cost is significant, or even observable, is a different question though!

Why should I use this?

Hxb is particularly interesting when you're working with portions of code that rarely, if ever, change. In that case, pre-compiling such code to hxb can give great performance benefits. One very nice case-study for this is go2hx, which deals with the entire Go standard library. Here is the --times output posted by @PXshadow:

Without hxb:

name        | time(s) |   % |  p% |      # | info
--------------------------------------------
analyzer    |   2.904 |  28 |  28 | 816305 |
filters     |   2.468 |  24 |  24 | 133831 |
typing      |   1.517 |  15 |  15 |   2876 |
generate    |   1.502 |  15 |  15 |      2 |
  hl        |   1.502 |  15 | 100 |      2 |
    write   |   0.452 |   4 |  30 |      1 |
macro       |   1.038 |  10 |  10 |   3532 |
  jit       |   0.039 |   0 |   4 |   1348 |
parsing     |   0.883 |   9 |   9 |    382 |
finalize    |   0.015 |   0 |   0 |      1 |
--------------------------------------------
total       |  10.328 | 100 | 100 | 956931 |

With hxb:

name                          | time(s) |   % |  p% |     # | info
-------------------------------------------------------------
generate                      |   1.329 |  55 |  55 |     2 |
  hl                          |   1.329 |  55 | 100 |     2 |
    write                     |   0.343 |  14 |  26 |     1 |
filters                       |   0.339 |  14 |  14 | 10953 |
typing                        |   0.315 |  13 |  13 |     2 |
hxb                           |   0.204 |   8 |   8 |  3860 |
  read                        |   0.204 |   8 | 100 |  3860 |
    EXD                       |   0.092 |   4 |  45 |   219 |
      stdgo_Go                |   0.036 |   1 |  39 |     2 |
      stdgo_math_bits_Bits    |   0.010 |   0 |  11 |     1 |
    CFD                       |   0.045 |   2 |  22 |   236 |
      stdgo_strconv_Strconv   |   0.015 |   1 |  34 |     1 |
    STR                       |   0.031 |   1 |  15 |   267 |
    MTF                       |   0.015 |   1 |   7 |   267 |
hxblib                        |   0.164 |   7 |   7 |   268 |
  get bytes                   |   0.161 |   7 |  98 |   267 |
macro                         |   0.056 |   2 |   2 |   777 |
  jit                         |   0.033 |   1 |  59 |   762 |
finalize                      |   0.017 |   1 |   1 |     1 |
analyzer                      |   0.010 |   0 |   0 |  5748 |
-------------------------------------------------------------
total                         |   2.436 | 100 | 100 | 21614 |

This is, of course, an extreme case, but shows the potential of hxb very nicely! Thanks to @AlexHaxe, we also have modified our benchmark runs to use hxb for the formatter benchmarks:

The way it works is that formatter uses --hxb and formatter_noio then has --hxb-lib to use that output. You can see the overhead of the former in the higher blue bar, and the benefit of the latter in the (much) lower one.

Closing thoughts

There will be bugs, for sure, so please report them! This has been the most complex addition to the compiler I've worked on so far, simply because of how integrated into the type-loading it is, and the type-loading itself is already the most complex part of the compiler. Fortunately, there has already been quite a bit of related "cleanup fallout" from the changes we're making here, so this PR's diff itself is not actually terribly big.

Rudy has been busy testing (and fixing) this with some real-life codebases, most importantly from Shiro Games. From what we can tell, their codebases work well with hxb, which already is a decently high bar. Thanks to him, I feel somewhat confident in this change now, although there's always a chance that even hello world won't work under some specific circumstance.

Thank you and happy hxb-testing!

[EliteMasterEric]

Additionally, the Haxe compilation cache uses hxb as its internal representation - exclusively for now. While we ultimately want to implement a hybrid memory/hxb cache for optimal performance, we need proper testing of the hxb part before we can look into that.

Interested to know, does this rework the existing HXCPP compile cache system? I was wondering about using hxb for compilation caching but if the PR does that already then that reduces the amount of work required.