LLVM Custom Module Linking: Add to CMake or remove from Make

[steven-johnson]

Our Makefile has special logic to link in the minimum parts of LLVM that we need, in order to minimize the size of libHalide. ("It made a large difference at the time we did it.")

We should evaluate current sizes of CMake vs Make libHalide results and see if the size difference is still worthwhile; if it isn't, we should remove it from Make (to simplify the world); if it is, we should add it to CMake.

[steven-johnson]

(On a related note: this technique relies heavily on the llvm-config tool, which is apparently considered somewhat deprecated by the LLVM team.)

[alexreinking]

which is apparently considered somewhat deprecated by the LLVM team

Do you have a reference for that?

[steven-johnson]

I don't recall, but I think @zvookin might

[alexreinking]

I know that you're not supposed to call that tool from CMake and they try to issue a warning if you do.

[shoaibkamil]

CMake currently does not produce a Halide static library that is usable without also having LLVM static libraries available. So, I think there's a few questions we need to figure out:

1. Do we want to produce static libHalide.a that includes the required LLVM objects?
2. Do we want to include only the minimal set of objects required for that?

I would really like the answer for (1) to be yes, but I'm indifferent about the size of the resulting library. Being able to use a self-contained libHalide.a is really useful.

[steven-johnson]

I think step one is to measure the current size difference in libHalide.so between Make and CMake and see if it's something that matters. (I'm assuming that the size of libHalide.a matters much less, as presumably a decent downstream linker will prune things appropriately.)

[alexreinking]

CMake currently does not produce a Halide static library that is usable without also having LLVM static libraries available. So, I think there's a few questions we need to figure out:

Correct, and it has never before.

1. Do we want to produce static libHalide.a that includes the required LLVM objects?

There is no built-in support for this using CMake because it is very platform and linker dependent. I did find a blog post that puts some effort into adding this functionality: https://cristianadam.eu/20190501/bundling-together-static-libraries-with-cmake/

But more natural would be to package the LLVM static libraries that we actually use alongside the Halide static library and add a dependency inside CMake on that.

2. Do we want to include only the minimal set of objects required for that?

As Steven said, the actual size difference is worth investigating. There are also unanswered questions regarding our blanket export of symbols, including inlined template instantiations, see #4651

Being able to use a self-contained libHalide.a is really useful.

Why? Because you can add just "/path/to/halide/libHalide.a" to the linker line? Wouldn't "/path/to/halide/libHalide.a /path/to/halide/LLVM*.a" be just about as easy?

[abadams]

Corporate users have custom weird build environments that are idiotic in a wide variety of ways. Single file dependencies can be workable where directories full of them are not depending on the specific flavor of idiocy.

[shoaibkamil]

On my macOS machine, with LLVM10 from homebrew, built with make:

-rwxr-xr-x 1 kamil staff 80M May 28 15:11 libHalide.dylib

built with cmake (same LLVM install):

-rwxr-xr-x 1 kamil staff 83M May 28 15:14 libHalide.dylib

[alexreinking]

Corporate users have custom weird build environments that are idiotic in a wide variety of ways. Single file dependencies can be workable where directories full of them are not depending on the specific flavor of idiocy.

Thanks for that context -- good to know.

[alexreinking]

So that's a ~3-4% difference? Doesn't sound huge to me. Could get a similar fluctuation based on the LLVM version, now?

Something worth looking into: https://github.com/google/bloaty

[alexreinking]

I get 90MB (86MB with Make) when compiling in Release mode against LLVM 9 on CMake / Ubuntu 18.04. Bloaty returns the following:

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  38.8%  34.6Mi  42.4%  34.6Mi    .text
  17.5%  15.6Mi  19.1%  15.6Mi    .data
  15.5%  13.8Mi  17.0%  13.8Mi    .rodata
   6.9%  6.13Mi   0.0%       0    .strtab
   4.8%  4.33Mi   5.3%  4.33Mi    .data.rel.ro
   4.6%  4.10Mi   5.0%  4.10Mi    .rela.dyn
   3.3%  2.95Mi   3.6%  2.95Mi    .dynstr
   3.1%  2.79Mi   3.4%  2.79Mi    .eh_frame
   2.2%  1.95Mi   0.0%       0    .symtab
   1.0%   953Ki   1.1%   953Ki    .dynsym
   0.5%   459Ki   0.5%   459Ki    .rela.plt
   0.5%   441Ki   0.5%   441Ki    .eh_frame_hdr
   0.0%       0   0.4%   367Ki    .bss
   0.3%   317Ki   0.4%   317Ki    .gnu.hash
   0.3%   308Ki   0.4%   308Ki    .gcc_except_table
   0.3%   306Ki   0.4%   306Ki    .plt
   0.2%   153Ki   0.2%   153Ki    .got.plt
   0.1%  79.5Ki   0.1%  79.4Ki    .gnu.version
   0.0%  30.6Ki   0.0%  30.5Ki    .got
   0.0%  6.00Ki   0.0%  2.29Ki    [14 Others]
   0.0%  4.54Ki   0.0%  4.46Ki    .init_array
 100.0%  89.3Mi 100.0%  81.6Mi    TOTAL

[abadams]

Some numbers from ubuntu:

All of LLVM's static libraries add up to 202 MB

cmake:
libHalide.a without LLVM is 44MB.
libHalide.so without LLVM is 113MB (or does this include LLVM?)
after stripping libHalide.so is 103M

make:
The LLVM object files we actually use add up to 191MB (i.e. nearly all of it)
libHalide.so with LLVM is 103MB (so yeah, looks like that 10MB was saved)
after stripping libHalide.so is 93MB
libHalide.a with LLVM is 210MB (not sure why this is less than 44 + 191)

[abadams]

So stripping llvm down to only the object files we use doesn't really matter anymore. Doing it is not really any harder than just concatenating static libraries, so no reason to disable it. But it's definitely not a necessary feature for a bundled libHalide.a

[steven-johnson]

Doing it is not really any harder than just concatenating static libraries, so no reason to disable it.

Well, yes and no: the recent hack I had to insert suggests that there may be brittleness here that will break us again at some point. I'm not in a rush to rip it out of Makefile, but IMHO this data suggests that if we encounter more issues of this sort in the future, we might consider doing so.

[abadams]

That hack wasn't related to the object file enumeration though was it? I thought that hack was related to the list of llvm static libs we include. We could just include all of them and assume the object file enumeration drops the unnecessary ones.

[alexreinking]

Using bloaty suggests that the difference is all in the .data section.

alex@alex-ubuntu:~/Development/Halide-Release$ ../bloaty-build/bloaty ../Halide/cmake-build-release/src/libHalide.so -- bin/libHalide.so
    FILE SIZE        VM SIZE    
 --------------  -------------- 
   +34% +3.98Mi   +34% +3.98Mi    .data
  [NEW]  +308Ki  [NEW]  +308Ki    .gcc_except_table
  +4.4%  +121Ki  +4.4%  +121Ki    .eh_frame
  -0.7%     -45  -0.7%     -47    [6 Others]
  -0.3%    -480  -0.3%    -480    .got.plt
  -0.8%    -626  -0.8%    -626    .gnu.version
 -25.1%    -919  [ = ]       0    [Unmapped]
  -0.3%    -960  -0.3%    -960    .plt
  -3.3% -1.04Ki  -3.3% -1.04Ki    .got
  -0.4% -1.21Ki  -0.4% -1.21Ki    .gnu.hash
  -0.3% -1.41Ki  -0.3% -1.41Ki    .rela.plt
  [ = ]       0  -0.6% -2.09Ki    .bss
  -0.9% -4.00Ki  -0.9% -4.00Ki    .eh_frame_hdr
  -0.8% -7.34Ki  -0.8% -7.34Ki    .dynsym
  -0.8% -25.2Ki  -0.8% -25.2Ki    .dynstr
  -1.4% -29.3Ki  [ = ]       0    .symtab
  -0.1% -37.5Ki  -0.1% -37.5Ki    .text
  -1.3% -84.3Ki  [ = ]       0    .strtab
  -2.4%  -108Ki  -2.4%  -108Ki    .data.rel.ro
  -2.8%  -122Ki  -2.8%  -122Ki    .rela.dyn
  -1.4%  -207Ki  -1.4%  -207Ki    .rodata
  +4.4% +3.78Mi  +5.0% +3.89Mi    TOTAL

[alexreinking]

I noticed that ".gcc_except_table" was new, which led me to conclude that the makefile builds without exceptions by default. CMake matches whatever LLVM reports. Recompiling with make WITH_EXCEPTIONS=1 adds a megabyte to the binary.

    FILE SIZE        VM SIZE    
 --------------  -------------- 
   +34% +3.98Mi   +34% +3.98Mi    .data
  +2.6% +73.1Ki  +2.6% +73.1Ki    .eh_frame
  +707% +2.34Ki  [ = ]       0    [Unmapped]
  -0.8%     -48  -0.5%     -32    [4 Others]
  -3.4% -1.09Ki  -3.4% -1.09Ki    .got
  -1.7% -1.38Ki  -1.7% -1.38Ki    .gnu.version
  -0.7% -2.08Ki  -0.7% -2.08Ki    .gcc_except_table
  [ = ]       0  -0.6% -2.09Ki    .bss
  -0.8% -2.67Ki  -0.8% -2.67Ki    .gnu.hash
  -1.8% -2.77Ki  -1.8% -2.77Ki    .got.plt
  -1.8% -5.53Ki  -1.8% -5.53Ki    .plt
  -1.8% -8.03Ki  -1.8% -8.03Ki    .eh_frame_hdr
  -1.8% -8.30Ki  -1.8% -8.30Ki    .rela.plt
  -1.7% -16.5Ki  -1.7% -16.5Ki    .dynsym
  -2.2% -45.8Ki  [ = ]       0    .symtab
  -1.9% -59.7Ki  -1.9% -59.7Ki    .dynstr
  -2.4%  -108Ki  -2.4%  -108Ki    .data.rel.ro
  -2.8%  -122Ki  -2.8%  -122Ki    .rela.dyn
  -2.1%  -132Ki  [ = ]       0    .strtab
  -1.4%  -207Ki  -1.4%  -207Ki    .rodata
  -1.7%  -604Ki  -1.7%  -604Ki    .text
  +3.2% +2.76Mi  +3.7% +2.93Mi    TOTAL

[alexreinking]

After #4978 the two builds are within 1MB of each other for me with or without exceptions.

[acolinisi]

FWIW, size when libHalide.so is linked against LLVM shared library (see PR #5015 for patch I needed to built this way):

$ bloaty /usr/lib64/libHalide.so
     VM SIZE                         FILE SIZE
 --------------                   --------------
  51.9%  12.0Mi .data              12.0Mi  52.1%
  32.4%  7.51Mi .text              7.51Mi  32.6%
   3.4%   797Ki .dynstr             797Ki   3.4%
   2.6%   622Ki .eh_frame           622Ki   2.6%
   2.4%   575Ki .gcc_except_table   575Ki   2.4%
   2.0%   480Ki .rela.dyn           480Ki   2.0%
   1.7%   398Ki .rodata             398Ki   1.7%
   0.9%   224Ki .dynsym             224Ki   1.0%
   0.6%   151Ki .data.rel.ro        151Ki   0.6%
   0.5%   107Ki .rela.plt           107Ki   0.5%
   0.4%  88.2Ki .bss                    0   0.0%
   0.3%  71.8Ki .plt               71.8Ki   0.3%
   0.3%  67.1Ki .eh_frame_hdr      67.1Ki   0.3%
   0.2%  55.5Ki .gnu.hash          55.5Ki   0.2%
   0.2%  35.9Ki .got.plt           35.9Ki   0.2%
   0.1%  18.7Ki .gnu.version       18.7Ki   0.1%
   0.0%  9.33Ki .got               9.33Ki   0.0%
   0.0%      23 [Unmapped]         5.31Ki   0.0%
   0.0%  2.38Ki [Other]            2.60Ki   0.0%
   0.0%     512 [ELF Headers]      2.12Ki   0.0%
   0.0%       8 [None]                  0   0.0%
 100.0%  23.1Mi TOTAL              23.1Mi 100.0%

$ ldd /usr/lib64/libHalide.so | grep LLVM
        libLLVM-10.so => /usr/lib/llvm/10/lib64/libLLVM-10.so (0x00007f0c74809000)

[alexreinking]

Note that with #5135 Halide's CMake build has gained the ability to bundle the LLVM static libraries inside libHalide.a (and also libHalide.lib as long as you're in Release mode!). We should see what the differences are, if any.