reduce binary size #17
Comments
|
From initial cursory investigation of Cronet, the library size is in the single digits Mb. Not suggesting anything necessarily, just putting out there a point of reference. |
|
I hacked up a basic binary just to see the effect of linking in Creating the binary took this patch: Just compiling a noop program that exits immediately yields 8.0K binary size: However, including the Stripping any unnecessary symbols cuts the size roughly in half: Bloaty output on the stripped binary: More analysis to come... |
|
Compiling with Stripping this binary takes off another 6MB: Next step is to remove unused objects from the binary (from unneeded includes). |
|
@tonya11en surprised it is this big. Can you confirm we are compiling without any extensions as well as compiling out hot restart, google gRPC, etc.? |
|
We have an extensions file for this repo ( There's also an Envoy extensions file in |
|
Looks like running the bazel build with However, Envoy and BoringSSL are being built with Envoy: BoringSSL: |
|
Made some changes to Envoy OSS to generate a size-optimized binary. Looks like it works. Here are the differences from the same Envoy SHA: This is a decent starting point for further analysis and measurement of effects of various changes. Here's a list of where I'd like to go next:
|
|
@tonya11en awesome work. Yes, please compile out all the things we are compiling out in mobile as a base point (please sync with @junr03 and @goaway on this). I would also love to see a per-object file breakdown of size. |
|
Learned this morning that setting Once envoyproxy/envoy#6960 gets merged in, that'll be a good starting point to begin breaking down the space usage of the library per-object as Matt mentioned. |
|
Uploading granular objdump of the size-optimized envoy lib for future analysis. |
|
@tonya11en any thoughts on how to:
cc @Reflejo who I know has done stuff like this in the past and can probably advise. @Reflejo I wonder if any of the scripts you wrote for LBS would be useful here? |
|
Looks like I can use bloaty to get the per-file breakdown. Building everything with the symbols now. I'll give an update today with the result. |
|
Attaching a by-file breakdown of the binary size contributions for an envoy lib built at Envoy commit I was also able to generate a breakdown of inlined code, but the text file is >70MB, so I will not attach. I'll follow up with some initial analysis later today or tomorrow morning. |
|
Thanks @tonya11en this is awesome. Is it possible to also generate a report filtered by directory? I think that would be useful just to look at entire things we might be able to get rid of. |
|
(Also, could you document/commit somewhere how to generate this data) |
|
Documenting this process in https://github.com/lyft/envoy-edge-dev/pull/95. I will transfer the document over to the envoy-mobile repo as once the organization of the code becomes apparent. |
|
I picked up where @tonya11en left off. I ran the same binary he was creating with a custom bloaty data source: This starts allowing us to aggregate size by files. The command ran was:
The binaries were created with:
Results: test_initial_06_19.txt Coarse Interesting observations:
Next steps:
|
06/20 updateWhat was done
iOSI built the Envoy.framework with the following command: Using this I can get size information of the composing object files with However, when I tried to get symbols from the object files I was not able to do so. e.g: although as far as I can tell the object file does have debug symbols: At this point I stopped working on iOS as having the debug symbols is important to actually see what is going on. @keith did mention I could try using Open questions:
AndroidI moved on to android where I built the envoy.aar with I also created an optimized stripped envoy.aar with The bloaty output is: Given that baseline I went in an deleted all logging (with ifdefs, and commenting some stuff out). The diff between libenvoy_jni.so was: So ~500k reduction by removing logging. To showcase a question that @mattklein123 has asked before. Using the dbg libenvoy_jni.so we can see what source files are larger in the envoy codebase: Open Questions:
|
|
@keith @Reflejo @mattklein123 @goaway let me know what you think of the progress above. Also please comment if you have other ideas at pieces of data to look at, or next steps to take. This is the first time I am running an investigation of this sort (beyond very small toy projects), so your prior expertise is very valuable. At this point I think that using the Android data compiled against arm, (albeit we have symbol resolution only in the -c dbg output), we probably have to make some calls about larger parts of envoy we can start compiling out. |
|
@junr03 A few quick observations:
$ otool -l bazel-out/darwin-fastbuild/bin/library/common/test_binary.dead
Section
sectname __cstring
segname __TEXT
addr 0x00000001004bf0a0
size 0x0000000000048f45
offset 4976800
align 2^4 (16)
(...)
$ dd if=bazel-out/darwin-fastbuild/bin/library/common/test_binary.dead bs=1 skip=4976800 count=`rax2 0x48f45` of=whatevthis is not a big deal because deflate compression on this section will be fairly effective. But this is obviously a low hanging fruit I think there are ~300K of these (on my stripped version of the binary).
- I will try to do a deeper analysis but please don't block on me. Hope this gives you more ideas where to look. |
|
Thanks @junr03 and @Reflejo. A couple of things from me:
@junr03 my advice is to actually just schedule 30-45 minutes with @Reflejo as I think the time will be well worth it. I can sit with you also for a bit once you do that, but I mainly want to make sure we have the right tooling and outputs in place to begin measurement. I don't think it's actually worth looking at where the space is being spent until we do that. Thank you! |
|
(You should be able to get full symbols on top of the opt build via
|
|
Thanks for the comments @Reflejo and @mattklein123 I do agree with you that the most representative way to look at the binary is by using the So I think the basic methodology we should use is (paraphrasing Matt's proposal and adding my own thoughts):
Then make changes and repeat, measuring deltas. Questions for you:
Let me know what you think. Once I have this down, and can iterate efficiently we can pair for a bit to follow up on your ideas (e.g the exception idea). I can schedule some time for monday @Reflejo. |
In general yes, but we need to also include tooling to actually see where the size is being spent. I would not worry about making any changes right now. Once we start making changes, yes, we need to definitely measure the deltas and keep track of the current size in CI so we don't regress without knowing it.
Sure, this would be optimal, but I thought there was some issue here. I'm also not sure why we can't effectively just do this with the iOS/Android targets but I don't know the details. Can we time box getting this working and circle back? I do know that many people in the community are compiling Envoy for ARM, but I think they do this on an ARM host vs. cross-compile, but I'm not sure. cc @moderation.
Yes, but let's not worry about changes right now. Let's get the methodology down on how to compile and measure what we have, and present a tabular view of where the size is being spent. |
Right now the tool that has given me most information is bloaty. However, part of the time I will setup with @Reflejo is to learn what his toolkit is for investigations like this. I will document so that it is repeatable in the future.
Yep, that is what I am doing right now. Won't extend further than this afternoon. I chatted with @moderation yesterday and they are compiling on native hosts. I chatted with @keith about cross compiling via bazel on x86 and he advised that was a rough path. The last path I am exploring today is Docker, which arguable supports multi-arch out of the box. |
|
If it really comes down to it just buy a Raspberry Pi or similar to do the investigation on, but it seems like that shouldn't be needed. |
|
Also, I don't know what bloaty is, but all of the data should be available via objdump and related tools, and can be scripted as needed. |
|
Hopefully @junr03 investigation into cross compilation pays off but if not I'm using a Pine64 RockPro64 SOC - https://www.pine64.org/rockpro64/. 4G board is $80. You'll want the big heat sink. Has enough memory and cores to compile stuff like Bazel, Envoy, CockroachDB etc. |
|
Not sure if it helps - the binary I built about 1 year ago - https://github.com/costinm/istio-build/releases - is 5M for android/arm (stripped). Maybe we can compare. Looking at the arm build for android - it is a bit smaller (#153) |
|
I'm not sure how many of the libraries truly need to be force loaded, but this blocks the |
Signed-off-by: Jose Nino jnino@lyft.com Description: This PR concludes binary size investigation slated for issue #17. The three deliverables of this PR are: 1. Developer documentation that solidifies the building and analysis platform used for binary size analysis. 2. A list of issues for next steps in binary size reduction under the perf/size label. 3. A final baseline size for the binary: As of https://github.com/lyft/envoy-mobile/tree/f17caebcfce09ec5dcda905dc8418fea4d382da7 The test_binary_size_size as built by the toolchain against the architecture described (arm64 with clang and lld) compiles to a stripped size of 8.9mb and a compressed size of 3mb. Additionally #181 will add CI jobs to add size regression analysis on every PR. Risk Level: low - add new bazel target and docs. Docs Changes: added developer documentation. Fixes #17
In order for us to be able to ship Envoy on the client we need to optimize for binary size.
We will need to reduce Envoy Mobile’s final application bloat (compressed) down to < 5MB to be able to ship in a production mobile application
The text was updated successfully, but these errors were encountered: