Umbrella Tracking Issue: Investigate and Resolve Kaniko OOM w/ multi-stage builds

[aaron-prindle]

Umbrella tracking issue for investigating and resolving Kaniko OOM w/ multi-stage builds. Steps include:

• Research root cause and develop repro case w/ github repo
• Implement fix for identified root cause (eg: memory leak, symlink cycles, etc.)

[aaron-prindle]

Related: #2033

[JeromeJu]

Thanks to the pointers Aaron. I've tried screening on the following issues:

Hard link issue that causes extra memory usage:
#1743

Overflow in filename that leads to cannot save file error caused by recursion related with otiai10Cpy.Copy when copy from "/", this now fails with filename too long instead of draining all the space. [update] this is a bug that needs to be fixed, but probably not in scope of memory usage related.

• Multi stage build ends with "no space left on device", no matter the free space you provide #2033
• kaniko cannot use scratch multi-stage build image #2787

Copy from /kaniko dir

• Better support for building custom Kaniko images with Kaniko #1881
• Kaniko recursion error while building docker image #2210

OOM:

• Multi-stage builds silently crashing #2249

[aaron-prindle]

This should be ok to close out now with the merge of #2863. Closing