document more clearly what the purpose of the --use-new-run feature is and how it works

[0x217]

How does the feature --use-new-run work?

I will be happy to explain :)
I didn't find information in the documentation.

[0x217]

JUMP

[aaron-prindle]

Unfortunately the maintainer who initally worked on --use-new-run as an experimental run mode no longer maintains the project. From poking around past PRs and going through the code here is what I have found:

Initial PR adding --use-new-run:
#1300

PR fixing some logic
#1379

Issue/Comment explaining --use-new-run logic:
#1305 (comment)
#1305 (comment)

(from #1305 (comment))

(The flag --use-new-run is a ) ... newer implementation of Run command which does not rely on snapshotting at all.
In this approach, in order to compute which files were changed, by creating a marker file before executing the Run command.
Then we walk the entire filesystem which takes about 1 to 3 seconds for 700Kfiles, to find all files whose ModTime is greater than the marker file.
With this new run command implementation, the total build time is reduced for 29 mins to 7 mins with a huge improvement of 75%

From looking at the code, here's how it works:

It runs a "marker" command like touch filemarker to create a temp file and get its mtime.
It then sleeps for 1 second to introduce a delay.
It executes the actual RUN command from the Dockerfile.
It uses find to walk the filesystem and detect any files newer than the marker file. These are the files changed by the RUN command.
It takes a snapshot tar of just these detected files.

The key differences vs the normal snapshotting behavior:
It doesn't take a snapshot of the full filesystem before the RUN command. This avoids the filesystem traversal on each RUN.
It only snapshots the files detected to have changed, not everything. This makes the snapshots much smaller.
It relies on the introduced delay to detect changes made by the command. Some file changes could potentially be missed.

So in summary, it trades off some accuracy (potential for missed files) for improved performance by avoiding the full filesystem snapshots. The resulting image should be the same, just built faster in most cases.

[aaron-prindle]

I will change this issue to be around updating the documentation of --use-new-run to include this additional information so other users don't encounter the same friction. Thanks for flagging this

[ariefrahmansyah]

Hi @aaron-prindle, thanks for the explanation!

I have follow-up questions as I'm still figuring out how --use-new-run and --snapshot-mode work together.

What happens if we use both --use-new-run and --snapshot-mode at the same time? Is the --snapshot-mode only being used to determine how to snapshot files detected by the new run? For example, from your explanation and my understanding, if I use --use-new-run and --snapshot-mode=full, will Kaniko still download the snapshot from cache and will Kaniko snapshot the detected files by new-run using full mode?