Caching does not work with ARG inside the Dockerfile #1910

apollo13 · 2022-02-02T13:06:15Z

Note to maintainers: Attached a patch in #1915 -- not sure how to test that though, hints welcome.

Actual behavior
I am running kaniko with --cache-copy-layers=true and it fails to utilize the cache in certain situations.

Expected behavior
I expect kaniko to reuse the cache, it seems as if the cache ids are generated wrongly

To Reproduce

Create the following dockerfile:

FROM alpine as build
ARG TEST=123
RUN apk add --no-cache tar

Run the following command multiple times and observe the cache not being used:

docker run -v `pwd`:/workspace gcr.io/kaniko-project/executor:bde904349e9b7797d14f12c7a2657bf79145d18e-debug --dockerfile /workspace/Dockerfile --context dir:///workspace/ --destination 10.0.0.6:5000/test --cache=true --cache-copy-layers=true

Additional Information

Dockerfile: See above
Build Context: empty aside from the Dockerfile
Kaniko Image (fully qualified with digest) gcr.io/kaniko-project/executor:bde904349e9b7797d14f12c7a2657bf79145d18e-debug (a commit from yesterday)

Looking at the debug output the cache key changes:

DEBU[0000] Skip resolving path /cache                   
DEBU[0001] No file found for cache key sha256:e7d88de73db3d3fd9b2d63aa7f447a10fd0220b7cbf39803c803f2af9ba256b3 stat /cache/sha256:e7d88de73db3d3fd9b2d63aa7f447a10fd0220b7cbf39803c803f2af9ba256b3: no such file or directory 
DEBU[0001] Image alpine not found in cache              
INFO[0001] Returning cached image manifest              
INFO[0002] Returning cached image manifest              
DEBU[0002] No file found for cache key sha256:e7d88de73db3d3fd9b2d63aa7f447a10fd0220b7cbf39803c803f2af9ba256b3 stat /cache/sha256:e7d88de73db3d3fd9b2d63aa7f447a10fd0220b7cbf39803c803f2af9ba256b3: no such file or directory 
DEBU[0002] Image alpine not found in cache              
INFO[0002] Returning cached image manifest              
DEBU[0002] optimize: composite key for command ARG TEST=123 {[sha256:e7d88de73db3d3fd9b2d63aa7f447a10fd0220b7cbf39803c803f2af9ba256b3 |1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ARG TEST=123]} 
DEBU[0002] optimize: cache key for command ARG TEST=123 186ec9fee220d1452a3af884e27fbb20dd3ef21eca7b68aaa316b058359fe73d 
DEBU[0002] optimize: composite key for command RUN apk add --no-cache tar {[sha256:e7d88de73db3d3fd9b2d63aa7f447a10fd0220b7cbf39803c803f2af9ba256b3 |1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ARG TEST=123 |2 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TEST=123 RUN apk add --no-cache tar]} 
DEBU[0002] optimize: cache key for command RUN apk add --no-cache tar a281ac04c35f422a7956387c1d32787139a79b22a87edd9f7933fbdd3fd65819 
INFO[0002] Checking for cached layer 10.0.0.6:5000/test/cache:a281ac04c35f422a7956387c1d32787139a79b22a87edd9f7933fbdd3fd65819... 
DEBU[0002] Failed to retrieve layer: GET http://10.0.0.6:5000/v2/test/cache/manifests/a281ac04c35f422a7956387c1d32787139a79b22a87edd9f7933fbdd3fd65819: MANIFEST_UNKNOWN: manifest unknown; map[Tag:a281ac04c35f422a7956387c1d32787139a79b22a87edd9f7933fbdd3fd65819] 
INFO[0002] No cached layer found for cmd RUN apk add --no-cache tar 
DEBU[0002] Key missing was: sha256:e7d88de73db3d3fd9b2d63aa7f447a10fd0220b7cbf39803c803f2af9ba256b3-|1-PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin-ARG TEST=123-|2-PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin-TEST=123-RUN apk add --no-cache tar 
INFO[0002] Unpacking rootfs as cmd RUN apk add --no-cache tar requires it. 
DEBU[0003] build: composite key for command ARG TEST=123 &{[sha256:e7d88de73db3d3fd9b2d63aa7f447a10fd0220b7cbf39803c803f2af9ba256b3 |2 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TEST=123 ARG TEST=123]} 
DEBU[0003] build: cache key for command ARG TEST=123 6c4bf2e8db648ae9c9b1db201aff474c5302bddb61b2444debd20ae0a00ebdb1 
INFO[0003] RUN apk add --no-cache tar                   
INFO[0003] args: [-c apk add --no-cache tar]            
INFO[0003] Running: [/bin/sh -c apk add --no-cache tar] 
DEBU[0004] build: composite key for command RUN apk add --no-cache tar &{[sha256:e7d88de73db3d3fd9b2d63aa7f447a10fd0220b7cbf39803c803f2af9ba256b3 |2 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TEST=123 ARG TEST=123 |2 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TEST=123 RUN apk add --no-cache tar]} 
DEBU[0004] build: cache key for command RUN apk add --no-cache tar a4329b9603a45f299011c221612a695286390059448babb87652103053db4209 
INFO[0004] Pushing layer 10.0.0.6:5000/test/cache:a4329b9603a45f299011c221612a695286390059448babb87652103053db4209 to cache now 
INFO[0004] Pushing image to 10.0.0.6:5000/test/cache:a4329b9603a45f299011c221612a695286390059448babb87652103053db4209 
INFO[0004] Pushed 10.0.0.6:5000/test/cache@sha256:ba36e231d062103b990c1106648f707f841039c6ade8cbb435b045af88479548 
DEBU[0004] mapping digest sha256:e53ebcc21ce659d505b6b9bf7fd251af6d118cf420abbc0ce94ec825b8de9a0f to cachekey a281ac04c35f422a7956387c1d32787139a79b22a87edd9f7933fbdd3fd65819

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use `--cache` flag
Please check if your dockerfile is a multistage dockerfile

The text was updated successfully, but these errors were encountered:

apollo13 · 2022-02-02T13:35:44Z

Comparing the composite keys I see:

|1 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ARG TEST
|2 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TEST=123 RUN apk add --no-cache tar

vs

|2 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TEST=123 ARG TEST
|2 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TEST=123 RUN apk add --no-cache tar

I am pretty sure that TEST=123 is wrong, this should always be only set after the ARG command ran.

…oogleContainerTools#1910 This seems to be required because otherwise build & optimize calculate different caches keys.

apollo13 · 2022-02-02T14:01:20Z

For my simple docker file this fix works:

diff --git a/pkg/executor/build.go b/pkg/executor/build.go
index 995c6737..45b3cae7 100644
--- a/pkg/executor/build.go
+++ b/pkg/executor/build.go
@@ -242,6 +242,13 @@ func (s *stageBuilder) optimize(compositeKey CompositeCache, cfg v1.Config) erro
                        return errors.Wrap(err, "failed to get files used from context")
                }
 
+               // Mutate the config for any commands that require it.
+               if command.MetadataOnly() {
+                       if err := command.ExecuteCommand(&cfg, s.args); err != nil {
+                               return err
+                       }
+               }
+
                compositeKey, err = s.populateCompositeKey(command, files, compositeKey, s.args, cfg.Env)
                if err != nil {
                        return err
@@ -272,13 +279,6 @@ func (s *stageBuilder) optimize(compositeKey CompositeCache, cfg v1.Config) erro
                                s.cmds[i] = cacheCmd
                        }
                }
-
-               // Mutate the config for any commands that require it.
-               if command.MetadataOnly() {
-                       if err := command.ExecuteCommand(&cfg, s.args); err != nil {
-                               return err
-                       }
-               }
        }
        return nil
 }

Since build() runs optimize it sees all the results of the metacommands even if the did not run yet. I think an actually fix needs to do even more, but I am still digging through.

apollo13 · 2022-02-02T14:04:40Z

Yeah my fix breaks down as soon as I have the following dockerfile:

FROM alpine as build
ARG TEST=123
ARG TEST2=234
RUN apk add --no-cache tar

with the following difference in the composite key:

{[
sha256:e7d88de73db3d3fd9b2d63aa7f447a10fd0220b7cbf39803c803f2af9ba256b3 
|2 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TEST=123 ARG TEST=123 
|3 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TEST=123 TEST2=234 ARG TEST2=234 
|3 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TEST=123 TEST2=234 RUN apk add --no-cache tar
]}

vs

{[
sha256:e7d88de73db3d3fd9b2d63aa7f447a10fd0220b7cbf39803c803f2af9ba256b3 
|3 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TEST=123 TEST2=234 ARG TEST=123 
|3 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TEST2=234 TEST=123 ARG TEST2=234 
|3 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TEST=123 TEST2=234 RUN apk add --no-cache tar
]}

…oogleContainerTools#1912.

apollo13 added a commit to apollo13/kaniko that referenced this issue Feb 2, 2022

Execute commands in optimize before updating the composite key. Fixes G…

4857eee

…oogleContainerTools#1910 This seems to be required because otherwise build & optimize calculate different caches keys.

apollo13 mentioned this issue Feb 2, 2022

Kaniko allows to access undefined variables when cache is enabled #1912

Open

apollo13 added a commit to apollo13/kaniko that referenced this issue Feb 2, 2022

Restore build args after optimize. Fixes GoogleContainerTools#1910, G…

0c3b97d

…oogleContainerTools#1912.

apollo13 linked a pull request that will close this issue Feb 2, 2022

Restore build args after optimize. Fixes #1910, #1912. #1915

Draft

4 tasks

GoogleContainerTools / kaniko Public

Caching does not work with ARG inside the Dockerfile #1910

Caching does not work with ARG inside the Dockerfile #1910

apollo13 commented Feb 2, 2022 •

edited

apollo13 commented Feb 2, 2022 •

edited

apollo13 commented Feb 2, 2022

apollo13 commented Feb 2, 2022

GoogleContainerTools / kaniko Public

Caching does not work with ARG inside the Dockerfile #1910

Caching does not work with ARG inside the Dockerfile #1910

Comments

apollo13 commented Feb 2, 2022 • edited

apollo13 commented Feb 2, 2022 • edited

apollo13 commented Feb 2, 2022

apollo13 commented Feb 2, 2022

apollo13 commented Feb 2, 2022 •

edited

apollo13 commented Feb 2, 2022 •

edited