Added rough guide on how to cache a docker image

[lawrencegripper]

This is a very rough guide on how to use the cache task to speed up docker builds. Would love to hear feedback on how to get this in a good state to merge.

[PRMerger17]

@lawrencegripper : Thanks for your contribution! The author(s) have been notified to review your proposed change.

[lawrencegripper]

Any thoughts on getting this merged or updated?

[ktoliver]

Hi @willsmythe - Could you review the pull request and indicate whether it can be merged? Thanks.

[willsmythe]

@johnterickson is probably the right person to review.

[ktoliver]

Hi @johnterickson - Are you able to review the PR and indicate whether it should be merged? Thanks.

Note that you don't show up as a user in the GitHub Assignees feature (top right), which indicates that your GitHub account isn't fully linked to your Microsoft account as described in Link your GitHub and Microsoft accounts; I can't assign the PR to you.

[johnterickson]

Thanks for submitting this @lawrencegripper !

Have you compared to what we have here: microsoft/azure-pipelines-tasks#11034 (comment) ?

We found "docker load" from local file to be slower than pulling it from a remote registry 😬

[lawrencegripper]

So the approach detailed here? https://github.com/fadnavistanmay/Scripts/blob/c7739a722528d7e27cc0f4f6e1adbaf4a1392977/docker.yml

In terms of simplicity I'd rather use the cache than have to setup container registries etc seems like a painful amount of admin. Competitors in this space do this nicely -> https://circleci.com/docs/2.0/docker-layer-caching/

One approach I tried get working was to identify the docker storage driver used and backup the layers directly from the /lib/docker file but the cache task wasn't running as a user with the appropriate permissions.

In terms of speed I've also observed the approach being slow - one reason appears to be because the throughput when downloading the image from cache is slower than I'd expect. I was expecting the cache to be a mounted pageblob/AzureFiles Premium onto the VM from Azure Blob rather than a download step (essentially bring the data to the node not have the node download the data).

The docker load call takes around 2mins for the 2.5GB image.

-------> Restoring docker image
Loaded image: devcontainer:latest

real	2m9.119s
user	0m0.608s
sys	0m1.735s

[stefankip]

Any updates?

[lawrencegripper]

Gonna close this one off, as there are different approaches maybe it's better suited to a blog post.

For discoverability it would be awesome to see one of the preferred approaches highlighted in the caching doc - even if only to say "Don't use the cache feature for docker images, push them to a repository instead it's better"

[stefankip]

@lawrencegripper I tried your take on caching docker layers and the buildkit one. Yours seems to only cache the first stage and not the second stage of the dockerfile I used: https://dev.azure.com/wirelab-test/Docker%20caching/_build/results?buildId=749&view=logs&j=12f1170f-54f2-53f3-20dd-22fc7dff55f9&t=efa2bfe1-554a-50c8-79b6-ef106ad3c7c2

Here is the buildkit run: https://dev.azure.com/wirelab-test/Docker%20caching/_build/results?buildId=750&view=logs&j=12f1170f-54f2-53f3-20dd-22fc7dff55f9&t=bd05475d-acb5-5619-3ccb-c46842dbc997