(pipelines): DockerCredential.docker_hub() silently fails to auth against Docker Hub

[james-mathiesen]

Using DockerCredential.docker_hub() from CDKPipeline will silently fail to auth against Docker Hub during all stages of the build. In our case this makes ContainerImage.from_asset() intermittently fail when executed via CDKPipeline and makes the overall CI/CD process unstable.

Because most users rely on public images on Docker Hub the failure mode is subtle. In most cases users will observe only intermittent issues due to Docker Hub enforcement of pull rate limits against the IP addresses in AWS CodeBuild infrastructure.

To reliably reproduce this issue one can create a free Docker Hub account and create a private container image. Construct a pipeline that uses from_asset() to build an image that depends on the private image. Finally, supply a DockerCredential structure to CDKPipeline that is expected to allow access to the private image. This approach will reliably demonstrate that the Docker Hub credentials are not being used.

Private images on Docker Hub are an unusual use case; however using Docker Hub credentials to pull public images from Docker Hub is common practice due to Docker Hub enforcing IP address rate limits for unauthenticated users. The IP address rate limits are especially painful when building on shared infrastructure such as AWS CodeBuild.

Docker Hub offers per-user rate limits for free accounts and it is common practice to setup free accounts to support automated builds. When using shared infrastructure such as AWS CodeBuild, Docker Hub credentials are a practical necessity.

Reproduction Steps

dockerhub_secret = secretsmanager.Secret.from_secret_name_v2(
        self,
        "pull-credentials",
       "dockerhub/image-pull-credentials"
)

pipeline = CdkPipeline(
            self,
            "CICDPipeline",
            docker_credentials=[
                pipelines.DockerCredential.docker_hub(dockerhub_secret),
            ],
...

Because the common failure mode is caused by rate limit enforcement on Docker Hub I cannot provide a copy/paste test that will reliably demonstrate the problem without also sharing valid Docker Hub credentials which I do not wish to do.

What did you expect to happen?

I expected the container asset stages in AWS CodeBuild to authenticate against Docker Hub when pulling image dependencies.

By authenticating I expected to be subject to a per-user pull rate limit on Docker Hub despite the build happening on shared infrastructure in AWS CodeBuild.

What actually happened?

The builds intermittently fail due to enforcement of IP address rate limits at Docker Hub.

Environment

• CDK CLI Version : 1.115.0
• Framework Version:
• Node.js Version: v14.17.2
• **OS : AWS CodeBuild (Amazon Linux) **
• Language (Version): Python 3.9.6

Other

Authentication fails because credHelpers in Docker's config.json does not work as the Docker documentation would lead one to believe.

When pulling a public image (e.g. FROM amazon/aws-cli:2.2.18) the lookup in credHelpers is for the key https://index.docker.io/v1/. When the credential helper program is executed the domain passed to the helper is index.docker.io. The CDK incorrectly presumes that both values are index.docker.io so the credential helper is never invoked. Because most images on Docker Hub are public the lack of authentication is not obvious except when pull rate limits are exceeded.

This is a workaround to the problem:

        pipeline = CdkPipeline(
            self,
            "CICDPipeline",
            docker_credentials=[
                pipelines.DockerCredential.docker_hub(dockerhub_secret),
                pipelines.DockerCredential.custom_registry("https://index.docker.io/v1/", dockerhub_secret),
            ],

The above workaround creates two very similar entries, one to support each stage of the auth process.

The entry for https://index.docker.io/v1/ is required to cause docker to invoke the CDK credential helper.

The entry for index.docker.io is required for the CDK credential helper to find a match for the domain passed into the credential helper.

---

This is 🐛 Bug Report

[nohack]

As mentioned by @james-mathiesen looks like docker credential helper is invoked for the dockers official registry when config.json has full url.

When we login to dockerhub, the below entry is added to config.json in auths section.

  "https://index.docker.io/v1/": {
      "auth": "dGhpc2lzd2h5asdasdasda......"
  }

If registry is official they use lookup to get server url and which is this

[nohack]

I think we can pass the full url to fetchDockerLoginCredentials in docker credential cdk asset and check the domain first and fallback to url as well for special cases like this.I doubt if there are any other cases which are similar.
I just pushed the above change here.

I did not reproduce the issue in itself.

[njlynch]

Thanks for the detailed write-up @james-mathiesen , and the patch and further info @nohack! Very useful.

It looks like -- in addition to the changes @nohack has posted -- we should also update the below line to store/point to the full URL instead of the domain. That will remove the need to double-configure the entry as seen in the original post.

aws-cdk/packages/@aws-cdk/pipelines/lib/docker-credentials.ts

Line 16 in 44330ec

return new ExternalDockerCredential('index.docker.io', secret, opts);

Looks like a relatively quick fix, followed by a painstaking testing process. :)

[mpetito-envative]

This workaround doesn't appear to be working. I still receive dockerhub rate limit errors in the CDK pipeline synth step after setting up a paid Dockerhub account and corresponding secret configuration in AWS.

var dockercreds = Amazon.CDK.AWS.SecretsManager.Secret.FromSecretNameV2(this, "DockerhubSecret", "dockerhub");

var pipeline = new CodePipeline(this, "Pipeline", new CodePipelineProps
{
    ...
    DockerEnabledForSynth = true,
    Synth = new ShellStep("Synth", new ShellStepProps
    {
        ...
    }),
    DockerCredentials = new[]
    {
        DockerCredential.DockerHub(dockercreds),
        DockerCredential.CustomRegistry("https://index.docker.io/v1/", dockercreds) // https://github.com/aws/aws-cdk/issues/15737
    }
});

I've verified that the IAM Role that CodeBuild is using has access to the secret. (Note: the originally generated IAM policy did not have access because the secret arn was suffixed with dockerhub-????? for some reason.)

I also get the following output in the synth step as expected:

[Container] 2021/10/14 14:15:55 Running command echo '
{
    "version": "1.0",
    "domainCredentials": {
        "index.docker.io": {
            "secretsManagerSecretId": "arn:aws:secretsmanager:us-east-1:<account-id>:secret:dockerhub"
        },
        "https://index.docker.io/v1/": {
            "secretsManagerSecretId": "arn:aws:secretsmanager:us-east-1:<account-id>:secret:dockerhub"
        }
    }
}
' > $HOME/.cdk/cdk-docker-creds.json

The asset that fails during synth is built using the node:lts docker image as part of an S3 bucket deployment.

var source = Source.Asset(..., new Amazon.CDK.AWS.S3.Assets.AssetOptions
{
    Bundling = new BundlingOptions
    {
        Image = DockerImage.FromRegistry("node:lts"),
        Command = new[]
        {
            "bash", "-c", string.Join(" && ", new []
            {
                "npm ci",
                "npm run build",
                "cp -r /asset-input/build/* /asset-output"
            })
        }
    },
    ...
});

[bobmoff]

The workaround works for me.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[bobmoff]

The workaround works for me.

Hmm. I guess I was just lucky, as now it is not working any more.

It works for

ContainerImage.fromAsset

but it fails when using

Source.asset('xxx', { bundling: { image: DockerImage.fromBuild('xxxx'), }, }),

I wonder if it has something to do with that DockerImage.fromBuild is created during synth, and the fromAsset is created during the Asset steps.

Hmm..