(core): Allow passing platform via BundlingOptions

[blimmer]

Describe the feature

It would be nice to be able to enforce the platform that any arbitrary docker bundling command uses via the BundlingOptions property.

Currently, this is not possible when using an existing Docker image via the BundlingOptions. You can specify this when building an image, but not when using an image as a builder:

aws-cdk/packages/aws-cdk-lib/aws-ecr-assets/lib/image-asset.ts

Lines 247 to 252 in 784cd0e

	/**
	* Platform to build for. _Requires Docker Buildx_.
	*
	* @default - no platform specified (the current machine architecture will be used)
	*/
	readonly platform?: Platform;

Use Case

We make use of the S3 Deployment BucketDeployment construct to package a Virtual Environment dependency zipfile for EMR Serverless (docs).

The process to produce the asset looks like this:

pyspark_env_asset = Source.asset(
    emr_job_root,
    asset_hash=FileSystem.fingerprint(
        str(Path(emr_job_root) / "requirements.txt")
    ),
    bundling=BundlingOptions(
        image=DockerImage.from_registry(
            f"REDACTED.dkr.ecr.us-west-2.amazonaws.com/data-services:emr-serverless-base-python-3-9-16"
        ),
        environment={
            "PIP_INDEX_URL": f"https://aws:{extract_code_artifact_token()}@REDACTED.d.codeartifact.us-west-2.amazonaws.com/pypi/REDACTED/simple/",
        },
        user="root",
        command=[
            "/bin/bash",
            "-c",
            " && ".join(
                [
                    "yum install zip -y",
                    "python3.9 -m venv venv --clear --copies",
                    "source venv/bin/activate",
                    "cp -r /usr/local/lib/python3.9/* ./venv/lib/python3.9/",
                    "python3.9 -m pip install --upgrade pip",
                    "pip install -r requirements.txt",
                    # package venv to archive.
                    # **Note** that you have to supply --python-prefix option
                    # to make sure python starts with the path where your
                    # copied libraries are present.
                    # Copying the python binary to the "environment" directory.
                    "venv-pack --quiet -f -o /tmp/pyspark_env.tar.gz --python-prefix /home/hadoop/environment",
                    # CDK assets are a bit weird - we need to zip up the tarball to make sure our desired name is retained in the
                    # final S3 Bucket.
                    "cd /tmp",
                    "zip /asset-output/venv-output.zip ./pyspark_env.tar.gz",
                ]
            ),
        ],
    ),
)

The issue is that our emr-serverless-base-python-3-9-16 is a Multi-Platform image. The EMR serverless runtime environment is linux/amd64, and our developer machines use linux/arm64 (Apple Silicon) processors. When we run the build from our local machines, Docker will automatically select the linux/arm64 variant. Therefore, as it stands, building the image from our local machines produces incompatible venvs.

Proposed Solution

Expose a platform option in the BundlingOptions property. When specified, the underlying docker run should use the platform flag as specified in the BundlingOptions.platform property.

Other Information

For now, I can work around this by using the DockerImage.from_build function which accepts a platform parameter.

So the image specification changes to look like this:

bundling=BundlingOptions(
    image=DockerImage.from_build(
        str(Path(__file__).parent / "emr_serverless_docker"),
        platform="linux/amd64"
    ),
	# everything else from the snippet above remains the same
)

The emr_serverless_docker folder contains one file (Dockerfile):

FROM --platform=linux/amd64 REDACTED.dkr.ecr.us-west-2.amazonaws.com/data-services:emr-serverless-base-python-3-9-16

(I'm not 100% sure I need to have the --platform specified here again, but I did it just to be safe).

This works and ensures that the linux/amd64 version is always selected no matter the host p

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

CDK version used

2.80.0

Environment details (OS name and version, etc.)

MacOS

[pahud]

related to #12472

Yes we should add this support. In fact I was trying to implement that but unfortunately didn't make it.

Making this as a p1 feature request. Any community PRs would be highly welcome and appreciated.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.