Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(aws-fargate-s3): Created new construct #591

Merged
merged 8 commits into from
Feb 19, 2022
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
lib/*.js
test/*.js
*.d.ts
coverage
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
lib/*.js
test/*.js
*.js.map
*.d.ts
node_modules
*.generated.ts
dist
.jsii

.LAST_BUILD
.nyc_output
coverage
.nycrc
.LAST_PACKAGE
*.snk
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Exclude typescript source and config
*.ts
tsconfig.json
coverage
.nyc_output
*.tgz
*.snk
*.tsbuildinfo

# Include javascript files and typescript declarations
!*.js
!*.d.ts

# Exclude jsii outdir
dist

# Include .jsii
!.jsii

# Include .jsii
!.jsii
107 changes: 107 additions & 0 deletions source/patterns/@aws-solutions-constructs/aws-fargate-s3/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
# aws-fargate-s3 module
<!--BEGIN STABILITY BANNER-->

---

![Stability: Experimental](https://img.shields.io/badge/stability-Experimental-important.svg?style=for-the-badge)

> All classes are under active development and subject to non-backward compatible changes or removal in any
> future version. These are not subject to the [Semantic Versioning](https://semver.org/) model.
> This means that while you may use them, you may need to update your source code when upgrading to a newer version of this package.

---
<!--END STABILITY BANNER-->

| **Reference Documentation**:| <span style="font-weight: normal">https://docs.aws.amazon.com/solutions/latest/constructs/</span>|
|:-------------|:-------------|
<div style="height:8px"></div>

| **Language** | **Package** |
|:-------------|-----------------|
|![Python Logo](https://docs.aws.amazon.com/cdk/api/latest/img/python32.png) Python|`aws_solutions_constructs.aws_fargate_s3`|
|![Typescript Logo](https://docs.aws.amazon.com/cdk/api/latest/img/typescript32.png) Typescript|`@aws-solutions-constructs/aws-fargate-s3`|
|![Java Logo](https://docs.aws.amazon.com/cdk/api/latest/img/java32.png) Java|`software.amazon.awsconstructs.services.fargates3`|

This AWS Solutions Construct implements an AWS Fargate service that can write/read to an Amazon S3 Bucket

Here is a minimal deployable pattern definition in Typescript:

``` typescript
import { FargateToS3, FargateToS3Props } from '@aws-solutions-constructs/aws-fargate-s3';

const props: FargateToS3Props = {
publicApi: true,
ecrRepositoryArn: "arn of a repo in ECR in your account",
});

new FargateToS3(stack, 'test-construct', props);
```

## Initializer
mickychetta marked this conversation as resolved.
Show resolved Hide resolved

``` text
new FargateToS3(scope: Construct, id: string, props: FargateToS3Props);
```

_Parameters_

* scope [`Construct`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_core.Construct.html)
* id `string`
* props [`FargateToS3Props`](#pattern-construct-props)

## Pattern Construct Props

| **Name** | **Type** | **Description** |
|:-------------|:----------------|-----------------|
| publicApi | boolean | Whether the construct is deploying a private or public API. This has implications for the VPC and ALB. |
| vpcProps? | [ec2.VpcProps](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-ec2.VpcProps.html) | Optional custom properties for a VPC the construct will create. This VPC will be used by the new ALB and any Private Hosted Zone the construct creates (that's why loadBalancerProps and privateHostedZoneProps can't include a VPC). Providing both this and existingVpc is an error. |
| existingVpc? | [ec2.IVpc](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-ec2.IVpc.html) | An existing VPC in which to deploy the construct. Providing both this and vpcProps is an error. If the client provides an existing load balancer and/or existing Private Hosted Zone, those constructs must exist in this VPC. |
| clusterProps? | [ecs.ClusterProps](https://docs.aws.amazon.com/cdk/api/v1/docs/@aws-cdk_aws-ecs.ClusterProps.html) | Optional properties to create a new ECS cluster. To provide an existing cluster, use the cluster attribute of fargateServiceProps. |
| ecrRepositoryArn? | string | The arn of an ECR Repository containing the image to use to generate the containers. Either this or the image property of containerDefinitionProps must be provided. format: arn:aws:ecr:*region*:*account number*:repository/*Repository Name* |
| ecrImageVersion? | string | The version of the image to use from the repository. Defaults to 'Latest' |
| containerDefinitionProps? | [ecs.ContainerDefinitionProps \| any](https://docs.aws.amazon.com/cdk/api/v1/docs/@aws-cdk_aws-ecs.ContainerDefinitionProps.html) | Optional props to define the container created for the Fargate Service (defaults found in fargate-defaults.ts) |
| fargateTaskDefinitionProps? | [ecs.FargateTaskDefinitionProps \| any](https://docs.aws.amazon.com/cdk/api/v1/docs/@aws-cdk_aws-ecs.FargateTaskDefinitionProps.html) | Optional props to define the Fargate Task Definition for this construct (defaults found in fargate-defaults.ts) |
| fargateServiceProps? | [ecs.FargateServiceProps \| any](https://docs.aws.amazon.com/cdk/api/v1/docs/@aws-cdk_aws-ecs.FargateServiceProps.html) | Optional values to override default Fargate Task definition properties (fargate-defaults.ts). The construct will default to launching the service is the most isolated subnets available (precedence: Isolated, Private and Public). Override those and other defaults here. |
| existingFargateServiceObject? | [ecs.FargateService](https://docs.aws.amazon.com/cdk/api/v1/docs/@aws-cdk_aws-ecs.FargateService.html) | A Fargate Service already instantiated (probably by another Solutions Construct). If this is specified, then no props defining a new service can be provided, including: existingImageObject, ecrImageVersion, containerDefintionProps, fargateTaskDefinitionProps, ecrRepositoryArn, fargateServiceProps, clusterProps, existingClusterInterface |
| existingContainerDefinitionObject? | [ecs.ContainerDefinition](https://docs.aws.amazon.com/cdk/api/v1/docs/@aws-cdk_aws-ecs.ContainerDefinition.html) | A container definition already instantiated as part of a Fargate service. This must be the container in the existingFargateServiceObject |
|existingBucketInterface?|[`s3.IBucket`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-s3.IBucket.html)|Existing S3 Bucket interface. Providing this property and `bucketProps` results in an error.|
|bucketProps?|[`s3.BucketProps`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-s3.BucketProps.html)|Optional user provided props to override the default props for the S3 Bucket.|
|loggingBucketProps?|[`s3.BucketProps`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-s3.BucketProps.html)|Optional user provided props to override the default props for the S3 Logging Bucket.|
|logS3AccessLogs?| boolean|Whether to turn on Access Logging for the S3 bucket. Creates an S3 bucket with associated storage costs for the logs. Enabling Access Logging is a best practice. default - true|
|bucketPermissions?|`string[]`|Optional bucket permissions to grant to the Lambda function. One or more of the following may be specified: `Delete`, `Put`, `Read`, `ReadWrite`, `Write`.|
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo - Lambda function?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These terms seem specific to DDB, there's no difference between Put and Write in S3 that I can think of easily. I think the correct list would be:

List
Put
Get
Delete

If we want to stay 'generic' so read/write are common terms across all constructs (which has appeal), then it would be:

Read (we would want to ensure this includes ListBucket)
Write
Delete

If the docs are correct and more than 1 can be specified, then there is no need to include ReadWrite - I haven't looked at lambda-s3 yet, but this seems like a mistake we made back then.

Let's go with the latter and make all constructs use the terms Read and Write for consistency.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is optional, the docs need to clearly state the default.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default bucket permissions in aws-lambda-s3 is ReadWrite which composes of Read, Write, and Delete operations so I will stick to to that for consistency. I allowed Read, Write, and Delete strings for bucketPermissions in order to create a more fine-grained policy. I removed Put as it is repetitive and made ReadWrite as default which separates confusion from inputs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will need to make following changes to aws-lambda-s3 if we agree with this.

|bucketArnEnvironmentVariableName?|string|Optional Name for the S3 bucket arn environment variable set for the container.|
|bucketEnvironmentVariableName?|string|Optional Name for the S3 bucket name environment variable set for the container.|

## Pattern Properties

| **Name** | **Type** | **Description** |
|:-------------|:----------------|-----------------|
| vpc | [ec2.IVpc](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-ec2.IVpc.html) | The VPC used by the construct (whether created by the construct or providedb by the client) |
| service | [ecs.FargateService](https://docs.aws.amazon.com/cdk/api/v1/docs/@aws-cdk_aws-ecs.FargateService.html) | The AWS Fargate service used by this construct (whether created by this construct or passed to this construct at initialization) |
| container | [ecs.ContainerDefinition](https://docs.aws.amazon.com/cdk/api/v1/docs/@aws-cdk_aws-ecs.ContainerDefinition.html) | The container associated with the AWS Fargate service in the service property. |
| s3Bucket? |[s3.IBucket](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-s3.IBucket.html)|Returns an instance of s3.Bucket created by the construct|
| s3BucketInterface |[`s3.IBucket`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-s3.IBucket.html)|Returns an instance of s3.IBucket created by the construct|
| s3LoggingBucket? | [s3.Bucket](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-s3.Bucket.html)|Returns an instance of s3.Bucket created by the construct|

## Default settings

Out of the box implementation of the Construct without any override will set the following defaults:

### AWS Fargate Service
* Sets up an AWS Fargate service
* Uses the existing service if provided
* Creates a new service if none provided.
* Service will run in isolated subnets if available, then private subnets if available and finally public subnets
* Adds environment variables to the container with the ARN and Name of the S3 Bucket
* Add permissions to the container IAM role allowing it to publish to the S3 Bucket

### Amazon S3 Bucket
* Sets up an Amazon S3 Bucket
* Uses an existing bucket if one is provided, otherwise creates a new one
* Adds an Interface Endpoint to the VPC for S3 (the service by default runs in Isolated or Private subnets)

## Architecture
![Architecture Diagram](architecture.png)

***
&copy; Copyright 2022 Amazon.com, Inc. or its affiliates. All Rights Reserved.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,234 @@
/**
* Copyright 2022 Amazon.com, Inc. or its affiliates. All Rights Reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
* with the License. A copy of the License is located at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES
* OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions
* and limitations under the License.
*/

import * as ec2 from "@aws-cdk/aws-ec2";
import * as s3 from "@aws-cdk/aws-s3";
// Note: To ensure CDKv2 compatibility, keep the import statement for Construct separate
import { Construct } from "@aws-cdk/core";
import * as defaults from "@aws-solutions-constructs/core";
import * as ecs from "@aws-cdk/aws-ecs";

export interface FargateToS3Props {
/**
* Optional custom properties for a VPC the construct will create. This VPC will
* be used by the new Fargate service the construct creates (that's
* why targetGroupProps can't include a VPC). Providing
* both this and existingVpc is an error. An S3 Interface
* endpoint will be included in this VPC.
*
* @default - none
*/
readonly vpcProps?: ec2.VpcProps;
/**
* An existing VPC in which to deploy the construct. Providing both this and
* vpcProps is an error. If the client provides an existing Fargate service,
* this value must be the VPC where the service is running. An S3 Interface
* endpoint will be added to this VPC.
*
* @default - none
*/
readonly existingVpc?: ec2.IVpc;
/**
* Whether the construct is deploying a private or public API. This has implications for the VPC deployed
* by this construct.
*
* @default - none
*/
readonly publicApi: boolean;
/**
* Optional properties to create a new ECS cluster
*/
readonly clusterProps?: ecs.ClusterProps;
/**
* The arn of an ECR Repository containing the image to use
* to generate the containers
*
* format:
* arn:aws:ecr:[region]:[account number]:repository/[Repository Name]
*/
readonly ecrRepositoryArn?: string;
/**
* The version of the image to use from the repository
*
* @default - 'latest'
*/
readonly ecrImageVersion?: string;
/*
* Optional props to define the container created for the Fargate Service
*
* defaults - fargate-defaults.ts
*/
readonly containerDefinitionProps?: ecs.ContainerDefinitionProps | any;
/*
* Optional props to define the Fargate Task Definition for this construct
*
* defaults - fargate-defaults.ts
*/
readonly fargateTaskDefinitionProps?: ecs.FargateTaskDefinitionProps | any;
/**
* Optional values to override default Fargate Task definition properties
* (fargate-defaults.ts). The construct will default to launching the service
* is the most isolated subnets available (precedence: Isolated, Private and
* Public). Override those and other defaults here.
*
* defaults - fargate-defaults.ts
*/
readonly fargateServiceProps?: ecs.FargateServiceProps | any;
/**
* A Fargate Service already instantiated (probably by another Solutions Construct). If
* this is specified, then no props defining a new service can be provided, including:
* existingImageObject, ecrImageVersion, containerDefintionProps, fargateTaskDefinitionProps,
* ecrRepositoryArn, fargateServiceProps, clusterProps, existingClusterInterface. If this value
* is provided, then existingContainerDefinitionObject must be provided as well.
*
* @default - none
*/
readonly existingFargateServiceObject?: ecs.FargateService;
/**
* Existing instance of S3 Bucket object, providing both this and `bucketProps` will cause an error.
*
* @default - None
*/
readonly existingBucketObj?: s3.IBucket;
/**
* Optional user provided props to override the default props for the S3 Bucket.
*
* @default - Default props are used
*/
readonly bucketProps?: s3.BucketProps;
/**
* Optional user provided props to override the default props for the S3 Logging Bucket.
*
* @default - Default props are used
*/
readonly loggingBucketProps?: s3.BucketProps
/**
* Whether to turn on Access Logs for the S3 bucket with the associated storage costs.
* Enabling Access Logging is a best practice.
*
* @default - true
*/
readonly logS3AccessLogs?: boolean;
/**
* Optional bucket permissions to grant to the Lambda function.
* One or more of the following may be specified: "Delete", "Put", "Read", "ReadWrite", "Write".
*
* @default - Read/write access is given to the Lambda function if no value is specified.
*/
readonly bucketPermissions?: string[];
/**
* Optional Name for the S3 bucket arn environment variable set for the container.
*
* @default - None
*/
readonly bucketArnEnvironmentVariableName?: string;
/**
* Optional Name for the S3 bucket name environment variable set for the container.
*
* @default - None
*/
readonly bucketEnvironmentVariableName?: string;
/*
* A container definition already instantiated as part of a Fargate service. This must
* be the container in the existingFargateServiceObject.
*
* @default - None
*/
readonly existingContainerDefinitionObject?: ecs.ContainerDefinition;
}

export class FargateToS3 extends Construct {
public readonly vpc: ec2.IVpc;
public readonly service: ecs.FargateService;
public readonly container: ecs.ContainerDefinition;
public readonly s3BucketInterface: s3.IBucket;
public readonly s3Bucket?: s3.Bucket;
public readonly s3LoggingBucket?: s3.Bucket;

constructor(scope: Construct, id: string, props: FargateToS3Props) {
super(scope, id);
defaults.CheckProps(props);
defaults.CheckFargateProps(props);

this.vpc = defaults.buildVpc(scope, {
existingVpc: props.existingVpc,
defaultVpcProps: props.publicApi ? defaults.DefaultPublicPrivateVpcProps() : defaults.DefaultIsolatedVpcProps(),
userVpcProps: props.vpcProps,
constructVpcProps: { enableDnsHostnames: true, enableDnsSupport: true }
});

defaults.AddAwsServiceEndpoint(scope, this.vpc, defaults.ServiceEndpointTypes.S3);

if (props.existingFargateServiceObject) {
this.service = props.existingFargateServiceObject;
// CheckFargateProps confirms that the container is provided
this.container = props.existingContainerDefinitionObject!;
} else {
[this.service, this.container] = defaults.CreateFargateService(
scope,
id,
this.vpc,
props.clusterProps,
props.ecrRepositoryArn,
props.ecrImageVersion,
props.fargateTaskDefinitionProps,
props.containerDefinitionProps,
props.fargateServiceProps
);
}

// Setup the S3 Bucket
let bucket: s3.IBucket;

if (!props.existingBucketObj) {
[this.s3Bucket, this.s3LoggingBucket] = defaults.buildS3Bucket(this, {
bucketProps: props.bucketProps,
loggingBucketProps: props.loggingBucketProps,
logS3AccessLogs: props.logS3AccessLogs
});
bucket = this.s3Bucket;
} else {
bucket = props.existingBucketObj;
}

this.s3BucketInterface = bucket;

// Add the requested or default bucket permissions
if (props.bucketPermissions) {
mickychetta marked this conversation as resolved.
Show resolved Hide resolved
if (props.bucketPermissions.includes('Delete')) {
bucket.grantDelete(this.service.taskDefinition.taskRole);
}
if (props.bucketPermissions.includes('Put')) {
bucket.grantPut(this.service.taskDefinition.taskRole);
}
if (props.bucketPermissions.includes('Read')) {
bucket.grantRead(this.service.taskDefinition.taskRole);
}
if (props.bucketPermissions.includes('ReadWrite')) {
bucket.grantReadWrite(this.service.taskDefinition.taskRole);
}
if (props.bucketPermissions.includes('Write')) {
bucket.grantWrite(this.service.taskDefinition.taskRole);
}
} else {
bucket.grantReadWrite(this.service.taskDefinition.taskRole);
}

// Add environment variables
const bucketArnEnvironmentVariableName = props.bucketArnEnvironmentVariableName || 'S3_BUCKET_ARN';
this.container.addEnvironment(bucketArnEnvironmentVariableName, this.s3BucketInterface.bucketArn);
const bucketEnvironmentVariableName = props.bucketEnvironmentVariableName || 'S3_BUCKET_NAME';
this.container.addEnvironment(bucketEnvironmentVariableName, this.s3BucketInterface.bucketName);

}
}
Loading