Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(ecs): support secret environment variables #2994

Merged
merged 30 commits into from
Jul 29, 2019
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
146f13e
feat(ecs): support secret environment variables
jogold Jun 21, 2019
5384d55
Merge branch 'master' into ecs-secrets
jogold Jun 21, 2019
bf91871
Merge branch 'master' into ecs-secrets
jogold Jun 21, 2019
a0bb714
add IAM permissions
jogold Jun 24, 2019
2a5155b
Merge branch 'master' into ecs-secrets
jogold Jun 24, 2019
d697273
disable-awslint:ref-via-interface
jogold Jun 24, 2019
d1c31bf
Merge branch 'master' into ecs-secrets
jogold Jun 26, 2019
3c75145
Merge branch 'master' into ecs-secrets
Jul 1, 2019
a242dca
Merge branch 'master' into ecs-secrets
jogold Jul 3, 2019
67b0abe
Merge branch 'ecs-secrets' of github.com:jogold/aws-cdk into ecs-secrets
jogold Jul 3, 2019
f8d0efc
scheduled task
jogold Jul 3, 2019
f3cb37a
typo
jogold Jul 4, 2019
39909d7
typo
jogold Jul 4, 2019
43f8244
typo
jogold Jul 4, 2019
3a6a374
Merge branch 'master' into ecs-secrets
jogold Jul 22, 2019
f300b28
secrets and non breaking
jogold Jul 22, 2019
bcc350a
JSDoc
jogold Jul 22, 2019
17e0964
non breaking renderContainerDefinition
jogold Jul 22, 2019
d629929
renderKV
jogold Jul 22, 2019
80d4495
base props
jogold Jul 22, 2019
199de19
polymorphism
jogold Jul 22, 2019
037f9e4
base props
jogold Jul 22, 2019
6cca252
JSDoc for queue env vars
jogold Jul 23, 2019
aedbbb8
make base props internal
jogold Jul 23, 2019
410cfd5
remove note about scheduled fargate tasks
jogold Jul 23, 2019
5f5cf12
update README
jogold Jul 23, 2019
6298b4d
Merge branch 'master' into ecs-secrets
Jul 23, 2019
865d7f2
revert base props
jogold Jul 23, 2019
a2032a6
Merge branch 'ecs-secrets' of github.com:jogold/aws-cdk into ecs-secrets
jogold Jul 23, 2019
f27f8c5
Merge branch 'master' into ecs-secrets
jogold Jul 25, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ export interface LoadBalancedServiceBaseProps {
*
* @default - No environment variables.
*/
readonly environment?: { [key: string]: string };
readonly environment?: { [key: string]: ecs.EnvironmentValue };
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like we might want to use some token magic here.

FWIW, we should have Tokenized representations of secret values already. Can we not use those in some way?

Copy link
Contributor Author

@jogold jogold Jun 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the Secrets property, CF expects the ARN of the secret/ssm param, the container will pull that value at startup and set it as env var avoiding passing secrets in clear text. Secret env vars are retrieved at runtime (always up to date), this is not the case with simple env vars (fixed at deploy time)

The class EnvironmentValue will feed either environment or secrets based on the type. This gives a better user experience (higher level of abstraction) than having to specify environment and/or secrets manually (both will become env vars in the running container at the end).

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ecs-taskdefinition-containerdefinitions.html#cfn-ecs-taskdefinition-containerdefinition-environment
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ecs-taskdefinition-secret.html

The support for the Secrets property was added in the latest release of CF (https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/ReleaseHistory.html).

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/specifying-sensitive-data.html


/**
* Whether to create an AWS log driver
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ export interface QueueProcessingServiceBaseProps {
*
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this really the default or will it always be passed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This part is not mine, it seems that it's always passed, I can update the doc.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, that would be nice

* @default 'QUEUE_NAME: queue.queueName'
*/
readonly environment?: { [key: string]: string };
readonly environment?: { [key: string]: ecs.EnvironmentValue };

/**
* A queue for which to process items from.
Expand Down Expand Up @@ -88,7 +88,7 @@ export abstract class QueueProcessingServiceBase extends cdk.Construct {
/**
* Environment variables that will include the queue name
*/
public readonly environment: { [key: string]: string };
public readonly environment: { [key: string]: ecs.EnvironmentValue };
/**
* The minimum number of tasks to run
*/
Expand Down Expand Up @@ -121,7 +121,7 @@ export abstract class QueueProcessingServiceBase extends cdk.Construct {
this.logDriver = enableLogging ? this.createAWSLogDriver(this.node.id) : undefined;

// Add the queue name to environment variables
this.environment = { ...(props.environment || {}), QUEUE_NAME: this.sqsQueue.queueName };
this.environment = { ...(props.environment || {}), QUEUE_NAME: ecs.EnvironmentValue.fromString(this.sqsQueue.queueName) };

// Determine the desired task count (minimum) and maximum scaling capacity
this.desiredCount = props.desiredTaskCount || 1;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ export interface ScheduledEc2TaskProps {
*
* @default none
*/
readonly environment?: { [key: string]: string };
readonly environment?: { [key: string]: ecs.EnvironmentValue };

/**
* The hard limit (in MiB) of memory to present to the container.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -649,11 +649,7 @@
"Cpu": 1,
"Environment": [
{
"Name": "name",
"Value": "TRIGGER"
},
{
"Name": "value",
"Name": "TRIGGER",
"Value": "CloudWatch Events"
}
],
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ class EventStack extends cdk.Stack {
desiredTaskCount: 2,
memoryLimitMiB: 512,
cpu: 1,
environment: { name: 'TRIGGER', value: 'CloudWatch Events' },
environment: { TRIGGER: ecs.EnvironmentValue.fromString('CloudWatch Events') },
schedule: events.Schedule.rate(cdk.Duration.minutes(1)),
});
/// !hide
Expand Down
12 changes: 6 additions & 6 deletions packages/@aws-cdk/aws-ecs-patterns/test/ec2/test.l3s.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,8 @@ export = {
image: ecs.ContainerImage.fromRegistry('test'),
desiredCount: 2,
environment: {
TEST_ENVIRONMENT_VARIABLE1: "test environment variable 1 value",
TEST_ENVIRONMENT_VARIABLE2: "test environment variable 2 value"
TEST_ENVIRONMENT_VARIABLE1: ecs.EnvironmentValue.fromString("test environment variable 1 value"),
TEST_ENVIRONMENT_VARIABLE2: ecs.EnvironmentValue.fromString("test environment variable 2 value")
}
});

Expand Down Expand Up @@ -96,8 +96,8 @@ export = {
image: ecs.ContainerImage.fromRegistry('test'),
desiredCount: 2,
environment: {
TEST_ENVIRONMENT_VARIABLE1: "test environment variable 1 value",
TEST_ENVIRONMENT_VARIABLE2: "test environment variable 2 value"
TEST_ENVIRONMENT_VARIABLE1: ecs.EnvironmentValue.fromString("test environment variable 1 value"),
TEST_ENVIRONMENT_VARIABLE2: ecs.EnvironmentValue.fromString("test environment variable 2 value")
}
});

Expand Down Expand Up @@ -153,8 +153,8 @@ export = {
desiredCount: 2,
enableLogging: false,
environment: {
TEST_ENVIRONMENT_VARIABLE1: "test environment variable 1 value",
TEST_ENVIRONMENT_VARIABLE2: "test environment variable 2 value"
TEST_ENVIRONMENT_VARIABLE1: ecs.EnvironmentValue.fromString("test environment variable 1 value"),
TEST_ENVIRONMENT_VARIABLE2: ecs.EnvironmentValue.fromString("test environment variable 2 value")
}
});

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -84,8 +84,8 @@ export = {
enableLogging: false,
desiredTaskCount: 2,
environment: {
TEST_ENVIRONMENT_VARIABLE1: "test environment variable 1 value",
TEST_ENVIRONMENT_VARIABLE2: "test environment variable 2 value"
TEST_ENVIRONMENT_VARIABLE1: ecs.EnvironmentValue.fromString("test environment variable 1 value"),
TEST_ENVIRONMENT_VARIABLE2: ecs.EnvironmentValue.fromString("test environment variable 2 value")
},
queue,
maxScalingCapacity: 5
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ export = {
desiredTaskCount: 2,
memoryLimitMiB: 512,
cpu: 2,
environment: { name: 'TRIGGER', value: 'CloudWatch Events' },
environment: { TRIGGER: ecs.EnvironmentValue.fromString('CloudWatch Events') },
schedule: events.Schedule.expression('rate(1 minute)')
});

Expand All @@ -111,11 +111,7 @@ export = {
Cpu: 2,
Environment: [
{
Name: "name",
Value: "TRIGGER"
},
{
Name: "value",
Name: "TRIGGER",
Value: "CloudWatch Events"
}
],
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -82,8 +82,8 @@ export = {
enableLogging: false,
desiredTaskCount: 2,
environment: {
TEST_ENVIRONMENT_VARIABLE1: "test environment variable 1 value",
TEST_ENVIRONMENT_VARIABLE2: "test environment variable 2 value"
TEST_ENVIRONMENT_VARIABLE1: ecs.EnvironmentValue.fromString("test environment variable 1 value"),
TEST_ENVIRONMENT_VARIABLE2: ecs.EnvironmentValue.fromString("test environment variable 2 value")
},
queue,
maxScalingCapacity: 5
Expand Down
75 changes: 64 additions & 11 deletions packages/@aws-cdk/aws-ecs/lib/container-definition.ts
Original file line number Diff line number Diff line change
@@ -1,11 +1,59 @@
import iam = require('@aws-cdk/aws-iam');
import secretsmanager = require('@aws-cdk/aws-secretsmanager');
import ssm = require('@aws-cdk/aws-ssm');
import cdk = require('@aws-cdk/cdk');
import { NetworkMode, TaskDefinition } from './base/task-definition';
import { ContainerImage, ContainerImageConfig } from './container-image';
import { CfnTaskDefinition } from './ecs.generated';
import { LinuxParameters } from './linux-parameters';
import { LogDriver, LogDriverConfig } from './log-drivers/log-driver';

/**
* Environment variable value type.
*/
export enum EnvironmentValueType {
/**
* A string in clear text.
*/
STRING = 'string',

/**
* The full ARN of the AWS Secrets Manager secret or the full ARN of the
* parameter in the AWS Systems Manager Parameter Store.
*/
SECRET = 'secret',
}

/**
* An environment variable value.
*/
export class EnvironmentValue {
/**
* Creates a environment variable value from a string.
jogold marked this conversation as resolved.
Show resolved Hide resolved
*/
public static fromString(value: string) {
return new EnvironmentValue(value, EnvironmentValueType.STRING);
}

/**
* Creates a environment variable value from a parameter stored in AWS
jogold marked this conversation as resolved.
Show resolved Hide resolved
* Systems Manager Parameter Store.
*/
public static fromSsmParameter(parameter: ssm.IParameter) {
return new EnvironmentValue(parameter.parameterArn, EnvironmentValueType.SECRET);
}

/**
* Creates a environment variable value from a secret stored in AWS Secrets
* Manager.
*/
public static fromSecretsManager(secret: secretsmanager.ISecret) {
return new EnvironmentValue(secret.secretArn, EnvironmentValueType.SECRET);
}

constructor(public readonly value: string, public readonly type: EnvironmentValueType) {}
}

export interface ContainerDefinitionOptions {
/**
* The image to use for a container.
Expand Down Expand Up @@ -81,7 +129,7 @@ export interface ContainerDefinitionOptions {
*
* @default - No environment variables.
*/
readonly environment?: { [key: string]: string };
readonly environment?: { [key: string]: EnvironmentValue };

/**
* Indicates whether the task stops if this container fails.
Expand Down Expand Up @@ -390,6 +438,17 @@ export class ContainerDefinition extends cdk.Construct {
* Render this container definition to a CloudFormation object
*/
public renderContainerDefinition(): CfnTaskDefinition.ContainerDefinitionProperty {
const environment = [];
const secrets = [];
for (const [k, v] of Object.entries(this.props.environment || {})) {
if (v.type === EnvironmentValueType.STRING) {
environment.push({ name: k, value: v.value });
}
if (v.type === EnvironmentValueType.SECRET) {
secrets.push({ name: k, valueFrom: v.value });
}
}

return {
command: this.props.command,
cpu: this.props.cpu,
Expand All @@ -415,8 +474,10 @@ export class ContainerDefinition extends cdk.Construct {
volumesFrom: this.volumesFrom.map(renderVolumeFrom),
workingDirectory: this.props.workingDirectory,
logConfiguration: this.logDriverConfig,
environment: this.props.environment && renderKV(this.props.environment, 'name', 'value'),
extraHosts: this.props.extraHosts && renderKV(this.props.extraHosts, 'hostname', 'ipAddress'),
environment: environment.length !== 0 ? environment : undefined,
secrets: secrets.length !== 0 ? secrets : undefined,
extraHosts: this.props.extraHosts && Object.entries(this.props.extraHosts)
.map(([k, v]) => ({ hostname: k, ipAddress: v })),
healthCheck: this.props.healthCheck && renderHealthCheck(this.props.healthCheck),
links: this.links,
linuxParameters: this.linuxParameters && this.linuxParameters.renderLinuxParameters(),
Expand Down Expand Up @@ -472,14 +533,6 @@ export interface HealthCheck {
readonly timeout?: cdk.Duration;
}

function renderKV(env: { [key: string]: string }, keyName: string, valueName: string): any {
jogold marked this conversation as resolved.
Show resolved Hide resolved
const ret = [];
for (const [key, value] of Object.entries(env)) {
ret.push({ [keyName]: key, [valueName]: value });
}
return ret;
}

function renderHealthCheck(hc: HealthCheck): CfnTaskDefinition.HealthCheckProperty {
return {
command: getHealthCheckCommand(hc),
Expand Down
67 changes: 66 additions & 1 deletion packages/@aws-cdk/aws-ecs/test/test.container-definition.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import { expect, haveResource, haveResourceLike } from '@aws-cdk/assert';
import secretsmanager = require('@aws-cdk/aws-secretsmanager');
import ssm = require('@aws-cdk/aws-ssm');
import cdk = require('@aws-cdk/cdk');
import { Test } from 'nodeunit';
import ecs = require('../lib');
Expand Down Expand Up @@ -265,7 +266,7 @@ export = {
image: ecs.ContainerImage.fromRegistry('test'),
memoryLimitMiB: 1024,
environment: {
TEST_ENVIRONMENT_VARIABLE: "test environment variable value"
TEST_ENVIRONMENT_VARIABLE: ecs.EnvironmentValue.fromString("test environment variable value")
}
});

Expand All @@ -285,6 +286,70 @@ export = {

},

'can add secret environment variables to the container definition'(test: Test) {
// GIVEN
const stack = new cdk.Stack();
const taskDefinition = new ecs.Ec2TaskDefinition(stack, 'TaskDef');

const secret = new secretsmanager.Secret(stack, 'Secret');
const parameter = ssm.StringParameter.fromSecureStringParameterAttributes(stack, 'Parameter', {
parameterName: '/name',
version: 1
});

// WHEN
taskDefinition.addContainer('cont', {
image: ecs.ContainerImage.fromRegistry('test'),
memoryLimitMiB: 1024,
environment: {
SECRET: ecs.EnvironmentValue.fromSecretsManager(secret),
PARAMETER: ecs.EnvironmentValue.fromSsmParameter(parameter),
}
});

// THEN
expect(stack).to(haveResourceLike('AWS::ECS::TaskDefinition', {
ContainerDefinitions: [
{
Secrets: [
{
Name: "SECRET",
ValueFrom: {
Ref: "SecretA720EF05"
}
},
{
Name: "PARAMETER",
ValueFrom: {
"Fn::Join": [
"",
[
"arn:",
{
Ref: "AWS::Partition"
},
":ssm:",
{
Ref: "AWS::Region"
},
":",
{
Ref: "AWS::AccountId"
},
":parameter/name"
]
]
}
},
]
}
]
}));

test.done();

},

'can add AWS logging to container definition'(test: Test) {
// GIVEN
const stack = new cdk.Stack();
Expand Down