-
Notifications
You must be signed in to change notification settings - Fork 4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(iot): add Action to put objects in S3 Buckets #17307
Changes from 2 commits
be3be36
b81a310
4b66370
0a37bdd
9023c01
6fc3248
a870670
f0a3480
f4c292c
d524a80
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,2 +1,32 @@ | ||
AWS Cloud Development Kit (AWS CDK) | ||
Copyright 2018-2021 Amazon.com, Inc. or its affiliates. All Rights Reserved. | ||
|
||
------------------------------------------------------------------------------- | ||
|
||
The AWS CDK includes the following third-party software/licensing: | ||
|
||
** case - https://www.npmjs.com/package/case | ||
Copyright (c) 2013 Nathan Bubna | ||
|
||
Permission is hereby granted, free of charge, to any person | ||
obtaining a copy of this software and associated documentation | ||
files (the "Software"), to deal in the Software without | ||
restriction, including without limitation the rights to use, | ||
copy, modify, merge, publish, distribute, sublicense, and/or sell | ||
copies of the Software, and to permit persons to whom the | ||
Software is furnished to do so, subject to the following | ||
conditions: | ||
|
||
The above copyright notice and this permission notice shall be | ||
included in all copies or substantial portions of the Software. | ||
|
||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, | ||
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES | ||
OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND | ||
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT | ||
HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, | ||
WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING | ||
FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR | ||
OTHER DEALINGS IN THE SOFTWARE. | ||
|
||
---------------- |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,2 +1,3 @@ | ||
export * from './cloudwatch-logs-action'; | ||
export * from './lambda-function-action'; | ||
export * from './s3-action'; |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,77 @@ | ||
import * as iam from '@aws-cdk/aws-iam'; | ||
import * as iot from '@aws-cdk/aws-iot'; | ||
import * as s3 from '@aws-cdk/aws-s3'; | ||
import { kebab as toKebabCase } from 'case'; | ||
import { singletonActionRole } from './private/role'; | ||
|
||
/** | ||
* Configuration properties of an action for s3. | ||
*/ | ||
export interface S3ActionProps { | ||
/** | ||
* The Amazon S3 canned ACL that controls access to the object identified by the object key. | ||
* @see https://docs.aws.amazon.com/AmazonS3/latest/userguide/acl-overview.html#canned-acl | ||
* | ||
* @default None | ||
*/ | ||
readonly cannedAcl?: s3.BucketAccessControl; | ||
|
||
/** | ||
* The path to the file where the data is written. | ||
* | ||
* Supports substitution templates. | ||
* @see https://docs.aws.amazon.com/iot/latest/developerguide/iot-substitution-templates.html | ||
* | ||
* @default '${topic()}/${timestamp()}' | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Should we have static, string-typed constants for these substitution templates? So that it's clear which ones can be used by which Actions? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think we cannot define all static, string-typed constants. Because users can use MQTT payload property defined by themselves in template. For example, there are following MQTT payload: {
"year": "2021",
"month": "Nov",
"day": "04",
"data": 123.4
} The user can define the bucket key as Are we on the same page? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't think we need to have all of them - but you have a few that are always available, right? Like There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. OK, I also want to make it as user-friendly as possible. So I'd like to be on same page!
And I thought 3 plans. PLAN1, implement a set of functionsIn light of the above, it might be better for us to implement a set of functions and for users to use them in template literals, as following: key: `\${${iot.Template.topic(2)}}/\${${iot.Template.timestamp()}}` PLAN2, just validationOr, is it better to implement validation as For substitution templates, I think we will be using a lot of regular expressions. PLAN3, implement
|
||
*/ | ||
readonly key?: string; | ||
|
||
/** | ||
* The IAM role that allows access to the S3. | ||
* | ||
* @default a new role will be created | ||
*/ | ||
readonly role?: iam.IRole; | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Will every Action have a There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @skinny85
For example, constructor(private readonly func: lambda.IFunction) {} So I thought there are no common property for all actions. But the way to align the arguments of the action's constructor is what I was also struggling with. What do you think? I'm open to any and all refactoring! There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ok, so if all of actions besides two will need There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. OK! I will! 🚀 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @skinny85 interface CommonActionProps {
}
intercace CommonAwsActionProps extends CommonActionProps {
role?: iam.IRole;
}
class Action {
constructor(props: CommonActionProps) {}
} Now, three Actions
constructor(func: lambda.IFunction) {}
constructor(logGroup: logs.ILogGroup, props: CloudWatchLogsActionProps) {}
constructor(bucket: s3.IBucket, props: S3PutObjectActionProps) {} So if first question is "yes", I will do this refactoring in another PR! There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. No. Let's leave the constructors as they are now. BTW, there's no point in having both |
||
} | ||
|
||
/** | ||
* The action to write the data from an MQTT message to an Amazon S3 bucket. | ||
*/ | ||
export class S3Action implements iot.IAction { | ||
yamatatsu marked this conversation as resolved.
Show resolved
Hide resolved
|
||
private readonly cannedAcl?: string; | ||
private readonly key?: string; | ||
private readonly role?: iam.IRole; | ||
|
||
/** | ||
* @param bucket The Amazon S3 bucket to which to write data. | ||
* @param props Optional properties to not use default | ||
*/ | ||
constructor(private readonly bucket: s3.IBucket, props: S3ActionProps = {}) { | ||
this.cannedAcl = props.cannedAcl; | ||
this.key = props.key; | ||
this.role = props.role; | ||
} | ||
|
||
bind(rule: iot.ITopicRule): iot.ActionConfig { | ||
const role = this.role ?? singletonActionRole(rule); | ||
role.addToPrincipalPolicy(this.putEventStatement(this.bucket)); | ||
|
||
return { | ||
configuration: { | ||
s3: { | ||
bucketName: this.bucket.bucketName, | ||
cannedAcl: this.cannedAcl && toKebabCase(this.cannedAcl.toString()), | ||
key: this.key ?? '${topic()}/${timestamp()}', | ||
roleArn: role.roleArn, | ||
}, | ||
}, | ||
}; | ||
} | ||
|
||
private putEventStatement(bucket: s3.IBucket) { | ||
return new iam.PolicyStatement({ | ||
actions: ['s3:PutObject'], | ||
skinny85 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
resources: [bucket.arnForObjects('*')], | ||
}); | ||
} | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,85 @@ | ||
{ | ||
"Resources": { | ||
"TopicRule40A4EA44": { | ||
"Type": "AWS::IoT::TopicRule", | ||
"Properties": { | ||
"TopicRulePayload": { | ||
"Actions": [ | ||
{ | ||
"S3": { | ||
"BucketName": { | ||
"Ref": "MyBucketF68F3FF0" | ||
}, | ||
"Key": "${topic()}/${timestamp()}", | ||
"RoleArn": { | ||
"Fn::GetAtt": [ | ||
"TopicRuleTopicRuleActionRole246C4F77", | ||
"Arn" | ||
] | ||
} | ||
} | ||
} | ||
], | ||
"AwsIotSqlVersion": "2016-03-23", | ||
"Sql": "SELECT topic(2) as device_id FROM 'device/+/data'" | ||
} | ||
} | ||
}, | ||
"TopicRuleTopicRuleActionRole246C4F77": { | ||
"Type": "AWS::IAM::Role", | ||
"Properties": { | ||
"AssumeRolePolicyDocument": { | ||
"Statement": [ | ||
{ | ||
"Action": "sts:AssumeRole", | ||
"Effect": "Allow", | ||
"Principal": { | ||
"Service": "iot.amazonaws.com" | ||
} | ||
} | ||
], | ||
"Version": "2012-10-17" | ||
} | ||
} | ||
}, | ||
"TopicRuleTopicRuleActionRoleDefaultPolicy99ADD687": { | ||
"Type": "AWS::IAM::Policy", | ||
"Properties": { | ||
"PolicyDocument": { | ||
"Statement": [ | ||
{ | ||
"Action": "s3:PutObject", | ||
"Effect": "Allow", | ||
"Resource": { | ||
"Fn::Join": [ | ||
"", | ||
[ | ||
{ | ||
"Fn::GetAtt": [ | ||
"MyBucketF68F3FF0", | ||
"Arn" | ||
] | ||
}, | ||
"/*" | ||
] | ||
] | ||
} | ||
} | ||
], | ||
"Version": "2012-10-17" | ||
}, | ||
"PolicyName": "TopicRuleTopicRuleActionRoleDefaultPolicy99ADD687", | ||
"Roles": [ | ||
{ | ||
"Ref": "TopicRuleTopicRuleActionRole246C4F77" | ||
} | ||
] | ||
} | ||
}, | ||
"MyBucketF68F3FF0": { | ||
"Type": "AWS::S3::Bucket", | ||
"UpdateReplacePolicy": "Delete", | ||
"DeletionPolicy": "Delete" | ||
} | ||
} | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,25 @@ | ||
/// !cdk-integ pragma:ignore-assets | ||
import * as iot from '@aws-cdk/aws-iot'; | ||
import * as s3 from '@aws-cdk/aws-s3'; | ||
import * as cdk from '@aws-cdk/core'; | ||
import * as actions from '../../lib'; | ||
|
||
const app = new cdk.App(); | ||
|
||
class TestStack extends cdk.Stack { | ||
constructor(scope: cdk.App, id: string, props?: cdk.StackProps) { | ||
super(scope, id, props); | ||
|
||
const topicRule = new iot.TopicRule(this, 'TopicRule', { | ||
sql: iot.IotSql.fromStringAsVer20160323("SELECT topic(2) as device_id FROM 'device/+/data'"), | ||
}); | ||
|
||
const bucket = new s3.Bucket(this, 'MyBucket', { | ||
removalPolicy: cdk.RemovalPolicy.DESTROY, | ||
}); | ||
topicRule.addAction(new actions.S3Action(bucket)); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Again, let's use a more sophisticated example of |
||
} | ||
} | ||
|
||
new TestStack(app, 'test-stack'); | ||
app.synth(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you illustrate the optional properties of
S3Action
in this example? Especially how to use the substitution templates.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh. It is important for S3Action! I will illustrate it!