New Pattern: aws-kinesisstreams-gluejob #122
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| lib/*.js | ||
| test/*.js | ||
| *.d.ts | ||
| coverage | ||
| test/lambda/index.js |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| lib/*.js | ||
| test/*.js | ||
| !test/lambda/* | ||
| *.js.map | ||
| *.d.ts | ||
| node_modules | ||
| *.generated.ts | ||
| dist | ||
| .jsii | ||
|
|
||
| .LAST_BUILD | ||
| .nyc_output | ||
| coverage | ||
| .nycrc | ||
| .LAST_PACKAGE | ||
| *.snk |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| # Exclude typescript source and config | ||
| *.ts | ||
| tsconfig.json | ||
| coverage | ||
| .nyc_output | ||
| *.tgz | ||
| *.snk | ||
| *.tsbuildinfo | ||
|
|
||
| # Include javascript files and typescript declarations | ||
| !*.js | ||
| !*.d.ts | ||
|
|
||
| # Exclude jsii outdir | ||
| dist | ||
|
|
||
| # Include .jsii | ||
| !.jsii | ||
|
|
||
| # Include .jsii | ||
| !.jsii |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,151 @@ | ||
| # aws-kinesisstreams-gluejob module | ||
|
|
||
| <!--BEGIN STABILITY BANNER--> | ||
|
|
||
| --- | ||
|
|
||
|  | ||
|
|
||
| > All classes are under active development and subject to non-backward compatible changes or removal in any | ||
| > future version. These are not subject to the [Semantic Versioning](https://semver.org/) model. | ||
| > This means that while you may use them, you may need to update your source code when upgrading to a newer version of this package. | ||
|
|
||
| --- | ||
|
|
||
| <!--END STABILITY BANNER--> | ||
|
|
||
| | **Reference Documentation**: | <span style="font-weight: normal">https://docs.aws.amazon.com/solutions/latest/constructs/</span> | | ||
| | :--------------------------- | :------------------------------------------------------------------------------------------------ | | ||
|
|
||
| <div style="height:8px"></div> | ||
|
|
||
| | **Language** | **Package** | | ||
| | :--------------------------------------------------------------------------------------------- | ------------------------------------------------------------- | | ||
| |  Python | `aws_solutions_constructs.aws_kinesis_streams_gluejob` | | ||
| |  Typescript | `@aws-solutions-constructs/aws-kinesisstreams-gluejob` | | ||
| |  Java | `software.amazon.awsconstructs.services.kinesisstreamgluejob` | | ||
|
|
||
| This AWS Solutions Construct deploys a Kinesis Stream and configures a AWS Glue Job to perform custom ETL transformation with the appropriate resources/properties for interaction and security. It also creates an S3 bucket where the python script for the AWS Glue Job can be uploaded | ||
|
|
||
| Here is a minimal deployable pattern definition in Typescript: | ||
|
|
||
| ```javascript | ||
| const fieldSchema: CfnTable.ColumnProperty [] = [{ | ||
| name: 'id', | ||
| type: 'int', | ||
| comment: 'Identifier for the record', | ||
| }, | ||
| { | ||
| name: 'name', | ||
| type: 'string', | ||
| comment: 'Name for the record', | ||
| }, | ||
| { | ||
| name: 'address', | ||
| type: 'string', | ||
| comment: 'Address for the record', | ||
| }, | ||
| { | ||
| name: 'value', | ||
| type: 'int', | ||
| comment: 'Value for the record', | ||
| }, | ||
| ] | ||
| ); | ||
|
|
||
| const _customEtlJob = new KinesisStreamGlueJob(this, 'CustomETL', { | ||
| glueJobProps: { | ||
| command: { | ||
| name: 'gluestreaming', | ||
| pythonVersion: '3', | ||
| scriptLocation: new Asset(this, 'ScriptLocation', { | ||
| path: `${__dirname}/../etl/transform.py` | ||
| }).s3ObjectUrl | ||
| } | ||
| }, | ||
| fieldSchema: fieldSchema | ||
| } | ||
| }); | ||
|
|
||
| ``` | ||
|
|
||
| ## Initializer | ||
|
|
||
| ```text | ||
| new KinesisStreamGlueJob(scope: Construct, id: string, props: KinesisStreamsToLambdaProps); | ||
| ``` | ||
|
|
||
| _Parameters_ | ||
|
|
||
| - scope [`Construct`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_core.Construct.html) | ||
| - id `string` | ||
| - props [`KinesisStreamGlueJobProps`](#pattern-construct-props) | ||
|
|
||
| ## Pattern Construct Props | ||
|
|
||
| | **Name** | **Type** | **Description** | | ||
| | :------------------ | :---------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ||
| | kinesisStreamProps? | [`kinesis.StreamProps`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-kinesis.StreamProps.html) | Optional user-provided props to override the default props for the Kinesis stream. | | ||
| | existingStreamObj? | [`kinesis.Stream`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-kinesis.Stream.html) | Existing instance of Kinesis Stream, if this is set then kinesisStreamProps is ignored. | | ||
| | glueJobProps? | [`cfnJob.CfnJobProps`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-glue.CfnJobProps.html) | User provided props to override the default props for the CfnJob. | | ||
| | existingGlueJob? | [`cfnJob.CfnJob`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-glue.CfnJob.html) | Existing CfnJob configuration, if this this set then glueJobProps is ignored. | | ||
| | fieldSchema? | [`CfnTable.ColumnProperty[]`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-glue.CfnTable.ColumnProperty.html) | Glue Database for this construct. If not provided the construct will create a new Glue Database. The database is where the schema for the data in Kinesis Data Streams is stored | | ||
| | database? | [`CfnDatabase`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-glue.CfnDatabase.html) | Glue Database for this construct. If not provided the construct will create a new Glue Database. The database is where the schema for the data in Kinesis Data Streams is stored | | ||
|
There was a problem hiding this comment. If we allow the client to pass existingDatabase, we should allow them the alternative of passing database props. Same for table (and also for bucket which we noted below). There was a problem hiding this comment. Ok, updating the interface. |
||
| | table? | [`CfnTable`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-glue.CfnTable.html) | Glue Table for this construct, If not provided the construct will create a new Table in the database. This table should define the schema for the records in the Kinesis Data Streams. Either @table or @fieldSchema is mandatory. If @table is provided then @fieldSchema is ignored | | ||
| | outputDataStore? | [`SinkDataStoreProps`](#sinkdatastoreprops) | The output datastore properties where the ETL output should be. Current datastore types suported is only S3. written | | ||
|
There was a problem hiding this comment. Incomplete sentence or typo at the end There was a problem hiding this comment. Yes, good catch, will be removed. |
||
|
|
||
| ## SinkDataStoreProps | ||
|
|
||
| | **Name** | **Type** | **Description** | | ||
| | :-------------- | :-------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ||
| | s3OutputBucket? | [`Bucket`](https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-s3.Bucket.html) | The output S3 location where the data should be written. The provided S3 bucket will be used to pass the output location to the etl script as an argument to the AWS Glue job. If not provided, the construct will create one | | ||
|
There was a problem hiding this comment. name should start with existing If we accept an existing bucket, we should also accept a BucketProps object. There was a problem hiding this comment. updating interface prop names |
||
| | datastoreStype | [`SinkStoreType`](#sinkstoretype) | Sink data store type | | ||
|
|
||
| ## SinkStoreType | ||
|
|
||
| Enumeration of data store types that could include S3, DynamoDB, DocumentDB, RDS or Redshift. Current construct implementation only supports S3, but potential to add other output types in the future | ||
|
|
||
| | **Name** | **Type** | **Description** | | ||
| | :------- | :------- | --------------- | | ||
| | S3 | `string` | S3 storage type | | ||
|
|
||
| # Default settings | ||
|
|
||
| Out of the box implementation of the Construct without any override will set the following defaults: | ||
|
|
||
| ### Amazon Kinesis Stream | ||
|
|
||
| - Configure least privilege access IAM role for Kinesis Stream | ||
| - Enable server-side encryption for Kinesis Stream using AWS Managed KMS Key | ||
| - Deploy best practices CloudWatch Alarms for the Kinesis Stream | ||
|
|
||
| ### Glue Job | ||
|
|
||
| - Create a Glue Security Config that configures encryption for CloudWatch, Job Bookmarks, and S3. CloudWatch and Job Bookmarks are encrypted using AWS Managed KMS Key created for AWS Glue Service. The S3 bucket is configured with SSE-S3 encryption mode | ||
| - Configure service role policies that allow AWS Glue to read from Kinesis Data Streams. | ||
|
|
||
| ### Glue Database | ||
|
|
||
| - A Glue database to add the table required to define a Glue Table structure and schema for the Kinesis stream | ||
|
|
||
| ### Glue Table | ||
|
|
||
| - A Table with storage descriptor and table input properties using the schema details provided for the records in the Kinesis Data Streams | ||
|
There was a problem hiding this comment. The description sounds too complicated and confusing, can it be simplified ? |
||
|
|
||
| ### IAM Role | ||
|
|
||
| - A job execution role that has privileges to read the ETL script from the S3 bucket location, read from the Kinesis Stream, and execute the Glue Job. The permissions to write to a specific location, is not configured by the construct. | ||
|
|
||
| ### Output S3 Bucket | ||
|
|
||
| - An S3 bucket to store the output of the ETL transformation. This bucket will be passed as an argument to the created glue job so that it can be used in the ETL script to write data into it | ||
|
|
||
| ## Architecture | ||
|
|
||
|  | ||
|
|
||
| ## Reference Implementation | ||
|
|
||
| A sample use case which uses this pattern is available under [`use_cases/aws-custom-glue-etl`](https://github.com/awslabs/aws-solutions-constructs/tree/master/source/use_cases/aws-custom-glue-etl). | ||
|
|
||
| © Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. | ||
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please remove/rollback these changes, it's not needed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
removing all the package.json entries