diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md new file mode 100644 index 00000000..d580f686 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -0,0 +1,31 @@ +--- +name: Bug report +about: Create a report to help us improve +title: '' +labels: bug +assignees: '' + +--- + +**Describe the bug** +A clear and concise description of what the bug is. + +**To Reproduce** +Steps to reproduce the behavior. + +**Expected behavior** +A clear and concise description of what you expected to happen. + +**Please complete the following information about the solution:** +- [ ] Version: [e.g. v1.0.0] +- [ ] Region: [e.g. us-east-1] +- [ ] Was the solution modified from the version published on this repository? +- [ ] If the answer to the previous question was yes, are the changes available on GitHub? +- [ ] Have you checked your [service quotas](https://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html) for the sevices this solution uses? +- [ ] Were there any errors in the CloudWatch Logs? + +**Screenshots** +If applicable, add screenshots to help explain your problem (please **DO NOT include sensitive information**). + +**Additional context** +Add any other context about the problem here. diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md new file mode 100644 index 00000000..d3d209fa --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.md @@ -0,0 +1,17 @@ +--- +name: Feature request +about: Suggest an idea for this solution +title: '' +labels: enhancement +assignees: '' + +--- + +**Is your feature request related to a problem? Please describe.** +A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] + +**Describe the feature you'd like** +A clear and concise description of what you want to happen. + +**Additional context** +Add any other context or screenshots about the feature request here. diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 6bdaa999..de50e4d9 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -2,5 +2,4 @@ *Description of changes:* - By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice. diff --git a/CHANGELOG.md b/CHANGELOG.md index 62b9ae80..4e8f1d46 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,30 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## [5.1.0] - 2020-04-30 +### Added +- Default encryption to SNS topic +- Environment variable to configure the AWS SDK to reuse TCP connections (https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/node-reusing-connections.html) +- SQS queue options at deployment to capture workflow outputs +- Support for accelerated transcoding in Elemental MediaCoonvert: https://aws.amazon.com/about-aws/whats-new/2019/10/announcing-new-aws-elemental-mediaconvert-features-for-accelerated-transcoding-dash-and-avc-video-quality/ +- support for Glacier deep archive + +### Changed +- Lambda functions runtime to latest available (Node.js 12) +- Build assets to include package-lock.json files +- Build and test commands to use _npm ci_ instead of _npm install_ +- Cloudformation template to use _AWS::Partition_ instead of _aws_ +- Logic to add MediaPackage VOD as a custom origin to CloudFront (it's now done as a custom resource when the stack is created / updated) + +### Fixed +- Links in README file +- fix to buildUrl function in output-validate lambda to support non root objects (https://github.com/awslabs/video-on-demand-on-aws/issues/61) +- fix mediainfo lambda function signing method error (https://github.com/awslabs/video-on-demand-on-aws/issues/670 + +### Removed +- _'use strict'_ directives + ## [5.0.0] - 2019-11-20 ### Added - MediaPackage VOD support @@ -35,4 +59,3 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Lambda functions (except for mediainfo) runtime to nodejs10.x - Mediainfo lambda function to python3.7 - Mediainfo executable version (from v0.7.92.1 to v19.09) - diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 2a0abf34..200680d3 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,14 +1,11 @@ # Contributing Guidelines - Thank you for your interest in contributing to our project. Whether it's a bug report, new feature, correction, or additional documentation, we greatly value feedback and contributions from our community. Please read through this document before submitting any issues or pull requests to ensure we have all the necessary information to effectively respond to your bug report or contribution. - ## Reporting Bugs/Feature Requests - We welcome you to use the GitHub issue tracker to report bugs or suggest features. When filing an issue, please check [existing open](https://github.com/awslabs/video-on-demand-on-aws/issues), or [recently closed](https://github.com/awslabs/video-on-demand-on-aws/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aclosed%20), issues to make sure somebody else hasn't already @@ -19,7 +16,6 @@ reported the issue. Please try to include as much information as you can. Detail * Any modifications you've made relevant to the bug * Anything unusual about your environment or deployment - ## Contributing via Pull Requests Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that: @@ -39,23 +35,17 @@ To send us a pull request, please: GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and [creating a pull request](https://help.github.com/articles/creating-a-pull-request/). - ## Finding contributions to work on Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels ((enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any ['help wanted'](https://github.com/awslabs/video-on-demand-on-aws/labels/help%20wanted) issues is a great place to start. - ## Code of Conduct This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct). For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact opensource-codeofconduct@amazon.com with any additional questions or comments. - ## Security issue notifications If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue. - ## Licensing - See the [LICENSE](https://github.com/awslabs/video-on-demand-on-aws/blob/master/LICENSE.txt) file for our project's licensing. We will ask you to confirm the licensing of your contribution. - We may ask you to sign a [Contributor License Agreement (CLA)](http://en.wikipedia.org/wiki/Contributor_License_Agreement) for larger changes. diff --git a/LICENSE.txt b/LICENSE.txt index f30fb6fb..e59e533e 100644 --- a/LICENSE.txt +++ b/LICENSE.txt @@ -187,7 +187,7 @@ same "printed page" as the copyright notice for easier identification within third-party archives. - Copyright 2019 - 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. + Copyright 2019 - 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. diff --git a/NOTICE.txt b/NOTICE.txt index a845ecb8..080a98b0 100644 --- a/NOTICE.txt +++ b/NOTICE.txt @@ -1,6 +1,6 @@ Video on Demand on AWS -Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. +Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. ********************** THIRD PARTY COMPONENTS diff --git a/README.md b/README.md index bd1bd557..37f71943 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ # Video on Demand on AWS How to implement a video-on-demand workflow on AWS leveraging AWS Step Functions, AWS Elemental MediaConvert, and AWS Elemental MediaPackage. -Source code for [Video on Demand on AWS](https://aws.amazon.com/answers/media-entertainment/video-on-demand-on-aws/) solution. +Source code for [Video on Demand on AWS](https://aws.amazon.com/solutions/video-on-demand-on-aws/) solution. ## On this Page - [Architecture Overview](#architecture-overview) @@ -10,6 +10,8 @@ Source code for [Video on Demand on AWS](https://aws.amazon.com/answers/media-en - [Source Metadata Option](#source-metadata-option) - [Encoding Templates](#encoding-templates) - [QVBR Mode](#qvbr-mode) +- [Accelerated Transcoding](#accelerated-transcoding) +- [Source Code](#source-code) - [Creating a custom Build](#creating-a-custom-build) - [Additional Resources](#additional-resources) @@ -35,6 +37,9 @@ The workflow configuration is set at deployment and is defined as environment va * **MediaConvert_Template_720p:** The name of the SD template in MediaConvert * **Source:** The name of the source S3 bucket * **WorkflowName:** Used to tag all of the MediaConvert encoding jobs +* **acceleratedTranscoding** Enabled Accelerated Transocding in MediaConvert. options include ENABLE, DISABLE, PREFERRED. for more detials please see: +* **enableSns** Send SNS notifications for the workflow results. +* **enableSqs** Send the workflow results to an SQS queue ### WorkFlow Triggers @@ -62,7 +67,7 @@ The only required field for the metadata file is the **srcVideo**. The workflow ``` { "srcVideo": "string", - "archiveSource": boolean, + "archiveSource": string, "frameCapture": boolean, "srcBucket": "string", "destBucket": "string", @@ -71,7 +76,12 @@ The only required field for the metadata file is the **srcVideo**. The workflow "jobTemplate_1080p": "string", "jobTemplate_720p": "string", "jobTemplate": "custom-job-template", - "inputRotate": "DEGREE_0|DEGREES_90|DEGREES_180|DEGREES_270|AUTO" + "inputRotate": "DEGREE_0|DEGREES_90|DEGREES_180|DEGREES_270|AUTO", + "captions": { + "srcFile": "string", + "fontSize": integer, + "fontColor": "WHITE|BLACK|YELLOW|RED|GREEN|BLUE" + } } ``` @@ -112,7 +122,18 @@ AWS MediaConvert Quality-defined Variable Bit-Rate (QVBR) control mode gets the For more detail please see [QVBR and MediaConvert](https://docs.aws.amazon.com/mediaconvert/latest/ug/cbr-vbr-qvbr.html). -## Source code (Node.js 10) +## Accelerated Transcoding +Version 5.1.0 introduces support for accelerated transcoding which is a pro tier feature of AWS Elemental MediaConvert. This feature can be configured when launching the template with one of the following options: + +* **ENABLED** All files upload will have acceleration enabled. Files that are not supported will not be processed and the workflow will fail +* **PREFERRED** All files uploaded will be processed but only supported files will have acceleration enabled, the workflow will not fail. +* **DISABLED** No acceleration. + +For more detail please see [Accelerated Transcoding](https://docs.aws.amazon.com/mediaconvert/latest/ug/accelerated-transcoding.html). + +## Source code + +### Node.js 12 * **archive-source:** Lambda function to tag the source video in s3 to enable the Glacier lifecycle policy. * **custom-resource:** Lambda backed CloudFormation custom resource to deploy MediaConvert templates configure S3 event notifications. * **dynamo:** Lambda function to Update DynamoDB. @@ -124,10 +145,10 @@ For more detail please see [QVBR and MediaConvert](https://docs.aws.amazon.com/m * **profiler:** Lambda function used to send publish and/or error notifications. * **step-functions:** Lambda function to trigger AWS Step Functions. -## Source code (Python 3.7) -> **Note**: The _mediainfo_ function uses the python3.7 runtime since the distributable was compiled on Amazon Linux, and the [Operating System for the node version 10 runtime](https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html) is Amazon Linux 2. +### Python 3.7 +* **mediainfo:** Lambda function to run [mediainfo](https://mediaarea.net/en/MediaInfo) on an S3 signed url. -* **mediainfo:** Lambda function to run mediainfo on s3 signed url. https://mediaarea.net/en/MediaInfo. bin/mediainfo must be made executable before deploying to lambda. +> ./source/mediainfo/bin/mediainfo must be made executable before deploying to lambda. ## Creating a custom build The solution can be deployed through the CloudFormation template available on the solution home page: [Video on Demand on AWS](https://aws.amazon.com/answers/media-entertainment/video-on-demand-on-aws/). @@ -135,8 +156,8 @@ To make changes to the solution, download or clone this repo, update the source ### Prerequisites: * [AWS Command Line Interface](https://aws.amazon.com/cli/) -* Node.js 10.x or later -* Python 3.7 or later +* Node.js 12.x or later +* Python 3.8 or later ### 1. Running unit tests for customization Run unit tests to make sure added customization passes the tests: @@ -156,14 +177,14 @@ aws s3 mb s3://my-bucket-us-east-1 Build the distributable: ``` chmod +x ./build-s3-dist.sh -./build-s3-dist.sh my-bucket video-on-demand-on-aws v5.0.0-custom +./build-s3-dist.sh my-bucket video-on-demand-on-aws version ``` > **Notes**: The _build-s3-dist_ script expects the bucket name as one of its parameters, and this value should not include the region suffix. Deploy the distributable to the Amazon S3 bucket in your account: ``` -aws s3 cp ./regional-s3-assets/ s3://my-bucket-us-east-1/video-on-demand-on-aws/v5.0.0-custom/ --recursive --acl bucket-owner-full-control +aws s3 cp ./regional-s3-assets/ s3://my-bucket-us-east-1/video-on-demand-on-aws/version/ --recursive --acl bucket-owner-full-control ``` ### 4. Launch the CloudFormation template. @@ -175,22 +196,22 @@ aws s3 cp ./regional-s3-assets/ s3://my-bucket-us-east-1/video-on-demand-on-aws/ ### Services - [AWS Elemental MediaConvert](https://aws.amazon.com/mediaconvert/) - [AWS Elemental MediaPackage](https://aws.amazon.com/mediapackage/) -- [AWS Step Functions](https://aws.amazon.com/mediapackage/) +- [AWS Step Functions](https://aws.amazon.com/step-functions/) - [AWS Lambda](https://aws.amazon.com/lambda/) - [Amazon CloudFront](https://aws.amazon.com/cloudfront/) - [OTT Workflows](https://www.elemental.com/applications/ott-workflows) - [QVBR and MediaConvert](https://docs.aws.amazon.com/mediaconvert/latest/ug/cbr-vbr-qvbr.html) ### Other Solutions and Demos -- [Live Streaming On AWS](https://aws.amazon.com/answers/media-entertainment/live-streaming/) -- [Media Analysis Solution](https://aws.amazon.com/answers/media-entertainment/media-analysis-solution/) +- [Live Streaming On AWS](https://aws.amazon.com/solutions/live-streaming-on-aws/) +- [Media Analysis Solution](https://aws.amazon.com/solutions/media-analysis-solution/) - [Live Streaming and Live to VOD Workshop](https://github.com/awslabs/speke-reference-server) - [Live to VOD with Machine Learning](https://github.com/aws-samples/aws-elemental-instant-video-highlights) - [Demo SPEKE Reference Server](https://github.com/awslabs/speke-reference-server) *** -Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. +Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. diff --git a/architecture.png b/architecture.png index 652bdc17..8ba4bf0e 100644 Binary files a/architecture.png and b/architecture.png differ diff --git a/deployment/build-s3-dist.sh b/deployment/build-s3-dist.sh index dd06d1d9..109f5696 100755 --- a/deployment/build-s3-dist.sh +++ b/deployment/build-s3-dist.sh @@ -67,17 +67,22 @@ chmod +x ./mediainfo/bin/mediainfo for folder in */ ; do cd "$folder" + function_name=${PWD##*/} - echo "Creating deployment package for $function_name" + zip_path="$build_dist_dir/$function_name.zip" + + echo "Creating deployment package for $function_name at $zip_path" if [ -e "package.json" ]; then rm -rf node_modules/ - npm install --production - rm package-lock.json + npm i --production - zip -q -r9 "$build_dist_dir/$function_name.zip" . - else - python3 setup.py build_pkg --zip-path="$build_dist_dir/$function_name.zip" + zip -q -r9 $zip_path . + elif [ -e "setup.py" ]; then + # If you're running this command on macOS and Python3 has been installed using Homebrew, you might see this issue: + # DistutilsOptionError: must supply either home or prefix/exec-prefix + # Please follow the workaround suggested on this StackOverflow answer: https://stackoverflow.com/a/44728772 + python3 setup.py build_pkg --zip-path=$zip_path fi cd .. diff --git a/deployment/run-unit-tests.sh b/deployment/run-unit-tests.sh index f9d9449c..c8d40e4a 100755 --- a/deployment/run-unit-tests.sh +++ b/deployment/run-unit-tests.sh @@ -32,5 +32,12 @@ npm test cd ../step-functions npm test +# If you're running these commands on macOS and Python3 has been installed using Homebrew, you might see this issue: +# DistutilsOptionError: must supply either home or prefix/exec-prefix +# Please follow the workaround suggested on this StackOverflow answer: https://stackoverflow.com/a/44728772 cd ../mediainfo -python3 setup.py test +rm -rf ./pytests && mkdir ./pytests +cp lambda_function.py ./test*.py ./pytests +pip3 install boto3 -t ./pytests +python3 -m unittest discover -s ./pytests -v +rm -rf ./pytests diff --git a/deployment/video-on-demand-on-aws.yaml b/deployment/video-on-demand-on-aws.yaml index a95bfb3d..4845ca58 100644 --- a/deployment/video-on-demand-on-aws.yaml +++ b/deployment/video-on-demand-on-aws.yaml @@ -1,6 +1,7 @@ -Description: '(SO0021) - Video On Demand workflow with AWS Step Functions, MediaConvert, MediaPackage, S3, CloudFront and DynamoDB. Version %%VERSION%%' +Description: "(SO0021) - Video On Demand workflow with AWS Step Functions, MediaConvert, MediaPackage, S3, CloudFront and DynamoDB. Version %%VERSION%%" Parameters: + AdminEmail: Description: Email address for SNS notifications (subscribed users will receive ingest, publishing, and error notifications) Type: String @@ -15,12 +16,13 @@ Parameters: - MetadataFile Glacier: - Description: If enabled, source assets will be tagged for archiving to Glacier once the workflow is complete + Description: If enabled, source assets will be tagged for archiving to Glacier or Glacier Deep Archive once the workflow is complete Type: String - Default: No + Default: DISABLED AllowedValues: - - Yes - - No + - DISABLED + - GLACIER + - DEEP_ARCHIVE FrameCapture: Description: If enabled, frame capture is added to the job submitted to MediaConvert @@ -38,6 +40,31 @@ Parameters: - Yes - No + EnableSns: + Description: Enable Ingest and Publish email notifications, error messages are not afeected by this parameter. + Type: String + Default: Yes + AllowedValues: + - Yes + - No + + EnableSqs: + Description: Publish the workflow results to an SQS queue to injest upstream + Type: String + Default: Yes + AllowedValues: + - Yes + - No + + AcceleratedTranscoding: + Description: Enable accelerated transcoding in AWS Elemental MediaConvert. PREFERRED will only use acceleration if the input files is supported. ENABLED accleration is applied to all files (this will fail for unsupported file types) see MediaConvert Documentation for more detail https://docs.aws.amazon.com/mediaconvert/latest/ug/accelerated-transcoding.html + Type: String + Default: PREFERRED + AllowedValues: + - ENABLED + - DISABLED + - PREFERRED + Metadata: AWS::CloudFormation::Interface: ParameterGroups: @@ -48,11 +75,14 @@ Metadata: - AdminEmail - WorkflowTrigger - Glacier + - EnableSns + - EnableSqs - Label: default: "AWS Elemental MediaConvert" Parameters: - FrameCapture + - AcceleratedTranscoding - Label: default: "AWS Elemental MediaPackage" @@ -69,13 +99,18 @@ Metadata: default: Enable Frame Capture EnableMediaPackage: default: Enable MediaPackage + AcceleratedTranscoding: + default: Accelerated Transcoding + EnableSns: + default: Enable SNS Notifications + EnableSqs: + default: Enable SQS Messaging Mappings: SourceCode: General: S3Bucket: "%%BUCKET_NAME%%" KeyPrefix: "%%SOLUTION_NAME%%/%%VERSION%%" - AnonymousData: SendAnonymousData: Data: Yes @@ -91,8 +126,7 @@ Resources: AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - - - Effect: Allow + - Effect: Allow Principal: Service: - lambda.amazonaws.com @@ -100,34 +134,24 @@ Resources: - sts:AssumeRole Path: "/" Policies: - - - PolicyName: !Sub "${AWS::StackName}-custom-resource" + - PolicyName: !Sub "${AWS::StackName}-custom-resource" PolicyDocument: Statement: - - - Effect: Allow + - Effect: Allow Action: - logs:CreateLogGroup - logs:CreateLogStream - logs:PutLogEvents Resource: - - !Sub "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/*" - - - Effect: Allow - Action: - - lambda:UpdateFunctionConfiguration - Resource: - - !Sub "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:*" - - - Effect: Allow + - !Sub "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/*" + - Effect: Allow Action: - s3:PutBucketNotification - s3:PutObject - s3:PutObjectAcl Resource: - - !Sub ${Source.Arn} - - - Effect: Allow + - !GetAtt Source.Arn + - Effect: Allow Action: - mediaconvert:CreatePreset - mediaconvert:CreateJobTemplate @@ -136,19 +160,21 @@ Resources: - mediaconvert:DescribeEndpoints - mediaconvert:ListJobTemplates Resource: - - !Sub "arn:aws:mediaconvert:${AWS::Region}:${AWS::AccountId}:*" - - - Effect: Allow + - !Sub "arn:${AWS::Partition}:mediaconvert:${AWS::Region}:${AWS::AccountId}:*" + - Effect: Allow Action: - mediapackage-vod:DeleteAsset - mediapackage-vod:DeletePackagingConfiguration + Resource: + - !Sub "arn:${AWS::Partition}:mediapackage-vod:${AWS::Region}:${AWS::AccountId}:assets/*" + - !Sub "arn:${AWS::Partition}:mediapackage-vod:${AWS::Region}:${AWS::AccountId}:packaging-configurations/packaging-config-*" + - Effect: Allow + Action: + - mediapackage-vod:DescribePackagingGroup - mediapackage-vod:DeletePackagingGroup Resource: - - !Sub "arn:aws:mediapackage-vod:${AWS::Region}:${AWS::AccountId}:assets/*" - - !Sub "arn:aws:mediapackage-vod:${AWS::Region}:${AWS::AccountId}:packaging-configurations/packaging-config-*" - - !Sub "arn:aws:mediapackage-vod:${AWS::Region}:${AWS::AccountId}:packaging-groups/*" - - - Effect: Allow + - !Sub "arn:${AWS::Partition}:mediapackage-vod:${AWS::Region}:${AWS::AccountId}:packaging-groups/${AWS::StackName}-packaging-group" + - Effect: Allow Action: - mediapackage-vod:CreatePackagingConfiguration - mediapackage-vod:CreatePackagingGroup @@ -156,6 +182,12 @@ Resources: - mediapackage-vod:ListPackagingConfigurations - mediapackage-vod:ListPackagingGroups Resource: "*" + - Effect: Allow + Action: + - cloudfront:GetDistributionConfig + - cloudfront:UpdateDistribution + Resource: + - !Sub "arn:${AWS::Partition}:cloudfront::${AWS::AccountId}:distribution/${CloudFront}" Metadata: cfn_nag: rules_to_suppress: @@ -168,24 +200,21 @@ Resources: AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - - - Effect: Allow + - Effect: Allow Principal: Service: - !Sub "states.${AWS::Region}.amazonaws.com" Action: - sts:AssumeRole Policies: - - - PolicyName: !Sub "${AWS::StackName}-stepfunctions-service-role" + - PolicyName: !Sub "${AWS::StackName}-stepfunctions-service-role" PolicyDocument: Statement: - - - Effect: Allow + - Effect: Allow Action: - lambda:InvokeFunction Resource: - - !Sub "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:*" + - !Sub "arn:${AWS::Partition}:lambda:${AWS::Region}:${AWS::AccountId}:function:*" Metadata: cfn_nag: rules_to_suppress: @@ -198,32 +227,28 @@ Resources: AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - - - Effect: Allow + - Effect: Allow Principal: Service: - - "mediaconvert.amazonaws.com" + - mediaconvert.amazonaws.com Action: - sts:AssumeRole Policies: - - - PolicyName: !Sub "${AWS::StackName}-mediatranscode-policy" + - PolicyName: !Sub "${AWS::StackName}-mediatranscode-policy" PolicyDocument: Statement: - - - Effect: Allow + - Effect: Allow Action: - s3:GetObject - s3:PutObject Resource: - !Sub "${Source.Arn}/*" - !Sub "${Destination.Arn}/*" - - - Effect: Allow + - Effect: Allow Action: - - "execute-api:Invoke" + - execute-api:Invoke Resource: - - !Sub "arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:*" + - !Sub "arn:${AWS::Partition}:execute-api:${AWS::Region}:${AWS::AccountId}:*" Metadata: cfn_nag: rules_to_suppress: @@ -236,20 +261,17 @@ Resources: AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - - - Effect: Allow + - Effect: Allow Principal: Service: - - "mediapackage.amazonaws.com" + - mediapackage.amazonaws.com Action: - sts:AssumeRole Policies: - - - PolicyName: !Sub "${AWS::StackName}-mediapackagevod-policy" + - PolicyName: !Sub "${AWS::StackName}-mediapackagevod-policy" PolicyDocument: Statement: - - - Effect: Allow + - Effect: Allow Action: - s3:GetObject - s3:GetBucketLocation @@ -293,18 +315,17 @@ Resources: Bucket: !Ref Destination PolicyDocument: Statement: - - + - Effect: Allow Action: - - "s3:GetObject" - Effect: "Allow" - Resource: !Sub "arn:aws:s3:::${Destination}/*" + - s3:GetObject + Resource: !Sub "arn:${AWS::Partition}:s3:::${Destination}/*" Principal: CanonicalUser: !GetAtt DestinationOriginAccessIdentity.S3CanonicalUserId EncodeCompleteRule: Type: AWS::Events::Rule Properties: - Name: !Sub ${AWS::StackName}-EncodeComplete + Name: !Sub "${AWS::StackName}-EncodeComplete" Description: MediaConvert Completed event rule EventPattern: source: @@ -314,16 +335,15 @@ Resources: - COMPLETE userMetadata: workflow: - - !Sub ${AWS::StackName} + - !Ref AWS::StackName Targets: - - - Arn: !GetAtt StepFunctions.Arn - Id: !Sub ${AWS::StackName}-StepFunctions + - Arn: !GetAtt StepFunctions.Arn + Id: !Sub "${AWS::StackName}-StepFunctions" EncodeErrorRule: Type: AWS::Events::Rule Properties: - Name: !Sub ${AWS::StackName}-EncodeError + Name: !Sub "${AWS::StackName}-EncodeError" Description: MediaConvert Error event rule EventPattern: source: @@ -333,29 +353,27 @@ Resources: - ERROR userMetadata: workflow: - - !Sub ${AWS::StackName} + - !Ref AWS::StackName Targets: - - - Arn: !GetAtt ErrorHandler.Arn - Id: !Sub ${AWS::StackName}-EncodeError + - Arn: !GetAtt ErrorHandler.Arn + Id: !Sub "${AWS::StackName}-EncodeError" DynamoDBTable: Type: AWS::DynamoDB::Table Properties: BillingMode: PAY_PER_REQUEST AttributeDefinitions: - - AttributeName: guid - AttributeType: S - - AttributeName: srcBucket - AttributeType: S - - AttributeName: startTime - AttributeType: S + - AttributeName: guid + AttributeType: S + - AttributeName: srcBucket + AttributeType: S + - AttributeName: startTime + AttributeType: S KeySchema: - - AttributeName: guid - KeyType: HASH + - AttributeName: guid + KeyType: HASH GlobalSecondaryIndexes: - - - IndexName: srcBucket-startTime-index + - IndexName: srcBucket-startTime-index KeySchema: - AttributeName: srcBucket KeyType: HASH @@ -369,9 +387,12 @@ Resources: rules_to_suppress: - id: W28 reason: "Table name is set to the stack name" + - id: W74 + reason: " The DynamoDB table is configured to use the default encryption" Source: DeletionPolicy: Retain + UpdateReplacePolicy: Retain Type: AWS::S3::Bucket Properties: LoggingConfiguration: @@ -379,16 +400,22 @@ Resources: LogFilePrefix: s3-access/ LifecycleConfiguration: Rules: - - - Id: !Sub ${AWS::StackName}-source-archive + - Id: !Sub "${AWS::StackName}-source-archive" TagFilters: - - - Key: !Sub ${AWS::StackName} - Value: archive + - Key: !Ref AWS::StackName + Value: GLACIER + Status: Enabled + Transitions: + - TransitionInDays: 1 + StorageClass: GLACIER + - Id: !Sub "${AWS::StackName}-source-deep-archive" + TagFilters: + - Key: !Ref AWS::StackName + Value: DEEP_ARCHIVE Status: Enabled Transitions: - TransitionInDays: 1 - StorageClass: Glacier + StorageClass: DEEP_ARCHIVE BucketEncryption: ServerSideEncryptionConfiguration: - ServerSideEncryptionByDefault: @@ -402,10 +429,11 @@ Resources: cfn_nag: rules_to_suppress: - id: W51 - reason: "Bucket is private and does not need a bucket policy" + reason: "Bucket does not need a bucket policy" Destination: DeletionPolicy: Retain + UpdateReplacePolicy: Retain Type: AWS::S3::Bucket Properties: LoggingConfiguration: @@ -429,6 +457,7 @@ Resources: Logs: DeletionPolicy: Retain + UpdateReplacePolicy: Retain Type: AWS::S3::Bucket Properties: AccessControl: LogDeliveryWrite @@ -447,17 +476,36 @@ Resources: - id: W35 reason: "Used to store access logs for other buckets" - id: W51 - reason: "Bucket is private and does not need a bucket policy" + reason: "Bucket does not need a bucket policy" SnsTopic: Type: AWS::SNS::Topic Properties: - DisplayName: !Sub ${AWS::StackName}-Notifications + DisplayName: !Sub "${AWS::StackName}-Notifications" + KmsMasterKeyId: alias/aws/sns Subscription: - - - Endpoint: !Ref AdminEmail + - Endpoint: !Ref AdminEmail Protocol: email + SqsQueue: + Type: AWS::SQS::Queue + Properties: + VisibilityTimeout: 120 + QueueName: !Sub ${AWS::StackName} + RedrivePolicy: + deadLetterTargetArn: !Sub ${SqsQueueDlq.Arn} + maxReceiveCount: 1 + KmsDataKeyReusePeriodSeconds: 300 + KmsMasterKeyId: alias/aws/sqs + + SqsQueueDlq: + Type: AWS::SQS::Queue + Properties: + VisibilityTimeout: 120 + QueueName: !Sub ${AWS::StackName}-dlq + KmsDataKeyReusePeriodSeconds: 300 + KmsMasterKeyId: alias/aws/sqs + DestinationOriginAccessIdentity: Type: AWS::CloudFront::CloudFrontOriginAccessIdentity Properties: @@ -496,6 +544,11 @@ Resources: PriceClass: PriceClass_100 ViewerCertificate: CloudFrontDefaultCertificate: true + Metadata: + cfn_nag: + rules_to_suppress: + - id: W70 + reason: "CloudFront automatically sets the security policy to TLSv1 when the distribution uses the CloudFront domain name (CloudFrontDefaultCertificate=true)" S3Config: DependsOn: CloudFront @@ -511,14 +564,14 @@ Resources: Type: Custom::LoadLambda Properties: ServiceToken: !GetAtt CustomResource.Arn - Resource: "EndPoint" + Resource: EndPoint MediaConvertTemplates: Type: Custom::LoadLambda Properties: ServiceToken: !GetAtt CustomResource.Arn - Resource: "MediaConvertTemplates" - StackName: !Sub ${AWS::StackName} + Resource: MediaConvertTemplates + StackName: !Ref AWS::StackName EndPoint: !GetAtt MediaConvertEndPoint.EndpointUrl EnableMediaPackage: !Ref EnableMediaPackage @@ -526,23 +579,28 @@ Resources: Type: Custom::LoadLambda Properties: ServiceToken: !GetAtt CustomResource.Arn - Resource: "MediaPackageVod" - StackName: !Sub ${AWS::StackName} + Resource: MediaPackageVod + StackName: !Ref AWS::StackName GroupId: !Sub "${AWS::StackName}-packaging-group" PackagingConfigurations: "HLS,DASH,MSS,CMAF" + DistributionId: !Ref CloudFront + EnableMediaPackage: !Ref EnableMediaPackage CustomResource: Type: AWS::Lambda::Function Properties: - FunctionName: !Sub ${AWS::StackName}-custom-resource + FunctionName: !Sub "${AWS::StackName}-custom-resource" Description: Used to deploy resources not supported by CloudFormation Handler: index.handler Role: !GetAtt CustomResourceRole.Arn Code: S3Bucket: !Join ["-", [!FindInMap ["SourceCode", "General", "S3Bucket"], Ref: "AWS::Region"]] S3Key: !Join ["/", [!FindInMap ["SourceCode", "General", "KeyPrefix"], "custom-resource.zip"]] - Runtime: nodejs10.x + Runtime: nodejs12.x Timeout: 180 + Environment: + Variables: + AWS_NODEJS_CONNECTION_REUSE_ENABLED: "1" StepFunctionsRole: Type: AWS::IAM::Role @@ -550,40 +608,35 @@ Resources: AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - - - Effect: Allow + - Effect: Allow Principal: Service: - lambda.amazonaws.com Action: - sts:AssumeRole Policies: - - - PolicyName: !Sub "${AWS::StackName}-step-functions-role" + - PolicyName: !Sub "${AWS::StackName}-step-functions-role" PolicyDocument: Statement: - - - Effect: Allow + - Effect: Allow Action: - states:StartExecution Resource: - - !Sub "arn:aws:states:${AWS::Region}:${AWS::AccountId}:stateMachine:${AWS::StackName}-ingest" - - !Sub "arn:aws:states:${AWS::Region}:${AWS::AccountId}:stateMachine:${AWS::StackName}-process" - - !Sub "arn:aws:states:${AWS::Region}:${AWS::AccountId}:stateMachine:${AWS::StackName}-publish" - - - Effect: Allow + - !Sub "arn:${AWS::Partition}:states:${AWS::Region}:${AWS::AccountId}:stateMachine:${AWS::StackName}-ingest" + - !Sub "arn:${AWS::Partition}:states:${AWS::Region}:${AWS::AccountId}:stateMachine:${AWS::StackName}-process" + - !Sub "arn:${AWS::Partition}:states:${AWS::Region}:${AWS::AccountId}:stateMachine:${AWS::StackName}-publish" + - Effect: Allow Action: - lambda:InvokeFunction Resource: - - !Sub "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${AWS::StackName}-error-handler" - - - Effect: Allow + - !Sub "arn:${AWS::Partition}:lambda:${AWS::Region}:${AWS::AccountId}:function:${AWS::StackName}-error-handler" + - Effect: Allow Action: - logs:CreateLogGroup - logs:CreateLogStream - logs:PutLogEvents Resource: - - !Sub "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/*" + - !Sub "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/*" Metadata: cfn_nag: rules_to_suppress: @@ -600,13 +653,14 @@ Resources: Code: S3Bucket: !Join ["-", [!FindInMap ["SourceCode", "General", "S3Bucket"], Ref: "AWS::Region"]] S3Key: !Join ["/", [!FindInMap ["SourceCode", "General", "KeyPrefix"], "step-functions.zip"]] - Runtime: nodejs10.x + Runtime: nodejs12.x Timeout: 120 Environment: Variables: - IngestWorkflow: !Sub "arn:aws:states:${AWS::Region}:${AWS::AccountId}:stateMachine:${AWS::StackName}-ingest" - ProcessWorkflow: !Sub "arn:aws:states:${AWS::Region}:${AWS::AccountId}:stateMachine:${AWS::StackName}-process" - PublishWorkflow: !Sub "arn:aws:states:${AWS::Region}:${AWS::AccountId}:stateMachine:${AWS::StackName}-publish" + AWS_NODEJS_CONNECTION_REUSE_ENABLED: "1" + IngestWorkflow: !Sub "arn:${AWS::Partition}:states:${AWS::Region}:${AWS::AccountId}:stateMachine:${AWS::StackName}-ingest" + ProcessWorkflow: !Sub "arn:${AWS::Partition}:states:${AWS::Region}:${AWS::AccountId}:stateMachine:${AWS::StackName}-process" + PublishWorkflow: !Sub "arn:${AWS::Partition}:states:${AWS::Region}:${AWS::AccountId}:stateMachine:${AWS::StackName}-publish" ErrorHandler: !GetAtt ErrorHandler.Arn InputValidateRole: @@ -615,38 +669,33 @@ Resources: AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - - - Effect: Allow + - Effect: Allow Principal: Service: - lambda.amazonaws.com Action: - sts:AssumeRole Policies: - - - PolicyName: !Sub "${AWS::StackName}-input-validate-role" + - PolicyName: !Sub "${AWS::StackName}-input-validate-role" PolicyDocument: Statement: - - - Effect: Allow + - Effect: Allow Action: - s3:GetObject Resource: - !Sub "${Source.Arn}/*" - - - Effect: Allow + - Effect: Allow Action: - lambda:InvokeFunction Resource: - - !Sub "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${AWS::StackName}-error-handler" - - - Effect: Allow + - !Sub "arn:${AWS::Partition}:lambda:${AWS::Region}:${AWS::AccountId}:function:${AWS::StackName}-error-handler" + - Effect: Allow Action: - logs:CreateLogGroup - logs:CreateLogStream - logs:PutLogEvents Resource: - - !Sub "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/*" + - !Sub "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/*" Metadata: cfn_nag: rules_to_suppress: @@ -656,19 +705,20 @@ Resources: InputValidate: Type: AWS::Lambda::Function Properties: - FunctionName: !Sub ${AWS::StackName}-input-validate + FunctionName: !Sub "${AWS::StackName}-input-validate" Description: Validates the input given to the workflow Handler: index.handler Role: !GetAtt InputValidateRole.Arn Code: S3Bucket: !Join ["-", [!FindInMap ["SourceCode", "General", "S3Bucket"], Ref: "AWS::Region"]] S3Key: !Join ["/", [!FindInMap ["SourceCode", "General", "KeyPrefix"], "input-validate.zip"]] - Runtime: nodejs10.x + Runtime: nodejs12.x Timeout: 120 Environment: Variables: + AWS_NODEJS_CONNECTION_REUSE_ENABLED: "1" ErrorHandler: !GetAtt ErrorHandler.Arn - WorkflowName: !Sub ${AWS::StackName} + WorkflowName: !Ref AWS::StackName Source: !Ref Source Destination: !Ref Destination FrameCapture: !Ref FrameCapture @@ -691,6 +741,9 @@ Resources: CloudFront: !GetAtt CloudFront.DomainName EnableMediaPackage: !Ref EnableMediaPackage InputRotate: DEGREE_0 + EnableSns: !Ref EnableSns + EnableSqs: !Ref EnableSqs + AcceleratedTranscoding: !Ref AcceleratedTranscoding MediainfoRole: Type: AWS::IAM::Role @@ -698,48 +751,44 @@ Resources: AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - - - Effect: Allow + - Effect: Allow Principal: Service: - lambda.amazonaws.com Action: - sts:AssumeRole Policies: - - - PolicyName: !Sub "${AWS::StackName}-mediainfo-role" + - PolicyName: !Sub "${AWS::StackName}-mediainfo-role" PolicyDocument: Statement: - - - Effect: Allow + - Effect: Allow Action: - s3:GetObject Resource: - !Sub "${Source.Arn}/*" - - - Effect: Allow + - Effect: Allow Action: - lambda:InvokeFunction Resource: - - !Sub "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${AWS::StackName}-error-handler" - - - Effect: Allow + - !Sub "arn:${AWS::Partition}:lambda:${AWS::Region}:${AWS::AccountId}:function:${AWS::StackName}-error-handler" + - Effect: Allow Action: - logs:CreateLogGroup - logs:CreateLogStream - logs:PutLogEvents Resource: - - !Sub "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/*" + - !Sub "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/*" Metadata: cfn_nag: rules_to_suppress: - id: W11 reason: "* is limited to one S3 bucket" + Mediainfo: Type: AWS::Lambda::Function Properties: - FunctionName: !Sub ${AWS::StackName}-mediainfo + FunctionName: !Sub "${AWS::StackName}-mediainfo" Description: Runs mediainfo on a pre-signed S3 URL Handler: lambda_function.lambda_handler Role: !GetAtt MediainfoRole.Arn @@ -758,38 +807,33 @@ Resources: AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - - - Effect: Allow + - Effect: Allow Principal: Service: - lambda.amazonaws.com Action: - sts:AssumeRole Policies: - - - PolicyName: !Sub "${AWS::StackName}-dynamo-role" + - PolicyName: !Sub "${AWS::StackName}-dynamo-role" PolicyDocument: Statement: - - - Effect: Allow + - Effect: Allow Action: - lambda:InvokeFunction Resource: - - !Sub "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${AWS::StackName}-error-handler" - - - Effect: Allow + - !Sub "arn:${AWS::Partition}:lambda:${AWS::Region}:${AWS::AccountId}:function:${AWS::StackName}-error-handler" + - Effect: Allow Action: - dynamodb:UpdateItem Resource: - - !Sub "arn:aws:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${DynamoDBTable}" - - - Effect: Allow + - !Sub "arn:${AWS::Partition}:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${DynamoDBTable}" + - Effect: Allow Action: - logs:CreateLogGroup - logs:CreateLogStream - logs:PutLogEvents Resource: - - !Sub "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/*" + - !Sub "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/*" Metadata: cfn_nag: rules_to_suppress: @@ -799,17 +843,18 @@ Resources: DynamodbUpdate: Type: AWS::Lambda::Function Properties: - FunctionName: !Sub ${AWS::StackName}-dynamo + FunctionName: !Sub "${AWS::StackName}-dynamo" Description: Updates DynamoDB with event data Handler: index.handler Role: !GetAtt DynamoUpdateRole.Arn Code: S3Bucket: !Join ["-", [!FindInMap ["SourceCode", "General", "S3Bucket"], Ref: "AWS::Region"]] S3Key: !Join ["/", [!FindInMap ["SourceCode", "General", "KeyPrefix"], "dynamo.zip"]] - Runtime: nodejs10.x + Runtime: nodejs12.x Timeout: 120 Environment: Variables: + AWS_NODEJS_CONNECTION_REUSE_ENABLED: "1" DynamoDBTable: !Ref DynamoDBTable ErrorHandler: !GetAtt ErrorHandler.Arn @@ -819,38 +864,33 @@ Resources: AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - - - Effect: Allow + - Effect: Allow Principal: Service: - lambda.amazonaws.com Action: - sts:AssumeRole Policies: - - - PolicyName: !Sub "${AWS::StackName}-profiler-role" + - PolicyName: !Sub "${AWS::StackName}-profiler-role" PolicyDocument: Statement: - - - Effect: Allow + - Effect: Allow Action: - lambda:InvokeFunction Resource: - - !Sub "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${AWS::StackName}-error-handler" - - - Effect: Allow + - !Sub "arn:${AWS::Partition}:lambda:${AWS::Region}:${AWS::AccountId}:function:${AWS::StackName}-error-handler" + - Effect: Allow Action: - dynamodb:GetItem Resource: - - !Sub "arn:aws:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${DynamoDBTable}" - - - Effect: Allow + - !Sub "arn:${AWS::Partition}:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${DynamoDBTable}" + - Effect: Allow Action: - logs:CreateLogGroup - logs:CreateLogStream - logs:PutLogEvents Resource: - - !Sub "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/*" + - !Sub "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/*" Metadata: cfn_nag: rules_to_suppress: @@ -860,17 +900,18 @@ Resources: Profiler: Type: AWS::Lambda::Function Properties: - FunctionName: !Sub ${AWS::StackName}-profiler + FunctionName: !Sub "${AWS::StackName}-profiler" Description: Sets an EncodeProfile based on mediainfo output Handler: index.handler Role: !GetAtt ProfilerRole.Arn Code: S3Bucket: !Join ["-", [!FindInMap ["SourceCode", "General", "S3Bucket"], Ref: "AWS::Region"]] S3Key: !Join ["/", [!FindInMap ["SourceCode", "General", "KeyPrefix"], "profiler.zip"]] - Runtime: nodejs10.x + Runtime: nodejs12.x Timeout: 120 Environment: Variables: + AWS_NODEJS_CONNECTION_REUSE_ENABLED: "1" DynamoDBTable: !Ref DynamoDBTable ErrorHandler: !GetAtt ErrorHandler.Arn @@ -880,45 +921,39 @@ Resources: AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - - - Effect: Allow + - Effect: Allow Principal: Service: - lambda.amazonaws.com Action: - sts:AssumeRole Policies: - - - PolicyName: !Sub "${AWS::StackName}-encode-role" + - PolicyName: !Sub "${AWS::StackName}-encode-role" PolicyDocument: Statement: - - - Effect: Allow + - Effect: Allow Action: - lambda:InvokeFunction Resource: - - !Sub "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${AWS::StackName}-error-handler" - - - Effect: Allow + - !Sub "arn:${AWS::Partition}:lambda:${AWS::Region}:${AWS::AccountId}:function:${AWS::StackName}-error-handler" + - Effect: Allow Action: - mediaconvert:CreateJob - mediaconvert:GetJobTemplate Resource: - - !Sub "arn:aws:mediaconvert:${AWS::Region}:${AWS::AccountId}:*" - - - Effect: Allow + - !Sub "arn:${AWS::Partition}:mediaconvert:${AWS::Region}:${AWS::AccountId}:*" + - Effect: Allow Action: - iam:PassRole Resource: - !GetAtt MediaConvertRole.Arn - - - Effect: Allow + - Effect: Allow Action: - logs:CreateLogGroup - logs:CreateLogStream - logs:PutLogEvents Resource: - - !Sub "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/*" + - !Sub "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/*" Metadata: cfn_nag: rules_to_suppress: @@ -928,17 +963,18 @@ Resources: Encode: Type: AWS::Lambda::Function Properties: - FunctionName: !Sub ${AWS::StackName}-encode + FunctionName: !Sub "${AWS::StackName}-encode" Description: Creates a MediaConvert encode job Handler: index.handler Role: !GetAtt EncodeRole.Arn Code: S3Bucket: !Join ["-", [!FindInMap ["SourceCode", "General", "S3Bucket"], Ref: "AWS::Region"]] S3Key: !Join ["/", [!FindInMap ["SourceCode", "General", "KeyPrefix"], "encode.zip"]] - Runtime: nodejs10.x + Runtime: nodejs12.x Timeout: 120 Environment: Variables: + AWS_NODEJS_CONNECTION_REUSE_ENABLED: "1" ErrorHandler: !GetAtt ErrorHandler.Arn MediaConvertRole: !GetAtt MediaConvertRole.Arn EndPoint: !GetAtt MediaConvertEndPoint.EndpointUrl @@ -949,38 +985,38 @@ Resources: AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - - - Effect: Allow + - Effect: Allow Principal: Service: - lambda.amazonaws.com Action: - sts:AssumeRole Policies: - - - PolicyName: !Sub "${AWS::StackName}-output-validate-role" + - PolicyName: !Sub "${AWS::StackName}-output-validate-role" PolicyDocument: Statement: - - - Effect: Allow + - Effect: Allow Action: - lambda:InvokeFunction Resource: - - !Sub "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${AWS::StackName}-error-handler" - - - Effect: Allow + - !Sub "arn:${AWS::Partition}:lambda:${AWS::Region}:${AWS::AccountId}:function:${AWS::StackName}-error-handler" + - Effect: Allow Action: - dynamodb:GetItem Resource: - - !Sub "arn:aws:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${DynamoDBTable}" - - - Effect: Allow + - !Sub "arn:${AWS::Partition}:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${DynamoDBTable}" + - Effect: Allow + Action: + - s3:ListBucket + Resource: + - !GetAtt Destination.Arn + - Effect: Allow Action: - logs:CreateLogGroup - logs:CreateLogStream - logs:PutLogEvents Resource: - - !Sub "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/*" + - !Sub "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/*" Metadata: cfn_nag: rules_to_suppress: @@ -990,17 +1026,18 @@ Resources: OutputValidate: Type: AWS::Lambda::Function Properties: - FunctionName: !Sub ${AWS::StackName}-output-validate + FunctionName: !Sub "${AWS::StackName}-output-validate" Description: Parses MediaConvert job output Handler: index.handler Role: !GetAtt OutputValidateRole.Arn Code: S3Bucket: !Join ["-", [!FindInMap ["SourceCode", "General", "S3Bucket"], Ref: "AWS::Region"]] S3Key: !Join ["/", [!FindInMap ["SourceCode", "General", "KeyPrefix"], "output-validate.zip"]] - Runtime: nodejs10.x + Runtime: nodejs12.x Timeout: 120 Environment: Variables: + AWS_NODEJS_CONNECTION_REUSE_ENABLED: "1" DynamoDBTable: !Ref DynamoDBTable ErrorHandler: !GetAtt ErrorHandler.Arn EndPoint: !GetAtt MediaConvertEndPoint.EndpointUrl @@ -1011,38 +1048,33 @@ Resources: AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - - - Effect: Allow + - Effect: Allow Principal: Service: - lambda.amazonaws.com Action: - sts:AssumeRole Policies: - - - PolicyName: !Sub "${AWS::StackName}-archive-source-role" + - PolicyName: !Sub "${AWS::StackName}-archive-source-role" PolicyDocument: Statement: - - - Effect: Allow + - Effect: Allow Action: - s3:PutObjectTagging Resource: - !Sub "${Source.Arn}/*" - - - Effect: Allow + - Effect: Allow Action: - lambda:InvokeFunction Resource: - - !Sub "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${AWS::StackName}-error-handler" - - - Effect: Allow + - !Sub "arn:${AWS::Partition}:lambda:${AWS::Region}:${AWS::AccountId}:function:${AWS::StackName}-error-handler" + - Effect: Allow Action: - logs:CreateLogGroup - logs:CreateLogStream - logs:PutLogEvents Resource: - - !Sub "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/*" + - !Sub "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/*" Metadata: cfn_nag: rules_to_suppress: @@ -1052,20 +1084,21 @@ Resources: ArchiveSource: Type: AWS::Lambda::Function Properties: - FunctionName: !Sub ${AWS::StackName}-archive-source + FunctionName: !Sub "${AWS::StackName}-archive-source" Description: Updates tags on source files to enable Glacier Handler: index.handler Role: !GetAtt ArchiveSourceRole.Arn Code: S3Bucket: !Join ["-", [!FindInMap ["SourceCode", "General", "S3Bucket"], Ref: "AWS::Region"]] S3Key: !Join ["/", [!FindInMap ["SourceCode", "General", "KeyPrefix"], "archive-source.zip"]] - Runtime: nodejs10.x + Runtime: nodejs12.x Timeout: 120 Environment: Variables: + AWS_NODEJS_CONNECTION_REUSE_ENABLED: "1" ErrorHandler: !GetAtt ErrorHandler.Arn - SnsNotificationRole: + SqsSendMessageRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: @@ -1080,7 +1113,7 @@ Resources: - sts:AssumeRole Policies: - - PolicyName: !Sub "${AWS::StackName}-sns-notification-role" + PolicyName: !Sub "${AWS::StackName}-sqs-publish-role" PolicyDocument: Statement: - @@ -1092,9 +1125,9 @@ Resources: - Effect: Allow Action: - - sns:Publish + - sqs:SendMessage Resource: - - !Ref SnsTopic + - !GetAtt SqsQueue.Arn - Effect: Allow Action: @@ -1109,20 +1142,78 @@ Resources: - id: W11 reason: "* is used so that the Lambda function can create log groups" + SqsSendMessage: + Type: AWS::Lambda::Function + Properties: + FunctionName: !Sub ${AWS::StackName}-sqs-publish + Description: Publish the workflow results to an SQS queue + Handler: index.handler + Role: !GetAtt SqsSendMessageRole.Arn + Code: + S3Bucket: !Join ["-", [!FindInMap ["SourceCode", "General", "S3Bucket"], Ref: "AWS::Region"]] + S3Key: !Join ["/", [!FindInMap ["SourceCode", "General", "KeyPrefix"], "sqs-publish.zip"]] + Runtime: nodejs12.x + Timeout: 120 + Environment: + Variables: + AWS_NODEJS_CONNECTION_REUSE_ENABLED: "1" + ErrorHandler: !GetAtt ErrorHandler.Arn + SqsQueue: !Ref SqsQueue + SnsNotificationRole: + Type: AWS::IAM::Role + Properties: + AssumeRolePolicyDocument: + Version: 2012-10-17 + Statement: + - + Effect: Allow + Principal: + Service: + - lambda.amazonaws.com + Action: + - sts:AssumeRole + Policies: + - PolicyName: !Sub "${AWS::StackName}-sns-notification-role" + PolicyDocument: + Statement: + - Effect: Allow + Action: + - lambda:InvokeFunction + Resource: + - !Sub "arn:${AWS::Partition}:lambda:${AWS::Region}:${AWS::AccountId}:function:${AWS::StackName}-error-handler" + - Effect: Allow + Action: + - sns:Publish + Resource: + - !Ref SnsTopic + - Effect: Allow + Action: + - logs:CreateLogGroup + - logs:CreateLogStream + - logs:PutLogEvents + Resource: + - !Sub "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/*" + Metadata: + cfn_nag: + rules_to_suppress: + - id: W11 + reason: "* is used so that the Lambda function can create log groups" + SnsNotification: Type: AWS::Lambda::Function Properties: - FunctionName: !Sub ${AWS::StackName}-sns-notification + FunctionName: !Sub "${AWS::StackName}-sns-notification" Description: Sends a notification when the encode job is completed Handler: index.handler Role: !GetAtt SnsNotificationRole.Arn Code: S3Bucket: !Join ["-", [!FindInMap ["SourceCode", "General", "S3Bucket"], Ref: "AWS::Region"]] S3Key: !Join ["/", [!FindInMap ["SourceCode", "General", "KeyPrefix"], "sns-notification.zip"]] - Runtime: nodejs10.x + Runtime: nodejs12.x Timeout: 120 Environment: Variables: + AWS_NODEJS_CONNECTION_REUSE_ENABLED: "1" ErrorHandler: !GetAtt ErrorHandler.Arn SnsTopic: !Ref SnsTopic @@ -1132,50 +1223,37 @@ Resources: AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - - - Effect: Allow + - Effect: Allow Principal: Service: - lambda.amazonaws.com Action: - sts:AssumeRole Policies: - - - PolicyName: !Sub "${AWS::StackName}-media-package-assets-role" + - PolicyName: !Sub "${AWS::StackName}-media-package-assets-role" PolicyDocument: Statement: - - - Effect: Allow + - Effect: Allow Action: - lambda:InvokeFunction Resource: - - !Sub "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${AWS::StackName}-error-handler" - - - Effect: Allow + - !Sub "arn:${AWS::Partition}:lambda:${AWS::Region}:${AWS::AccountId}:function:${AWS::StackName}-error-handler" + - Effect: Allow Action: - iam:PassRole Resource: - !GetAtt MediaPackageVodRole.Arn - - - Effect: Allow + - Effect: Allow Action: - logs:CreateLogGroup - logs:CreateLogStream - logs:PutLogEvents Resource: - - !Sub "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/*" - - - Effect: Allow + - !Sub "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/*" + - Effect: Allow Action: - mediapackage-vod:CreateAsset Resource: "*" - - - Effect: Allow - Action: - - cloudfront:GetDistributionConfig - - cloudfront:UpdateDistribution - Resource: - - !Sub "arn:aws:cloudfront::${AWS::AccountId}:distribution/${CloudFront}" Metadata: cfn_nag: rules_to_suppress: @@ -1185,20 +1263,21 @@ Resources: MediaPackageAssets: Type: AWS::Lambda::Function Properties: - FunctionName: !Sub ${AWS::StackName}-media-package-assets + FunctionName: !Sub "${AWS::StackName}-media-package-assets" Description: Ingests an asset into MediaPackage-VOD Handler: index.handler Role: !GetAtt MediaPackageAssetsRole.Arn Code: S3Bucket: !Join ["-", [!FindInMap ["SourceCode", "General", "S3Bucket"], Ref: "AWS::Region"]] S3Key: !Join ["/", [!FindInMap ["SourceCode", "General", "KeyPrefix"], "media-package-assets.zip"]] - Runtime: nodejs10.x + Runtime: nodejs12.x Timeout: 300 Environment: Variables: - DistributionId: !Ref CloudFront + AWS_NODEJS_CONNECTION_REUSE_ENABLED: "1" ErrorHandler: !GetAtt ErrorHandler.Arn GroupId: !GetAtt MediaPackageVod.GroupId + GroupDomainName: !GetAtt MediaPackageVod.GroupDomainName MediaPackageVodRole: !GetAtt MediaPackageVodRole.Arn ErrorHandlerRole: @@ -1207,38 +1286,33 @@ Resources: AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - - - Effect: Allow + - Effect: Allow Principal: Service: - lambda.amazonaws.com Action: - sts:AssumeRole Policies: - - - PolicyName: !Sub "${AWS::StackName}-error-handler-role" + - PolicyName: !Sub "${AWS::StackName}-error-handler-role" PolicyDocument: Statement: - - - Effect: Allow + - Effect: Allow Action: - sns:Publish Resource: - !Ref SnsTopic - - - Effect: Allow + - Effect: Allow Action: - dynamodb:UpdateItem Resource: - - !Sub "arn:aws:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${DynamoDBTable}" - - - Effect: Allow + - !Sub "arn:${AWS::Partition}:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${DynamoDBTable}" + - Effect: Allow Action: - logs:CreateLogGroup - logs:CreateLogStream - logs:PutLogEvents Resource: - - !Sub "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/*" + - !Sub "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/*" Metadata: cfn_nag: rules_to_suppress: @@ -1255,17 +1329,18 @@ Resources: Code: S3Bucket: !Join ["-", [!FindInMap ["SourceCode", "General", "S3Bucket"], Ref: "AWS::Region"]] S3Key: !Join ["/", [!FindInMap ["SourceCode", "General", "KeyPrefix"], "error-handler.zip"]] - Runtime: nodejs10.x + Runtime: nodejs12.x Timeout: 120 Environment: Variables: + AWS_NODEJS_CONNECTION_REUSE_ENABLED: "1" DynamoDBTable: !Ref DynamoDBTable SnsTopic: !Ref SnsTopic IngestWorkflow: Type: AWS::StepFunctions::StateMachine Properties: - StateMachineName: !Sub ${AWS::StackName}-ingest + StateMachineName: !Sub "${AWS::StackName}-ingest" DefinitionString: !Sub | { "StartAt": "Input Validate", @@ -1283,7 +1358,18 @@ Resources: "DynamoDB Update": { "Type": "Task", "Resource": "${DynamodbUpdate.Arn}", - "Next": "SNS Notification" + "Next": "SNS Choice" + }, + "SNS Choice": { + "Type": "Choice", + "Choices": [ + { + "Variable": "$.enableSns", + "BooleanEquals": true, + "Next": "SNS Notification" + } + ], + "Default": "Process Execute" }, "SNS Notification": { "Type": "Task", @@ -1302,7 +1388,7 @@ Resources: ProcessWorkflow: Type: AWS::StepFunctions::StateMachine Properties: - StateMachineName: !Sub ${AWS::StackName}-process + StateMachineName: !Sub "${AWS::StackName}-process" DefinitionString: !Sub | { "Comment": "Process StateMachine to create MediaConvert Encoding Jobs", @@ -1311,6 +1397,102 @@ Resources: "Profiler": { "Type": "Task", "Resource": "${Profiler.Arn}", + "Next": "Encoding Profile Check" + }, + "Encoding Profile Check": { + "Type": "Choice", + "Choices": [ + { + "Variable": "$.isCustomTemplate", + "BooleanEquals": true, + "Next": "Custom jobTemplate" + }, + { + "Variable": "$.encodingProfile ", + "NumericEquals": 2160, + "Next": "jobTemplate 2160p" + }, + { + "Variable": "$.encodingProfile ", + "NumericEquals": 1080, + "Next": "jobTemplate 1080p" + }, + { + "Variable": "$.encodingProfile ", + "NumericEquals": 720, + "Next": "jobTemplate 720p" + } + ] + }, + "jobTemplate 2160p": { + "Type": "Pass", + "Next": "Accelerated Transcoding Check" + }, + "jobTemplate 1080p": { + "Type": "Pass", + "Next": "Accelerated Transcoding Check" + }, + "jobTemplate 720p": { + "Type": "Pass", + "Next": "Accelerated Transcoding Check" + }, + "Custom jobTemplate": { + "Type": "Pass", + "Next": "Accelerated Transcoding Check" + }, + "Accelerated Transcoding Check": { + "Type": "Choice", + "Choices": [ + { + "Variable": "$.acceleratedTranscoding", + "StringEquals": "ENABLED", + "Next": "Enabled" + }, + { + "Variable": "$.acceleratedTranscoding", + "StringEquals": "PREFERRED", + "Next": "Preferred" + }, + { + "Variable": "$.acceleratedTranscoding", + "StringEquals": "DISABLED", + "Next": "Disabled" + } + ] + }, + "Enabled": { + "Type": "Pass", + "Next": "Frame Capture Check" + }, + "Preferred": { + "Type": "Pass", + "Next": "Frame Capture Check" + }, + "Disabled": { + "Type": "Pass", + "Next": "Frame Capture Check" + }, + "Frame Capture Check": { + "Type": "Choice", + "Choices": [ + { + "Variable": "$.frameCapture", + "BooleanEquals": true, + "Next": "Frame Capture" + }, + { + "Variable": "$.frameCapture", + "BooleanEquals": false, + "Next": "No Frame Capture" + } + ] + }, + "Frame Capture": { + "Type": "Pass", + "Next": "Encode Job Submit" + }, + "No Frame Capture": { + "Type": "Pass", "Next": "Encode Job Submit" }, "Encode Job Submit": { @@ -1330,7 +1512,7 @@ Resources: PublishWorkflow: Type: AWS::StepFunctions::StateMachine Properties: - StateMachineName: !Sub ${AWS::StackName}-publish + StateMachineName: !Sub "${AWS::StackName}-publish" DefinitionString: !Sub | { "StartAt": "Validate Encoding Outputs", @@ -1345,13 +1527,23 @@ Resources: "Choices": [ { "Variable": "$.archiveSource", - "BooleanEquals": true, - "Next": "Archive Source" + "StringEquals": "GLACIER", + "Next": "Archive" + }, + { + "Variable": "$.archiveSource", + "StringEquals": "DEEP_ARCHIVE", + "Next": "Deep Archive" } ], "Default": "MediaPackage Choice" }, - "Archive Source": { + "Archive": { + "Type": "Task", + "Resource": "${ArchiveSource.Arn}", + "Next": "MediaPackage Choice" + }, + "Deep Archive": { "Type": "Task", "Resource": "${ArchiveSource.Arn}", "Next": "MediaPackage Choice" @@ -1375,11 +1567,42 @@ Resources: "DynamoDB Update": { "Type": "Task", "Resource": "${DynamodbUpdate.Arn}", - "Next": "SNS Notification" + "Next": "SQS Choice" + }, + "SQS Choice": { + "Type": "Choice", + "Choices": [ + { + "Variable": "$.enableSqs", + "BooleanEquals": true, + "Next": "SQS Send Message" + } + ], + "Default": "SNS Choice" + }, + "SQS Send Message": { + "Type": "Task", + "Resource": "${SqsSendMessage.Arn}", + "Next": "SNS Choice" + }, + "SNS Choice": { + "Type": "Choice", + "Choices": [ + { + "Variable": "$.enableSns", + "BooleanEquals": true, + "Next": "SNS Notification" + } + ], + "Default": "Complete" }, "SNS Notification": { "Type": "Task", "Resource": "${SnsNotification.Arn}", + "Next": "Complete" + }, + "Complete": { + "Type": "Pass", "End": true } } @@ -1391,24 +1614,25 @@ Resources: Type: Custom::UUID Properties: ServiceToken: !GetAtt CustomResource.Arn - Resource: "UUID" + Resource: UUID AnonymousMetric: Condition: Metrics Type: Custom::LoadLambda Properties: ServiceToken: !GetAtt CustomResource.Arn - SolutionId: "SO0021" + SolutionId: SO0021 UUID: !GetAtt Uuid.UUID Version: "%%VERSION%%" Transcoder: MediaConvert WorkflowTrigger: !Ref WorkflowTrigger Glacier: !Ref Glacier FrameCapture: !Ref FrameCapture - Resource: "AnonymousMetric" + Resource: AnonymousMetric EnableMediaPackage: !Ref EnableMediaPackage Outputs: + DynamoDBTable: Description: DynamoDB Table Value: !Ref DynamoDBTable @@ -1439,3 +1663,20 @@ Outputs: Value: !GetAtt Uuid.UUID Export: Name: !Join [":", [!Ref "AWS::StackName", UUID]] + + SnsTopic: + Description: SNS Notification Topic + Value: !Ref SnsTopic + Export: + Name: !Join [":", [!Ref "AWS::StackName", SnsTopic]] + + SqsURL: + Description: AmazonSQS Queue URL + Value: !Ref SqsQueue + Export: + Name: !Join [":", [!Ref "AWS::StackName", SqsQueue]] + + SqsARN: + Description: AmazonSQS Queue ARN + Value: + !Sub ${SqsQueue.Arn} \ No newline at end of file diff --git a/source/archive-source/index.js b/source/archive-source/index.js index 3aad3664..edb4f78e 100644 --- a/source/archive-source/index.js +++ b/source/archive-source/index.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * @@ -26,12 +26,12 @@ exports.handler = async (event) => { Tagging: { TagSet: [ { - Key: "guid", + Key: 'guid', Value: event.guid }, { Key: process.env.AWS_LAMBDA_FUNCTION_NAME.slice(0, -15), - Value: 'archive' + Value: event.archiveSource } ] } diff --git a/source/archive-source/lib/error.js b/source/archive-source/lib/error.js index 1c5c2ac6..53b6be3f 100644 --- a/source/archive-source/lib/error.js +++ b/source/archive-source/lib/error.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * @@ -20,10 +20,10 @@ let errHandler = async (event, _err) => { try { let payload = { - "guid": event.guid, - "event": event, - "function": process.env.AWS_LAMBDA_FUNCTION_NAME, - "error": _err.toString() + 'guid': event.guid, + 'event': event, + 'function': process.env.AWS_LAMBDA_FUNCTION_NAME, + 'error': _err.toString() }; let params = { diff --git a/source/archive-source/lib/index.spec.js b/source/archive-source/lib/index.spec.js index 543fc1c2..7d257d35 100644 --- a/source/archive-source/lib/index.spec.js +++ b/source/archive-source/lib/index.spec.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * @@ -11,31 +11,31 @@ * and limitations under the License. * *********************************************************************************************************************/ -let expect = require('chai').expect; -var path = require('path'); -let AWS = require('aws-sdk-mock'); +const expect = require('chai').expect; +const path = require('path'); +const AWS = require('aws-sdk-mock'); AWS.setSDK(path.resolve('./node_modules/aws-sdk')); -let lambda = require('../index.js'); +const lambda = require('../index.js'); describe('#SOURCE ARCHIVE::', () => { - let _event = { - guid: "1234", - srcVideo: "example.mpg", - srcBucket: "bucket" + const _event = { + guid: '1234', + srcVideo: 'example.mpg', + srcBucket: 'bucket', + archiveSource: 'GLACIER' + }; process.env.ErrorHandler = 'error_handler'; process.env.AWS_LAMBDA_FUNCTION_NAME = 'Lambda'; - afterEach(() => { - AWS.restore('S3'); - }); + afterEach(() => AWS.restore('S3')); it('should return "SUCCESS" when s3 tag object returns success', async () => { AWS.mock('S3', 'putObjectTagging', Promise.resolve()); - let response = await lambda.handler(_event); + const response = await lambda.handler(_event); expect(response.guid).to.equal('1234'); }); diff --git a/source/archive-source/package.json b/source/archive-source/package.json index 8b1b54e9..b395cac7 100644 --- a/source/archive-source/package.json +++ b/source/archive-source/package.json @@ -2,21 +2,20 @@ "name": "vod-archive-source", "version": "1.0.0", "engines": { - "node": ">=8" + "node": ">=12" }, "description": "ingest workflow for video on demand", "main": "index.js", "scripts": { - "pretest": "npm install --quiet", + "pretest": "npm i --quiet", "test": "mocha lib/*.spec.js" }, - "dependencies": { - "aws-sdk": "^2.574.0" - }, + "dependencies": {}, "devDependencies": { - "aws-sdk-mock": "^4.5.0", + "aws-sdk": "^2.609.0", + "aws-sdk-mock": "^5.0.0", "chai": "^4.2.0", - "mocha": "^6.2.2" + "mocha": "^7.0.0" }, "private": true, "author": { diff --git a/source/custom-resource/index.js b/source/custom-resource/index.js index 84fd3829..5189eb7a 100644 --- a/source/custom-resource/index.js +++ b/source/custom-resource/index.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * @@ -41,17 +41,15 @@ exports.handler = async (event, context) => { await MediaConvert.createTemplates(config); break; - case ('UUID'): - responseData = { - UUID: uuidv4() - }; + case 'UUID': + responseData = { UUID: uuidv4() }; break; - case ('AnonymousMetric'): + case 'AnonymousMetric': await Metrics.send(config); break; - case ('MediaPackageVod'): + case 'MediaPackageVod': responseData = await MediaPackage.create(config); break; @@ -73,14 +71,16 @@ exports.handler = async (event, context) => { await MediaConvert.updateTemplates(config); break; + case 'MediaPackageVod': + responseData = await MediaPackage.update(config); + break; + default: console.log(config.Resource, ': update not supported, sending success response'); } } if (event.RequestType === 'Delete') { switch (config.Resource) { - // Feature/so-vod-173 limit on the number of custom presets per region, - // deleting on a stack delete case 'MediaConvertTemplates': await MediaConvert.deleteTemplates(config); break; @@ -94,7 +94,7 @@ exports.handler = async (event, context) => { } } - let response = await cfn.send(event, context, 'SUCCESS', responseData); + const response = await cfn.send(event, context, 'SUCCESS', responseData); console.log(`RESPONSE:: ${JSON.stringify(responseData, null, 2)}`); console.log(`CFN STATUS:: ${response}`); } catch (err) { diff --git a/source/custom-resource/lib/cfn/index.js b/source/custom-resource/lib/cfn/index.js index 305f10df..b2c9927c 100644 --- a/source/custom-resource/lib/cfn/index.js +++ b/source/custom-resource/lib/cfn/index.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * @@ -16,32 +16,28 @@ const axios = require('axios'); let sendResponse = async (event, context, responseStatus, responseData) => { let data; - try { - let responseBody = JSON.stringify({ - Status: responseStatus, - Reason: "See the details in CloudWatch Log Stream: " + context.logStreamName, - PhysicalResourceId: event.LogicalResourceId, - StackId: event.StackId, - RequestId: event.RequestId, - LogicalResourceId: event.LogicalResourceId, - Data: responseData - }); + const responseBody = JSON.stringify({ + Status: responseStatus, + Reason: 'See the details in CloudWatch Log Stream: ' + context.logStreamName, + PhysicalResourceId: event.LogicalResourceId, + StackId: event.StackId, + RequestId: event.RequestId, + LogicalResourceId: event.LogicalResourceId, + Data: responseData + }); - let params = { - url: event.ResponseURL, - port: 443, - method: "put", - headers: { - "content-type": "", - "content-length": responseBody.length - }, - data: responseBody - }; + const params = { + url: event.ResponseURL, + port: 443, + method: 'put', + headers: { + 'content-type': '', + 'content-length': responseBody.length + }, + data: responseBody + }; - data = await axios(params); - } catch (err) { - throw err; - } + data = await axios(params); return data.status; }; diff --git a/source/custom-resource/lib/cfn/index.spec.js b/source/custom-resource/lib/cfn/index.spec.js index 438ad693..04473a23 100644 --- a/source/custom-resource/lib/cfn/index.spec.js +++ b/source/custom-resource/lib/cfn/index.spec.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * @@ -15,33 +15,33 @@ const axios = require('axios'); const expect = require('chai').expect; const MockAdapter = require('axios-mock-adapter'); -let lambda = require('./index.js'); -let _event = { - RequestType: "Create", - ServiceToken: "arn:aws:lambda", - ResponseURL: "https://cloudformation", - StackId: "arn:aws:cloudformation", - RequestId: "63e8ffa2-3059-4607-a450-119d473c73bc", - LogicalResourceId: "Uuid", - ResourceType: "Custom::UUID", +const lambda = require('./index.js'); +const _event = { + RequestType: 'Create', + ServiceToken: 'arn:aws:lambda', + ResponseURL: 'https://cloudformation', + StackId: 'arn:aws:cloudformation', + RequestId: '63e8ffa2-3059-4607-a450-119d473c73bc', + LogicalResourceId: 'Uuid', + ResourceType: 'Custom::UUID', ResourceProperties: { - ServiceToken: "arn:aws:lambda", - Resource: "abc" + ServiceToken: 'arn:aws:lambda', + Resource: 'abc' } }; -let _context = { +const _context = { logStreamName: 'cloudwatch' }; -let _responseStatus = 'ok'; -let _responseData = { +const _responseStatus = 'ok'; +const _responseData = { test: 'testing' }; describe('#CFN RESONSE::', () => { it('should return "200" on a send cfn response sucess', async () => { - let mock = new MockAdapter(axios); + const mock = new MockAdapter(axios); mock.onPut().reply(200, {}); lambda.send(_event, _context, _responseStatus, _responseData, (err, res) => { @@ -50,11 +50,11 @@ describe('#CFN RESONSE::', () => { }); it('should return "Network Error" on connection timedout', async () => { - let mock = new MockAdapter(axios); + const mock = new MockAdapter(axios); mock.onPut().networkError(); await lambda.send(_event, _context, _responseStatus, _responseData).catch(err => { - expect(err.toString()).to.equal("Error: Network Error"); + expect(err.toString()).to.equal('Error: Network Error'); }); }); }); diff --git a/source/custom-resource/lib/mediaconvert/index.js b/source/custom-resource/lib/mediaconvert/index.js index 8f3520be..da85fc31 100644 --- a/source/custom-resource/lib/mediaconvert/index.js +++ b/source/custom-resource/lib/mediaconvert/index.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * @@ -17,7 +17,6 @@ const AWS = require('aws-sdk'); const CATEGORY = 'VOD'; const DESCRIPTION = 'video on demand on aws'; -// Feature/so-vod-173 QVBR versions of the default system presets. const qvbrPresets = [ { name: '_Mp4_Avc_Aac_16x9_1280x720p_24Hz_4.5Mbps_qvbr', @@ -130,16 +129,11 @@ const mediaPackageTemplates = [ // Get the Account regional MediaConvert endpoint for making API calls const GetEndpoints = async () => { const mediaconvert = new AWS.MediaConvert(); + const data = await mediaconvert.describeEndpoints().promise(); - try { - let data = await mediaconvert.describeEndpoints().promise(); - - return { - EndpointUrl: data.Endpoints[0].Url - }; - } catch (err) { - throw err; - } + return { + EndpointUrl: data.Endpoints[0].Url + }; }; const _createPresets = async (instance, presets, stackName) => { @@ -187,66 +181,56 @@ const Create = async (config) => { let presets = []; let templates = []; - try { - if (config.EnableMediaPackage === 'true') { - // Use qvbr presets but Media Package templates - presets = qvbrPresets; - templates = mediaPackageTemplates; - } else { - // Use qvbr presets and templates - presets = qvbrPresets; - templates = qvbrTemplates; - } - - await _createPresets(mediaconvert, presets, config.StackName); - await _createTemplates(mediaconvert, templates, config.StackName); - } catch (err) { - throw err; + if (config.EnableMediaPackage === 'true') { + // Use qvbr presets but Media Package templates + presets = qvbrPresets; + templates = mediaPackageTemplates; + } else { + // Use qvbr presets and templates + presets = qvbrPresets; + templates = qvbrTemplates; } + await _createPresets(mediaconvert, presets, config.StackName); + await _createTemplates(mediaconvert, templates, config.StackName); + return 'success'; }; -// Feature/so-vod-176 Support for stack update const Update = async (config) => { const mediaconvert = new AWS.MediaConvert({ endpoint: config.EndPoint, region: process.env.AWS_REGION }); - try { - let enableMediaPackage = 'false'; - - // Check if the curent templates are MediaPackage or not. - let data = await mediaconvert.listJobTemplates({ Category: CATEGORY }).promise(); - data.JobTemplates.forEach(template => { - if (template.Name === config.StackName + '_Ott_720p_Avc_Aac_16x9_mvod') { - enableMediaPackage = 'true'; - } - }); + let enableMediaPackage = 'false'; + + // Check if the curent templates are MediaPackage or not. + let data = await mediaconvert.listJobTemplates({ Category: CATEGORY }).promise(); + data.JobTemplates.forEach(template => { + if (template.Name === config.StackName + '_Ott_720p_Avc_Aac_16x9_mvod') { + enableMediaPackage = 'true'; + } + }); - if (config.EnableMediaPackage != enableMediaPackage) { - if (config.EnableMediaPackage == 'true') { - console.log('Deleting qvbr templates and creating MediaPackage templates'); - await _deleteTemplates(mediaconvert, qvbrTemplates, config.StackName); - await _createTemplates(mediaconvert, mediaPackageTemplates, config.StackName); - } else { - console.log('Deleting MediaPackage templates and creating qvbr templates'); - await _deleteTemplates(mediaconvert, mediaPackageTemplates, config.StackName); - await _createTemplates(mediaconvert, qvbrTemplates, config.StackName); - } + if (config.EnableMediaPackage != enableMediaPackage) { + if (config.EnableMediaPackage == 'true') { + console.log('Deleting qvbr templates and creating MediaPackage templates'); + await _deleteTemplates(mediaconvert, qvbrTemplates, config.StackName); + await _createTemplates(mediaconvert, mediaPackageTemplates, config.StackName); } else { - console.log('No changes to the MediaConvert templates'); + console.log('Deleting MediaPackage templates and creating qvbr templates'); + await _deleteTemplates(mediaconvert, mediaPackageTemplates, config.StackName); + await _createTemplates(mediaconvert, qvbrTemplates, config.StackName); } - } catch (err) { - throw err; + } else { + console.log('No changes to the MediaConvert templates'); } return 'success'; }; const _deletePresets = async (instance, presets, stackName) => { - // Delete custom presets for (let preset of presets) { let name = stackName + preset.name; @@ -256,7 +240,6 @@ const _deletePresets = async (instance, presets, stackName) => { }; const _deleteTemplates = async (instance, templates, stackName) => { - // Delete custom templates for (let tmpl of templates) { let name = stackName + tmpl.name; @@ -265,8 +248,6 @@ const _deleteTemplates = async (instance, templates, stackName) => { } }; -// Feature/so-vod-173 limit on the number of custom presets per region, -// deleting on a stack delete const Delete = async (config) => { const mediaconvert = new AWS.MediaConvert({ endpoint: config.EndPoint, diff --git a/source/custom-resource/lib/mediaconvert/index.spec.js b/source/custom-resource/lib/mediaconvert/index.spec.js index bcc8627f..0703dd71 100644 --- a/source/custom-resource/lib/mediaconvert/index.spec.js +++ b/source/custom-resource/lib/mediaconvert/index.spec.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * @@ -11,7 +11,6 @@ * and limitations under the License. * *********************************************************************************************************************/ -'use strict'; const expect = require('chai').expect; const path = require('path'); const AWS = require('aws-sdk-mock'); @@ -52,7 +51,7 @@ describe('#MEDIACONVERT::', () => { AWS.mock('MediaConvert', 'createPreset', Promise.resolve()); AWS.mock('MediaConvert', 'createJobTemplate', Promise.resolve(data)); - let response = await lambda.createTemplates(_config); + const response = await lambda.createTemplates(_config); expect(response).to.equal('success'); }); @@ -65,7 +64,7 @@ describe('#MEDIACONVERT::', () => { return Promise.resolve(data); }); - let response = await lambda.createTemplates(_mediaPackageConfig); + const response = await lambda.createTemplates(_mediaPackageConfig); expect(response).to.equal('success'); expect(name.endsWith('_mvod')).to.be.true; }); @@ -84,7 +83,7 @@ describe('#MEDIACONVERT::', () => { it('should return "SUCCESS" on describeEndpoints', async () => { AWS.mock('MediaConvert', 'describeEndpoints', Promise.resolve(data)); - let response = await lambda.getEndpoint(_config); + const response = await lambda.getEndpoint(_config); expect(response.EndpointUrl).to.equal('https://test.com'); }); @@ -101,7 +100,7 @@ describe('#MEDIACONVERT::', () => { it('should return "SUCCESS" on update templates', async () => { AWS.mock('MediaConvert', 'listJobTemplates', Promise.resolve(update_data)); - let response = await lambda.updateTemplates(_config); + const response = await lambda.updateTemplates(_config); expect(response).to.equal('success'); }); @@ -185,7 +184,7 @@ describe('#MEDIACONVERT::', () => { AWS.mock('MediaConvert', 'deletePreset', Promise.resolve()); AWS.mock('MediaConvert', 'deleteJobTemplate', Promise.resolve()); - let response = await lambda.deleteTemplates(_config); + const response = await lambda.deleteTemplates(_config); expect(response).to.equal('success'); }); @@ -198,7 +197,7 @@ describe('#MEDIACONVERT::', () => { return Promise.resolve(); }); - let response = await lambda.deleteTemplates(_mediaPackageConfig); + const response = await lambda.deleteTemplates(_mediaPackageConfig); expect(response).to.equal('success'); expect(name.endsWith('_mvod')).to.be.true; }); diff --git a/source/custom-resource/lib/mediapackage/cloudfront.js b/source/custom-resource/lib/mediapackage/cloudfront.js new file mode 100644 index 00000000..ace8f5da --- /dev/null +++ b/source/custom-resource/lib/mediapackage/cloudfront.js @@ -0,0 +1,104 @@ +/********************************************************************************************************************* + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * * + * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * + * with the License. A copy of the License is located at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES * + * OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions * + * and limitations under the License. * + *********************************************************************************************************************/ + +const AWS = require('aws-sdk'); +const originId = 'vodMPOrigin'; + +module.exports.addCustomOrigin = async (distributionId, domainName) => { + if (!distributionId) { + throw new Error('distributionId must be informed'); + } + + if (!domainName) { + throw new Error('domainName must be informed'); + } + + const cloudFront = new AWS.CloudFront(); + const response = await cloudFront.getDistributionConfig({ Id: distributionId }).promise(); + const config = response.DistributionConfig; + + const originExists = config.Origins.Items.some(item => item.Id === originId); + if (originExists) { + console.log(`Origin ${originId} has already been added to distribution ${distributionId}`); + return; + } + + console.log(`Adding MediaPackage as origin to distribution ${distributionId}`); + const url = new URL(domainName); + + const customOrigin = { + Id: originId, + DomainName: url.hostname, + OriginPath: '', + CustomHeaders: { Quantity: 0 }, + CustomOriginConfig: { + HTTPPort: 80, + HTTPSPort: url.port || 443, + OriginProtocolPolicy: 'https-only', + OriginSslProtocols: { Quantity: 1, Items: ['TLSv1.2'] }, + OriginReadTimeout: 30, + OriginKeepaliveTimeout: 5 + } + }; + + config.Origins.Quantity = config.Origins.Items.push(customOrigin); + + const customBehavior = { + PathPattern: 'out/*', + TargetOriginId: originId, + ForwardedValues: { + QueryString: false, + Cookies: { Forward: 'none' }, + Headers: { Quantity: 0 }, + QueryStringCacheKeys: { Quantity: 0 } + }, + TrustedSigners: { Enabled: false, Quantity: 0 }, + ViewerProtocolPolicy: 'redirect-to-https', + MinTTL: 0, + AllowedMethods: { + Quantity: 2, + Items: ['HEAD', 'GET'], + CachedMethods: { Quantity: 2, Items: ['HEAD', 'GET'] } + }, + SmoothStreaming: false, + DefaultTTL: 86400, + MaxTTL: 31536000, + Compress: false, + LambdaFunctionAssociations: { Quantity: 0 }, + FieldLevelEncryptionId: '' + }; + + if (config.CacheBehaviors.Items) { + config.CacheBehaviors.Items.push(customBehavior); + } else { + config.CacheBehaviors.Items = [customBehavior]; + } + + config.CacheBehaviors.Quantity = config.CacheBehaviors.Items.length; + + try { + const params = { + Id: distributionId, + DistributionConfig: config, + IfMatch: response.ETag + }; + + await cloudFront.updateDistribution(params).promise(); + console.log(`Origins:: ${JSON.stringify(config.Origins.Items, null, 2)}`); + console.log(`Cache behaviors:: ${JSON.stringify(config.CacheBehaviors.Items, null, 2)}`); + } catch (error) { + if (error.code !== 'PreconditionFailed') { + throw error; + } + } +}; diff --git a/source/media-package-assets/lib/cloudfront.spec.js b/source/custom-resource/lib/mediapackage/cloudfront.spec.js similarity index 50% rename from source/media-package-assets/lib/cloudfront.spec.js rename to source/custom-resource/lib/mediapackage/cloudfront.spec.js index 988bbb44..62c3fb4d 100644 --- a/source/media-package-assets/lib/cloudfront.spec.js +++ b/source/custom-resource/lib/mediapackage/cloudfront.spec.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * @@ -19,64 +19,13 @@ const AWS = require('aws-sdk-mock'); AWS.setSDK(path.resolve('./node_modules/aws-sdk')); const helper = require('./cloudfront'); - -const _distributionId = 'distribution-id'; -const _domainName = 'bogus.cloudfront.net'; -const _mediaPackageEndpoints = [ - { - PackagingConfigurationId: 'packaging-config-hls', - Url: 'https://test.com/out/index.m3u8' - } -]; - -const _distConfig = { - ETag: 'some-etag', - DistributionConfig: { - CallerReference: 'some-caller-reference', - Origins: { - Quantity: 1, - Items: [{ - Id: 's3Origin', - DomainName: 'some-bucket.s3.us-east-1.amazonaws.com', - OriginPath: '', - CustomHeaders: { - Quantity: 0 - }, - S3OriginConfig: { - OriginAccessIdentity: 'origin-access-identity/cloudfront/some-oai' - } - }] - }, - DefaultCacheBehavior: { - TargetOriginId: 's3Origin', - ForwardedValues: { - QueryString: false, - Cookies: { Forward: 'none' }, - Headers: { - Quantity: 3, - Items: [ - 'Access-Control-Request-Headers', - 'Access-Control-Request-Method', - 'Origin' - ] - }, - QueryStringCacheKeys: { Quantity: 0 } - }, - TrustedSigners: { Enabled: false, Quantity: 0 }, - ViewerProtocolPolicy: 'allow-all', - MinTTL: 0 - }, - CacheBehaviors: { Quantity: 0 }, - Comment: '', - Enabled: true - } -}; +const testAssets = require('./test-assets'); describe('#CLOUDFRONT::', () => { - describe('validation', () => { + describe('Validation', () => { it('should throw an exception when distribution id is undefined', async () => { try { - await helper.convertEndpoints(undefined, _domainName, _mediaPackageEndpoints); + await helper.addCustomOrigin(undefined, testAssets.DomainName); } catch (error) { expect(error).to.not.be.null; return; @@ -87,18 +36,7 @@ describe('#CLOUDFRONT::', () => { it('should throw an exception when domain name is undefined', async () => { try { - await helper.convertEndpoints(_distributionId, undefined, _mediaPackageEndpoints); - } catch (error) { - expect(error).to.not.be.null; - return; - } - - expect.fail('exception should have been thrown'); - }); - - it('should throw an exception when endpoints is empty', async () => { - try { - await helper.convertEndpoints(_distributionId, _domainName, []); + await helper.addCustomOrigin(testAssets.DistributionId, undefined); } catch (error) { expect(error).to.not.be.null; return; @@ -108,30 +46,17 @@ describe('#CLOUDFRONT::', () => { }); }); - describe('api', () => { + describe('Api', () => { afterEach(() => AWS.restore('CloudFront')); - it('should throw an exception when getDistributionConfig fails', async () => { - try { - AWS.mock('CloudFront', 'getDistributionConfig', Promise.reject('some error')); - - await helper.convertEndpoints(_distributionId, _domainName, _mediaPackageEndpoints); - } catch (error) { - expect(error).to.not.be.null; - return; - } - - expect.fail('exception should have been thrown'); - }); - it('should throw an exception when updateDistribution fails with something other than PreconditionFailed', async () => { try { - const config = _.cloneDeep(_distConfig); + const config = _.cloneDeep(testAssets.ConfigurationWithS3); AWS.mock('CloudFront', 'getDistributionConfig', Promise.resolve(config)); AWS.mock('CloudFront', 'updateDistribution', Promise.reject({ code: 'some error' })); - await helper.convertEndpoints(_distributionId, _domainName, _mediaPackageEndpoints); + await helper.addCustomOrigin(testAssets.DistributionId, testAssets.DomainName); } catch (error) { expect(error).to.not.be.null; return; @@ -141,23 +66,38 @@ describe('#CLOUDFRONT::', () => { }); it('should not throw an exception when updateDistribution fails with PreconditionFailed', async () => { - const config = _.cloneDeep(_distConfig); + const config = _.cloneDeep(testAssets.ConfigurationWithS3); AWS.mock('CloudFront', 'getDistributionConfig', Promise.resolve(config)); AWS.mock('CloudFront', 'updateDistribution', Promise.reject({ code: 'PreconditionFailed' })); - const response = await helper.convertEndpoints(_distributionId, _domainName, _mediaPackageEndpoints); - expect(response).not.to.be.undefined; + await helper.addCustomOrigin(testAssets.DistributionId, testAssets.DomainName); + }); + + it('should not add origin if it already exists', async () => { + let callCount = 0; + + const config = _.cloneDeep(testAssets.ConfigurationWithS3); + config.DistributionConfig.Origins.Quantity = 2; + config.DistributionConfig.Origins.Items.push({ Id: 'vodMPOrigin' }); + + AWS.mock('CloudFront', 'getDistributionConfig', Promise.resolve(config)); + AWS.mock('CloudFront', 'updateDistribution', () => { + callCount++; + return Promise.resolve(); + }); + + await helper.addCustomOrigin(testAssets.DistributionId, testAssets.DomainName); + expect(callCount).to.equal(0); }); it('should succeed with valid parameters', async () => { - const config = _.cloneDeep(_distConfig); + const config = _.cloneDeep(testAssets.ConfigurationWithS3); AWS.mock('CloudFront', 'getDistributionConfig', Promise.resolve(config)); AWS.mock('CloudFront', 'updateDistribution', Promise.resolve()); - const response = await helper.convertEndpoints(_distributionId, _domainName, _mediaPackageEndpoints); - expect(response).to.deep.equal({ 'HLS': 'https://bogus.cloudfront.net/out/index.m3u8' }); + await helper.addCustomOrigin(testAssets.DistributionId, testAssets.DomainName); }); }); }); diff --git a/source/custom-resource/lib/mediapackage/index.js b/source/custom-resource/lib/mediapackage/index.js index f731a68a..92946b9f 100644 --- a/source/custom-resource/lib/mediapackage/index.js +++ b/source/custom-resource/lib/mediapackage/index.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * @@ -13,13 +13,14 @@ const AWS = require('aws-sdk'); const crypto = require('crypto'); +const cloudfrontHelper = require('./cloudfront'); const DEFAULT_SEGMENT_LENGTH = 6; const DEFAULT_PROGRAM_DATETIME_INTERVAL = 60; const DEFAULT_MANIFEST_NAME = 'index'; -const getHlsParameters = (groupId, randomId) => ({ - Id: `packaging-config-${randomId}-hls`, +const getHlsParameters = (groupId, configId) => ({ + Id: configId, PackagingGroupId: groupId, HlsPackage: { HlsManifests: [{ @@ -34,8 +35,8 @@ const getHlsParameters = (groupId, randomId) => ({ } }); -const getDashParameters = (groupId, randomId) => ({ - Id: `packaging-config-${randomId}-dash`, +const getDashParameters = (groupId, configId) => ({ + Id: configId, PackagingGroupId: groupId, DashPackage: { DashManifests: [{ @@ -47,8 +48,8 @@ const getDashParameters = (groupId, randomId) => ({ } }); -const getMssParameters = (groupId, randomId) => ({ - Id: `packaging-config-${randomId}-mss`, +const getMssParameters = (groupId, configId) => ({ + Id: configId, PackagingGroupId: groupId, MssPackage: { MssManifests: [{ @@ -58,8 +59,8 @@ const getMssParameters = (groupId, randomId) => ({ } }); -const getCmafParameters = (groupId, randomId) => ({ - Id: `packaging-config-${randomId}-cmaf`, +const getCmafParameters = (groupId, configId) => ({ + Id: configId, PackagingGroupId: groupId, CmafPackage: { HlsManifests: [{ @@ -69,7 +70,7 @@ const getCmafParameters = (groupId, randomId) => ({ ProgramDateTimeIntervalSeconds: DEFAULT_PROGRAM_DATETIME_INTERVAL, RepeatExtXKey: false }], - SegmentDurationSeconds: DEFAULT_SEGMENT_LENGTH, + SegmentDurationSeconds: DEFAULT_SEGMENT_LENGTH } }); @@ -77,30 +78,28 @@ const create = async (properties) => { const mediaPackageVod = new AWS.MediaPackageVod(); const randomId = crypto.randomBytes(8).toString('hex'); - const groupId = properties.GroupId; - await mediaPackageVod.createPackagingGroup({ Id: groupId }).promise(); - + const packagingGroup = await mediaPackageVod.createPackagingGroup({ Id: properties.GroupId }).promise(); let created = false; - let configurations = Array.from(new Set(properties.PackagingConfigurations.split(','))); + const configurations = Array.from(new Set(properties.PackagingConfigurations.split(','))); for (let config of configurations) { let params = {}; switch (config.toLowerCase()) { case 'hls': - params = getHlsParameters(groupId, randomId); + params = getHlsParameters(packagingGroup.Id, `packaging-config-${randomId}-hls`); break; case 'dash': - params = getDashParameters(groupId, randomId); + params = getDashParameters(packagingGroup.Id, `packaging-config-${randomId}-dash`); break; case 'mss': - params = getMssParameters(groupId, randomId); + params = getMssParameters(packagingGroup.Id, `packaging-config-${randomId}-mss`); break; case 'cmaf': - params = getCmafParameters(groupId, randomId); + params = getCmafParameters(packagingGroup.Id, `packaging-config-${randomId}-cmaf`); break; default: @@ -120,22 +119,35 @@ const create = async (properties) => { throw new Error('At least one valid packaging configuration must be informed.'); } + await cloudfrontHelper.addCustomOrigin(properties.DistributionId, packagingGroup.DomainName); + return { - GroupId: groupId + GroupId: packagingGroup.Id, + GroupDomainName: packagingGroup.DomainName + }; +}; + +const update = async (properties) => { + const mediaPackageVod = new AWS.MediaPackageVod(); + const packagingGroup = await mediaPackageVod.describePackagingGroup({ Id: properties.GroupId }).promise(); + + if (properties.EnableMediaPackage == 'true') { + await cloudfrontHelper.addCustomOrigin(properties.DistributionId, packagingGroup.DomainName); + } + + return { + GroupId: packagingGroup.Id, + GroupDomainName: packagingGroup.DomainName }; }; const purge = async (properties) => { const mediaPackageVod = new AWS.MediaPackageVod(); const groupId = properties.GroupId; - let token; do { - const response = await mediaPackageVod.listAssets({ - PackagingGroupId: groupId, - NextToken: token - }).promise(); + const response = await mediaPackageVod.listAssets({ PackagingGroupId: groupId, NextToken: token }).promise(); for (let asset of response.Assets) { await mediaPackageVod.deleteAsset({ Id: asset.Id }).promise(); @@ -146,10 +158,7 @@ const purge = async (properties) => { } while (token); do { - const response = await mediaPackageVod.listPackagingConfigurations({ - PackagingGroupId: groupId, - NextToken: token - }).promise(); + const response = await mediaPackageVod.listPackagingConfigurations({ PackagingGroupId: groupId, NextToken: token }).promise(); for (let config of response.PackagingConfigurations) { await mediaPackageVod.deletePackagingConfiguration({ Id: config.Id }).promise(); @@ -175,5 +184,6 @@ const purge = async (properties) => { module.exports = { create, + update, purge -}; \ No newline at end of file +}; diff --git a/source/custom-resource/lib/mediapackage/index.spec.js b/source/custom-resource/lib/mediapackage/index.spec.js index 5a8cc24c..52e05f26 100644 --- a/source/custom-resource/lib/mediapackage/index.spec.js +++ b/source/custom-resource/lib/mediapackage/index.spec.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * @@ -11,24 +11,26 @@ * and limitations under the License. * *********************************************************************************************************************/ -'use strict'; const expect = require('chai').expect; const path = require('path'); const AWS = require('aws-sdk-mock'); AWS.setSDK(path.resolve('./node_modules/aws-sdk')); const lambda = require('./index.js'); +const testAssets = require('./test-assets'); -const _stackName = 'test'; +const _stackName = 'test-stack'; const _groupId = 'test-packaging-group'; const validParameters = { StackName: _stackName, GroupId: _groupId, - PackagingConfigurations: 'HLS,DASH' + PackagingConfigurations: 'HLS,DASH', + DistributionId: testAssets.DistributionId, + EnableMediaPackage: 'true' }; -const assetsResponse = { +const listAssetsResponse = { Assets: [{ Arn: 'asset-arn', Id: 'asset-id', @@ -39,7 +41,7 @@ const assetsResponse = { }] }; -const configurationsResponse = { +const listConfigurationsResponse = { PackagingConfigurations: [{ Arn: 'config-arn', Id: 'MSS', @@ -53,13 +55,22 @@ const configurationsResponse = { }] }; +const groupResponse = { + Id: _groupId, + DomainName: testAssets.DomainName +}; + describe('#MEDIAPACKAGE-VOD::', () => { - afterEach(() => AWS.restore('MediaPackageVod')); + afterEach(() => { + AWS.restore('MediaPackageVod'); + AWS.restore('CloudFront'); + }); describe('Create', () => { it('should succeed with valid parameters', async () => { - AWS.mock('MediaPackageVod', 'createPackagingGroup', Promise.resolve()); + AWS.mock('MediaPackageVod', 'createPackagingGroup', Promise.resolve(groupResponse)); AWS.mock('MediaPackageVod', 'createPackagingConfiguration', Promise.resolve()); + AWS.mock('CloudFront', 'getDistributionConfig', Promise.resolve(testAssets.ConfigurationWithMP)); const response = await lambda.create(validParameters); expect(response.GroupId).to.equal(_groupId); @@ -68,16 +79,18 @@ describe('#MEDIAPACKAGE-VOD::', () => { it('should ignore duplicate configurations', async () => { let callCount = 0; - AWS.mock('MediaPackageVod', 'createPackagingGroup', Promise.resolve()); + AWS.mock('MediaPackageVod', 'createPackagingGroup', Promise.resolve(groupResponse)); AWS.mock('MediaPackageVod', 'createPackagingConfiguration', () => { callCount++; return Promise.resolve(); }); + AWS.mock('CloudFront', 'getDistributionConfig', Promise.resolve(testAssets.ConfigurationWithMP)); const duplicateConfigParams = { StackName: _stackName, GroupId: _groupId, - PackagingConfigurations: 'HLS,DASH,HLS' + PackagingConfigurations: 'HLS,DASH,HLS', + DistributionId: testAssets.DistributionId }; const response = await lambda.create(duplicateConfigParams); @@ -86,13 +99,15 @@ describe('#MEDIAPACKAGE-VOD::', () => { }); it('should succeed when at least one valid configuration is informed', async () => { - AWS.mock('MediaPackageVod', 'createPackagingGroup', Promise.resolve()); + AWS.mock('MediaPackageVod', 'createPackagingGroup', Promise.resolve(groupResponse)); AWS.mock('MediaPackageVod', 'createPackagingConfiguration', Promise.resolve()); + AWS.mock('CloudFront', 'getDistributionConfig', Promise.resolve(testAssets.ConfigurationWithMP)); const singleInvalidParams = { StackName: _stackName, GroupId: _groupId, - PackagingConfigurations: 'HLS,bogus' + PackagingConfigurations: 'HLS,bogus', + DistributionId: testAssets.DistributionId }; const response = await lambda.create(singleInvalidParams); @@ -114,7 +129,7 @@ describe('#MEDIAPACKAGE-VOD::', () => { }); it('should fail when no valid configurations are informed', async () => { - AWS.mock('MediaPackageVod', 'createPackagingGroup', Promise.resolve()); + AWS.mock('MediaPackageVod', 'createPackagingGroup', Promise.resolve(groupResponse)); const invalidParameters = { StackName: _stackName, @@ -133,7 +148,7 @@ describe('#MEDIAPACKAGE-VOD::', () => { }); it('should fail when createPackagingConfiguration throws an exception', async () => { - AWS.mock('MediaPackageVod', 'createPackagingGroup', Promise.resolve()); + AWS.mock('MediaPackageVod', 'createPackagingGroup', Promise.resolve(groupResponse)); AWS.mock('MediaPackageVod', 'createPackagingConfiguration', Promise.reject('some error')); try { @@ -146,12 +161,66 @@ describe('#MEDIAPACKAGE-VOD::', () => { expect.fail('exception should have been thrown'); }); + + it('should fail when getDistributionConfig throws an exception', async () => { + AWS.mock('MediaPackageVod', 'createPackagingGroup', Promise.resolve(groupResponse)); + AWS.mock('MediaPackageVod', 'createPackagingConfiguration', Promise.resolve()); + AWS.mock('CloudFront', 'getDistributionConfig', Promise.reject('some error')); + + try { + await lambda.create(validParameters); + } catch (error) { + expect(error).to.not.be.null; + expect(error).to.equal('some error'); + return; + } + + expect.fail('exception should have been thrown'); + }); + }); + + describe('Update', () => { + it('should not try to add the origin if mediapackage is not enabled', async () => { + let callCount = 0; + + AWS.mock('MediaPackageVod', 'describePackagingGroup', Promise.resolve(groupResponse)); + AWS.mock('CloudFront', 'getDistributionConfig', Promise.resolve(testAssets.ConfigurationWithS3)); + AWS.mock('CloudFront', 'updateDistribution', () => { + callCount++; + return Promise.resolve(); + }); + + const disabledMediaPackageParams = { + StackName: _stackName, + GroupId: _groupId, + PackagingConfigurations: 'HLS,DASH', + DistributionId: testAssets.DistributionId, + EnableMediaPackage: 'false' + }; + + await lambda.update(disabledMediaPackageParams); + expect(callCount).to.equal(0); + }); + + it('should add the origin if it does not exist', async () => { + let callCount = 0; + + AWS.mock('MediaPackageVod', 'describePackagingGroup', Promise.resolve(groupResponse)); + AWS.mock('CloudFront', 'getDistributionConfig', Promise.resolve(testAssets.ConfigurationWithS3)); + AWS.mock('CloudFront', 'updateDistribution', () => { + callCount++; + return Promise.resolve(); + }); + + await lambda.update(validParameters); + expect(callCount).to.equal(1); + }); }); describe('Delete', () => { it('should succeed when group exists', async () => { - AWS.mock('MediaPackageVod', 'listAssets', Promise.resolve(assetsResponse)); - AWS.mock('MediaPackageVod', 'listPackagingConfigurations', Promise.resolve(configurationsResponse)); + AWS.mock('MediaPackageVod', 'listAssets', Promise.resolve(listAssetsResponse)); + AWS.mock('MediaPackageVod', 'listPackagingConfigurations', Promise.resolve(listConfigurationsResponse)); AWS.mock('MediaPackageVod', 'deleteAsset', Promise.resolve()); AWS.mock('MediaPackageVod', 'deletePackagingConfiguration', Promise.resolve()); @@ -179,8 +248,8 @@ describe('#MEDIAPACKAGE-VOD::', () => { }); it('should fail when exception is unknown', async () => { - AWS.mock('MediaPackageVod', 'listAssets', Promise.resolve(assetsResponse)); - AWS.mock('MediaPackageVod', 'listPackagingConfigurations', Promise.resolve(configurationsResponse)); + AWS.mock('MediaPackageVod', 'listAssets', Promise.resolve(listAssetsResponse)); + AWS.mock('MediaPackageVod', 'listPackagingConfigurations', Promise.resolve(listConfigurationsResponse)); AWS.mock('MediaPackageVod', 'deleteAsset', Promise.resolve()); AWS.mock('MediaPackageVod', 'deletePackagingConfiguration', Promise.resolve()); diff --git a/source/custom-resource/lib/mediapackage/test-assets.js b/source/custom-resource/lib/mediapackage/test-assets.js new file mode 100644 index 00000000..2f5e15b8 --- /dev/null +++ b/source/custom-resource/lib/mediapackage/test-assets.js @@ -0,0 +1,58 @@ +const ConfigurationWithS3 = { + ETag: 'some-etag', + DistributionConfig: { + CallerReference: 'some-caller-reference', + Origins: { + Quantity: 1, + Items: [{ + Id: 's3Origin', + DomainName: 'some-bucket.s3.us-east-1.amazonaws.com', + OriginPath: '', + CustomHeaders: { + Quantity: 0 + }, + S3OriginConfig: { + OriginAccessIdentity: 'origin-access-identity/cloudfront/some-oai' + } + }] + }, + DefaultCacheBehavior: { + TargetOriginId: 's3Origin', + ForwardedValues: { + QueryString: false, + Cookies: { Forward: 'none' }, + Headers: { + Quantity: 3, + Items: [ + 'Access-Control-Request-Headers', + 'Access-Control-Request-Method', + 'Origin' + ] + }, + QueryStringCacheKeys: { Quantity: 0 } + }, + TrustedSigners: { Enabled: false, Quantity: 0 }, + ViewerProtocolPolicy: 'allow-all', + MinTTL: 0 + }, + CacheBehaviors: { Quantity: 0 }, + Comment: '', + Enabled: true + } +}; + +const ConfigurationWithMP = { + DistributionConfig: { + Origins: { + Items: [{ Id: 'vodMPOrigin' }], + Quantity: 1 + } + } +}; + +module.exports = { + ConfigurationWithS3, + ConfigurationWithMP, + DistributionId: 'distribution-id', + DomainName: 'https://random-id.egress.mediapackage-vod.us-east-1.amazonaws.com' +}; diff --git a/source/custom-resource/lib/metrics/index.js b/source/custom-resource/lib/metrics/index.js index 7539fc37..ce8e2741 100644 --- a/source/custom-resource/lib/metrics/index.js +++ b/source/custom-resource/lib/metrics/index.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * @@ -25,29 +25,24 @@ const sanitizeData = (config) => { const send = async (config) => { let data; - try { - const metrics = { - Solution: config.SolutionId, - UUID: config.UUID, - TimeStamp: moment().utc().toISOString(), - Data: sanitizeData(config) - }; - - const params = { - method: 'post', - port: 443, - url: 'https://metrics.awssolutionsbuilder.com/generic', - headers: { - 'Content-Type': 'application/json' - }, - data: metrics - }; - - // Send Metrics & retun status code. - data = await axios(params); - } catch (err) { - throw err; - } + const metrics = { + Solution: config.SolutionId, + UUID: config.UUID, + TimeStamp: moment().utc().toISOString(), + Data: sanitizeData(config) + }; + + const params = { + method: 'post', + port: 443, + url: 'https://metrics.awssolutionsbuilder.com/generic', + headers: { + 'Content-Type': 'application/json' + }, + data: metrics + }; + + data = await axios(params); return data.status; }; diff --git a/source/custom-resource/lib/metrics/index.spec.js b/source/custom-resource/lib/metrics/index.spec.js index 50f60552..e8e0e26a 100644 --- a/source/custom-resource/lib/metrics/index.spec.js +++ b/source/custom-resource/lib/metrics/index.spec.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * @@ -15,7 +15,7 @@ const axios = require('axios'); const expect = require('chai').expect; const MockAdapter = require('axios-mock-adapter'); -let lambda = require('./index.js'); +const lambda = require('./index.js'); const _config = { SolutionId: 'SO0021', @@ -26,7 +26,7 @@ const _config = { describe('#SEND METRICS', () => { it('should return "200" on a send metrics sucess', async () => { - let mock = new MockAdapter(axios); + const mock = new MockAdapter(axios); mock.onPost().reply(200, {}); lambda.send(_config, (_err, res) => { @@ -35,11 +35,11 @@ describe('#SEND METRICS', () => { }); it('should return "Network Error" on connection timedout', async () => { - let mock = new MockAdapter(axios); + const mock = new MockAdapter(axios); mock.onPut().networkError(); await lambda.send(_config).catch(err => { - expect(err.toString()).to.equal("Error: Request failed with status code 404"); + expect(err.toString()).to.equal('Error: Request failed with status code 404'); }); }); diff --git a/source/custom-resource/lib/s3/index.js b/source/custom-resource/lib/s3/index.js index 127ebd27..438a39d9 100644 --- a/source/custom-resource/lib/s3/index.js +++ b/source/custom-resource/lib/s3/index.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * @@ -16,15 +16,15 @@ const AWS = require('aws-sdk'); let PutNotification = async (config) => { const s3 = new AWS.S3(); - try { - let params; + let params; - switch (config.WorkflowTrigger) { - case 'VideoFile': - params = { - Bucket: config.Source, - NotificationConfiguration: { - LambdaFunctionConfigurations: [{ + switch (config.WorkflowTrigger) { + case 'VideoFile': + params = { + Bucket: config.Source, + NotificationConfiguration: { + LambdaFunctionConfigurations: [ + { Events: ['s3:ObjectCreated:*'], LambdaFunctionArn: config.IngestArn, Filter: { @@ -84,43 +84,39 @@ let PutNotification = async (config) => { } } } - ] - } - }; + ] + } + }; - console.log('configuring S3 event for Video'); - await s3.putBucketNotificationConfiguration(params).promise(); - break; + console.log(`Configuring S3 event for ${config.WorkflowTrigger}`); + await s3.putBucketNotificationConfiguration(params).promise(); + break; - case 'MetadataFile': - params = { - Bucket: config.Source, - NotificationConfiguration: { - LambdaFunctionConfigurations: [{ - Events: ['s3:ObjectCreated:*'], - LambdaFunctionArn: config.IngestArn, - Filter: { - Key: { - FilterRules: [{ - Name: 'suffix', - Value: 'json' - }] - } + case 'MetadataFile': + params = { + Bucket: config.Source, + NotificationConfiguration: { + LambdaFunctionConfigurations: [{ + Events: ['s3:ObjectCreated:*'], + LambdaFunctionArn: config.IngestArn, + Filter: { + Key: { + FilterRules: [{ + Name: 'suffix', + Value: 'json' + }] } } - ] - } - }; + }] + } + }; - console.log('configuring S3 event for Metadata'); - await s3.putBucketNotificationConfiguration(params).promise(); - break; + console.log(`Configuring S3 event for ${config.WorkflowTrigger}`); + await s3.putBucketNotificationConfiguration(params).promise(); + break; - default: - throw new Error(`Unknown WorkflowTrigger: ${config.WorkflowTrigger}`); - } - } catch (err) { - throw err; + default: + throw new Error(`Unknown WorkflowTrigger: ${config.WorkflowTrigger}`); } return 'success'; diff --git a/source/custom-resource/lib/s3/index.spec.js b/source/custom-resource/lib/s3/index.spec.js index b0418744..c3b4c8dc 100644 --- a/source/custom-resource/lib/s3/index.spec.js +++ b/source/custom-resource/lib/s3/index.spec.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * @@ -11,7 +11,6 @@ * and limitations under the License. * *********************************************************************************************************************/ -'use strict'; const expect = require('chai').expect; const path = require('path'); const AWS = require('aws-sdk-mock'); diff --git a/source/custom-resource/package.json b/source/custom-resource/package.json index cf4bdb8e..74d6f50a 100644 --- a/source/custom-resource/package.json +++ b/source/custom-resource/package.json @@ -2,25 +2,26 @@ "name": "vod-custom-resource", "version": "1.0.0", "engines": { - "node": ">=8" + "node": ">=12" }, "description": "cfn custom resources for video on demand", "main": "index.js", "scripts": { - "pretest": "npm install --quiet", + "pretest": "npm i --quiet", "test": "mocha lib/**/*.spec.js" }, "dependencies": { - "aws-sdk": "^2.574.0", - "axios": "^0.19.0", + "axios": "^0.19.2", "moment": "^2.24.0", - "uuid": "^3.3.3" + "uuid": "^3.4.0" }, "devDependencies": { - "aws-sdk-mock": "^4.5.0", - "mocha": "^6.2.2", + "aws-sdk": "^2.609.0", + "aws-sdk-mock": "^5.0.0", + "mocha": "^7.0.0", "chai": "^4.2.0", - "axios-mock-adapter": "^1.17.0" + "axios-mock-adapter": "^1.17.0", + "lodash": "^4.17.15" }, "private": true, "author": { diff --git a/source/dynamo/index.js b/source/dynamo/index.js index cc55d55f..4bceb296 100644 --- a/source/dynamo/index.js +++ b/source/dynamo/index.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * diff --git a/source/dynamo/lib/error.js b/source/dynamo/lib/error.js index 1c5c2ac6..53b6be3f 100644 --- a/source/dynamo/lib/error.js +++ b/source/dynamo/lib/error.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * @@ -20,10 +20,10 @@ let errHandler = async (event, _err) => { try { let payload = { - "guid": event.guid, - "event": event, - "function": process.env.AWS_LAMBDA_FUNCTION_NAME, - "error": _err.toString() + 'guid': event.guid, + 'event': event, + 'function': process.env.AWS_LAMBDA_FUNCTION_NAME, + 'error': _err.toString() }; let params = { diff --git a/source/dynamo/lib/index.spec.js b/source/dynamo/lib/index.spec.js index d6af2cb5..f156cb77 100644 --- a/source/dynamo/lib/index.spec.js +++ b/source/dynamo/lib/index.spec.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * @@ -20,8 +20,8 @@ const lambda = require('../index.js'); describe('#DYNAMODB UPDATE::', () => { const _event = { - guid: "SUCCESS", - hello: "from AWS mock" + guid: 'SUCCESS', + hello: 'from AWS mock' }; process.env.ErrorHandler = 'error_handler'; @@ -33,7 +33,7 @@ describe('#DYNAMODB UPDATE::', () => { it('should return "SUCCESS" when db put returns success', async () => { AWS.mock('DynamoDB.DocumentClient', 'update', Promise.resolve()); - let response = await lambda.handler(_event); + const response = await lambda.handler(_event); expect(response.guid).to.equal('SUCCESS'); }); diff --git a/source/dynamo/package.json b/source/dynamo/package.json index 1955d0bb..305b443c 100644 --- a/source/dynamo/package.json +++ b/source/dynamo/package.json @@ -2,23 +2,22 @@ "name": "vod-dynamo", "version": "1.0.0", "engines": { - "node": ">=8" + "node": ">=12" }, "description": "dynamo helper for video on demand", "main": "index.js", "scripts": { - "pretest": "npm install --quiet", + "pretest": "npm i --quiet", "test": "mocha lib/*.spec.js" }, - "dependencies": { - "aws-sdk": "^2.574.0" - }, + "dependencies": {}, "devDependencies": { - "aws-sdk-mock": "^4.5.0", + "aws-sdk": "^2.609.0", + "aws-sdk-mock": "^5.0.0", "chai": "^4.2.0", - "mocha": "^6.2.2", - "sinon": "^7.5.0", - "sinon-chai": "^3.3.0" + "mocha": "^7.0.0", + "sinon": "^8.1.1", + "sinon-chai": "^3.4.0" }, "private": true, "author": { diff --git a/source/encode/index.js b/source/encode/index.js index 6b026d07..a2f89a6b 100644 --- a/source/encode/index.js +++ b/source/encode/index.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * @@ -88,7 +88,7 @@ const getFrameGroup = (event, outputPath) => ({ } }, Outputs: [{ - NameModifier: '_tumb', + NameModifier: '_thumb', ContainerSettings: { Container: 'RAW' }, @@ -225,6 +225,14 @@ exports.handler = async (event) => { job.Settings.OutputGroups.push(frameCapture); } + //if enabled the TimeCodeConfig needs to be set to ZEROBASED not passthrough + //https://docs.aws.amazon.com/mediaconvert/latest/ug/job-requirements.html + if (event.acceleratedTranscoding === 'PREFERRED' || event.acceleratedTranscoding === 'ENABLED') { + job.AccelerationSettings = {"Mode" : event.acceleratedTranscoding} + job.Settings.TimecodeConfig = {"Source" : "ZEROBASED"} + job.Settings.Inputs[0].TimecodeSource = "ZEROBASED" + } + let data = await mediaconvert.createJob(job).promise(); event.encodingJob = job; event.encodeJobId = data.Job.Id; diff --git a/source/encode/lib/error.js b/source/encode/lib/error.js index 1c5c2ac6..53b6be3f 100644 --- a/source/encode/lib/error.js +++ b/source/encode/lib/error.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * @@ -20,10 +20,10 @@ let errHandler = async (event, _err) => { try { let payload = { - "guid": event.guid, - "event": event, - "function": process.env.AWS_LAMBDA_FUNCTION_NAME, - "error": _err.toString() + 'guid': event.guid, + 'event': event, + 'function': process.env.AWS_LAMBDA_FUNCTION_NAME, + 'error': _err.toString() }; let params = { diff --git a/source/encode/lib/index.spec.js b/source/encode/lib/index.spec.js index 96ca7499..977a74c5 100644 --- a/source/encode/lib/index.spec.js +++ b/source/encode/lib/index.spec.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * @@ -28,7 +28,8 @@ describe('#ENCODE::', () => { jobTemplate: 'jobTemplate', srcVideo: 'video.mp4', srcBucket: 'src', - destBucket: 'dest' + destBucket: 'dest', + acceleratedTranscoding:'PREFERRED' }; const _withframe = { @@ -37,7 +38,8 @@ describe('#ENCODE::', () => { srcVideo: 'video.mp4', srcBucket: 'src', destBucket: 'dest', - frameCapture: true + frameCapture: true, + acceleratedTranscoding:'ENABLED' }; const data = { @@ -88,7 +90,8 @@ describe('#ENCODE::', () => { srcVideo: 'video.mp4', srcBucket: 'src', destBucket: 'dest', - isCustomTemplate: true + isCustomTemplate: true, + acceleratedTranscoding:'DISABLED' }; const customTemplate = { @@ -132,7 +135,8 @@ describe('#ENCODE::', () => { srcVideo: 'video.mp4', srcBucket: 'src', destBucket: 'dest', - isCustomTemplate: false + isCustomTemplate: false, + acceleratedTranscoding:'DISABLED' }; AWS.mock('MediaConvert', 'getJobTemplate', Promise.resolve(tmpl)); diff --git a/source/encode/package.json b/source/encode/package.json index 9fb664ba..33431952 100644 --- a/source/encode/package.json +++ b/source/encode/package.json @@ -2,24 +2,24 @@ "name": "vod-encode", "version": "1.0.0", "engines": { - "node": ">=8" + "node": ">=12" }, "description": "process workflow for video on demand", "main": "index.js", "scripts": { - "pretest": "npm install --quiet", + "pretest": "npm i --quiet", "test": "mocha lib/*.spec.js" }, "dependencies": { - "aws-sdk": "^2.574.0", "lodash": "^4.17.15" }, "devDependencies": { - "aws-sdk-mock": "^4.5.0", + "aws-sdk": "^2.609.0", + "aws-sdk-mock": "^5.0.0", "chai": "^4.2.0", - "mocha": "^6.2.2", - "sinon": "^7.5.0", - "sinon-chai": "^3.3.0" + "mocha": "^7.0.0", + "sinon": "^8.1.1", + "sinon-chai": "^3.4.0" }, "private": true, "author": { diff --git a/source/error-handler/index.js b/source/error-handler/index.js index 04b91f66..2a79b452 100644 --- a/source/error-handler/index.js +++ b/source/error-handler/index.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * @@ -29,73 +29,69 @@ exports.handler = async (event) => { url, msg; - try { - if (event.function) { - url = 'https://console.aws.amazon.com/cloudwatch/home?region=' + process.env.AWS_REGION + '#logStream:group=/aws/lambda/' + event.function; - guid = event.guid; - values = { - ':st': 'Error', - ':ea': event.function, - ':em': event.error, - ':ed': url - }; + if (event.function) { + url = 'https://console.aws.amazon.com/cloudwatch/home?region=' + process.env.AWS_REGION + '#logStream:group=/aws/lambda/' + event.function; + guid = event.guid; + values = { + ':st': 'Error', + ':ea': event.function, + ':em': event.error, + ':ed': url + }; - // Msg update to match DynamoDB entry - msg = { - guid: guid, - workflowStatus: 'Error', - workflowErrorAt: event.function, - errorMessage: event.error, - errorDetails: url - }; - } + // Msg update to match DynamoDB entry + msg = { + guid: guid, + workflowStatus: 'Error', + workflowErrorAt: event.function, + errorMessage: event.error, + errorDetails: url + }; + } - if (event.detail) { - url = 'https://console.aws.amazon.com/mediaconvert/home?region=' + process.env.AWS_REGION + '#/jobs/summary/' + event.detail.jobId; - guid = event.detail.userMetadata.guid; - values = { - ':st': 'Error', - ':ea': 'Encoding', - ':em': JSON.stringify(event, null, 2), - ':ed': url - }; + if (event.detail) { + url = 'https://console.aws.amazon.com/mediaconvert/home?region=' + process.env.AWS_REGION + '#/jobs/summary/' + event.detail.jobId; + guid = event.detail.userMetadata.guid; + values = { + ':st': 'Error', + ':ea': 'Encoding', + ':em': JSON.stringify(event, null, 2), + ':ed': url + }; - // Msg update to match DynamoDB entry - msg = { - guid: guid, - workflowStatus: 'Error', - workflowErrorAt: 'Encoding', - errorMessage: event.detail.errorMessage, - errorDetails: url, - }; - } + // Msg update to match DynamoDB entry + msg = { + guid: guid, + workflowStatus: 'Error', + workflowErrorAt: 'Encoding', + errorMessage: event.detail.errorMessage, + errorDetails: url, + }; + } - console.log(JSON.stringify(msg, null, 2)); + console.log(JSON.stringify(msg, null, 2)); - // Update DynamoDB - let params = { - TableName: process.env.DynamoDBTable, - Key: { - guid: guid - }, - UpdateExpression: 'SET workflowStatus = :st,' + 'workflowErrorAt = :ea,' + 'errorMessage = :em,' + 'errorDetails = :ed', - ExpressionAttributeValues: values - }; + // Update DynamoDB + let params = { + TableName: process.env.DynamoDBTable, + Key: { + guid: guid + }, + UpdateExpression: 'SET workflowStatus = :st,' + 'workflowErrorAt = :ea,' + 'errorMessage = :em,' + 'errorDetails = :ed', + ExpressionAttributeValues: values + }; - await dynamo.update(params).promise(); + await dynamo.update(params).promise(); - // Feature/so-vod-173 match SNS data structure with the SNS Notification - // Function for consistency. - params = { - Message: JSON.stringify(msg, null, 2), - Subject: ' workflow Status:: Error: ' + guid, - TargetArn: process.env.SnsTopic - }; + // Feature/so-vod-173 match SNS data structure with the SNS Notification + // Function for consistency. + params = { + Message: JSON.stringify(msg, null, 2), + Subject: ' workflow Status:: Error: ' + guid, + TargetArn: process.env.SnsTopic + }; - await sns.publish(params).promise(); - } catch (err) { - throw err; - } + await sns.publish(params).promise(); return event; }; diff --git a/source/error-handler/lib/index.spec.js b/source/error-handler/lib/index.spec.js index 366dc005..0790b3e9 100644 --- a/source/error-handler/lib/index.spec.js +++ b/source/error-handler/lib/index.spec.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * @@ -11,23 +11,23 @@ * and limitations under the License. * *********************************************************************************************************************/ -let expect = require('chai').expect; -var path = require('path'); -let AWS = require('aws-sdk-mock'); +const expect = require('chai').expect; +const path = require('path'); +const AWS = require('aws-sdk-mock'); AWS.setSDK(path.resolve('./node_modules/aws-sdk')); -let lambda = require('../index.js'); +const lambda = require('../index.js'); describe('#ERROR HANDLER::', () => { - let _lambda = { - guid: "1234", - error: "LAMBDA", - function: "workflow", + const _lambda = { + guid: '1234', + error: 'LAMBDA', + function: 'workflow', }; - let _encode = { - guid: "12345678", - error: "MEDIACONVERT", + const _encode = { + guid: '12345678', + error: 'MEDIACONVERT', errorMessage: 'Encoding Error', detail: { jobId: '1111111', @@ -46,7 +46,7 @@ describe('#ERROR HANDLER::', () => { AWS.mock('DynamoDB.DocumentClient', 'update', Promise.resolve()); AWS.mock('SNS', 'publish', Promise.resolve()); - let response = await lambda.handler(_lambda) + const response = await lambda.handler(_lambda); expect(response.error).to.equal('LAMBDA'); }); @@ -54,7 +54,7 @@ describe('#ERROR HANDLER::', () => { AWS.mock('DynamoDB.DocumentClient', 'update', Promise.resolve()); AWS.mock('SNS', 'publish', Promise.resolve()); - let response = await lambda.handler(_encode) + const response = await lambda.handler(_encode); expect(response.error).to.equal('MEDIACONVERT'); }); diff --git a/source/error-handler/package.json b/source/error-handler/package.json index 16f67a08..c2b96ba7 100644 --- a/source/error-handler/package.json +++ b/source/error-handler/package.json @@ -2,23 +2,22 @@ "name": "vod-error-handler", "version": "1.0.0", "engines": { - "node": ">=8" + "node": ">=12" }, "description": "error handler for video on demand", "main": "index.js", "scripts": { - "pretest": "npm install --quiet", + "pretest": "npm i --quiet", "test": "mocha lib/*.spec.js" }, - "dependencies": { - "aws-sdk": "^2.574.0" - }, + "dependencies": {}, "devDependencies": { - "aws-sdk-mock": "^4.5.0", + "aws-sdk": "^2.609.0", + "aws-sdk-mock": "^5.0.0", "chai": "^4.2.0", - "mocha": "^6.2.2", - "sinon": "^7.5.0", - "sinon-chai": "^3.3.0" + "mocha": "^7.0.0", + "sinon": "^8.1.1", + "sinon-chai": "^3.4.0" }, "private": true, "author": { diff --git a/source/input-validate/index.js b/source/input-validate/index.js index aa107ee4..e384721b 100644 --- a/source/input-validate/index.js +++ b/source/input-validate/index.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * @@ -20,11 +20,10 @@ exports.handler = async (event) => { const s3 = new AWS.S3(); let data; - let params; try { // Default configuration for the workflow is built using the enviroment variables. - // Any parameter in config can be overwriten using an metadata file or API call. + // Any parameter in config can be overwriten using a metadata file. data = { guid: event.guid, startTime: moment().utc().toISOString(), @@ -35,53 +34,44 @@ exports.handler = async (event) => { destBucket: process.env.Destination, cloudFront: process.env.CloudFront, frameCapture: JSON.parse(process.env.FrameCapture), - archiveSource: JSON.parse(process.env.ArchiveSource), + archiveSource: process.env.ArchiveSource, jobTemplate_2160p: process.env.MediaConvert_Template_2160p, jobTemplate_1080p: process.env.MediaConvert_Template_1080p, jobTemplate_720p: process.env.MediaConvert_Template_720p, - - // https://github.com/awslabs/video-on-demand-on-aws/pull/27 - // Rotation support - inputRotate: process.env.InputRotate + inputRotate: process.env.InputRotate, + acceleratedTranscoding: process.env.AcceleratedTranscoding, + enableSns:JSON.parse(process.env.EnableSns), + enableSqs:JSON.parse(process.env.EnableSqs) }; switch (event.workflowTrigger) { case 'Metadata': - // Parse Metadata File console.log('Validating Metadata file::'); - let key = decodeURIComponent(event.Records[0].s3.object.key.replace(/\+/g, " ")); + + const key = decodeURIComponent(event.Records[0].s3.object.key.replace(/\+/g, " ")); data.srcMetadataFile = key; // Download json metadata file from s3 - params = { - Bucket: data.srcBucket, - Key: key - }; - - let metadata = await s3.getObject(params).promise(); - let metadataFile = JSON.parse(metadata.Body); + const metadata = await s3.getObject({ Bucket: data.srcBucket, Key: key }).promise(); - // Check metadata for srcVideo - if (!metadataFile.srcVideo) throw new Error('srcVideo is not defined in metadata::', metadataFile); + const metadataFile = JSON.parse(metadata.Body); + if (!metadataFile.srcVideo) { + throw new Error('srcVideo is not defined in metadata::', metadataFile); + } // https://github.com/awslabs/video-on-demand-on-aws/pull/23 // Normalize key in order to support different casing Object.keys(metadataFile).forEach((key) => { - let normalizedKey = key.charAt(0).toLowerCase() + key.substr(1); + const normalizedKey = key.charAt(0).toLowerCase() + key.substr(1); data[normalizedKey] = metadataFile[key]; }); // Check source file is accessible in s3 - params = { - Bucket: data.srcBucket, - Key: data.srcVideo - }; + await s3.headObject({ Bucket: data.srcBucket, Key: data.srcVideo }).promise(); - await s3.headObject(params).promise(); break; case 'Video': - // Updating config with source video and data data.srcVideo = decodeURIComponent(event.Records[0].s3.object.key.replace(/\+/g, " ")); break; diff --git a/source/input-validate/lib/error.js b/source/input-validate/lib/error.js index 1c5c2ac6..53b6be3f 100644 --- a/source/input-validate/lib/error.js +++ b/source/input-validate/lib/error.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * @@ -20,10 +20,10 @@ let errHandler = async (event, _err) => { try { let payload = { - "guid": event.guid, - "event": event, - "function": process.env.AWS_LAMBDA_FUNCTION_NAME, - "error": _err.toString() + 'guid': event.guid, + 'event': event, + 'function': process.env.AWS_LAMBDA_FUNCTION_NAME, + 'error': _err.toString() }; let params = { diff --git a/source/input-validate/lib/index.spec.js b/source/input-validate/lib/index.spec.js index 6ebddbe9..e8ac1dcd 100644 --- a/source/input-validate/lib/index.spec.js +++ b/source/input-validate/lib/index.spec.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * @@ -11,12 +11,12 @@ * and limitations under the License. * *********************************************************************************************************************/ -let expect = require('chai').expect; -var path = require('path'); -let AWS = require('aws-sdk-mock'); +const expect = require('chai').expect; +const path = require('path'); +const AWS = require('aws-sdk-mock'); AWS.setSDK(path.resolve('./node_modules/aws-sdk')); -let lambda = require('../index.js'); +const lambda = require('../index.js'); describe('#INPUT VALIDATE::', () => { process.env.ErrorHandler = 'error_handler'; @@ -25,8 +25,11 @@ describe('#INPUT VALIDATE::', () => { process.env.Source = 'source_bucket'; process.env.EnableMediaPackage = 'true'; process.env.InputRotate = 'DEGREE_0'; + process.env.AcceleratedTranscoding = 'DISABLED' + process.env.EnableSns = 'true'; + process.env.EnableSqs = 'true'; - let _video = { + const _video = { workflowTrigger: 'Video', guid: '1234-1223232-212121', Records: [{ @@ -38,7 +41,7 @@ describe('#INPUT VALIDATE::', () => { }] }; - let _json = { + const _json = { workflowTrigger: 'Metadata', guid: '1234-1223232-212121', Records: [{ @@ -50,24 +53,22 @@ describe('#INPUT VALIDATE::', () => { }] }; - afterEach(() => { - AWS.restore('S3'); - }); + afterEach(() => AWS.restore('S3')); - it('should return "SUCCESS" when validating souce video', async () => { - let response = await lambda.handler(_video); + it('should succeed when processing valid source video', async () => { + const response = await lambda.handler(_video); expect(response.srcVideo).to.equal('video.mp4'); }); - it('should return "SUCCESS" when validating metadata', async () => { - let validMetadata = { + it('should succeed when processing valid metadata', async () => { + const validMetadata = { "Body": '{"srcVideo": "video_from_json.mp4", "archiveSource": false, "frameCapture": false, "srcBucket": "other-source", "jobTemplate_720p": "other-template"}' }; AWS.mock('S3', 'getObject', Promise.resolve(validMetadata)); AWS.mock('S3', 'headObject', Promise.resolve()); - let response = await lambda.handler(_json); + const response = await lambda.handler(_json); expect(response.srcVideo).to.equal('video_from_json.mp4'); expect(response.archiveSource).to.be.false; expect(response.frameCapture).to.be.false; @@ -78,26 +79,26 @@ describe('#INPUT VALIDATE::', () => { }); it('should always use MediaPackage env variable', async () => { - let metadata = { + const metadata = { "Body": '{"srcVideo": "video_from_json.mp4", "enableMediaPackage": false }' }; AWS.mock('S3', 'getObject', Promise.resolve(metadata)); AWS.mock('S3', 'headObject', Promise.resolve()); - let response = await lambda.handler(_json); + const response = await lambda.handler(_json); expect(response.enableMediaPackage).to.be.true; }); it('should correctly handle metadata in PascalCase', async () => { - let invalidMetadata = { + const invalidMetadata = { "Body": '{"srcVideo": "video_from_json.mp4", "ArchiveSource": false, "FrameCapture": false, "SrcBucket": "other-source", "inputRotate": "AUTO" }' }; AWS.mock('S3', 'getObject', Promise.resolve(invalidMetadata)); AWS.mock('S3', 'headObject', Promise.resolve()); - let response = await lambda.handler(_json); + const response = await lambda.handler(_json); expect(response.srcVideo).to.equal('video_from_json.mp4'); expect(response.archiveSource).to.be.false; expect(response.frameCapture).to.be.false; @@ -105,13 +106,11 @@ describe('#INPUT VALIDATE::', () => { expect(response.inputRotate).to.equal('AUTO'); }); - it('should return "S3 GET ERROR" when validating metadata', async () => { + it('should fail when getting object from S3 throws an exception', async () => { AWS.mock('S3', 'getObject', Promise.reject('S3 GET ERROR')); AWS.mock('S3', 'headObject', Promise.resolve()); AWS.mock('Lambda', 'invoke', Promise.resolve()); - await lambda.handler(_json).catch(err => { - expect(err).to.equal('S3 GET ERROR'); - }); + await lambda.handler(_json).catch(err => expect(err).to.equal('S3 GET ERROR')); }); }); diff --git a/source/input-validate/package.json b/source/input-validate/package.json index 4715926f..ae6a9ce5 100644 --- a/source/input-validate/package.json +++ b/source/input-validate/package.json @@ -2,24 +2,24 @@ "name": "vod-input-validation", "version": "1.0.0", "engines": { - "node": ">=8" + "node": ">=12" }, "description": "ingest workflow for video on demand", "main": "index.js", "scripts": { - "pretest": "npm install --quiet", + "pretest": "npm i --quiet", "test": "mocha lib/*.spec.js" }, "dependencies": { - "aws-sdk": "^2.574.0", "moment": "^2.24.0" }, "devDependencies": { - "aws-sdk-mock": "^4.5.0", + "aws-sdk": "^2.609.0", + "aws-sdk-mock": "^5.0.0", "chai": "^4.2.0", - "mocha": "^6.2.2", - "sinon": "^7.5.0", - "sinon-chai": "^3.3.0" + "mocha": "^7.0.0", + "sinon": "^8.1.1", + "sinon-chai": "^3.4.0" }, "private": true, "author": { diff --git a/source/media-package-assets/index.js b/source/media-package-assets/index.js index 067196d0..32412e80 100644 --- a/source/media-package-assets/index.js +++ b/source/media-package-assets/index.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * @@ -14,7 +14,6 @@ const AWS = require('aws-sdk'); const crypto = require('crypto'); const error = require('./lib/error'); -const cloudfrontHelper = require('./lib/cloudfront'); const buildArnFromUri = (s3Uri) => { const S3_URI_ID = 's3://'; @@ -27,6 +26,26 @@ const buildArnFromUri = (s3Uri) => { return `arn:aws:s3:::${source}`; }; +/** + * Converts the assets to be advertised behind CloudFront (instead of going directly to MediaPackage). For instance: + * https://endpoint-id.egress.mediapackage-vod.us-east-1.amazonaws.com/out/v1/asset-id/index.m3u8 + * => + * https://ditribution-id.cloudfront.net/out/v1/asset-id/index.m3u8 + */ +const convertEndpoints = (egressEndpoints, cloudFrontEndpoint) => { + const url = new URL(process.env.GroupDomainName); + let updatedEndpoints = {}; + + egressEndpoints.forEach(endpoint => { + const parts = endpoint.PackagingConfigurationId.split('-'); + const config = parts.pop().toUpperCase(); + + updatedEndpoints[config] = endpoint.Url.replace(url.hostname, cloudFrontEndpoint); + }); + + return updatedEndpoints; +}; + const handler = async (event) => { console.log(`REQUEST:: ${JSON.stringify(event, null, 2)}`); @@ -45,12 +64,7 @@ const handler = async (event) => { console.log(`Ingesting asset:: ${JSON.stringify(params, null, 2)}`); const response = await mediaPackageVod.createAsset(params).promise(); - event.egressEndpoints = await cloudfrontHelper.convertEndpoints( - process.env.DistributionId, - event.cloudFront, - response.EgressEndpoints - ); - + event.egressEndpoints = convertEndpoints(response.EgressEndpoints, event.cloudFront); console.log(`ENDPOINTS:: ${JSON.stringify(event.egressEndpoints, null, 2)}`); } catch (err) { await error.handler(event, err); diff --git a/source/media-package-assets/lib/cloudfront.js b/source/media-package-assets/lib/cloudfront.js deleted file mode 100644 index 62f80d46..00000000 --- a/source/media-package-assets/lib/cloudfront.js +++ /dev/null @@ -1,125 +0,0 @@ -/********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * - * * - * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * - * with the License. A copy of the License is located at * - * * - * http://www.apache.org/licenses/LICENSE-2.0 * - * * - * or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES * - * OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions * - * and limitations under the License. * - *********************************************************************************************************************/ - -const AWS = require('aws-sdk'); - -module.exports.convertEndpoints = async (distributionId, domainName, mediaPackageEndpoints) => { - if (!distributionId) { - throw new Error('distributionId must be informed'); - } - - if (!domainName) { - throw new Error('domainName must be informed'); - } - - if (!mediaPackageEndpoints || mediaPackageEndpoints.length === 0) { - throw new Error('mediaPackageEndpoints must contain at least one value'); - } - - const cloudFront = new AWS.CloudFront(); - const originId = 'vodMPOrigin'; - const url = new URL(mediaPackageEndpoints[0].Url); - - const response = await cloudFront.getDistributionConfig({ Id: distributionId }).promise(); - const config = response.DistributionConfig; - - const originExists = config.Origins.Items.some(item => item.Id === originId); - if (!originExists) { - console.log(`Adding MediaPackage as origin to distribution ${distributionId}`); - - const customOrigin = { - Id: originId, - DomainName: url.hostname, - OriginPath: '', - CustomHeaders: { Quantity: 0 }, - CustomOriginConfig: { - HTTPPort: 80, - HTTPSPort: url.port || 443, - OriginProtocolPolicy: 'https-only', - OriginSslProtocols: { Quantity: 1, Items: ['TLSv1.2'] }, - OriginReadTimeout: 30, - OriginKeepaliveTimeout: 5 - } - }; - - config.Origins.Quantity = config.Origins.Items.push(customOrigin); - - const customBehavior = { - PathPattern: 'out/*', - TargetOriginId: originId, - ForwardedValues: { - QueryString: false, - Cookies: { Forward: 'none' }, - Headers: { Quantity: 0 }, - QueryStringCacheKeys: { Quantity: 0 } - }, - TrustedSigners: { Enabled: false, Quantity: 0 }, - ViewerProtocolPolicy: 'redirect-to-https', - MinTTL: 0, - AllowedMethods: { - Quantity: 2, - Items: ['HEAD', 'GET'], - CachedMethods: { Quantity: 2, Items: ['HEAD', 'GET'] } - }, - SmoothStreaming: false, - DefaultTTL: 86400, - MaxTTL: 31536000, - Compress: false, - LambdaFunctionAssociations: { Quantity: 0 }, - FieldLevelEncryptionId: '' - }; - - if (config.CacheBehaviors.Items) { - config.CacheBehaviors.Items.push(customBehavior); - } else { - config.CacheBehaviors.Items = [customBehavior]; - } - - config.CacheBehaviors.Quantity = config.CacheBehaviors.Items.length; - - try { - const params = { - Id: distributionId, - DistributionConfig: config, - IfMatch: response.ETag - }; - - await cloudFront.updateDistribution(params).promise(); - console.log(`Origins:: ${JSON.stringify(config.Origins.Items, null, 2)}`); - console.log(`Cache behaviors:: ${JSON.stringify(config.CacheBehaviors.Items, null, 2)}`); - } catch (error) { - /* - The updateDistribution operation throws a PreconditionFailed error if - the ETag returned by getDistributionConfig is not valid anymore. - - This means that the distribution was already updated, probably by another - file uploaded to S3 at the same time. We can safely ignore this error since - the custom origin only has to be added once. - */ - if (error.code !== 'PreconditionFailed') { - throw error; - } - } - } - - let updatedEndpoints = {}; - - mediaPackageEndpoints.forEach(endpoint => { - const parts = endpoint.PackagingConfigurationId.split('-'); - const config = parts.pop().toUpperCase(); - - updatedEndpoints[config] = endpoint.Url.replace(url.hostname, domainName); - }); - - return updatedEndpoints; -}; diff --git a/source/media-package-assets/lib/error.js b/source/media-package-assets/lib/error.js index 1c5c2ac6..53b6be3f 100644 --- a/source/media-package-assets/lib/error.js +++ b/source/media-package-assets/lib/error.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * @@ -20,10 +20,10 @@ let errHandler = async (event, _err) => { try { let payload = { - "guid": event.guid, - "event": event, - "function": process.env.AWS_LAMBDA_FUNCTION_NAME, - "error": _err.toString() + 'guid': event.guid, + 'event': event, + 'function': process.env.AWS_LAMBDA_FUNCTION_NAME, + 'error': _err.toString() }; let params = { diff --git a/source/media-package-assets/lib/index.spec.js b/source/media-package-assets/lib/index.spec.js index 3d6bdff7..c4ec3da1 100644 --- a/source/media-package-assets/lib/index.spec.js +++ b/source/media-package-assets/lib/index.spec.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * @@ -18,16 +18,18 @@ AWS.setSDK(path.resolve('./node_modules/aws-sdk')); const lambda = require('../index.js'); -const _createAssetResponse = { +const _domainName = 'https://random-id.egress.mediapackage-vod.us-east-1.amazonaws.com'; + +const createAssetResponse = { Arn: 'asset-arn', EgressEndpoints: [ { PackagingConfigurationId: 'packaging-config-hls', - Url: 'https://test.com/out/index.m3u8' + Url: `${_domainName}/out/index.m3u8` }, { PackagingConfigurationId: 'packaging-config-dash', - Url: 'https://test.com/out/index.mpd' + Url: `${_domainName}/out/index.mpd` } ], Id: 'asset-id', @@ -37,31 +39,21 @@ const _createAssetResponse = { SourceRoleArn: 'test-vod-role' }; -const _cloudfrontConfig = { - DistributionConfig: { - Origins: { - Items: [{ Id: 'vodMPOrigin' }], - Quantity: 1 - } - } -}; - describe('#INGEST::', () => { describe('Asset', () => { process.env.DistributionId = 'distribution-id'; process.env.GroupId = 'test-packaging-group'; + process.env.GroupDomainName = _domainName; process.env.MediaPackageVodRole = 'test-vod-role'; process.env.ErrorHandler = 'error_handler'; afterEach(() => { AWS.restore('MediaPackageVod'); AWS.restore('Lambda'); - AWS.restore('CloudFront'); }); it('should succeed with valid parameters', async () => { - AWS.mock('MediaPackageVod', 'createAsset', Promise.resolve(_createAssetResponse)); - AWS.mock('CloudFront', 'getDistributionConfig', Promise.resolve(_cloudfrontConfig)); + AWS.mock('MediaPackageVod', 'createAsset', Promise.resolve(createAssetResponse)); const event = { guid: 'ce791447-277c-4a8c-9a0d-c3ed28b495ea', diff --git a/source/media-package-assets/package.json b/source/media-package-assets/package.json index 69496389..31504fb6 100644 --- a/source/media-package-assets/package.json +++ b/source/media-package-assets/package.json @@ -2,24 +2,22 @@ "name": "vod-media-package-asset", "version": "1.0.0", "engines": { - "node": ">=8" + "node": ">=12" }, "description": "ingest function for video on demand", "main": "index.js", "scripts": { - "pretest": "npm install --quiet", + "pretest": "npm i --quiet", "test": "mocha lib/*.spec.js" }, - "dependencies": { - "aws-sdk": "^2.574.0" - }, + "dependencies": {}, "devDependencies": { - "aws-sdk-mock": "^4.5.0", + "aws-sdk": "^2.609.0", + "aws-sdk-mock": "^5.0.0", "chai": "^4.2.0", - "mocha": "^6.2.2", - "sinon": "^7.5.0", - "sinon-chai": "^3.3.0", - "lodash": "^4.17.15" + "mocha": "^7.0.0", + "sinon": "^8.1.1", + "sinon-chai": "^3.4.0" }, "private": true, "author": { diff --git a/source/mediainfo/lambda_function.py b/source/mediainfo/lambda_function.py index b923d3e3..51b0e1fe 100644 --- a/source/mediainfo/lambda_function.py +++ b/source/mediainfo/lambda_function.py @@ -99,7 +99,8 @@ def parse_text_attributes(track): def get_signed_url(bucket, obj): SIGNED_URL_EXPIRATION = 60 * 60 * 2 - s3_client = boto3.client('s3') + AWS_REGION = os.environ['AWS_REGION'] + s3_client = boto3.client('s3', endpoint_url=f'https://s3.{AWS_REGION}.amazonaws.com', region_name=AWS_REGION) return s3_client.generate_presigned_url( 'get_object', diff --git a/source/mediainfo/requirements.txt b/source/mediainfo/requirements.txt deleted file mode 100644 index 22d1e3bc..00000000 --- a/source/mediainfo/requirements.txt +++ /dev/null @@ -1 +0,0 @@ -boto3==1.9.221 \ No newline at end of file diff --git a/source/mediainfo/setup.py b/source/mediainfo/setup.py index 1d0d17d3..d5be5fd3 100644 --- a/source/mediainfo/setup.py +++ b/source/mediainfo/setup.py @@ -37,11 +37,7 @@ def finalize_options(self): assert self.zip_path is not None, 'Invalid zip_path' def run(self): - run_bash_command('rm -rf ./pypackage') - run_bash_command('pip3 install -r requirements.txt -t ./pypackage') - run_bash_command(f'cd pypackage && zip -rq9 {self.zip_path} . && cd ..') - run_bash_command(f'zip -g {self.zip_path} lambda_function.py requirements.txt ./bin/*') - run_bash_command('rm -rf ./pypackage') + run_bash_command(f'zip -rq9 {self.zip_path} lambda_function.py ./bin/*') class UnitTestsCommand(TestCommand): description = 'Run unit tests' @@ -49,7 +45,6 @@ class UnitTestsCommand(TestCommand): def run_tests(self): run_bash_command('rm -rf ./pytests && mkdir ./pytests') run_bash_command('cp lambda_function.py ./test*.py ./pytests') - run_bash_command('pip3 install -r requirements.txt -t ./pytests') run_bash_command('python3 -m unittest discover -s ./pytests -v') run_bash_command('rm -rf ./pytests') diff --git a/source/output-validate/index.js b/source/output-validate/index.js index 3fae99b6..ad01f56f 100644 --- a/source/output-validate/index.js +++ b/source/output-validate/index.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * @@ -15,94 +15,119 @@ const AWS = require('aws-sdk'); const error = require('./lib/error.js'); const moment = require('moment'); -const buildUrl = (originalValue) => originalValue.slice(5).split('/').splice(1, 3).join('/'); +const buildUrl = (originalValue) => originalValue.slice(5).split('/').splice(1).join('/'); exports.handler = async (event) => { - console.log(`REQUEST:: ${JSON.stringify(event, null, 2)}`); + console.log(`REQUEST:: ${JSON.stringify(event, null, 2)}`); - const dynamo = new AWS.DynamoDB.DocumentClient({ - region: process.env.AWS_REGION - }); + const dynamo = new AWS.DynamoDB.DocumentClient({ + region: process.env.AWS_REGION + }); - let data = {}; + const s3 = new AWS.S3(); - try { - // Get Config from DynamoDB (data required for the workflow) - let params = { - TableName: process.env.DynamoDBTable, - Key: { - guid: event.detail.userMetadata.guid, - } - }; + let data = {}; - data = await dynamo.get(params).promise(); - data = data.Item; - - data.encodingOutput = event; - data.workflowStatus = 'Complete'; - data.endTime = moment().utc().toISOString(); - - // Parse MediaConvert Output and generate CloudFront URLS. - event.detail.outputGroupDetails.forEach(output => { - console.log(`${output.type} found in outputs`); + try { + // Get Config from DynamoDB (data required for the workflow) + let params = { + TableName: process.env.DynamoDBTable, + Key: { + guid: event.detail.userMetadata.guid, + } + }; - switch (output.type) { - case 'HLS_GROUP': - data.hlsPlaylist = output.playlistFilePaths[0]; - data.hlsUrl = `https://${data.cloudFront}/${buildUrl(data.hlsPlaylist)}`; + data = await dynamo.get(params).promise(); + data = data.Item; - break; + data.encodingOutput = event; + data.workflowStatus = 'Complete'; + data.endTime = moment().utc().toISOString(); - case 'DASH_ISO_GROUP': - data.dashPlaylist = output.playlistFilePaths[0]; - data.dashUrl = `https://${data.cloudFront}/${buildUrl(data.dashPlaylist)}`; + // Parse MediaConvert Output and generate CloudFront URLS. + event.detail.outputGroupDetails.forEach(output => { + console.log(`${output.type} found in outputs`); - break; + switch (output.type) { + case 'HLS_GROUP': + data.hlsPlaylist = output.playlistFilePaths[0]; + data.hlsUrl = `https://${data.cloudFront}/${buildUrl(data.hlsPlaylist)}`; - case 'FILE_GROUP': - let files = []; - let urls = []; + break; - output.outputDetails.forEach((file) => { - files.push(file.outputFilePaths[0]); - urls.push(`https://${data.cloudFront}/${buildUrl(file.outputFilePaths[0])}`); - }); + case 'DASH_ISO_GROUP': + data.dashPlaylist = output.playlistFilePaths[0]; + data.dashUrl = `https://${data.cloudFront}/${buildUrl(data.dashPlaylist)}`; - if (files[0].split('.').pop() === 'mp4') { - data.mp4Outputs = files; - data.mp4Urls = urls; - } + break; - if (files[0].split('.').pop() === 'jpg') { - data.thumbNail = files; - data.thumbNailUrl = urls; - } + case 'FILE_GROUP': + let files = []; + let urls = []; + output.outputDetails.forEach((file) => { + + if (file.outputFilePaths) { + files.push(file.outputFilePaths[0]); + urls.push(`https://${data.cloudFront}/${buildUrl(file.outputFilePaths[0])}`); + } + }); + + if (files.length >0 && files[0].split('.').pop() === 'mp4') { + data.mp4Outputs = files; + data.mp4Urls = urls; + } - break; + break; - case 'MS_SMOOTH_GROUP': - data.mssPlaylist = output.playlistFilePaths[0]; - data.mssUrl = `https://${data.cloudFront}/${buildUrl(data.mssPlaylist)}`; + case 'MS_SMOOTH_GROUP': + data.mssPlaylist = output.playlistFilePaths[0]; + data.mssUrl = `https://${data.cloudFront}/${buildUrl(data.mssPlaylist)}`; - break; + break; - case 'CMAF_GROUP': - data.cmafDashPlaylist = output.playlistFilePaths[0]; - data.cmafDashUrl = `https://${data.cloudFront}/${buildUrl(data.cmafDashPlaylist)}`; + case 'CMAF_GROUP': + data.cmafDashPlaylist = output.playlistFilePaths[0]; + data.cmafDashUrl = `https://${data.cloudFront}/${buildUrl(data.cmafDashPlaylist)}`; - data.cmafHlsPlaylist = output.playlistFilePaths[1]; - data.cmafHlsUrl = `https://${data.cloudFront}/${buildUrl(data.cmafHlsPlaylist)}`; + data.cmafHlsPlaylist = output.playlistFilePaths[1]; + data.cmafHlsUrl = `https://${data.cloudFront}/${buildUrl(data.cmafHlsPlaylist)}`; - break; + break; - default: - throw new Error('Could not parse MediaConvert output'); - } - }); - } catch (err) { - await error.handler(event, err); - throw err; + default: + throw new Error('Could not parse MediaConvert output'); } - - return data; + }); + + /** + * feature: if framcapture and accelerated are both enabled the tumbnails are not listed in the CloudWatch + * output. adding a function to get the last image from the list of images. + */ + if (data.frameCapture) { + + data.thumbNails = []; + data.thumbNailsUrls = []; + + params = { + Bucket: data.destBucket, + Prefix: `${data.guid}/thumbnails/`, + }; + + let thumbNails = await s3.listObjects(params).promise(); + + if (thumbNails.Contents.legnth !=0) { + let lastImg = thumbNails.Contents.pop(); + data.thumbNails.push(`s3://${data.destBucket}/${lastImg.Key}`); + data.thumbNailsUrls.push(`https://${data.cloudFront}/${lastImg.Key}`); + } else { + throw new Error('MediaConvert Thumbnails not found in S3'); + } + + } + + } catch (err) { + await error.handler(event, err); + throw err; + } + return data; }; diff --git a/source/output-validate/lib/error.js b/source/output-validate/lib/error.js index 1c5c2ac6..53b6be3f 100644 --- a/source/output-validate/lib/error.js +++ b/source/output-validate/lib/error.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * @@ -20,10 +20,10 @@ let errHandler = async (event, _err) => { try { let payload = { - "guid": event.guid, - "event": event, - "function": process.env.AWS_LAMBDA_FUNCTION_NAME, - "error": _err.toString() + 'guid': event.guid, + 'event': event, + 'function': process.env.AWS_LAMBDA_FUNCTION_NAME, + 'error': _err.toString() }; let params = { diff --git a/source/output-validate/lib/index.spec.js b/source/output-validate/lib/index.spec.js index 27cc08c5..c4fc1537 100644 --- a/source/output-validate/lib/index.spec.js +++ b/source/output-validate/lib/index.spec.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * @@ -35,10 +35,30 @@ describe('#OUTPUT VALIDATE::', () => { const data = { Item: { guid: '1234', - cloudFront: 'cloudfront' + cloudFront: 'cloudfront', + destBucket: 'vod-destination', + frameCapture: false } }; + const img_data = { + Item: { + guid: '1234', + cloudFront: 'cloudfront', + destBucket: 'vod-destination', + frameCapture: true + } + }; + + const imgList = { + Contents: [ + { + Key: "12345/thumbnails/dude3.000.jpg", + + } + ] + } + process.env.ErrorHandler = 'error_handler'; afterEach(() => AWS.restore('DynamoDB.DocumentClient')); @@ -46,28 +66,28 @@ describe('#OUTPUT VALIDATE::', () => { it('should return "SUCCESS" on parsing CMAF MSS output', async () => { AWS.mock('DynamoDB.DocumentClient', 'get', Promise.resolve(data)); - let response = await lambda.handler(_events.cmafMss); - expect(response.mssPlaylist).to.equal('s3://voodoo-destination-1w8dqfz7w8cq3/12345/mss/big_bunny.ism'); + const response = await lambda.handler(_events.cmafMss); + expect(response.mssPlaylist).to.equal('s3://vod-destination/12345/mss/big_bunny.ism'); expect(response.mssUrl).to.equal('https://cloudfront/12345/mss/big_bunny.ism'); - expect(response.cmafDashPlaylist).to.equal('s3://voodoo-destination-1w8dqfz7w8cq3/12345/cmaf/big_bunny.mpd'); + expect(response.cmafDashPlaylist).to.equal('s3://vod-destination/12345/cmaf/big_bunny.mpd'); expect(response.cmafDashUrl).to.equal('https://cloudfront/12345/cmaf/big_bunny.mpd'); }); it('should return "SUCCESS" on parsing HLS DASH output', async () => { AWS.mock('DynamoDB.DocumentClient', 'get', Promise.resolve(data)); - let response = await lambda.handler(_events.hlsDash); - expect(response.hlsPlaylist).to.equal('s3://vod4-destination-fr0ao9hz7tbo/12345/hls/dude.m3u8'); + const response = await lambda.handler(_events.hlsDash); + expect(response.hlsPlaylist).to.equal('s3://vod-destination/12345/hls/dude.m3u8'); expect(response.hlsUrl).to.equal('https://cloudfront/12345/hls/dude.m3u8'); - expect(response.dashPlaylist).to.equal('s3://vod4-destination-fr0ao9hz7tbo/12345/dash/dude.mpd'); + expect(response.dashPlaylist).to.equal('s3://vod-destination/12345/dash/dude.mpd'); expect(response.dashUrl).to.equal('https://cloudfront/12345/dash/dude.mpd'); }); it('should return "SUCCESS" on parsing MP4 output', async () => { AWS.mock('DynamoDB.DocumentClient', 'get', Promise.resolve(data)); - let response = await lambda.handler(_events.mp4); - expect(response.mp4Outputs[0]).to.equal('s3://vod4-destination-fr0ao9hz7tbo/12345/mp4/dude_3.0Mbps.mp4'); + const response = await lambda.handler(_events.mp4); + expect(response.mp4Outputs[0]).to.equal('s3://vod-destination/12345/mp4/dude_3.0Mbps.mp4'); expect(response.mp4Urls[0]).to.equal('https://cloudfront/12345/mp4/dude_3.0Mbps.mp4'); }); @@ -88,3 +108,4 @@ describe('#OUTPUT VALIDATE::', () => { }); }); }); + diff --git a/source/output-validate/lib/test-events.js b/source/output-validate/lib/test-events.js index 4e986da1..b3694b45 100644 --- a/source/output-validate/lib/test-events.js +++ b/source/output-validate/lib/test-events.js @@ -9,19 +9,19 @@ const CmafMss = { outputGroupDetails: [ { playlistFilePaths: [ - 's3://voodoo-destination-1w8dqfz7w8cq3/12345/cmaf/big_bunny.mpd', - 's3://voodoo-destination-1w8dqfz7w8cq3/12345/cmaf/big_bunny.m3u8' + 's3://vod-destination/12345/cmaf/big_bunny.mpd', + 's3://vod-destination/12345/cmaf/big_bunny.m3u8' ], type: 'CMAF_GROUP' }, { outputDetails: [{ outputFilePaths: [ - 's3://voodoo-destination-1w8dqfz7w8cq3/12345/mss/big_bunny.ismv' + 's3://vod-destination/12345/mss/big_bunny.ismv' ] }], playlistFilePaths: [ - 's3://voodoo-destination-1w8dqfz7w8cq3/12345/mss/big_bunny.ism' + 's3://vod-destination/12345/mss/big_bunny.ism' ], type: 'MS_SMOOTH_GROUP' } @@ -41,13 +41,13 @@ const HlsDash = { outputGroupDetails: [ { playlistFilePaths: [ - 's3://vod4-destination-fr0ao9hz7tbo/12345/hls/dude.m3u8' + 's3://vod-destination/12345/hls/dude.m3u8' ], type: 'HLS_GROUP' }, { playlistFilePaths: [ - 's3://vod4-destination-fr0ao9hz7tbo/12345/dash/dude.mpd' + 's3://vod-destination/12345/dash/dude.mpd' ], type: 'DASH_ISO_GROUP' } @@ -79,7 +79,7 @@ const Mp4 = { outputDetails: [ { outputFilePaths: [ - 's3://vod4-destination-fr0ao9hz7tbo/12345/mp4/dude_3.0Mbps.mp4' + 's3://vod-destination/12345/mp4/dude_3.0Mbps.mp4' ], durationInMs: 13471, videoDetails: { diff --git a/source/output-validate/package.json b/source/output-validate/package.json index dba151b1..20670643 100644 --- a/source/output-validate/package.json +++ b/source/output-validate/package.json @@ -2,25 +2,25 @@ "name": "vod-output-validation", "version": "1.0.0", "engines": { - "node": ">=8" + "node": ">=12" }, "description": "ingest workflow for video on demand", "main": "index.js", "scripts": { - "pretest": "npm install --quiet", + "pretest": "npm i --quiet", "test": "mocha lib/*.spec.js" }, "dependencies": { - "aws-sdk": "^2.574.0", "moment": "^2.24.0", - "uuid": "^3.3.3" + "uuid": "^3.4.0" }, "devDependencies": { - "aws-sdk-mock": "^4.5.0", + "aws-sdk": "^2.609.0", + "aws-sdk-mock": "^5.0.0", "chai": "^4.2.0", - "mocha": "^6.2.2", - "sinon": "^7.5.0", - "sinon-chai": "^3.3.0" + "mocha": "^7.0.0", + "sinon": "^8.1.1", + "sinon-chai": "^3.4.0" }, "private": true, "author": { diff --git a/source/profiler/index.js b/source/profiler/index.js index 4aeb4a50..9b15b3ce 100644 --- a/source/profiler/index.js +++ b/source/profiler/index.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * diff --git a/source/profiler/lib/error.js b/source/profiler/lib/error.js index 1c5c2ac6..53b6be3f 100644 --- a/source/profiler/lib/error.js +++ b/source/profiler/lib/error.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * @@ -20,10 +20,10 @@ let errHandler = async (event, _err) => { try { let payload = { - "guid": event.guid, - "event": event, - "function": process.env.AWS_LAMBDA_FUNCTION_NAME, - "error": _err.toString() + 'guid': event.guid, + 'event': event, + 'function': process.env.AWS_LAMBDA_FUNCTION_NAME, + 'error': _err.toString() }; let params = { diff --git a/source/profiler/lib/index.spec.js b/source/profiler/lib/index.spec.js index b5ccab19..c07d0c73 100644 --- a/source/profiler/lib/index.spec.js +++ b/source/profiler/lib/index.spec.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * @@ -46,7 +46,7 @@ describe('#PROFILER::', () => { it('should return "SUCCESS" on profile set', async () => { AWS.mock('DynamoDB.DocumentClient', 'get', Promise.resolve(data)); - let response = await lambda.handler(_event); + const response = await lambda.handler(_event); expect(response.jobTemplate).to.equal('tmpl3'); expect(response.frameCaptureHeight).to.equal(720); expect(response.frameCaptureWidth).to.equal(1280); @@ -56,7 +56,7 @@ describe('#PROFILER::', () => { it('should return "SUCCESS" using a custom template', async () => { AWS.mock('DynamoDB.DocumentClient', 'get', Promise.resolve(data)); - let response = await lambda.handler(_tmpl_event); + const response = await lambda.handler(_tmpl_event); expect(response.jobTemplate).to.equal('customTemplate'); expect(response.frameCaptureHeight).to.equal(720); expect(response.frameCaptureWidth).to.equal(1280); diff --git a/source/profiler/package.json b/source/profiler/package.json index dca2e6b7..0b7dc5ce 100644 --- a/source/profiler/package.json +++ b/source/profiler/package.json @@ -2,23 +2,22 @@ "name": "vod-profiler", "version": "1.0.0", "engines": { - "node": ">=8" + "node": ">=12" }, "description": "process workflow for video on demand", "main": "index.js", "scripts": { - "pretest": "npm install --quiet", + "pretest": "npm i --quiet", "test": "mocha lib/*.spec.js" }, - "dependencies": { - "aws-sdk": "^2.574.0" - }, + "dependencies": {}, "devDependencies": { - "aws-sdk-mock": "^4.5.0", + "aws-sdk": "^2.609.0", + "aws-sdk-mock": "^5.0.0", "chai": "^4.2.0", - "mocha": "^6.2.2", - "sinon": "^7.5.0", - "sinon-chai": "^3.3.0" + "mocha": "^7.0.0", + "sinon": "^8.1.1", + "sinon-chai": "^3.4.0" }, "private": true, "author": { diff --git a/source/sns-notification/index.js b/source/sns-notification/index.js index 97c1662a..c46e548a 100644 --- a/source/sns-notification/index.js +++ b/source/sns-notification/index.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * diff --git a/source/sns-notification/lib/error.js b/source/sns-notification/lib/error.js index 1c5c2ac6..53b6be3f 100644 --- a/source/sns-notification/lib/error.js +++ b/source/sns-notification/lib/error.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * @@ -20,10 +20,10 @@ let errHandler = async (event, _err) => { try { let payload = { - "guid": event.guid, - "event": event, - "function": process.env.AWS_LAMBDA_FUNCTION_NAME, - "error": _err.toString() + 'guid': event.guid, + 'event': event, + 'function': process.env.AWS_LAMBDA_FUNCTION_NAME, + 'error': _err.toString() }; let params = { diff --git a/source/sns-notification/lib/index.spec.js b/source/sns-notification/lib/index.spec.js index c7864697..8a9bd06c 100644 --- a/source/sns-notification/lib/index.spec.js +++ b/source/sns-notification/lib/index.spec.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * @@ -31,7 +31,7 @@ describe('#SNS::', () => { startTime: 'now', srcVideo: 'video.mp4', workflowStatus: 'Complete', - jobTemplate_2160p: 'jobTemplate_720p', + jobTemplate_720p: 'jobTemplate_720p', jobTemplate_1080p: 'jobTemplate_1080p', jobTemplate_2160p: 'jobTemplate_1080p' }; diff --git a/source/sns-notification/package.json b/source/sns-notification/package.json index d433b782..7a7aef33 100644 --- a/source/sns-notification/package.json +++ b/source/sns-notification/package.json @@ -2,23 +2,22 @@ "name": "vod-notification", "version": "1.0.0", "engines": { - "node": ">=8" + "node": ">=12" }, "description": "sns function for video on demand", "main": "index.js", "scripts": { - "pretest": "npm install --quiet", + "pretest": "npm i --quiet", "test": "mocha lib/*.spec.js" }, - "dependencies": { - "aws-sdk": "^2.574.0" - }, + "dependencies": {}, "devDependencies": { - "aws-sdk-mock": "^4.5.0", + "aws-sdk": "^2.609.0", + "aws-sdk-mock": "^5.0.0", "chai": "^4.2.0", - "mocha": "^6.2.2", - "sinon": "^7.5.0", - "sinon-chai": "^3.3.0" + "mocha": "^7.0.0", + "sinon": "^8.1.1", + "sinon-chai": "^3.4.0" }, "private": true, "author": { diff --git a/source/sqs-publish/index.js b/source/sqs-publish/index.js new file mode 100644 index 00000000..54fb75de --- /dev/null +++ b/source/sqs-publish/index.js @@ -0,0 +1,42 @@ +/********************************************************************************************************************* + * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * * + * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * + * with the License. A copy of the License is located at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES * + * OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions * + * and limitations under the License. * + *********************************************************************************************************************/ + +const AWS = require('aws-sdk'); +const error = require('./lib/error.js'); + + +exports.handler = async (event) => { + console.log(`REQUEST:: ${JSON.stringify(event, null, 2)}`); + + const sqs = new AWS.SQS({ + region: process.env.AWS_REGION + }); + + try { + + console.log(`SEND SQS:: ${JSON.stringify(event, null, 2)}`); + + let params = { + MessageBody: JSON.stringify(event, null, 2), + QueueUrl: process.env.SqsQueue + }; + + await sqs.sendMessage(params).promise(); + + } catch (err) { + await error.handler(event, err); + throw err; + } + + return event; +}; diff --git a/source/sqs-publish/lib/error.js b/source/sqs-publish/lib/error.js new file mode 100644 index 00000000..1c5c2ac6 --- /dev/null +++ b/source/sqs-publish/lib/error.js @@ -0,0 +1,45 @@ +/********************************************************************************************************************* + * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * * + * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * + * with the License. A copy of the License is located at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES * + * OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions * + * and limitations under the License. * + *********************************************************************************************************************/ + +const AWS = require('aws-sdk'); + +let errHandler = async (event, _err) => { + const lambda = new AWS.Lambda({ + region: process.env.AWS_REGION + }); + + try { + let payload = { + "guid": event.guid, + "event": event, + "function": process.env.AWS_LAMBDA_FUNCTION_NAME, + "error": _err.toString() + }; + + let params = { + FunctionName: process.env.ErrorHandler, + Payload: JSON.stringify(payload, null, 2) + }; + + await lambda.invoke(params).promise(); + } catch (err) { + console.log(err); + throw err; + } + + return 'success'; +}; + +module.exports = { + handler: errHandler +}; diff --git a/source/sqs-publish/lib/index.spec.js b/source/sqs-publish/lib/index.spec.js new file mode 100644 index 00000000..2282cfd1 --- /dev/null +++ b/source/sqs-publish/lib/index.spec.js @@ -0,0 +1,48 @@ +/********************************************************************************************************************* + * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * * + * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * + * with the License. A copy of the License is located at * + * * + * http://www.apache.org/licenses/LICENSE-2.0 * + * * + * or in the 'license' file accompanying this file. This file is distributed on an 'AS IS' BASIS, WITHOUT WARRANTIES * + * OR CONDITIONS OF ANY KIND, express or implied. See the License for the specific language governing permissions * + * and limitations under the License. * + *********************************************************************************************************************/ + +const expect = require('chai').expect; +const path = require('path'); +const AWS = require('aws-sdk-mock'); +AWS.setSDK(path.resolve('./node_modules/aws-sdk')); + +const lambda = require('../index.js'); + +describe('#SNS::', () => { + const _event = { + guid: '12345678', + startTime: 'now', + srcVideo: 'video.mp4', + }; + + process.env.ErrorHandler = 'error_handler'; + process.env.SqsQueue = 'https://sqs.amazonaws.com/1234' + + afterEach(() => AWS.restore('SQS')); + + it('should return "success" on send SQS message', async () => { + AWS.mock('SQS', 'sendMessage', Promise.resolve()); + + const response = await lambda.handler(_event); + expect(response.srcVideo).to.equal('video.mp4'); + }); + + it('should return "SQS ERROR" when send SQS message fails', async () => { + AWS.mock('SQS', 'sendMessage', Promise.reject('SQS ERROR')); + AWS.mock('Lambda', 'invoke', Promise.resolve()); + + await lambda.handler(_event).catch(err => { + expect(err).to.equal('SQS ERROR'); + }); + }); +}); diff --git a/source/sqs-publish/package.json b/source/sqs-publish/package.json new file mode 100644 index 00000000..5afc6a7f --- /dev/null +++ b/source/sqs-publish/package.json @@ -0,0 +1,27 @@ +{ + "name": "vod-notification", + "version": "1.0.0", + "engines": { + "node": ">=12" + }, + "description": "sqs function for video on demand", + "main": "index.js", + "scripts": { + "pretest": "npm i --quiet", + "test": "mocha lib/*.spec.js" + }, + "dependencies": {}, + "devDependencies": { + "aws-sdk": "^2.609.0", + "aws-sdk-mock": "^4.5.0", + "chai": "^4.2.0", + "mocha": "^6.2.2", + "sinon": "^7.5.0", + "sinon-chai": "^3.3.0" + }, + "private": true, + "author": { + "name": "aws-solutions-builder" + }, + "license": "Apache-2.0" +} diff --git a/source/step-functions/index.js b/source/step-functions/index.js index 4539875e..bc85e5b9 100644 --- a/source/step-functions/index.js +++ b/source/step-functions/index.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * diff --git a/source/step-functions/lib/error.js b/source/step-functions/lib/error.js index 1c5c2ac6..53b6be3f 100644 --- a/source/step-functions/lib/error.js +++ b/source/step-functions/lib/error.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * @@ -20,10 +20,10 @@ let errHandler = async (event, _err) => { try { let payload = { - "guid": event.guid, - "event": event, - "function": process.env.AWS_LAMBDA_FUNCTION_NAME, - "error": _err.toString() + 'guid': event.guid, + 'event': event, + 'function': process.env.AWS_LAMBDA_FUNCTION_NAME, + 'error': _err.toString() }; let params = { diff --git a/source/step-functions/lib/index.spec.js b/source/step-functions/lib/index.spec.js index 8d1b4135..3b6b40bc 100644 --- a/source/step-functions/lib/index.spec.js +++ b/source/step-functions/lib/index.spec.js @@ -1,5 +1,5 @@ /********************************************************************************************************************* - * Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved. * + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. * * * * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance * * with the License. A copy of the License is located at * @@ -11,12 +11,12 @@ * and limitations under the License. * *********************************************************************************************************************/ -let expect = require('chai').expect; -var path = require('path'); -let AWS = require('aws-sdk-mock'); +const expect = require('chai').expect; +const path = require('path'); +const AWS = require('aws-sdk-mock'); AWS.setSDK(path.resolve('./node_modules/aws-sdk')); -let lambda = require('../index.js'); +const lambda = require('../index.js'); describe('#STEP FUNCTIONS::', () => { process.env.IngestWorkflow = 'INGEST'; @@ -24,23 +24,21 @@ describe('#STEP FUNCTIONS::', () => { process.env.PublishWorkflow = 'PUBLISH'; process.env.ErrorHandler = 'error_handler'; - let _ingest = { - Records: [ - { - s3: { - object: { - key: "big_bunny.mp4", - } + const _ingest = { + Records: [{ + s3: { + object: { + key: 'big_bunny.mp4', } } - ] + }] }; - let _process = { + const _process = { guid: '1234' }; - let _publish = { + const _publish = { detail: { userMetadata: { guid: '1234' @@ -48,39 +46,37 @@ describe('#STEP FUNCTIONS::', () => { } }; - let _error = {}; + const _error = {}; - let data = { - executionArn: "arn" + const data = { + executionArn: 'arn' }; - afterEach(() => { - AWS.restore('StepFunctions'); - }); + afterEach(() => AWS.restore('StepFunctions')); it('should return "success" on Ingest Execute success', async () => { AWS.mock('StepFunctions', 'startExecution', Promise.resolve(data)); - let response = await lambda.handler(_ingest); + const response = await lambda.handler(_ingest); expect(response).to.equal('success'); }); it('should return "success" on process Execute success', async () => { AWS.mock('StepFunctions', 'startExecution', Promise.resolve(data)); - let response = await lambda.handler(_process); + const response = await lambda.handler(_process); expect(response).to.equal('success'); }); it('should return "success" on publish Execute success', async () => { AWS.mock('StepFunctions', 'startExecution', Promise.resolve(data)); - let response = await lambda.handler(_publish); + const response = await lambda.handler(_publish); expect(response).to.equal('success'); }); it('should return "ERROR" with an invalid event object', async () => { - AWS.mock('StepFunctions', 'startExecution', Promise.resolve('data')); + AWS.mock('StepFunctions', 'startExecution', Promise.resolve(data)); AWS.mock('Lambda', 'invoke', Promise.resolve()); await lambda.handler(_error).catch(err => { diff --git a/source/step-functions/package.json b/source/step-functions/package.json index eb20fdd5..a280c331 100644 --- a/source/step-functions/package.json +++ b/source/step-functions/package.json @@ -2,25 +2,25 @@ "name": "vod-step-functions", "version": "1.0.0", "engines": { - "node": ">=8" + "node": ">=12" }, "description": "ingest workflow for video on demand", "main": "index.js", "scripts": { - "pretest": "npm install --quiet", + "pretest": "npm i --quiet", "test": "mocha lib/*.spec.js" }, "dependencies": { - "aws-sdk": "^2.574.0", "moment": "^2.24.0", - "uuid": "^3.3.3" + "uuid": "^3.4.0" }, "devDependencies": { - "aws-sdk-mock": "^4.5.0", + "aws-sdk": "^2.609.0", + "aws-sdk-mock": "^5.0.0", "chai": "^4.2.0", - "mocha": "^6.2.2", - "sinon": "^7.5.0", - "sinon-chai": "^3.3.0" + "mocha": "^7.0.0", + "sinon": "^8.1.1", + "sinon-chai": "^3.4.0" }, "private": true, "author": {