(aws-kinesis): fails to deploy on-demand stream

[igilham]

What is the problem?

Error from CloudFormation when using Kinesis Data Streams in on-demand mode with shardCount specified.

ShardCount is not expected when StreamMode=ON_DEMAND

Reproduction Steps

Create a new stream with a streamMode set.

new Stream(this, 'KinesisStream', {
    streamName: 'my-ingest-stream',
    shardCount: 1, // optional
    streamMode: StreamMode.ON_DEMAND,
});

What did you expect to happen?

Deploy a stream in on-demand mode, possibly informed by the shardCount for an initial scaling hint.

What actually happened?

Error from CloudFormation

ShardCount is not expected when StreamMode=ON_DEMAND

CDK CLI Version

1.137.0 (build bfbdf64)

Framework Version

1.137.0

Node.js Version

14.17.3

OS

MacOS

Language

Typescript

Language Version

4.5.2

Other information

CloudFormation's documentation does not inform the user that setting the StreamMode to ON_DEMNAD is an error. Can we raise a bug with the right team to get the documentation improved?

Removing the shardCount field from the CDK user code does not fix the error because internally, CDK always sets the shardCount to 1 if it is left undefined by the user.

The solution in CDK is to remove shardCount from the underlying CfnStream if the user did not provide the field. CloudFormation will default to 1 anyway if needed, so CDK's default is not doing anything useful here.

We should also add a validator to ensure that shardCount and StreamMode: 'ON_DEMAND' are not set at the same time when generating the template.

An end-to-end test through CloudFormation would have caught this bug before release. I'm not aware that there is any automated testing of this type at present.

[igilham]

Since I wrote the code for this feature, I can only apologise for not testing it through CloudFormation before shipping it.

[otaviomacedo]

@igilham do you think you can submit another PR with these fixes?

An end-to-end test through CloudFormation would have caught this bug before release. I'm not aware that there is any automated testing of this type at present.

There are integration tests all over the CDK code base. You can take a look and write your own. It's pretty straightforward.

[igilham]

I'd be happy to fix the bug when I have time. It'll probably be next week. I'm not familiar with all the testing layers so doing a thorough job may not be such a quick fix but I'm happy to take a look.

If anyone wants to do it sooner than I can, I won't be offended but I'll aim to pick this up soon if nobody else has.

[igilham]

@otaviomacedo I'm making a start on this. Can you help me to understand how the L1 validators are generated? Do they come directly from the CloudFormation spec or are they generated from somewhere else in the repo?

Edit: Nevermind. I found a similar solution in DynamoDB's tables so I've copied the existing pattern.

[d4r3topk]

Have you tested the code for both PROVISIONED and ON_DEMAND options to check whether it works for the following scenarios.

• PROVISIONED without a shard count - should default to 1
• PROVISIONED with a shard count
• ON_DEMAND without shard count

[igilham]

@d4r3topk I'm working through these scenarios now.

I initially thought it would be easiest to remove default values from the generated CloudFormation and allow the upstream service's defaults to take over. Now that I've been spending a bit more time with CDK's codebase, this seems not to be the preferred pattern, though. CDK likes to be explicit where possible.

I'm now following the example set by Table in the DynamoDB module. This uses guard statements in the L2 constructor to ensure the correct defaults are applied in each scenario, or an error is thrown if the user sets incompatible options.

I can't get the tests to run on my computer so I'm relying on the CodeBuild infrastructure connected to the Pull Request. I hope to finish getting it ready today.

[d4r3topk]

@igilham I have a PR ready w/ a fix for this one but as you are working on it, I'll wait for your PR. I am looking to deploy this in a personal app ASAP. Please let me know if you face any issues with it.

[igilham]

I appreciate your enthusiasm and your support with code review. I'm trying to get this into production at work, myself.

I intend to keep moving this forward tomorrow (UTC). I'd be happy to collaborate but I'm aware that multiple PRs and discussions can slow things down. It may be fastest if we can incorporate your suggestions in the open PR, assuming you're happy with how it's going. You're always free to do your own thing or offer another solution, of course. Either way, we get the feature we need so I'd be happy.

[d4r3topk]

Oh yeah the commit looks fine to me. I haven't gone into reviewing changes in other modules test files for this change yet. But the core change looks good to me and I can cross-verify its working for me locally.

[d4r3topk]

I think one of the approvers of the repository has to review it as well. Kind note @kaizen3031593

[pierre-vr]

⚠️ Edit: Caution this might only work for creating a new stream from scratch (see comment below)

In the meantime, a quick fix is to manually set the shardCount prop to undefined on the Cfn resource:

const stream = new Stream(this, 'KinesisStream', {
    streamName: 'my-ingest-stream',
    streamMode: StreamMode.ON_DEMAND,
});

;(stream.node.defaultChild as CfnStream).shardCount = undefined

I have tested it and I am able to confirm that "ShardCount" is no longer present in the Cfn template.

[pierre-vr]

Actually, this might work for creating a new stream from scratch with ON_DEMAND mode. However, when trying to update an existing stream with provisioned shards to ON_DEMAND, the CloudFormation update fails with "Rate exceeded for stream {streamName} [...]".

[igilham]

I have confirmed @pierre-vr's work-around is successful for streams that have been manually marked as on-demand and therefore drifted from the deployed stack template, which had a specific shardCount.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[Harshit22]

@igilham
What is the mitigation for the Cloudformation deployment validation failure as CDK still defaults ShardCount as 1 internally.

CDK code

        const dataArchivalKinesisStream = new Stream(this, 'KinesisStream', {
            streamName: `${props.stageName}-${props.realm}-DataArchivalKinesisStream`,
            retentionPeriod: Duration.days(14),
            streamMode: StreamMode.ON_DEMAND,
        });

Excerpt from snapshot test generated artifact

    "KinesisStream": Object {
      "Properties": Object {
        "Name": "test-NA-DataArchivalKinesisStream",
        "RetentionPeriodHours": 336,
        "ShardCount": 1, // <------------------------- Issue
        "StreamEncryption": Object {
          "Fn::If": Array [
            "AwsCdkKinesisEncryptedStreamsUnsupportedRegions",
            Object {
              "Ref": "AWS::NoValue",
            },
            Object {
              "EncryptionType": "KMS",
              "KeyId": "alias/aws/kinesis",
            },
          ],
        },
        "StreamModeDetails": Object {
          "StreamMode": "ON_DEMAND",
        },
      },
      "Type": "AWS::Kinesis::Stream",
    },

Am I missing something?

[igilham]

@Harshit22 This issue is fixed in CDK versions 1.138.2 and 2.5.0.

The buggy version can be worked around using the code posted above by @pierre-vr.

Modifying it to match your example:

const dataArchivalKinesisStream = new Stream(this, 'KinesisStream', {
    streamName: `${props.stageName}-${props.realm}-DataArchivalKinesisStream`,
    retentionPeriod: Duration.days(14),
    streamMode: StreamMode.ON_DEMAND,
});
(dataArchivalKinesisStreamnode.defaultChild as CfnStream).shardCount = undefined

I have just created the following minimal test project to demonstrate that the fix works (using CDK 2.5.0):

app.ts

import { Construct } from 'constructs';
import { App, Duration, Stack, StackProps } from 'aws-cdk-lib';
import { Stream, StreamMode } from 'aws-cdk-lib/aws-kinesis';

const app = new App();

class MyStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);
    new Stream(this, 'stream', {
      streamName: 'my-stream',
      retentionPeriod: Duration.days(14),
      streamMode: StreamMode.ON_DEMAND
    });
  }
}

new MyStack(app, 'MyStack');

app.synth();

cdk.json

{
  "app": "npx ts-node --prefer-ts-exts lib/index.ts"
}

command to run

npx cdk synth

Output cdk.out/MyStack.template.json

{
  "Resources": {
    "stream19075594": {
      "Type": "AWS::Kinesis::Stream",
      "Properties": {
        "Name": "my-stream",
        "RetentionPeriodHours": 336,
        "StreamEncryption": {
          "Fn::If": [
            "AwsCdkKinesisEncryptedStreamsUnsupportedRegions",
            {
              "Ref": "AWS::NoValue"
            },
            {
              "EncryptionType": "KMS",
              "KeyId": "alias/aws/kinesis"
            }
          ]
        },
        "StreamModeDetails": {
          "StreamMode": "ON_DEMAND"
        }
      },
      "Metadata": {
        "aws:cdk:path": "MyStack/stream/Resource"
      }
    },
    "CDKMetadata": {
      "Type": "AWS::CDK::Metadata",
      "Properties": {
        "Analytics": "v2:deflate64:H4sIAAAAAAAA/yWIXQ5EMBCAz+K9HSziAG7AAaTbjmSUadIpHjbuvhpP388HOqgKc4m2zuuNvvCbkrFePWv2xCgkeUU0uxoWfu3OOgR2lChwjhElHNHirTg4hFXKs26h7qEpViHS8eBEO8L48g+nSx5idwAAAA=="
      },
      "Metadata": {
        "aws:cdk:path": "MyStack/CDKMetadata/Default"
      },
      "Condition": "CDKMetadataAvailable"
    }
  },
  "Conditions": {
    "AwsCdkKinesisEncryptedStreamsUnsupportedRegions": {
      "Fn::Or": [
        {
          "Fn::Equals": [
            {
              "Ref": "AWS::Region"
            },
            "cn-north-1"
          ]
        },
        {
          "Fn::Equals": [
            {
              "Ref": "AWS::Region"
            },
            "cn-northwest-1"
          ]
        }
      ]
    },
    "CDKMetadataAvailable": {
      "Fn::Or": [
        {
          "Fn::Or": [
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "af-south-1"
              ]
            },
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "ap-east-1"
              ]
            },
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "ap-northeast-1"
              ]
            },
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "ap-northeast-2"
              ]
            },
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "ap-south-1"
              ]
            },
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "ap-southeast-1"
              ]
            },
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "ap-southeast-2"
              ]
            },
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "ca-central-1"
              ]
            },
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "cn-north-1"
              ]
            },
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "cn-northwest-1"
              ]
            }
          ]
        },
        {
          "Fn::Or": [
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "eu-central-1"
              ]
            },
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "eu-north-1"
              ]
            },
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "eu-south-1"
              ]
            },
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "eu-west-1"
              ]
            },
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "eu-west-2"
              ]
            },
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "eu-west-3"
              ]
            },
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "me-south-1"
              ]
            },
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "sa-east-1"
              ]
            },
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "us-east-1"
              ]
            },
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "us-east-2"
              ]
            }
          ]
        },
        {
          "Fn::Or": [
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "us-west-1"
              ]
            },
            {
              "Fn::Equals": [
                {
                  "Ref": "AWS::Region"
                },
                "us-west-2"
              ]
            }
          ]
        }
      ]
    }
  },
  "Parameters": {
    "BootstrapVersion": {
      "Type": "AWS::SSM::Parameter::Value<String>",
      "Default": "/cdk-bootstrap/hnb659fds/version",
      "Description": "Version of the CDK Bootstrap resources in this environment, automatically retrieved from SSM Parameter Store. [cdk:skip]"
    }
  },
  "Rules": {
    "CheckBootstrapVersion": {
      "Assertions": [
        {
          "Assert": {
            "Fn::Not": [
              {
                "Fn::Contains": [
                  [
                    "1",
                    "2",
                    "3",
                    "4",
                    "5"
                  ],
                  {
                    "Ref": "BootstrapVersion"
                  }
                ]
              }
            ]
          },
          "AssertDescription": "CDK bootstrap stack version 6 required. Please run 'cdk bootstrap' with a recent version of the CDK CLI."
        }
      ]
    }
  }
}

[Harshit22]

Thanks @igilham for detailed workaround ! Upgrading to CDK version 1.138.2 fixed the issue.

[ivandres73]

Had this issue today.
Fixed it by removing package-lock.json and node_modules/

rm package-lock.json node_modules/ -rf

then, reinstalling all dependencies

npm install

doing that updated my packages to version 1.139.0 which included the fix.
Thank you @igilham