(cfnspec): resources not defined in us-east-1 cfn specification are not found as l1 resources in CDK

[SamStephens]

Describe the bug

The CDK has no L1 constructs for resources that are not declared in us-east-1.

The one example we have of constructs that are only declared in regions other than us-east-1 is device-farm, which is only present in the us-west-2 AWS CloudFormation resource specification, as the AWS Device Farm service is only available in us-west-2.

Credit to @workeitel for discovering this in #17893.

Expected Behavior

The CDK should look at all regional AWS CloudFormation resource specifications when discovering resources.

Current Behavior

Even though the AWS CloudFormation resource specification is regional, the CDK only looks at us-east-1.

Reproduction Steps

CDK lacks L1 resources for Device Farm, even though they are present in the us-west-2 AWS CloudFormation resource specification.

Possible Solution

Rather than relying on any individual AWS CloudFormation resource specification, merge all the AWS CloudFormation resource specifications together to update the spec. It looks like it wouldn't be very difficult to hash merge the PropertyTypes and ResourceTypes together to get a superset of all the properties and resources available across all regions.

Additional Information/Context

No response

CDK CLI Version

2.46.0

Framework Version

No response

Node.js Version

Not relevant

OS

Not relevant

Language

Typescript, Python, .NET, Java, Go

Language Version

No response

Other information

No response

[workeitel]

Downloading all files from https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-resource-specification.html and merging them with: jq -S -s 'reduce .[] as $item ({}; . *= $item)' *.json us-east-1.json > merged.json does the trick. Notice the us-east-1.json file comes last to reduce the diff. A bunch of properties like Documentation links, UpdateType, DuplicatesAllowed, Required or PrimitiveType have regional differences.

After creating the merged.json cdk can be updated with [:cfnspec] $ npx bash -x build-tools/update.sh /tmp/merged.json

Eliminating the property changes the remaining diff for v105.0.0 is:

## New Resource Types

* AWS::DeviceFarm::DevicePool
* AWS::DeviceFarm::InstanceProfile
* AWS::DeviceFarm::NetworkProfile
* AWS::DeviceFarm::Project
* AWS::DeviceFarm::TestGridProject
* AWS::DeviceFarm::VPCEConfiguration

## Attribute Changes

* AWS::ApiGateway::RestApi RestApiId (__added__)
* AWS::EC2::TransitGateway TransitGatewayArn (__added__)
* AWS::Logs::SubscriptionFilter FilterName (__added__)
* AWS::StepFunctions::Activity Id (__added__)
* AWS::WAFv2::RuleGroup AvailableLabels (__added__)
* AWS::WAFv2::RuleGroup ConsumedLabels (__added__)

## Property Changes

* AWS::Config::OrganizationConfigRule OrganizationCustomPolicyRuleMetadata (__added__)
* AWS::Events::Rule BarName (__added__)
* AWS::GameLift::Fleet LogPaths (__added__)
* AWS::GameLift::Fleet ServerLaunchParameters (__added__)
* AWS::GameLift::Fleet ServerLaunchPath (__added__)
* AWS::Lambda::EventSourceMapping ScalingConfig (__added__)
* AWS::MediaPackage::Channel HlsIngest (__added__)

## Property Type Changes

* AWS::Config::OrganizationConfigRule.OrganizationCustomPolicyRuleMetadata (__added__)
* AWS::GameLift::Build.S3Location (__added__)
* AWS::Lambda::EventSourceMapping.ScalingConfig (__added__)
* AWS::MediaPackage::PackagingConfiguration.DashPackage IncludeIframeOnlyStream (__added__)

I assume hard coding all URLs and merging before executing update.sh is the easiest. Deep-merging them correctly might be bit tricky. Overall it raises the question of regional differences in L1 constructs 🤔

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[mrgrain]

Summary:

• chore(cfnspec): support us-west-2 only services #23928 adds support to including services only available in some regions (that are not us-east-1)
• This does not cover differences in properties. The assumption is that property differences can always be dealt with by using overrides.
• It is also using an explict allow list. New services won't be added automatically. Please continue to create new issues for any services that might still be missing, ping me or open a PR yourself.
• The team is working on a longer term solution to incoporate different spec sources as well as potentially making L1s region aware. No details and timeline on this yet.