-
Notifications
You must be signed in to change notification settings - Fork 3.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
state machine : Dynamic passing of bucket and key to distributed map #29409
Comments
Thank you for the feature request. We welcome and appreciate your PR for this. |
I faced the same issue and still wanted to use the I came up with a dirty, hacky workaround and extended the export interface ExtendedS3CsvItemReaderProps extends sfn.S3CsvItemReaderProps {
readonly "key.$"?: string;
}
class ExtendedS3CsvItemReader extends sfn.S3CsvItemReader {
readonly dynamicKey: string | undefined;
constructor(props: ExtendedS3CsvItemReaderProps) {
super(props);
this.dynamicKey = props["key.$"];
}
public render() {
const rendered = super.render();
if (this.dynamicKey) {
// ignores the provided `key` and uses `key.$` instead
rendered.Parameters = {
Bucket: rendered.Parameters.Bucket,
"Key.$": this.dynamicKey,
};
}
return rendered;
}
} You can use it just like the existing I may dig deeper and provide a PR (with a polished implementation of course 😜). |
**Why are these changes required?** - For `DistributedMap` state of StepFunctions, `IItemReader` currently only allows S3 bucket as input source to be declared statically in CDK. - In other words, current CDK implementation only caters to static use-case where we know either `bucket` or `bucketName` (from which we can create `IBucket`) and pass it to `IItemReader`. - Whereas via AWS Console, if we create `DistributedMap` manually, then we can also convey S3 source dynamically using State Input / JsonPath. - In other words, for dynamic use-case, we will neither have `bucket` nor `bucketName` i.e. we only know state input variable which will convey `bucketName` e.g. `$.bucketName`. - So, if we want to use `IItemReader` for dynamic use case also, then we will: - (1) need to make `bucket: IBucket` optional (which will be breaking - how? e.g. if some dev is currently accessing `bucket` via `reader.bucket` then dev now needs to add check for `undefined`) - (2) then add another optional fields to convey state input variable names (e.g. $.bucketName $.key $.prefix) - Therefore, to avoid introducing breaking change, we can follow `*Path` convention of StepFunctions and introduce `IItemReaderPath` for dynamic use-case. (closes aws#29409) **What changes are being made?** - Add `IItemReaderPath` interface (and its pros interface) - Add `S3ObjectsItemReaderPath` as one of many concrete classes of `IItemReaderPath` - this class also helps with unit-testing and integration-testing. - Modify `index` to export new constructs - Modify `DistributedMap` (and its props) to allow `itemReaderPath?` (which will be mutually exclusive with `itemReader?` and `itemsPath?`) and utilise it for render **How are these changes tested?** - Via new unit-tests for `DistributedMap` (via `yarn build+test`) - Via new integration test for `S3ObjectsItemReaderPath` (with snapshot created) - Via `yarn build --directory test/aws-stepfunctions/test && yarn integ test/aws-stepfunctions/test/integ.item-reader-path-s3-object.js && yarn integ-runner --update-on-failed` - Verified expected step function execution result during snapshot creation
Assigned this issue to @ChakshuGupta13, I will try to review your PR by end of this month. If I don't, feel free to ping me directly. |
**Why are these changes required?** - For `DistributedMap` state of StepFunctions, `IItemReader` currently only allows S3 bucket as input source to be declared statically in CDK. - In other words, current CDK implementation only caters to static use-case where we know either `bucket` or `bucketName` (from which we can create `IBucket`) and pass it to `IItemReader`. - Whereas via AWS Console, if we create `DistributedMap` manually, then we can also convey S3 source dynamically using State Input / JsonPath. - In other words, for dynamic use-case, we will neither have `bucket` nor `bucketName` i.e. we only know state input variable which will convey `bucketName` e.g. `$.bucketName`. - So, if we want to use `IItemReader` for dynamic use case also, then we will: - (1) need to make `bucket: IBucket` optional (which will be breaking - how? e.g. if some dev is currently accessing `bucket` via `reader.bucket` then dev now needs to add check for `undefined`) - (2) then add another optional fields to convey state input variable names (e.g. $.bucketName $.key $.prefix) - Therefore, to avoid introducing breaking change, we can follow `*Path` convention of StepFunctions and introduce `IItemReaderPath` for dynamic use-case. (closes aws#29409) **What changes are being made?** - Add `IItemReaderPath` interface (and its pros interface) - Add `S3ObjectsItemReaderPath` as one of many concrete classes of `IItemReaderPath` - this class also helps with unit-testing and integration-testing. - Modify `index` to export new constructs - Modify `DistributedMap` (and its props) to allow `itemReaderPath?` (which will be mutually exclusive with `itemReader?` and `itemsPath?`) and utilise it for render **How are these changes tested?** - Via new unit-tests for `DistributedMap` (via `yarn build+test`) - Via new integration test for `S3ObjectsItemReaderPath` (with snapshot created) - Via `yarn build --directory test/aws-stepfunctions/test && yarn integ test/aws-stepfunctions/test/integ.item-reader-path-s3-object.js && yarn integ-runner --update-on-failed` - Verified expected step function execution result during snapshot creation
**Why are these changes required?** - For `DistributedMap` state of StepFunctions, `IItemReader` currently only allows S3 bucket as input source to be declared statically in CDK. - In other words, current CDK implementation only caters to static use-case where we know either `bucket` or `bucketName` (from which we can create `IBucket`) and pass it to `IItemReader`. - Whereas via AWS Console, if we create `DistributedMap` manually, then we can also convey S3 source dynamically using State Input / JsonPath. - In other words, for dynamic use-case, we will neither have `bucket` nor `bucketName` i.e. we only know state input variable which will convey `bucketName` e.g. `$.bucketName`. - So, if we want to use `IItemReader` for dynamic use case also, then we will: - (1) need to make `bucket: IBucket` optional (which will be breaking - how? e.g. if some dev is currently accessing `bucket` via `reader.bucket` then dev now needs to add check for `undefined`) - (2) then add another optional fields to convey state input variable names (e.g. $.bucketName $.key $.prefix) - Therefore, to avoid introducing breaking change, we can follow `*Path` convention of StepFunctions and introduce `IItemReaderPath` for dynamic use-case. (closes aws#29409) **What changes are being made?** - Add `IItemReaderPath` interface (and its pros interface) - Add `S3ObjectsItemReaderPath` as one of many concrete classes of `IItemReaderPath` - this class also helps with unit-testing and integration-testing. - Modify `index` to export new constructs - Modify `DistributedMap` (and its props) to allow `itemReaderPath?` (which will be mutually exclusive with `itemReader?` and `itemsPath?`) and utilise it for render **How are these changes tested?** - Via new unit-tests for `DistributedMap` (via `yarn build+test`) - Via new integration test for `S3ObjectsItemReaderPath` (with snapshot created) - Via `yarn build --directory test/aws-stepfunctions/test && yarn integ test/aws-stepfunctions/test/integ.item-reader-path-s3-object.js && yarn integ-runner --update-on-failed` - Verified expected step function execution result during snapshot creation
FYI: We can still currently utilise dynamic parameters except
This code snippet produces following cdk.out template:
|
**Why are these changes required?** - For `DistributedMap` state of StepFunctions, `IItemReader` currently only allows S3 bucket as input source to be declared statically in CDK. - In other words, current CDK implementation only caters to static use-case where we know either `bucket` or `bucketName` (from which we can create `IBucket`) and pass it to `IItemReader`. - Whereas via AWS Console, if we create `DistributedMap` manually, then we can also convey S3 source dynamically using State Input / JsonPath. - In other words, for dynamic use-case, we will neither have `bucket` nor `bucketName` i.e. we only know state input variable which will convey `bucketName` e.g. `$.bucketName`. - So, if we want to use `IItemReader` for dynamic use case also, then we will: - (1) need to make `bucket: IBucket` optional (which will be breaking - how? e.g. if some dev is currently accessing `bucket` via `reader.bucket` then dev now needs to add check for `undefined`) - (2) then add another optional fields to convey state input variable names (e.g. $.bucketName $.key $.prefix) - Therefore, to avoid introducing breaking change, we can follow `*Path` convention of StepFunctions and introduce `IItemReaderPath` for dynamic use-case. (closes aws#29409) **What changes are being made?** - Add `IItemReaderPath` interface (and its pros interface) - Add `S3ObjectsItemReaderPath` as one of many concrete classes of `IItemReaderPath` - this class also helps with unit-testing and integration-testing. - Modify `index` to export new constructs - Modify `DistributedMap` (and its props) to allow `itemReaderPath?` (which will be mutually exclusive with `itemReader?` and `itemsPath?`) and utilise it for render **How are these changes tested?** - Via new unit-tests for `DistributedMap` (via `yarn build+test`) - Via new integration test for `S3ObjectsItemReaderPath` (with snapshot created) - Via `yarn build --directory test/aws-stepfunctions/test && yarn integ test/aws-stepfunctions/test/integ.item-reader-path-s3-object.js && yarn integ-runner --update-on-failed` - Verified expected step function execution result during snapshot creation
Describe the feature
Currently there is no means of passing a bucket and key dynamically to a distributed Map state using CDK. This functionality is available in the states language using JSONPath along the lines of
"ItemReader": {
"Resource": "arn:aws:states:::s3:getObject",
"ReaderConfig": {
"InputType": "JSON"
},
"Parameters": {
"Bucket.$": "$.Payload.bucket",
"Key.$": "$.Payload.key"
}
}
Use Case
I want to define my state machine using CDK rather than the states language
Proposed Solution
Create an IItemReader that can take a string in place of a bucket.
Other Information
No response
Acknowledgements
CDK version used
2.128.0
Environment details (OS name and version, etc.)
MacOs Sonoma
The text was updated successfully, but these errors were encountered: