Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add support for Step Functions #827

Merged
merged 38 commits into from
Oct 16, 2018
Merged
Show file tree
Hide file tree
Changes from 32 commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
6a76727
Model the Amazon States Language
adilosa Jul 12, 2018
d74e4c8
Change to classes to make jsii happy
Jul 14, 2018
61f310b
Added ChoiceState, validations, and basic serialization tests
adilosa Jul 16, 2018
1fbdee3
Add method and class documentation
Jul 26, 2018
b4bad6b
Add unit tests
Jul 27, 2018
f33cd7b
Merge remote-tracking branch 'origin/master' into huijbers/step-funct…
Aug 28, 2018
a7deb69
WIP
Aug 28, 2018
a5dd0b3
Merge remote-tracking branch 'origin/master' into huijbers/step-funct…
Aug 28, 2018
813f4a4
First iteration of object model and builder
Aug 29, 2018
385ae1f
WIP
Sep 3, 2018
4a1172c
Make an activity be possible to be the resource for a Task
Sep 3, 2018
9e54761
Making all tests pass with mutable interface
Sep 3, 2018
c4e849f
Add TODO
Sep 3, 2018
0c32d6d
Some more tests
Sep 3, 2018
2b402f9
Add defaultRetry, get rid of internal ugliness in public API
Sep 3, 2018
9a80554
Add integration test
Sep 3, 2018
766d0c8
Add metrics
Sep 4, 2018
006f98e
Metric tests for activities
Sep 4, 2018
abb85ef
Add comments to all states
Sep 4, 2018
dc74909
Add resultPath to error handler
Sep 4, 2018
b23fe58
Add some more supported input/output/resultpaths
Sep 4, 2018
bd4f1da
Support "Result" field on "Pass"
Sep 4, 2018
9645ade
Small fixes to deal with oddities of mutable state
Sep 4, 2018
c53238d
Shorten generated name if necessary
Sep 4, 2018
d46454d
Merge remote-tracking branch 'origin/master' into huijbers/step-funct…
Sep 21, 2018
f8d534e
Update to latest API on 'master'
Sep 21, 2018
9a84c20
Finish first iteration
Sep 21, 2018
beebb71
Review comments
Oct 9, 2018
a9a5fa4
Merge remote-tracking branch 'origin/master' into huijbers/step-funct…
Oct 9, 2018
6c979a0
Merge remote-tracking branch 'origin/master' into huijbers/step-funct…
Oct 9, 2018
8307cd3
Update to new API
Oct 9, 2018
584645d
Work around jsii bug that does not register implicitly created proper…
Oct 9, 2018
fe43fbd
Removing obsolete DESIGN_NOTES
rix0rrr Oct 10, 2018
ffe3533
Merge remote-tracking branch 'origin/master' into huijbers/step-funct…
Oct 11, 2018
4f792c8
Merge remote-tracking branch 'origin/master' into huijbers/step-funct…
Oct 12, 2018
b88b2ff
Merge remote-tracking branch 'origin/master' into huijbers/step-funct…
Oct 15, 2018
ae693ab
Method renames
Oct 16, 2018
0e8cc0e
Merge branch 'huijbers/step-functions' of github.com:awslabs/aws-cdk …
Oct 16, 2018
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion examples/cdk-examples-typescript/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,8 @@
"@aws-cdk/aws-sns": "^0.10.0",
"@aws-cdk/aws-sqs": "^0.10.0",
"@aws-cdk/cdk": "^0.10.0",
"@aws-cdk/runtime-values": "^0.10.0"
"@aws-cdk/runtime-values": "^0.10.0",
"@aws-cdk/aws-stepfunctions": "^0.10.0"
},
"repository": {
"url": "https://github.com/awslabs/aws-cdk.git",
Expand Down
16 changes: 15 additions & 1 deletion packages/@aws-cdk/aws-lambda/lib/lambda-ref.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import events = require('@aws-cdk/aws-events');
import iam = require('@aws-cdk/aws-iam');
import logs = require('@aws-cdk/aws-logs');
import s3n = require('@aws-cdk/aws-s3-notifications');
import stepfunctions = require('@aws-cdk/aws-stepfunctions');
import cdk = require('@aws-cdk/cdk');
import { cloudformation } from './lambda.generated';
import { Permission } from './permission';
Expand Down Expand Up @@ -37,7 +38,7 @@ export interface FunctionRefProps {

export abstract class FunctionRef extends cdk.Construct
implements events.IEventRuleTarget, logs.ILogSubscriptionDestination, s3n.IBucketNotificationDestination,
ec2.IConnectable {
ec2.IConnectable, stepfunctions.IStepFunctionsTaskResource {

/**
* Creates a Lambda function object which represents a function not defined
Expand Down Expand Up @@ -333,6 +334,19 @@ export abstract class FunctionRef extends cdk.Construct
};
}

public asStepFunctionsTaskResource(_callingTask: stepfunctions.Task): stepfunctions.StepFunctionsTaskResourceProps {
return {
resourceArn: this.functionArn,
metricPrefixSingular: 'LambdaFunction',
metricPrefixPlural: 'LambdaFunctions',
metricDimensions: { LambdaFunctionArn: this.functionArn },
policyStatements: [new iam.PolicyStatement()
.addResource(this.functionArn)
.addActions("lambda:InvokeFunction")
]
};
}

private parsePermissionPrincipal(principal?: iam.PolicyPrincipal) {
if (!principal) {
return undefined;
Expand Down
1 change: 1 addition & 0 deletions packages/@aws-cdk/aws-lambda/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,7 @@
"@aws-cdk/aws-s3": "^0.10.0",
"@aws-cdk/aws-s3-notifications": "^0.10.0",
"@aws-cdk/aws-sqs": "^0.10.0",
"@aws-cdk/aws-stepfunctions": "^0.10.0",
"@aws-cdk/cdk": "^0.10.0",
"@aws-cdk/cx-api": "^0.10.0"
},
Expand Down
216 changes: 216 additions & 0 deletions packages/@aws-cdk/aws-stepfunctions/DESIGN_NOTES.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,216 @@
- Goal: should be possible to define structures such as IfThenElse() by users.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can see why, but every time you add the ability to express a construct that's not in ASL, this creates the risk that if ASL subsequently wants to add the same thing, probably with subtly varying semantics, things get awkward. I'd think there's a case for, in V1.0, sticking pretty close to what ASL can do.

We'd like the usage of these constructs to look more or less like they would look
in regular programming languages. It should look roughly like this:

task
.then(new IfThenElse(
new Condition(),
task.then(task).then(task),
task.then(task))
.then(task)

- Goal: should be possible to define reusable-recurring pieces with parameters,
and then reuse them.

new Parallel([
new DoSomeWork({ quality: 'high' }),
new DoSomeWork({ quality: 'medium' }),
new DoSomeWork({ quality: 'low' }),
])

- Goal: States defined in the same StateMachineDefinition share a scope and cannot refer
to states outside that scope. StateMachineDefinition can be exploded into other
StateMachineDefinitions.

- Goal: you shouldn't HAVE to define a StateMachineDefinition to use in a Parallel
(even though should render as such when expanding to ASL). The following should also work:

new Parallel([
task.then(task).then(task),
task.then(task),
task
]);

Regardless of how the states get into the Parallel, it should not be possible for them
to jump outside the Parallel branch.

- Other kind of syntax:

task1.then(task2).then(task3).goto(task1); // <--- ends chain

- Interface we need from State:

interface IState {
next(chainable): Chainable;
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not an explicit task.endMachine() to model "End": true? In the future State Machines are going to need to be event-driven, which means multiple threads of execution. So being able to explicitly signal terminating the state machine is distinct and useful.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, this whole file shouldn't have been here anymore, sorry. It's mostly outdated too.


Right now, absence of a Next is assumed to be an End state, and it doesn't necessarily end the whole machine (might also end a branch in a Parallel).

Can you elaborate a bit on the upcoming change, I don't quite get what it is.

goto(state): void; // Use this for terminators as well?

first(): State;
allReachable(): State[];
}

StateMachineDefinition
- All targeted states must be part of the same StateMachineDefinition
enclosure.
- States can be "lifted" into a separate enclosure by being a target of the
Parallel branch.
- Must be able to enumerate all reachable states, so that it can error on
them. Every task can only be the target of a .next() once, but it can
be the target of multiple goto()s.

- Contraints: Because of JSII:

- No overloading (Golang)
- No generics (Golang)
- No return type covariance (C#)

Lack of overloading means that in order to not have a different function call for
every state type, we'll have to do runtime checks (for example, not all states
can be next()'ed onto, and we cannot use the type system to detect and appropriately
constrain this).

Bonus: runtime checks are going to help dynamically-typed users.

- Constraint: Trying to next() onto a Chainable that doens't have a next(), runtime error;
Lack of overloads make it impossible to do otherwise.

- State machine definition should be frozen (and complete) after being chained to.

Nextable: Pass, Task, Wait, Parallel
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All of these can be terminating.

Terminating: Succeed, Fail
Weird: Choice

class Blah extends StateMachine {
constructor() {
// define statemachine here
}
}

Use as:

new StateMachine(..., {
definintion: new Blah()
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo

});

But also:

const def = new StateMachineDefinition();
const task = new Task(def, ...);
task.then(new Blah())// <-- expand (does that introduce naming? It should!)
.then(new Task(...); // <-- appends transition to every end state!

And also:

const p = new Parallel();
p.parallel(new Blah()); // <-- leave as state machine

But not:

const p = new Parallel();
blah = new Blah();
p.parallel(blah);

blah.then(...); // <-- should be immutable!!

And also:

const task1 = new Task1();
const task2 = new Task1();

const p = new Parallel();
p.parallel(task1); // <-- convert to state machine
p.parallel(task2);

class FrozenStateMachine {
render();
}

TO CHECK
- Can SMDef be mutable?

QUESTION
- Do branches in Parallel allow Timeout/Version?
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not on the branch itself, but obvs there can be task states in a branch with timeouts.


PROBLEMS ENCOUNTERED
--------------------

task1.catch(handler).then(task2)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, the option (b) doesn't make sense, since the Handler has its own "Next", there's no reason to think that it'd want to go to task2.

So I think that if you're going to put a .catch() in one of these fluent expressions, you just have to accept that the catch() might fling you off somewhere else in the state machine if an error happens.


Does this mean:

(a) task1 -> task2
|
+---> handler

Or does this mean:

(b) task1 -------> task2
| ^
+--> handler --+

In the case of simply writing this, you probably want (a), but
in the case of a larger composition:

someStateMachine.then(task2)

You want behavior (b) in case someStateMachine = task1.catch(handler).

How to distinguish the two? The problem is in the .then() operator, but the
only way to solve it is with more operators (.close(), or otherwise) which is
going to confuse people 99% of the time.

If we make everything mutable, we can simply do the narrow .then() definition
(open transitions only include task1), and people can manually add more
transitions after handler if they want to (in an immutable system, there's no
good way to "grab" that handler object and add to the transitions later).
Also, in the mutable representation, goto's are easier to represent (as in:
need no special representation, we can simply use .then()).

Next complication however: if we do the mutable thing, and we can add to the
state machine in fragments, there's no object that represents the ENTIRE
state machine with all states and transitions. How do we get the full state
machine? Either we enumerate all children of a StateMachineDefinition object,
or we begin at the start state and crawl all accessible states. In the former
case, we can warn/error if not all states are used in the SM.

Then there is the construct model API. I would like to allow:

```ts
class SomeConstruct extends cdk.Construct {
constructor(parent: cdk.Construct) {
const task1 = new Task(this, ...);
const task2 = new Task(this, ...);

// ALLOW THIS
new StateMachine(..., {
definition: task1.then(task2)
});

// ALLOW THIS
task1.then(task2);
new StateMachine(..., {
definition: task1
});

new StateMachineDefinition(this, {
});
}
}
```

It's desirable to just have a StateMachineDefinition own and list everything,
but that kind of requires that for Parallel you need a StateMachineDefinition
as well (but probably without appending names).

---

Notes: use a complicated object model to hide ugly parts of the API from users
(by using "internal details" classes as capabilities).

Decouple state chainer from states.

Decouple rendering from states to allow elidable states.

'then' is a reserved word in Ruby
'catch' is a reserved word in many languages

https://halyph.com/blog/2016/11/28/prog-lang-reserved-words.html
Loading