Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[functionbeat] Expose cloudformation error when waiting for status #8912

Closed
ph opened this issue Nov 2, 2018 · 3 comments
Closed

[functionbeat] Expose cloudformation error when waiting for status #8912

ph opened this issue Nov 2, 2018 · 3 comments
Assignees

Comments

@ph
Copy link
Contributor

ph commented Nov 2, 2018

Currently, we only wait for Completed or Failed events for cloudformation; it's not the best thing for debugging. Instead, we should also monitor Stack Events and report them to the console as they appear.

This way it makes it a lot easier to debug why a stack failed to apply or not.

@ph
Copy link
Contributor Author

ph commented Nov 28, 2018

To simplify writing other feature I am currently working on this, The DescribeStacksEvents is the API that you want to hit to retrieve any events related to a specific stack.

The problem is this API actually returns all the events for a stack, are you cannot say retrieve events after a specific date, so you actually have to do something like this.

  • Fetch Events
  • Move cursor to event for a resource type of stack
  • Event must have one of the following status: CREATE_, UPDATE_ or DELETE_*
  • We must assert the time to make sure its older than the beginning of our stack. (some jitter might be necessary)

When the following match this mean we have our first relevant event 🎉
Retrieve calls is a matter of doing multiple call over the API using the NextToken option to scope the pagination results.

BUT, When a stack is currently in progress nextToken can be nil because we have finished reading all the current events. So we also need to keep track of events that we have already received.

@ph
Copy link
Contributor Author

ph commented Nov 28, 2018

I think the easiest way is to implement this:

  • Have a goroutine that check only the status of a stack.
  • Have a goroutine that only do the events reporting.

When a stack operation is detected to be complete by the first go routine we say to the second one to stop on next event.

@ph
Copy link
Contributor Author

ph commented Nov 28, 2018

This seems to work pretty well, I have the full output of the cloudformation event, this mean when a create stack job fails I can do a nice cleanup job.

ph added a commit to ph/beats that referenced this issue Dec 6, 2018
When creating, updating or deleting a stack on AWS a lot of different
things can go wrong, to name a few:

- Permissions
- Missing resources
- Missing files

Before when we were deploying a new cloudformation template we only
displayed information when running in debug mode internal
log statements and the final status from cloudformation.
This was actually hidding all the work that was
happening behind the scene and when needed to debug something we
were required to log into the UI and inspect manually the stack event.

This commit changes the behavior and will instead starts a goroutine that
will listen to events coming from the DescribeStackEvent's API.

The API is not the most flexible to work with, since there are no way to
specify at what point in time we are interested in receiving events and
we cannot say give me all the events from a specific Stack Change.
So instead we retrieve the list of events and skip through it until we
position ourself to when a new stack event is created. Functionbeat
stack events are small so it doesn't take too much time to position
ourself.

After we periodically poll the events and keep track of pagination, any
already seen events are ignored and everything else is printed to the
console in Info mode.

When you are using one of the deploy, update or delete subcommand and if the
stack fail you will get a stack failed error but you will be able to
inspect the log and find where and why it fails.

Example output in verbose:

```
2018-12-06T14:49:14.855-0500    INFO    instance/beat.go:597    Home path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat] Config path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat] Data path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat/data] Logs path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat/logs]
2018-12-06T14:49:14.855-0500    INFO    instance/beat.go:604    Beat UUID: 693706c5-688b-4673-a8af-e0d931894fdc
2018-12-06T14:49:28.292-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::CloudFormation::Stack, LogicalResourceId: fnb-mylogs-stack, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: User Initiated
2018-12-06T14:49:32.405-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:32.405-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:32.405-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:34.465-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:34.465-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:49:44.804-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:49:46.861-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:48.927-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:50.982-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:57.197-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:57.197-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:50:03.403-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:50:03.403-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_COMPLETE
Function: mylogs, deploy successful
```

Fixes: elastic#8912
@ph ph closed this as completed in #9409 Dec 13, 2018
ph added a commit that referenced this issue Dec 13, 2018
* Display CloudFormation StackEvent in the CLI

When creating, updating or deleting a stack on AWS a lot of different
things can go wrong, to name a few:

- Permissions
- Missing resources
- Missing files

Before when we were deploying a new cloudformation template we only
displayed information when running in debug mode internal
log statements and the final status from cloudformation.
This was actually hidding all the work that was
happening behind the scene and when needed to debug something we
were required to log into the UI and inspect manually the stack event.

This commit changes the behavior and will instead starts a goroutine that
will listen to events coming from the DescribeStackEvent's API.

The API is not the most flexible to work with, since there are no way to
specify at what point in time we are interested in receiving events and
we cannot say give me all the events from a specific Stack Change.
So instead we retrieve the list of events and skip through it until we
position ourself to when a new stack event is created. Functionbeat
stack events are small so it doesn't take too much time to position
ourself.

After we periodically poll the events and keep track of pagination, any
already seen events are ignored and everything else is printed to the
console in Info mode.

When you are using one of the deploy, update or delete subcommand and if the
stack fail you will get a stack failed error but you will be able to
inspect the log and find where and why it fails.

Example output in verbose:

```
2018-12-06T14:49:14.855-0500    INFO    instance/beat.go:597    Home path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat] Config path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat] Data path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat/data] Logs path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat/logs]
2018-12-06T14:49:14.855-0500    INFO    instance/beat.go:604    Beat UUID: 693706c5-688b-4673-a8af-e0d931894fdc
2018-12-06T14:49:28.292-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::CloudFormation::Stack, LogicalResourceId: fnb-mylogs-stack, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: User Initiated
2018-12-06T14:49:32.405-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:32.405-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:32.405-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:34.465-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:34.465-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:49:44.804-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:49:46.861-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:48.927-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:50.982-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:57.197-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:57.197-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:50:03.403-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:50:03.403-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_COMPLETE
Function: mylogs, deploy successful
```

Fixes: #8912
ph added a commit to ph/beats that referenced this issue Dec 13, 2018
* Display CloudFormation StackEvent in the CLI

When creating, updating or deleting a stack on AWS a lot of different
things can go wrong, to name a few:

- Permissions
- Missing resources
- Missing files

Before when we were deploying a new cloudformation template we only
displayed information when running in debug mode internal
log statements and the final status from cloudformation.
This was actually hidding all the work that was
happening behind the scene and when needed to debug something we
were required to log into the UI and inspect manually the stack event.

This commit changes the behavior and will instead starts a goroutine that
will listen to events coming from the DescribeStackEvent's API.

The API is not the most flexible to work with, since there are no way to
specify at what point in time we are interested in receiving events and
we cannot say give me all the events from a specific Stack Change.
So instead we retrieve the list of events and skip through it until we
position ourself to when a new stack event is created. Functionbeat
stack events are small so it doesn't take too much time to position
ourself.

After we periodically poll the events and keep track of pagination, any
already seen events are ignored and everything else is printed to the
console in Info mode.

When you are using one of the deploy, update or delete subcommand and if the
stack fail you will get a stack failed error but you will be able to
inspect the log and find where and why it fails.

Example output in verbose:

```
2018-12-06T14:49:14.855-0500    INFO    instance/beat.go:597    Home path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat] Config path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat] Data path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat/data] Logs path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat/logs]
2018-12-06T14:49:14.855-0500    INFO    instance/beat.go:604    Beat UUID: 693706c5-688b-4673-a8af-e0d931894fdc
2018-12-06T14:49:28.292-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::CloudFormation::Stack, LogicalResourceId: fnb-mylogs-stack, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: User Initiated
2018-12-06T14:49:32.405-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:32.405-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:32.405-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:34.465-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:34.465-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:49:44.804-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:49:46.861-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:48.927-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:50.982-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:57.197-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:57.197-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:50:03.403-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:50:03.403-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_COMPLETE
Function: mylogs, deploy successful
```

Fixes: elastic#8912
(cherry picked from commit 54314dd)
ph added a commit that referenced this issue Dec 13, 2018
#9518)

Cherry-pick of PR #9409 to 6.x branch. Original message: 

When creating, updating or deleting a stack on AWS a lot of different
things can go wrong, to name a few:

- Permissions
- Missing resources
- Missing files

Before when we were deploying a new cloudformation template we only
displayed information when running in debug mode internal
log statements and the final status from cloudformation.
This was actually hidding all the work that was
happening behind the scene and when needed to debug something we
were required to log into the UI and inspect manually the stack event.

This commit changes the behavior and will instead starts a goroutine that
will listen to events coming from the DescribeStackEvent's API.

The API is not the most flexible to work with, since there are no way to
specify at what point in time we are interested in receiving events and
we cannot say give me all the events from a specific Stack Change.
So instead we retrieve the list of events and skip through it until we
position ourself to when a new stack event is created. Functionbeat
stack events are small so it doesn't take too much time to position
ourself.

After we periodically poll the events and keep track of pagination, any
already seen events are ignored and everything else is printed to the
console in Info mode.

When you are using one of the deploy, update or delete subcommand and if the
stack fail you will get a stack failed error but you will be able to
inspect the log and find where and why it fails.

Example output in verbose:

```
2018-12-06T14:49:14.855-0500    INFO    instance/beat.go:597    Home path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat] Config path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat] Data path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat/data] Logs path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat/logs]
2018-12-06T14:49:14.855-0500    INFO    instance/beat.go:604    Beat UUID: 693706c5-688b-4673-a8af-e0d931894fdc
2018-12-06T14:49:28.292-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::CloudFormation::Stack, LogicalResourceId: fnb-mylogs-stack, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: User Initiated
2018-12-06T14:49:32.405-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:32.405-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:32.405-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:34.465-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:34.465-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:49:44.804-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:49:46.861-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:48.927-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:50.982-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:57.197-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:57.197-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:50:03.403-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:50:03.403-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_COMPLETE
Function: mylogs, deploy successful
```

Fixes: #8912
ph added a commit to ph/beats that referenced this issue Dec 13, 2018
* Display CloudFormation StackEvent in the CLI

When creating, updating or deleting a stack on AWS a lot of different
things can go wrong, to name a few:

- Permissions
- Missing resources
- Missing files

Before when we were deploying a new cloudformation template we only
displayed information when running in debug mode internal
log statements and the final status from cloudformation.
This was actually hidding all the work that was
happening behind the scene and when needed to debug something we
were required to log into the UI and inspect manually the stack event.

This commit changes the behavior and will instead starts a goroutine that
will listen to events coming from the DescribeStackEvent's API.

The API is not the most flexible to work with, since there are no way to
specify at what point in time we are interested in receiving events and
we cannot say give me all the events from a specific Stack Change.
So instead we retrieve the list of events and skip through it until we
position ourself to when a new stack event is created. Functionbeat
stack events are small so it doesn't take too much time to position
ourself.

After we periodically poll the events and keep track of pagination, any
already seen events are ignored and everything else is printed to the
console in Info mode.

When you are using one of the deploy, update or delete subcommand and if the
stack fail you will get a stack failed error but you will be able to
inspect the log and find where and why it fails.

Example output in verbose:

```
2018-12-06T14:49:14.855-0500    INFO    instance/beat.go:597    Home path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat] Config path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat] Data path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat/data] Logs path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat/logs]
2018-12-06T14:49:14.855-0500    INFO    instance/beat.go:604    Beat UUID: 693706c5-688b-4673-a8af-e0d931894fdc
2018-12-06T14:49:28.292-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::CloudFormation::Stack, LogicalResourceId: fnb-mylogs-stack, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: User Initiated
2018-12-06T14:49:32.405-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:32.405-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:32.405-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:34.465-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:34.465-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:49:44.804-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:49:46.861-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:48.927-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:50.982-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:57.197-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:57.197-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:50:03.403-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:50:03.403-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_COMPLETE
Function: mylogs, deploy successful
```

Fixes: elastic#8912
(cherry picked from commit 54314dd)
ph added a commit that referenced this issue Dec 27, 2018
* Display CloudFormation StackEvent in the CLI

When creating, updating or deleting a stack on AWS a lot of different
things can go wrong, to name a few:

- Permissions
- Missing resources
- Missing files

Before when we were deploying a new cloudformation template we only
displayed information when running in debug mode internal
log statements and the final status from cloudformation.
This was actually hidding all the work that was
happening behind the scene and when needed to debug something we
were required to log into the UI and inspect manually the stack event.

This commit changes the behavior and will instead starts a goroutine that
will listen to events coming from the DescribeStackEvent's API.

The API is not the most flexible to work with, since there are no way to
specify at what point in time we are interested in receiving events and
we cannot say give me all the events from a specific Stack Change.
So instead we retrieve the list of events and skip through it until we
position ourself to when a new stack event is created. Functionbeat
stack events are small so it doesn't take too much time to position
ourself.

After we periodically poll the events and keep track of pagination, any
already seen events are ignored and everything else is printed to the
console in Info mode.

When you are using one of the deploy, update or delete subcommand and if the
stack fail you will get a stack failed error but you will be able to
inspect the log and find where and why it fails.

Example output in verbose:

```
2018-12-06T14:49:14.855-0500    INFO    instance/beat.go:597    Home path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat] Config path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat] Data path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat/data] Logs path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat/logs]
2018-12-06T14:49:14.855-0500    INFO    instance/beat.go:604    Beat UUID: 693706c5-688b-4673-a8af-e0d931894fdc
2018-12-06T14:49:28.292-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::CloudFormation::Stack, LogicalResourceId: fnb-mylogs-stack, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: User Initiated
2018-12-06T14:49:32.405-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:32.405-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:32.405-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:34.465-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:34.465-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:49:44.804-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:49:46.861-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:48.927-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:50.982-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:57.197-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:57.197-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:50:03.403-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:50:03.403-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_COMPLETE
Function: mylogs, deploy successful
```

Fixes: #8912
(cherry picked from commit 54314dd)
leweafan pushed a commit to leweafan/beats that referenced this issue Apr 28, 2023
…9526)

* Display CloudFormation StackEvent in the CLI

When creating, updating or deleting a stack on AWS a lot of different
things can go wrong, to name a few:

- Permissions
- Missing resources
- Missing files

Before when we were deploying a new cloudformation template we only
displayed information when running in debug mode internal
log statements and the final status from cloudformation.
This was actually hidding all the work that was
happening behind the scene and when needed to debug something we
were required to log into the UI and inspect manually the stack event.

This commit changes the behavior and will instead starts a goroutine that
will listen to events coming from the DescribeStackEvent's API.

The API is not the most flexible to work with, since there are no way to
specify at what point in time we are interested in receiving events and
we cannot say give me all the events from a specific Stack Change.
So instead we retrieve the list of events and skip through it until we
position ourself to when a new stack event is created. Functionbeat
stack events are small so it doesn't take too much time to position
ourself.

After we periodically poll the events and keep track of pagination, any
already seen events are ignored and everything else is printed to the
console in Info mode.

When you are using one of the deploy, update or delete subcommand and if the
stack fail you will get a stack failed error but you will be able to
inspect the log and find where and why it fails.

Example output in verbose:

```
2018-12-06T14:49:14.855-0500    INFO    instance/beat.go:597    Home path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat] Config path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat] Data path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat/data] Logs path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat/logs]
2018-12-06T14:49:14.855-0500    INFO    instance/beat.go:604    Beat UUID: 693706c5-688b-4673-a8af-e0d931894fdc
2018-12-06T14:49:28.292-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::CloudFormation::Stack, LogicalResourceId: fnb-mylogs-stack, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: User Initiated
2018-12-06T14:49:32.405-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:32.405-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:32.405-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:34.465-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:34.465-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:49:44.804-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:49:46.861-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:48.927-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:50.982-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_IN_PROGRESS
2018-12-06T14:49:53.054-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:57.197-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2018-12-06T14:49:57.197-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:50:03.403-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_COMPLETE
2018-12-06T14:50:03.403-0500    INFO    [aws]   aws/op_cloudformation.go:106    Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_COMPLETE
Function: mylogs, deploy successful
```

Fixes: elastic#8912
(cherry picked from commit c7d013d)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant