Cherry-pick #9409 to 6.x: Display CloudFormation StackEvent in the CLI #9518
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Display CloudFormation StackEvent in the CLI When creating, updating or deleting a stack on AWS a lot of different things can go wrong, to name a few: - Permissions - Missing resources - Missing files Before when we were deploying a new cloudformation template we only displayed information when running in debug mode internal log statements and the final status from cloudformation. This was actually hidding all the work that was happening behind the scene and when needed to debug something we were required to log into the UI and inspect manually the stack event. This commit changes the behavior and will instead starts a goroutine that will listen to events coming from the DescribeStackEvent's API. The API is not the most flexible to work with, since there are no way to specify at what point in time we are interested in receiving events and we cannot say give me all the events from a specific Stack Change. So instead we retrieve the list of events and skip through it until we position ourself to when a new stack event is created. Functionbeat stack events are small so it doesn't take too much time to position ourself. After we periodically poll the events and keep track of pagination, any already seen events are ignored and everything else is printed to the console in Info mode. When you are using one of the deploy, update or delete subcommand and if the stack fail you will get a stack failed error but you will be able to inspect the log and find where and why it fails. Example output in verbose: ``` 2018-12-06T14:49:14.855-0500 INFO instance/beat.go:597 Home path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat] Config path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat] Data path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat/data] Logs path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat/logs] 2018-12-06T14:49:14.855-0500 INFO instance/beat.go:604 Beat UUID: 693706c5-688b-4673-a8af-e0d931894fdc 2018-12-06T14:49:28.292-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::CloudFormation::Stack, LogicalResourceId: fnb-mylogs-stack, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: User Initiated 2018-12-06T14:49:32.405-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:32.405-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:32.405-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:34.465-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:34.465-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:49:44.804-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:49:46.861-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:48.927-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:50.982-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:57.197-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:57.197-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:50:03.403-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:50:03.403-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_COMPLETE Function: mylogs, deploy successful ``` Fixes: elastic#8912 (cherry picked from commit 54314dd)
urso
approved these changes
Dec 13, 2018
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Cherry-pick of PR #9409 to 6.x branch. Original message:
When creating, updating or deleting a stack on AWS a lot of different
things can go wrong, to name a few:
Before when we were deploying a new cloudformation template we only
displayed information when running in debug mode internal
log statements and the final status from cloudformation.
This was actually hidding all the work that was
happening behind the scene and when needed to debug something we
were required to log into the UI and inspect manually the stack event.
This commit changes the behavior and will instead starts a goroutine that
will listen to events coming from the DescribeStackEvent's API.
The API is not the most flexible to work with, since there are no way to
specify at what point in time we are interested in receiving events and
we cannot say give me all the events from a specific Stack Change.
So instead we retrieve the list of events and skip through it until we
position ourself to when a new stack event is created. Functionbeat
stack events are small so it doesn't take too much time to position
ourself.
After we periodically poll the events and keep track of pagination, any
already seen events are ignored and everything else is printed to the
console in Info mode.
When you are using one of the deploy, update or delete subcommand and if the
stack fail you will get a stack failed error but you will be able to
inspect the log and find where and why it fails.
Example output in verbose:
Fixes: #8912