-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[functionbeat] Expose cloudformation error when waiting for status #8912
Comments
To simplify writing other feature I am currently working on this, The The problem is this API actually returns all the events for a stack, are you cannot say retrieve events after a specific date, so you actually have to do something like this.
When the following match this mean we have our first relevant event 🎉 BUT, When a stack is currently in progress nextToken can be nil because we have finished reading all the current events. So we also need to keep track of events that we have already received. |
I think the easiest way is to implement this:
When a stack operation is detected to be complete by the first go routine we say to the second one to stop on next event. |
This seems to work pretty well, I have the full output of the cloudformation event, this mean when a create stack job fails I can do a nice cleanup job. |
When creating, updating or deleting a stack on AWS a lot of different things can go wrong, to name a few: - Permissions - Missing resources - Missing files Before when we were deploying a new cloudformation template we only displayed information when running in debug mode internal log statements and the final status from cloudformation. This was actually hidding all the work that was happening behind the scene and when needed to debug something we were required to log into the UI and inspect manually the stack event. This commit changes the behavior and will instead starts a goroutine that will listen to events coming from the DescribeStackEvent's API. The API is not the most flexible to work with, since there are no way to specify at what point in time we are interested in receiving events and we cannot say give me all the events from a specific Stack Change. So instead we retrieve the list of events and skip through it until we position ourself to when a new stack event is created. Functionbeat stack events are small so it doesn't take too much time to position ourself. After we periodically poll the events and keep track of pagination, any already seen events are ignored and everything else is printed to the console in Info mode. When you are using one of the deploy, update or delete subcommand and if the stack fail you will get a stack failed error but you will be able to inspect the log and find where and why it fails. Example output in verbose: ``` 2018-12-06T14:49:14.855-0500 INFO instance/beat.go:597 Home path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat] Config path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat] Data path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat/data] Logs path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat/logs] 2018-12-06T14:49:14.855-0500 INFO instance/beat.go:604 Beat UUID: 693706c5-688b-4673-a8af-e0d931894fdc 2018-12-06T14:49:28.292-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::CloudFormation::Stack, LogicalResourceId: fnb-mylogs-stack, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: User Initiated 2018-12-06T14:49:32.405-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:32.405-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:32.405-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:34.465-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:34.465-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:49:44.804-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:49:46.861-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:48.927-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:50.982-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:57.197-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:57.197-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:50:03.403-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:50:03.403-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_COMPLETE Function: mylogs, deploy successful ``` Fixes: elastic#8912
* Display CloudFormation StackEvent in the CLI When creating, updating or deleting a stack on AWS a lot of different things can go wrong, to name a few: - Permissions - Missing resources - Missing files Before when we were deploying a new cloudformation template we only displayed information when running in debug mode internal log statements and the final status from cloudformation. This was actually hidding all the work that was happening behind the scene and when needed to debug something we were required to log into the UI and inspect manually the stack event. This commit changes the behavior and will instead starts a goroutine that will listen to events coming from the DescribeStackEvent's API. The API is not the most flexible to work with, since there are no way to specify at what point in time we are interested in receiving events and we cannot say give me all the events from a specific Stack Change. So instead we retrieve the list of events and skip through it until we position ourself to when a new stack event is created. Functionbeat stack events are small so it doesn't take too much time to position ourself. After we periodically poll the events and keep track of pagination, any already seen events are ignored and everything else is printed to the console in Info mode. When you are using one of the deploy, update or delete subcommand and if the stack fail you will get a stack failed error but you will be able to inspect the log and find where and why it fails. Example output in verbose: ``` 2018-12-06T14:49:14.855-0500 INFO instance/beat.go:597 Home path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat] Config path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat] Data path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat/data] Logs path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat/logs] 2018-12-06T14:49:14.855-0500 INFO instance/beat.go:604 Beat UUID: 693706c5-688b-4673-a8af-e0d931894fdc 2018-12-06T14:49:28.292-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::CloudFormation::Stack, LogicalResourceId: fnb-mylogs-stack, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: User Initiated 2018-12-06T14:49:32.405-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:32.405-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:32.405-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:34.465-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:34.465-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:49:44.804-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:49:46.861-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:48.927-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:50.982-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:57.197-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:57.197-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:50:03.403-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:50:03.403-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_COMPLETE Function: mylogs, deploy successful ``` Fixes: #8912
* Display CloudFormation StackEvent in the CLI When creating, updating or deleting a stack on AWS a lot of different things can go wrong, to name a few: - Permissions - Missing resources - Missing files Before when we were deploying a new cloudformation template we only displayed information when running in debug mode internal log statements and the final status from cloudformation. This was actually hidding all the work that was happening behind the scene and when needed to debug something we were required to log into the UI and inspect manually the stack event. This commit changes the behavior and will instead starts a goroutine that will listen to events coming from the DescribeStackEvent's API. The API is not the most flexible to work with, since there are no way to specify at what point in time we are interested in receiving events and we cannot say give me all the events from a specific Stack Change. So instead we retrieve the list of events and skip through it until we position ourself to when a new stack event is created. Functionbeat stack events are small so it doesn't take too much time to position ourself. After we periodically poll the events and keep track of pagination, any already seen events are ignored and everything else is printed to the console in Info mode. When you are using one of the deploy, update or delete subcommand and if the stack fail you will get a stack failed error but you will be able to inspect the log and find where and why it fails. Example output in verbose: ``` 2018-12-06T14:49:14.855-0500 INFO instance/beat.go:597 Home path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat] Config path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat] Data path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat/data] Logs path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat/logs] 2018-12-06T14:49:14.855-0500 INFO instance/beat.go:604 Beat UUID: 693706c5-688b-4673-a8af-e0d931894fdc 2018-12-06T14:49:28.292-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::CloudFormation::Stack, LogicalResourceId: fnb-mylogs-stack, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: User Initiated 2018-12-06T14:49:32.405-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:32.405-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:32.405-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:34.465-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:34.465-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:49:44.804-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:49:46.861-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:48.927-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:50.982-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:57.197-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:57.197-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:50:03.403-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:50:03.403-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_COMPLETE Function: mylogs, deploy successful ``` Fixes: elastic#8912 (cherry picked from commit 54314dd)
#9518) Cherry-pick of PR #9409 to 6.x branch. Original message: When creating, updating or deleting a stack on AWS a lot of different things can go wrong, to name a few: - Permissions - Missing resources - Missing files Before when we were deploying a new cloudformation template we only displayed information when running in debug mode internal log statements and the final status from cloudformation. This was actually hidding all the work that was happening behind the scene and when needed to debug something we were required to log into the UI and inspect manually the stack event. This commit changes the behavior and will instead starts a goroutine that will listen to events coming from the DescribeStackEvent's API. The API is not the most flexible to work with, since there are no way to specify at what point in time we are interested in receiving events and we cannot say give me all the events from a specific Stack Change. So instead we retrieve the list of events and skip through it until we position ourself to when a new stack event is created. Functionbeat stack events are small so it doesn't take too much time to position ourself. After we periodically poll the events and keep track of pagination, any already seen events are ignored and everything else is printed to the console in Info mode. When you are using one of the deploy, update or delete subcommand and if the stack fail you will get a stack failed error but you will be able to inspect the log and find where and why it fails. Example output in verbose: ``` 2018-12-06T14:49:14.855-0500 INFO instance/beat.go:597 Home path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat] Config path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat] Data path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat/data] Logs path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat/logs] 2018-12-06T14:49:14.855-0500 INFO instance/beat.go:604 Beat UUID: 693706c5-688b-4673-a8af-e0d931894fdc 2018-12-06T14:49:28.292-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::CloudFormation::Stack, LogicalResourceId: fnb-mylogs-stack, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: User Initiated 2018-12-06T14:49:32.405-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:32.405-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:32.405-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:34.465-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:34.465-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:49:44.804-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:49:46.861-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:48.927-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:50.982-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:57.197-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:57.197-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:50:03.403-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:50:03.403-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_COMPLETE Function: mylogs, deploy successful ``` Fixes: #8912
* Display CloudFormation StackEvent in the CLI When creating, updating or deleting a stack on AWS a lot of different things can go wrong, to name a few: - Permissions - Missing resources - Missing files Before when we were deploying a new cloudformation template we only displayed information when running in debug mode internal log statements and the final status from cloudformation. This was actually hidding all the work that was happening behind the scene and when needed to debug something we were required to log into the UI and inspect manually the stack event. This commit changes the behavior and will instead starts a goroutine that will listen to events coming from the DescribeStackEvent's API. The API is not the most flexible to work with, since there are no way to specify at what point in time we are interested in receiving events and we cannot say give me all the events from a specific Stack Change. So instead we retrieve the list of events and skip through it until we position ourself to when a new stack event is created. Functionbeat stack events are small so it doesn't take too much time to position ourself. After we periodically poll the events and keep track of pagination, any already seen events are ignored and everything else is printed to the console in Info mode. When you are using one of the deploy, update or delete subcommand and if the stack fail you will get a stack failed error but you will be able to inspect the log and find where and why it fails. Example output in verbose: ``` 2018-12-06T14:49:14.855-0500 INFO instance/beat.go:597 Home path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat] Config path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat] Data path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat/data] Logs path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat/logs] 2018-12-06T14:49:14.855-0500 INFO instance/beat.go:604 Beat UUID: 693706c5-688b-4673-a8af-e0d931894fdc 2018-12-06T14:49:28.292-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::CloudFormation::Stack, LogicalResourceId: fnb-mylogs-stack, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: User Initiated 2018-12-06T14:49:32.405-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:32.405-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:32.405-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:34.465-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:34.465-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:49:44.804-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:49:46.861-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:48.927-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:50.982-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:57.197-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:57.197-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:50:03.403-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:50:03.403-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_COMPLETE Function: mylogs, deploy successful ``` Fixes: elastic#8912 (cherry picked from commit 54314dd)
* Display CloudFormation StackEvent in the CLI When creating, updating or deleting a stack on AWS a lot of different things can go wrong, to name a few: - Permissions - Missing resources - Missing files Before when we were deploying a new cloudformation template we only displayed information when running in debug mode internal log statements and the final status from cloudformation. This was actually hidding all the work that was happening behind the scene and when needed to debug something we were required to log into the UI and inspect manually the stack event. This commit changes the behavior and will instead starts a goroutine that will listen to events coming from the DescribeStackEvent's API. The API is not the most flexible to work with, since there are no way to specify at what point in time we are interested in receiving events and we cannot say give me all the events from a specific Stack Change. So instead we retrieve the list of events and skip through it until we position ourself to when a new stack event is created. Functionbeat stack events are small so it doesn't take too much time to position ourself. After we periodically poll the events and keep track of pagination, any already seen events are ignored and everything else is printed to the console in Info mode. When you are using one of the deploy, update or delete subcommand and if the stack fail you will get a stack failed error but you will be able to inspect the log and find where and why it fails. Example output in verbose: ``` 2018-12-06T14:49:14.855-0500 INFO instance/beat.go:597 Home path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat] Config path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat] Data path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat/data] Logs path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat/logs] 2018-12-06T14:49:14.855-0500 INFO instance/beat.go:604 Beat UUID: 693706c5-688b-4673-a8af-e0d931894fdc 2018-12-06T14:49:28.292-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::CloudFormation::Stack, LogicalResourceId: fnb-mylogs-stack, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: User Initiated 2018-12-06T14:49:32.405-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:32.405-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:32.405-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:34.465-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:34.465-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:49:44.804-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:49:46.861-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:48.927-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:50.982-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:57.197-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:57.197-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:50:03.403-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:50:03.403-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_COMPLETE Function: mylogs, deploy successful ``` Fixes: #8912 (cherry picked from commit 54314dd)
…9526) * Display CloudFormation StackEvent in the CLI When creating, updating or deleting a stack on AWS a lot of different things can go wrong, to name a few: - Permissions - Missing resources - Missing files Before when we were deploying a new cloudformation template we only displayed information when running in debug mode internal log statements and the final status from cloudformation. This was actually hidding all the work that was happening behind the scene and when needed to debug something we were required to log into the UI and inspect manually the stack event. This commit changes the behavior and will instead starts a goroutine that will listen to events coming from the DescribeStackEvent's API. The API is not the most flexible to work with, since there are no way to specify at what point in time we are interested in receiving events and we cannot say give me all the events from a specific Stack Change. So instead we retrieve the list of events and skip through it until we position ourself to when a new stack event is created. Functionbeat stack events are small so it doesn't take too much time to position ourself. After we periodically poll the events and keep track of pagination, any already seen events are ignored and everything else is printed to the console in Info mode. When you are using one of the deploy, update or delete subcommand and if the stack fail you will get a stack failed error but you will be able to inspect the log and find where and why it fails. Example output in verbose: ``` 2018-12-06T14:49:14.855-0500 INFO instance/beat.go:597 Home path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat] Config path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat] Data path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat/data] Logs path: [/Users/ph/go/src/github.com/elastic/beats/x-pack/functionbeat/logs] 2018-12-06T14:49:14.855-0500 INFO instance/beat.go:604 Beat UUID: 693706c5-688b-4673-a8af-e0d931894fdc 2018-12-06T14:49:28.292-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::CloudFormation::Stack, LogicalResourceId: fnb-mylogs-stack, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: User Initiated 2018-12-06T14:49:32.405-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:32.405-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:32.405-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:34.465-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:34.465-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbmylogsLogGroup, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:49:44.804-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: IAMRoleLambdaExecution, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:49:46.861-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:48.927-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:50.982-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbmylogs, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdafilter, ResourceStatus: CREATE_IN_PROGRESS 2018-12-06T14:49:53.054-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:57.197-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated 2018-12-06T14:49:57.197-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbmylogsSubscriptionFilterawslambdaall, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:50:03.403-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission1, ResourceStatus: CREATE_COMPLETE 2018-12-06T14:50:03.403-0500 INFO [aws] aws/op_cloudformation.go:106 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbmylogsPermission0, ResourceStatus: CREATE_COMPLETE Function: mylogs, deploy successful ``` Fixes: elastic#8912 (cherry picked from commit c7d013d)
Currently, we only wait for Completed or Failed events for cloudformation; it's not the best thing for debugging. Instead, we should also monitor Stack Events and report them to the console as they appear.
This way it makes it a lot easier to debug why a stack failed to apply or not.
The text was updated successfully, but these errors were encountered: