Skip to content
This repository has been archived by the owner on Jun 29, 2022. It is now read-only.

Handle OS interrupts in lokoctl to fix leaking terraform process. #483

Merged
merged 3 commits into from
Jun 4, 2020

Conversation

ipochi
Copy link
Member

@ipochi ipochi commented May 26, 2020

Fixes #329

Registers a Signal handler for terraform process and components installation/deletion.

Before running the lokoctl cluster apply

ps ef | grep terraform | wc -l
0
module.packet-liger.data.ct_config.controller-install-ignitions[0]: Refreshing state...
module.worker-madrid.data.ct_config.install-ignitions: Refreshing state...
module.worker-madrid.packet_device.nodes[0]: Creating...
module.packet-liger.packet_device.controllers[0]: Creating...
module.worker-madrid.packet_device.nodes[0]: Still creating... [10s elapsed]
module.packet-liger.packet_device.controllers[0]: Still creating... [10s elapsed]
^CInterrupt received.
Please wait for Terraform to exit or data loss may occur.
Gracefully shutting down...
Stopping operation...

After Ctrl + C

ps ef | grep terraform | wc -l
0

@ipochi ipochi requested review from iaguis and invidian May 26, 2020 12:43
@ipochi ipochi force-pushed the imran/signal-handling branch 2 times, most recently from f765f95 to 09d7f89 Compare May 27, 2020 05:02
Copy link
Member

@invidian invidian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some thoughts, please have a look.

pkg/signals/signals.go Outdated Show resolved Hide resolved
pkg/signals/signals.go Outdated Show resolved Hide resolved
pkg/signals/signals.go Outdated Show resolved Hide resolved
pkg/signals/signals.go Outdated Show resolved Hide resolved
pkg/signals/signals.go Outdated Show resolved Hide resolved
pkg/signals/signals.go Outdated Show resolved Hide resolved
pkg/signals/signals.go Outdated Show resolved Hide resolved
pkg/signals/signals.go Outdated Show resolved Hide resolved
pkg/components/util/install.go Outdated Show resolved Hide resolved
iaguis
iaguis previously requested changes May 29, 2020
Copy link
Contributor

@iaguis iaguis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some partial review I did the other day.

pkg/signals/signals.go Outdated Show resolved Hide resolved
pkg/terraform/executor.go Outdated Show resolved Hide resolved
pkg/signals/signals.go Outdated Show resolved Hide resolved
pkg/signals/signals.go Outdated Show resolved Hide resolved
pkg/signals/signals.go Outdated Show resolved Hide resolved
pkg/signals/signals.go Outdated Show resolved Hide resolved
pkg/signals/signals.go Outdated Show resolved Hide resolved
@ipochi
Copy link
Member Author

ipochi commented Jun 3, 2020

After a call with @invidian we found out that terraform apply handles the interrupts but terraform init doesn't and this behavior is reflected the same in lokoctl as well.

An upstream issue has been filed for the same.

hashicorp/terraform#25118

@invidian @iaguis please re-review.

@ipochi ipochi force-pushed the imran/signal-handling branch from 09d7f89 to 0800591 Compare June 3, 2020 17:39
@ipochi ipochi requested review from iaguis and invidian June 3, 2020 17:40
@ipochi ipochi dismissed stale reviews from iaguis and invidian June 3, 2020 17:40

please re-review

@ipochi ipochi force-pushed the imran/signal-handling branch 2 times, most recently from 7530cbb to 84da10b Compare June 4, 2020 07:44
iaguis
iaguis previously requested changes Jun 4, 2020
Copy link
Contributor

@iaguis iaguis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have some nits and a general comment.

From my testing it seems when I interrupt lokoctl at, say, terraform init this doesn't stop lokoctl. I'm aware terraform init doesn't handle interrupts but I would expect lokoctl to stop after terraform init is finished instead of continuing with the next steps (terraform plan and so on). This works fine if I do two interrupts for example when terraform plan is running, I guess because the process ends with exit status 1 and we propagate the error.

Should we add a channel to send interrupt information from the signal handler to the Executor so we can os.Exit() or error out after the command is finished if we get an interrupt?

pkg/terraform/executor.go Outdated Show resolved Hide resolved
pkg/terraform/executor.go Outdated Show resolved Hide resolved
pkg/terraform/signals.go Show resolved Hide resolved
@ipochi
Copy link
Member Author

ipochi commented Jun 4, 2020

I have some nits and a general comment.

From my testing it seems when I interrupt lokoctl at, say, terraform init this doesn't stop lokoctl. I'm aware terraform init doesn't handle interrupts but I would expect lokoctl to stop after terraform init is finished instead of continuing with the next steps (terraform plan and so on). This works fine if I do two interrupts for example when terraform plan is running, I guess because the process ends with exit status 1 and we propagate the error.

Should we add a channel to send interrupt information from the signal handler to the Executor so we can os.Exit() or error out after the command is finished if we get an interrupt?

It would become quite messy, besides there would be two branches where it would be handle the differently if the current process is terraform version/init vs terraform plan/apply/refresh

Nevertheless, I am opening up a follow up issue that tracks this limitation and and will also link the upstream issue in the follow up issue.

- Fix issue #329, to handle OS interrupts.

- signalHandler returns an instance of handler, currently we only
  deal with the signal os.Interrupt.

- stop terminates listening of interrupts once the terraform process has
  released its resources by either successful exit for forced
  termination. This way there are no leaking terraform processes.

Signed-off-by: Imran Pochi <imran@kinvolk.io>
@ipochi ipochi force-pushed the imran/signal-handling branch from 84da10b to 9fbbd02 Compare June 4, 2020 13:05
ipochi added 2 commits June 4, 2020 18:38
- Registers signal handler for the terraform command.

Signed-off-by: Imran Pochi <imran@kinvolk.io>
Signed-off-by: Imran Pochi <imran@kinvolk.io>
@ipochi
Copy link
Member Author

ipochi commented Jun 4, 2020

Follow up issue created #555

Copy link
Member

@invidian invidian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ipochi ipochi dismissed iaguis’s stale review June 4, 2020 13:37

Addressed your feedback, follow up issue created, please re-review changes.

Copy link
Contributor

@iaguis iaguis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks!

@ipochi ipochi merged commit 7416b5f into master Jun 4, 2020
@ipochi ipochi deleted the imran/signal-handling branch June 4, 2020 13:50
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Leaking Terraform processes when terminating a cluster operation
3 participants