Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Go lang bug causes controller to CrashLoopBackoff #411

Closed
cyrilcros opened this issue May 18, 2020 · 1 comment · Fixed by #415
Closed

Go lang bug causes controller to CrashLoopBackoff #411

cyrilcros opened this issue May 18, 2020 · 1 comment · Fixed by #415
Labels
bug pr-merged
Milestone
v0.12.4

Comments

@cyrilcros
Copy link

cyrilcros commented May 18, 2020 •
edited
Loading

I updated to Ubuntu 20.04, and with SealedSecrets v0.12.3 / v0.12.1 I kept seeing CrashLoopBackOff errors from the controller.
The relevant part of the controller logs starts here:

runtime: mlock of signal stack failed: 12
runtime: increase the mlock limit (ulimit -l) or
runtime: update your kernel to 5.3.15+, 5.4.2+, or 5.5+
fatal error: mlock failed

runtime stack:
runtime.throw(0x139a45f, 0xc)
        /home/travis/.gimme/versions/go1.14.linux.amd64/src/runtime/panic.go:1112 +0x72
runtime.mlockGsignal(0xc00020b080)

My kernel is 5.4.0-29 - and by the way this what ubuntu currently ships - it seems to be related to this go-lang issue: golang/go#37436

Please consider updating your docker image go version to 1.14.2 or above. Thanks!

PS: I will add the controller still kind of works, it fails after a seemingly random amount of time.
@mkmik mkmik added the bug label May 25, 2020
@mkmik mkmik mentioned this issue May 25, 2020
Merged
@helpr helpr bot added the pr-available label May 25, 2020
bors bot added a commit that referenced this issue May 26, 2020
@bors
Merge #415
29441fc 
415: Fix travis Go version r=mkmik a=mkmik

Turns out all our released binaries are compiled with Go 1.14.0 because .travis.yaml
contains the `1.14` version spec which doesn't imply "latest version of 1.14 branch"

(I checked that on our released binaries following the instructions on https://dave.cheney.net/2017/06/20/how-to-find-out-which-go-version-built-your-binary)

A user reported an issue (#411) that is likely caused by a toolchain bug (golang/go#37436)
which is fixed in 1.14.1 (2020/03/19).

Several sealed-secret releases happened after that and we didn't inherit the fix.

Given the strong backwards compat guarantees the Go team commits to,
I think it's a good idea to just build with the latest version of the toolchain.

Closes #411

Co-authored-by: Marko Mikulicic <mkmik@vmware.com>
@bors bors bot closed this as completed in 63b08af May 26, 2020
@bors bors bot closed this as completed in #415 May 26, 2020
@helpr helpr bot added pr-merged and removed pr-available labels May 26, 2020
@mkmik mkmik added this to the v0.12.4 milestone May 26, 2020
@mkmik
Copy link
Collaborator

mkmik commented May 26, 2020

Rebuilt with Go 1.14.3 and released in https://github.com/bitnami-labs/sealed-secrets/releases/tag/v0.12.4

Closing this issue optimistically, please reopen if it's still a problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug pr-merged
Projects
None yet
Milestone
v0.12.4
Development

Successfully merging a pull request may close this issue.

Fix travis Go version bitnami-labs/sealed-secrets
2 participants
@mkmik @cyrilcros