CIS scans throw panic on kind clusters frequently

[jzvmw]

We noticed CIS scans failed to run on kind clusters in our tests pretty frequently recently. We found the following logs in sonobuoy-kube-bench-master-daemon-set pods that seem like sonobuoy threw panic when it began to run the scans. The scans run on Sonobuoy 1.17 and 1.16.3 kind clusters

Logs
runtime: mlock of signal stack failed: 12 runtime: increase the mlock limit (ulimit -l) or runtime: update your kernel to 5.3.15+, 5.4.2+, or 5.5+ fatal error: mlock failed runtime stack: runtime.throw(0x9c4bde, 0xc) /usr/local/go/src/runtime/panic.go:1112 +0x72 runtime.mlockGsignal(0xc000304300) /usr/local/go/src/runtime/os_linux_x86.go:72 +0x107 runtime.mpreinit(0xc000234700) /usr/local/go/src/runtime/os_linux.go:341 +0x78 runtime.mcommoninit(0xc000234700) /usr/local/go/src/runtime/proc.go:630 +0x108 runtime.allocm(0xc000051000, 0x9eb858, 0x0) /usr/local/go/src/runtime/proc.go:1390 +0x14e runtime.newm(0x9eb858, 0xc000051000) /usr/local/go/src/runtime/proc.go:1704 +0x39 runtime.startm(0x0, 0xc000107301) /usr/local/go/src/runtime/proc.go:1869 +0x12a runtime.wakep(...) /usr/local/go/src/runtime/proc.go:1953 runtime.resetspinning() /usr/local/go/src/runtime/proc.go:2415 +0x93 runtime.schedule() /usr/local/go/src/runtime/proc.go:2527 +0x2de runtime.mstart1() /usr/local/go/src/runtime/proc.go:1104 +0x8e runtime.mstart() /usr/local/go/src/runtime/proc.go:1062 +0x6e goroutine 1 [syscall]: syscall.Syscall(0x3, 0xc, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/syscall/asm_linux_amd64.s:18 +0x5 syscall.Close(0xc, 0xc00000d820, 0x4) /usr/local/go/src/syscall/zsyscall_linux_amd64.go:285 +0x40 syscall.forkExec(0x9c1fb7, 0x7, 0xc0002c0930, 0x3, 0x3, 0xc0003a1190, 0x45, 0x46283ba300000400, 0xc00047b000) /usr/local/go/src/syscall/exec_unix.go:209 +0x39f syscall.StartProcess(...) /usr/local/go/src/syscall/exec_unix.go:248 os.startProcess(0x9c1fb7, 0x7, 0xc0002c0930, 0x3, 0x3, 0xc0003a1328, 0x0, 0x0, 0x0) /usr/local/go/src/os/exec_posix.go:52 +0x2c0 os.StartProcess(0x9c1fb7, 0x7, 0xc0002c0930, 0x3, 0x3, 0xc0003a1328, 0x45, 0x0, 0x203000) /usr/local/go/src/os/exec.go:102 +0x7c os/exec.(*Cmd).Start(0xc00053ab00, 0x503801, 0xc000120cd0) /usr/local/go/src/os/exec/exec.go:417 +0x50c os/exec.(*Cmd).Run(0xc00053ab00, 0xc000120cd0, 0x2) /usr/local/go/src/os/exec/exec.go:337 +0x2b os/exec.(*Cmd).Output(0xc00053ab00, 0x7, 0xc0003a1480, 0x2, 0x2, 0xc00053ab00) /usr/local/go/src/os/exec/exec.go:541 +0x88 github.com/aquasecurity/kube-bench/check.isShellCommand(0xc0004ec380, 0x9, 0xe3c401) /go/src/github.com/aquasecurity/kube-bench/check/check.go:253 +0xf9 github.com/aquasecurity/kube-bench/check.runExecCommands(0xc000023740, 0x30, 0xc00012f460, 0x3, 0x4, 0xc0002c0780, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/aquasecurity/kube-bench/check/check.go:290 +0x84 github.com/aquasecurity/kube-bench/check.performTest(0xc000023740, 0x30, 0xc00012f460, 0x3, 0x4, 0xc000526b10, 0x0, 0x0, 0xc0002c06c0, 0x0, ...) /go/src/github.com/aquasecurity/kube-bench/check/check.go:270 +0xbd github.com/aquasecurity/kube-bench/check.(*Check).run(0xc000529000, 0xc0003a1948, 0xc000108f80) /go/src/github.com/aquasecurity/kube-bench/check/check.go:133 +0x219 github.com/aquasecurity/kube-bench/check.(*defaultRunner).Run(0xe3b458, 0xc000529000, 0x1, 0x3) /go/src/github.com/aquasecurity/kube-bench/check/check.go:100 +0x2b github.com/aquasecurity/kube-bench/check.(*Controls).RunChecks(0xc00002c480, 0xa8ce00, 0xe3b458, 0xc000108f80, 0x101, 0xc000108f80, 0x0, 0x0) /go/src/github.com/aquasecurity/kube-bench/check/controls.go:101 +0x19e github.com/aquasecurity/kube-bench/cmd.runChecks(0xc00024d7ec, 0x6, 0xc00024d7e0, 0x17) /go/src/github.com/aquasecurity/kube-bench/cmd/common.go:120 +0x68e github.com/aquasecurity/kube-bench/cmd.run(0xc000258260, 0x1, 0x1, 0xc000206e60, 0x7, 0xc000206e01, 0x7) /go/src/github.com/aquasecurity/kube-bench/cmd/run.go:67 +0x1e8 github.com/aquasecurity/kube-bench/cmd.glob..func4(0xe065e0, 0xc000232090, 0x0, 0x9) /go/src/github.com/aquasecurity/kube-bench/cmd/run.go:49 +0x362 github.com/spf13/cobra.(*Command).execute(0xe065e0, 0xc000232000, 0x9, 0x9, 0xe065e0, 0xc000232000) /go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:766 +0x29d github.com/spf13/cobra.(*Command).ExecuteC(0xe06f60, 0xe3b458, 0x0, 0x0) /go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:852 +0x2ea github.com/spf13/cobra.(*Command).Execute(...) /go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:800 github.com/aquasecurity/kube-bench/cmd.Execute() /go/src/github.com/aquasecurity/kube-bench/cmd/root.go:115 +0x55 main.main() /go/src/github.com/aquasecurity/kube-bench/main.go:22 +0x20 goroutine 18 [chan receive]: github.com/golang/glog.(*loggingT).flushDaemon(0xe109a0) /go/pkg/mod/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:882 +0x8b created by github.com/golang/glog.init.0 /go/pkg/mod/github.com/golang/glog@v0.0.0-20160126235308-23def4e6c14b/glog.go:410 +0x26f Sleeping for 1h to avoid daemonset restart

kind version: v1.16.3
sonobuoy version: we are using github.com/zubron/sonobuoy v1.11.5-prerelease.1.0.20200706195956-8ef2fd901589 because of some dependency reasons

[vladimirvivien]

@jzvmw After researching this error it looks it may have to do with Go 1.14 (as outlined here) possibly which was used to build the current version of this plugin. As stated below in the runtime error:

runtime: mlock of signal stack failed: 12 runtime: increase the mlock limit (ulimit -l) or runtime: update your kernel to 5.3.15+, 5.4.2+, or 5.5+ fatal error: mlock failed runtime stack

You can try to change the ulimit to see if that helps. In the mean times, I am going to open a PR to update the the version of kube-bench project to use the latest image which uses Go 1.15.