Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Segmentation violation #3811

Closed
tomasgiden opened this issue Nov 3, 2022 · 0 comments · Fixed by #3812
Closed

Segmentation violation #3811

tomasgiden opened this issue Nov 3, 2022 · 0 comments · Fixed by #3812
Assignees
Labels
bug Something isn't working

Comments

@tomasgiden
Copy link
Contributor

Report

Starting up, Keda Operator has a segmentation violation in logger. Presumably because Azure Blob Scaler is missing to store the initialized logger on line 70 in azure_blob_scaler.go. I will contribute a PR and link to this issue.

Expected Behavior

I expect Keda Operator to not have a segmentation violation. If my scaler credentials are bad, I expect Keda to handle it gracefully.

Actual Behavior

Keda crashes cyclically.

Steps to Reproduce the Problem

  1. Add a ScaledJob with azure blob scaling with something wrong in the credentials to blob storage (I presume)
  2. Sit back and watch keda operator crash

Logs from KEDA operator

LogEntry
" /workspace/pkg/scaling/scale_handler.go:108 +0x465"
created by github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).HandleScalableObject
" /workspace/pkg/scaling/scale_handler.go:149 +0x31c"
"github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop(0xc000863650, {0x3cae988, 0xc0011982c0}, 0xc0004e9900, {0x34ee760, 0xc0011a4240}, {0x3c8b6e8, 0xc000e7ff68})"
" /workspace/pkg/scaling/scale_handler.go:286 +0x317"
"github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers(0xc000863650, {0x3cae988, 0xc0011982c0}, {0x34ee760, 0xc0011a4240}, {0x3c8b6e8, 0xc000e7ff68})"
" /workspace/pkg/scaling/cache/scalers_cache.go:125 +0xda"
"github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).IsScaledJobActive(0xc00004dd10, {0x3cae988, 0xc0011982c0}, 0xc0011a4240)"
" /workspace/pkg/scaling/cache/scalers_cache.go:263 +0x332"
"github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).getScaledJobMetrics(0xc0002dc690, {0x3cae988, 0xc0011982c0}, 0xc0011a4240)"
" /workspace/pkg/scalers/azure_blob_scaler.go:193 +0xa5"
"github.com/kedacore/keda/v2/pkg/scalers.(*azureBlobScaler).IsActive(0xc0011a21e0, {0x3cae988, 0xc0011982c0})"
" /go/pkg/mod/github.com/go-logr/logr@v1.2.3/logr.go:279 +0x80"
"github.com/go-logr/logr.Logger.Error({{0x0, 0x0}, 0x35a828d}, {0x3c59720, 0xc000412dc0}, {0x35a49bb, 0x6}, {0x0, 0x0, 0x0})"
goroutine 350 [running]:

[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x585a40]
panic: runtime error: invalid memory address or nil pointer dereference
"2022-11-03T17:37:27Z INFO Initializing Scaling logic according to ScaledJob Specification {""controller"": ""scaledjob"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""ScaledJob"", ""scaledJob"": {""name"":""reconstruction"",""namespace"":""app-env-dtu""}, ""namespace"": ""app-env-dtu"", ""name"": ""reconstruction"", ""reconcileID"": ""386dff46-d3f6-4e03-99bc-a2d86ca1a35d""}"
"2022-11-03T17:37:27Z INFO RolloutStrategy: gradual, Not deleting jobs owned by the previous version of the scaleJob {""controller"": ""scaledjob"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""ScaledJob"", ""scaledJob"": {""name"":""reconstruction"",""namespace"":""app-env-dtu""}, ""namespace"": ""app-env-dtu"", ""name"": ""reconstruction"", ""reconcileID"": ""386dff46-d3f6-4e03-99bc-a2d86ca1a35d""}"
"2022-11-03T17:37:27Z INFO Reconciling ScaledJob {""controller"": ""scaledjob"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""ScaledJob"", ""scaledJob"": {""name"":""reconstruction"",""namespace"":""app-env-dtu""}, ""namespace"": ""app-env-dtu"", ""name"": ""reconstruction"", ""reconcileID"": ""386dff46-d3f6-4e03-99bc-a2d86ca1a35d""}"
"2022-11-03T17:37:27Z INFO Initializing Scaling logic according to ScaledJob Specification {""controller"": ""scaledjob"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""ScaledJob"", ""scaledJob"": {""name"":""preprocessing"",""namespace"":""app-env-dtu""}, ""namespace"": ""app-env-dtu"", ""name"": ""preprocessing"", ""reconcileID"": ""6e92f8cf-d4c8-43f2-848a-5060ef1dbeae""}"
"2022-11-03T17:37:27Z INFO RolloutStrategy: gradual, Not deleting jobs owned by the previous version of the scaleJob {""controller"": ""scaledjob"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""ScaledJob"", ""scaledJob"": {""name"":""preprocessing"",""namespace"":""app-env-dtu""}, ""namespace"": ""app-env-dtu"", ""name"": ""preprocessing"", ""reconcileID"": ""6e92f8cf-d4c8-43f2-848a-5060ef1dbeae""}"
"2022-11-03T17:37:27Z INFO Reconciling ScaledJob {""controller"": ""scaledjob"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""ScaledJob"", ""scaledJob"": {""name"":""preprocessing"",""namespace"":""app-env-dtu""}, ""namespace"": ""app-env-dtu"", ""name"": ""preprocessing"", ""reconcileID"": ""6e92f8cf-d4c8-43f2-848a-5060ef1dbeae""}"
"2022-11-03T17:37:27Z INFO RolloutStrategy: gradual, Not deleting jobs owned by the previous version of the scaleJob {""controller"": ""scaledjob"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""ScaledJob"", ""scaledJob"": {""name"":""big-reconstruction"",""namespace"":""app-env-dtu""}, ""namespace"": ""app-env-dtu"", ""name"": ""big-reconstruction"", ""reconcileID"": ""ba3f1142-a3bd-4c3d-a1c7-a07b1e103dc9""}"
"2022-11-03T17:37:27Z INFO Reconciling ScaledJob {""controller"": ""scaledjob"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""ScaledJob"", ""scaledJob"": {""name"":""big-reconstruction"",""namespace"":""app-env-dtu""}, ""namespace"": ""app-env-dtu"", ""name"": ""big-reconstruction"", ""reconcileID"": ""ba3f1142-a3bd-4c3d-a1c7-a07b1e103dc9""}"
"2022-11-03T17:37:27Z INFO Starting workers {""controller"": ""triggerauthentication"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""TriggerAuthentication"", ""worker count"": 1}"
"2022-11-03T17:37:27Z INFO Starting workers {""controller"": ""scaledobject"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""ScaledObject"", ""worker count"": 5}"
"2022-11-03T17:37:27Z INFO Starting workers {""controller"": ""clustertriggerauthentication"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""ClusterTriggerAuthentication"", ""worker count"": 1}"
"2022-11-03T17:37:27Z INFO Starting workers {""controller"": ""scaledjob"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""ScaledJob"", ""worker count"": 1}"
"2022-11-03T17:37:27Z INFO KubeAPIWarningLogger autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+  use autoscaling/v2 HorizontalPodAutoscaler"
"2022-11-03T17:37:27Z INFO Starting Controller {""controller"": ""clustertriggerauthentication"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""ClusterTriggerAuthentication""}"
"2022-11-03T17:37:27Z INFO Starting EventSource {""controller"": ""clustertriggerauthentication"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""ClusterTriggerAuthentication"", ""source"": ""kind source: *v1alpha1.ClusterTriggerAuthentication""}"
"2022-11-03T17:37:27Z INFO Starting Controller {""controller"": ""scaledjob"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""ScaledJob""}"
"2022-11-03T17:37:27Z INFO Starting EventSource {""controller"": ""scaledjob"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""ScaledJob"", ""source"": ""kind source: *v1alpha1.ScaledJob""}"
"2022-11-03T17:37:27Z INFO Starting Controller {""controller"": ""triggerauthentication"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""TriggerAuthentication""}"
"2022-11-03T17:37:27Z INFO Starting EventSource {""controller"": ""triggerauthentication"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""TriggerAuthentication"", ""source"": ""kind source: *v1alpha1.TriggerAuthentication""}"
"2022-11-03T17:37:27Z INFO Starting Controller {""controller"": ""scaledobject"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""ScaledObject""}"
"2022-11-03T17:37:27Z INFO Starting EventSource {""controller"": ""scaledobject"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""ScaledObject"", ""source"": ""kind source: *v2beta2.HorizontalPodAutoscaler""}"
"2022-11-03T17:37:27Z INFO Starting EventSource {""controller"": ""scaledobject"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""ScaledObject"", ""source"": ""kind source: *v1alpha1.ScaledObject""}"
I1103 17:37:27.219554       1 leaderelection.go:258] successfully acquired lease keda/operator.keda.sh
"2022-11-03T17:37:27Z INFO Initializing Scaling logic according to ScaledJob Specification {""controller"": ""scaledjob"", ""controllerGroup"": ""keda.sh"", ""controllerKind"": ""ScaledJob"", ""scaledJob"": {""name"":""big-reconstruction"",""namespace"":""app-env-dtu""}, ""namespace"": ""app-env-dtu"", ""name"": ""big-reconstruction"", ""reconcileID"": ""ba3f1142-a3bd-4c3d-a1c7-a07b1e103dc9""}"
"2022-11-03T17:37:11Z INFO Starting server {""path"": ""/metrics"", ""kind"": ""metrics"", ""addr"": ""[::]:8080""}"
"2022-11-03T17:37:11Z INFO Starting server {""kind"": ""health probe"", ""addr"": ""[::]:8081""}"
I1103 17:37:11.962951       1 leaderelection.go:248] attempting to acquire leader lease keda/operator.keda.sh...
2022-11-03T17:37:11Z INFO setup Go OS/Arch: linux/amd64
2022-11-03T17:37:11Z INFO setup Go Version: go1.17.13
2022-11-03T17:37:11Z INFO setup Git Commit: a4a118201214e7abdeebad72cbe337b9856f8191
2022-11-03T17:37:11Z INFO setup KEDA Version: 2.8.0
2022-11-03T17:37:11Z INFO setup Starting manager
"2022-11-03T17:37:11Z INFO setup Running on Kubernetes 1.24 {""version"": ""v1.24.6""}"
"2022-11-03T17:37:11Z INFO controller-runtime.metrics Metrics server is starting to listen {""addr"": "":8080""}"```


### KEDA Version

2.8.0

### Kubernetes Version

1.24

### Platform

Microsoft Azure

### Scaler Details

Azube Blob Scaler

### Anything else?

_No response_
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

1 participant