Cluster-global http-add-on: scale-up from zero doesn't work

[morganchristiansson]

Report

First try running http-add-on. Scaling deployment/redmine in namespace redmine with keda in namespace keda .

Getting error from keda-operator below. It's trying to connect to keda-add-ons-http-external-scaler.redmine.svc.cluster.local which exists in namespace keda. It's incorrectly using redmine namespace that HTTPScaledObject is in.

(I also created empty configmap/keda-http-routing-table in redmine ns. Not sure it's relevant.)

BTW http-add-on chart is missing rbac configmaps patch. Will create PR soon. kedacore/charts@main...morganchristiansson:fix-http-add-on

Expected Behavior

scale-up from zero should work

Actual Behavior

503 service unavailable from nginx-ingress - interceptor is not receiving requests

Steps to Reproduce the Problem

1. Deployed keda chart (in keda ns)
2. Deployed keda http-add-on chart with images.tag: canary
3. Create HTTPScaledObject for redmine in redmine ns

Logs from KEDA HTTP operator

$ kubectl -n redmine get event
LAST SEEN   TYPE      REASON             OBJECT                     MESSAGE
3m29s       Warning   KEDAScalerFailed   scaledobject/redmine-app   rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup keda-add-ons-http-external-scaler.redmine.svc.cluster.local on 10.43.0.10:53: no such host"

NOTE: logs from keda-operator - not keda-http-operator

$ kubectl -n keda logs keda-operator-54bc565bf8-gs27c
<snip>
2022-01-17T03:56:56.692Z        ERROR   external_scaler error calling IsActive on external scaler       {"error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: lookup keda-add-ons-http-external-scaler.redmine.svc.cluster.local on 10.43.0.10:53: no such host\""}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
        /workspace/pkg/scaling/scale_handler.go:261
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
        /workspace/pkg/scaling/scale_handler.go:146

$ kubectl -n keda logs -f keda-add-ons-http-controller-manager-7f74dd98cd-vmj28 -c keda-add-ons-http-operator
<snip>
2022-01-15T04:46:30.645Z        INFO    controllers.HTTPScaledObject    Reconciliation start    {"HTTPScaledObject.Namespace": "redmine", "HTTPScaledObject.Name": "redmine"}
2022-01-15T04:46:30.645Z        INFO    controllers.HTTPScaledObject    Reconciling HTTPScaledObject    {"HTTPScaledObject.Namespace": "redmine", "HTTPScaledObject.Name": "redmine", "Namespace": "redmine", "DeploymentName": "redmine"}
2022-01-15T04:46:30.645Z        INFO    controllers.HTTPScaledObject    Creating scaled objects {"reconciler.appObjects": "addObjects", "HTTPScaledObject.name": "redmine", "HTTPScaledObject.namespace": "redmine", "external scaler host name": "keda-add-ons-http-external-scaler.keda:9090"}
2022-01-15T04:46:30.646Z        INFO    controllers.HTTPScaledObject    Creating App ScaledObject       {"reconciler.appObjects": "addObjects", "HTTPScaledObject.name": "redmine", "HTTPScaledObject.namespace": "redmine", "ScaledObject": {"Object":{"apiVersion":"keda.sh/v1alpha1","kind":"ScaledObject","metadata":{"labels":{"app":"kedahttp-redmine-app","name":"redmine-app"},"name":"redmine-app","namespace":"redmine"},"spec":{"advanced":{"restoreToOriginalReplicaCount":true},"maxReplicaCount":1,"minReplicaCount":0,"pollingInterval":1,"scaleTargetRef":{"kind":"Deployment","name":"redmine"},"triggers":[{"metadata":{"host":"redmine.int.mog.se","scalerAddress":"keda-add-ons-http-external-scaler.keda:9090"},"type":"external-push"}]}}}}
2022-01-15T04:46:30.671Z        INFO    controllers.HTTPScaledObject    User app scaled object already exists, moving on        {"reconciler.appObjects": "addObjects", "HTTPScaledObject.name": "redmine", "HTTPScaledObject.namespace": "redmine"}
2022-01-15T04:46:30.672Z        ERROR   controllers.HTTPScaledObject.addAndUpdateRoutingTable   could not add host to routing table, progressing anyway {"reconciler.appObjects": "addObjects", "HTTPScaledObject.name": "redmine", "HTTPScaledObject.namespace": "redmine", "host": "redmine.int.mog.se", "error": "host redmine.int.mog.se is already registered in the routing table"}
github.com/kedacore/http-add-on/operator/controllers.(*HTTPScaledObjectReconciler).Reconcile
        /go/src/github.com/kedacore/http-add-on/operator/controllers/httpscaledobject_controller.go:116
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.3/pkg/internal/controller/controller.go:114
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.3/pkg/internal/controller/controller.go:311
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.3/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.3/pkg/internal/controller/controller.go:227
2022-01-15T04:46:30.679Z        INFO    controllers.HTTPScaledObject    Updating status on HTTPScaledObject     {"HTTPScaledObject.Namespace": "redmine", "HTTPScaledObject.Name": "redmine", "resource version": "123771734"}
2022-01-15T04:46:30.699Z        INFO    controllers.HTTPScaledObject    Updated status on HTTPScaledObject      {"HTTPScaledObject.Namespace": "redmine", "HTTPScaledObject.Name": "redmine", "resource version": "123771736"}
2022-01-15T04:46:30.699Z        INFO    controllers.HTTPScaledObject    Updating status on HTTPScaledObject     {"HTTPScaledObject.Namespace": "redmine", "HTTPScaledObject.Name": "redmine", "resource version": "123771736"}
2022-01-15T04:46:30.717Z        INFO    controllers.HTTPScaledObject    Updated status on HTTPScaledObject      {"HTTPScaledObject.Namespace": "redmine", "HTTPScaledObject.Name": "redmine", "resource version": "123771737"}
2022-01-15T04:46:30.717Z        INFO    controllers.HTTPScaledObject    Reconcile success       {"HTTPScaledObject.Namespace": "redmine", "HTTPScaledObject.Name": "redmine"}

What version of the KEDA HTTP Addon are you running?

http-add-on canary docker images with keda 2.5.0

Kubernetes Version

1.21

Platform

Other

Anything else?

Related to #269

[arschles]

@morganchristiansson thanks for this. the operator is what sets up the ScaledObject, and it uses the NAMESPACE environment variable for the hostname of the external scaler (this is changed from the behavior before #269). Is that value set to keda in your deployment?

Also, thanks for that patch! I'll look out for the PR.

[morganchristiansson]

NAMESPACE is not set, KEDA_HTTP_SCALER_TARGET_ADMIN_NAMESPACE is set. I checked both http-add-on chart deployment-scaler.yaml and pods running process environment.

After setting NAMESPACE in deployment/keda-add-ons-http-external-scaler it successfully scales up target, but fails to forward initial request.

kubectl -n keda port-forward svc/keda-add-ons-http-interceptor-proxy 8080:8080 &
curl -H 'Host: redmine.int.mog.se' localhost:8080
error on backend (context marked done while waiting for deployment redmine to reach > 0 replicas (context deadline exceeded))

I'm guessing NAMESPACE is needed in other pods too..... It's not intercepting requests from nginx-ingress...

[arschles]

@morganchristiansson what about KEDA_HTTP_OPERATOR_NAMESPACE? Is it possible to send me the YAML you used to deploy the operator, interceptor and scaler?

[arschles]

@morganchristiansson are you still interesting in this? If so, I'm happy to continue working to get it solved. Just let me know.

[morganchristiansson]

Sorry have been away and other things to focus on. After reinstalling with latest helm charts and default image tags I just got it all to work.

I did not immediately realize ingress should forward to serviceName: keda-add-ons-http-interceptor-proxy and be in keda namespace which caused scale up not to happen.

Thanks this is great.

[arschles]

@morganchristiansson great to hear. out of curiousity, what documentation did you read to get started? I want to make sure wherever you looked is updated to make it clear that the service name to which ingress should forward is keda-add-ons-http-interceptor-proxy so that the next person isn't confused

[morganchristiansson]

I was reading the walkthrough https://github.com/kedacore/http-add-on/blob/main/docs/walkthrough.md it does mention ingressNamespace which sets namespace on ingress resource in xkcd example chart. Most charts do not support providing servicename or namespace to ingress. So I'm creating ingress outside the chart.

[morganchristiansson]

Also was it considered to create proxy-interceptor pod per HTTPScaledObject with labels from service selector.

Then ingress and service selector forward to proxy-interceptor pod per service selector
Don't need routing-table configmap - each proxy-interceptor belongs to HTTPScaledObject.
Don't need Host: header - can handle any protocol.
proxy-interceptor pod only needed when scaled to zero. Can terminate when app is scaled up. All requests go directly to app pods per service.

Naively this seems like a good idea. Anyhow it's working fine as is. I guess you loose out on monitoring active requests to discover that it's idle and will scale to zero.

[arschles]

Thanks @morganchristiansson !