following documentation throws errors

[dariuszbz]

let's follow the steps (i put them into one list - your welcome)

install helm with choco

install choco

Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))

install helm

choco install kubernetes-helm

add keda chart

helm repo add kedacore https://kedacore.github.io/charts

make sure all charts in your repo are updated

helm repo update

make a new k8s namespace for keda

kubectl create namespace keda

install keda

helm install keda kedacore/keda --namespace keda

##install http addon
helm install http-add-on kedacore/keda-add-ons-http --namespace keda

clone repo https://github.com/kedacore/http-add-on

cd to the cloned folder

install test applicattion

helm install xkcd ./examples/xkcd -n keda

until now all is well. Great!

now you suggest:
kubectl create -f -n keda examples/v0.0.2/httpscaledobject.yaml

first error:
error: Unexpected args: [keda examples/v0.0.2/httpscaledobject.yaml]
See 'kubectl create -h' for help and examples

Let's fix that:
kubectl create -n keda -f ./examples/v0.0.2/httpscaledobject.yaml

and we get error:
error: error validating "./examples/v0.0.2/httpscaledobject.yaml": error validating data: ValidationError(HTTPScaledObject.spec): missing required field "host" in sh.keda.http.v1alpha1.HTTPScaledObject.spec; if you choose to ignore these errors, turn validation off with --validate=false

Let us try the latest version:

kubectl create -n keda -f ./examples/v0.2.0/httpscaledobject.yaml

but we get:
Error from server (AlreadyExists): error when creating "./examples/v0.2.0/httpscaledobject.yaml": httpscaledobjects.http.keda.sh "xkcd" already exists.

I'd assume it is unnecessary step, but how can we update scaler ? I'm not an expert so please: could you tell me how to update scaler to scale up/down on every X http requests.

following your documentation: https://github.com/kedacore/http-add-on/blob/main/docs/ref/v0.2.0/http_scaled_object.md
I was trying to add:

targetPendingRequests: X

under: spec.scaleTagetRef

as part of httpscaledobject.yaml

but when I try to deploy I got:
Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest: error validating "": error validating data: ValidationError(HTTPScaledObject.spec.scaleTargetRef): unknown field "targetPendingRequests" in sh.keda.http.v1alpha1.HTTPScaledObject.spec.scaleTargetRef

Thank You in advance!

[arschles]

Hi @dariuszbz - after you did the helm install xkcd step, you are actually done. I'll modify the documentation to be more clear that you don't have to do the kubectl create httpscaledobject.yaml step if you did the helm install xkcd one.

After the helm install xkcd command completed, does your app scale properly?

[dariuszbz]

Hello @arschles. Thank You for your prompt feedback. I don't know what is that mean "does your app scale properly" ? I can describe how it actually works.

Just after installation the xkcd pod doesn't exist.

I installed ingress:

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm install ingress-nginx ingress-nginx/ingress-nginx -n keda

and after calling:
curl -H "Host: myhost.com"

the xkcd Pod is created. btw: pod disappears after some time if there is no requests - and it is awesome :) !

What I want is to be able to setup scaling option. Let's say 10 http requests per pod. Is it possible, please? And if yes, could you describe how to do it, please.

Thank You!

[dariuszbz]

Hello again. I did some digging and I think I've found.

Consider you definition:
https://github.com/kedacore/http-add-on/blob/main/docs/ref/v0.2.0/http_scaled_object.md

kind: HTTPScaledObject
apiVersion: http.keda.sh/v1alpha1
metadata:
name: xkcd
spec:
host: "myhost.com"
scaleTargetRef:
deployment: xkcd
service: xkcd
port: 8080
targetPendingRequests: 100

and implementation:
kind: HTTPScaledObject
apiVersion: http.keda.sh/v1alpha1
metadata:
name: {{ include "xkcd.fullname" . }}
spec:
host: {{ .Values.host }}
scaleTargetRef:
deployment: {{ include "xkcd.fullname" . }}
service: {{ include "xkcd.fullname" . }}
port: 8080
replicas:
min: {{ .Values.autoscaling.http.minReplicas }}
max: {{ .Values.autoscaling.http.maxReplicas }}

and comparing to:
https://pkg.go.dev/github.com/kedacore/http-add-on@v0.2.0/operator/api/v1alpha1#HTTPScaledObjectSpec

type HTTPScaledObjectSpec struct {
// The host to route. All requests with this host in the "Host"
// header will be routed to the Service and Port specified
// in the scaleTargetRef
Host string json:"host"
// The name of the deployment to route HTTP requests to (and to autoscale). Either this
// or Image must be set
ScaleTargetRef *ScaleTargetRef json:"scaleTargetRef"
// (optional) Replica information
//+optional
Replicas ReplicaStruct json:"replicas,omitempty"
//(optional) Target metric value
TargetPendingRequests int32 json:"targetPendingRequests,omitempty" description:"The target metric value for the HPA (Default 100)"
}
doesn't match.

Implementation should be:
kind: HTTPScaledObject
apiVersion: http.keda.sh/v1alpha1
metadata:
name: {{ include "xkcd.fullname" . }}
spec:
host: {{ .Values.host }}
scaleTargetRef:
deployment: {{ include "xkcd.fullname" . }}
service: {{ include "xkcd.fullname" . }}
port: 8080
replicas:
min: {{ .Values.autoscaling.http.minReplicas }}
max: {{ .Values.autoscaling.http.maxReplicas }}
targetPendingRequests: {{ .Values.httpscaledobject.targetPendingRequests }}

but even than it won't work throwing an error:
Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest: error validating "": error validating data: ValidationError(HTTPScaledObject.spec): unknown field "targetPendingRequests" in sh.keda.http.v1alpha1.HTTPScaledObject.spec

I don't know how to move forward and install http addon with custom value for pending request - and this is my goal.

Any advice?

[arschles]

What I want is to be able to setup scaling option. Let's say 10 http requests per pod. Is it possible, please? And if yes, could you describe how to do it, please.

@dariuszbz there's no way to specify a max value of requests per pod at the moment. The closest thing we have for that is the targetPendingRequests field. That value is the target number of total requests that should be in flight at one time, across all the pods. If there are more in flight than that number, the HTTP Addon will scale up. Similarly, if there are fewer, it will scale down.

I don't know how to move forward and install http addon with custom value for pending request - and this is my goal.

If you're looking to set that targetPendingRequests value from the xkcd helm chart, there's no way to do that currently. I can submit a PR to add that feature, though. Would that help you?

[arschles]

Just to add another note, you can always submit this value by building your own HTTPScaledObject yaml file - it's just not configured in the xkcd helm chart specifically, so I can add that if you'd like.

[dariuszbz]

Thank You @arschles any help - custom object would be the best at the moment, but PR as a future, as well as updating the documentation ;)

[dariuszbz]

Sorry ... it's me again. How does it work " That value is the target number of total requests that should be in flight at one time, across all the pods. If there are more in flight than that number, the HTTP Addon will scale up." ?

let's say there is one pod (literally one), and targetPendingRequests value is 200. when there are 201 requests, the add-on will create a second pod. And what will happen if request numbers increases and after a few second there are 2000 request. Do we still have two pods only?

[dariuszbz]

I don't give up and want to make it work :)
still wont work.

http object:

kind: HTTPScaledObject
apiVersion: http.keda.sh/v1alpha1
metadata:
name: k8sdemo
spec:
host: "myhost.com"
scaleTargetRef:
deployment: k8sdemo
service: k8sdemo
port: 80
## targetPendingRequests: 10
replicas:
min: 1 # change spec.replicas.min to 1
max: 10

keda and add-on installed in keda-system namespace , an app installed in k8sdemo namespace.

logs from pod-keda-operator (NOTE: there is no such a thing "k8sdemo-app" in my cluster)

2022-01-18T10:47:38.252Z INFO controller.scaledobject Reconciling ScaledObject {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "k8sdemo-app", "namespace": "keda-system"}
2022-01-18T10:47:38.262Z ERROR controller.scaledobject Target resource doesn't exist {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "k8sdemo-app", "namespace": "keda-system", "resource": "apps/v1.Deployment", "name": "k8sdemo", "error": "deployments.apps "k8sdemo" not found"}
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
/workspace/controllers/keda/scaledobject_controller.go:179
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.3/pkg/internal/controller/controller.go:114
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.3/pkg/internal/controller/controller.go:311
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.3/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.3/pkg/internal/controller/controller.go:227
2022-01-18T10:47:38.263Z ERROR controller.scaledobject ScaledObject doesn't have correct scaleTargetRef specification {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "k8sdemo-app", "namespace": "keda-system", "error": "deployments.apps "k8sdemo" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.3/pkg/internal/controller/controller.go:311
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.3/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.3/pkg/internal/controller/controller.go:227
2022-01-18T10:47:38.271Z ERROR controller.scaledobject Reconciler error {"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "k8sdemo-app", "namespace": "keda-system", "error": "deployments.apps "k8sdemo" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.3/pkg/internal/controller/controller.go:227

I

[arschles]

Sorry about the delay @dariuszbz!

---

Thank You @arschles any help - custom object would be the best at the moment, but PR as a future, as well as updating the documentation ;)

I've added the ability to set the targetPendingRequests field in #373 and updated the documentation in #372

---

let's say there is one pod (literally one), and targetPendingRequests value is 200. when there are 201 requests, the add-on will create a second pod. And what will happen if request numbers increases and after a few second there are 2000 request. Do we still have two pods only?

targetPendingRequests says that we want no more than 200 pending requests in the system at any time, which means that if 201 requests come in, that doesn't necessarily mean we are going to scale up because the single pod may be able to process all of them quickly enough. Similarly, if 2000 requests come in and the two pods can process them quickly enough, KEDA will never see 2000 pending requests and will never scale up.

On the other hand, if the two pods can't process them fast enough, there will be N pending requests, and KEDA uses the Kubernetes horizontal pod autoscaler to calculate the number of pods to scale to. The generic formula is listed at https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#algorithm-details, but here it is rewritten using HTTP specific nomenclature:

desiredReplicas = ceil[currentReplicas * ( currentNumPendingRequests / targetPendingRequests )]

---

still wont work.

The HTTPScaledObject you submitted to the cluster is telling the HTTP Addon (and KEDA, under the hood) to do two things:

1. Route incoming traffic to myhost.com to the k8sdemo Service on port 80
2. Scale the k8sdemo Deployment up and down according to the pending number of requests for myhost.com

So, what those logs are saying is that KEDA can't find your Deployment or Service, and therefore can't do any scaling. Is it possible for you to change your HTTPScaledObject to point to a valid Deployment and Service in your cluster?

[dariuszbz]

Thank You @arschles .
I understand your explanation. I really do :) and I knew that.

the problem is there is k8sdemo Deployment, and k8sdemo Service. They are in namespace: k8sdemo.
Keda is in different namespace: keda-system.

Where do you want me to install http-add-on: k8s-demo namespace or keda-system namespace.
2.
Where do you want me to put httpscaledobject: k8s-demo or keda-system?

[arschles]

@dariuszbz ah, sorry - I misunderstood what you said. Thought you were saying that k8s-demo didn't exist. I'll put answers inline to your questions below:

Where do you want me to install http-add-on: k8s-demo namespace or keda-system namespace.

can you install it into the k8s-demo namespace? in the v0.2.0 release and earlier, it should be installed into the same namespace as the app it's trying to scale (this requirement will be relaxed in the coming v0.3.0 release). KEDA can be installed in keda-system, but I usually install everything (KEDA, HTTP Addon, and the app itself) into the same namespace for simplicity.

Where do you want me to put httpscaledobject: k8s-demo or keda-system?

Can you please install the HTTPScaledObject into k8s-demo as well?

[dariuszbz]

Thank You @arschles .

I've done as you described.

Now the keda-oparator:

2022-01-24T09:54:14.935Z ERROR external_scaler error calling IsActive on external scaler {"error": "rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup keda-add-ons-http-external-scaler on 10.0.0.10:53: no such host""}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
/workspace/pkg/scaling/scale_handler.go:261
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
/workspace/pkg/scaling/scale_handler.go:146

I assume it's not bad as it refer to external scaler, and external scaler logs:

{"level":"error","ts":1643017917.1607754,"logger":"GetMetricSpec","caller":"scaler/handlers.go:127","msg":"error getting target for host","host":"myhost.com","error":"Target not found","stacktrace":"main.(*impl).GetMetricSpec\n\t/go/src/github.com/kedacore/http-add-on/scaler/handlers.go:127\ngithub.com/kedacore/http-add-on/proto._ExternalScaler_GetMetricSpec_Handler\n\t/go/src/github.com/kedacore/http-add-on/proto/scaler_grpc.pb.go:178\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1279\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1608\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:923"}
{"level":"error","ts":1643017917.1875103,"logger":"IsActive","caller":"scaler/handlers.go:72","msg":"Given host was not found in queue count map","host":"myhost.com","allCounts":{},"error":"host 'myhost.com' not found in counts","stacktrace":"main.(*impl).IsActive\n\t/go/src/github.com/kedacore/http-add-on/scaler/handlers.go:72\ngithub.com/kedacore/http-add-on/proto._ExternalScaler_IsActive_Handler\n\t/go/src/github.com/kedacore/http-add-on/proto/scaler_grpc.pb.go:139\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1279\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:1608\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.41.0/server.go:923"}

I run the following line to run test:
for($i = 0; $i -le 300; $i = $i + 1) { curl -H "myhost.com" xxxxxxx }

but i don't see scaling effect. :(

httpobject:

"kind: HTTPScaledObject
apiVersion: http.keda.sh/v1alpha1
metadata:
name: k8sdemo
spec:
host: "myhost.com"
scaleTargetRef:
deployment: k8sdemo
service: k8sdemo
port: 80
##targetPendingRequests: 10
replicas:
min: 1 # change spec.replicas.min to 1
max: 10"

btw: targetPendingRequests is still no available. Throws an error during deployment, so it is commented in above yaml.

I did helm repo update. install keda into keda-system namespace, and add-on, httpobject and the app into the k8sdemo namespace.

Could you advise please? I think we are one step to complete my tests. Thank You!

[dariuszbz]

@arschles - hello mate. any advice?

[arschles]

sorry @dariuszbz - I was preparing for and then releasing v0.3.0. Regarding this:

2022-01-24T09:54:14.935Z ERROR external_scaler error calling IsActive on external scaler {"error": "rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup keda-add-ons-http-external-scaler on 10.0.0.10:53: no such host""}

I suspect something went wrong when the HTTP Addon was installed to your cluster. can you show me the output of your kubectl get pods in the k8s-demo namespace? it should look something like this:

┌───────────────────>
│~ 
└─> kubectl get pods -n kedahttp1
NAME                                                    READY   STATUS    RESTARTS   AGE
keda-add-ons-http-controller-manager-77cd79449b-2nkqs   2/2     Running   0          19h
keda-add-ons-http-external-scaler-5db478d9ff-2g5js      1/1     Running   0          19h
keda-add-ons-http-interceptor-74f47468cb-2xh56          1/1     Running   0          19h
keda-operator-78bf54cc6d-n5n9d                          1/1     Running   2          19d
keda-operator-metrics-apiserver-697bdc5996-rtxsj        1/1     Running   1          47d

Also, now that v0.3.0 is out, it might be helpful to install that version, since it adds the ability for the HTTP addon to work across all namespaces. I suspect that will help with this problem. We're almost there :)

[dariuszbz]

Thank You, and appreciate your feedback.
Requested output is as follow:
kubectl get pods -n k8sdemo

NAME READY STATUS RESTARTS AGE
k8sdemo-845844d7cb-fzjbh 1/1 Running 0 2d22h
keda-add-ons-http-controller-manager-866884c8cc-wc6bj 2/2 Running 3 6d1h
keda-add-ons-http-external-scaler-64ccd7d5cc-rpmkl 1/1 Running 12 7d
keda-add-ons-http-interceptor-595b68f4db-7kvmr 1/1 Running 11 7d

I'm going to reinstall with 0.3.0 now.

Can't wait to make it work :)

[dariuszbz]

I did upgrade:

helm uninstall http-add-on
helm repo update

$namespace = "k8sdemo"
helm install -n $namespace http-add-on kedacore/keda-add-ons-http

requested output is:
kubectl get pods -n k8sdemo

NAME READY STATUS RESTARTS AGE
k8sdemo-845844d7cb-fzjbh 1/1 Running 0 2d22h
keda-add-ons-http-controller-manager-74c8d684c5-568wq 2/2 Running 0 9m18s
keda-add-ons-http-external-scaler-9544cdff8-5cnsg 1/1 Running 0 9m18s
keda-add-ons-http-interceptor-776845f4d9-wz7xw 1/1 Running 0 9m18s

the error is still appearing:
2022-01-31T11:01:00.427Z ERROR external_scaler error calling IsActive on external scaler {"error": "rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup keda-add-ons-http-external-scaler on 10.0.0.10:53: no such host""}
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
/workspace/pkg/scaling/scale_handler.go:261
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
/workspace/pkg/scaling/scale_handler.go:146

and in external scaler pod:
{"level":"error","ts":1643626719.6587505,"logger":"IsActive","caller":"scaler/handlers.go:75","msg":"Given host was not found in queue count map","host":"myhost.com","allCounts":{},"error":"host 'myhost.com' not found in counts","stacktrace":"main.(*impl).IsActive\n\t/go/src/github.com/kedacore/http-add-on/scaler/handlers.go:75\ngithub.com/kedacore/http-add-on/proto._ExternalScaler_IsActive_Handler\n\t/go/src/github.com/kedacore/http-add-on/proto/scaler_grpc.pb.go:139\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.43.0/server.go:1282\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.43.0/server.go:1616\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.43.0/server.go:921"}

and from keda-operator pod:
E0129 11:34:57.724625 1 status.go:71] apiserver received an error that is not an metav1.Status: &url.Error{Op:"Get", URL:"https://10.0.0.1:443/apis/keda.sh/v1alpha1/namespaces/k8sdemo/scaledobjects?labelSelector=scaledobject.keda.sh%2Fname%3Dk8sdemo-app", Err:(*errors.errorString)(0xc00011e0b0)}: Get "https://10.0.0.1:443/apis/keda.sh/v1alpha1/namespaces/k8sdemo/scaledobjects?labelSelector=scaledobject.keda.sh%2Fname%3Dk8sdemo-app": context canceled
E0129 11:34:57.724798 1 writers.go:117] apiserver was unable to write a JSON response: http: Handler timeout
E0129 11:34:57.725915 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
E0129 11:34:57.727052 1 writers.go:130] apiserver was unable to write a fallback JSON response: http: Handler timeout
I0129 11:34:57.728198 1 trace.go:205] Trace[671764567]: "List" url:/apis/external.metrics.k8s.io/v1beta1/namespaces/k8sdemo/s0-myhost.com,user-agent:kube-controller-manager/v1.21.7 (linux/amd64) kubernetes/1f86634/system:serviceaccount:kube-system:horizontal-pod-autoscaler,audit-id:4c5e9339-a50d-4213-946b-2405204a670b,client:172.31.18.227,accept:application/vnd.kubernetes.protobuf, /,protocol:HTTP/2.0 (29-Jan-2022 11:34:41.445) (total time: 16282ms):
Trace[671764567]: [16.282515416s] [16.282515416s] END
E0129 11:34:57.728752 1 timeout.go:135] post-timeout activity - time-elapsed: 3.956581ms, GET "/apis/external.metrics.k8s.io/v1beta1/namespaces/k8sdemo/s0-myhost.com" result:
I0129 15:06:20.840284 1 trace.go:205] Trace[2122939447]: "List" url:/apis/external.metrics.k8s.io/v1beta1/namespaces/k8sdemo/s0-myhost.com,user-agent:kube-controller-manager/v1.21.7 (linux/amd64) kubernetes/1f86634/system:serviceaccount:kube-system:horizontal-pod-autoscaler,audit-id:eddc73ad-fc69-4f10-94a0-abf3f58da9e8,client:172.31.5.183,accept:application/vnd.kubernetes.protobuf, /,protocol:HTTP/2.0 (29-Jan-2022 15:06:20.326) (total time: 513ms):
Trace[2122939447]: ---"Listing from storage done" 513ms (15:06:20.840)
Trace[2122939447]: [513.830339ms] [513.830339ms] END
W0129 17:18:06.308411 1 clientconn.go:1315] [core] grpc: addrConn.createTransport failed to connect to {keda-add-ons-http-external-scaler.k8sdemo.svc.cluster.local:9090 keda-add-ons-http-external-scaler.k8sdemo.svc.cluster.local:9090 0 }. Err: connection error: desc = "transport: Error while dialing dial tcp 10.0.61.65:9090: connect: connection refused"
E0129 17:18:06.308474 1 deleg.go:144] keda_metrics_adapter/external_scaler "msg"="error" "error"="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 10.0.61.65:9090: connect: connection refused""
E0129 17:18:06.308500 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"No matching metrics found for s0-myhost.com"}: No matching metrics found for s0-myhost.com
E0130 02:31:02.176834 1 deleg.go:144] keda_metrics_adapter/external_scaler "msg"="error" "error"="rpc error: code = Unknown desc = host 'myhost.com' not found in counts"
E0130 02:31:02.179471 1 deleg.go:144] keda_metrics_adapter/external_scaler "msg"="error" "error"="rpc error: code = Unknown desc = host 'myhost.com' not found in counts"
E0130 02:31:02.194346 1 provider.go:124] keda_metrics_adapter/provider "msg"="error getting metric for scaler" "error"="rpc error: code = Unknown desc = host 'myhost.com' not found in counts" "scaledObject.Name"="k8sdemo-app" "scaledObject.Namespace"="k8sdemo" "scaler"={}
E0130 02:31:02.194381 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"No matching metrics found for s0-myhost.com"}: No matching metrics found for s0-myhost.com
E0130 04:52:23.825913 1 deleg.go:144] keda_metrics_adapter/external_scaler "msg"="error" "error"="rpc error: code = Unknown desc = host 'myhost.com' not found in counts"
E0130 04:52:23.828378 1 deleg.go:144] keda_metrics_adapter/external_scaler "msg"="error" "error"="rpc error: code = Unknown desc = host 'myhost.com' not found in counts"
E0130 04:52:23.840626 1 provider.go:124] keda_metrics_adapter/provider "msg"="error getting metric for scaler" "error"="rpc error: code = Unknown desc = host 'myhost.com' not found in counts" "scaledObject.Name"="k8sdemo-app" "scaledObject.Namespace"="k8sdemo" "scaler"={}
E0130 04:52:23.840660 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"No matching metrics found for s0-myhost.com"}: No matching metrics found for s0-myhost.com
W0130 05:18:07.061427 1 clientconn.go:1315] [core] grpc: addrConn.createTransport failed to connect to {keda-add-ons-http-external-scaler.k8sdemo.svc.cluster.local:9090 keda-add-ons-http-external-scaler.k8sdemo.svc.cluster.local:9090 0 }. Err: connection error: desc = "transport: Error while dialing dial tcp 10.0.61.65:9090: connect: connection refused"
E0130 05:18:07.061663 1 deleg.go:144] keda_metrics_adapter/external_scaler "msg"="error" "error"="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp 10.0.61.65:9090: connect: connection refused""
E0130 05:18:07.061739 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"No matching metrics found for s0-myhost.com"}: No matching metrics found for s0-myhost.com
I0130 05:18:07.062031 1 trace.go:205] Trace[1373082432]: "List" url:/apis/external.metrics.k8s.io/v1beta1/namespaces/k8sdemo/s0-myhost.com,user-agent:kube-controller-manager/v1.21.7 (linux/amd64) kubernetes/1f86634/system:serviceaccount:kube-system:horizontal-pod-autoscaler,audit-id:500b8e58-0962-4542-b235-ca54ec3f5cab,client:172.31.5.183,accept:application/vnd.kubernetes.protobuf, /,protocol:HTTP/2.0 (30-Jan-2022 05:18:06.036) (total time: 1025ms):
Trace[1373082432]: [1.025136954s] [1.025136954s] END
I0130 11:36:40.492794 1 trace.go:205] Trace[653398731]: "List" url:/apis/external.metrics.k8s.io/v1beta1/namespaces/k8sdemo/s0-myhost.com,user-agent:kube-controller-manager/v1.21.7 (linux/amd64) kubernetes/1f86634/system:serviceaccount:kube-system:horizontal-pod-autoscaler,audit-id:9e6f274e-5b11-4b9c-b8a7-afb950874754,client:172.31.5.183,accept:application/vnd.kubernetes.protobuf, /,protocol:HTTP/2.0 (30-Jan-2022 11:36:39.976) (total time: 516ms):
Trace[653398731]: ---"Listing from storage done" 515ms (11:36:40.492)
Trace[653398731]: [516.016277ms] [516.016277ms] END
I0131 06:12:38.465399 1 trace.go:205] Trace[349145759]: "List" url:/apis/external.metrics.k8s.io/v1beta1/namespaces/k8sdemo/s0-myhost.com,user-agent:kube-controller-manager/v1.21.7 (linux/amd64) kubernetes/1f86634/system:serviceaccount:kube-system:horizontal-pod-autoscaler,audit-id:4066dc40-f03b-4219-9517-bdbbe66fe83d,client:172.31.13.161,accept:application/vnd.kubernetes.protobuf, /,protocol:HTTP/2.0 (31-Jan-2022 06:12:37.958) (total time: 507ms):
Trace[349145759]: ---"Listing from storage done" 506ms (06:12:38.465)
Trace[349145759]: [507.049484ms] [507.049484ms] END
E0131 10:57:35.953392 1 deleg.go:144] keda_metrics_adapter/external_scaler "msg"="error" "error"="rpc error: code = Unknown desc = Target not found"
E0131 10:57:35.953572 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"No matching metrics found for s0-myhost.com"}: No matching metrics found for s0-myhost.com

[arschles]

ok, thanks for trying that. do you have any network security policies defined in your cluster?

[dariuszbz]

Hi @arschles . Nop. No security. I'm happy to setup a qcall with screen sharing if it helps.

[arschles]

hm, yes let's do that. are you on the Kubernetes slack? if so, can you send me a message so we can set it up? my name is arschles (same as my GitHub name)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity.