Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

e2e tests fail with pods not coming up. #4725

Closed
vaikas opened this issue Jan 12, 2021 · 6 comments · Fixed by #4741
Closed

e2e tests fail with pods not coming up. #4725

vaikas opened this issue Jan 12, 2021 · 6 comments · Fixed by #4741
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@vaikas
Copy link
Contributor

vaikas commented Jan 12, 2021

Describe the bug

Here's one example
https://github.com/knative/eventing/pull/4724/checks?check_run_id=1687176531

Every now and then we observe a failure where the pods are supposedly not coming up even though they are. Working theory is that things get started, chaos duck kills something and somehow the list gets screwed because of this:

Run kubectl -v 9 wait pod --for=condition=Ready -n ${SYSTEM_NAMESPACE} -l '!job-name'
I0112 10:00:29.155380   26156 loader.go:375] Config loaded from file:  /home/runner/.kube/config
I0112 10:00:29.178703   26156 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?labelSelector=%21job-name'
I0112 10:00:29.192251   26156 round_trippers.go:443] GET https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?labelSelector=%21job-name 200 OK in 13 milliseconds
I0112 10:00:29.192674   26156 round_trippers.go:449] Response Headers:
I0112 10:00:29.192867   26156 round_trippers.go:452]     Cache-Control: no-cache, private
I0112 10:00:29.193424   26156 round_trippers.go:452]     Content-Type: application/json
I0112 10:00:29.193938   26156 round_trippers.go:452]     Date: Tue, 12 Jan 2021 10:00:29 GMT
I0112 10:00:29.194886   26156 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"3093"},"items":[{"metadata":{"name":"chaosduck-5d8694d86b-7qf9q","generateName":"chaosduck-5d8694d86b-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/chaosduck-5d8694d86b-7qf9q","uid":"09cfacd4-9c8b-4514-84c4-2b7e8ce3c4cf","resourceVersion":"3092","creationTimestamp":"2021-01-12T09:59:10Z","deletionTimestamp":"2021-01-12T09:59:42Z","deletionGracePeriodSeconds":30,"labels":{"app":"chaosduck","pod-template-hash":"5d8694d86b"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"chaosduck-5d8694d86b","uid":"330ec909-410d-422b-873c-e48c708a728c","controller":true,"blockOwnerDeletion":true}],"managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2021-01-12T09:59:10Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:app":{},"f:pod-template-hash":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"330ec909-410d-422b-873c-e48c708a728c\"}":{".":{},"f:apiVersion":{},"f:blockOwnerDeletion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:spec":{"f:containers":{"k:{\"name\":\"chaosduck\"}":{".":{},"f:args":{},"f:env":{".":{},"k:{\"name\":\"SYSTEM_NAMESPACE\"}":{".":{},"f:name":{},"f:valueFrom":{".":{},"f:fieldRef":{".":{},"f:apiVersion":{},"f:fieldPath":{}}}}},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:securityContext":{".":{},"f:allowPrivilegeEscalation":{}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-01-12T10:00:28Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.1.10\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"chaosduck-token-njg99","secret":{"secretName":"chaosduck-token-njg99","defaultMode":420}}],"containers":[{"name":"chaosduck","image":"kind.local/knative.dev/pkg/leaderelection/chaosduck:eedd2bc811c5c0f3a058521009893f7a53c48afce45ce12f6ca24806a69f43b8","args":["-disable=imc-controller","-disable=imc-dispatcher"],"env":[{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}}],"resources":{},"volumeMounts":[{"name":"chaosduck-token-njg99","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"allowPrivilegeEscalation":false}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"chaosduck","serviceAccount":"chaosduck","nodeName":"kind-worker","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T09:59:10Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-01-12T10:00:28Z","reason":"ContainersNotReady","message":"containers with unready status: [chaosduck]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-01-12T10:00:28Z","reason":"ContainersNotReady","message":"containers with unready status: [chaosduck]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T09:59:10Z"}],"hostIP":"172.18.0.2","podIP":"10.244.1.10","podIPs":[{"ip":"10.244.1.10"}],"startTime":"2021-01-12T09:59:10Z","containerStatuses":[{"name":"chaosduck","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-01-12T09:59:12Z","finishedAt":"2021-01-12T10:00:23Z","containerID":"containerd://ae44d1a0b5c17a9b1b08cd1c9bf823afe89b71a637269d31608a4faded961cbb"}},"lastState":{},"ready":false,"restartCount":0,"image":"kind.local/knative.dev/pkg/leaderelection/chaosduck:eedd2bc811c5c0f3a058521009893f7a53c48afce45ce12f6ca24806a69f43b8","imageID":"sha256:fdeed9cea55a31608971f0d1b049e9cdfae8d25e26daf366f089a4117fc2be2d","containerID":"containerd://ae44d1a0b5c17a9b1b08cd1c9bf823afe89b71a637269d31608a4faded961cbb","started":false}],"qosClass":"BestEffort"}},{"metadata":{"name":"eventing-controller-67d57bbcb9-c7n6n","generateName":"eventing-controller-67d57bbcb9-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/eventing-controller-67d57bbcb9-c7n6n","uid":"6b4c1e0c-9fef-4a45-b11b-571e588d2fb3","resourceVersion":"2704","creationTimestamp":"2021-01-12T09:59:57Z","labels":{"app":"eventing-controller","eventing.knative.dev/release":"devel","pod-template-hash":"67d57bbcb9"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"eventing-controller-67d57bbcb9","uid":"e982bee4-1ba6-4ec5-b03b-9dde520664e0","controller":true,"blockOwnerDeletion":true}],"managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2021-01-12T09:59:57Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:app":{},"f:eventing.knative.dev/release":{},"f:pod-template-hash":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"e982bee4-1ba6-4ec5-b03b-9dde520664e0\"}":{".":{},"f:apiVersion":{},"f:blockOwnerDeletion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:spec":{"f:affinity":{".":{},"f:podAntiAffinity":{".":{},"f:preferredDuringSchedulingIgnoredDuringExecution":{}}},"f:containers":{"k:{\"name\":\"eventing-controller\"}":{".":{},"f:env":{".":{},"k:{\"name\":\"APISERVER_RA_IMAGE\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"CONFIG_LOGGING_NAME\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"CONFIG_OBSERVABILITY_NAME\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"METRICS_DOMAIN\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"POD_NAME\"}":{".":{},"f:name":{},"f:valueFrom":{".":{},"f:fieldRef":{".":{},"f:apiVersion":{},"f:fieldPath":{}}}},"k:{\"name\":\"SYSTEM_NAMESPACE\"}":{".":{},"f:name":{},"f:valueFrom":{".":{},"f:fieldRef":{".":{},"f:apiVersion":{},"f:fieldPath":{}}}}},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:ports":{".":{},"k:{\"containerPort\":8008,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}},"k:{\"containerPort\":9090,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}}},"f:resources":{".":{},"f:requests":{".":{},"f:cpu":{},"f:memory":{}}},"f:securityContext":{".":{},"f:allowPrivilegeEscalation":{}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-01-12T10:00:03Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.1.16\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"eventing-controller-token-zbv2z","secret":{"secretName":"eventing-controller-token-zbv2z","defaultMode":420}}],"containers":[{"name":"eventing-controller","image":"kind.local/knative.dev/eventing/cmd/controller:419b856bcf2e2e3a256c999936453bbe8a86d479b97b400aa919a548de283b18","ports":[{"name":"metrics","containerPort":9090,"protocol":"TCP"},{"name":"profiling","containerPort":8008,"protocol":"TCP"}],"env":[{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"CONFIG_LOGGING_NAME","value":"config-logging"},{"name":"CONFIG_OBSERVABILITY_NAME","value":"config-observability"},{"name":"METRICS_DOMAIN","value":"knative.dev/eventing"},{"name":"APISERVER_RA_IMAGE","value":"kind.local/knative.dev/eventing/cmd/apiserver_receive_adapter:c18172a78588abbbe7d006194c8a330547c9b7793495ae99078003769382a295"},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}}],"resources":{"requests":{"cpu":"100m","memory":"100Mi"}},"volumeMounts":[{"name":"eventing-controller-token-zbv2z","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent","securityContext":{"allowPrivilegeEscalation":false}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"eventing-controller","serviceAccount":"eventing-controller","nodeName":"kind-worker","securityContext":{},"affinity":{"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":100,"podAffinityTerm":{"labelSelector":{"matchLabels":{"app":"eventing-controller"}},"topologyKey":"kubernetes.io/hostname"}}]}},"schedulerName":"default-scheduler","toler [truncated 59478 chars]
I0112 10:00:29.212305   26156 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dchaosduck-5d8694d86b-7qf9q'
I0112 10:00:29.214537   26156 round_trippers.go:443] GET https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dchaosduck-5d8694d86b-7qf9q 200 OK in 1 milliseconds
I0112 10:00:29.214552   26156 round_trippers.go:449] Response Headers:
I0112 10:00:29.214557   26156 round_trippers.go:452]     Date: Tue, 12 Jan 2021 10:00:29 GMT
I0112 10:00:29.214560   26156 round_trippers.go:452]     Cache-Control: no-cache, private
I0112 10:00:29.214563   26156 round_trippers.go:452]     Content-Type: application/json
I0112 10:00:29.214705   26156 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"3093"},"items":[{"metadata":{"name":"chaosduck-5d8694d86b-7qf9q","generateName":"chaosduck-5d8694d86b-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/chaosduck-5d8694d86b-7qf9q","uid":"09cfacd4-9c8b-4514-84c4-2b7e8ce3c4cf","resourceVersion":"3092","creationTimestamp":"2021-01-12T09:59:10Z","deletionTimestamp":"2021-01-12T09:59:42Z","deletionGracePeriodSeconds":30,"labels":{"app":"chaosduck","pod-template-hash":"5d8694d86b"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"chaosduck-5d8694d86b","uid":"330ec909-410d-422b-873c-e48c708a728c","controller":true,"blockOwnerDeletion":true}],"managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2021-01-12T09:59:10Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:app":{},"f:pod-template-hash":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"330ec909-410d-422b-873c-e48c708a728c\"}":{".":{},"f:apiVersion":{},"f:blockOwnerDeletion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:spec":{"f:containers":{"k:{\"name\":\"chaosduck\"}":{".":{},"f:args":{},"f:env":{".":{},"k:{\"name\":\"SYSTEM_NAMESPACE\"}":{".":{},"f:name":{},"f:valueFrom":{".":{},"f:fieldRef":{".":{},"f:apiVersion":{},"f:fieldPath":{}}}}},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:securityContext":{".":{},"f:allowPrivilegeEscalation":{}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-01-12T10:00:28Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.1.10\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"chaosduck-token-njg99","secret":{"secretName":"chaosduck-token-njg99","defaultMode":420}}],"containers":[{"name":"chaosduck","image":"kind.local/knative.dev/pkg/leaderelection/chaosduck:eedd2bc811c5c0f3a058521009893f7a53c48afce45ce12f6ca24806a69f43b8","args":["-disable=imc-controller","-disable=imc-dispatcher"],"env":[{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}}],"resources":{},"volumeMounts":[{"name":"chaosduck-token-njg99","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"allowPrivilegeEscalation":false}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"chaosduck","serviceAccount":"chaosduck","nodeName":"kind-worker","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T09:59:10Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-01-12T10:00:28Z","reason":"ContainersNotReady","message":"containers with unready status: [chaosduck]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-01-12T10:00:28Z","reason":"ContainersNotReady","message":"containers with unready status: [chaosduck]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T09:59:10Z"}],"hostIP":"172.18.0.2","podIP":"10.244.1.10","podIPs":[{"ip":"10.244.1.10"}],"startTime":"2021-01-12T09:59:10Z","containerStatuses":[{"name":"chaosduck","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2021-01-12T09:59:12Z","finishedAt":"2021-01-12T10:00:23Z","containerID":"containerd://ae44d1a0b5c17a9b1b08cd1c9bf823afe89b71a637269d31608a4faded961cbb"}},"lastState":{},"ready":false,"restartCount":0,"image":"kind.local/knative.dev/pkg/leaderelection/chaosduck:eedd2bc811c5c0f3a058521009893f7a53c48afce45ce12f6ca24806a69f43b8","imageID":"sha256:fdeed9cea55a31608971f0d1b049e9cdfae8d25e26daf366f089a4117fc2be2d","containerID":"containerd://ae44d1a0b5c17a9b1b08cd1c9bf823afe89b71a637269d31608a4faded961cbb","started":false}],"qosClass":"BestEffort"}}]}
I0112 10:00:29.215222   26156 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dchaosduck-5d8694d86b-7qf9q&resourceVersion=3093&watch=true'
I0112 10:00:29.215819   26156 round_trippers.go:443] GET https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dchaosduck-5d8694d86b-7qf9q&resourceVersion=3093&watch=true 200 OK in 0 milliseconds
I0112 10:00:29.215828   26156 round_trippers.go:449] Response Headers:
I0112 10:00:29.215831   26156 round_trippers.go:452]     Cache-Control: no-cache, private
I0112 10:00:29.215834   26156 round_trippers.go:452]     Content-Type: application/json
I0112 10:00:29.215837   26156 round_trippers.go:452]     Date: Tue, 12 Jan 2021 10:00:29 GMT
I0112 10:00:59.216616   26156 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Deventing-controller-67d57bbcb9-c7n6n'
I0112 10:00:59.218402   26156 round_trippers.go:443] GET https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Deventing-controller-67d57bbcb9-c7n6n 200 OK in 1 milliseconds
I0112 10:00:59.218412   26156 round_trippers.go:449] Response Headers:
I0112 10:00:59.218416   26156 round_trippers.go:452]     Cache-Control: no-cache, private
I0112 10:00:59.218423   26156 round_trippers.go:452]     Content-Type: application/json
I0112 10:00:59.218426   26156 round_trippers.go:452]     Date: Tue, 12 Jan 2021 10:00:59 GMT
I0112 10:00:59.218629   26156 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"3522"},"items":[{"metadata":{"name":"eventing-controller-67d57bbcb9-c7n6n","generateName":"eventing-controller-67d57bbcb9-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/eventing-controller-67d57bbcb9-c7n6n","uid":"6b4c1e0c-9fef-4a45-b11b-571e588d2fb3","resourceVersion":"2704","creationTimestamp":"2021-01-12T09:59:57Z","labels":{"app":"eventing-controller","eventing.knative.dev/release":"devel","pod-template-hash":"67d57bbcb9"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"eventing-controller-67d57bbcb9","uid":"e982bee4-1ba6-4ec5-b03b-9dde520664e0","controller":true,"blockOwnerDeletion":true}],"managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2021-01-12T09:59:57Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:app":{},"f:eventing.knative.dev/release":{},"f:pod-template-hash":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"e982bee4-1ba6-4ec5-b03b-9dde520664e0\"}":{".":{},"f:apiVersion":{},"f:blockOwnerDeletion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:spec":{"f:affinity":{".":{},"f:podAntiAffinity":{".":{},"f:preferredDuringSchedulingIgnoredDuringExecution":{}}},"f:containers":{"k:{\"name\":\"eventing-controller\"}":{".":{},"f:env":{".":{},"k:{\"name\":\"APISERVER_RA_IMAGE\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"CONFIG_LOGGING_NAME\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"CONFIG_OBSERVABILITY_NAME\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"METRICS_DOMAIN\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"POD_NAME\"}":{".":{},"f:name":{},"f:valueFrom":{".":{},"f:fieldRef":{".":{},"f:apiVersion":{},"f:fieldPath":{}}}},"k:{\"name\":\"SYSTEM_NAMESPACE\"}":{".":{},"f:name":{},"f:valueFrom":{".":{},"f:fieldRef":{".":{},"f:apiVersion":{},"f:fieldPath":{}}}}},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:ports":{".":{},"k:{\"containerPort\":8008,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}},"k:{\"containerPort\":9090,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}}},"f:resources":{".":{},"f:requests":{".":{},"f:cpu":{},"f:memory":{}}},"f:securityContext":{".":{},"f:allowPrivilegeEscalation":{}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-01-12T10:00:03Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.1.16\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"eventing-controller-token-zbv2z","secret":{"secretName":"eventing-controller-token-zbv2z","defaultMode":420}}],"containers":[{"name":"eventing-controller","image":"kind.local/knative.dev/eventing/cmd/controller:419b856bcf2e2e3a256c999936453bbe8a86d479b97b400aa919a548de283b18","ports":[{"name":"metrics","containerPort":9090,"protocol":"TCP"},{"name":"profiling","containerPort":8008,"protocol":"TCP"}],"env":[{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"CONFIG_LOGGING_NAME","value":"config-logging"},{"name":"CONFIG_OBSERVABILITY_NAME","value":"config-observability"},{"name":"METRICS_DOMAIN","value":"knative.dev/eventing"},{"name":"APISERVER_RA_IMAGE","value":"kind.local/knative.dev/eventing/cmd/apiserver_receive_adapter:c18172a78588abbbe7d006194c8a330547c9b7793495ae99078003769382a295"},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}}],"resources":{"requests":{"cpu":"100m","memory":"100Mi"}},"volumeMounts":[{"name":"eventing-controller-token-zbv2z","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent","securityContext":{"allowPrivilegeEscalation":false}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"eventing-controller","serviceAccount":"eventing-controller","nodeName":"kind-worker","securityContext":{},"affinity":{"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":100,"podAffinityTerm":{"labelSelector":{"matchLabels":{"app":"eventing-controller"}},"topologyKey":"kubernetes.io/hostname"}}]}},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T09:59:57Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T10:00:03Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T10:00:03Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T09:59:57Z"}],"hostIP":"172.18.0.2","podIP":"10.244.1.16","podIPs":[{"ip":"10.244.1.16"}],"startTime":"2021-01-12T09:59:57Z","containerStatuses":[{"name":"eventing-controller","state":{"running":{"startedAt":"2021-01-12T10:00:02Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"kind.local/knative.dev/eventing/cmd/controller:419b856bcf2e2e3a256c999936453bbe8a86d479b97b400aa919a548de283b18","imageID":"sha256:d775b6b8769ee1e7671481fe8bbef0b44c51276bf4924898ed0b28aef3fe7c87","containerID":"containerd://4b91885600e42139dab1940c34f59052c14a23b40f1cd61095581a7a18407d5c","started":true}],"qosClass":"Burstable"}}]}
I0112 10:00:59.219514   26156 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Deventing-webhook-9d555fbd4-hnnbm'
I0112 10:00:59.221086   26156 round_trippers.go:443] GET https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Deventing-webhook-9d555fbd4-hnnbm 200 OK in 1 milliseconds
I0112 10:00:59.221110   26156 round_trippers.go:449] Response Headers:
I0112 10:00:59.221114   26156 round_trippers.go:452]     Content-Length: 140
I0112 10:00:59.221135   26156 round_trippers.go:452]     Date: Tue, 12 Jan 2021 10:00:59 GMT
I0112 10:00:59.221141   26156 round_trippers.go:452]     Cache-Control: no-cache, private
I0112 10:00:59.221144   26156 round_trippers.go:452]     Content-Type: application/json
I0112 10:00:59.221156   26156 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"3522"},"items":[]}
I0112 10:00:59.221278   26156 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Deventing-webhook-9d555fbd4-hnnbm&resourceVersion=3522&watch=true'
I0112 10:00:59.221971   26156 round_trippers.go:443] GET https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Deventing-webhook-9d555fbd4-hnnbm&resourceVersion=3522&watch=true 200 OK in 0 milliseconds
I0112 10:00:59.221985   26156 round_trippers.go:449] Response Headers:
I0112 10:00:59.221989   26156 round_trippers.go:452]     Content-Type: application/json
I0112 10:00:59.221993   26156 round_trippers.go:452]     Date: Tue, 12 Jan 2021 10:00:59 GMT
I0112 10:00:59.221996   26156 round_trippers.go:452]     Cache-Control: no-cache, private
pod/eventing-controller-67d57bbcb9-c7n6n condition met
I0112 10:01:29.222736   26156 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Deventing-webhook-9d555fbd4-st8fw'
I0112 10:01:29.225821   26156 round_trippers.go:443] GET https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Deventing-webhook-9d555fbd4-st8fw 200 OK in 3 milliseconds
I0112 10:01:29.225836   26156 round_trippers.go:449] Response Headers:
I0112 10:01:29.225841   26156 round_trippers.go:452]     Cache-Control: no-cache, private
I0112 10:01:29.225845   26156 round_trippers.go:452]     Content-Type: application/json
I0112 10:01:29.225848   26156 round_trippers.go:452]     Date: Tue, 12 Jan 2021 10:01:29 GMT
I0112 10:01:29.227030   26156 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"3944"},"items":[{"metadata":{"name":"eventing-webhook-9d555fbd4-st8fw","generateName":"eventing-webhook-9d555fbd4-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/eventing-webhook-9d555fbd4-st8fw","uid":"57a04376-a85c-4f30-9e0f-c5e032260288","resourceVersion":"2743","creationTimestamp":"2021-01-12T09:59:57Z","labels":{"app":"eventing-webhook","pod-template-hash":"9d555fbd4","role":"eventing-webhook"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"eventing-webhook-9d555fbd4","uid":"c2be7162-0148-475b-ab0a-ae0226596635","controller":true,"blockOwnerDeletion":true}],"managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2021-01-12T09:59:57Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:app":{},"f:pod-template-hash":{},"f:role":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"c2be7162-0148-475b-ab0a-ae0226596635\"}":{".":{},"f:apiVersion":{},"f:blockOwnerDeletion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:spec":{"f:affinity":{".":{},"f:podAntiAffinity":{".":{},"f:preferredDuringSchedulingIgnoredDuringExecution":{}}},"f:containers":{"k:{\"name\":\"eventing-webhook\"}":{".":{},"f:env":{".":{},"k:{\"name\":\"CONFIG_LOGGING_NAME\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"METRICS_DOMAIN\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"POD_NAME\"}":{".":{},"f:name":{},"f:valueFrom":{".":{},"f:fieldRef":{".":{},"f:apiVersion":{},"f:fieldPath":{}}}},"k:{\"name\":\"SINK_BINDING_SELECTION_MODE\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"SYSTEM_NAMESPACE\"}":{".":{},"f:name":{},"f:valueFrom":{".":{},"f:fieldRef":{".":{},"f:apiVersion":{},"f:fieldPath":{}}}},"k:{\"name\":\"WEBHOOK_NAME\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"WEBHOOK_PORT\"}":{".":{},"f:name":{},"f:value":{}}},"f:image":{},"f:imagePullPolicy":{},"f:livenessProbe":{".":{},"f:failureThreshold":{},"f:httpGet":{".":{},"f:httpHeaders":{},"f:path":{},"f:port":{},"f:scheme":{}},"f:initialDelaySeconds":{},"f:periodSeconds":{},"f:successThreshold":{},"f:timeoutSeconds":{}},"f:name":{},"f:ports":{".":{},"k:{\"containerPort\":8008,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}},"k:{\"containerPort\":8443,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}},"k:{\"containerPort\":9090,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}}},"f:readinessProbe":{".":{},"f:failureThreshold":{},"f:httpGet":{".":{},"f:httpHeaders":{},"f:path":{},"f:port":{},"f:scheme":{}},"f:periodSeconds":{},"f:successThreshold":{},"f:timeoutSeconds":{}},"f:resources":{".":{},"f:limits":{".":{},"f:cpu":{},"f:memory":{}},"f:requests":{".":{},"f:cpu":{},"f:memory":{}}},"f:securityContext":{".":{},"f:allowPrivilegeEscalation":{}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-01-12T10:00:04Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.1.17\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"eventing-webhook-token-ljck4","secret":{"secretName":"eventing-webhook-token-ljck4","defaultMode":420}}],"containers":[{"name":"eventing-webhook","image":"kind.local/knative.dev/eventing/cmd/webhook:b5a5737377c81d19b91ebf347e1a853232f74273063e4fa8669b2e5b7babfa55","ports":[{"name":"https-webhook","containerPort":8443,"protocol":"TCP"},{"name":"metrics","containerPort":9090,"protocol":"TCP"},{"name":"profiling","containerPort":8008,"protocol":"TCP"}],"env":[{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"CONFIG_LOGGING_NAME","value":"config-logging"},{"name":"METRICS_DOMAIN","value":"knative.dev/eventing"},{"name":"WEBHOOK_NAME","value":"eventing-webhook"},{"name":"WEBHOOK_PORT","value":"8443"},{"name":"SINK_BINDING_SELECTION_MODE","value":"exclusion"},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}}],"resources":{"limits":{"cpu":"200m","memory":"200Mi"},"requests":{"cpu":"20m","memory":"20Mi"}},"volumeMounts":[{"name":"eventing-webhook-token-ljck4","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"livenessProbe":{"httpGet":{"path":"/","port":8443,"scheme":"HTTPS","httpHeaders":[{"name":"k-kubelet-probe","value":"webhook"}]},"initialDelaySeconds":20,"timeoutSeconds":1,"periodSeconds":1,"successThreshold":1,"failureThreshold":3},"readinessProbe":{"httpGet":{"path":"/","port":8443,"scheme":"HTTPS","httpHeaders":[{"name":"k-kubelet-probe","value":"webhook"}]},"timeoutSeconds":1,"periodSeconds":1,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent","securityContext":{"allowPrivilegeEscalation":false}}],"restartPolicy":"Always","terminationGracePeriodSeconds":300,"dnsPolicy":"ClusterFirst","serviceAccountName":"eventing-webhook","serviceAccount":"eventing-webhook","nodeName":"kind-worker","securityContext":{},"affinity":{"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":100,"podAffinityTerm":{"labelSelector":{"matchLabels":{"app":"eventing-webhook"}},"topologyKey":"kubernetes.io/hostname"}}]}},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":false},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T09:59:58Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T10:00:04Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T10:00:04Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T09:59:57Z"}],"hostIP":"172.18.0.2","podIP":"10.244.1.17","podIPs":[{"ip":"10.244.1.17"}],"startTime":"2021-01-12T09:59:58Z","containerStatuses":[{"name":"eventing-webhook","state":{"running":{"startedAt":"2021-01-12T10:00:02Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"kind.local/knative.dev/eventing/cmd/webhook:latest","imageID":"sha256:ac72d6366971fda96b76839e59ef67b3f31eb66bc34f6225ec6bbe0a3433f81b","containerID":"containerd://0c8891f22e9e43092d29227f2c6e226147b8fb5beca5c56f20256986655fd7c1","started":true}],"qosClass":"Burstable"}}]}
pod/eventing-webhook-9d555fbd4-st8fw condition met
I0112 10:01:29.237752   26156 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dimc-controller-7586dff887-t68d6'
I0112 10:01:29.246275   26156 round_trippers.go:443] GET https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dimc-controller-7586dff887-t68d6 200 OK in 8 milliseconds
I0112 10:01:29.246367   26156 round_trippers.go:449] Response Headers:
I0112 10:01:29.246382   26156 round_trippers.go:452]     Cache-Control: no-cache, private
I0112 10:01:29.246386   26156 round_trippers.go:452]     Content-Type: application/json
I0112 10:01:29.246389   26156 round_trippers.go:452]     Date: Tue, 12 Jan 2021 10:01:29 GMT
I0112 10:01:29.246476   26156 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"3944"},"items":[{"metadata":{"name":"imc-controller-7586dff887-t68d6","generateName":"imc-controller-7586dff887-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/imc-controller-7586dff887-t68d6","uid":"9c736774-1c5a-4b12-a33f-4229c23a59a4","resourceVersion":"1625","creationTimestamp":"2021-01-12T09:58:57Z","labels":{"messaging.knative.dev/channel":"in-memory-channel","messaging.knative.dev/role":"controller","pod-template-hash":"7586dff887"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"imc-controller-7586dff887","uid":"b630de97-8394-4c7a-821f-3027bf31474f","controller":true,"blockOwnerDeletion":true}],"managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2021-01-12T09:58:57Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:messaging.knative.dev/channel":{},"f:messaging.knative.dev/role":{},"f:pod-template-hash":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"b630de97-8394-4c7a-821f-3027bf31474f\"}":{".":{},"f:apiVersion":{},"f:blockOwnerDeletion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:spec":{"f:containers":{"k:{\"name\":\"controller\"}":{".":{},"f:env":{".":{},"k:{\"name\":\"CONFIG_LOGGING_NAME\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"CONFIG_OBSERVABILITY_NAME\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"DISPATCHER_IMAGE\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"METRICS_DOMAIN\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"POD_NAME\"}":{".":{},"f:name":{},"f:valueFrom":{".":{},"f:fieldRef":{".":{},"f:apiVersion":{},"f:fieldPath":{}}}},"k:{\"name\":\"SYSTEM_NAMESPACE\"}":{".":{},"f:name":{},"f:valueFrom":{".":{},"f:fieldRef":{".":{},"f:apiVersion":{},"f:fieldPath":{}}}}},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:ports":{".":{},"k:{\"containerPort\":8008,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}},"k:{\"containerPort\":9090,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}}},"f:resources":{},"f:securityContext":{".":{},"f:allowPrivilegeEscalation":{}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-01-12T09:59:00Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.1.8\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"imc-controller-token-gd6dc","secret":{"secretName":"imc-controller-token-gd6dc","defaultMode":420}}],"containers":[{"name":"controller","image":"kind.local/knative.dev/eventing/cmd/in_memory/channel_controller:98bfd3ab9f876164ad87fd19cbd26efd22dad07de8423dcc9e67322bf359a7f6","ports":[{"name":"metrics","containerPort":9090,"protocol":"TCP"},{"name":"profiling","containerPort":8008,"protocol":"TCP"}],"env":[{"name":"CONFIG_LOGGING_NAME","value":"config-logging"},{"name":"CONFIG_OBSERVABILITY_NAME","value":"config-observability"},{"name":"METRICS_DOMAIN","value":"knative.dev/inmemorychannel-controller"},{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"DISPATCHER_IMAGE","value":"kind.local/knative.dev/eventing/cmd/in_memory/channel_dispatcher:8760c84ce6cbc008658d839bbfd4dfcb9650d8bafeeb4d9d2d2f3aa0820ec542"},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}}],"resources":{},"volumeMounts":[{"name":"imc-controller-token-gd6dc","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"allowPrivilegeEscalation":false}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"imc-controller","serviceAccount":"imc-controller","nodeName":"kind-worker","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T09:58:57Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T09:58:59Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T09:58:59Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T09:58:57Z"}],"hostIP":"172.18.0.2","podIP":"10.244.1.8","podIPs":[{"ip":"10.244.1.8"}],"startTime":"2021-01-12T09:58:57Z","containerStatuses":[{"name":"controller","state":{"running":{"startedAt":"2021-01-12T09:58:59Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"kind.local/knative.dev/eventing/cmd/in_memory/channel_controller:98bfd3ab9f876164ad87fd19cbd26efd22dad07de8423dcc9e67322bf359a7f6","imageID":"sha256:7f8cde280706e080d687a631e1d54b12bda522fbb8f68ec78ebf3102089490c6","containerID":"containerd://67a6b63f1fbe30a92ac2fe551252e280232d3a0ac81705064c8a432a207e2336","started":true}],"qosClass":"BestEffort"}}]}
I0112 10:01:29.247872   26156 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dimc-dispatcher-6d95d7b95f-dlkgb'
I0112 10:01:29.250385   26156 round_trippers.go:443] GET https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dimc-dispatcher-6d95d7b95f-dlkgb 200 OK in 2 milliseconds
I0112 10:01:29.250394   26156 round_trippers.go:449] Response Headers:
I0112 10:01:29.250398   26156 round_trippers.go:452]     Content-Type: application/json
I0112 10:01:29.250402   26156 round_trippers.go:452]     Date: Tue, 12 Jan 2021 10:01:29 GMT
I0112 10:01:29.250404   26156 round_trippers.go:452]     Cache-Control: no-cache, private
I0112 10:01:29.250470   26156 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"3944"},"items":[{"metadata":{"name":"imc-dispatcher-6d95d7b95f-dlkgb","generateName":"imc-dispatcher-6d95d7b95f-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/imc-dispatcher-6d95d7b95f-dlkgb","uid":"e70bfd18-0910-4a73-ac8b-c41f0dcace37","resourceVersion":"1636","creationTimestamp":"2021-01-12T09:58:57Z","labels":{"messaging.knative.dev/channel":"in-memory-channel","messaging.knative.dev/role":"dispatcher","pod-template-hash":"6d95d7b95f"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"imc-dispatcher-6d95d7b95f","uid":"2a31963a-0715-4464-8446-e17b936cbf60","controller":true,"blockOwnerDeletion":true}],"managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2021-01-12T09:58:57Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:messaging.knative.dev/channel":{},"f:messaging.knative.dev/role":{},"f:pod-template-hash":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"2a31963a-0715-4464-8446-e17b936cbf60\"}":{".":{},"f:apiVersion":{},"f:blockOwnerDeletion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:spec":{"f:affinity":{".":{},"f:podAntiAffinity":{".":{},"f:preferredDuringSchedulingIgnoredDuringExecution":{}}},"f:containers":{"k:{\"name\":\"dispatcher\"}":{".":{},"f:env":{".":{},"k:{\"name\":\"CONFIG_LOGGING_NAME\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"CONFIG_OBSERVABILITY_NAME\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"CONTAINER_NAME\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"METRICS_DOMAIN\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"POD_NAME\"}":{".":{},"f:name":{},"f:valueFrom":{".":{},"f:fieldRef":{".":{},"f:apiVersion":{},"f:fieldPath":{}}}},"k:{\"name\":\"SYSTEM_NAMESPACE\"}":{".":{},"f:name":{},"f:valueFrom":{".":{},"f:fieldRef":{".":{},"f:apiVersion":{},"f:fieldPath":{}}}}},"f:image":{},"f:imagePullPolicy":{},"f:livenessProbe":{".":{},"f:failureThreshold":{},"f:httpGet":{".":{},"f:path":{},"f:port":{},"f:scheme":{}},"f:initialDelaySeconds":{},"f:periodSeconds":{},"f:successThreshold":{},"f:timeoutSeconds":{}},"f:name":{},"f:ports":{".":{},"k:{\"containerPort\":8080,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}},"k:{\"containerPort\":9090,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}}},"f:readinessProbe":{".":{},"f:failureThreshold":{},"f:httpGet":{".":{},"f:path":{},"f:port":{},"f:scheme":{}},"f:periodSeconds":{},"f:successThreshold":{},"f:timeoutSeconds":{}},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-01-12T09:59:01Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.1.9\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"imc-dispatcher-token-dwmfc","secret":{"secretName":"imc-dispatcher-token-dwmfc","defaultMode":420}}],"containers":[{"name":"dispatcher","image":"kind.local/knative.dev/eventing/cmd/in_memory/channel_dispatcher:8760c84ce6cbc008658d839bbfd4dfcb9650d8bafeeb4d9d2d2f3aa0820ec542","ports":[{"name":"http","containerPort":8080,"protocol":"TCP"},{"name":"metrics","containerPort":9090,"protocol":"TCP"}],"env":[{"name":"CONFIG_LOGGING_NAME","value":"config-logging"},{"name":"CONFIG_OBSERVABILITY_NAME","value":"config-observability"},{"name":"METRICS_DOMAIN","value":"knative.dev/inmemorychannel-dispatcher"},{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"CONTAINER_NAME","value":"dispatcher"}],"resources":{},"volumeMounts":[{"name":"imc-dispatcher-token-dwmfc","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"livenessProbe":{"httpGet":{"path":"/healthz","port":8080,"scheme":"HTTP"},"initialDelaySeconds":5,"timeoutSeconds":1,"periodSeconds":2,"successThreshold":1,"failureThreshold":3},"readinessProbe":{"httpGet":{"path":"/healthz","port":8080,"scheme":"HTTP"},"timeoutSeconds":1,"periodSeconds":2,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"imc-dispatcher","serviceAccount":"imc-dispatcher","nodeName":"kind-worker","securityContext":{},"affinity":{"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":100,"podAffinityTerm":{"labelSelector":{"matchLabels":{"messaging.knative.dev/channel":"in-memory-channel","messaging.knative.dev/role":"dispatcher"}},"topologyKey":"kubernetes.io/hostname"}}]}},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T09:58:58Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T09:59:01Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T09:59:01Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T09:58:57Z"}],"hostIP":"172.18.0.2","podIP":"10.244.1.9","podIPs":[{"ip":"10.244.1.9"}],"startTime":"2021-01-12T09:58:58Z","containerStatuses":[{"name":"dispatcher","state":{"running":{"startedAt":"2021-01-12T09:58:59Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"kind.local/knative.dev/eventing/cmd/in_memory/channel_dispatcher:8760c84ce6cbc008658d839bbfd4dfcb9650d8bafeeb4d9d2d2f3aa0820ec542","imageID":"sha256:79923e21da63842826fb533d9cf02634fbbff3d5592f40d8bb597f1f2dad8349","containerID":"containerd://92dc143c1a270e950fcb1fbbf03c202010a7177405dda2e9641cfe4109047914","started":true}],"qosClass":"BestEffort"}}]}
I0112 10:01:29.251340   26156 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dmt-broker-controller-5b755c9445-n5gcp'
I0112 10:01:29.253561   26156 round_trippers.go:443] GET https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dmt-broker-controller-5b755c9445-n5gcp 200 OK in 2 milliseconds
I0112 10:01:29.253589   26156 round_trippers.go:449] Response Headers:
I0112 10:01:29.253593   26156 round_trippers.go:452]     Cache-Control: no-cache, private
I0112 10:01:29.253596   26156 round_trippers.go:452]     Content-Type: application/json
I0112 10:01:29.253600   26156 round_trippers.go:452]     Date: Tue, 12 Jan 2021 10:01:29 GMT
I0112 10:01:29.253698   26156 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"3944"},"items":[{"metadata":{"name":"mt-broker-controller-5b755c9445-n5gcp","generateName":"mt-broker-controller-5b755c9445-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/mt-broker-controller-5b755c9445-n5gcp","uid":"e254ee15-f7bb-468f-a88b-7ad4dd9b91c4","resourceVersion":"2714","creationTimestamp":"2021-01-12T09:59:57Z","labels":{"app":"mt-broker-controller","eventing.knative.dev/release":"devel","pod-template-hash":"5b755c9445"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"mt-broker-controller-5b755c9445","uid":"22a7054a-d532-47bc-a1f4-f74e7df14440","controller":true,"blockOwnerDeletion":true}],"managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2021-01-12T09:59:57Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:app":{},"f:eventing.knative.dev/release":{},"f:pod-template-hash":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"22a7054a-d532-47bc-a1f4-f74e7df14440\"}":{".":{},"f:apiVersion":{},"f:blockOwnerDeletion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:spec":{"f:affinity":{".":{},"f:podAntiAffinity":{".":{},"f:preferredDuringSchedulingIgnoredDuringExecution":{}}},"f:containers":{"k:{\"name\":\"mt-broker-controller\"}":{".":{},"f:env":{".":{},"k:{\"name\":\"BROKER_INJECTION_DEFAULT\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"CONFIG_LOGGING_NAME\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"CONFIG_OBSERVABILITY_NAME\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"METRICS_DOMAIN\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"POD_NAME\"}":{".":{},"f:name":{},"f:valueFrom":{".":{},"f:fieldRef":{".":{},"f:apiVersion":{},"f:fieldPath":{}}}},"k:{\"name\":\"SYSTEM_NAMESPACE\"}":{".":{},"f:name":{},"f:valueFrom":{".":{},"f:fieldRef":{".":{},"f:apiVersion":{},"f:fieldPath":{}}}}},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:ports":{".":{},"k:{\"containerPort\":8008,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}},"k:{\"containerPort\":9090,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}}},"f:resources":{".":{},"f:requests":{".":{},"f:cpu":{},"f:memory":{}}},"f:securityContext":{".":{},"f:allowPrivilegeEscalation":{}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-01-12T10:00:04Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.1.19\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"eventing-controller-token-zbv2z","secret":{"secretName":"eventing-controller-token-zbv2z","defaultMode":420}}],"containers":[{"name":"mt-broker-controller","image":"kind.local/knative.dev/eventing/cmd/mtchannel_broker:bb5df6b90a0333c8ff41bac30b6cb121322b5b72cbe820014f076d68f769d4c8","ports":[{"name":"metrics","containerPort":9090,"protocol":"TCP"},{"name":"profiling","containerPort":8008,"protocol":"TCP"}],"env":[{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"CONFIG_LOGGING_NAME","value":"config-logging"},{"name":"CONFIG_OBSERVABILITY_NAME","value":"config-observability"},{"name":"METRICS_DOMAIN","value":"knative.dev/eventing"},{"name":"BROKER_INJECTION_DEFAULT","value":"false"},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}}],"resources":{"requests":{"cpu":"100m","memory":"100Mi"}},"volumeMounts":[{"name":"eventing-controller-token-zbv2z","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent","securityContext":{"allowPrivilegeEscalation":false}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"eventing-controller","serviceAccount":"eventing-controller","nodeName":"kind-worker","securityContext":{},"affinity":{"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":100,"podAffinityTerm":{"labelSelector":{"matchLabels":{"app":"mt-broker-controller"}},"topologyKey":"kubernetes.io/hostname"}}]}},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T09:59:57Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T10:00:03Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T10:00:03Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T09:59:57Z"}],"hostIP":"172.18.0.2","podIP":"10.244.1.19","podIPs":[{"ip":"10.244.1.19"}],"startTime":"2021-01-12T09:59:57Z","containerStatuses":[{"name":"mt-broker-controller","state":{"running":{"startedAt":"2021-01-12T10:00:03Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"kind.local/knative.dev/eventing/cmd/mtchannel_broker:bb5df6b90a0333c8ff41bac30b6cb121322b5b72cbe820014f076d68f769d4c8","imageID":"sha256:50af5a38ed83ff1f8ab299d90a4ace8f71622fb295c91bd88f59a147ffc31912","containerID":"containerd://3be031e553b3e8b8b3dc5fbf2879cf2eb30eae70764d05a86d9df2c1455b4f9e","started":true}],"qosClass":"Burstable"}}]}
I0112 10:01:29.254835   26156 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dmt-broker-filter-56864974c6-mhndh'
I0112 10:01:29.257245   26156 round_trippers.go:443] GET https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dmt-broker-filter-56864974c6-mhndh 200 OK in 2 milliseconds
I0112 10:01:29.257255   26156 round_trippers.go:449] Response Headers:
I0112 10:01:29.257259   26156 round_trippers.go:452]     Date: Tue, 12 Jan 2021 10:01:29 GMT
I0112 10:01:29.257263   26156 round_trippers.go:452]     Cache-Control: no-cache, private
I0112 10:01:29.257266   26156 round_trippers.go:452]     Content-Type: application/json
I0112 10:01:29.257336   26156 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"3944"},"items":[{"metadata":{"name":"mt-broker-filter-56864974c6-mhndh","generateName":"mt-broker-filter-56864974c6-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/mt-broker-filter-56864974c6-mhndh","uid":"1ad381c0-d16f-45ea-adb5-65c2eeda678f","resourceVersion":"1556","creationTimestamp":"2021-01-12T09:58:53Z","labels":{"eventing.knative.dev/brokerRole":"filter","eventing.knative.dev/release":"devel","pod-template-hash":"56864974c6"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"mt-broker-filter-56864974c6","uid":"af87df77-d4f6-44e4-96ac-f270802e29c5","controller":true,"blockOwnerDeletion":true}],"managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2021-01-12T09:58:53Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:eventing.knative.dev/brokerRole":{},"f:eventing.knative.dev/release":{},"f:pod-template-hash":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"af87df77-d4f6-44e4-96ac-f270802e29c5\"}":{".":{},"f:apiVersion":{},"f:blockOwnerDeletion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:spec":{"f:containers":{"k:{\"name\":\"filter\"}":{".":{},"f:env":{".":{},"k:{\"name\":\"CONFIG_LOGGING_NAME\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"CONFIG_OBSERVABILITY_NAME\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"CONTAINER_NAME\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"FILTER_PORT\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"METRICS_DOMAIN\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"NAMESPACE\"}":{".":{},"f:name":{},"f:valueFrom":{".":{},"f:fieldRef":{".":{},"f:apiVersion":{},"f:fieldPath":{}}}},"k:{\"name\":\"POD_NAME\"}":{".":{},"f:name":{},"f:valueFrom":{".":{},"f:fieldRef":{".":{},"f:apiVersion":{},"f:fieldPath":{}}}},"k:{\"name\":\"SYSTEM_NAMESPACE\"}":{".":{},"f:name":{},"f:valueFrom":{".":{},"f:fieldRef":{".":{},"f:apiVersion":{},"f:fieldPath":{}}}}},"f:image":{},"f:imagePullPolicy":{},"f:livenessProbe":{".":{},"f:failureThreshold":{},"f:httpGet":{".":{},"f:path":{},"f:port":{},"f:scheme":{}},"f:initialDelaySeconds":{},"f:periodSeconds":{},"f:successThreshold":{},"f:timeoutSeconds":{}},"f:name":{},"f:ports":{".":{},"k:{\"containerPort\":8080,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}},"k:{\"containerPort\":9092,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}}},"f:readinessProbe":{".":{},"f:failureThreshold":{},"f:httpGet":{".":{},"f:path":{},"f:port":{},"f:scheme":{}},"f:periodSeconds":{},"f:successThreshold":{},"f:timeoutSeconds":{}},"f:resources":{".":{},"f:requests":{".":{},"f:cpu":{},"f:memory":{}}},"f:securityContext":{".":{},"f:allowPrivilegeEscalation":{}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-01-12T09:58:58Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.1.4\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"mt-broker-filter-token-vzvsr","secret":{"secretName":"mt-broker-filter-token-vzvsr","defaultMode":420}}],"containers":[{"name":"filter","image":"kind.local/knative.dev/eventing/cmd/mtbroker/filter:8501edf5fc2ba4801350f9c1ae59f630c241e43b2b8b878c8aef4a82f4addb9a","ports":[{"name":"http","containerPort":8080,"protocol":"TCP"},{"name":"metrics","containerPort":9092,"protocol":"TCP"}],"env":[{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"CONTAINER_NAME","value":"filter"},{"name":"CONFIG_LOGGING_NAME","value":"config-logging"},{"name":"CONFIG_OBSERVABILITY_NAME","value":"config-observability"},{"name":"METRICS_DOMAIN","value":"knative.dev/internal/eventing"},{"name":"FILTER_PORT","value":"8080"}],"resources":{"requests":{"cpu":"100m","memory":"100Mi"}},"volumeMounts":[{"name":"mt-broker-filter-token-vzvsr","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"livenessProbe":{"httpGet":{"path":"/healthz","port":8080,"scheme":"HTTP"},"initialDelaySeconds":5,"timeoutSeconds":1,"periodSeconds":2,"successThreshold":1,"failureThreshold":3},"readinessProbe":{"httpGet":{"path":"/healthz","port":8080,"scheme":"HTTP"},"timeoutSeconds":1,"periodSeconds":2,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent","securityContext":{"allowPrivilegeEscalation":false}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"mt-broker-filter","serviceAccount":"mt-broker-filter","nodeName":"kind-worker","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T09:58:54Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T09:58:57Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T09:58:57Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T09:58:54Z"}],"hostIP":"172.18.0.2","podIP":"10.244.1.4","podIPs":[{"ip":"10.244.1.4"}],"startTime":"2021-01-12T09:58:54Z","containerStatuses":[{"name":"filter","state":{"running":{"startedAt":"2021-01-12T09:58:57Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"kind.local/knative.dev/eventing/cmd/mtbroker/filter:8501edf5fc2ba4801350f9c1ae59f630c241e43b2b8b878c8aef4a82f4addb9a","imageID":"sha256:8e1d5d28a422c59142cc6373717fe43cfe789d64aea1934182559696aea4ed85","containerID":"containerd://73f9a11cb6a9fb688967d980eee02527e659d6ec2e7d36ed28075949ede29807","started":true}],"qosClass":"Burstable"}}]}
I0112 10:01:29.258291   26156 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dmt-broker-ingress-6db4d947c6-2kpxk'
I0112 10:01:29.259776   26156 round_trippers.go:443] GET https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dmt-broker-ingress-6db4d947c6-2kpxk 200 OK in 1 milliseconds
I0112 10:01:29.259785   26156 round_trippers.go:449] Response Headers:
I0112 10:01:29.259802   26156 round_trippers.go:452]     Date: Tue, 12 Jan 2021 10:01:29 GMT
I0112 10:01:29.259807   26156 round_trippers.go:452]     Cache-Control: no-cache, private
I0112 10:01:29.259810   26156 round_trippers.go:452]     Content-Type: application/json
I0112 10:01:29.259956   26156 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"3944"},"items":[{"metadata":{"name":"mt-broker-ingress-6db4d947c6-2kpxk","generateName":"mt-broker-ingress-6db4d947c6-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/mt-broker-ingress-6db4d947c6-2kpxk","uid":"7a1d7410-24ca-4080-b618-232f987cb35d","resourceVersion":"1640","creationTimestamp":"2021-01-12T09:58:56Z","labels":{"eventing.knative.dev/brokerRole":"ingress","eventing.knative.dev/release":"devel","pod-template-hash":"6db4d947c6"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"mt-broker-ingress-6db4d947c6","uid":"dc53aaa5-1236-408f-a7f1-217476bb6a59","controller":true,"blockOwnerDeletion":true}],"managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2021-01-12T09:58:56Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:eventing.knative.dev/brokerRole":{},"f:eventing.knative.dev/release":{},"f:pod-template-hash":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"dc53aaa5-1236-408f-a7f1-217476bb6a59\"}":{".":{},"f:apiVersion":{},"f:blockOwnerDeletion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:spec":{"f:containers":{"k:{\"name\":\"ingress\"}":{".":{},"f:env":{".":{},"k:{\"name\":\"CONFIG_LOGGING_NAME\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"CONFIG_OBSERVABILITY_NAME\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"CONTAINER_NAME\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"INGRESS_PORT\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"METRICS_DOMAIN\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"NAMESPACE\"}":{".":{},"f:name":{},"f:valueFrom":{".":{},"f:fieldRef":{".":{},"f:apiVersion":{},"f:fieldPath":{}}}},"k:{\"name\":\"POD_NAME\"}":{".":{},"f:name":{},"f:valueFrom":{".":{},"f:fieldRef":{".":{},"f:apiVersion":{},"f:fieldPath":{}}}},"k:{\"name\":\"SYSTEM_NAMESPACE\"}":{".":{},"f:name":{},"f:valueFrom":{".":{},"f:fieldRef":{".":{},"f:apiVersion":{},"f:fieldPath":{}}}}},"f:image":{},"f:imagePullPolicy":{},"f:livenessProbe":{".":{},"f:failureThreshold":{},"f:httpGet":{".":{},"f:path":{},"f:port":{},"f:scheme":{}},"f:initialDelaySeconds":{},"f:periodSeconds":{},"f:successThreshold":{},"f:timeoutSeconds":{}},"f:name":{},"f:ports":{".":{},"k:{\"containerPort\":8080,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}},"k:{\"containerPort\":9092,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}}},"f:readinessProbe":{".":{},"f:failureThreshold":{},"f:httpGet":{".":{},"f:path":{},"f:port":{},"f:scheme":{}},"f:periodSeconds":{},"f:successThreshold":{},"f:timeoutSeconds":{}},"f:resources":{".":{},"f:requests":{".":{},"f:cpu":{},"f:memory":{}}},"f:securityContext":{".":{},"f:allowPrivilegeEscalation":{}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-01-12T09:59:01Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.1.5\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"mt-broker-ingress-token-6gvdp","secret":{"secretName":"mt-broker-ingress-token-6gvdp","defaultMode":420}}],"containers":[{"name":"ingress","image":"kind.local/knative.dev/eventing/cmd/mtbroker/ingress:4f670eba8a372765410d5b52a57ee00c1d175712eff9f521c29fe9d2d251fbb3","ports":[{"name":"http","containerPort":8080,"protocol":"TCP"},{"name":"metrics","containerPort":9092,"protocol":"TCP"}],"env":[{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"CONTAINER_NAME","value":"ingress"},{"name":"CONFIG_LOGGING_NAME","value":"config-logging"},{"name":"CONFIG_OBSERVABILITY_NAME","value":"config-observability"},{"name":"METRICS_DOMAIN","value":"knative.dev/internal/eventing"},{"name":"INGRESS_PORT","value":"8080"}],"resources":{"requests":{"cpu":"100m","memory":"100Mi"}},"volumeMounts":[{"name":"mt-broker-ingress-token-6gvdp","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"livenessProbe":{"httpGet":{"path":"/healthz","port":8080,"scheme":"HTTP"},"initialDelaySeconds":5,"timeoutSeconds":1,"periodSeconds":2,"successThreshold":1,"failureThreshold":3},"readinessProbe":{"httpGet":{"path":"/healthz","port":8080,"scheme":"HTTP"},"timeoutSeconds":1,"periodSeconds":2,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent","securityContext":{"allowPrivilegeEscalation":false}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"mt-broker-ingress","serviceAccount":"mt-broker-ingress","nodeName":"kind-worker","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T09:58:56Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T09:59:01Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T09:59:01Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T09:58:56Z"}],"hostIP":"172.18.0.2","podIP":"10.244.1.5","podIPs":[{"ip":"10.244.1.5"}],"startTime":"2021-01-12T09:58:56Z","containerStatuses":[{"name":"ingress","state":{"running":{"startedAt":"2021-01-12T09:58:58Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"kind.local/knative.dev/eventing/cmd/mtbroker/ingress:4f670eba8a372765410d5b52a57ee00c1d175712eff9f521c29fe9d2d251fbb3","imageID":"sha256:d44dec927dc9ca6b33f00514b12204cca3e0c94a7d6523952e475340cc3eaa45","containerID":"containerd://774015d44c9d28709681a1b8f438f9ed1af2a441d8447f0ad7eb7467a67d9cc4","started":true}],"qosClass":"Burstable"}}]}
I0112 10:01:29.260944   26156 round_trippers.go:423] curl -k -v -XGET  -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" -H "Accept: application/json" 'https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dsugar-controller-84b77b5675-hkdsj'
I0112 10:01:29.262569   26156 round_trippers.go:443] GET https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dsugar-controller-84b77b5675-hkdsj 200 OK in 1 milliseconds
I0112 10:01:29.262578   26156 round_trippers.go:449] Response Headers:
I0112 10:01:29.262583   26156 round_trippers.go:452]     Cache-Control: no-cache, private
I0112 10:01:29.262586   26156 round_trippers.go:452]     Content-Type: application/json
I0112 10:01:29.262589   26156 round_trippers.go:452]     Date: Tue, 12 Jan 2021 10:01:29 GMT
I0112 10:01:29.262701   26156 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"3944"},"items":[{"metadata":{"name":"sugar-controller-84b77b5675-hkdsj","generateName":"sugar-controller-84b77b5675-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/sugar-controller-84b77b5675-hkdsj","uid":"a400ee78-8ed6-4a70-9468-c80655d01695","resourceVersion":"2737","creationTimestamp":"2021-01-12T09:59:57Z","labels":{"eventing.knative.dev/role":"sugar-controller","pod-template-hash":"84b77b5675"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"sugar-controller-84b77b5675","uid":"6cfce264-5610-4279-b116-7a8eaf847ea1","controller":true,"blockOwnerDeletion":true}],"managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2021-01-12T09:59:57Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:eventing.knative.dev/role":{},"f:pod-template-hash":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"6cfce264-5610-4279-b116-7a8eaf847ea1\"}":{".":{},"f:apiVersion":{},"f:blockOwnerDeletion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:spec":{"f:containers":{"k:{\"name\":\"controller\"}":{".":{},"f:env":{".":{},"k:{\"name\":\"CONFIG_LOGGING_NAME\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"CONFIG_OBSERVABILITY_NAME\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"METRICS_DOMAIN\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"POD_NAME\"}":{".":{},"f:name":{},"f:valueFrom":{".":{},"f:fieldRef":{".":{},"f:apiVersion":{},"f:fieldPath":{}}}},"k:{\"name\":\"SYSTEM_NAMESPACE\"}":{".":{},"f:name":{},"f:valueFrom":{".":{},"f:fieldRef":{".":{},"f:apiVersion":{},"f:fieldPath":{}}}}},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:ports":{".":{},"k:{\"containerPort\":8008,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}},"k:{\"containerPort\":9090,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}}},"f:resources":{},"f:securityContext":{".":{},"f:allowPrivilegeEscalation":{}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-01-12T10:00:04Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.1.18\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"eventing-controller-token-zbv2z","secret":{"secretName":"eventing-controller-token-zbv2z","defaultMode":420}}],"containers":[{"name":"controller","image":"kind.local/knative.dev/eventing/cmd/sugar_controller:9ded56c077e7ad33f29ad4aa513d9bb146a8e73a841c83b6448b4b97bcb2c9ec","ports":[{"name":"metrics","containerPort":9090,"protocol":"TCP"},{"name":"profiling","containerPort":8008,"protocol":"TCP"}],"env":[{"name":"CONFIG_LOGGING_NAME","value":"config-logging"},{"name":"CONFIG_OBSERVABILITY_NAME","value":"config-observability"},{"name":"METRICS_DOMAIN","value":"knative.dev/sugar-controller"},{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}}],"resources":{},"volumeMounts":[{"name":"eventing-controller-token-zbv2z","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"allowPrivilegeEscalation":false}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"eventing-controller","serviceAccount":"eventing-controller","nodeName":"kind-worker","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T09:59:58Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T10:00:04Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T10:00:04Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T09:59:57Z"}],"hostIP":"172.18.0.2","podIP":"10.244.1.18","podIPs":[{"ip":"10.244.1.18"}],"startTime":"2021-01-12T09:59:58Z","containerStatuses":[{"name":"controller","state":{"running":{"startedAt":"2021-01-12T10:00:03Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"kind.local/knative.dev/eventing/cmd/sugar_controller:9ded56c077e7ad33f29ad4aa513d9bb146a8e73a841c83b6448b4b97bcb2c9ec","imageID":"sha256:253637b7806b4c97b73e834ae273aa556041f90c09b60563400051efc94904e0","containerID":"containerd://6afd6540150eb896e7736042d41f0d77d676e2fc5e3402d0f4066e0517efa424","started":true}],"qosClass":"BestEffort"}}]}
I0112 10:01:29.263749   26156 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dzipkin-764f9875c6-w5rg8'
I0112 10:01:29.265209   26156 round_trippers.go:443] GET https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dzipkin-764f9875c6-w5rg8 200 OK in 1 milliseconds
I0112 10:01:29.265219   26156 round_trippers.go:449] Response Headers:
I0112 10:01:29.265223   26156 round_trippers.go:452]     Cache-Control: no-cache, private
I0112 10:01:29.265228   26156 round_trippers.go:452]     Content-Type: application/json
I0112 10:01:29.265232   26156 round_trippers.go:452]     Date: Tue, 12 Jan 2021 10:01:29 GMT
I0112 10:01:29.265526   26156 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"3944"},"items":[{"metadata":{"name":"zipkin-764f9875c6-w5rg8","generateName":"zipkin-764f9875c6-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/zipkin-764f9875c6-w5rg8","uid":"a3f3576a-a77e-4aab-bc3c-fe752e5a0e77","resourceVersion":"2261","creationTimestamp":"2021-01-12T09:59:10Z","labels":{"app":"zipkin","pod-template-hash":"764f9875c6"},"annotations":{"sidecar.istio.io/inject":"false"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"zipkin-764f9875c6","uid":"d0fb1f80-0502-4b03-acc4-fbbcc541788c","controller":true,"blockOwnerDeletion":true}],"managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2021-01-12T09:59:10Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:sidecar.istio.io/inject":{}},"f:generateName":{},"f:labels":{".":{},"f:app":{},"f:pod-template-hash":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"d0fb1f80-0502-4b03-acc4-fbbcc541788c\"}":{".":{},"f:apiVersion":{},"f:blockOwnerDeletion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:spec":{"f:containers":{"k:{\"name\":\"zipkin\"}":{".":{},"f:env":{".":{},"k:{\"name\":\"POD_NAMESPACE\"}":{".":{},"f:name":{},"f:valueFrom":{".":{},"f:fieldRef":{".":{},"f:apiVersion":{},"f:fieldPath":{}}}}},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:ports":{".":{},"k:{\"containerPort\":9411,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:protocol":{}}},"f:resources":{".":{},"f:limits":{".":{},"f:memory":{}},"f:requests":{".":{},"f:memory":{}}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-01-12T09:59:34Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.1.11\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"default-token-sptbl","secret":{"secretName":"default-token-sptbl","defaultMode":420}}],"containers":[{"name":"zipkin","image":"docker.io/openzipkin/zipkin:2.13.0","ports":[{"containerPort":9411,"protocol":"TCP"}],"env":[{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}}],"resources":{"limits":{"memory":"1000Mi"},"requests":{"memory":"256Mi"}},"volumeMounts":[{"name":"default-token-sptbl","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"kind-worker","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T09:59:10Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T09:59:34Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T09:59:34Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T09:59:10Z"}],"hostIP":"172.18.0.2","podIP":"10.244.1.11","podIPs":[{"ip":"10.244.1.11"}],"startTime":"2021-01-12T09:59:10Z","containerStatuses":[{"name":"zipkin","state":{"running":{"startedAt":"2021-01-12T09:59:33Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"docker.io/openzipkin/zipkin:2.13.0","imageID":"docker.io/openzipkin/zipkin@sha256:2f3139c6c2975ce4e53717fc07da628efdf2c0d2b3dbd7de890369b9450fd21d","containerID":"containerd://de6b9781bb120e5f3799753d0940bdef1fec3072b0189360012702d4215b621b","started":true}],"qosClass":"Burstable"}}]}
F0112 10:01:29.266090   26156 helpers.go:115] timed out waiting for the condition on pods/chaosduck-5d8694d86b-7qf9q
timed out waiting for the condition on pods/eventing-webhook-9d555fbd4-hnnbm
goroutine 1 [running]:
k8s.io/klog/v2.stacks(0xc000010001, 0xc00086ca20, 0xc2, 0x114)
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:996 +0xb9
k8s.io/klog/v2.(*loggingT).output(0x4d2aec0, 0xc000000003, 0x0, 0x0, 0xc0007f4cb0, 0x48a9af4, 0xa, 0x73, 0x41d400)
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:945 +0x191
k8s.io/klog/v2.(*loggingT).printDepth(0x4d2aec0, 0x3, 0x0, 0x0, 0x2, 0xc0011a7a10, 0x1, 0x1)
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:718 +0x165
k8s.io/klog/v2.FatalDepth(...)
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:1449
k8s.io/kubectl/pkg/cmd/util.fatal(0xc000db21e0, 0x94, 0x1)
	/go/src/github.com/openshift/oc/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:93 +0x1f0
k8s.io/kubectl/pkg/cmd/util.checkErr(0x3482280, 0xc0002189e0, 0x31c65e8)
	/go/src/github.com/openshift/oc/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:177 +0x8b5
k8s.io/kubectl/pkg/cmd/util.CheckErr(...)
	/go/src/github.com/openshift/oc/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:115
k8s.io/kubectl/pkg/cmd/wait.NewCmdWait.func1(0xc00104e000, 0xc001026e00, 0x1, 0x8)
	/go/src/github.com/openshift/oc/vendor/k8s.io/kubectl/pkg/cmd/wait/wait.go:120 +0xad
github.com/spf13/cobra.(*Command).execute(0xc00104e000, 0xc001026d80, 0x8, 0x8, 0xc00104e000, 0xc001026d80)
	/go/src/github.com/openshift/oc/vendor/github.com/spf13/cobra/command.go:846 +0x2c2
github.com/spf13/cobra.(*Command).ExecuteC(0xc0000e2000, 0x7, 0xc0000e2000, 0x7)
	/go/src/github.com/openshift/oc/vendor/github.com/spf13/cobra/command.go:950 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
	/go/src/github.com/openshift/oc/vendor/github.com/spf13/cobra/command.go:887
main.main()
	/go/src/github.com/openshift/oc/cmd/oc/oc.go:110 +0x885

goroutine 6 [chan receive, 1 minutes]:
k8s.io/klog/v2.(*loggingT).flushDaemon(0x4d2aec0)
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:1131 +0x8b
created by k8s.io/klog/v2.init.0
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:416 +0xd8

goroutine 22 [select, 1 minutes]:
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x31c6510, 0x347e280, 0xc000996000, 0x1, 0xc0000a6360)
	/go/src/github.com/openshift/oc/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:167 +0x149
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x31c6510, 0x12a05f200, 0x0, 0xc0007f8d01, 0xc0000a6360)
	/go/src/github.com/openshift/oc/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/go/src/github.com/openshift/oc/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
k8s.io/apimachinery/pkg/util/wait.Forever(0x31c6510, 0x12a05f200)
	/go/src/github.com/openshift/oc/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:81 +0x4f
created by k8s.io/component-base/logs.InitLogs
	/go/src/github.com/openshift/oc/vendor/k8s.io/component-base/logs/logs.go:58 +0x8a

goroutine 10 [chan receive, 1 minutes]:
k8s.io/klog.(*loggingT).flushDaemon(0x4d2ade0)
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/klog.go:1010 +0x8b
created by k8s.io/klog.init.0
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/klog.go:411 +0xd8

goroutine 134 [IO wait]:
internal/poll.runtime_pollWait(0x7f5bd512cf78, 0x72, 0x3482f20)
	/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/runtime/netpoll.go:220 +0x55
internal/poll.(*pollDesc).wait(0xc0014de418, 0x72, 0xc00153c000, 0x5797, 0x5797)
	/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0014de400, 0xc00153c000, 0x5797, 0x5797, 0x0, 0x0, 0x0)
	/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/internal/poll/fd_unix.go:159 +0x1b1
net.(*netFD).Read(0xc0014de400, 0xc00153c000, 0x5797, 0x5797, 0x203000, 0x78699b, 0xc0014d72e0)
	/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc0003033f8, 0xc00153c000, 0x5797, 0x5797, 0x0, 0x0, 0x0)
	/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/net/net.go:182 +0x8e
crypto/tls.(*atLeastReader).Read(0xc0002182e0, 0xc00153c000, 0x5797, 0x5797, 0x1a, 0x5792, 0xc001163710)
	/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/crypto/tls/conn.go:779 +0x62
bytes.(*Buffer).ReadFrom(0xc0014d7400, 0x347a260, 0xc0002182e0, 0x41d785, 0x2bdcba0, 0x2ed2dc0)
	/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/bytes/buffer.go:204 +0xb1
crypto/tls.(*Conn).readFromUntil(0xc0014d7180, 0x347f7c0, 0xc0003033f8, 0x5, 0xc0003033f8, 0x9)
	/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/crypto/tls/conn.go:801 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc0014d7180, 0x0, 0x0, 0xc001163d18)
	/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/crypto/tls/conn.go:608 +0x115
crypto/tls.(*Conn).readRecord(...)
	/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/crypto/tls/conn.go:576
crypto/tls.(*Conn).Read(0xc0014d7180, 0xc00150e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/crypto/tls/conn.go:1252 +0x15f
bufio.(*Reader).Read(0xc0014f5920, 0xc0001948f8, 0x9, 0x9, 0xc001163d18, 0x31c9f00, 0x9bbceb)
	/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/bufio/bufio.go:227 +0x222
io.ReadAtLeast(0x347a000, 0xc0014f5920, 0xc0001948f8, 0x9, 0x9, 0x9, 0xc000062070, 0x0, 0x347a520)
	/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/io/io.go:314 +0x87
io.ReadFull(...)
	/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/io/io.go:333
golang.org/x/net/http2.readFrameHeader(0xc0001948f8, 0x9, 0x9, 0x347a000, 0xc0014f5920, 0x0, 0x0, 0xc00168e780, 0x0)
	/go/src/github.com/openshift/oc/vendor/golang.org/x/net/http2/frame.go:237 +0x89
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0001948c0, 0xc00168e780, 0x0, 0x0, 0x0)
	/go/src/github.com/openshift/oc/vendor/golang.org/x/net/http2/frame.go:492 +0xa5
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc001163fa8, 0x0, 0x0)
	/go/src/github.com/openshift/oc/vendor/golang.org/x/net/http2/transport.go:1794 +0xd8
golang.org/x/net/http2.(*ClientConn).readLoop(0xc001382600)
	/go/src/github.com/openshift/oc/vendor/golang.org/x/net/http2/transport.go:1716 +0x6f
created by golang.org/x/net/http2.(*Transport).newClientConn
	/go/src/github.com/openshift/oc/vendor/golang.org/x/net/http2/transport.go:695 +0x66e
pod/imc-controller-7586dff887-t68d6 condition met
pod/imc-dispatcher-6d95d7b95f-dlkgb condition met
pod/mt-broker-controller-5b755c9445-n5gcp condition met
pod/mt-broker-filter-56864974c6-mhndh condition met
pod/mt-broker-ingress-6db4d947c6-2kpxk condition met
pod/sugar-controller-84b77b5675-hkdsj condition met
pod/zipkin-764f9875c6-w5rg8 condition met
Error: Process completed with exit code 255.

Yet, after collecting the system diagnostics we see this:

Run kubectl -n${SYSTEM_NAMESPACE} get pods
NAME                                    READY   STATUS    RESTARTS   AGE
eventing-controller-67d57bbcb9-c7n6n    1/1     Running   0          93s
eventing-webhook-9d555fbd4-st8fw        1/1     Running   0          93s
imc-controller-7586dff887-t68d6         1/1     Running   0          2m33s
imc-dispatcher-6d95d7b95f-dlkgb         1/1     Running   0          2m33s
mt-broker-controller-5b755c9445-n5gcp   1/1     Running   0          93s
mt-broker-filter-56864974c6-mhndh       1/1     Running   0          2m37s
mt-broker-ingress-6db4d947c6-2kpxk      1/1     Running   0          2m34s
sugar-controller-84b77b5675-hkdsj       1/1     Running   0          93s
zipkin-764f9875c6-w5rg8                 1/1     Running   0          2m20s

A clear and concise description of what the bug is.

Expected behavior
A clear and concise description of what you expected to happen.

To Reproduce
Steps to reproduce the behavior.

Knative release version
HEAD

Additional context
Add any other context about the problem here such as proposed priority
@vaikas vaikas added the kind/bug Categorizes issue or PR as related to a bug. label Jan 12, 2021
@vaikas vaikas mentioned this issue Jan 12, 2021
Merged
@vaikas
Copy link
Contributor Author

vaikas commented Jan 13, 2021

Failed again here:
https://github.com/knative/eventing/pull/4729/checks?check_run_id=1694274951

Run kubectl -v 9 wait pod --for=condition=Ready -n ${SYSTEM_NAMESPACE} -l '!job-name'
  kubectl -v 9 wait pod --for=condition=Ready -n ${SYSTEM_NAMESPACE} -l '!job-name'
  shell: /bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    GOPATH: /home/runner/work/eventing/eventing
    GO111MODULE: off
    KO_DOCKER_REPO: kind.local
    SYSTEM_NAMESPACE: knative-eventing
    CLUSTER_SUFFIX: c482504125.local
    ARTIFACTS: /home/runner/work/eventing/eventing/artifacts
    GOROOT: /opt/hostedtoolcache/go/1.15.6/x64
I0113 10:36:52.886381   25628 loader.go:375] Config loaded from file:  /home/runner/.kube/config
I0113 10:36:52.892363   25628 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?labelSelector=%21job-name'
I0113 10:36:52.914279   25628 round_trippers.go:443] GET https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?labelSelector=%21job-name 200 OK in 21 milliseconds
I0113 10:36:52.914645   25628 round_trippers.go:449] Response Headers:
I0113 10:36:52.914658   25628 round_trippers.go:452]     Cache-Control: no-cache, private
I0113 10:36:52.914662   25628 round_trippers.go:452]     Content-Type: application/json
I0113 10:36:52.914665   25628 round_trippers.go:452]     Date: Wed, 13 Jan 2021 10:36:52 GMT
I0113 10:36:52.915130   25628 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"2902"},"items":[{"metadata":{"name":"chaosduck-f665c9dc7-zkdbf","generateName":"chaosduck-f665c9dc7-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/chaosduck-f665c9dc7-zkdbf","uid":"c2c2d30a-a755-4a67-9289-beb684e8fbce","resourceVersion":"1892","creationTimestamp":"2021-01-13T10:35:41Z","deletionTimestamp":"2021-01-13T10:36:14Z","deletionGracePeriodSeconds":30,"labels":{"app":"chaosduck","pod-template-hash":"f665c9dc7"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"chaosduck-f665c9dc7","uid":"be93d508-83a9-428f-94a5-a218a6835edc","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"chaosduck-token-lwvbb","secret":{"secretName":"chaosduck-token-lwvbb","defaultMode":420}}],"containers":[{"name":"chaosduck","image":"kind.local/knative.dev/pkg/leaderelection/chaosduck:967fb52fb64b86752de93a2833a15b705c07660c858b0b27a4fd4fd89094663a","args":["-disable=imc-controller","-disable=imc-dispatcher"],"env":[{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}}],"resources":{},"volumeMounts":[{"name":"chaosduck-token-lwvbb","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"allowPrivilegeEscalation":false}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"chaosduck","serviceAccount":"chaosduck","nodeName":"kind-worker","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:42Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:45Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:45Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:41Z"}],"hostIP":"172.18.0.3","podIP":"10.244.1.10","podIPs":[{"ip":"10.244.1.10"}],"startTime":"2021-01-13T10:35:42Z","containerStatuses":[{"name":"chaosduck","state":{"running":{"startedAt":"2021-01-13T10:35:44Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"kind.local/knative.dev/pkg/leaderelection/chaosduck:967fb52fb64b86752de93a2833a15b705c07660c858b0b27a4fd4fd89094663a","imageID":"sha256:9ff310726fe092fe547795899132afed3195fd2651a9202a863738a1682934dd","containerID":"containerd://7d90d940150d5ce8ebb702417f38e4c0f8a4b9dc578f1e374de3c83313cf90dd","started":true}],"qosClass":"BestEffort"}},{"metadata":{"name":"eventing-controller-c895f6756-cf6dg","generateName":"eventing-controller-c895f6756-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/eventing-controller-c895f6756-cf6dg","uid":"3827b98b-f6ac-4551-b24e-f6d336cd8e53","resourceVersion":"2645","creationTimestamp":"2021-01-13T10:36:28Z","labels":{"app":"eventing-controller","eventing.knative.dev/release":"devel","pod-template-hash":"c895f6756"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"eventing-controller-c895f6756","uid":"93116175-b845-4b5c-94fe-9c4668bd0ef5","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"eventing-controller-token-6hxjg","secret":{"secretName":"eventing-controller-token-6hxjg","defaultMode":420}}],"containers":[{"name":"eventing-controller","image":"kind.local/knative.dev/eventing/cmd/controller:a695ea7ccc558e8ab6434dcdd1a7c5cb4eabb84c3cd596235473b2e59ef18158","ports":[{"name":"metrics","containerPort":9090,"protocol":"TCP"},{"name":"profiling","containerPort":8008,"protocol":"TCP"}],"env":[{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"CONFIG_LOGGING_NAME","value":"config-logging"},{"name":"CONFIG_OBSERVABILITY_NAME","value":"config-observability"},{"name":"METRICS_DOMAIN","value":"knative.dev/eventing"},{"name":"APISERVER_RA_IMAGE","value":"kind.local/knative.dev/eventing/cmd/apiserver_receive_adapter:8d4f85c32cc62b2731fd86a859d7305d261f94d6471e2f8f6db165a02c70c888"},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}}],"resources":{"requests":{"cpu":"100m","memory":"100Mi"}},"volumeMounts":[{"name":"eventing-controller-token-6hxjg","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent","securityContext":{"allowPrivilegeEscalation":false}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"eventing-controller","serviceAccount":"eventing-controller","nodeName":"kind-worker","securityContext":{},"affinity":{"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":100,"podAffinityTerm":{"labelSelector":{"matchLabels":{"app":"eventing-controller"}},"topologyKey":"kubernetes.io/hostname"}}]}},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":false},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:30Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:35Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:35Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:28Z"}],"hostIP":"172.18.0.3","podIP":"10.244.1.18","podIPs":[{"ip":"10.244.1.18"}],"startTime":"2021-01-13T10:36:30Z","containerStatuses":[{"name":"eventing-controller","state":{"running":{"startedAt":"2021-01-13T10:36:34Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"kind.local/knative.dev/eventing/cmd/controller:a695ea7ccc558e8ab6434dcdd1a7c5cb4eabb84c3cd596235473b2e59ef18158","imageID":"sha256:c9b859f4ecfcd11881660bcb48e03d44ad10991335bc4aee9d53a4894f148bfc","containerID":"containerd://50c64ef119ae94f0de6df42a6dc16ea384bd1eb5111b95d183d3389ce46016ff","started":true}],"qosClass":"Burstable"}},{"metadata":{"name":"eventing-webhook-bc5f465f4-mv6r6","generateName":"eventing-webhook-bc5f465f4-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/eventing-webhook-bc5f465f4-mv6r6","uid":"f0aed95c-93c4-4ae0-b1a4-add0bd06aac5","resourceVersion":"2595","creationTimestamp":"2021-01-13T10:35:44Z","deletionTimestamp":"2021-01-13T10:41:28Z","deletionGracePeriodSeconds":300,"labels":{"app":"eventing-webhook","pod-template-hash":"bc5f465f4","role":"eventing-webhook"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"eventing-webhook-bc5f465f4","uid":"a3fb1319-83e9-4a2c-927b-339c71893048","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"eventing-webhook-token-cmvdm","secret":{"secretName":"eventing-webhook-token-cmvdm","defaultMode":420}}],"containers":[{"name":"eventing-webhook","image":"kind.local/knative.dev/eventing/cmd/webhook:169708841d7a6371b096e69b3015117a3ea94aa52f33a23861151bb0d7d229a7","ports":[{"name":"https-webhook","containerPort":8443,"protocol":"TCP"},{"name":"metrics","containerPort":9090,"protocol":"TCP"},{"name":"profiling","containerPort":8008,"protocol":"TCP"}],"env":[{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"CONFIG_LOGGING_NAME","value":"config-logging"},{"name":"METRICS_DOMAIN","value":"knative.dev/eventing"},{"name":"WEBHOOK_NAME","value":"eventing-webhook"},{"name":"WEBHOOK_PORT","value":"8443"},{"name":"SINK_BINDING_SELECTION_MODE","value":"exclusion"},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}}],"resources":{"limits":{"cpu":"200m","memory":"200Mi"},"requests":{"cpu":"20m","memory":"20Mi"}},"volumeMounts":[{"name":"eventing-webhook-token-cmvdm","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"livenessProbe":{"httpGet":{"path":"/","port":8443,"scheme":"HTTPS","httpHeaders":[{"name":"k-kubelet-probe","value":"webhook"}]},"initialDelaySeconds":20,"timeoutSeconds":1,"periodSeconds":1,"successThreshold":1,"failureThreshold":3},"readinessProbe":{"httpGet":{"path":"/","port":8443,"scheme":"HTTPS","httpHeaders":[{"name":"k-kubelet-probe","value":"webhook"}]},"timeoutSeconds":1,"periodSeconds":1,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent","securityContext":{"allowPrivilegeEscalation":false}}],"restartPolicy":"Always","terminationGracePeriodSeconds":300,"dnsPolicy":"ClusterFirst","serviceAccountName":"eventing-webhook","serviceAccount":"eventing-webhook","nodeName":"kind-worker","securityContext":{},"affinity":{"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":100,"podAffinityTerm":{"labelSelector":{"matchLabels":{"app":"eventing-webhook"}},"topologyKey":"kubernetes.io/hostname"}}]}},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":false},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null, [truncated 30407 chars]
I0113 10:36:52.918471   25628 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dchaosduck-f665c9dc7-zkdbf'
I0113 10:36:52.925233   25628 round_trippers.go:443] GET https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dchaosduck-f665c9dc7-zkdbf 200 OK in 6 milliseconds
I0113 10:36:52.925247   25628 round_trippers.go:449] Response Headers:
I0113 10:36:52.925252   25628 round_trippers.go:452]     Cache-Control: no-cache, private
I0113 10:36:52.925255   25628 round_trippers.go:452]     Content-Type: application/json
I0113 10:36:52.925258   25628 round_trippers.go:452]     Content-Length: 3092
I0113 10:36:52.925261   25628 round_trippers.go:452]     Date: Wed, 13 Jan 2021 10:36:52 GMT
I0113 10:36:52.925394   25628 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"2902"},"items":[{"metadata":{"name":"chaosduck-f665c9dc7-zkdbf","generateName":"chaosduck-f665c9dc7-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/chaosduck-f665c9dc7-zkdbf","uid":"c2c2d30a-a755-4a67-9289-beb684e8fbce","resourceVersion":"1892","creationTimestamp":"2021-01-13T10:35:41Z","deletionTimestamp":"2021-01-13T10:36:14Z","deletionGracePeriodSeconds":30,"labels":{"app":"chaosduck","pod-template-hash":"f665c9dc7"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"chaosduck-f665c9dc7","uid":"be93d508-83a9-428f-94a5-a218a6835edc","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"chaosduck-token-lwvbb","secret":{"secretName":"chaosduck-token-lwvbb","defaultMode":420}}],"containers":[{"name":"chaosduck","image":"kind.local/knative.dev/pkg/leaderelection/chaosduck:967fb52fb64b86752de93a2833a15b705c07660c858b0b27a4fd4fd89094663a","args":["-disable=imc-controller","-disable=imc-dispatcher"],"env":[{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}}],"resources":{},"volumeMounts":[{"name":"chaosduck-token-lwvbb","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"allowPrivilegeEscalation":false}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"chaosduck","serviceAccount":"chaosduck","nodeName":"kind-worker","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:42Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:45Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:45Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:41Z"}],"hostIP":"172.18.0.3","podIP":"10.244.1.10","podIPs":[{"ip":"10.244.1.10"}],"startTime":"2021-01-13T10:35:42Z","containerStatuses":[{"name":"chaosduck","state":{"running":{"startedAt":"2021-01-13T10:35:44Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"kind.local/knative.dev/pkg/leaderelection/chaosduck:967fb52fb64b86752de93a2833a15b705c07660c858b0b27a4fd4fd89094663a","imageID":"sha256:9ff310726fe092fe547795899132afed3195fd2651a9202a863738a1682934dd","containerID":"containerd://7d90d940150d5ce8ebb702417f38e4c0f8a4b9dc578f1e374de3c83313cf90dd","started":true}],"qosClass":"BestEffort"}}]}
I0113 10:36:52.926378   25628 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Deventing-controller-c895f6756-cf6dg'
I0113 10:36:52.928973   25628 round_trippers.go:443] GET https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Deventing-controller-c895f6756-cf6dg 200 OK in 2 milliseconds
I0113 10:36:52.929023   25628 round_trippers.go:449] Response Headers:
I0113 10:36:52.929072   25628 round_trippers.go:452]     Cache-Control: no-cache, private
I0113 10:36:52.929083   25628 round_trippers.go:452]     Content-Type: application/json
I0113 10:36:52.929146   25628 round_trippers.go:452]     Content-Length: 3952
I0113 10:36:52.929159   25628 round_trippers.go:452]     Date: Wed, 13 Jan 2021 10:36:52 GMT
I0113 10:36:52.929242   25628 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"2902"},"items":[{"metadata":{"name":"eventing-controller-c895f6756-cf6dg","generateName":"eventing-controller-c895f6756-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/eventing-controller-c895f6756-cf6dg","uid":"3827b98b-f6ac-4551-b24e-f6d336cd8e53","resourceVersion":"2645","creationTimestamp":"2021-01-13T10:36:28Z","labels":{"app":"eventing-controller","eventing.knative.dev/release":"devel","pod-template-hash":"c895f6756"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"eventing-controller-c895f6756","uid":"93116175-b845-4b5c-94fe-9c4668bd0ef5","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"eventing-controller-token-6hxjg","secret":{"secretName":"eventing-controller-token-6hxjg","defaultMode":420}}],"containers":[{"name":"eventing-controller","image":"kind.local/knative.dev/eventing/cmd/controller:a695ea7ccc558e8ab6434dcdd1a7c5cb4eabb84c3cd596235473b2e59ef18158","ports":[{"name":"metrics","containerPort":9090,"protocol":"TCP"},{"name":"profiling","containerPort":8008,"protocol":"TCP"}],"env":[{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"CONFIG_LOGGING_NAME","value":"config-logging"},{"name":"CONFIG_OBSERVABILITY_NAME","value":"config-observability"},{"name":"METRICS_DOMAIN","value":"knative.dev/eventing"},{"name":"APISERVER_RA_IMAGE","value":"kind.local/knative.dev/eventing/cmd/apiserver_receive_adapter:8d4f85c32cc62b2731fd86a859d7305d261f94d6471e2f8f6db165a02c70c888"},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}}],"resources":{"requests":{"cpu":"100m","memory":"100Mi"}},"volumeMounts":[{"name":"eventing-controller-token-6hxjg","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent","securityContext":{"allowPrivilegeEscalation":false}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"eventing-controller","serviceAccount":"eventing-controller","nodeName":"kind-worker","securityContext":{},"affinity":{"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":100,"podAffinityTerm":{"labelSelector":{"matchLabels":{"app":"eventing-controller"}},"topologyKey":"kubernetes.io/hostname"}}]}},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":false},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:30Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:35Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:35Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:28Z"}],"hostIP":"172.18.0.3","podIP":"10.244.1.18","podIPs":[{"ip":"10.244.1.18"}],"startTime":"2021-01-13T10:36:30Z","containerStatuses":[{"name":"eventing-controller","state":{"running":{"startedAt":"2021-01-13T10:36:34Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"kind.local/knative.dev/eventing/cmd/controller:a695ea7ccc558e8ab6434dcdd1a7c5cb4eabb84c3cd596235473b2e59ef18158","imageID":"sha256:c9b859f4ecfcd11881660bcb48e03d44ad10991335bc4aee9d53a4894f148bfc","containerID":"containerd://50c64ef119ae94f0de6df42a6dc16ea384bd1eb5111b95d183d3389ce46016ff","started":true}],"qosClass":"Burstable"}}]}
I0113 10:36:52.930211   25628 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Deventing-webhook-bc5f465f4-mv6r6'
pod/chaosduck-f665c9dc7-zkdbf condition met
pod/eventing-controller-c895f6756-cf6dg condition met
I0113 10:36:52.932381   25628 round_trippers.go:443] GET https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Deventing-webhook-bc5f465f4-mv6r6 200 OK in 2 milliseconds
I0113 10:36:52.932399   25628 round_trippers.go:449] Response Headers:
I0113 10:36:52.932403   25628 round_trippers.go:452]     Cache-Control: no-cache, private
I0113 10:36:52.932407   25628 round_trippers.go:452]     Content-Type: application/json
I0113 10:36:52.932410   25628 round_trippers.go:452]     Date: Wed, 13 Jan 2021 10:36:52 GMT
I0113 10:36:52.932555   25628 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"2902"},"items":[{"metadata":{"name":"eventing-webhook-bc5f465f4-mv6r6","generateName":"eventing-webhook-bc5f465f4-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/eventing-webhook-bc5f465f4-mv6r6","uid":"f0aed95c-93c4-4ae0-b1a4-add0bd06aac5","resourceVersion":"2595","creationTimestamp":"2021-01-13T10:35:44Z","deletionTimestamp":"2021-01-13T10:41:28Z","deletionGracePeriodSeconds":300,"labels":{"app":"eventing-webhook","pod-template-hash":"bc5f465f4","role":"eventing-webhook"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"eventing-webhook-bc5f465f4","uid":"a3fb1319-83e9-4a2c-927b-339c71893048","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"eventing-webhook-token-cmvdm","secret":{"secretName":"eventing-webhook-token-cmvdm","defaultMode":420}}],"containers":[{"name":"eventing-webhook","image":"kind.local/knative.dev/eventing/cmd/webhook:169708841d7a6371b096e69b3015117a3ea94aa52f33a23861151bb0d7d229a7","ports":[{"name":"https-webhook","containerPort":8443,"protocol":"TCP"},{"name":"metrics","containerPort":9090,"protocol":"TCP"},{"name":"profiling","containerPort":8008,"protocol":"TCP"}],"env":[{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"CONFIG_LOGGING_NAME","value":"config-logging"},{"name":"METRICS_DOMAIN","value":"knative.dev/eventing"},{"name":"WEBHOOK_NAME","value":"eventing-webhook"},{"name":"WEBHOOK_PORT","value":"8443"},{"name":"SINK_BINDING_SELECTION_MODE","value":"exclusion"},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}}],"resources":{"limits":{"cpu":"200m","memory":"200Mi"},"requests":{"cpu":"20m","memory":"20Mi"}},"volumeMounts":[{"name":"eventing-webhook-token-cmvdm","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"livenessProbe":{"httpGet":{"path":"/","port":8443,"scheme":"HTTPS","httpHeaders":[{"name":"k-kubelet-probe","value":"webhook"}]},"initialDelaySeconds":20,"timeoutSeconds":1,"periodSeconds":1,"successThreshold":1,"failureThreshold":3},"readinessProbe":{"httpGet":{"path":"/","port":8443,"scheme":"HTTPS","httpHeaders":[{"name":"k-kubelet-probe","value":"webhook"}]},"timeoutSeconds":1,"periodSeconds":1,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent","securityContext":{"allowPrivilegeEscalation":false}}],"restartPolicy":"Always","terminationGracePeriodSeconds":300,"dnsPolicy":"ClusterFirst","serviceAccountName":"eventing-webhook","serviceAccount":"eventing-webhook","nodeName":"kind-worker","securityContext":{},"affinity":{"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":100,"podAffinityTerm":{"labelSelector":{"matchLabels":{"app":"eventing-webhook"}},"topologyKey":"kubernetes.io/hostname"}}]}},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":false},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:44Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:31Z","reason":"ContainersNotReady","message":"containers with unready status: [eventing-webhook]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:31Z","reason":"ContainersNotReady","message":"containers with unready status: [eventing-webhook]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:44Z"}],"hostIP":"172.18.0.3","podIP":"10.244.1.15","podIPs":[{"ip":"10.244.1.15"}],"startTime":"2021-01-13T10:35:44Z","containerStatuses":[{"name":"eventing-webhook","state":{"running":{"startedAt":"2021-01-13T10:35:46Z"}},"lastState":{},"ready":false,"restartCount":0,"image":"kind.local/knative.dev/eventing/cmd/webhook:169708841d7a6371b096e69b3015117a3ea94aa52f33a23861151bb0d7d229a7","imageID":"sha256:3ca196d729701392f6b0dd1adc6838612cf64e20cbadfa3d3bac820622342de5","containerID":"containerd://95b7a4b38426f0c6762988a6ea9abc39f68db70e4c4667326ebb1488a5d58f41","started":true}],"qosClass":"Burstable"}}]}
I0113 10:36:52.933010   25628 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Deventing-webhook-bc5f465f4-mv6r6&resourceVersion=2902&watch=true'
I0113 10:36:52.934812   25628 round_trippers.go:443] GET https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Deventing-webhook-bc5f465f4-mv6r6&resourceVersion=2902&watch=true 200 OK in 1 milliseconds
I0113 10:36:52.934825   25628 round_trippers.go:449] Response Headers:
I0113 10:36:52.934829   25628 round_trippers.go:452]     Cache-Control: no-cache, private
I0113 10:36:52.934833   25628 round_trippers.go:452]     Content-Type: application/json
I0113 10:36:52.934836   25628 round_trippers.go:452]     Date: Wed, 13 Jan 2021 10:36:52 GMT
I0113 10:37:22.935488   25628 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Deventing-webhook-bc5f465f4-zrq9g'
I0113 10:37:22.937585   25628 round_trippers.go:443] GET https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Deventing-webhook-bc5f465f4-zrq9g 200 OK in 2 milliseconds
I0113 10:37:22.937604   25628 round_trippers.go:449] Response Headers:
I0113 10:37:22.937609   25628 round_trippers.go:452]     Cache-Control: no-cache, private
I0113 10:37:22.937612   25628 round_trippers.go:452]     Content-Type: application/json
I0113 10:37:22.937616   25628 round_trippers.go:452]     Date: Wed, 13 Jan 2021 10:37:22 GMT
I0113 10:37:22.937773   25628 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"3326"},"items":[{"metadata":{"name":"eventing-webhook-bc5f465f4-zrq9g","generateName":"eventing-webhook-bc5f465f4-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/eventing-webhook-bc5f465f4-zrq9g","uid":"6acec832-0948-4356-b8bc-9e0f64f9427e","resourceVersion":"2670","creationTimestamp":"2021-01-13T10:36:28Z","labels":{"app":"eventing-webhook","pod-template-hash":"bc5f465f4","role":"eventing-webhook"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"eventing-webhook-bc5f465f4","uid":"a3fb1319-83e9-4a2c-927b-339c71893048","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"eventing-webhook-token-cmvdm","secret":{"secretName":"eventing-webhook-token-cmvdm","defaultMode":420}}],"containers":[{"name":"eventing-webhook","image":"kind.local/knative.dev/eventing/cmd/webhook:169708841d7a6371b096e69b3015117a3ea94aa52f33a23861151bb0d7d229a7","ports":[{"name":"https-webhook","containerPort":8443,"protocol":"TCP"},{"name":"metrics","containerPort":9090,"protocol":"TCP"},{"name":"profiling","containerPort":8008,"protocol":"TCP"}],"env":[{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"CONFIG_LOGGING_NAME","value":"config-logging"},{"name":"METRICS_DOMAIN","value":"knative.dev/eventing"},{"name":"WEBHOOK_NAME","value":"eventing-webhook"},{"name":"WEBHOOK_PORT","value":"8443"},{"name":"SINK_BINDING_SELECTION_MODE","value":"exclusion"},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}}],"resources":{"limits":{"cpu":"200m","memory":"200Mi"},"requests":{"cpu":"20m","memory":"20Mi"}},"volumeMounts":[{"name":"eventing-webhook-token-cmvdm","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"livenessProbe":{"httpGet":{"path":"/","port":8443,"scheme":"HTTPS","httpHeaders":[{"name":"k-kubelet-probe","value":"webhook"}]},"initialDelaySeconds":20,"timeoutSeconds":1,"periodSeconds":1,"successThreshold":1,"failureThreshold":3},"readinessProbe":{"httpGet":{"path":"/","port":8443,"scheme":"HTTPS","httpHeaders":[{"name":"k-kubelet-probe","value":"webhook"}]},"timeoutSeconds":1,"periodSeconds":1,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent","securityContext":{"allowPrivilegeEscalation":false}}],"restartPolicy":"Always","terminationGracePeriodSeconds":300,"dnsPolicy":"ClusterFirst","serviceAccountName":"eventing-webhook","serviceAccount":"eventing-webhook","nodeName":"kind-worker","securityContext":{},"affinity":{"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":100,"podAffinityTerm":{"labelSelector":{"matchLabels":{"app":"eventing-webhook"}},"topologyKey":"kubernetes.io/hostname"}}]}},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":false},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:31Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:36Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:36Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:29Z"}],"hostIP":"172.18.0.3","podIP":"10.244.1.19","podIPs":[{"ip":"10.244.1.19"}],"startTime":"2021-01-13T10:36:31Z","containerStatuses":[{"name":"eventing-webhook","state":{"running":{"startedAt":"2021-01-13T10:36:35Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"kind.local/knative.dev/eventing/cmd/webhook:169708841d7a6371b096e69b3015117a3ea94aa52f33a23861151bb0d7d229a7","imageID":"sha256:3ca196d729701392f6b0dd1adc6838612cf64e20cbadfa3d3bac820622342de5","containerID":"containerd://1b05f6547a8c6073eeff892307a053e2c3b3bed2d94b1bc9453dde45796d332a","started":true}],"qosClass":"Burstable"}}]}
I0113 10:37:22.938466   25628 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dimc-controller-676bd4f69-856x5'
pod/eventing-webhook-bc5f465f4-zrq9g condition met
I0113 10:37:22.944534   25628 round_trippers.go:443] GET https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dimc-controller-676bd4f69-856x5 200 OK in 6 milliseconds
pod/imc-controller-676bd4f69-856x5 condition met
I0113 10:37:22.944553   25628 round_trippers.go:449] Response Headers:
I0113 10:37:22.944557   25628 round_trippers.go:452]     Cache-Control: no-cache, private
I0113 10:37:22.944561   25628 round_trippers.go:452]     Content-Type: application/json
I0113 10:37:22.944564   25628 round_trippers.go:452]     Content-Length: 3632
I0113 10:37:22.944568   25628 round_trippers.go:452]     Date: Wed, 13 Jan 2021 10:37:22 GMT
I0113 10:37:22.944594   25628 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"3326"},"items":[{"metadata":{"name":"imc-controller-676bd4f69-856x5","generateName":"imc-controller-676bd4f69-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/imc-controller-676bd4f69-856x5","uid":"c9f03481-08c5-4283-858a-a2919365613d","resourceVersion":"1526","creationTimestamp":"2021-01-13T10:35:28Z","labels":{"messaging.knative.dev/channel":"in-memory-channel","messaging.knative.dev/role":"controller","pod-template-hash":"676bd4f69"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"imc-controller-676bd4f69","uid":"21816286-b917-4b9b-83d4-d934758322ee","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"imc-controller-token-5m8c8","secret":{"secretName":"imc-controller-token-5m8c8","defaultMode":420}}],"containers":[{"name":"controller","image":"kind.local/knative.dev/eventing/cmd/in_memory/channel_controller:a6d5565bf562e93a1f0911a82e7a513ee77535879002b5b2d72303fe1eb59b12","ports":[{"name":"metrics","containerPort":9090,"protocol":"TCP"},{"name":"profiling","containerPort":8008,"protocol":"TCP"}],"env":[{"name":"CONFIG_LOGGING_NAME","value":"config-logging"},{"name":"CONFIG_OBSERVABILITY_NAME","value":"config-observability"},{"name":"METRICS_DOMAIN","value":"knative.dev/inmemorychannel-controller"},{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"DISPATCHER_IMAGE","value":"kind.local/knative.dev/eventing/cmd/in_memory/channel_dispatcher:32b708f1d26d514c6459873cff70a513aca349e7d775709b058a11ffdc3a5147"},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}}],"resources":{},"volumeMounts":[{"name":"imc-controller-token-5m8c8","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"allowPrivilegeEscalation":false}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"imc-controller","serviceAccount":"imc-controller","nodeName":"kind-worker","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":false},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:28Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:31Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:31Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:28Z"}],"hostIP":"172.18.0.3","podIP":"10.244.1.9","podIPs":[{"ip":"10.244.1.9"}],"startTime":"2021-01-13T10:35:28Z","containerStatuses":[{"name":"controller","state":{"running":{"startedAt":"2021-01-13T10:35:31Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"kind.local/knative.dev/eventing/cmd/in_memory/channel_controller:latest","imageID":"sha256:b28daf1512a5c35ee6c6ba69b18a86e3b3c2d8a824f5f5828994fa84b0a9afee","containerID":"containerd://62df33837bbd4e833be11ef1b529256ef5a7c03fd8a3be301a85ea9f36a0b811","started":true}],"qosClass":"BestEffort"}}]}
I0113 10:37:22.945167   25628 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dimc-dispatcher-576559d88f-j58p9'
pod/imc-dispatcher-576559d88f-j58p9 condition met
I0113 10:37:22.950680   25628 round_trippers.go:443] GET https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dimc-dispatcher-576559d88f-j58p9 200 OK in 5 milliseconds
pod/mt-broker-controller-84fc78474-jzbdg condition met
I0113 10:37:22.950689   25628 round_trippers.go:449] Response Headers:
pod/mt-broker-filter-69f97b6df5-v6x6v condition met
I0113 10:37:22.950693   25628 round_trippers.go:452]     Cache-Control: no-cache, private
pod/mt-broker-ingress-5d6df454bf-pnpm6 condition met
I0113 10:37:22.950697   25628 round_trippers.go:452]     Content-Type: application/json
pod/sugar-controller-59f989cc7-ldn8w condition met
I0113 10:37:22.950700   25628 round_trippers.go:452]     Content-Length: 4080
pod/zipkin-7fcb9b4ff8-xljj9 condition met
I0113 10:37:22.950703   25628 round_trippers.go:452]     Date: Wed, 13 Jan 2021 10:37:22 GMT
I0113 10:37:22.950731   25628 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"3326"},"items":[{"metadata":{"name":"imc-dispatcher-576559d88f-j58p9","generateName":"imc-dispatcher-576559d88f-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/imc-dispatcher-576559d88f-j58p9","uid":"b3798024-dd26-4a01-afdc-be8c4d0fa755","resourceVersion":"1584","creationTimestamp":"2021-01-13T10:35:28Z","labels":{"messaging.knative.dev/channel":"in-memory-channel","messaging.knative.dev/role":"dispatcher","pod-template-hash":"576559d88f"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"imc-dispatcher-576559d88f","uid":"608cd40f-4f67-45a5-b2d4-69e6a16448cf","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"imc-dispatcher-token-blmn7","secret":{"secretName":"imc-dispatcher-token-blmn7","defaultMode":420}}],"containers":[{"name":"dispatcher","image":"kind.local/knative.dev/eventing/cmd/in_memory/channel_dispatcher:32b708f1d26d514c6459873cff70a513aca349e7d775709b058a11ffdc3a5147","ports":[{"name":"http","containerPort":8080,"protocol":"TCP"},{"name":"metrics","containerPort":9090,"protocol":"TCP"}],"env":[{"name":"CONFIG_LOGGING_NAME","value":"config-logging"},{"name":"CONFIG_OBSERVABILITY_NAME","value":"config-observability"},{"name":"METRICS_DOMAIN","value":"knative.dev/inmemorychannel-dispatcher"},{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"CONTAINER_NAME","value":"dispatcher"}],"resources":{},"volumeMounts":[{"name":"imc-dispatcher-token-blmn7","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"livenessProbe":{"httpGet":{"path":"/healthz","port":8080,"scheme":"HTTP"},"initialDelaySeconds":5,"timeoutSeconds":1,"periodSeconds":2,"successThreshold":1,"failureThreshold":3},"readinessProbe":{"httpGet":{"path":"/healthz","port":8080,"scheme":"HTTP"},"timeoutSeconds":1,"periodSeconds":2,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"imc-dispatcher","serviceAccount":"imc-dispatcher","nodeName":"kind-worker","securityContext":{},"affinity":{"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":100,"podAffinityTerm":{"labelSelector":{"matchLabels":{"messaging.knative.dev/channel":"in-memory-channel","messaging.knative.dev/role":"dispatcher"}},"topologyKey":"kubernetes.io/hostname"}}]}},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":false},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:28Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:33Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:33Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:28Z"}],"hostIP":"172.18.0.3","podIP":"10.244.1.8","podIPs":[{"ip":"10.244.1.8"}],"startTime":"2021-01-13T10:35:28Z","containerStatuses":[{"name":"dispatcher","state":{"running":{"startedAt":"2021-01-13T10:35:31Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"kind.local/knative.dev/eventing/cmd/in_memory/channel_dispatcher:latest","imageID":"sha256:b4d2446c83a01b0762c1c907e5337f83d77e2137c6d0d560467d1bb0b0a68888","containerID":"containerd://e29c0fcb2dea4a6bb02513408230e200bf674cff4720a1669287d23ecabedd1a","started":true}],"qosClass":"BestEffort"}}]}
I0113 10:37:22.951375   25628 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dmt-broker-controller-84fc78474-jzbdg'
I0113 10:37:22.952825   25628 round_trippers.go:443] GET https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dmt-broker-controller-84fc78474-jzbdg 200 OK in 1 milliseconds
I0113 10:37:22.952835   25628 round_trippers.go:449] Response Headers:
I0113 10:37:22.952839   25628 round_trippers.go:452]     Date: Wed, 13 Jan 2021 10:37:22 GMT
I0113 10:37:22.952842   25628 round_trippers.go:452]     Cache-Control: no-cache, private
I0113 10:37:22.952845   25628 round_trippers.go:452]     Content-Type: application/json
I0113 10:37:22.952848   25628 round_trippers.go:452]     Content-Length: 3857
I0113 10:37:22.952873   25628 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"3326"},"items":[{"metadata":{"name":"mt-broker-controller-84fc78474-jzbdg","generateName":"mt-broker-controller-84fc78474-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/mt-broker-controller-84fc78474-jzbdg","uid":"42cd9a3c-9a43-4a50-873e-885764bc289f","resourceVersion":"2612","creationTimestamp":"2021-01-13T10:36:28Z","labels":{"app":"mt-broker-controller","eventing.knative.dev/release":"devel","pod-template-hash":"84fc78474"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"mt-broker-controller-84fc78474","uid":"eb855823-b150-4f3b-b8b1-412cb896f24f","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"eventing-controller-token-6hxjg","secret":{"secretName":"eventing-controller-token-6hxjg","defaultMode":420}}],"containers":[{"name":"mt-broker-controller","image":"kind.local/knative.dev/eventing/cmd/mtchannel_broker:371a2ce1db2b6af3b71f1bb6248035c293e7ca5f247f674c362398f1ec49868a","ports":[{"name":"metrics","containerPort":9090,"protocol":"TCP"},{"name":"profiling","containerPort":8008,"protocol":"TCP"}],"env":[{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"CONFIG_LOGGING_NAME","value":"config-logging"},{"name":"CONFIG_OBSERVABILITY_NAME","value":"config-observability"},{"name":"METRICS_DOMAIN","value":"knative.dev/eventing"},{"name":"BROKER_INJECTION_DEFAULT","value":"false"},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}}],"resources":{"requests":{"cpu":"100m","memory":"100Mi"}},"volumeMounts":[{"name":"eventing-controller-token-6hxjg","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent","securityContext":{"allowPrivilegeEscalation":false}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"eventing-controller","serviceAccount":"eventing-controller","nodeName":"kind-worker","securityContext":{},"affinity":{"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":100,"podAffinityTerm":{"labelSelector":{"matchLabels":{"app":"mt-broker-controller"}},"topologyKey":"kubernetes.io/hostname"}}]}},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":false},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:29Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:34Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:34Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:28Z"}],"hostIP":"172.18.0.3","podIP":"10.244.1.16","podIPs":[{"ip":"10.244.1.16"}],"startTime":"2021-01-13T10:36:29Z","containerStatuses":[{"name":"mt-broker-controller","state":{"running":{"startedAt":"2021-01-13T10:36:34Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"kind.local/knative.dev/eventing/cmd/mtchannel_broker:371a2ce1db2b6af3b71f1bb6248035c293e7ca5f247f674c362398f1ec49868a","imageID":"sha256:137b256f9100db6159c8906661c029814bd4aadcb53cf99fee4875f61644011f","containerID":"containerd://44a2d4c111f9292f0c52c49fa3d63b566c6cfa4ce0a6a7a3864a70ffa886e813","started":true}],"qosClass":"Burstable"}}]}
I0113 10:37:22.953442   25628 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dmt-broker-filter-69f97b6df5-v6x6v'
I0113 10:37:22.954856   25628 round_trippers.go:443] GET https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dmt-broker-filter-69f97b6df5-v6x6v 200 OK in 1 milliseconds
I0113 10:37:22.954864   25628 round_trippers.go:449] Response Headers:
I0113 10:37:22.954880   25628 round_trippers.go:452]     Cache-Control: no-cache, private
I0113 10:37:22.954883   25628 round_trippers.go:452]     Content-Type: application/json
I0113 10:37:22.954889   25628 round_trippers.go:452]     Content-Length: 4058
I0113 10:37:22.954892   25628 round_trippers.go:452]     Date: Wed, 13 Jan 2021 10:37:22 GMT
I0113 10:37:22.955052   25628 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"3326"},"items":[{"metadata":{"name":"mt-broker-filter-69f97b6df5-v6x6v","generateName":"mt-broker-filter-69f97b6df5-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/mt-broker-filter-69f97b6df5-v6x6v","uid":"cdd763a1-a5b0-437d-9742-16be293a82e1","resourceVersion":"1517","creationTimestamp":"2021-01-13T10:35:23Z","labels":{"eventing.knative.dev/brokerRole":"filter","eventing.knative.dev/release":"devel","pod-template-hash":"69f97b6df5"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"mt-broker-filter-69f97b6df5","uid":"cba052aa-09a4-4ed4-8a89-3a897d933412","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"mt-broker-filter-token-vsvrp","secret":{"secretName":"mt-broker-filter-token-vsvrp","defaultMode":420}}],"containers":[{"name":"filter","image":"kind.local/knative.dev/eventing/cmd/mtbroker/filter:1215b68433d1ab8979e55e5ea38c5f78c37818ce9f3873d0ec659ebe76bcddc8","ports":[{"name":"http","containerPort":8080,"protocol":"TCP"},{"name":"metrics","containerPort":9092,"protocol":"TCP"}],"env":[{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"CONTAINER_NAME","value":"filter"},{"name":"CONFIG_LOGGING_NAME","value":"config-logging"},{"name":"CONFIG_OBSERVABILITY_NAME","value":"config-observability"},{"name":"METRICS_DOMAIN","value":"knative.dev/internal/eventing"},{"name":"FILTER_PORT","value":"8080"}],"resources":{"requests":{"cpu":"100m","memory":"100Mi"}},"volumeMounts":[{"name":"mt-broker-filter-token-vsvrp","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"livenessProbe":{"httpGet":{"path":"/healthz","port":8080,"scheme":"HTTP"},"initialDelaySeconds":5,"timeoutSeconds":1,"periodSeconds":2,"successThreshold":1,"failureThreshold":3},"readinessProbe":{"httpGet":{"path":"/healthz","port":8080,"scheme":"HTTP"},"timeoutSeconds":1,"periodSeconds":2,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent","securityContext":{"allowPrivilegeEscalation":false}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"mt-broker-filter","serviceAccount":"mt-broker-filter","nodeName":"kind-worker","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":false},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:23Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:31Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:31Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:23Z"}],"hostIP":"172.18.0.3","podIP":"10.244.1.4","podIPs":[{"ip":"10.244.1.4"}],"startTime":"2021-01-13T10:35:23Z","containerStatuses":[{"name":"filter","state":{"running":{"startedAt":"2021-01-13T10:35:28Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"kind.local/knative.dev/eventing/cmd/mtbroker/filter:1215b68433d1ab8979e55e5ea38c5f78c37818ce9f3873d0ec659ebe76bcddc8","imageID":"sha256:60f93ac8439bca5e71de1c154db2368dea719b63425ae43d6691cd212a825705","containerID":"containerd://5fe88480ab2ed8384e10246857aefd98756a821eb40126db54ff2d76df5a6a8b","started":true}],"qosClass":"Burstable"}}]}
I0113 10:37:22.955843   25628 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dmt-broker-ingress-5d6df454bf-pnpm6'
I0113 10:37:22.958044   25628 round_trippers.go:443] GET https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dmt-broker-ingress-5d6df454bf-pnpm6 200 OK in 2 milliseconds
I0113 10:37:22.958052   25628 round_trippers.go:449] Response Headers:
I0113 10:37:22.958056   25628 round_trippers.go:452]     Content-Length: 4016
I0113 10:37:22.958060   25628 round_trippers.go:452]     Date: Wed, 13 Jan 2021 10:37:22 GMT
I0113 10:37:22.958063   25628 round_trippers.go:452]     Cache-Control: no-cache, private
I0113 10:37:22.958066   25628 round_trippers.go:452]     Content-Type: application/json
I0113 10:37:22.958093   25628 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"3326"},"items":[{"metadata":{"name":"mt-broker-ingress-5d6df454bf-pnpm6","generateName":"mt-broker-ingress-5d6df454bf-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/mt-broker-ingress-5d6df454bf-pnpm6","uid":"0c2f4674-3da3-4e4a-8fa1-4dafd2eb855b","resourceVersion":"1579","creationTimestamp":"2021-01-13T10:35:23Z","labels":{"eventing.knative.dev/brokerRole":"ingress","eventing.knative.dev/release":"devel","pod-template-hash":"5d6df454bf"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"mt-broker-ingress-5d6df454bf","uid":"2f3dee37-c0f8-494a-8448-857b467dc315","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"mt-broker-ingress-token-4k524","secret":{"secretName":"mt-broker-ingress-token-4k524","defaultMode":420}}],"containers":[{"name":"ingress","image":"kind.local/knative.dev/eventing/cmd/mtbroker/ingress:52b8032f8f0ccabcc25b8a9a526a338d7e50a173e4b480097e278dda8fa59871","ports":[{"name":"http","containerPort":8080,"protocol":"TCP"},{"name":"metrics","containerPort":9092,"protocol":"TCP"}],"env":[{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}},{"name":"CONTAINER_NAME","value":"ingress"},{"name":"CONFIG_LOGGING_NAME","value":"config-logging"},{"name":"CONFIG_OBSERVABILITY_NAME","value":"config-observability"},{"name":"METRICS_DOMAIN","value":"knative.dev/internal/eventing"},{"name":"INGRESS_PORT","value":"8080"}],"resources":{"requests":{"cpu":"100m","memory":"100Mi"}},"volumeMounts":[{"name":"mt-broker-ingress-token-4k524","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"livenessProbe":{"httpGet":{"path":"/healthz","port":8080,"scheme":"HTTP"},"initialDelaySeconds":5,"timeoutSeconds":1,"periodSeconds":2,"successThreshold":1,"failureThreshold":3},"readinessProbe":{"httpGet":{"path":"/healthz","port":8080,"scheme":"HTTP"},"timeoutSeconds":1,"periodSeconds":2,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent","securityContext":{"allowPrivilegeEscalation":false}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"mt-broker-ingress","serviceAccount":"mt-broker-ingress","nodeName":"kind-worker","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":false},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:25Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:33Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:33Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:23Z"}],"hostIP":"172.18.0.3","podIP":"10.244.1.5","podIPs":[{"ip":"10.244.1.5"}],"startTime":"2021-01-13T10:35:25Z","containerStatuses":[{"name":"ingress","state":{"running":{"startedAt":"2021-01-13T10:35:29Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"kind.local/knative.dev/eventing/cmd/mtbroker/ingress:latest","imageID":"sha256:538f35844d92c5172fcdbf148cd0a2b6bbd6adea6d43d043452eb013d2814a79","containerID":"containerd://a9f5f06fbeccfb51d7f002962fe8c7eec7711731ed7b8fea851733bb0106d25c","started":true}],"qosClass":"Burstable"}}]}
I0113 10:37:22.958710   25628 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dsugar-controller-59f989cc7-ldn8w'
I0113 10:37:22.960182   25628 round_trippers.go:443] GET https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dsugar-controller-59f989cc7-ldn8w 200 OK in 1 milliseconds
I0113 10:37:22.960190   25628 round_trippers.go:449] Response Headers:
I0113 10:37:22.960194   25628 round_trippers.go:452]     Cache-Control: no-cache, private
I0113 10:37:22.960197   25628 round_trippers.go:452]     Content-Type: application/json
I0113 10:37:22.960278   25628 round_trippers.go:452]     Content-Length: 3476
I0113 10:37:22.960282   25628 round_trippers.go:452]     Date: Wed, 13 Jan 2021 10:37:22 GMT
I0113 10:37:22.960352   25628 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"3326"},"items":[{"metadata":{"name":"sugar-controller-59f989cc7-ldn8w","generateName":"sugar-controller-59f989cc7-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/sugar-controller-59f989cc7-ldn8w","uid":"82a96098-e9fd-48e4-b61e-685c73196bf0","resourceVersion":"2609","creationTimestamp":"2021-01-13T10:36:28Z","labels":{"eventing.knative.dev/role":"sugar-controller","pod-template-hash":"59f989cc7"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"sugar-controller-59f989cc7","uid":"6ba17fce-4163-41e8-90dd-4584e2278069","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"eventing-controller-token-6hxjg","secret":{"secretName":"eventing-controller-token-6hxjg","defaultMode":420}}],"containers":[{"name":"controller","image":"kind.local/knative.dev/eventing/cmd/sugar_controller:a34f3e24a6d56caf6fffa73e954f4e5b77682aa0f021d8503d25c0c9bea418e8","ports":[{"name":"metrics","containerPort":9090,"protocol":"TCP"},{"name":"profiling","containerPort":8008,"protocol":"TCP"}],"env":[{"name":"CONFIG_LOGGING_NAME","value":"config-logging"},{"name":"CONFIG_OBSERVABILITY_NAME","value":"config-observability"},{"name":"METRICS_DOMAIN","value":"knative.dev/sugar-controller"},{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}}],"resources":{},"volumeMounts":[{"name":"eventing-controller-token-6hxjg","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"allowPrivilegeEscalation":false}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"eventing-controller","serviceAccount":"eventing-controller","nodeName":"kind-worker","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":false},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:30Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:34Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:34Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:29Z"}],"hostIP":"172.18.0.3","podIP":"10.244.1.17","podIPs":[{"ip":"10.244.1.17"}],"startTime":"2021-01-13T10:36:30Z","containerStatuses":[{"name":"controller","state":{"running":{"startedAt":"2021-01-13T10:36:34Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"kind.local/knative.dev/eventing/cmd/sugar_controller:a34f3e24a6d56caf6fffa73e954f4e5b77682aa0f021d8503d25c0c9bea418e8","imageID":"sha256:68097895c55555c0beb9844f3e0ca4abca3f79d4f23218b372066bcc2fc8fe84","containerID":"containerd://86b4971a6963515dc77a2a2a9c5054681939ec3a4aa0ab06a114d0a745019db6","started":true}],"qosClass":"BestEffort"}}]}
I0113 10:37:22.960858   25628 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dzipkin-7fcb9b4ff8-xljj9'
I0113 10:37:22.962181   25628 round_trippers.go:443] GET https://127.0.0.1:38131/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Dzipkin-7fcb9b4ff8-xljj9 200 OK in 1 milliseconds
I0113 10:37:22.962206   25628 round_trippers.go:449] Response Headers:
I0113 10:37:22.962213   25628 round_trippers.go:452]     Cache-Control: no-cache, private
I0113 10:37:22.962217   25628 round_trippers.go:452]     Content-Type: application/json
I0113 10:37:22.962220   25628 round_trippers.go:452]     Content-Length: 2895
I0113 10:37:22.962223   25628 round_trippers.go:452]     Date: Wed, 13 Jan 2021 10:37:22 GMT
I0113 10:37:22.962429   25628 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"3326"},"items":[{"metadata":{"name":"zipkin-7fcb9b4ff8-xljj9","generateName":"zipkin-7fcb9b4ff8-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/zipkin-7fcb9b4ff8-xljj9","uid":"62d7e52c-bb16-414b-9154-4d45655c1d04","resourceVersion":"2174","creationTimestamp":"2021-01-13T10:35:42Z","labels":{"app":"zipkin","pod-template-hash":"7fcb9b4ff8"},"annotations":{"sidecar.istio.io/inject":"false"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"zipkin-7fcb9b4ff8","uid":"2afd3171-0986-4051-9b40-3defa5be7a13","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"default-token-c4sfx","secret":{"secretName":"default-token-c4sfx","defaultMode":420}}],"containers":[{"name":"zipkin","image":"docker.io/openzipkin/zipkin:2.13.0","ports":[{"containerPort":9411,"protocol":"TCP"}],"env":[{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}}],"resources":{"limits":{"memory":"1000Mi"},"requests":{"memory":"256Mi"}},"volumeMounts":[{"name":"default-token-c4sfx","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"kind-worker","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:42Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:04Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:36:04Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-13T10:35:42Z"}],"hostIP":"172.18.0.3","podIP":"10.244.1.11","podIPs":[{"ip":"10.244.1.11"}],"startTime":"2021-01-13T10:35:42Z","containerStatuses":[{"name":"zipkin","state":{"running":{"startedAt":"2021-01-13T10:36:03Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"docker.io/openzipkin/zipkin:2.13.0","imageID":"docker.io/openzipkin/zipkin@sha256:2f3139c6c2975ce4e53717fc07da628efdf2c0d2b3dbd7de890369b9450fd21d","containerID":"containerd://54aa896f65831a6f59da3923ce4c49a271d6cc6dfd610bc9d8eb7eeb7417bcf4","started":true}],"qosClass":"Burstable"}}]}
F0113 10:37:22.962774   25628 helpers.go:115] error: timed out waiting for the condition on pods/eventing-webhook-bc5f465f4-mv6r6
goroutine 1 [running]:
k8s.io/klog/v2.stacks(0xc000010001, 0xc0000fc7e0, 0x82, 0xd3)
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:996 +0xb9
k8s.io/klog/v2.(*loggingT).output(0x4d2aec0, 0xc000000003, 0x0, 0x0, 0xc000b18690, 0x48a9af4, 0xa, 0x73, 0x41d400)
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:945 +0x191
k8s.io/klog/v2.(*loggingT).printDepth(0x4d2aec0, 0x3, 0x0, 0x0, 0x2, 0xc0013b7a10, 0x1, 0x1)
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:718 +0x165
k8s.io/klog/v2.FatalDepth(...)
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:1449
k8s.io/kubectl/pkg/cmd/util.fatal(0xc00134d620, 0x53, 0x1)
	/go/src/github.com/openshift/oc/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:93 +0x1f0
k8s.io/kubectl/pkg/cmd/util.checkErr(0x347a520, 0xc0015d2180, 0x31c65e8)
	/go/src/github.com/openshift/oc/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:188 +0x945
k8s.io/kubectl/pkg/cmd/util.CheckErr(...)
	/go/src/github.com/openshift/oc/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:115
k8s.io/kubectl/pkg/cmd/wait.NewCmdWait.func1(0xc000bdadc0, 0xc000b0bf80, 0x1, 0x8)
	/go/src/github.com/openshift/oc/vendor/k8s.io/kubectl/pkg/cmd/wait/wait.go:120 +0xad
github.com/spf13/cobra.(*Command).execute(0xc000bdadc0, 0xc000b0bf00, 0x8, 0x8, 0xc000bdadc0, 0xc000b0bf00)
	/go/src/github.com/openshift/oc/vendor/github.com/spf13/cobra/command.go:846 +0x2c2
github.com/spf13/cobra.(*Command).ExecuteC(0xc001242b00, 0x7, 0xc001242b00, 0x7)
	/go/src/github.com/openshift/oc/vendor/github.com/spf13/cobra/command.go:950 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
	/go/src/github.com/openshift/oc/vendor/github.com/spf13/cobra/command.go:887
main.main()
	/go/src/github.com/openshift/oc/cmd/oc/oc.go:110 +0x885

goroutine 4 [chan receive]:
k8s.io/klog/v2.(*loggingT).flushDaemon(0x4d2aec0)
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:1131 +0x8b
created by k8s.io/klog/v2.init.0
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:416 +0xd8

goroutine 7 [chan receive]:
k8s.io/klog.(*loggingT).flushDaemon(0x4d2ade0)
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/klog.go:1010 +0x8b
created by k8s.io/klog.init.0
	/go/src/github.com/openshift/oc/vendor/k8s.io/klog/klog.go:411 +0xd8

goroutine 24 [select]:
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x31c6510, 0x347e280, 0xc0011bf8f0, 0x1, 0xc0000a0360)
	/go/src/github.com/openshift/oc/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:167 +0x149
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x31c6510, 0x12a05f200, 0x0, 0xc0002d6201, 0xc0000a0360)
	/go/src/github.com/openshift/oc/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/go/src/github.com/openshift/oc/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
k8s.io/apimachinery/pkg/util/wait.Forever(0x31c6510, 0x12a05f200)
	/go/src/github.com/openshift/oc/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:81 +0x4f
created by k8s.io/component-base/logs.InitLogs
	/go/src/github.com/openshift/oc/vendor/k8s.io/component-base/logs/logs.go:58 +0x8a

goroutine 125 [IO wait]:
internal/poll.runtime_pollWait(0x7f7b5407e0d8, 0x72, 0x3482f20)
	/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/runtime/netpoll.go:220 +0x55
internal/poll.(*pollDesc).wait(0xc00102ee18, 0x72, 0xc001150000, 0x49b1, 0x49b1)
	/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
	/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00102ee00, 0xc001150000, 0x49b1, 0x49b1, 0x0, 0x0, 0x0)
	/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/internal/poll/fd_unix.go:159 +0x1b1
net.(*netFD).Read(0xc00102ee00, 0xc001150000, 0x49b1, 0x49b1, 0x203000, 0x78699b, 0xc0010d59e0)
	/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc0000108d0, 0xc001150000, 0x49b1, 0x49b1, 0x0, 0x0, 0x0)
	/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/net/net.go:182 +0x8e
crypto/tls.(*atLeastReader).Read(0xc00166a760, 0xc001150000, 0x49b1, 0x49b1, 0xb69, 0x49ac, 0xc001139710)
	/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/crypto/tls/conn.go:779 +0x62
bytes.(*Buffer).ReadFrom(0xc0010d5b00, 0x347a260, 0xc00166a760, 0x41d785, 0x2bdcba0, 0x2ed2dc0)
	/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/bytes/buffer.go:204 +0xb1
crypto/tls.(*Conn).readFromUntil(0xc0010d5880, 0x347f7c0, 0xc0000108d0, 0x5, 0xc0000108d0, 0xb58)
	/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/crypto/tls/conn.go:801 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc0010d5880, 0x0, 0x0, 0xc001139d18)
	/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/crypto/tls/conn.go:608 +0x115
crypto/tls.(*Conn).readRecord(...)
	/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/crypto/tls/conn.go:576
crypto/tls.(*Conn).Read(0xc0010d5880, 0xc000fdf000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/crypto/tls/conn.go:1252 +0x15f
bufio.(*Reader).Read(0xc000246840, 0xc0005ded58, 0x9, 0x9, 0xc001139d18, 0x31c9f00, 0x9bbceb)
	/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/bufio/bufio.go:227 +0x222
io.ReadAtLeast(0x347a000, 0xc000246840, 0xc0005ded58, 0x9, 0x9, 0x9, 0xc000062070, 0x0, 0x347a520)
	/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/io/io.go:314 +0x87
io.ReadFull(...)
	/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/io/io.go:333
golang.org/x/net/http2.readFrameHeader(0xc0005ded58, 0x9, 0x9, 0x347a000, 0xc000246840, 0x0, 0x0, 0xc00165ce10, 0x0)
	/go/src/github.com/openshift/oc/vendor/golang.org/x/net/http2/frame.go:237 +0x89
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0005ded20, 0xc00165ce10, 0x0, 0x0, 0x0)
	/go/src/github.com/openshift/oc/vendor/golang.org/x/net/http2/frame.go:492 +0xa5
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc001139fa8, 0x0, 0x0)
	/go/src/github.com/openshift/oc/vendor/golang.org/x/net/http2/transport.go:1794 +0xd8
golang.org/x/net/http2.(*ClientConn).readLoop(0xc0009da780)
	/go/src/github.com/openshift/oc/vendor/golang.org/x/net/http2/transport.go:1716 +0x6f
created by golang.org/x/net/http2.(*Transport).newClientConn
	/go/src/github.com/openshift/oc/vendor/golang.org/x/net/http2/transport.go:695 +0x66e
Error: Process completed with exit code 255.

@vaikas
Copy link
Contributor Author

vaikas commented Jan 14, 2021

Ok, so moar logs:

b8769ee1e7671481fe8bbef0b44c51276bf4924898ed0b28aef3fe7c87","containerID":"containerd://4b91885600e42139dab1940c34f59052c14a23b40f1cd61095581a7a18407d5c","started":true}],"qosClass":"Burstable"}}]}
2021-01-12T10:00:59.2281962Z I0112 10:00:59.219514   26156 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Deventing-webhook-9d555fbd4-hnnbm'
2021-01-12T10:00:59.2286072Z I0112 10:00:59.221086   26156 round_trippers.go:443] GET https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Deventing-webhook-9d555fbd4-hnnbm 200 OK in 1 milliseconds
2021-01-12T10:00:59.2287553Z I0112 10:00:59.221110   26156 round_trippers.go:449] Response Headers:
2021-01-12T10:00:59.2288508Z I0112 10:00:59.221114   26156 round_trippers.go:452]     Content-Length: 140
2021-01-12T10:00:59.2289124Z I0112 10:00:59.221135   26156 round_trippers.go:452]     Date: Tue, 12 Jan 2021 10:00:59 GMT
2021-01-12T10:00:59.2289934Z I0112 10:00:59.221141   26156 round_trippers.go:452]     Cache-Control: no-cache, private
2021-01-12T10:00:59.2291195Z I0112 10:00:59.221144   26156 round_trippers.go:452]     Content-Type: application/json
2021-01-12T10:00:59.2293471Z I0112 10:00:59.221156   26156 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"3522"},"items":[]}
2021-01-12T10:00:59.2295809Z I0112 10:00:59.221278   26156 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Deventing-webhook-9d555fbd4-hnnbm&resourceVersion=3522&watch=true'
2021-01-12T10:00:59.2298840Z I0112 10:00:59.221971   26156 round_trippers.go:443] GET https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Deventing-webhook-9d555fbd4-hnnbm&resourceVersion=3522&watch=true 200 OK in 0 milliseconds
2021-01-12T10:00:59.2300203Z I0112 10:00:59.221985   26156 round_trippers.go:449] Response Headers:
2021-01-12T10:00:59.2301048Z I0112 10:00:59.221989   26156 round_trippers.go:452]     Content-Type: application/json
2021-01-12T10:00:59.2302207Z I0112 10:00:59.221993   26156 round_trippers.go:452]     Date: Tue, 12 Jan 2021 10:00:59 GMT
2021-01-12T10:00:59.2303262Z I0112 10:00:59.221996   26156 round_trippers.go:452]     Cache-Control: no-cache, private
2021-01-12T10:00:59.2305010Z pod/eventing-controller-67d57bbcb9-c7n6n condition met
2021-01-12T10:01:29.2231482Z I0112 10:01:29.222736   26156 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: kubectl/openshift (linux/amd64) kubernetes/ffd6836" 'https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Deventing-webhook-9d555fbd4-st8fw'
2021-01-12T10:01:29.2261284Z I0112 10:01:29.225821   26156 round_trippers.go:443] GET https://127.0.0.1:34667/api/v1/namespaces/knative-eventing/pods?fieldSelector=metadata.name%3Deventing-webhook-9d555fbd4-st8fw 200 OK in 3 milliseconds
2021-01-12T10:01:29.2263069Z I0112 10:01:29.225836   26156 round_trippers.go:449] Response Headers:
2021-01-12T10:01:29.2264303Z I0112 10:01:29.225841   26156 round_trippers.go:452]     Cache-Control: no-cache, private
2021-01-12T10:01:29.2265563Z I0112 10:01:29.225845   26156 round_trippers.go:452]     Content-Type: application/json
2021-01-12T10:01:29.2266867Z I0112 10:01:29.225848   26156 round_trippers.go:452]     Date: Tue, 12 Jan 2021 10:01:29 GMT
2021-01-12T10:01:29.2311596Z I0112 10:01:29.227030   26156 request.go:1097] Response Body: {"kind":"PodList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/namespaces/knative-eventing/pods","resourceVersion":"3944"},"items":[{"metadata":{"name":"eventing-webhook-9d555fbd4-st8fw","generateName":"eventing-webhook-9d555fbd4-","namespace":"knative-eventing","selfLink":"/api/v1/namespaces/knative-eventing/pods/eventing-webhook-9d555fbd4-st8fw","uid":"57a04376-a85c-4f30-9e0f-c5e032260288","resourceVersion":"2743","creationTimestamp":"2021-01-12T09:59:57Z","labels":{"app":"eventing-webhook","pod-template-hash":"9d555fbd4","role":"eventing-webhook"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"eventing-webhook-9d555fbd4","uid":"c2be7162-0148-475b-ab0a-ae0226596635","controller":true,"blockOwnerDeletion":true}],"managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2021-01-12T09:59:57Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:generateName":{},"f:labels":{".":{},"f:app":{},"f:pod-template-hash":{},"f:role":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"c2be7162-0148-475b-ab0a-ae0226596635\"}":{".":{},"f:apiVersion":{},"f:blockOwnerDeletion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:spec":{"f:affinity":{".":{},"f:podAntiAffinity":{".":{},"f:preferredDuringSchedulingIgnoredDuringExecution":{}}},"f:containers":{"k:{\"name\":\"eventing-webhook\"}":{".":{},"f:env":{".":{},"k:{\"name\":\"CONFIG_LOGGING_NAME\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"METRICS_DOMAIN\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"POD_NAME\"}":{".":{},"f:name":{},"f:valueFrom":{".":{},"f:fieldRef":{".":{},"f:apiVersion":{},"f:fieldPath":{}}}},"k:{\"name\":\"SINK_BINDING_SELECTION_MODE\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"SYSTEM_NAMESPACE\"}":{".":{},"f:name":{},"f:valueFrom":{".":{},"f:fieldRef":{".":{},"f:apiVersion":{},"f:fieldPath":{}}}},"k:{\"name\":\"WEBHOOK_NAME\"}":{".":{},"f:name":{},"f:value":{}},"k:{\"name\":\"WEBHOOK_PORT\"}":{".":{},"f:name":{},"f:value":{}}},"f:image":{},"f:imagePullPolicy":{},"f:livenessProbe":{".":{},"f:failureThreshold":{},"f:httpGet":{".":{},"f:httpHeaders":{},"f:path":{},"f:port":{},"f:scheme":{}},"f:initialDelaySeconds":{},"f:periodSeconds":{},"f:successThreshold":{},"f:timeoutSeconds":{}},"f:name":{},"f:ports":{".":{},"k:{\"containerPort\":8008,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}},"k:{\"containerPort\":8443,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}},"k:{\"containerPort\":9090,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:name":{},"f:protocol":{}}},"f:readinessProbe":{".":{},"f:failureThreshold":{},"f:httpGet":{".":{},"f:httpHeaders":{},"f:path":{},"f:port":{},"f:scheme":{}},"f:periodSeconds":{},"f:successThreshold":{},"f:timeoutSeconds":{}},"f:resources":{".":{},"f:limits":{".":{},"f:cpu":{},"f:memory":{}},"f:requests":{".":{},"f:cpu":{},"f:memory":{}}},"f:securityContext":{".":{},"f:allowPrivilegeEscalation":{}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:serviceAccount":{},"f:serviceAccountName":{},"f:terminationGracePeriodSeconds":{}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-01-12T10:00:04Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.244.1.17\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"eventing-webhook-token-ljck4","secret":{"secretName":"eventing-webhook-token-ljck4","defaultMode":420}}],"containers":[{"name":"eventing-webhook","image":"kind.local/knative.dev/eventing/cmd/webhook:b5a5737377c81d19b91ebf347e1a853232f74273063e4fa8669b2e5b7babfa55","ports":[{"name":"https-webhook","containerPort":8443,"protocol":"TCP"},{"name":"metrics","containerPort":9090,"protocol":"TCP"},{"name":"profiling","containerPort":8008,"protocol":"TCP"}],"env":[{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}},{"name":"CONFIG_LOGGING_NAME","value":"config-logging"},{"name":"METRICS_DOMAIN","value":"knative.dev/eventing"},{"name":"WEBHOOK_NAME","value":"eventing-webhook"},{"name":"WEBHOOK_PORT","value":"8443"},{"name":"SINK_BINDING_SELECTION_MODE","value":"exclusion"},{"name":"POD_NAME","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.name"}}}],"resources":{"limits":{"cpu":"200m","memory":"200Mi"},"requests":{"cpu":"20m","memory":"20Mi"}},"volumeMounts":[{"name":"eventing-webhook-token-ljck4","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"livenessProbe":{"httpGet":{"path":"/","port":8443,"scheme":"HTTPS","httpHeaders":[{"name":"k-kubelet-probe","value":"webhook"}]},"initialDelaySeconds":20,"timeoutSeconds":1,"periodSeconds":1,"successThreshold":1,"failureThreshold":3},"readinessProbe":{"httpGet":{"path":"/","port":8443,"scheme":"HTTPS","httpHeaders":[{"name":"k-kubelet-probe","value":"webhook"}]},"timeoutSeconds":1,"periodSeconds":1,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent","securityContext":{"allowPrivilegeEscalation":false}}],"restartPolicy":"Always","terminationGracePeriodSeconds":300,"dnsPolicy":"ClusterFirst","serviceAccountName":"eventing-webhook","serviceAccount":"eventing-webhook","nodeName":"kind-worker","securityContext":{},"affinity":{"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":100,"podAffinityTerm":{"labelSelector":{"matchLabels":{"app":"eventing-webhook"}},"topologyKey":"kubernetes.io/hostname"}}]}},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":false},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T09:59:58Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T10:00:04Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T10:00:04Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-01-12T09:59:57Z"}],"hostIP":"172.18.0.2","podIP":"10.244.1.17","podIPs":[{"ip":"10.244.1.17"}],"startTime":"2021-01-12T09:59:58Z","containerStatuses":[{"name":"eventing-webhook","state":{"running":{"startedAt":"2021-01-12T10:00:02Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"kind.local/knative.dev/eventing/cmd/webhook:latest","imageID":"sha256:ac72d6366971fda96b76839e59ef67b3f31eb66bc34f6225ec6bbe0a3433f81b","containerID":"containerd://0c8891f22e9e43092d29227f2c6e226147b8fb5beca5c56f20256986655fd7c1","started":true}],"qosClass":"Burstable"}}]}
2021-01-12T10:01:29.2349822Z pod/eventing-webhook-9d555fbd4-st8fw condition met

We're trying to fetch two different webhooks, one is found, other is not.

It fails because the first is not found there:

2021-01-12T10:01:30.2204077Z F0112 10:01:29.266090   26156 helpers.go:115] timed out waiting for the condition on pods/chaosduck-5d8694d86b-7qf9q
2021-01-12T10:01:30.2205303Z timed out waiting for the condition on pods/eventing-webhook-9d555fbd4-hnnbm

And the st8fw pod is found there to be running:

2021-01-12T10:01:30.5300104Z NAME                                    READY   STATUS    RESTARTS   AGE
2021-01-12T10:01:30.5302609Z eventing-controller-67d57bbcb9-c7n6n    1/1     Running   0          93s
2021-01-12T10:01:30.5304221Z eventing-webhook-9d555fbd4-st8fw        1/1     Running   0          93s
2021-01-12T10:01:30.5305729Z imc-controller-7586dff887-t68d6         1/1     Running   0          2m33s
2021-01-12T10:01:30.5307784Z imc-dispatcher-6d95d7b95f-dlkgb         1/1     Running   0          2m33s
2021-01-12T10:01:30.5309366Z mt-broker-controller-5b755c9445-n5gcp   1/1     Running   0          93s
2021-01-12T10:01:30.5311139Z mt-broker-filter-56864974c6-mhndh       1/1     Running   0          2m37s
2021-01-12T10:01:30.5312478Z mt-broker-ingress-6db4d947c6-2kpxk      1/1     Running   0          2m34s
2021-01-12T10:01:30.5313840Z sugar-controller-84b77b5675-hkdsj       1/1     Running   0          93s
2021-01-12T10:01:30.5315209Z zipkin-764f9875c6-w5rg8                 1/1     Running   0          2m20s

From timestamps, I'd guess chaos killed them, but I can't seem to find the logs for chaos. Will look into it more. But furthermore will look at the waiting code to see if there's something there that we can look. I'd think we only care that the webhook is up, not that a particular webhook pod with a specific name is up.

@vaikas
Copy link
Contributor Author

vaikas commented Jan 14, 2021

Yeah, so the step that does this is:

    - name: Wait for things to be up
      run: |
        kubectl -v 9 wait pod --for=condition=Ready -n ${SYSTEM_NAMESPACE} -l '!job-name'

Which seems to turn into list calls where it is looking for specific pod names in the lists.

@pierDipi
Copy link
Member

I can't seem to find the logs for chaos

chaos is disabled on KinD tests.

eventing/.github/workflows/kind-e2e.yaml

Lines 161 to 162 in d74dc25

# Be KinD to these tests.
kubectl scale -n${SYSTEM_NAMESPACE} deployment/chaosduck --replicas=0

I'd think we only care that the webhook is up, not that a particular webhook pod with a specific name is up

Do we have more than one replica?

@pierDipi
Copy link
Member

I'm testing on KinD eventing-kafka-broker and failures I see are never related to the eventing webhook not coming up.
However, since creating a Kafka cluster is slow and contains some dynamic resources to be created by the operator we added a sleep to prevent, "yes resources are ready but not all are there tho", so I guess eventually the webhook recovers.

@vaikas vaikas mentioned this issue Jan 14, 2021
Merged
@vaikas
Copy link
Contributor Author

vaikas commented Jan 14, 2021

@pierDipi hehe, that explains the no logs and disproves my theory :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Use wait_for_pods instead of kubectl wait for pods. Fix #4725 vaikas/eventing
2 participants
@vaikas @pierDipi