Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create the oauthConfig session secrets #152

Closed
wants to merge 1 commit into from

Conversation

mrogers950
Copy link
Contributor

@mrogers950 mrogers950 commented Nov 30, 2018

This PR starts filling in the default oauthConfig, by generating the session secrets to include in the openshift-kube-apiserver pod. Getting the oauthConfig filled in here to start a working oauth server is part of openshift/origin#21580

/cc @openshift/sig-auth @smarterclayton @derekwaynecarr

@openshift-ci-robot openshift-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Nov 30, 2018
@openshift-ci-robot
Copy link

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: mrogers950
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: deads2k

If they are not already assigned, you can assign the PR to them by writing /assign @deads2k in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@@ -76,7 +76,7 @@ oauthConfig:
sessionConfig:
sessionMaxAgeSeconds: 300
sessionName: ssn
sessionSecretsFile: ""
sessionSecretsFile: /var/run/secrets/session-secret/secrets
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

doesn't this break the bootstrap phase?

"github.com/openshift/cluster-kube-apiserver-operator/pkg/operator/v311_00_assets"
"github.com/openshift/library-go/pkg/operator/resource/resourceapply"
"github.com/openshift/library-go/pkg/operator/resource/resourcemerge"
"github.com/openshift/library-go/pkg/operator/resource/resourceread"
v1helpers "github.com/openshift/library-go/pkg/operator/v1helpers"
)

var (
Scheme = runtime.NewScheme()
Codecs = serializer.NewCodecFactory(Scheme)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do these have to be public?

@@ -76,7 +76,7 @@ oauthConfig:
sessionConfig:
sessionMaxAgeSeconds: 300
sessionName: ssn
sessionSecretsFile: ""
sessionSecretsFile: /var/run/secrets/session-secret/secrets
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wouldn't expect session configuration until after the oeprator is up.

@@ -40,3 +42,6 @@ spec:
- hostPath:
path: /etc/kubernetes/static-pod-resources/kube-apiserver-pod-REVISION
name: resource-dir
- secret:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

must be optional

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

must be optional

err, must be removed. You only get static-pod-resources

@@ -0,0 +1,33 @@
package crypto
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

library-go?

@@ -106,6 +127,33 @@ func manageKubeApiserverConfigMap_v311_00_to_latest(client coreclientv1.ConfigMa
return resourceapply.ApplyConfigMap(client, requiredConfigMap)
}

func manageSessionSecret_v311_00_to_latest(client coreclientv1.SecretsGetter) (*corev1.Secret, bool, error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this approach doesn't look right. We know this is likely to move and we know that it needs to rotate. Make a different control loop that manages creation and rotation by itself. Then you simply list the name to be copied down.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, ok. I'll revise this.

@deads2k
Copy link
Contributor

deads2k commented Dec 3, 2018

Pretty sure that this results in a pod that brings down the kube-apiserver and never comes back up because the static pod cannot mount a secret.

@openshift-bot
Copy link
Contributor

@mrogers950: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-bot openshift-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 3, 2018
@openshift-ci-robot openshift-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 4, 2018
@mrogers950
Copy link
Contributor Author

@deads2k @sttts I've updated with what should be closer to the right approach. There's a controller that creates the initial session secret and regenerates it on delete/update. Then I added a config observer that checks to see if the secret is there and updates sessionSecretsFile with the static pod resource path.
/cc @enj

@mrogers950
Copy link
Contributor Author

/retest

1 similar comment
@mrogers950
Copy link
Contributor Author

/retest

return err
}

func (sc *SessionSecretController) updateSessionSecret() error {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@deads2k wanted this to handle rotation, meaning you need the old data as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Specifically, what is the rotation behavior that we want?

return
}

// Generate the session secret if it doesn't exist initially.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This just looks weird to me.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In what way?

"k8s.io/client-go/tools/cache"
"k8s.io/client-go/util/workqueue"

legacyv1 "github.com/openshift/api/legacyconfig/v1"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, do not import this

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without that I'm unclear on how to handle the encoding for SessionSecret

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@deads2k osin still expects it to be in the legacy format.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@deads2k osin still expects it to be in the legacy format.

serialize by hand if need be. Do not rely on this package. You should fix it to be bilingual

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should fix it to be bilingual

If you do this, then you won't have a problem

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know what you mean by bilingual here..

@mrogers950
Copy link
Contributor Author

This update addresses nits and other comments and handles the initial creation of session-secret by using a watch on the openshift-kube-apiserver-* pods, rather than doing it all in the Run method.

Like @enj has mentioned we want to be able to cycle the session secret, but as to which event to trigger this on is not clear. Should a delete of session-secret result in a recreation containing the original SessionSecret with an addition of a new one to the SessionSecrets slice? How about on update with a check for some annotation to tell it to rotate?
The current push just handles the initial creation, I suppose that is good enough for the moment.

Also, manual serialization of SessionSecrets instead of relying on the current legacy encoder might be easier than it sounds to me, is there a good example of how to do this properly?

@mrogers950
Copy link
Contributor Author

I've pushed an update that removes the dependency on the legacy package. I was able to confirm that the manual encoding I'm doing now matches the legacy encoder. (turned out to be doing JSON not YAML)

@mrogers950
Copy link
Contributor Author

/retest

cache.FilteringResourceEventHandler{
FilterFunc: isKubeAPIServerPod,
Handler: cache.ResourceEventHandlerFuncs{
AddFunc: sc.enqueueSecret,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sets off my spidey sense. We need to be level driven and this is watching for an edge. Why would we do this once ever?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There needs to be something that triggers the creation of the secret initially, so watching for the pod that will be the consumer of it seems to be the most logical to me. If there is a better solution please suggest.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There needs to be something that triggers the creation of the secret initially, so watching for the pod that will be the consumer of it seems to be the most logical to me. If there is a better solution please suggest.

? You're trying to create a thing when the namespace is present, right? Why not watch for the namespace?


_, err = listers.SecretLister.Secrets(sessionSecretNamespace).Get(sessionSecretName)
if errors.IsNotFound(err) {
glog.Warningf("session secret %s/%s not found", sessionSecretNamespace, sessionSecretName)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use the event recorder if it is this important

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't really seem event-worthy to me. there will be some time before the secret is created so even warning it might be chatty.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't really seem event-worthy to me. there will be some time before the secret is created so even warning it might be chatty.

You're debugging a cluster where this secret isn't present. Would you like this information? Then it's not chatty.

@mrogers950
Copy link
Contributor Author

@deads2k @enj updated.

@mrogers950
Copy link
Contributor Author

/retest

},
}
_, err = sc.secretClient.Secrets(secret.Namespace).Create(secret)
return err
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ignore already exists

if err != nil {
errs = append(errs, err)
}
if currentSessionSecretsFilePath == sessionSecretPath {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no

observedConfig := map[string]interface{}{}
oauthConfigSessionSecretsFilePath := []string{"oauthConfig", "sessionConfig", "sessionSecretsFile"}

currentSessionSecretsFilePath, _, err := unstructured.NestedString(existingConfig, oauthConfigSessionSecretsFilePath...)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

build the prevObservedConfig and use it like all the others

if err != nil {
errs = append(errs, err)
}
if currentSessionSecretsFilePath == sessionSecretPath {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

secret is deleted after it is created, this block prevents the value from ever being cleared.

@mrogers950
Copy link
Contributor Author

I've revised the controller and config observer to handle a removal of the field if the secret is deleted. Although the config appears to update correctly, now I'm seeing one of the kube-apiserver pods staying stuck in a crashloop (the mounted session-secret file is not available) and I can't figure out why that is. It appears to never catch up with the latest config. Here's what the log looks like around at time the secret is created the first time and the config gets updated:

W1207 01:37:59.632179       1 observe_sessionsecret.go:45] session secret openshift-kube-apiserver/session-secret not found
I1207 01:37:59.633013       1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-cluster-kube-apiserver-operator", Name:"openshift-cluster-kube-apiserver-operator", UID:"11b9dc90-f9be-11e8-b5f5-0a32278910d2", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'ObserveRestrictedCIDRFailed' Required networking.podCIDR field is not set in install-config
I1207 01:38:04.840369       1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-cluster-kube-apiserver-operator", Name:"openshift-cluster-kube-apiserver-operator", UID:"11b9dc90-f9be-11e8-b5f5-0a32278910d2", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapUpdated' Updated ConfigMap/kube-apiserver-pod -n openshift-kube-apiserver because it changed
I1207 01:38:08.641554       1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-cluster-kube-apiserver-operator", Name:"openshift-cluster-kube-apiserver-operator", UID:"11b9dc90-f9be-11e8-b5f5-0a32278910d2", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapUpdated' Updated ConfigMap/config -n openshift-kube-apiserver because it changed
I1207 01:38:10.837478       1 deployment_controller.go:95] new revision 3 triggered by "secrets \"session-secret-2\" not found"
I1207 01:38:12.039352       1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-cluster-kube-apiserver-operator", Name:"openshift-cluster-kube-apiserver-operator", UID:"11b9dc90-f9be-11e8-b5f5-0a32278910d2", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapUpdated' Updated ConfigMap/kube-apiserver-pod-3 -n openshift-kube-apiserver because it changed
I1207 01:38:13.241799       1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-cluster-kube-apiserver-operator", Name:"openshift-cluster-kube-apiserver-operator", UID:"11b9dc90-f9be-11e8-b5f5-0a32278910d2", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapUpdated' Updated ConfigMap/config-3 -n openshift-kube-apiserver because it changed
I1207 01:38:19.142365       1 config_observer_controller.go:115] writing updated observedConfig: {"admissionPluginConfig":{"openshift.io/RestrictedEndpointsAdmission":{"configuration":{"restrictedCIDRs":["172.30.0.0/16"]}}},"imagePolicyConfig":{"internalRegistryHostname":"image-registry.openshift-image-registry.svc:5000"},"

A: servicesSubnet":"172.30.0.0/16","storageConfig":{"urls":["https://mrogersdev-etcd-0.devcluster.openshift.com:2379","https://mrogersdev-etcd-1.devcluster.openshift.com:2379","https://mrogersdev-etcd-2.devcluster.openshift.com:2379"]}}

B: oauthConfig":{"sessionConfig":{"sessionSecretsFile":"/etc/kubernetes/static-pod-resources/secrets/session-secret/secret"}},"servicesSubnet":"172.30.0.0/16","storageConfig":{"urls":["https://mrogersdev-etcd-0.devcluster.openshift.com:2379","https://mrogersdev-etcd-1.devcluster.openshift.com:2379","https://mrogersdev-etcd-2.devcluster.openshift.com:2379"]}}

I1207 01:38:19.144188       1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-cluster-kube-apiserver-operator", Name:"openshift-cluster-kube-apiserver-operator", UID:"11b9dc90-f9be-11e8-b5f5-0a32278910d2", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'ObserveRestrictedCIDRFailed' Required networking.podCIDR field is not set in install-config
I1207 01:38:19.159151       1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-cluster-kube-apiserver-operator", Name:"openshift-cluster-kube-apiserver-operator", UID:"11b9dc90-f9be-11e8-b5f5-0a32278910d2", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ObservedConfigChanged' Writing updated observed config
I1207 01:38:19.175376       1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-cluster-kube-apiserver-operator", Name:"openshift-cluster-kube-apiserver-operator", UID:"11b9dc90-f9be-11e8-b5f5-0a32278910d2", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for operator openshift-cluster-kube-apiserver-operator changed
I1207 01:38:22.441303       1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-cluster-kube-apiserver-operator", Name:"openshift-cluster-kube-apiserver-operator", UID:"11b9dc90-f9be-11e8-b5f5-0a32278910d2", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'SecretCreated' Created Secret/session-secret-3 -n openshift-kube-apiserver because it was missing
E1207 01:38:22.459100       1 deployment_controller.go:218] Operation cannot be fulfilled on kubeapiserveroperatorconfigs.kubeapiserver.operator.openshift.io "instance": the object has been modified; please apply your changes to the latest version and try again
E1207 01:38:22.459126       1 deployment_controller.go:259] key failed with : Operation cannot be fulfilled on kubeapiserveroperatorconfigs.kubeapiserver.operator.openshift.io "instance": the object has been modified; please apply your changes to the latest version and try again
I1207 01:38:23.438223       1 deployment_controller.go:95] new revision 3 triggered by "configmap/kube-apiserver-pod has changed"
I1207 01:38:23.640014       1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-cluster-kube-apiserver-operator", Name:"openshift-cluster-kube-apiserver-operator", UID:"11b9dc90-f9be-11e8-b5f5-0a32278910d2", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapUpdated' Updated ConfigMap/config -n openshift-kube-apiserver because it changed
I1207 01:38:25.441802       1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-cluster-kube-apiserver-operator", Name:"openshift-cluster-kube-apiserver-operator", UID:"11b9dc90-f9be-11e8-b5f5-0a32278910d2", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'ConfigMapUpdated' Updated ConfigMap/config-3 -n openshift-kube-apiserver because it changed
I1207 01:38:32.063074       1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-cluster-kube-apiserver-operator", Name:"openshift-cluster-kube-apiserver-operator", UID:"11b9dc90-f9be-11e8-b5f5-0a32278910d2", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RevisionCreate' Revision 3 created because configmap/kube-apiserver-pod has changed
I1207 01:38:32.087628       1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-cluster-kube-apiserver-operator", Name:"openshift-cluster-kube-apiserver-operator", UID:"11b9dc90-f9be-11e8-b5f5-0a32278910d2", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for operator openshift-cluster-kube-apiserver-operator changed
I1207 01:38:35.436823       1 installer_controller.go:240] "ip-10-0-25-126.ec2.internal" needs new revision 3
I1207 01:38:35.436908       1 installer_controller.go:249] "ip-10-0-25-126.ec2.internal" moving to (v1.NodeStatus) {
 NodeName: (string) (len=27) "ip-10-0-25-126.ec2.internal",
 CurrentRevision: (int32) 2,
 TargetRevision: (int32) 3,
 LastFailedRevision: (int32) 0,
 LastFailedRevisionErrors: ([]string) <nil>
}
I1207 01:38:35.461871       1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-cluster-kube-apiserver-operator", Name:"openshift-cluster-kube-apiserver-operator", UID:"11b9dc90-f9be-11e8-b5f5-0a32278910d2", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for operator openshift-cluster-kube-apiserver-operator changed
I1207 01:38:36.842765       1 installer_controller.go:229] "ip-10-0-25-126.ec2.internal" is in transition to 3, but has not made progress
I1207 01:38:38.036185       1 installer_controller.go:229] "ip-10-0-25-126.ec2.internal" is in transition to 3, but has not made progress
I1207 01:38:39.149865       1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-cluster-kube-apiserver-operator", Name:"openshift-cluster-kube-apiserver-operator", UID:"11b9dc90-f9be-11e8-b5f5-0a32278910d2", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'ObserveRestrictedCIDRFailed' Required networking.podCIDR field is not set in install-config
E1207 01:38:39.150190       1 event.go:212] Unable to write event: 'Patch https://172.30.0.1:443/api/v1/namespaces/openshift-cluster-kube-apiserver-operator/events/openshift-cluster-kube-apiserver-operator.156de939335e8bd0: dial tcp 172.30.0.1:443: connect: connection refused' (may retry after sleeping)
E1207 01:38:39.520767       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=671, ErrCode=NO_ERROR, debug=""
E1207 01:38:39.521295       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=671, ErrCode=NO_ERROR, debug=""
E1207 01:38:39.522377       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=671, ErrCode=NO_ERROR, debug=""
E1207 01:38:39.522692       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=671, ErrCode=NO_ERROR, debug=""
E1207 01:38:39.525607       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=671, ErrCode=NO_ERROR, debug=""
E1207 01:38:39.525736       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=671, ErrCode=NO_ERROR, debug=""
E1207 01:38:39.525828       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=671, ErrCode=NO_ERROR, debug=""
E1207 01:38:39.527079       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=671, ErrCode=NO_ERROR, debug=""
E1207 01:38:39.527219       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=671, ErrCode=NO_ERROR, debug=""
E1207 01:38:39.527307       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=671, ErrCode=NO_ERROR, debug=""
E1207 01:38:39.527392       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=671, ErrCode=NO_ERROR, debug=""
E1207 01:38:39.527476       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=671, ErrCode=NO_ERROR, debug=""
E1207 01:38:39.527551       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=671, ErrCode=NO_ERROR, debug=""
E1207 01:38:39.527634       1 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=671, ErrCode=NO_ERROR, debug=""
W1207 01:38:39.891254       1 reflector.go:341] github.com/openshift/cluster-kube-apiserver-operator/vendor/k8s.io/client-go/informers/factory.go:130: watch of *v1.ConfigMap ended with: too old resource version: 16522 (16527)
I1207 01:38:40.173144       1 installer_controller.go:229] "ip-10-0-25-126.ec2.internal" is in transition to 3, but has not made progress
W1207 01:38:40.251234       1 reflector.go:341] github.com/openshift/cluster-kube-apiserver-operator/pkg/generated/informers/externalversions/factory.go:101: watch of *v1alpha1.KubeAPIServerOperatorConfig ended with: too old resource version: 16630 (16691)
W1207 01:38:40.357404       1 reflector.go:341] github.com/openshift/cluster-kube-apiserver-operator/vendor/github.com/openshift/client-go/config/informers/externalversions/factory.go:101: watch of *v1.Image ended with: too old resource version: 9323 (16696)
I1207 01:38:41.409278       1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-cluster-kube-apiserver-operator", Name:"openshift-cluster-kube-apiserver-operator", UID:"11b9dc90-f9be-11e8-b5f5-0a32278910d2", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for operator openshift-cluster-kube-apiserver-operator changed
I1207 01:38:42.040417       1 installer_controller.go:229] "ip-10-0-25-126.ec2.internal" is in transition to 3, but has not made progress
I1207 01:38:44.040272       1 installer_controller.go:229] "ip-10-0-25-126.ec2.internal" is in transition to 3, but has not made progress
I1207 01:38:45.843211       1 installer_controller.go:229] "ip-10-0-25-126.ec2.internal" is in transition to 3, but has not made progress
I1207 01:38:47.638638       1 installer_controller.go:229] "ip-10-0-25-126.ec2.internal" is in transition to 3, but has not made progress
I1207 01:38:51.638295       1 installer_controller.go:229] "ip-10-0-25-126.ec2.internal" is in transition to 3, but has not made progress

@deads2k
Copy link
Contributor

deads2k commented Dec 17, 2018

/test all

@openshift-ci-robot
Copy link

@mrogers950: The following test failed, say /retest to rerun them all:

Test name Commit Details Rerun command
ci/prow/e2e-aws 5173411 link /test e2e-aws

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@openshift-ci-robot
Copy link

@mrogers950: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 3, 2019
@mrogers950
Copy link
Contributor Author

We'll be handling the session secrets file in the auth operator at this point.

@mrogers950 mrogers950 closed this Jan 14, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants