Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SendonDeployment with name > 31 characters #556

Closed
sasvaritoni opened this issue May 8, 2019 · 2 comments
Closed

SendonDeployment with name > 31 characters #556

sasvaritoni opened this issue May 8, 2019 · 2 comments
Labels
Milestone

Comments

@sasvaritoni
Copy link
Contributor

sasvaritoni commented May 8, 2019

seldonio/cluster-manager:0.2.6

When I try to create a seldondeployment with a name of lenght 32 or more, it will go to error state:

"name": "longname30longname30longname3012"
...
    "status": {
        "description": "Failed to create deployment seldon-6cbb0f8",
        "state": "Failed"
    }

In the logs of Seldon cluster manager:

2019-05-08 16:43:34.134 ERROR 1 --- [pool-1-thread-1] i.s.c.k.SeldonDeploymentControllerImpl   : Error creating deployment:{
  "metadata": {
    "selfLink": "",
    "resourceVersion": ""
  },
  "status": "Failure",
  "message": "Deployment.apps \"seldon-6cbb0f8\" is invalid: [metadata.labels: Invalid value: \"longname30longname30longname3012-longname30longname30longname3012\": must be no more than 63 characters, spec.selector.matchLabels: Invalid value: \"longname30longname30longname3012-longname30longname30longname3012\": must be no more than 63 characters, spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{\"seldon-app\":\"longname30longname30longname3012-longname30longname30longname3012\", \"seldon-app-longname30longname30longname3012\":\"seldon-vmx-vmx171-5000-img-vgg16-v0-0-1-4961034\"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: invalid label selector.]",
  "reason": "Invalid",
  "details": {
    "name": "seldon-6cbb0f8",
    "group": "apps",
    "kind": "Deployment",
    "causes": [{
      "reason": "FieldValueInvalid",
      "message": "Invalid value: \"longname30longname30longname3012-longname30longname30longname3012\": must be no more than 63 characters",
      "field": "metadata.labels"
    }, {
      "reason": "FieldValueInvalid",
      "message": "Invalid value: \"longname30longname30longname3012-longname30longname30longname3012\": must be no more than 63 characters",
      "field": "metadata.labels"
    }, {
      "reason": "FieldValueInvalid",
      "message": "Invalid value: \"longname30longname30longname3012-longname30longname30longname3012\": must be no more than 63 characters",
      "field": "spec.selector.matchLabels"
    }, {
      "reason": "FieldValueInvalid",
      "message": "Invalid value: v1.LabelSelector{MatchLabels:map[string]string{\"seldon-app-longname30longname30longname3012\":\"seldon-vmx-vmx171-5000-img-vgg16-v0-0-1-4961034\", \"seldon-app\":\"longname30longname30longname3012-longname30longname30longname3012\"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: invalid label selector.",
      "field": "spec.selector"
    }],
    "retryAfterSeconds": 0,
    "uid": ""
  },
  "code": 422
}

2019-05-08 16:43:34.135 ERROR 1 --- [pool-1-thread-1] i.s.c.k.SeldonDeploymentControllerImpl   : Failed to create deployment 

io.seldon.clustermanager.k8s.SeldonDeploymentException: Failed to create deployment seldon-6cbb0f8
	at io.seldon.clustermanager.k8s.SeldonDeploymentControllerImpl.createDeployments(SeldonDeploymentControllerImpl.java:94) ~[classes!/:0.2.6]
	at io.seldon.clustermanager.k8s.SeldonDeploymentControllerImpl.createOrReplaceSeldonDeployment(SeldonDeploymentControllerImpl.java:306) ~[classes!/:0.2.6]
	at io.seldon.clustermanager.k8s.SeldonDeploymentWatcher.processWatch(SeldonDeploymentWatcher.java:85) [classes!/:0.2.6]
	at io.seldon.clustermanager.k8s.SeldonDeploymentWatcher.watchSeldonMLDeployments(SeldonDeploymentWatcher.java:196) [classes!/:0.2.6]
	at io.seldon.clustermanager.k8s.SeldonDeploymentWatcher.watch(SeldonDeploymentWatcher.java:228) [classes!/:0.2.6]
	at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source) ~[na:na]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_181]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_181]
	at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65) [spring-context-4.3.20.RELEASE.jar!/:4.3.20.RELEASE]
	at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-4.3.20.RELEASE.jar!/:4.3.20.RELEASE]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_181]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_181]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_181]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]

When the name has length of 30 characters, it works fine.
It seems that the deployment name is actually repeated twice in that metadata, so the real limitation on length this way is 31 characters, not 63.

Is this a bug? Or am I doing something wrong?

Full json:

kubectl get seldondeployment longname30longname30longname3012 -o json
{
    "apiVersion": "machinelearning.seldon.io/v1alpha2",
    "kind": "SeldonDeployment",
    "metadata": {
        "annotations": {
            "ksonnet.io/managed": "{\"pristine\":\"H4sIAAAAAAAA/8xSwU7cQAy99zN8TpcERCvlDKoqlXbVoF4qtDIzJnGZsUczk4gVyr9Xk8AW1B7YW2+W7ff8nu1HwMA/KCZWgRY8moGFHGEUln6TyFmVDevJ1KALA55CBfcsFlroltoFBad7T5KhAk8ZLWaE9hEc3pJLJcIQoIWVqsCTilAupEZ9UCnYFpxKL+jprP5X1JzCXEGJ39K6dqaAprT7ByrgFMgsckQ0Y2aVRZ09GNhNhz1MDVQQov4ik3dvnXqUwBDJsskaE7Q//xJ1qL7WNFdwWFkXyKzYZ2NGJSMLPVGyx76ImfzDeyJsPjbteV3XJ+z73dT3zYd2qjf1pjhdWrejc1t1bPbQwue7r5q3kdJ6WMcTCaW0jXpLZdYdshsjXQ+R0qDOQntWwYBiHcVSzyZ0au4pL240lgsPOQeY5wpYODO6C3K478io2ARtU9cVBIqs9pA7ryCNxlBKLwY1FWT2pGP+gz1q9ZHQ8n/uZlI3errSUXK55s188+JKHZlIa76CTNGzLK/zKaKh7euhzTPXSlN4+ohhWB5mYGcjyUpEYoOyLB7zPpRlfr/sro976yfg1beLyy/HISMFxwYX/zfz/O43AAAA//8BAAD//078TreaBAAA\"}",
            "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"machinelearning.seldon.io/v1alpha2\",\"kind\":\"SeldonDeployment\",\"metadata\":{\"name\":\"longname30longname30longname3012\",\"namespace\":\"mmx\",\"selfLink\":\"/apis/machinelearning.seldon.io/v1alpha2/namespaces/mmx/seldondeployments/longname30longname30longname3012\",\"uid\":\"66740d40-71b0-11e9-8667-0050569a816e\",\"generation\":1,\"creationTimestamp\":\"2019-05-08T16:43:25Z\",\"labels\":{\"app\":\"seldon\",\"app.kubernetes.io/deploy-manager\":\"ksonnet\",\"ksonnet.io/component\":\"longname30longname30longname3012\"},\"annotations\":{\"ksonnet.io/managed\":\"{\\\"pristine\\\":\\\"H4sIAAAAAAAA/8xSwU7cQAy99zN8TpcERCvlDKoqlXbVoF4qtDIzJnGZsUczk4gVyr9Xk8AW1B7YW2+W7ff8nu1HwMA/KCZWgRY8moGFHGEUln6TyFmVDevJ1KALA55CBfcsFlroltoFBad7T5KhAk8ZLWaE9hEc3pJLJcIQoIWVqsCTilAupEZ9UCnYFpxKL+jprP5X1JzCXEGJ39K6dqaAprT7ByrgFMgsckQ0Y2aVRZ09GNhNhz1MDVQQov4ik3dvnXqUwBDJsskaE7Q//xJ1qL7WNFdwWFkXyKzYZ2NGJSMLPVGyx76ImfzDeyJsPjbteV3XJ+z73dT3zYd2qjf1pjhdWrejc1t1bPbQwue7r5q3kdJ6WMcTCaW0jXpLZdYdshsjXQ+R0qDOQntWwYBiHcVSzyZ0au4pL240lgsPOQeY5wpYODO6C3K478io2ARtU9cVBIqs9pA7ryCNxlBKLwY1FWT2pGP+gz1q9ZHQ8n/uZlI3errSUXK55s188+JKHZlIa76CTNGzLK/zKaKh7euhzTPXSlN4+ohhWB5mYGcjyUpEYoOyLB7zPpRlfr/sro976yfg1beLyy/HISMFxwYX/zfz/O43AAAA//8BAAD//078TreaBAAA\\\"}\"},\"clusterName\":\"\"},\"spec\":{\"name\":\"longname30longname30longname3012\",\"predictors\":[{\"name\":\"longname30longname30longname3012\",\"graph\":{\"name\":\"longname30longname30longname3012\",\"type\":\"MODEL\",\"endpoint\":{\"type\":\"REST\"}},\"componentSpecs\":[{\"spec\":{\"containers\":[{\"name\":\"longname30longname30longname3012\",\"image\":\"vmx-vmx171:5000/img_vgg16:v0.0.1\",\"livenessProbe\":{\"handler\":{\"tcpSocket\":{\"port\":\"http\"}},\"initialDelaySeconds\":100,\"timeoutSeconds\":1,\"periodSeconds\":5,\"successThreshold\":1,\"failureThreshold\":3},\"readinessProbe\":{\"handler\":{\"tcpSocket\":{\"port\":\"http\"}},\"initialDelaySeconds\":100,\"timeoutSeconds\":1,\"periodSeconds\":5,\"successThreshold\":1,\"failureThreshold\":3},\"imagePullPolicy\":\"IfNotPresent\"}],\"terminationGracePeriodSeconds\":1}}],\"replicas\":1,\"annotations\":{\"predictor_version\":\"v1\"}}],\"annotations\":{\"deployment_version\":\"v1\",\"project_name\":\"longname30longname30longname3012\"}},\"status\":{\"state\":\"Failed\",\"description\":\"Failed to create deployment seldon-6cbb0f8\"}}\n"
        },
        "clusterName": "",
        "creationTimestamp": "2019-05-08T16:43:25Z",
        "generation": 1,
        "labels": {
            "app": "seldon",
            "app.kubernetes.io/deploy-manager": "ksonnet",
            "ksonnet.io/component": "longname30longname30longname3012"
        },
        "name": "longname30longname30longname3012",
        "namespace": "mmx",
        "resourceVersion": "28705471",
        "selfLink": "/apis/machinelearning.seldon.io/v1alpha2/namespaces/mmx/seldondeployments/longname30longname30longname3012",
        "uid": "66740d40-71b0-11e9-8667-0050569a816e"
    },
    "spec": {
        "annotations": {
            "deployment_version": "v1",
            "project_name": "longname30longname30longname3012"
        },
        "name": "longname30longname30longname3012",
        "predictors": [
            {
                "annotations": {
                    "predictor_version": "v1"
                },
                "componentSpecs": [
                    {
                        "spec": {
                            "containers": [
                                {
                                    "image": "vmx-vmx171:5000/img_vgg16:v0.0.1",
                                    "imagePullPolicy": "IfNotPresent",
                                    "livenessProbe": {
                                        "failureThreshold": 3,
                                        "handler": {
                                            "tcpSocket": {
                                                "port": "http"
                                            }
                                        },
                                        "initialDelaySeconds": 100,
                                        "periodSeconds": 5,
                                        "successThreshold": 1,
                                        "timeoutSeconds": 1
                                    },
                                    "name": "longname30longname30longname3012",
                                    "readinessProbe": {
                                        "failureThreshold": 3,
                                        "handler": {
                                            "tcpSocket": {
                                                "port": "http"
                                            }
                                        },
                                        "initialDelaySeconds": 100,
                                        "periodSeconds": 5,
                                        "successThreshold": 1,
                                        "timeoutSeconds": 1
                                    }
                                }
                            ],
                            "terminationGracePeriodSeconds": 1
                        }
                    }
                ],
                "graph": {
                    "endpoint": {
                        "type": "REST"
                    },
                    "name": "longname30longname30longname3012",
                    "type": "MODEL"
                },
                "name": "longname30longname30longname3012",
                "replicas": 1
            }
        ]
    },
    "status": {
        "description": "Failed to create deployment seldon-6cbb0f8",
        "state": "Failed"
    }
}
@ukclivecox ukclivecox added the bug label May 8, 2019
@ukclivecox ukclivecox added this to the 0.2.x milestone May 8, 2019
@ukclivecox
Copy link
Contributor

This is probably a bug. We hash the core deployment names if the name is too long. But it looks like there is a leak into the labels metadata.

@ukclivecox
Copy link
Contributor

Fixed by SeldonIO/seldon-operator#5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants