Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Singularity dies when deploy.resources.numPorts=0 and BRIDGE mode docker container has only LITERAL port mappings #926

Closed
whitedr opened this issue Feb 29, 2016 · 0 comments · Fixed by #927
Assignees

Comments

@whitedr
Copy link
Contributor

whitedr commented Feb 29, 2016

Submitting a deploy request with the following containerInfo and resources section in it:

{
    "containerInfo": {
      "docker": {
        "image": "docker.yb0t.cc/riemann:0.2.10",
        "network": "BRIDGE",
        "portMappings": [
          {
            "containerPortType": "LITERAL",
            "containerPort": 5555,
            "hostPortType": "LITERAL",
            "hostPort": 4255,
            "protocol": "tcp"
          },
          {
            "containerPortType": "LITERAL",
            "containerPort": 5556,
            "hostPortType": "LITERAL",
            "hostPort": 4256,
            "protocol": "tcp"
          },
          {
            "containerPortType": "LITERAL",
            "containerPort": 4567,
            "hostPortType": "LITERAL",
            "hostPort": 4257,
            "protocol": "tcp"
          }
        ],
        "privileged": true
      },
      "type": "DOCKER"
    },
    "resources": {
      "cpus": 1,
      "memoryMb": 1024,
      "numPorts": 0
    }
  }

causes singularity to die/abort with the following exception info in the log:

2016-02-29T14:55:08.03091 ERROR [2016-02-29 14:55:08,021] com.hubspot.singularity.mesos.SingularityMesosScheduler: Received fatal error while accepting offers - will decline all available offers
2016-02-29T14:55:08.03094 ! java.lang.ArrayIndexOutOfBoundsException: 0
2016-02-29T14:55:08.03094 ! at com.hubspot.mesos.MesosUtils.getPorts(MesosUtils.java:87) ~[SingularityService-shaded.jar:0.4.9]
2016-02-29T14:55:08.03094 ! at com.hubspot.singularity.mesos.SingularityMesosTaskBuilder.buildTask(SingularityMesosTaskBuilder.java:93) ~[SingularityService-shaded.jar:0.4.9]
2016-02-29T14:55:08.03094 ! at com.hubspot.singularity.mesos.SingularityMesosScheduler.match(SingularityMesosScheduler.java:236) ~[SingularityService-shaded.jar:0.4.9]
2016-02-29T14:55:08.03095 ! at com.hubspot.singularity.mesos.SingularityMesosScheduler.resourceOffers(SingularityMesosScheduler.java:174) ~[SingularityService-shaded.jar:0.4.9]
2016-02-29T14:55:08.03095 ! at com.hubspot.singularity.mesos.SingularityMesosScheduler$$EnhancerByGuice$$4737360b.CGLIB$resourceOffers$3(<generated>) [SingularityService-shaded.jar:0.4.9]
2016-02-29T14:55:08.03096 ! at com.hubspot.singularity.mesos.SingularityMesosScheduler$$EnhancerByGuice$$4737360b$$FastClassByGuice$$a1320235.invoke(<generated>) [SingularityService-shaded.jar:0.4.9]
2016-02-29T14:55:08.03096 ! at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228) [SingularityService-shaded.jar:0.4.9]
2016-02-29T14:55:08.03096 ! at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:75) [SingularityService-shaded.jar:0.4.9]
2016-02-29T14:55:08.03096 ! at com.palominolabs.metrics.guice.TimedInterceptor.invoke(TimedInterceptor.java:47) [SingularityService-shaded.jar:0.4.9]
2016-02-29T14:55:08.03097 ! at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:75) [SingularityService-shaded.jar:0.4.9]
2016-02-29T14:55:08.03097 ! at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:55) [SingularityService-shaded.jar:0.4.9]
2016-02-29T14:55:08.03097 ! at com.hubspot.singularity.mesos.SingularityMesosScheduler$$EnhancerByGuice$$4737360b.resourceOffers(<generated>) [SingularityService-shaded.jar:0.4.9]
2016-02-29T14:55:08.03097 ! at com.hubspot.singularity.mesos.SingularityMesosSchedulerDelegator.resourceOffers(SingularityMesosSchedulerDelegator.java:175) [SingularityService-shaded.jar:0.4.9]
2016-02-29T14:55:08.03097 ERROR [2016-02-29 14:55:08,025] com.hubspot.singularity.mesos.SingularityMesosSchedulerDelegator: Scheduler threw an uncaught exception - exiting
2016-02-29T14:55:08.03098 ! java.lang.ArrayIndexOutOfBoundsException: 0
2016-02-29T14:55:08.03098 ! at com.hubspot.mesos.MesosUtils.getPorts(MesosUtils.java:87) ~[SingularityService-shaded.jar:0.4.9]
2016-02-29T14:55:08.03098 ! at com.hubspot.singularity.mesos.SingularityMesosTaskBuilder.buildTask(SingularityMesosTaskBuilder.java:93) ~[SingularityService-shaded.jar:0.4.9]
2016-02-29T14:55:08.03098 ! at com.hubspot.singularity.mesos.SingularityMesosScheduler.match(SingularityMesosScheduler.java:236) ~[SingularityService-shaded.jar:0.4.9]
2016-02-29T14:55:08.03098 ! at com.hubspot.singularity.mesos.SingularityMesosScheduler.resourceOffers(SingularityMesosScheduler.java:174) ~[SingularityService-shaded.jar:0.4.9]
2016-02-29T14:55:08.03098 ! at com.palominolabs.metrics.guice.TimedInterceptor.invoke(TimedInterceptor.java:47) ~[SingularityService-shaded.jar:0.4.9]
2016-02-29T14:55:08.03098 ! at com.hubspot.singularity.mesos.SingularityMesosSchedulerDelegator.resourceOffers(SingularityMesosSchedulerDelegator.java:175) ~[SingularityService-shaded.jar:0.4.9]
2016-02-29T14:55:08.03099 ERROR [2016-02-29 14:55:08,025] com.hubspot.singularity.SingularityAbort: Singularity on vagrant-devops-singularity is aborting due to UNRECOVERABLE_ERROR
2016-02-29T14:55:08.03099 WARN  [2016-02-29 14:55:08,025] com.hubspot.singularity.SingularityAbort: Couldn't send abort mail because no SMTP configuration is present
2016-02-29T14:55:08.03099 INFO  [2016-02-29 14:55:08,028] com.hubspot.singularity.SingularityAbort: Attempting to flush logs and wait 00:00.100 ...
2016-02-29T14:55:08.19300 I0229 14:55:08.182867  7619 sched.cpp:1805] Asked to abort the driver

Essentially what we're trying to do here is run a container in bridge mode where all the ports are statically allocated. Since no dynamically allocated ports are needed (i.e. no FROM_OFFER ports in the port mappings), I've set resources.numPorts=0. It seems this combination clearly makes singularity unhappy. In looking at this, I was wondering why resources.numPorts is even needed and why it can't be dynamically determined by the number of FROM_OFFER ports in the port mappings section?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants