option for docker-in-docker setup

[sadovnikov]

I tried to resolve two different problems: 1) failure to run docker-version of ES framework on minimesos and 2) failures of ES framework System Test https://ci.infra.container-solutions.com/view/All/job/ElasticSearch_SystemTest/. Seems both of them have the same underlying reason, which is being "worked-around" by test-framework-docker in minimesos project itself.

Let's assume a mesos agent runs on a host (container) with IP 127.0.0.3. It gets an assigned task to start container and let an application inside that container to start listening to a port, let's say 8080. The expectation is that application will become accessible under 127.0.0.3:8080, so it actually occupies the given port. However the new container gets a new IP address (127.0.0.4) and application becomes available at 127.0.0.4:8080 and attempts to access it at 127.0.0.3:8080 fail. ES even sets port mapping for the given port, but this succeeds only for the first agent - the second gets "port already in use". The system test of test-framework-docker cheats by looking up IP addresses of started containers and using them to access application.

To help resolving the problem with port mapping I logged https://issues.apache.org/jira/browse/MESOS-4215. However the proper solution is to keep the new containers on mesos agents, coming back to docker-in-docker setup. The serious drawback of this setup is necessity to pull images by mesos agents (instead of re-using images from the host). However by not-having a better idea, I'm going to create a brach in minimesos project, where I add possibility to start cluster in docker-in-docker mode, and will test my assumptions there.

[frankscholten]

Can you test and check if these problems still occur for https://github.com/mesos/elasticsearch 0.7.0?

After discussing with the team we came to the conclusion that supporting Docker-in-docker is more correct but very difficult to suppor. In general I think that it is easier to run frameworks without containers on minimesos like Jar based frameworks or other languages. It also speeds up development because you don't have to build images. Let's work on #156 to show an example of running Jar based frameworks.

[sashkachan]

--net=container:NAME_or_ID — Tells Docker to put this container’s processes inside
 of the network stack that has already been created inside of another container. 
The new container’s processes will be confined to their own filesystem and process 
list and resource limits, but will share the same IP address and port 
numbers as the first container, and processes on the two containers will be 
able to connect to each other over the loopback interface.

If we create a docker wrapper for mesos-agent to use to start executors with --net:mesos-agent-container-id, then they will share namespace, that should solve the issue, right?

[frankscholten]

The problem here is not Docker in Docker but the fact that in a regular Mesos cluster the agent process and containers use the same network stack while in a minimesos cluster
the agent uses its own network stack.

There are a few ways to deal with this.

1 - Run the Mesos agent in host mode

This will be fixed in #316

2 - Use Mesos' IP per container http://mesos.apache.org/documentation/latest/networking-for-mesos-managed-containers

We should experiment with this feature at some point.

[sadovnikov]

reported in #401 too

[frankscholten]

Closing since CNI and IP per container has landed in Mesos 1.0.0 we should look into that instead.