[feature] Static IP / Interface Alias support - similar to Docker MacVlan

[james-masson]

TL/DR This is a feature request - while CNI drivers can be used with java/exec drivers, you can't pass a static IP like you can with the docker driver's ipv4_address

---

There's a variety of software out there that requires fixed IP addresses. The ones I work with are typically UDP based message buses

Putting this traffic through NAT, or changing the apparent IP when a job moves from agent to agent will break the software.

With Docker, the solution is MacVLAN and ipv4_address - as it allows the container to take the IP with it, as it moves around the fleet of servers.

With Java/exec - it doesn't look like this is possible, I'd have to write something as part of the startup routine which creates the IP alias, and somehow manage to always clean it up at the end - without fail.

Anybody tackled this before with Nomad? Any good ways to achieve this without Dockerising or otherwise rewriting a fleet of existing java applications?

Ideally, Nomad would have a low level concept of an IP address per job, and be able to create/destroy interface aliases associated with it. Stretch goal of having something preventing IP re-use.

[shoenig]

Hi @james-masson have you looked into making use of Nomad's CNI plugin abilities?

https://www.hashicorp.com/blog/multi-interface-networking-and-cni-plugins-in-nomad-0-12
https://www.nomadproject.io/docs/job-specification/network#container-network-interface-cni

BTW, a lot of times these larger architecture questions get better traction on Discuss, where other users chip in with their own experiences.

[james-masson]

Thanks @shoenig

So you're saying that I can use my regular CNI networking plugins, but with the java task driver? Obviously I'd have to add a java param to bind to the interface created....

I can see how the syntax of the Job DSL would support that, but wasn't sure about compatibility - I'll have a play, thanks!

[schmichael]

So you're saying that I can use my regular CNI networking plugins, but with the java task driver?

Yes! Any task driver that supports group networks (Docker, exec, java, etc) should support your CNI plugin. Hopefuly the links @shoenig posted sent you in the right direction but feel free to reopen this issue if you run into any problems!

[james-masson]

Thanks @schmichael and @shoenig

Unless I'm missing something, it looks like Nomad's lacks the ability to supply a specific static IP to the task using Exec/Java driver.

With the docker driver I can use ipv4_address to supply this - but not with Exec/Java - the CNI driver does support this, but there's no way to pass it with Nomad network {} stanza that I can see.

I realise that I can get quasi-static IP as a side effect of some of the CNI plugins, but that's not what's required - we're looking for an equivalent to docker's ipv4_address.

[james-masson]

@schmichael @shoenig - can you re-open this please?

This is definitely a feature request now

[passcod]

We also desire this. Or some way to pass arbitrary data to CNI, but might as well make use of CNI's own ips capability feature.

[Amier3]

@james-masson @passcod

Re-opening this for now so we can reignite a discussion on it 😄

[passcod]

I have left the job that I used Nomad at, and do not use it currently, so I'm unsubscribing, sorry!

[nvx]

Even with Docker I suspect using the ipv4_address isn't really the right option when using CNI.

Having a look at how this might be implemented, it looks like being able to configure CNI_ARGS KV pairs in the network stanza that then get applied here:

nomad/client/allocrunner/networking_cni.go

Line 100 in 4b81337

if res, err = c.cni.Setup(ctx, alloc.ID, spec.Path, cni.WithCapabilityPortMap(getPortMapping(alloc, c.ignorePortMappingHostIP))); err != nil {

The last param of c.cni.Setup() is variadic, for each KV provided going into a cni.WithArgs() call https://pkg.go.dev/github.com/containerd/go-cni#WithArgs and passed in to Setup(). This would allow specifying the IP parameter which is handled by eg the host-local CNI ipam plugin.

Ideally the cni args would permit templating on the values to avoid IP conflicts when there is more than one count.

[Allan-Nava]

Even with Docker I suspect using the ipv4_address isn't really the right option when using CNI.

Having a look at how this might be implemented, it looks like being able to configure CNI_ARGS KV pairs in the network stanza that then get applied here:

nomad/client/allocrunner/networking_cni.go

Line 100 in 4b81337

if res, err = c.cni.Setup(ctx, alloc.ID, spec.Path, cni.WithCapabilityPortMap(getPortMapping(alloc, c.ignorePortMappingHostIP))); err != nil {

The last param of c.cni.Setup() is variadic, for each KV provided going into a cni.WithArgs() call https://pkg.go.dev/github.com/containerd/go-cni#WithArgs and passed in to Setup(). This would allow specifying the IP parameter which is handled by eg the host-local CNI ipam plugin.

Ideally the cni args would permit templating on the values to avoid IP conflicts when there is more than one count.

An example to add static ip?

[dani]

Also face this issue. I need to assign fixed IP to some of my containers, and connect them "directly" to the network without NAT or portmap. In my case, it's because I'm running a few dozen FTP servers as containers which are each reachable through a dedicated IPSec tunnels by remote endpoints (and each P2 of those tunnels only allows a single /32). I can assign a fixed IP to container only if I create a custom docker network on each node, eg

docker network create -d macvlan --subnet 10.99.5.0/24 --gateway 10.99.5.1 -o parent=ens18 foobarnet

Then, in my task, I can assign the IP I want

  task "ftp-client-128" {
    driver = "docker"
    config {
        image   = "ftpserver"
        ipv4_address = "10.99.5.128"
        network_mode = "foobarnet"
      }
    }
  }

I'd prefer using CNI to have Nomad handling everything, but without the possibility to pass CNI_ARGS from Nomad to the CNI plugin, I don't see a way to acheive this. It'd probably be even better if Nomad could handle ipvlan/macvlan natively without needing any CNI config (but beeing able to do it with CNI would be a great first step) :-)

[FibreFoX]

I am experimenting with ipvlan, and can confirm #9525 (comment) is working.

One thing I just want to add: this does not work when using group.network.mode = "bridge".

I was experimenting with a little setup (pihole) having a companion sidecar task (running unbound), where "the bridge network" was meant to have the static IP (via ipvlan) while both tasks are sharing that network (I was trying to avoid using consul, because I feel it would not be needed in my bridge-scenario). Not sure if this is even creatable using docker itself.

Does anyone have any idea how to archieve this using ipvlan, either the CNI-way or the "create it via docker locally first"?
(multiple interfaces are not a thing yet #11085 )