wrong --net settings and cni subnet config ignored

[scicco]

Hello,

I'm having trouble with IP lease.

Here are the relevant configuration files:

whoami

root

cat /etc/issue

Ubuntu 22.04.1 LTS \n \l

$> singularity-compose -v

0.1.18

$> cat /usr/local/etc/singularity/singularity.conf | grep "allow net"

allow net networks = bridge, fakeroot
allow net users = deployer
allow net groups = deployer

$> cat /usr/local/etc/singularity/network/00_bridge.conflist (the same IP range for /usr/local/etc/singularity/network/40_fakeroot.conflist)

{
    "cniVersion": "0.4.0",
    "name": "bridge",
    "plugins": [
        {
            "type": "bridge",
            "bridge": "sbr0",
            "isGateway": true,
            "ipMasq": true,
            "ipam": {
                "type": "host-local",
                "subnet": "192.168.20.0/24",
                "routes": [
                    { "dst": "0.0.0.0/0" }
                ]
            }
        },
        {
            "type": "firewall"
        },
        {
            "type": "portmap",
            "capabilities": {"portMappings": true},
            "snat": true
        }
    ]
}

#singularity-compose.yml
version: "2.0"
instances:
  cg-cache:
    name: redis
    run:
      background: true
    build:
      context: .
      recipe: redis.def
    start:
      background: true
      options:
        - "env-file=./env-file.sh"
      command: redis-server --bind 0.0.0.0 --requirepass ${REDIS_PASSWORD}
    volumes:
      - ./env-file.sh:/.singularity.d/env/env-file.sh
    network:
      enable: true
      allocate_ip: true
      ports:
        - "6379:6379"

#redis.def
BootStrap:docker
From:redis:latest

%runscript

    exec /usr/local/bin/docker-entrypoint.sh "$@"

#env-file.sh
export REDIS_PASSWORD=ThisIsNotAStrongPassword!

# running output of:
singularity-compose --debug up
Creating redis1
DEBUG singularity instance start --bind /home/deployer/cgp/singularity-test/env-file.sh:/.singularity.d/env/env-file.sh --bind /home/deployer/cgp/singularity-test/resolv.conf:/etc/resolv.conf --bind /home/deployer/cgp/singularity-test/etc.hosts:/etc/hosts --net --network none --network-args "IP=10.22.0.2" --env-file=./env-file.sh --hostname redis1 --writable-tmpfs /home/deployer/cgp/singularity-test/redis.sif redis1
DEBUG singularity run  instance://redis1
singularity run instance://redis1

$> singularity instance list

INSTANCE NAME    PID      IP    IMAGE
redis1           19524          /home/deployer/cgp/singularity-test/redis.sif

$> singularity-compose ps

INSTANCES  NAME         PID     IP              IMAGE
1         redis1	19524		redis.sif

I think --net --network none --network-args "IP=10.22.0.2" inside singularity-compose debug message
is causing the issue because:

1. it doesn't use bridge
2. it doesn't use my bridge network config: 192.168.20.0/24 instead is using default one (10.22.0.x)

What I'm doing wrong?

Thank you in advance

[vsoch]

We probably need to add support to be able to detect that if singularity doesn’t do it. And you’ve tried with sudo?

I’m not super experienced with custom networking - @pierlauro @PauloMigAlmeida do you have any insights?

[PauloMigAlmeida]

I agree with @vsoch, we need to add support for other variations of network configs if we want to cover these use cases.

@scicco One workaround I had to recently do and I believe it might work for you too (remains to be seen) is to "disable the network" on singularity-compose so it doesn't try add the --net *** params and potentially singularity can pick up the hosting OS network configs (assuming that the bridge config is the default one).

  redis:
    image: cvatredis.sif
    start:
      options:
        - containall
    network:
      enable: false

PS.: if you do that, you won't be able to list the IP addresses when you run singularity instance list as a side-effect of the workaround. Give that a spin to see if that works.

[vsoch]

Thank you @PauloMigAlmeida ! And if there is something extra we can do to make this easier, let's chat about that too.

[scicco]

We probably need to add support to be able to detect that if singularity doesn’t do it. And you’ve tried with sudo?

I've tried with sudo, unfortunately I got the same result

[vsoch]

okay, perhaps try the workaround suggested by @PauloMigAlmeida then?

[scicco]

okay, perhaps try the workaround suggested by @PauloMigAlmeida then?

It does not work either, here is the step done:

$> cat singularity-compose.yml

version: "2.0"
instances:
  cg-cache:
    name: redis
    run:
      background: true
    build:
      context: .
      recipe: redis.def
    start:
      background: true
      options:
        - "env-file=./env-file.sh"
      command: redis-server --bind 0.0.0.0 --requirepass ${REDIS_PASSWORD}
    volumes:
      - ./env-file.sh:/.singularity.d/env/env-file.sh
    network:
      enable: false

$> singularity-compose --debug up -d
DEBUG singularity instance start --bind /home/deployer/cgp/singularity-test2/env-file.sh:/.singularity.d/env/env-file.sh --bind /home/deployer/cgp/singularity-test2/resolv.conf:/etc/resolv.conf --bind /home/deployer/cgp/singularity-test2/etc.hosts:/etc/hosts --env-file=./env-file.sh --hostname redis1 --writable-tmpfs /home/deployer/cgp/singularity-test2/redis.sif redis1
DEBUG singularity run  instance://redis1
singularity run instance://redis1

$> singularity instance list
INSTANCE NAME    PID      IP    IMAGE
redis1           15001          /home/deployer/cgp/singularity-test2/redis.sif

[scicco]

What I'm trying to achieve with singularity-compose is the same as the following script steps:

#!/bin/bash

set -eo pipefail

echo "[1/6] stopping existing instance"
singularity instance stop redis* || true

# (docker-compose build --no-cache equivalent) #OPTIONAL
echo "[2/6] cleaning cache and old image"
singularity cache clean -f || true

echo "[3/6] removing old sif image" #OPTIONAL
rm -f redis.sif

echo "[4/6] building sif image"
singularity build redis.sif redis.def || true

echo "[5/6] spinning up a new redis instance"
singularity instance start --hostname redis --writable-tmpfs --net --network=static-redis --network-args "portmap=6379:6379/tcp" $PWD/redis.sif redis

echo "[6/6] launching redis server in background"
singularity exec instance://redis redis-server --bind 0.0.0.0 --requirepass ThisIsNotAStrongPassword! &

$> cat singularity-compose.yml

version: "2.0"
instances:
  cg-cache:
    name: redis
    run:
      background: true
    build:
      context: .
      recipe: redis.def
    start:
      background: true
      options:
        - "env-file=./env-file.sh"
      command: redis-server --bind 0.0.0.0 --requirepass ${REDIS_PASSWORD}
    volumes:
      - ./env-file.sh:/.singularity.d/env/env-file.sh
    network:
      enable: true
      allocate_ip: true
      ports:
        - 6379:6379

I also needed to add the following static cni config file to have a specific IP but using default bridge with 192.168.20.0/24 will work and add IP to the instance:

$> cat /usr/local/etc/singularity/network/50_static-redis.conflist

{
  "cniVersion": "0.4.0",
  "name": "static-redis",
  "plugins": [
    {
      "type": "bridge",
      "bridge": "sbr0",
      "isGateway": true,
      "ipMasq": true,
      "ipam": {
        "type": "static",
        "addresses": [
          {
            "address": "192.168.20.10/24",
            "gateway": "192.168.20.254"
          }
        ],
        "routes": [
          { "dst": "0.0.0.0/0" }
        ]
      }
    },
    {
      "type": "firewall"
    },
    {
      "type": "portmap",
      "capabilities": {
        "portMappings": true
      },
      "snat": true
    }
  ]
}

if I run all commands I got this result:

$> singularity instance list

INSTANCE NAME    PID      IP               IMAGE
redis            15354    192.168.20.10    /home/deployer/cgp/singularity-test/redis.sif

and now it's reachable from redis-cli on localhost (thanks to port forwarding) or either with

redis-cli -h 192.168.20.10

and auth <PASSWORD> must be used to access db (for both commands)

[scicco]

I've created a test repo to simplify debug process. Hope this helps to clarify better my issue

[vsoch]

This is perfect! I'll make some time soon to work on this for you, stay tuned!

[vsoch]

@scicco how does the above work for you without any sudo? It simply does not work for me without sudo - if you add fakeroot, fakeroot isn't allowed to use the custom bridge.

[scicco]

@scicco how does the above work for you without any sudo? It simply does not work for me without sudo - if you add fakeroot, fakeroot isn't allowed to use the custom bridge.

I'm not using fakeroot and running singularity as root user to avoid any permission related issue. If you check inside files of my test repo you should not see any usage of fakeroot. Anyway to use a network I think you have to add it inside singularity.conf. In the test repo I'm launching it with root.

Anyway the following command (according to guide and this one) should allow you to use fakeroot and bridge:

#as root user or using sudo
echo "allow net networks = bridge, fakeroot" >> /etc/singularity/singularity.conf

I'll double check again my test repo to be sure

[vsoch]

Ok gotcha. Since most users don’t by default run as root I’ll modify your example files to have it. I’ll have more time to look at this again tomorrow evening.

[vsoch]

Gotcha - I think I should be able to finish this up and do a PR tonight. Stay tuned!

[vsoch]

See #61

[scicco]

I think #61 will address only the use case of a singularity-compose.yml config file containing network.args.
If the config doesn't use it it still fails.

It should be useful to support --network options so that it's possible to use a command like:

singularity instance start --hostname redis --writable-tmpfs --net --network static --network-args "portmap=6379:6379/tcp" $PWD/redis.sif redis as in the custom example you added

where --network redis-static allow us to a cni config different from bridge or fakeroot

the new singularity-compose.yml could be:

version: "2.0"
instances:
  cg-cache:
    name: redis
    run:
      background: true
    build:
      context: .
      recipe: redis.def
    start:
      background: true
    volumes:
      - ./env-file.sh:/.singularity.d/env/env-file.sh
    network:
      enable: true
      type: static-redis #this is used to specify which type use, default is bridge when enabled == true or none if enable == false
      allocate_ip: true

      # this should not be needed      
      #args:
      # - '"portmap=6379:6379/tcp"'
      ports:
        - 6379:6379

I can open a separate issue for the --network feature is it helps

[vsoch]

Ahh I think I follow - so the example would have type: 50_static-redis to correspond to the prefix (without extension) of the CNI config?

[scicco]

it should match the name property of the cni configuration file, so in this case would be static-redis.

I've updated the previous comment, I wrongly used static instead of static-redis

[scicco]

I've added a PR (#62) to try to address the --network "custom" type

[vsoch]

Is there some additional plugin I need for singularity?

FATAL:   container creation failed: plugin type="bridge" failed (add): invalid CIDR 10.22.0.2: invalid CIDR address: 10.22.0.2

The way I did it works, this one does not.

[scicco]

That IP is probably leased by cni configuration when unchanged, so in this case:
/usr/local/etc/singularity/network/00_bridge.conflist if you are using bridge (no fakeroot)
or /usr/local/etc/singularity/network/40_fakeroot.configlist are specifying plugins.ipam.subnet property, which probably should be changed in 192.168.20.0/24 (as my example) or another private subnet that is valid

10.22.0.x is also present as default value inside the codebase in

singularity-compose/scompose/client/__init__.py

Line 145 in e301c95

default="10.22.0.0/16",

singularity-compose/scompose/project/project.py

Line 243 in e301c95

def get_ip_lookup(self, names, bridge="10.22.0.0/16"):

singularity-compose/scompose/project/project.py

Line 454 in e301c95

self, names=None, writable_tmpfs=True, bridge="10.22.0.0/16", no_resolv=False

singularity-compose/scompose/project/project.py

Line 466 in e301c95

bridge="10.22.0.0/16",

singularity-compose/scompose/project/project.py

Line 487 in e301c95

bridge="10.22.0.0/16",

[vsoch]

I think I just need a full example that reproduces what you have, if that might be possible!

[scicco]

Ok, sure. Can you run singularity-compose without fakeroot options and use root? I'll modify my example repo to reproduce it

[vsoch]

Yep! If you go to the examples I linked in the PR the README instructions and scripts are exactly what I was running.

[scicco]

I've run your codebase version: https://github.com/singularityhub/singularity-compose/tree/e301c950c177ca6201769cce8d85243ce9158879

as root user:

$> singularity-compose up -d
Creating redis1
singularity run instance://redis1

$> singularity instance list
INSTANCE NAME    PID     IP           IMAGE
redis1           3063    10.22.0.2    /prj/singularity-compose/singularity-compose-examples/v2.0/custom-network/redis.sif

It works, but it's not using /usr/local/etc/singularity/network/00_bridge.conflist setup plugins.ipam.subnet parameter: its using 10.22.0.2 instead of 192.168.20.0/24 (but let's skip this problem for another time)

with my repo commit: https://github.com/scicco/singularity-compose/tree/25bd22491967217949cd3caae74c7f7b212055ca

running as root:

singularity-compose up -d
Creating redis1
singularity run instance://redis1

$> singularity instance list
INSTANCE NAME    PID     IP           IMAGE
redis1           4243    10.22.0.2    /prj/singularity-compose/singularity-compose-examples/v2.0/custom-network/redis.sif

it's the same behaviour

[scicco]

I've updated #62, adding a link to example repo with updated files for custom network

[vsoch]

@scicco I still get the same error about the port mapping - when you mention:

That IP is probably leased by cni configuration when unchanged, so in this case:
/usr/local/etc/singularity/network/00_bridge.conflist if you are using bridge (no fakeroot)
or /usr/local/etc/singularity/network/40_fakeroot.configlist are specifying plugins.ipam.subnet property, which probably should be changed in 192.168.20.0/24 (as my example) or another private subnet that is valid

This needs to be an explicit part of the example, so a new user (and myself) can reproduce what you do, to a T, and have it work. Currently doing that I get the same error. I see your bridge file has the same ips that are in the statis-redis:

{
    "cniVersion": "0.4.0",
    "name": "bridge",
    "plugins": [
        {
            "type": "bridge",
            "bridge": "sbr0",
            "isGateway": true,
            "ipMasq": true,
            "ipam": {
                "type": "host-local",
                "subnet": "192.168.20.0/24",
                "routes": [
                    { "dst": "0.0.0.0/0" }
                ]
            }
        },
        {
            "type": "firewall"
        },
        {
            "type": "portmap",
            "capabilities": {"portMappings": true},
            "snat": true
        }
    ]
}

So ideally there can be an instruction for using this file without re-writing over a default singularity configuration? Or if that is the only way, then a clear example for how to do that (and undo it). Thanks!

[scicco]

@vsoch can you try with this changes?

1. Ensure to have 10.22.0.0/16 inside /usr/local/etc/network/00_bridge.conflist

cat /usr/local/etc/network/00_bridge.conflist

{
    "cniVersion": "0.4.0",
    "name": "bridge",
    "plugins": [
        {
            "type": "bridge",
            "bridge": "sbr0",
            "isGateway": true,
            "ipMasq": true,
            "ipam": {
                "type": "host-local",
                "subnet": "10.22.0.0/16",
                "routes": [
                    { "dst": "0.0.0.0/0" }
                ]
            }
        },
        {
            "type": "firewall"
        },
        {
            "type": "portmap",
            "capabilities": {"portMappings": true},
            "snat": true
        }
    ]
}

and try to modify rebuild.sh script like this:
(it's adapted from this version.

#!/bin/bash

set -eo pipefail

echo "[1/8] stopping existing instance"
singularity-compose stop || true

echo "[2/8] cleanup sbr0 network interface"
sudo ip link delete sbr0 || true

echo "[4/8] cleaning cache and old image"
singularity cache clean -f || true

echo "[5/8] removing old sif image" #OPTIONAL
rm -f redis.sif

echo "[6/8] removing etc.hosts and resolv.conf"
rm -f etc.hosts
rm -f resolv.conf

echo "[7/8] running singularity-compose"
singularity-compose build && singularity-compose up -d

echo "[8/8] show instances"
singularity-compose ps

Now rebuild.sh should work

#61 does not work currently if you have a subnet class different from bridge.

I have tried to address this problem with the changes inside #62, where I check the right subnet to use instead of using always 10.22.0.0/16

[vsoch]

Could you show me the underlying command being run (e.g., add --debug to singularity-compose)?

[scicco]

@vsoch sure. I've fixed and renamed the custom configuration because it was wrong (it used the same bridge name sbr0 which caused problems and switched into bridge cni config typology).

$> cat /usr/local/etc/network/50_redis.conflist

{
    "cniVersion": "0.4.0",
    "name": "redis",
    "plugins": [
        {
            "type": "bridge",
            "bridge": "srr0",
            "isGateway": true,
            "ipMasq": true,
            "ipam": {
                "type": "host-local",
                "subnet": "192.168.20.0/24",
                "routes": [
                    { "dst": "0.0.0.0/0" }
                ]
            }
        },
        {
            "type": "firewall"
        },
        {
            "type": "portmap",
            "capabilities": {"portMappings": true},
            "snat": true
        }
    ]
}

$> cat /usr/local/etc/network/00_bridge.conflist

{
    "cniVersion": "0.4.0",
    "name": "bridge",
    "plugins": [
        {
            "type": "bridge",
            "bridge": "sbr0",
            "isGateway": true,
            "ipMasq": true,
            "ipam": {
                "type": "host-local",
                "subnet": "10.22.0.0/16",
                "routes": [
                    { "dst": "0.0.0.0/0" }
                ]
            }
        },
        {
            "type": "firewall"
        },
        {
            "type": "portmap",
            "capabilities": {"portMappings": true},
            "snat": true
        }
    ]
}

$> cat singularity-compose.yml

version: "2.0"
instances:
  cg-cache:
    name: redis
    run:
      background: true
    build:
      context: .
      recipe: redis.def
    start:
      background: true
    volumes:
      - ./env-file.sh:/.singularity.d/env/env-file.sh
    network:
      enable: true
      allocate_ip: true
      type: redis
      ports:
        - 6379:6379

singularity-compose --debug up

Creating redis1
DEBUG network_type is: redis
DEBUG singularity instance start --bind /prj/singularity-compose-examples/v2.0/custom-network/env-file.sh:/.singularity.d/env/env-file.sh --bind /prj/singularity-compose-examples/v2.0/custom-network/resolv.conf:/etc/resolv.conf --bind /prj/singularity-compose-examples/v2.0/custom-network/etc.hosts:/etc/hosts --net --network redis --network-args "IP=192.168.20.2" --hostname redis1 --writable-tmpfs /prj/singularity-compose-examples/v2.0/custom-network/redis.sif redis1
DEBUG singularity run  instance://redis1
singularity run instance://redis1

singularity-compose ps

INSTANCES  NAME         PID     IP              IMAGE
1         redis1	57204	192.168.20.2	redis.sif

You can find the latest configuration files here and singularity-compose version with latest fixes here in #62

[vsoch]

Thank you @scicco ! I will give this another try later today. One tweak to the files above - it looks like you renamed the network to just "redis" but the compose is looking for redis-static (I'll tweak this locally so no worries about needing to update)!

[scicco]

@vsoch you're right, I fixed the previous reply with the proper type name. singularity-compose.yml inside custom examples folder is already using type: redis. Thank you for pointing it out 😄

[vsoch]

We are getting closer! Hopefully this isn't an issue with my system:

ERROR:   could not delete networks: plugin type="bridge" failed (delete): running [/usr/sbin/iptables -t nat -D POSTROUTING -s 192.168.20.2 -j CNI-628185cdfa8bb97a360ff8a2 -m comment --comment name: "static-redis" id: "3990930" --wait]: exit status 2: iptables v1.8.4 (legacy): Couldn't load target `CNI-628185cdfa8bb97a360ff8a2':No such file or directory