Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Communication between containers on separate networks #5805

Closed
aparhatus opened this issue Apr 14, 2020 · 43 comments · Fixed by #14958
Closed

Communication between containers on separate networks #5805

aparhatus opened this issue Apr 14, 2020 · 43 comments · Fixed by #14958
Assignees
Labels
CNI Bug with CNI networking for root containers kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. network Networking related issue or feature planning Accepted and planned for future release

Comments

@aparhatus
Copy link

aparhatus commented Apr 14, 2020

Is communication between rootful containers on separate networks allowed by design? In Docker, containers on separate networks cannot access each other out of the box, whereas here they appear to:

[root@sandbox ~]# podman run --network network1 --name container1 -d docker.io/library/nginx
cb9d8e5ac20f25bf52866d1e468d139aff2b33de53b9c143d20fe1fed200fe97
[root@sandbox ~]# podman inspect -f "{{ .NetworkSettings.Networks.network1.IPAddress }}" container1
10.89.0.4
[root@sandbox ~]# podman run --network network2 --rm -it curlimages/curl 10.89.0.4
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...

The networks were created with podman network create <name>.

Is there any way to get the same behaviour as with Docker, where containers cannot reach each other directly in this fashion and instead have to go through published ports on the host?

Output of podman version:

Version:            1.8.2
RemoteAPI Version:  1
Go Version:         go1.12.12
OS/Arch:            linux/amd64
@mheon
Copy link
Member

mheon commented Apr 14, 2020

No way at present, though I imagine you could add firewall rules manually to restrict networks from reaching each other.

I imagine that CNI does have some way of doing this, given its origins in Kube, so that is something else worth researching.

@github-actions
Copy link

A friendly reminder that this issue had no activity for 30 days.

@rhatdan
Copy link
Member

rhatdan commented May 15, 2020

@baude @mheon @mccv1r0 Any further info on this?

@mheon
Copy link
Member

mheon commented May 15, 2020

Nope. We should investigate further, though, this sounds like something we should support.

@mccv1r0
Copy link
Collaborator

mccv1r0 commented May 15, 2020

Is communication between rootful containers on separate networks allowed by design?

This is (should be) a function of what you set the iptables FORWARD chain to allow.

@mheon mheon added CNI Bug with CNI networking for root containers kind/bug Categorizes issue or PR as related to a bug. labels Jun 2, 2020
@rhatdan
Copy link
Member

rhatdan commented Sep 10, 2020

@mccv1r0 @mheon Anyone ever work on this?

@mccv1r0
Copy link
Collaborator

mccv1r0 commented Sep 10, 2020

No. CNI has no way of knowing what you want the network you create to do/not do with respect to other "networks" (e.g. bridges) on the host system.

@mheon
Copy link
Member

mheon commented Sep 10, 2020

@rhatdan I think we'll need another CNI plugin, similar to the dnsname plugin, to do this.

@github-actions
Copy link

A friendly reminder that this issue had no activity for 30 days.

@rhatdan
Copy link
Member

rhatdan commented Dec 24, 2020

@baude PTAL Seems like something for you.

@baude
Copy link
Member

baude commented Dec 24, 2020

@rhatdan i dont think this is something podman should tread into. i think users should have to add fw rules, unless there is something in cni we can flip a bit on.

@rhatdan
Copy link
Member

rhatdan commented Dec 26, 2020

It would be nice if they were isolated by default. But yes having CNI implement this would be best.

@AkihiroSuda
Copy link
Collaborator

This sounds like a security issue. Can we document this in the man page?

@AkihiroSuda
Copy link
Collaborator

The following iptables commands seem to successfully isolate cni-podman1 and cni-podman2:

iptables -A INPUT -i cni-podman1 -j MARK --set-mark 1
iptables -A INPUT -i cni-podman2 -j MARK --set-mark 2
iptables -A FORWARD -o cni-podman1 -m mark --mark 0 -j ACCEPT
iptables -A FORWARD -o cni-podman1 -m mark ! --mark 1 -j REJECT
iptables -A FORWARD -o cni-podman2 -m mark --mark 0 -j ACCEPT
iptables -A FORWARD -o cni-podman2 -m mark ! --mark 2 -j REJECT

The number of commands is O(N) where N is the number of the networks (not the number of the containers).

I'm planning to implement this as a CNI plugin, but I'd like to get some design review before doing that. RFC.

@mheon
Copy link
Member

mheon commented Jan 14, 2021

Can we make a separate chain in FORWARD for each network? We could have the separate chains default to REJECT which would only require a single ACCEPT rule for each new chain. Keeps the overall FORWARD chain smaller.

@mheon
Copy link
Member

mheon commented Jan 14, 2021

Sorry, a separate chain which FORWARD would transfer to for each network

@mccv1r0
Copy link
Collaborator

mccv1r0 commented Jan 14, 2021

It looks like, to enforce isolation between docker networks, docker added two chans:

Chain DOCKER-ISOLATION-STAGE-1
Chain DOCKER-ISOLATION-STAGE-2

Stage 1 has a rule for each bridge sending to somewhere off bridge:

 pkts   bytes   target    prot opt  in            out          source               destination     
 1498 77256 DOCKER-ISOLATION-STAGE-2  all    --    docker0 !docker0  0.0.0.0/0            0.0.0.0/0   

Stage 2 also has rules for all their bridges where the output is another docker created network:

 pkts   bytes target     prot opt in     out         source               destination     
    0     0       DROP     all    --  *      docker0  0.0.0.0/0            0.0.0.0/0           

Thus anything from a docker network to another docker network is isolated.

As each network is created/removed, these chains are updated.

If there is a way for docker (or podman) to explicitly allow network x to talk to network y via run command option, I suspect that they explicitly add ACCEPT rules, probably to stage-2.

@AkihiroSuda
Copy link
Collaborator


iptables -N CNI-ISOLATION-STAGE-1
iptables -N CNI-ISOLATION-STAGE-2
# NOTE: "-j CNI-ISOLATION-STAGE-1" needs to be before "CNI-FORWARD" created by CNI firewall plugin. So we use -I here.
iptables -I FORWARD -j CNI-ISOLATION-STAGE-1
iptables -A CNI-ISOLATION-STAGE-1 -i cni-podman1 ! -o cni-podman1 -j CNI-ISOLATION-STAGE-2
iptables -A CNI-ISOLATION-STAGE-1 -i cni-podman2 ! -o cni-podman2 -j CNI-ISOLATION-STAGE-2
iptables -A CNI-ISOLATION-STAGE-1 -j RETURN
iptables -A CNI-ISOLATION-STAGE-2 -o cni-podman1 -j DROP
iptables -A CNI-ISOLATION-STAGE-2 -o cni-podman2 -j DROP
iptables -A CNI-ISOLATION-STAGE-2 -j RETURN

This seems to work as well.

@AkihiroSuda
Copy link
Collaborator

Implemented as a CNI plugin (/opt/cni/bin/isolation): https://github.com/AkihiroSuda/cni-isolation

I'm planning to propose this to CNI upstream (https://github.com/containernetworking/plugins).

@AkihiroSuda
Copy link
Collaborator

AkihiroSuda commented Jan 18, 2021

Submitted a proposal to CNI upstream: containernetworking/plugins#573

EDIT PR: containernetworking/plugins#584

@github-actions
Copy link

github-actions bot commented Mar 9, 2021

A friendly reminder that this issue had no activity for 30 days.

@rhatdan
Copy link
Member

rhatdan commented Jan 11, 2022

@AkihiroSuda Any progress?

@cdoern
Copy link
Contributor

cdoern commented Mar 31, 2022

@flouthoc I am looking to dive into some rust/netavark work. @rhatdan @Luap99 you think this would be a good thing to try as a first issue I got here from containers/netavark#154

@flouthoc
Copy link
Collaborator

@cdoern Sure, please feel free to take this one. I think this falls more into category of iptables/firewall stuff.

@flouthoc flouthoc assigned cdoern and unassigned flouthoc Mar 31, 2022
@rhatdan
Copy link
Member

rhatdan commented Mar 31, 2022

Sure, if you think you can handle it, and @Luap99 and @flouthoc and @mheon don't have time to work on it.

@rhatdan
Copy link
Member

rhatdan commented May 17, 2022

@cdoern Are you working on this?

@cdoern
Copy link
Contributor

cdoern commented May 17, 2022

yes @rhatdan containers/netavark#275 is the first step I am about to open a c/common PR for the option handling as well

@rhatdan
Copy link
Member

rhatdan commented May 17, 2022

Great

@cdoern
Copy link
Contributor

cdoern commented Jul 11, 2022

@Luap99 do we need to modify podman to expose this option?

@Luap99
Copy link
Member

Luap99 commented Jul 11, 2022

No but we have to add the option to the network create man page

@umohnani8
Copy link
Member

umohnani8 commented Jul 13, 2022

This has been fixed upstream.

@cdoern can you please verify this? Please close if so.

@cdoern
Copy link
Contributor

cdoern commented Jul 13, 2022

yes, a new network option --isolate was added in netavark.

@cdoern cdoern closed this as completed Jul 13, 2022
@Luap99
Copy link
Member

Luap99 commented Jul 13, 2022

We still have to document that in the man page.

@Luap99 Luap99 reopened this Jul 13, 2022
@cdoern
Copy link
Contributor

cdoern commented Jul 13, 2022

We still have to document that in the man page.

I will do that today, nice catch

cdoern added a commit to cdoern/podman that referenced this issue Jul 18, 2022

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
[CI:DOCS]

document the podman network create -o=isolate which allows networks to cut themselves off
from external connections.

resolves containers#5805

Signed-off-by: Charlie Doern <cdoern@redhat.com>
mheon pushed a commit to mheon/libpod that referenced this issue Jul 26, 2022
[CI:DOCS]

document the podman network create -o=isolate which allows networks to cut themselves off
from external connections.

resolves containers#5805

Signed-off-by: Charlie Doern <cdoern@redhat.com>
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 20, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 20, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
CNI Bug with CNI networking for root containers kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. network Networking related issue or feature planning Accepted and planned for future release
Projects
None yet
Development

Successfully merging a pull request may close this issue.