Docker Networking Model(3) — Container Network Access

6 min readApr 15, 2023

This article is the third in a series of articles. For those who are unfamiliar, please refer to the first two articles in the series:

Preface

In the previous article, we built a Bridge network model step by step using commands and finally succeeded in allowing two containers to access each other through the ping command. However, neither of these two containers could access the internet, and there was still a mystery of the iptables command left in the end. Therefore, in this article, we will clarify this information as a conclusion.

The following articles will explore from three perspectives, along with three main axes, which are:

How containers access each other
How containers actively access external services
How external networks actively access containers

The third point of functionality is the meaning of the docker -p feature, so the example will also explain in detail how to use it.

Each perspective has three main axes to explore, which are:

Who to send the packet to
Who to process the packet
Who filters the packet

In technical terms, the above three concepts are roughly the following items, but this article will not go into too much detail on each component:

routing table + forwarding table
kernel + iptables
conntrack q

Environment

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.3 LTS
Release:        18.04
Codename:       bionic

$ uname -a
Linux k8s-dev 4.15.0-72-generic #81-Ubuntu SMP Tue Nov 26 12:20:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

$ docker --version
Docker version 19.03.13, build 4484c46d9d

How containers access each other

This experiment is based on the previous environment to observe what actually happened with a simple ICMP packet.

First, in our environment, packets on the same network segment were specifically set, so access in this environment will be relatively simple.

The entire environment architecture is as follows:

The two containers C1 & C2 in the architecture are connected to each other through the Linux Bridge (hwchiu0) and related veth virtual network cards that we created.

Who to send the packet to

Previously, after setting IP for eth0 through ifconfig, the Kernel sets a Routing rule for the network interface, telling the system which packets to send to which network interface.

Let’s look at the status of the two containers C1 & C2.

○ → docker exec c1 route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.55.66.0      0.0.0.0         255.255.255.0   U     0      0        0 eth0

○ → docker exec c2 route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.55.66.0      0.0.0.0         255.255.255.0   U     0      0        0 eth0

These two messages are simple, telling the system that when it sees a packet which sent to 10.55.66.0/24 in the future, it should send it out through the eth0 network interface.

However, since the eth0 network interface is composed of veth, veth is like a pipe. When the packet arrives at eth0, it will immediately run out from the other end of veth0/veth1.

The concept is as follows:

According to the above concept, packets from the current two containers to 10.55.66.0/24 will eventually appear on veth0/veth1 on the host.

Who handles the packet?

When the packet enters the eth0 network interface and then appears from veth0/veth1, because veth0/veth1 itself is attached to the Linux Bridge (hwchiu0), the packet will be handled by the Linux Bridge!

The handling is divided into two parts (the actual packet handling sequence is not discussed here).

Linux Bridge has a mechanism to decide how to transfer packets
ebtables (this concept is ignored in this article)

The Linux Bridge forwarding mechanism is based on MAC Address to determine how to transfer packets, and this conversion table can be referred to as the Forwarding Table, forwarding packets based on MAC Address.

The following is an example. When the packet enters the Linux Bridge, it calculate which port to send the packet out based on the target MAC Address of the packet.

Therefore, in our example, the packets between C1/C2 containers are handled and forwarded through this mechanism.

Who filters the packets?

After the above rules are in place, C1/C2 containers should be able to transmit packets to each other, but in practice, there is a problem. Our PING will not go through because iptables secretly intervenes in the processing and drops packets that do not meet the rules.

There are two issues here:

Why do iptables sneak in to process
Why packets that do not meet the rules be dropped instead of packets that meet the rules

By default, iptables does not interfere with any Linux Bridge forwarding packets. There is a switch here that needs to be turned on, which is also turned on by Docker after installation.

→ cat /proc/sys/net/bridge/bridge-nf-call-iptables
1

Here we can do an experiment:

Confirm that the current network is not working
Turn off the switch mentioned above so that iptables will not interfere
Resend the ping
At this point, we will find that containers can transmit packets through ICMP.

$ docker exec -it c1 ping 10.55.66.3 -c1 -W1
PING 10.55.66.3 (10.55.66.3) 56(84) bytes of data.

--- 10.55.66.3 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms


$ echo "0" | sudo tee /proc/sys/net/bridge/bridge-nf-call-iptables
0

$ docker exec -it c1 ping 10.55.66.3 -c1 -W1
PING 10.55.66.3 (10.55.66.3) 56(84) bytes of data.
64 bytes from 10.55.66.3: icmp_seq=1 ttl=64 time=0.025 ms

--- 10.55.66.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.025/0.025/0.025/0.000 ms

$ echo "1" | sudo tee /proc/sys/net/bridge/bridge-nf-call-iptables
1

Once iptables is instructed to interfere with the management of packets in the Linux Bridge, all packets passing through will be checked by iptables.

In a Docker environment, a whitelist mechanism is adopted. If there is no instruction to pass, the packets will be dropped. This is why our packets will not pass. There are two solutions here:

Change to a blacklist concept, where packets are defaulted to pass.
Add relevant rules to allow our packets to pass.

The experiment in the previous article was based on (1), and here, by using “iptables -P FORWARD ACCEPT,” it is explained that the default for FORWARD packet processing is ACCEPT to accept them.

$ docker exec -it c1 ping 10.55.66.3 -c1 -W1
PING 10.55.66.3 (10.55.66.3) 56(84) bytes of data.

--- 10.55.66.3 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms


$ sudo iptables -P FORWARD ACCEPT

$ docker exec -it c1 ping 10.55.66.3 -c1 -W1
PING 10.55.66.3 (10.55.66.3) 56(84) bytes of data.
64 bytes from 10.55.66.3: icmp_seq=1 ttl=64 time=0.039 ms

--- 10.55.66.3 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.039/0.039/0.039/0.000 ms

# recover
$ sudo iptables -P FORWARD DROP

Conclusion:

The transfer of packets between different containers on the same node is primarily handled by Linux Bridge, which uses the Forwarding Table to determine how to transfer the packets and uses iptables to determine whether the packets can pass or not.

By default, Docker will allow iptables to intervene in the management of packets by Linux Bridge, and it adopts a whitelist mechanism, where any packets that do not meet the rules will be dropped. Docker’s approach is to add relevant rules to facilitate the connection of packets.

The next article will discuss How containers actively access external services. Stay tuned!