You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/chi_edge/networking.md
+150-1Lines changed: 150 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,9 +6,125 @@ While Neutron is still running, it is only used for floating IPs, and end-users
6
6
7
7
Instead, the Calico CNI plugin for kubernetes is managing most networking, and all container -> container networking, and neutron and zun are configured to provide a minimal "shim" around this.
8
8
9
+
## Network Details
10
+
11
+
### Openstack's Perspective
12
+
As far as Openstack knows, there are only two networks, "Public" and "Caliconet", connected by a router.
13
+
14
+
This router is implemented by means of a linux network namespace, openvswitch ports (qg_iface and qr_iface) which map layer 2 traffic into that namespace, and a series of IPTables rules which configure routing between qg_iface and qr_iface.
15
+
By default, the `qg_iface` acts as the default route within the namespace, and other links (each a separare qr_iface) route to their local subnets.
16
+
For technical details, refer to [Neutron's Layer 3 Internals](https://docs.openstack.org/neutron/latest/contributor/internals/layer3.html)
When a floating IP is attached, neutron binds that address to `qg_iface` in the router namespace, and also sets up NAT to forward traffic to the mapped internal address.
39
+
40
+
We'll refer to the external address as `floating_ip`, and the internal one as `fixed_ip`.
1. Initially, a packet arriving from outside has some `source_address=foo`, and `destination_address=floating_ip`.
52
+
2. It arrives at qg_iface, and iptables applies "destination nat (DNAT)" to rewrite the destination IP address. Now, `source_address=foo` and `destination_address=fixed_ip`
53
+
3. As `destination_address=fixed_ip`, the routing table indicates it should be forwarded via `qr_iface`, and it's sent off into caliconet
54
+
4. If a host were listening on `fixed_ip`, and connected to `caliconet` at layer 2, it would receive this packet and be able to respond. Its reply would have `source_address=fixed_ip` and `destination_address=foo`.
55
+
5. On the way back out, the packet would be received at `qr_iface`, and IPtables apply source NAT (SNAT), rewriting `source_address=fixed_ip` to `source_address=floating_ip`
56
+
6. the packet then leaves via `qg_iface`, having `source_address=floating_ip`,`destination_address=foo`, and is routed back to the intiial sender.
57
+
58
+
59
+
### Calico's Perspective
60
+
61
+
Calico sees the world differently, and is primarily concerned about routing between kubernetes hosts. Using the below diagram, we'll look at how traffic flows between a few different pairs of endpoints.
62
+
63
+
```mermaid
64
+
flowchart LR
65
+
66
+
67
+
clusternet[Cluster Network 10.3.0.0/24]
68
+
clusternet --- localip1
69
+
clusternet --- localip2
70
+
clusternet --- gw[Gateway:
71
+
10.3.0.1/24
72
+
10.8.8.1/24
73
+
]
74
+
75
+
subgraph host2[Kubernetes Host 2]
76
+
localip2[Local IP 2\n 10.3.0.12/24]---podcidr2
77
+
subgraph podcidr2[PodCIDR 192.168.72.0/24]
78
+
pod2a[Pod2a 192.168.72.12]
79
+
pod2b[Pod2b 192.168.72.13]
80
+
end
81
+
note2[
82
+
0.0.0.0/0 via 10.3.0.1
83
+
192.168.71.0/24 via 10.3.0.11
84
+
192.168.72.0/24 via 10.3.0.12
85
+
]
86
+
end
87
+
subgraph host1[Kubernetes Host 1]
88
+
localip1[Local IP 1\n 10.3.0.11/24]---podcidr1
89
+
subgraph podcidr1[PodCIDR: 192.168.71.0/24]
90
+
pod1a[Pod1a 192.168.71.10]
91
+
pod1b[Pod1b 192.168.71.11]
92
+
end
93
+
note1[
94
+
0.0.0.0/0 via 10.3.0.1
95
+
192.168.71.0/24 via 10.3.0.11
96
+
192.168.72.0/24 via 10.3.0.12
97
+
]
98
+
end
99
+
subgraph "outside"
100
+
gw --- external[external 10.8.8.8/24]
101
+
note3[Route: 0.0.0.0/0 via 10.8.8.1]
102
+
end
103
+
```
104
+
105
+
We start with the assumption that Local IP 1 and Local IP 2 can communicate directly with each other at layer 2, as it's the simplest.
106
+
107
+
* Traffic between two pods on the same host, or between the host's local IP and its own pods, is sent directly with no NAT or encapsulation.
108
+
* Traffic from pods on one host to the local IP of another host (or other "endpoints" calico is aware of), is also sent directly, as the source and destination addresses still match the locally connected routes. (Depending on config, it could also be routed via local-ip)
109
+
* Traffic from pods on one host to pods on another host is routed, using the destination host's "local ip" as the next-hop
110
+
* in contrast, traffic from pods to the "outside" has source NAT applied before leaving the host, since the network outside of Calico wouldn't be aware of how to return traffic to the pod IPs.
All neutron "sees" is that some addresses in the "calico-subnet" send traffic, respond to messages, and so on.
39
188
Calico's BGP routing handles getting ipv4 traffic from neutron to the actual destination, which may traverse multiple kubnernetes nodes before reaching a container.
0 commit comments