Distributed SNAT - local gateway mode
In this post I am going to dive into proof of concept (POC) implementation details of local gateway SNAT mode. Instead of being transferred to network node SNAT traffic will utilize additional local gateway interfaces to skip one hop for 'north-south' traffic. POC implementation was based on neutron stable/mitaka branch that is pretty outdated these days. However implementation concept not changes a lot.
SNAT implementation at DVR
SNAT feature in Distributed virtual router (DVR) implementation remains centralized:
Figure 1 shows single VM deployed at Compute node.
- Data plane address of Compute node: 22.214.171.124
- Data plane address of Network node: 126.96.36.199
- External network 172.24.2.x
- VM's address on private network: 10.0.0.4
Traffic flows as following:
- Packet with destination address outside of cloud VM enters internal bridge (br-int)
- Target MAC address lays in a Router namespace with on interface with IP address: 10.0.0.1
- Router identifies packet as 'north-south' and route it towards IP address 10.0.0.3 at network node
- Packet passes from internal bridge to tunnel bridge
- Tunneling packet according to defined tunnel type (VXLAN) and send to network node
- At end of tunnel packet is passed to internal bridge at network node
- Packet arrives to SNAT namespace where target IP address resides
- Packet is routed outside
SNAT implementation - local gateway modeUnlike reference implementation SNAT in local gateway mode not requires network node.
We create additional network namespace that hosts SNAT external address. Outgoing 'north-south' packet is routed locally from router namespace to SNAT namespace and then outside.
This implementation consumes 2 additional address per (tenant, router) pair. If addresses are taken from predefined external pool and traffic passes later another NAT beyond cloud boundary this approach will work well.
Router and SNAT namespace are connected in a way that mimics DVR floating IP implementation
Addresses on veth pair endpoint interfaces are taken from different IP range
External addresses can be assigned to to SNAT namespace on demand. Once first tenant VM is deployed on given compute host neutron implementation can allocate address additional address pair and setup local routing rules.
POC implementation is however uses CLI based external local gateway allocation which is not desired for production case.
Figure 4 shows CLI command that extends router external information and defines local gateway on specified L3 agent.
Extended router information is listed below.
Cloud deployment has 2 compute nodes and network node. Router is deployed in DVR mode. Agent gateway information holds compute host alternative gateway address which is denoted in this post as 'local gateway'.
Unlike router new internal structure - having local gateway port on every compute node with different IP addresses - Horizon graphical outline was not part of this feature POC. Besides multiple gateway ports brake traditional gateway visual presentation.
However port view is up to date in Horizon:
Besides gateway port at network node, 2 other local gateway ports are present:
Other key implementation aspectsDefault gateway port can be overridden on agents with dvr-local-snat mode
- When gateway is overridden traffic is routed outside locally avoiding Network Node
- Alternative gateway port on this agent is added
- Alternative gateway port on this agent is removed
- Internal interface (scheduled VM) is added/removed
- Default gateway port is set/cleared
- All alternative gateway ports are removed to prevent ambiguity