Distributed SNAT - local gateway mode


In this post I am going to dive into proof of concept (POC) implementation details of local gateway SNAT mode. Instead of being transferred to network node SNAT traffic will utilize additional local gateway interfaces to skip one hop for 'north-south' traffic. POC implementation was based on neutron stable/mitaka branch that is pretty outdated these days. However implementation concept not changes a lot.

SNAT implementation at DVR


SNAT feature in Distributed virtual router (DVR) implementation remains centralized:
figure 1

 Figure 1 shows single VM deployed at Compute node. 

  • Data plane address of Compute node: 182.0.0.2
  • Data plane address of Network node: 182.0.0.3
  • External network 172.24.2.x
  • VM's address on private network: 10.0.0.4
Traffic flows as following:
  1. Packet with destination address outside of cloud VM enters internal bridge (br-int)
  2. Target MAC address lays in a Router namespace with on interface with IP address: 10.0.0.1
  3. Router identifies packet as 'north-south'  and route it towards IP address 10.0.0.3 at network node
  4. Packet passes from internal bridge to tunnel bridge
  5. Tunneling packet  according to defined tunnel type (VXLAN) and send to network node
  6. At end of tunnel packet is passed to internal bridge at network node
  7. Packet arrives to SNAT namespace where target IP address resides
  8. Packet is routed outside

SNAT implementation - local gateway mode

Unlike reference implementation SNAT in local gateway mode not requires network node.
figure 2

 We create additional network namespace that hosts SNAT external address. Outgoing 'north-south' packet is routed locally from router namespace to SNAT namespace and then outside.

 This implementation consumes 2 additional address per (tenant, router) pair. If addresses are taken from predefined external pool and traffic passes later another NAT beyond cloud boundary this approach will work well.

 Router and SNAT namespace are connected in a way that mimics DVR floating IP implementation
figure 3
Addresses on veth pair endpoint interfaces are taken from different IP range

 External addresses can be assigned to to SNAT namespace on demand. Once first tenant VM is deployed on given compute host neutron implementation can allocate address additional address pair and setup local routing rules.

 POC implementation is however uses CLI based external local gateway allocation which is not desired for production case.
figure 4

 Figure 4 shows CLI command that extends router external information and defines local gateway on specified L3 agent.

 Extended router information is listed below.
figure 5

 Cloud deployment has 2 compute nodes and network node. Router is deployed in DVR mode. Agent gateway information holds compute host alternative gateway address which is denoted in this post as 'local gateway'.

 Unlike router new internal structure - having local gateway port on every compute node with different IP addresses - Horizon graphical outline was not part of this feature POC. Besides multiple gateway ports brake traditional gateway visual presentation.

 However port view is up to date in Horizon:
figure 6

 Besides gateway port at network node, 2 other local gateway ports are present:

Other key implementation aspects

Default gateway port can be overridden on agents with dvr-local-snat mode 
  • When gateway is overridden traffic is routed outside locally avoiding Network Node 
Database update notification may cause ‘re-wiring’ in following cases:
  • Alternative gateway port on this agent is added
  • Alternative gateway port on this agent is removed
  • Internal interface (scheduled VM) is added/removed
  • Default gateway port is set/cleared
  •      All alternative gateway ports are removed to prevent ambiguity

Comments

Post a Comment

Popular posts from this blog

SSH as a service

Image
This post starts a series of posts devoted to newly created SSH cloud service Motivation for SSH service SSH was and remains a handy tool for all network administrators. At cloud deployment w e need a way to remotely SSH a VM in the cloud that is not connected to the internet from outside of the cloud. Such task requires VPN. Among disadvantages of VPN I can mention:   VPN does not support internal, non-routable networks  VPN is less secured since it allows access to all devices on all routable networks in the cloud  Requires a special client   Error-prone – client might set the VPN server as its default GW which will direct all its outgoing traffic to the cloud Tenant administrator wants to setup service that will handle network connectivity, hide address details from outside user and at the same time save cloud floating IP addresses from being attached/detached according to current tenant users immediate request. User  wa...
Read more

Distributed SNAT - examining alternatives

Image
SNAT feature remains one of the most common network features in cloud network deployment that still has no agreed distributed solution. While Distrubuted Virtual router solves local connectivity for floating IP and simplifies east-west VM communication SNAT is still deployed at Network node and remains a traffic bottleneck. figure 1 Above figure has 2 deployed tenants 'green' and 'orange'. SNAT is performed centrally at network node. Possible solutions The are number of proposed solution to decentralize SNAT. Each solution has its own benefits and drawbacks. 1. SNAT per (tenant, router) pair The most straightforward solution is to perform SNAT at compute node router instance.  However while DVR deployment is copying internal subnet router address across compute nodes router's IP address on external network can not be copied. Such a deployment will consume extra external address per (router, tenant) pair. figure 2 Maximum address consumption...
Read more