Distributed SNAT - examining alternatives

SNAT feature remains one of the most common network features in cloud network deployment that still has no agreed distributed solution. While Distrubuted Virtual router solves local connectivity for floating IP and simplifies east-west VM communication SNAT is still deployed at Network node and remains a traffic bottleneck.
figure 1

 Above figure has 2 deployed tenants 'green' and 'orange'. SNAT is performed centrally at network node.

Possible solutions

The are number of proposed solution to decentralize SNAT. Each solution has its own benefits and drawbacks.

1. SNAT per (tenant, router) pair

The most straightforward solution is to perform SNAT at compute node router instance. However while DVR deployment is copying internal subnet router address across compute nodes router's IP address on external network can not be copied. Such a deployment will consume extra external address per (router, tenant) pair.
figure 2

 Maximum address consumption equals to:
   [# of compute nodes] * [# of tenants]

 Cloud deployment may consume external address in a lazy way to optimize external address consumption. External address should be allocated and assigned only when first VM of requested tenant is 'born' on chosen compute node and connected routable internal network. At above figure only when VM2 of 'orange' tenant is born external address 172.24.4.2 got allocated.

 For cloud deployment with high limit of external addresses or deployment having another NAT beyond cloud edge this NAT model may give additional value. Neutron database should track all additional gateway router ports. External addresses should be provisioned implicitly from separate external address pool. 

 Proof of concept implementation for this SNAT model base on neutron stable/mitaka branch is here and is being discussed in this post. This implementation has number easing assumptions that require modification in future enhancement round.
Among them:

  • Explicit external address allocation per (router, compute node) pair that required client API modification.

2. SNAT per compute node

This SNAT model limits external address consumption to single address per compute node greatly optimizing network resources use. However this model has at least one noticeable drawback. When external service serves multiple tenants via single external IP address - service abuse performed by VM of one tenant may cause denial of service to VM of another tenant that lays on same compute host. 

figure 3

Above figure shows that SNAT rule implemented by both tenants translating multiple tenant VMs addresses into single external address. Reverse translation should restore tenant information IP/MAC address and other tenant related information. On order to visualize this complex translation figure 3 shows two routing entities. In reality this translation could be performed by different routing tables in same routing entity.

DragonFlow implementation

This model was proposed and implemented in pretty elegant manner with DragonFlow service plugin of neutron. 

 DragonFlow deploys Ryu SDN controller on every compute node. When Nova scheduler asks Neutron server to allocate network port for VM the following happens:



  1. Neutron server passes port allocation request to DragonFlow ML2 plugin. ML2 plugin pushes new neutron port information into separate DragonFlow database in parallel to regular neutron DB update
  2. ML2 plugin informs relevant compute node where VM should be 'born' regarding port  update.
  3. L3 agent on target compute node fetch port information from database and build a Ryu style update event with port information in it.
  4. Ryu SDN controller push update event to all registered SDN application
  5. Every application that registered for this specific event [local neutron port created] may insert/update OVS flows 

New SDN application that takes care of install/uninstall SNAT related OVS flows was the only new component to enable distributed SNAT model.


Useful resources:


Comments

Post a Comment

Popular posts from this blog

SSH as a service

Image
This post starts a series of posts devoted to newly created SSH cloud service Motivation for SSH service SSH was and remains a handy tool for all network administrators. At cloud deployment w e need a way to remotely SSH a VM in the cloud that is not connected to the internet from outside of the cloud. Such task requires VPN. Among disadvantages of VPN I can mention:   VPN does not support internal, non-routable networks  VPN is less secured since it allows access to all devices on all routable networks in the cloud  Requires a special client   Error-prone – client might set the VPN server as its default GW which will direct all its outgoing traffic to the cloud Tenant administrator wants to setup service that will handle network connectivity, hide address details from outside user and at the same time save cloud floating IP addresses from being attached/detached according to current tenant users immediate request. User  wa...
Read more

Distributed SNAT - local gateway mode

Image
In this post I am going to dive into proof of concept (POC) implementation details of local gateway SNAT mode. Instead of being transferred to network node SNAT traffic will utilize additional local gateway interfaces to skip one hop for 'north-south' traffic.  POC implementation was based on neutron stable/mitaka branch that is pretty outdated these days. However implementation concept not changes a lot. SNAT implementation at DVR SNAT feature in Distributed virtual router (DVR) implementation remains centralized: figure 1 Figure 1 shows single VM deployed at Compute node.  Data plane address of Compute node: 182.0.0.2 Data plane address of Network node: 182.0.0.3 External network 172.24.2.x VM's address on private network: 10.0.0.4 Traffic flows as following: Packet with destination address outside of cloud VM enters internal bridge (br-int) Target MAC address lays in a Router namespace with on interface with IP address: 10.0.0.1 Router iden...
Read more