Introduction to MN (Part 6 – VLAN L2 Gateway)

In this article we discuss how to connect MidoNet virtual workloads to physical workloads. This post is the sixth in a series intended to familiarize users with MidoNet’s overlay networking models:

  • Part 1 covered MN’s Provider Router.
  • Part 2 covered Tenant Routers and Networks.
  • Part 3 covered how MN Agent simulates packet traversal of the virtual topology in order to compute a flow rule.
  • Part 4 covered Security Groups and MidoNet’s low-level Rule-Chain model.
  • Part 5 explained how MidoNet distributes flow state to support of stateful network services.

A Layer 2 Gateway allows connecting an overlay L2 switches, i.e. a MN bridge that renders a Neutron network, to a physical L2 segment. This is typically used to allow L2 connectivity between VM instances in an OpenStack environment and physical servers (or VM instances) on physical L2 segments external to OpenStack.

For example, imagine you have an already installed/running cloud and have Neutron network “Network 1” (rendered by Virtual Bridge 1 in MidoNet’s low-level model). There are three VM instances on Network 1 and you’d like these VM instances to have L2 access to three physical servers on a physical L2 segment somewhere in your network. The servers might be a database cluster. The following diagram illustrates how you’d like the overlay network to look: you’d like the physical segment to be accessible via a port of your Network (equivalently, via a port of the virtual bridge). In the diagram, any traffic from the VM instances to the physical servers, and from the physical servers to the VM instances, will transit port P4 of Virtual Bridge 1.

Blog L2 Gateways 1

How you configure this in MidoNet and in your physical network depends on what your physical network design and the physical switch capabilities. We’ll start by trying to use concepts we’ve already learned in previous posts and imagining that the servers’ physical L2 segment is VLAN 10 on an 802.1q compatible switch. The diagram below illustrates one possible physical setup as well as the mapping to the overlay topology.

  • Let’s assume that the physical switch has a free port and is within distance to run an Ethernet cable from that port to a commodity x86 server where MidoNet is installed. The MidoNet host need not be a compute host, but in this example let’s assume it’s Compute Host 1.
  • Let’s also imagine that Host 1 has a free NIC, eth1. Host 1 (and Host 2 as well) has a NIC, eth0, that it uses for to send/receive tunneled overlay traffic to/from other compute hosts. If Host 1 were single-homed we could set up some software bridging to accomplish the setup. For simplicity, we’ll assume Host 1 has an extra NIC.
  • You run a cable from the physical switch’s free port to Host1’s eth1 and you configure the switch’s port to carry untagged VLAN 10 traffic.
  • You use MidoNet’s API (Neutron does not yet have a L2 Gateway API) to create a free port, P4, on Virtual Bridge 1, and to bind P4 to Host1/eth1. MidoNet Agent on Host 1 learns about that port-interface binding from MidoNet’s state cluster (powered by Apache ZooKeeper) and plugs eth1 directly into the local datapath.
  • For the sake of this example, let’s also imagine that VM1 is running on Host 1, while VM2 and VM3 are running on Host 2. As usual for MidoNet hosts, Host 1 and Host 2 both have access to an IP fabric and can tunnel (GRE/VXLAN) overlay traffic to each other.

What happens when Server1 sends an L2 broadcast packet, for example an ARP request to VM2’s MAC?

  1. The switch will flood the broadcast packet to all ports in VLAN 10. The packet will therefore be emitted (untagged) from the port towards Host 1’s eth1.
  2. When the packet arrives at Host 1’s eth1, MidoNet interprets it as arriving at Bridge 1’s P4. Bridge 1 learns that Server 1’s MAC is on P4 (i.e. Bridge 1’s MAC-table gets an entry mapping Server 1’s MAC to P4).
  3. Normally, the Bridge’s ARP table has entries for the IPs and MACs of VM-facing ports (this was explained in Part 2 and Part 3). However, for the sake of this example, let’s assume that you’ve disabled the virtual bridge’s ARP table. The Simulation on Host 1 therefore determines that Bridge 1 would flood the packet from all its port (except P4, the ingress port). The MN Agent on Host 1 therefore has the datapath emit one copy of the packet locally (from the datapath port corresponding to P1 and therefore also VM1’s software interface/tap) , and tunnel one copy to Host 2 to be emitted from the datapath ports corresponding to Bridge1’s ports P2 and P3.
  4. VM2 will answer with an ARP reply directed at Server 1’s MAC. The MN Agent at Host 2 interprets the packet as arriving at port P2 of Bridge 1. The MAC-table entry for Server 1’s MAC should have propagated from the MN Agent on Host 1 to MN’s State Database and from there to MN Agents on other hosts, and particularly Host 2. The Simulation on Host 2 will therefore determine that ARP reply (a unicast) should be emitted from Bridge 1’s port P4. As a result the packet is tunneled from Host 2 to Host 1, with a tunnel key that encodes P4’s UUID.
  5. Upon receiving the tunneled packet, Host 5 will decapsulate it and decode the tunnel key to P4, map P4 to the datapath port for eth1, and consequently emit the ARP reply from eth1.
  6. The physical switch will receive the ARP reply, learn VM2’s MAC, and the forward the ARP reply to Server 1.


Blog L2 Gateways 2

So this setup accomplishes L2 connectivity between the physical servers and the VM instances. But there are several problems with this approach:

  • eth1 and Compute Host 1 are SPOFs (Single points of Failure) for connecting Network1 to the physical servers
  • one physical switch port will be used for each Network that needs to be connected to physical servers
  • you have to make manual changes to the physical network each time you have another request to bridge virtual and physical workloads.

MidoNet’s VLAN L2 Gateway is meant to address these problems while continuing to work with the 802.1q physical switch fabric. In a future post we’ll describe the VXLAN L2 Gateway which solves these problems and more but requires the network operator to have a VTEP (VXLAN Tunnel Endpoint) connected to the physical workloads.


Blog L2 Gateways 3

In the diagram above, you can see that a new overlay/virtual device has been introduced: a VLAN-aware (802.1Q compatible) Bridge2. Bridge1’s P4 is now an interior port (meaning it connects to another overlay device and emits traffic that remains in the overlay topology) and links to Bridge2 at an untagged VLAN 10 port. Bridge2 has a trunk port which is bound to Host1/eth1 (where P4 was bound in the earlier/naive setup. Also, and importantly, the physical switch port facing Host1/eth1 must be put in TRUNK mode (earlier it was untagged VLAN 10).

What happens when Server1 sends an L2 broadcast packet, for example an ARP request to VM2’s MAC?

  1. The physical switch floods the packet, it’s tagged with VLAN10 and emitted to Host1/eth1.
  2. MN interprets the packet as arriving on Bridge2’s trunk port; Bridge2 learns Server1’s MAC. The packet must be untagged and flooded from all VLAN 10 untagged ports (MN will only ever allow one, in order to prevent bridging loops). The packet is therefore emitted by Bridge2 on the port facing Bridge1.
  3. Bridge1 receives the packet on P4 and learns Server1’s MAC, floods it to all its port (except the ingress). All VMs on Bridge1 receive the packet. Only VM2 constructs an ARP reply.
  4. VM2’s ARP reply packet ingresses Bridge1 on port P2. The unicast destination MAC matches the just-learned MAC-table entry, so Bridge1 emits the packet from P4.
  5. The packet ingresses Bridge2 at the VLAN 10 untagged port. The unicast destination MAC matches the MAC-table entry from step #2, so Bridge2 tags the packet with VLAN 10 and emits it from the trunk port.
  6. Since the trunk port is bound to Host1/eth1, the tagged packet is emitted from that interface and arrives at the physical switch’s trunk port. The physical switch learns VM2’s MAC on VLAN10, untags the packet and forwards it to Server1.

The introduction of Bridge2 now allows the following:

  • Connect other vlan-agnostic virtual bridges (Neutron networks) to Bridge2. Each one will require a new untagged port on Bridge2 with a unique VLAN id. These are overlay objects and are therefore instantly provisioned.
  • Add a second trunk port to Bridge 2, bound to Host3/eth1. Host3/eth1 is connected via physical cable to another trunk port on the physical switch. This removes the SPOFs. The setup is manual but is only performed once. Thereafter, new vlan gateways require only new ports on Bridge2.


Blog L2 Gateways 4


Finally, what about STP (Spanning Tree Protocol) and its variants? MidoNet’s VLAN-aware bridge does not implement STP. However, it recognizes STP packets and:

  • drops any arriving on a non-trunk port (i.e. on the untagged VLAN ports)
  • if an STP packet arrives on Trunk1, it’s forwarded from Trunk2, and vice-versa.

This STP pass-through allows connecting the VLAN L2 Gateway Nodes/interfaces (Host1/eth1 and Host3/eth1 in our example) to different physical switches that are part of the same L2 fabric. That eliminates the physical switch as a SPOF. STP in the physical fabric, combined with mac-learning on the vlan-aware overlay bridge’s trunk ports allows outgoing overlay traffic to choose the correct trunk port.


2 thoughts on “Introduction to MN (Part 6 – VLAN L2 Gateway)

Leave a Reply

Your email address will not be published. Required fields are marked *