In this article we discuss how OpenStack Security Groups are implemented in MidoNet. This post is the fourth in a series intended to familiarize users with MidoNet’s overlay networking models.
- Part 1 covered MN’s Provider Router.
- Part 2 covered Tenant Routers and Networks.
- Part 3 covered how MN Agent simulates packet traversing of the overlay topology.
Neutron Security Groups
In OpenStack the term “Security Group” refers to two concepts:
- A white-list of rules each of which identifies some kind of allowed traffic. Traffic may be identified by any of the following: source IP prefix, destination IP prefix, IP protocol, L4 source port, L4 destination port, ICMP type, ICMP code, direction (traffic from the VM is checked against egress rules, traffic to the VM is checked against ingress rules).
- All the ports that incorporate a Security Group’s rules in their port-level firewalls. Therefore, traffic may also be identified by the Security Group of the source port or destination port.
Neutron network ports (those bound to VM instances) can be assigned one or more Security Groups. Any forward traffic (regardless of direction: from/to the VM) that isn’t explicitly allowed by some Security Group rule is dropped. Return traffic is allowed without matching any Security Group rule.
Each Tenant (the OpenStack term is Project but I prefer Tenant) has a default Security Group (initially empty) and may create any number of new Security Groups (Security Groups are not shared across Tenants). Any VM-facing network port that isn’t explicitly assigned specific Security Groups will be assigned the Tenant’s default group. Security Groups may be edited (creating or deleting rules) at any time in Horizon or via the Neutron CLI. A change to a Security Group is reflected almost immediately at all the ports assigned to that Security Group.
Note that VM-facing network ports also have anti-spoofing rules automatically applied to them. Remember that at port-creation time, Neutron selects the port’s MAC and IP addresses. Each port automatically gets anti-spoofing rules. These rules are different from Security Group rules in that 1) they can match on L2 fields, 2) they’re specific to the port’s addresses and therefore cannot be shared across ports (in contrast, think of Security Groups as rule sets that can be shared). A recent version of Neutron (Icehouse?) enabled toggling the anti-spoofing rules on a per-port basis in order to allow a VM instance to act as an appliance/router, receiving and forwarding traffic whose L3 source and destination are different than the VM’s.
To sum up, here’s how the VM’s port-level firewall treats a packet emitted from the VM:
- First check for spoofing. Drop the packet if its source MAC and IP don’t match the addresses reserved for the VM when the VM’s network port was created.
- Then, allow the packet if it can be identified as a reply/response to traffic that was allowed towards the VM.
- Then, allow the packet if it matches any egress rule in the Security Groups assigned to the VM’s network port.
- Finally, drop the packet if it wasn’t allowed in step 2 or 3.
Treatment for traffic to the VM is similar (spoofing will check destination MAC and IP; ingress Security Group rules are checked in step 3).
MN Agents do not understand Security Groups. Instead they understand the corresponding low-level models: Rules and Rule-Chains. Rule-Chains are a powerful mechanism that allows MN Agents to implement many different high-level network capabilities.
MN Rules are distinct from Security Group rules. MN Rules have richer matching/filtering and richer actions. MN Rules have condition statements that can express boolean logic over clauses that match most L2-4 header fields (practically the same set matched by OpenFlow). MN Rules also allow:
- wildcards, bit-masks, or ranges on most L2-4 header fields;
- clauses matching the packet’s ingress port on a specific virtual device
- clauses matching the packet’s egress port of a specific virtual device (the egress port is known, for example, if the rule is evaluated in a virtual router’s post-routing stage);
- Tenant/owner of the port where the packet entered the overlay/virtual network.
A Rule specifies an action to apply to traffic that matches the rule’s condition statement (think of it as a filter). Possible actions include: accept, drop, static source NAT, dynamic source network address and port translation (NAPT), static DNAT, dynamic DNAT. Some actions that are currently in development are: trace (marks the packet for tracing through the overlay and underlay network), and meter (counts flows, packets, or bytes).
Rule-Chains are ordered lists of Rules. Rule-Chains are referenced by virtual devices to implement some of their packet-processing logic. For example, MidoNet virtual routers have pre-routing and post-routing rule-chains; virtual bridges have pre-forwarding and post-forwarding rule-chains; router and bridge ports have inbound and outbound rule-chains.
Rule-Chains themselves are stateless and can be shared by multiple virtual devices. Think of them as a program specifying how packets should be treated. Therefore, some rule actions exist purely for flow-control: continue, jump and return.
The best way to understand Rule-Chains is to jump into an example: MidoNet’s implementation of Neutron Security Groups and anti-spoofing rules.
Security Groups in MidoNet
Consider the following scenario. For simplicity we’ll consider only ingress traffic to VMs (and therefore only ingress Security Group rules):
- Tenant A has two Security Groups:
- SG1 allows ICMP and HTTP traffic.
- SG2 allows SSH traffic.
- Tenant A has two VMs:
- VM1 (with MAC1 and IP1) is bound to network Port1, which belongs to both Security Groups.
- VM2 (with MAC2 and IP2) is bound to network Port2, which belongs only to SG1 (SSH to this VM is not allowed).
The following diagram shows how the ingress Security Group and anti-spoofing rules are translated and organized into MidoNet Rule-Chains.
When a Security Group is created, e.g. SG1, MidoNet’s low-level topology gets two Rule-Chains: SG1-Ingress will hold translations of the Group’s ingress rules; SG2-Egress will hold translations of the Group’s egress rules. When a ingress Security Group Rule is added, a corresponding low-level MN Rule is added to SG1-Ingress. Similarly, a new egress Security Group Rule results in a new MN Rule in SG1-Egress. A Security Group rule is translated to a MN Rule with a condition matching the same traffic and Action set to “ACCEPT”.
When Neutron Port1 is created, MidoNet’s low-level model gets a new port on the MN virtual bridge corresponding to Port1’s network. The MN port has two Rule-Chains: Port1-Ingress filters traffic to the VM; Port1-Egress filters traffic from the VM. These Chains are immediately populated with anti-spoofing rules (because the port’s MAC and IP are already known).
When a Security Group is assigned to a Neutron port, for example SG1 is assigned to Port1, the Port1-Ingress gets an unconditional rule whose JUMP action causes the flow of control (during Simulation) to pass to SG1-Ingress. Similarly, Port1-Egress gets an unconditional rule whose action is Jump to SG1-Egress.
Putting this all together, since Port1 is assigned to both SG1 and SG2, the Rule-Chain named Port1-Ingress ends up with these rules:
- DROP traffic whose destination MAC is not MAC1
- DROP traffic whose destination IP is not IP1
- ACCEPT reply traffic
- JUMP to SG1-Ingress
- JUMP to SG2-Ingress
- DROP (unconditional, drops everything that gets here)
Notice that Port1-Ingress and Port1-Egress are Rule-Chains that are only used by Port1. In contrast, SG1-Ingress and SG1-Egress are both potentially shared Rule-Chains. They will be jumped-to by rules in every port that belongs to Security Group SG1.
Part 3 described the MN Agent’s packet-processing logic, and particularly the Simulation stage. So here we’ll just jump right into how Rule-Chains behave during Simulation. Assume an SSH packet (to IP1 and TCP port 22 and of course having missed in the kernel and user-space flow tables) is in the Simulation stage. Assume the Simulation has determined that the packet would arrive at the virtual bridge (corresponding to a Neutron network) and would be emitted from Port1. The Simulation would proceed to check how Port1’s outbound (outbound, from the perspective of the bridge) filter would behave. This means evaluating the packet against the Rule-Chain named Port1-Ingress (the chain name reflect Neutron’s VM-based perspective). Here’s what the Simulation would do:
- Check Rule #1’s condition against the packet. Assume the packet is for MAC1, so the condition does not match the packet. Move on to the next rule.
- Check Rule #2’s condition against the packet. The packet is for IP1, so the condition does not match. Move on to the next rule.
- Check Rule #3’s condition. This requires doing a lookup in the local flow-state cache using a key constructed by flipping the packet’s source and destination IP addresses and L4 ports. Assume this is a new SSH connection, no flow-state is found. Move on to the next rule.
- Rule #4 is unconditional. JUMP to Chain SG1. The Simulation loads the representation of SG1.
- Evaluate SG1’s Rule #1’s condition. That matches ICMP, this packet’s SSH, so move on to the next rule.
- Evaluate SG2’s Rule #2’s condition. That matches HTTP, this packet’s SSH, so move on to the next rule.
- There’s no next rule, so return from SG1 to the calling JUMP rule. Move on to the next rule.
- Rule #5 is unconditional. JUMP to Chain SG1. The Simulation loads the representation of SG1.
- Evaluate SG1’s Rule #1’s condition. That matches SSH and therefore applies to this packet. The Action is ACCEPT. Break out of Chain evaluation with result ACCEPT.
- The Simulation moves on to next stage in the device’s packet processing. For a pre-routing chain, the next stage would be routing. For a bridge port’s Inbound filter, the next stage would be entering the bridge itself. In this case, an Outbound filter, the next stage is to actually emit the packet from the port (it wasn’t dropped).
It’s not part of the Rule-Chain evaluation, but remember from Part 3 that since this is an exterior port, the Simulation terminates with the result that the original packet should have its headers modified to what they looked like during Port1’s simulation (e.g. if they traversed a router, the L2 addresses would be modified) and then be emitted from Port1. The virtual/physical translation stage maps Port1 to the compute host where VM1 resides, e.g. Host5. If VM1 is remote, a datapath flow rule is installed to match all similar packets and emit them (forward them in OVS kmod’s terminology) from a tunnel port with outer IP address set to Host5’s IP. The tunnel key will be set to encode Port1’s ID so that the MN Agent at Host5 can emit the packet/flow from the corresponding datapath port without doing a Simulation. The ingress host of a flow does the simulation and the egress host (the recipient of tunnel traffic) trusts the result.