FreeBSD Jails are a really useful way to isolate
and run processes in a container under FreeBSD. You can either create thick jails similar which allow
different versions of a whole isolated FreeBSD OS, or you can create thin or service jails that share
resources and are very lightweight.
Regardless, you need to attach a network to your jail so you can expose services. There are a number of ways to achieve this, but I chose to use VNET Jails to keep my jail isolated from my host machine.
However as is the curse of being Firstyear, I encountered a bug. I noticed very poor throughput to the jail in the order of 100kb/s when the host was able provide 10gbit to a client. After a lot of investigation, it turns out that LRO (Large Receive Offload) on my network card was interacting with the epair network device and causing the issue (even through a VM). I have reported this to the FreeBSD bugzilla.
But in the meantime I still needed a solution to my problem. I noticed that disabling LRO, while I improved network performance, it was still in the order of 1GBit instead of 10GBit.
In this case I decided to setup the jail with host mode networking, but to isolate the jail into it's own FIB (Forwarding Information Base).
What is a FIB?
You may know this better as a routing table - it is how your computer (or router) makes decisions about where traffic should be routed to. Routing tables always match more-specific route when they decide where to send traffic.
As an example:
# netstat -rn
Destination Gateway Flags Netif Expire
default 172.24.10.1 UGS bridge0
127.0.0.1 link#4 UH lo0
172.24.10.0/24 link#5 U bridge0
172.24.10.2 link#4 UHS lo0
In this example, if you were to ping 127.0.0.1 the route table shows that this should be sent via
the device lo0, and that the network is directly attached to that interfaces (Gateway = link). If
we were to ping 172.24.10.1 this would be sent via bridge1 (as 172.24.10.1 is part of the subnet 172.24.10.0/24)
and that 172.24.10.1 should be on that network (Gateway = link). Finally if we were to ping 103.2.119.199
then since no subnets match, we fall back to the default route, and the traffic is sent via the gateway
router at 172.24.10.1.
So Why Would You Need Multiple FIBs?
Imagine our network is laid out like so. You'll notice the FIB from above is from the Server in this example.
┌─────────────┐ ┌─────────────┐
│ router │ │ Server │
│┌───────────┐│ │┌───────────┐│
││172.24.10.1│◀─────────────┤│172.24.10.2││
│├───────────┤│ │└───────────┘│
││172.24.11.1││ │ │
│├───────────┤│ │ │
││172.24.12.1│◀──────┐ │ │
└┴───────────┴┘ │ └─────────────┘
│
│ ┌───────────────┐
│ │ Laptop │
│ │┌─────────────┐│
└──────┤│172.24.12.100││
│└─────────────┘│
│ │
│ │
│ │
└───────────────┘
When our laptop contacts the server, it has to go via the router. When the server replies to the laptop, since the laptop's address is not in the server's FIB, it uses the default route for the return traffic.
Now let's add another interface on the server, but attached to a separate VLAN (Virtual LAN).
┌─────────────┐ ┌─────────────┐
│ router │ │ Server │
│┌───────────┐│ │┌───────────┐│
││172.24.10.1│◀─────────────┤│172.24.10.2││
│├───────────┤│ │├───────────┤│
││172.24.11.1│◀─────────────┤│172.24.11.2││
│├───────────┤│ │└───────────┘│
││172.24.12.1│◀──────┐ │ │
└┴───────────┴┘ │ └─────────────┘
│
│ ┌───────────────┐
│ │ Laptop │
│ │┌─────────────┐│
└──────┤│172.24.12.100││
│└─────────────┘│
│ │
│ │
│ │
└───────────────┘
Our servers FIB would update to:
# netstat -rn
Destination Gateway Flags Netif Expire
default 172.24.10.1 UGS bridge0
127.0.0.1 link#4 UH lo0
172.24.10.0/24 link#5 U bridge0
172.24.10.2 link#4 UHS lo0
172.24.11.0/24 link#6 U bridge1
172.24.11.2 link#4 UHS lo0
So when our laptop (172.24.12.100) contacts the server on 172.24.10.2, everything works as before.
But if our laptop contacts the server on 172.24.11.2 it will fail. Why?
Because we created a triangular route.
┌─────────────┐ ┌─────────────┐
│ router │ │ Server │
│┌───────────┐│ │┌───────────┐│
││172.24.10.1│◀─X───3.──────┤│172.24.10.2││
│├───────────┤│ │├───────────┤│
││172.24.11.1│├─────2.──────▶│172.24.11.2││
│├───────────┤│ │└───────────┘│
││172.24.12.1│◀──────┐ │ │
└┴───────────┴┘ │ └─────────────┘
│
1. ┌───────────────┐
│ │ Laptop │
│ │┌─────────────┐│
└──────┤│172.24.12.100││
│└─────────────┘│
│ │
│ │
│ │
└───────────────┘
First the traffic from our laptop goes to the router (1.), which sends the packet to the server on 172.24.11.2 (2.). The server then processes the packet and needs to reply to the laptop. However, since our route table doesn't have 172.24.12.0/24, we fall back to the default route. So now the response from 172.24.11.2 is sent out via bridge0 - Not bridge1 (3.) !!! As a result the router will drop the response as it's source network (172.24.11.2) doesn't match the actual network subnet (172.24.10.0/24)
To resolve this we want to isolate each bridge with their own FIBs, so that they each have their own default routes.
The End Result
So when this is completed (on FreeBSD) you will have two (or more!) FIBs available, which you can
inspect with netstat. Notice the -F X where X is the FIB number.
# netstat -F 0 -rn
Routing tables
Internet:
Destination Gateway Flags Netif Expire
default 172.24.10.1 UGS bridge1
127.0.0.1 link#4 UH lo0
172.24.10.0/24 link#5 U bridge1
172.24.10.22 link#4 UHS lo0
# netstat -F 1 -rn
Routing tables (fib: 1)
Internet:
Destination Gateway Flags Netif Expire
default 172.24.11.1 UGS bridge2
127.0.0.1 link#4 UHS lo0
172.24.11.0/24 link#6 U bridge2
172.24.11.131 link#4 UHS lo0
Here you can see there is a separate FIB for bridge1 and bride11, and they have different default gateways.
Setting Up FIBs
Setup the number of FIBs you want in /boot/loader.conf
# /boot/loader.conf
net.fibs=2
When you create your interfaces in rc.conf, attach the FIB to your interface.
# Setup your tagged VLANs
vlans_ix0="1 2"
# Create the bridges
cloned_interfaces="bridge1 bridge2"
# Up the physical interface
ifconfig_ix0="up"
# Up the VLAN tagged 1
ifconfig_ix0_1="up"
# Add the VLAN 1 to bridge 1 and set an IP. This defaults to FIB 0
ifconfig_bridge1="inet 172.24.10.2/24 addm ix0.1"
# Add the defaultroute to FIB 0
defaultrouter="172.24.10.1"
# Repeat for VLAN 2
ifconfig_ix0_2="up"
# Add VLAN 2 to bridge 2
ifconfig_bridge2="addm ix0.2"
# Add the address to bridge 2 *and* assign it to FIB 1
ifconfig_bridge2_alias0="inet 172.24.11.131/24 fib 1"
# Add routes to FIB 1
static_routes="fibnetwork fibdefault"
route_fibnetwork="-net 172.24.11.0/24 -interface bridge11 -fib 1"
route_fibdefault="default 172.24.11.1 -fib 1"
Reboot your machine.
Now you can test your new routes - the command setfib executes a command under the specified FIB.
setfib -F 0 traceroute ....
setfib -F 1 traceroute ....
Now you have to configure the jail to run in the second FIB. Thankfully you just use "host mode" networking and it will automatically attach to the right FIB if you use an IP from that FIB.
# /etc/jail.conf.d/test.conf
test {
...
ip4.addr = 172.24.11.131;
}
Happy Gaoling!