Running a (semi-)stateless linux router for private network

It comes to no surprise that you should always avoid nf_conntrack when you can. Many problems can arise from having your linux keeping track of every connection being made. There are some optimisation that you can do for it but there is always the maximum connections you have to worry about (in linux, it defaults to 65536). Malicious users know this and with very few machines can easily fill the conntrack table (especially if it hasn't been optimised). Even without malicious users in mind, if you're expecting high load to your website, it's very easy to reach that limit.

Most often, servers don't need to keep track of all connections. You only need tracking if you need to filter packets in iptables based on previous established connections. For servers that only serve a simple purpose like with only port 80 (and maybe 21) open, don't require that. In those instances, you can disable connection tracking.

All connections are tracked in the kernel module nf_conntrack. Disabling it on a server that's expecting high load is relatively easy and there are guides on how to do this. However, if you're trying to run a NAT router, things get slightly complicated. In order to NAT something, you need to keep track of those connections so you can deliver packets from the outside network to the internal network. Because of this, running a stateless linux router is close to impossible.

Let's assume we have the following network:

Our network is pretty simple. We'll be running some services on the internal network and exposing them to the internet. We will also allow the servers on the internal network to browse the internet. Finally, we want to load balance exposed services on the internal network. From this we can draw the following assumptions:

  • All services on the internal network are load balanced on router01
  • All servers on the internal network can only access HTTP and HTTPS on the internet

With those assumptions in mind while configuring our network, let's get started and masquerade outgoing packets on router01:

iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE  

This unfortunately enforces nf_conntrack kernel module and all connections are now being tracked automatically. Fortunately there is a way to mark packets to not be tracked with the flag aptly named NOTRACK. In order to run a (semi-)stateless router, we need to flag all packets that don't require masquerading with it.

The first kind of connections we don't need masquerading is everything from router01 to the outside world. Since we'll be servicing HTTP and HTTPS from router01 (for load balancing), let's untrack those:

iptables -t raw -A PREROUTING -d -p tcp --dport 80 -j NOTRACK  
iptables -t raw -A PREROUTING -d -p tcp --dport 443 -j NOTRACK  
iptables -t raw -A OUTPUT -s -p tcp --sport 80 -j NOTRACK  
iptables -t raw -A OUTPUT -s -p tcp --sport 443 -j NOTRACK  

One thing to keep in mind while configuring iptables is the flow of packets and the inspection points we can manipulate.

From the above flow picture, we can see that in order to target and catch all packets, we need to filter both OUTPUT and PREROUTING. Lastly the only table that supports NOTRACK is the raw table.

Moving on, the next thing we don't need to track is any direct traffic to and from the internal network:

iptables -t raw -A PREROUTING -s -d -j NOTRACK  
iptables -t raw -A OUTPUT -s -d -j NOTRACK  

Finally, we will only be allowing servers on the internal network to only access SSH, HTTP and HTTPS on the internet. Those connections need to be tracked and masqueraded. That means that anything incoming from the internet that isn't from those ports does not need to be tracked as those will be dropped automatically.

iptables -t raw -A PREROUTING -i eth0 -m multiport -p tcp ! --sport 22,80,443 -j NOTRACK  

We also want to do the above to all udp packets except those that come from 53 as 53 allows internal servers to make outgoing DNS requests:

iptables -t raw -A PREROUTING -i eth0 -m multiport -p udp ! --sport 53 -j NOTRACK  

With these rules, our raw table should look something like this:

# iptables -t raw -L
Chain PREROUTING (policy ACCEPT)  
target     prot opt source               destination  
NOTRACK    tcp  --  anywhere             anywhere            multiport sports ! ssh,http,https  
NOTRACK    udp  --  anywhere             anywhere            multiport sports ! domain  
NOTRACK    tcp  --  anywhere          tcp dpt:http  
NOTRACK    tcp  --  anywhere          tcp dpt:https  
NOTRACK    all  -- 

Chain OUTPUT (policy ACCEPT)  
target     prot opt source               destination  
NOTRACK    tcp  --        anywhere            tcp spt:http  
NOTRACK    tcp  --        anywhere            tcp spt:https  
NOTRACK    all  --   

There you have it, you have successfully configured a (semi-)stateless router. The only connections being tracked are outgoing DNS, SSH, HTTP and HTTPS requests from inside the internal network. Now let's lock things up.

First things first is to filter all forwarding rules. we want to ALLOW all outgoing requests and we also want to ALLOW all packets from the internet coming from established connections so our internal servers can receive their outgoing browsing requests. Everything else we want to drop.

iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT  
iptables -A FORWARD -i eth1 -o eth0 -j ACCEPT  
iptables -A FORWARD -j DROP  

Next we want to filter all input rules. Since we trust our internal network, we will allow all packets from it. We also want to only accept packets from the internet on ports 80, 443 and 22 as our router will be load balancing incoming HTTP and HTTPS requests to internal servers. Finally, we also want to allow established connections:

iptables -A INPUT -m multiport -p tcp --dport 22,80,443 -j ACCEPT  
iptables -A INPUT -i eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT  
iptables -A INPUT -j DROP  

Personally, I would allow the following on all servers, both internal servers and on router01:

iptables -A INPUT -i eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT  
iptables -A INPUT -p icmp -j ACCEPT  
iptables -A INPUT -i lo -j ACCEPT  
iptables -A INPUT -s -j ACCEPT  

And there you have it, a (semi-)stateless linux router.

The above method was used to create a salt-managed private network and you can view all the for it on github, both the pillar files and state files.

Show Comments