Using Iptables To Track and Enforce Data Cap

OpenWrt Firewall

A plethora of reading in the top hits: Google "traffic accounting iptables" Read OpenWrt Wiki :: Firewall. Reference the Advanced Bash Scripting Guide for shell script syntax.

OpenWrt comes configured with a pretty good "sane default" firewall configuration. One thing I noticed was there were rules inserted with comments, like "/ user chain for xxxxxx /", with chain names already there for some user-made rules to tie in to.

Since we want everything else to work normally, and just work with traffic being forwarded between interfaces, everything I come up with here can go into the FILTER table under the FORWARD section.

There are a couple details in our way:

  • A 3G/4G modem is not always 'connected'. The connection can and will drop.
  • The firewall will be reset every time a network interface comes up.

Any custom firewall config can be added to /etc/firewall.user, in the form of shell commands. However, firewall.user is executed at the tail end of a reset, which means that we cannot use it to capture byte counter information from before the reset.

There is a point at which we can place a script to do stuff just before the firewall gets reset, using OpenWrt's hotplugd. Look in:

/
├── etc
│   └── hotplud.d
│       └── iface
│           ├── iface
│           ├── 00-netstate
│           ├── 10-qos
│           ├── 15-teql
│           ├── 20-firewall             # this script does the reset
│           ├── 20-ntpclient
│           ├── 30-6relay
│           ├── 30-relay
│           └── 50-miniupnpd

Each of the scripts in that directory are run on every ifup/ifdown/ifupdate of any interface.

To keep from clobbering perfectly a perfectly good setup, I'm going to leave 20-firewall alone, and instead put my script into a file that will execute just before, 19-traffic-accounting.

Traffic Accounting Overview

I'm going to put in two rules to account for traffic coming and going. They will match traffic based on the incoming and outgoing interface, '3g-wan', since the IP address gets dynamically set. Since this is going to be attached to the FORWARD chain, it is guaranteed to be data that is using up our monthly quota.

TODO:  Figure out background usage for a 3G/4G connection is,
e.g. do the LCP echoes of the PPP link count?

TODO:  Play with this some more after the first month to see if rules
in the INPUT and OUTPUT chains would be more appropriate.

I can also add a rule into the FORWARD chain that uses iptables quota, and cut off the internet when the monthly cap is hit.

So, I want to save the byte counters to disk in 19-traffic-accounting, and restore them from disk in /etc/firewall.user.

/etc/firewall.user:

# This file is interpreted as shell script.
# Put your custom iptables rules here, they will
# be executed with each firewall (re-)start.

# these names are just stubs for byte counters
iptables -N forward_upcount
iptables -N forward_downcount

# type our counts to integers, so as to avoid certain syntax errors
declare -i downcount
declare -i upcount
# initialize the counts to default of 0
downcount=0
upcount=0
# ...and if there is a value already stored to disk, load them
[ -e /downcount.txt ] && downcount=$(cat /downcount.txt)
[ -e /upcount.txt ] && upcount=$(cat /upcount.txt)

# create the two rules, matching incoming and outgoing interface,
# and initialize the byte counters
# 'forwarding_rule' is the stub that OpenWrt leaves open for us to work with
iptables -A forwarding_rule -i 3g-wan -j forward_downcount -c 0 $downcount -m comment --comment "downcount"
iptables -A forwarding_rule -o 3g-wan -j forward_upcount -c 0 $upcount -m comment --comment "upcount"

# and now the quota rule chain
# first, create the chain name
iptables -N forwarding_quota 
# as long we are below the quota, we want to do NOTHING further to the packet in this chain, so RETURN to where we were
iptables -A forwarding_quota -i 3g-wan -m quota --quota 21474836480 -c 0 $downcount -j RETURN -m comment --comment "downquota"
# if we are over the quota, we drop everything in the next rules
# maybe I could do it one rule, but I'd like to see how much I'm dropping in either direction
iptables -A forwarding_quota -i 3g-wan -j DROP
iptables -A forwarding_quota -o 3g-wan -j DROP
# now attach this to the forwarding_rule
iptables -A forwarding_rule -j forwarding_quota

# Internal uci firewall chains are flushed and recreated on reload, so
# put custom rules into the root chains e.g. INPUT or FORWARD or into the
# special user chains, e.g. input_wan_rule or postrouting_lan_rule.

This adds the following functionality when the firewall starts:

  • add two rules, one each for counting bytes coming and going over the 3g-wan interface, restoring the byte counts from a couple files

  • add a quota chain, also restoring the byte count, with a hard limit of 20 gigabytes that starts dropping traffic when the limit is reached

Now, to add a way to save the byte counts in the first place.

/etc/hotplug.d/iface/19-traffic-accounting:

#!/bin/sh
# This file needs to be named 19-something, so it gets run before 20-firewall

declare -i olddowncount
[ -e /downcount.txt ] && olddowncount=$(cat /downcount.txt)
declare -i oldupcount
[ -e /upcount.txt ] && oldupcount=$(cat /upcount.txt)

declare -i downcount
declare -i upcount

logger -t firewall "Saving bandwidth usage due to $ACTION of $INTERFACE ($DEVICE)"

# Now, we need to get the byte counts out the FILTER table, FORWARD chain
# grep the comment field, and awk the bytes out
downcount=$(iptables -v -n -x -L | grep downcount | awk '{print $2}')
upcount=$(iptables -v -n -x -L | grep upcount | awk '{print $2}')

logger -t firewall "Downcount of $downcount"
logger -t firewall "Upcount of $upcount"

# now, save it somewhere the firewall.user can load it from
# but only if the new value is higher than the old value
# because the value should only ever increase during a month
[ "$downcount" -gt "$olddowncount" ] && echo "$downcount" > /downcount.txt
[ "$upcount" -gt "$oldupcount" ] && echo "$upcount" > /upcount.txt

# it couldn't hurt to save it to a monthly log file, too
# but we can't put that in /var/log, because in OpenWrt that's a tmpfs mountpoint
logfile=$(date +%Y-%m.log)
fulldate=$(date +%Y-%m-%d\ %H:%M:%S)
echo "$fulldate :: $downcount :: $upcount :: $ACTION" >> /root/$logfile
exit 0

It probably wouldn't hurt to use cron to log the usage at intervals, as well as reset the count at the first of the month.

/log_usage.sh:

#!/bin/sh

declare -i olddowncount
[ -e /downcount.txt ] && olddowncount=$(cat /downcount.txt)
declare -i oldupcount
[ -e /upcount.txt ] && oldupcount=$(cat /upcount.txt)

declare -i downcount
declare -i upcount

downcount=$(iptables -v -n -x -L | grep downcount | awk '{print $2}')
upcount=$(iptables -v -n -x -L | grep upcount | awk '{print $2}')

[ "$downcount" -gt "$olddowncount" ] && echo "$downcount" > /downcount.txt
[ "$upcount" -gt "$oldupcount" ] && echo "$upcount" > /upcount.txt

logfile=$(date +%Y-%m.log)
fulldate=$(date +%Y-%m-%d\ %H:%M:%S)
echo "$fulldate :: $downcount :: $upcount :: LOG" >> /root/$logfile
exit 0

/reset-quota.sh:

#!/bin/sh

# reset-quota.sh
# This script should be run at midnight on the 1st day of the month

# It's the end of a month, and the beginning of the next!
# MORE DATA, YAY

# First, let us save the current state to our logs
# This is a bit sticky, since at the time of running, this script does
# not know what file is 'last month'
# but it wouldn't hurt in any way to simply put it into this month
declare -i downcount
declare -i upcount

downcount=$(iptables -v -n -x -L | grep downcount | awk '{print $2}')
upcount=$(iptables -v -n -x -L | grep upcount | awk '{print $2}')

logfile=$(date +%Y-%m.log)
fulldate=$(date +%Y-%m-%d\ %H:%M:%S)
echo "$fulldate :: $downcount :: $upcount :: MONTH_FINISH" >> /root/$logfile

# Next, let's zero the counters in the iptables
iptables -Z

# Next, let us zero out the stored counters
echo 0 > /downcount.txt
echo 0 > /upcount.txt

# Everything should be ready and counting 
exit 0

and finally, the crontab entries:

0 0 1 * * /reset-quota.sh
*/5 * * * * /log-usage.sh