ip

Configuring Multiple Default Routes in Linux « Darien Kindlund’s Blog

Evan Davies

• 4 min read

Configuring Multiple Default Routes in Linux « Darien Kindlund’s Blog.

Assume you have a Linux system with more than one network interface card (NIC) — say eth0 and eth1. By default, administrators can define a single, default route (on eth0). However, if you receive traffic (i.e., ICMP pings) on eth1, the return traffic will go out eth0 by default.

This can be a bit of a problem — especially when the two NICs share the same parent network and you’re trying to preserve sane traffic flows. In a nutshell, this post will explain how you can ensure traffic going into eth0 goes out only on eth0, as well as enforce all traffic going into eth1 goes out only on eth1.

You’ve found the one post that actually explains this issue; your googling has paid off. You wouldn’t believe how many advanced Linux routing websites out there explain how to route everything including your kitchen sink — yet fail to clearly explain something as simple as this.

As always, we’ll explain by example. Assume the following:

  • eth0 - 10.10.70.38 netmask 255.255.255.0
  • eth0's gateway is: 10.10.70.254
  • eth1 - 192.168.7.126 netmask 255.255.255.0
  • eth1's gateway is: 192.168.7.1

    • First, you’ll need to make sure your Linux kernel has support for “policy routing” enabled. (As a reference, I’m using a v2.6.13-gentoo-r5 kernel.)

    During the kernel compilation process, you’ll want to:

    cd /usr/src/linux
    make menuconfig
    Select "Networking --->"
    Select "Networking options --->"
    [*] TCP/IP networking
    [*] IP: advanced router
    Choose IP: FIB lookup algorithm (FIB_HASH)
    [*] IP: policy routing
    [*] IP: use netfilter MARK value as routing key

    Next, you’ll want to download, compile, and install the iproute2 [1] utilities. (Most Linux distributions have binary packages for this utility.) Once installed, typing ip route show should bring up your system’s routing table. Type man ip for more information about this utility, in general.

    Speaking of which, assume the system’s initial route configuration looks like this:

    # netstat -anr
    Kernel IP routing table
    Destination Gateway Genmask Flags MSS Window irtt Iface
    192.168.7.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
    10.10.70.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
    0.0.0.0 192.168.7.1 0.0.0.0 UG 0 0 0 eth1

    So, basically, the system is using eth1 as the default route. If anyone pings 192.168.7.126, then the response packets will properly go out eth1 to the upstream gateway of 192.168.7.1. But what about pinging 10.10.70.38? Sure, the incoming ICMP packets will properly arrive on eth0, but the outgoing response packets will be sent out via eth1! That’s bad.

    Here’s how to fix this issue. Borrowing the method from a really sketchy website [2], you’ll first need to create a new policy routing table entry within the /etc/iproute2/rt_tables. Let’s call it table #1, named “admin” (for routing administrative traffic onto eth0).

    # echo "1 admin" >> /etc/iproute2/rt_tables

    Next, we’re going to set a couple of new entries within this “admin” table. Specifically, we’ll provide information about eth0‘s local /24 subnet, along with eth0‘s default gateway.

    ip route add 10.10.70.0/24 dev eth0 src 10.10.70.38 table admin
    ip route add default via 10.10.70.254 dev eth0 table admin

    At this point, you’ve created a new, isolated routing table named “admin” that really isn’t used by the OS just yet. Why? Because we still need to create a rule referencing how the OS should use this table. For starters, type ip rule show to see your current policy routing ruleset. Here’s what an empty ruleset looks like:

    0: from all lookup local
    32766: from all lookup main
    32767: from all lookup default

    Without going into all the boring details, each rule entry is evaluated in ascending order. The main gist is that your normal main routing table appears as entry 32766 in this list. (This would be the normal route table you’d see when you type netstat -anr.)

    We’re now going to create two new rule entries, that will be evaluated before the main rule entry.

    ip rule add from 10.10.70.38/32 table admin
    ip rule add to 10.10.70.38/32 table admin

    Typing ip rule show now shows the following policy routing rulesets:

    0: from all lookup local
    32764: from all to 10.10.70.38 lookup admin
    32765: from 10.10.70.38 lookup admin
    32766: from all lookup main
    32767: from all lookup default

    Rule 32764 specifies that for all traffic going to eth0‘s IP, make sure to use the “admin” routing table, instead of the “main” one. Likewise, rule 32765 indicates that for all traffic originating from eth0‘s IP, make sure to use the “admin” routing table as well. For all other packets, use the “main” routing table. In order to commit these changes, it’s a good idea to type ip route flush cache.

    Congratulations! You’re system should now properly route traffic to these two different default gateways. For more than 2 NICs, repeat the table/rule creation process as necessary.

    Please provide comments, if you find any errors or have corrections to this post. I don’t claim that this method will work for everyone; this information is designed primarily to preserve my sanity, when configuring routing on future multi-NIC Linux systems.

    References:
    [1] http://www.policyrouting.org
    [2] http://www.linuxhorizon.ro/iproute2.html