Success: connecting multiple ISP's to one gateway (was: Re: Equal Cost Multipath)

Zygo Blaxell (
17 Jun 1999 07:42:06 -0400

In article <7k21b0$khh$>,
Zygo Blaxell <> wrote:
>In article <>,
>jamal <> wrote:
>>>What this does is:
>>>1. Permanently disable the load balancing that was working, (with the
>>>same gateway on both interfaces; see my other posting on this
>>>thread) and
>>>2. Nothing else, as far as I can tell.
>When I try to combine that with extra routing tables for port 80, the
>load balancing goes away under mysterious circumstances.
>It looks like I'm using a code path that nobody else has tested before.

OK, I've looked at it, and it's fairly clear that what I was originally
doing only worked at all due to unintentional features or perhaps sheer

By using the tools properly I can now make things work fairly well except
in one case, and that case isn't my highest priority.

What I did:

# Mark packets from the private network for special routing.
ipchains -A input -i myNETdev -d 0/0 80 -p tcp -m 1

# The standard masquerading line (simplified for clarity)
ipchains -A forward -j MASQ

# This is effectively what badISP does, except they use REJECT.
# If the routing is working right, this rule should never be used.
# The -l makes any exceptions show up in the logs for debugging.
ipchains -A output -i badISPdev -d 0/0 80 -p tcp -j DENY -l

# This is effectively what goodISP does, except they use REJECT.
# The -l makes any exceptions show up in the logs for debugging.
ipchains -A output -i goodISPdev -s ! goodISPipaddress -j DENY -l

# Filter out "admin prohibited filter" packets, which cause some nasty
# effects (e.g. Netscape shuts down _all_ open connections on a page if it
# sees one of these on a new connection start!). Log them so they can be
# detected.
ipchains -A input -s 0/0 3 -d 0/0 13 -p icmp -j DENY -l

# Set up default route using equal cost multipath (ip = iproute2/ip/ip).
# Get rid of anything that might have been set up by dhcpcd or Red Hat.
while ip route del default; do echo Deleting default routes...; done

# Install new flow-balancing route, meaning each new IP address that we
# want to send packets to gets one of the two routes at random.
# Use 'weight' to bias the randomness.
ip route add default nexthop via goodISPgateway dev goodISPdev weight 1 \
nexthop via badISPgateway dev badISPdev weight 1

# Routes for ISP's news servers, members-only areas, etc.
# These only work if accessed from their respective interfaces, so
# don't try to load-balance them. Note that port 80 is still blocked by
# badISP, even on their own subnet (!), and so I let the fwmark below
# take precedence over these routes.
ip route add goodISPsubnet/8 via goodISPgateway dev goodISPdev
ip route add badISPsubnet/16 via badISPgateway dev badISPdev

# Set up a routing table for routing packets marked by ipchains.
# Note: the parameter for 'fwmark' is a hexadecimal number. It took me
# a _long_ time to notice that. Somebody might want to document that.
# I now use 1-digit, unambiguous integers for table and mark numbers.
# In any case, this is a routing table that is used only when the destination
# of the connection is TCP port 80--only goodISP can carry that traffic.
ip rule del fwmark 1 # for idempotency
ip rule add fwmark 1 lookup 1
ip route add table 1 default gw goodISPgateway dev goodISPdev

Note: new "ip rule"s are added near the top of the list, after the
implied routes (e.g. broadcast and loopback). So if a packet is marked
with a 1, then I use routing table 1, otherwise, I use the default table.
The only rule I need to override is the default route; otherwise the
usual semantics are sufficient.

One thing still doesn't work: outgoing HTTP connections from the local
host. There's a 50% probability that I'll be able to connect to any given
IP address on port 80, depending on which route is chosen.

One interesting quirk: The IP masquerading code seems to use the right
local IP address for both external interfaces; however, the local socket
code does not. In fact, it seems to go out of its way to be wrong most
of the time. Then again, it's late, and it does seem to be random after
all--I'm just being really really unlucky.

Zygo Blaxell, Linux Engineer, Corel Corporation. (work) or (play).  Opinions above are my own, not Corel's.
Linux hysterical 2.0.35 #1 Aug 9 17:26 EDT 1998 i486 up 10 days, 7:24
Linux washu 2.2.8 #2 Thu May 13 21:14 EDT 1999 i686 up 5 days, 11:01
To unsubscribe from this list: send the line "unsubscribe linux-net" in
the body of a message to