fwmark based routing stopped working in 2.6.32

From: Nebojsa Trpkovic
Date: Fri Jan 29 2010 - 08:13:46 EST


hello.

I have two ADSL links on eth2 and eth3.

ADSL1 (eth2) with IP 10.5.18.18 is default gateway in main routing table.

ADSL2 (eth3) with IP 10.5.18.22 is used just for marked packets:
###################################################
#!/bin/bash
ip route add default via 10.5.18.22 dev eth3 table 20
ip rule add fwmark 0x351 table 20
ip rule add fwmark 0x352 table 20
ip rule add fwmark 0x353 table 20
ip route flush cache
###################################################

everything worked fine for years using kernels 2.6.24 and 2.6.29.
recently I upgraded to 2.6.32-r2 and traffic through ADSL2 stopped.

the moment I delete table 20 and ip rules, everything works fine:
I can set both ADSL1 or ADSL2 as default gateway and they will work.

again, the moment I start making routing decision considering firewall
marks, I get traffic only on ADSL1 (main table default gw) interface.

I've found out that when I mark ICMP protocol with 0x351 fwmark and try
too ping something, ping packets are sent via eth3 indeed:
iptraf detailed eth3 statistics shows that there are constatnly outgoing
ICMP packages.

even more interesting is fact that there is exactly the same number of
incoming ICMP packages, but my ping output is empty:
there is no "Destination Host Unreachable" or similar - nothing.

this leeds me to believe that ICMP packages are routed right, I receive
some answer, but those answer packages are discarded.

so, I've flushed all firewall rules except marking for ICMP, and added
explicit
###################################################
iptables -t mangle -A OUTPUT -p ICMP -j MARK --set-mark 0x351
###################################################
that didn't help.

I've added explicit rule
###################################################
iptables -I INPUT -i eth3 -j ACCEPT
###################################################
that didn't help.

I've checked, and my source route verification is turned off for these
ifaces:
###################################################
etc # sysctl net.ipv4.conf.default.rp_filter
net.ipv4.conf.default.rp_filter = 1
etc # sysctl net.ipv4.conf.eth2.rp_filter
net.ipv4.conf.eth2.rp_filter = 0
etc # sysctl net.ipv4.conf.eth3.rp_filter
net.ipv4.conf.eth3.rp_filter = 0
###################################################
changing that to "=1" doesn't solve the problem.

tcpdump on eth3 after 3 pings to 216.239.34.10
###################################################
ping -I eth3 -c3 216.239.34.10
PING 216.239.34.10 (216.239.34.10) from 10.5.18.21 eth3: 56(84) bytes of
data.

--- 216.239.34.10 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2006ms
###################################################
###################################################
13:24:23.556436 00:23:54:07:e9:6a > 00:90:d0:da:d2:06, ethertype IPv4
(0x0800), length 98: 10.5.18.21 > 216.239.34.10: ICMP echo request, id
51300, seq 1, length 64
13:24:23.605304 00:90:d0:da:d2:06 > 00:23:54:07:e9:6a, ethertype IPv4
(0x0800), length 98: 216.239.34.10 > 10.5.18.21: ICMP echo reply, id
51300, seq 1, length 64
13:24:24.555536 00:23:54:07:e9:6a > 00:90:d0:da:d2:06, ethertype IPv4
(0x0800), length 98: 10.5.18.21 > 216.239.34.10: ICMP echo request, id
51300, seq 2, length 64
13:24:24.603520 00:90:d0:da:d2:06 > 00:23:54:07:e9:6a, ethertype IPv4
(0x0800), length 98: 216.239.34.10 > 10.5.18.21: ICMP echo reply, id
51300, seq 2, length 64
13:24:25.563105 00:23:54:07:e9:6a > 00:90:d0:da:d2:06, ethertype IPv4
(0x0800), length 98: 10.5.18.21 > 216.239.34.10: ICMP echo request, id
51300, seq 3, length 64
13:24:25.610497 00:90:d0:da:d2:06 > 00:23:54:07:e9:6a, ethertype IPv4
(0x0800), length 98: 216.239.34.10 > 10.5.18.21: ICMP echo reply, id
51300, seq 3, length 64
###################################################

so, I'm definitely getting those packets back, but system ignoress them.

any idea what could go wrong and why does my system discard packages
from eth3 if they are not routed by main ruting table?

any info on what could be changed between kernels 2.6.29 and 2.6.32
regarding this issue?

thx a lot.

regards,
nebojsa trpkovic
--
To unsubscribe from this list: send the line "unsubscribe linux-net" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html