Problem of TCP bandwidth drops when change MTU to a small value

[Weilong Chen]

Hi, when we change the mtu to a small value, for example, ifconfig eth0 
mtu 68, IPERF test shows there's a great bandwidth drop while previous 
kernel versions don't.
Git bisect find the differences is from the patch 
28d35bcdd3925e7293408cdb8aa5f2aac5f0d6e3 (net: ipv4: don't let PMTU 
updates increase route MTU). After this patch, the stack send packets 
use a small mtu.

This problem can be reproduced easily on a qemu-kvm platform with virt 
nic E1000:
# ethtool -i eth0
driver: e1000
version: 7.3.21-k8-NAPI
firmware-version:
expansion-rom-version:
bus-info: 0000:00:03.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
# # iperf -c 9.81.3.11 -t 3
------------------------------------------------------------
Client connecting to 9.81.3.11, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 9.83.1.202 port 44644 connected with 9.81.3.11 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 3.0 sec   336 MBytes   938 Mbits/sec
# ifconfig eth0 mtu 68
# iperf -c 9.81.3.11 -t 3
------------------------------------------------------------
Client connecting to 9.81.3.11, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 9.83.1.202 port 44646 connected with 9.81.3.11 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 3.3 sec  3.62 MBytes  9.18 Mbits/sec