Re: Tuning Linux for faster high-latency outbound transfers

From: Ketil Froyn
Date: Wed Jan 27 2010 - 15:05:13 EST

On Wed, Jan 27, 2010 at 3:27 PM, Matt Zagrabelny <mzagrabe@xxxxxxxxx> wrote:
> On Wed, 2010-01-27 at 11:45 +0100, Ketil Froyn wrote:
>> Hi,
>> I am based in Norway, and I'm benchmarking upload and download speeds
>> to and from a US site (Rackspace Cloud Files). I'm on a 1Gbit link
>> with a 100Mbit firewall limiting my speeds, but I routinely see both
>> uploads and downloads at 20Mbit, and spikes up to 50Mbit. Ping latency
>> between my site and the US site is about 170ms.
>> The speeds I'm seeing for single large file transfers when benchmarking are:
>> uploads: 90kb/s - 140kb/s
>> downloads: 450kb/s - 500kb/s
> So are you seeing 20Mb/s or 90 - 140 and 450 - 500 kb/s ?

I regularly see 20Mb/s on downloads from outside our network by
customers of the service I'm running. 90-140kb/s is the speed I get
when uploading files to Rackspace, and 450-500 when downloading from

> How many hops away are you from the destination system?

Traceroute shows 14 hops before I get * * * after 3 hops in Rackspace's own net.

> Are you sure there is no shaping, throttling, or other contention
> (besides the firewall) between your system and the destination?

There's none on our side by us or our ISP. I asked Rackspace, they
said they allow much faster transfers than what I see.

> Use iperf (aptitude install iperf) to saturate the link. You can send
> both TCP and UDP packets/streams.

I can't test that against Rackspace, but since I have access to a
virtual host in the US anyway, I can test against that instead. It has
approx 130ms rtt. Testing, I see bandwidth numbers varying from
682Kbit/s to 1.59Mbit/s. I got some even lower numbers when I tried to
increase the window size.

I've also noticed some other things:

1. ping shows a ~1% packet loss (2 packets out of 120 missing).
2. scp transfers start at about 360kb/s, and then fall, completing a
file at an average of about 85kb/s (680Kbit/s)
3. running 2 scp in parallell, they also started faster and then fell,
ending at about 82kb/s each (~1.3Mbit/s total)
4. running 4 scp in parallell, they got an average of about 78kb/s
each, and gave a total of ~2.4Mbit/s transferred

So the network appears to be able to handle more data transferred than
a single transfer can handle, so I guess it should be able to tune
this somehow? Or do I just need to run many transfers in parallel?

(I may have been inaccurate with my use of kb/Kbit, but I mean
kilobyte and kilobit, respectively, *1024 in both cases.)

Regards, Ketil
To unsubscribe from this list: send the line "unsubscribe linux-net" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at