Re: slow performance on disk/network i/o full speed after drop_caches

From: Stefan Priebe - Profihost AG
Date: Thu Aug 25 2011 - 22:54:50 EST


Hi Wu,

Ah you are running an older kernel that didn't show all the vmstat
numbers. But still it's revealing that node 0 is used heavily and node
1 is almost idle. So I won't be surprised to see most free pages lie
in node 1.
I'm running a 2.6.38 kernel.

There is at least a numastat proc file.
grep . /sys/devices/system/node/node*/numastat
/sys/devices/system/node/node0/numastat:numa_hit 5958586
/sys/devices/system/node/node0/numastat:numa_miss 0
/sys/devices/system/node/node0/numastat:numa_foreign 0
/sys/devices/system/node/node0/numastat:interleave_hit 4191
/sys/devices/system/node/node0/numastat:local_node 5885189
/sys/devices/system/node/node0/numastat:other_node 73397
/sys/devices/system/node/node1/numastat:numa_hit 488922
/sys/devices/system/node/node1/numastat:numa_miss 0
/sys/devices/system/node/node1/numastat:numa_foreign 0
/sys/devices/system/node/node1/numastat:interleave_hit 4187
/sys/devices/system/node/node1/numastat:local_node 386741
/sys/devices/system/node/node1/numastat:other_node 102181

modified it a little bit:
~# while [ true ]; do ps -eo
user,pid,tid,class,rtprio,ni,pri,psr,pcpu,vsz,rss,pmem,stat,wchan:28,cmd
| grep scp | grep -v grep; sleep 1; done

root 12409 12409 TS - 0 19 0 59.8 42136 1724 0.0 Ss
poll_schedule_timeout scp -t /tmp/

It's mostly doing poll() waits. There must be some dependency on
something other to make progress. Would you post the full ps output
for all tasks, and even better, run
complete ps output:
http://pastebin.com/raw.php?i=b948svzN

echo t> /proc/sysrq-trigger
sadly i wa sonly able to grab the output in this crazy format:
http://pastebin.com/raw.php?i=MBXvvyH1

Hope that still helps.

Thanks Stefan
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/