Re: slow performance on disk/network i/o full speed after drop_caches

From: Stefan Priebe - Profihost AG
Date: Thu Aug 25 2011 - 05:00:34 EST

Next message: Tejun Heo: "Re: [PATCH 2/6] cgroup: improve old cgroup handling incgroup_attach_proc()"
Previous message: Benjamin Tissoires: "[PATCH v3] HID: Add full support for Logitech Unifying receivers"
In reply to: Wu Fengguang: "Re: slow performance on disk/network i/o full speed afterdrop_caches"
Next in thread: Wu Fengguang: "Re: slow performance on disk/network i/o full speed afterdrop_caches"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Am 24.08.2011 11:33, schrieb Wu Fengguang:

On Wed, Aug 24, 2011 at 05:01:03PM +0800, Stefan Priebe - Profihost AG wrote:

sync&& echo 3>/proc/sys/vm/drop_caches&& sleep 2&& echo 0
/proc/sys/vm/drop_caches

Another way to get it working again is to stop some processes. Could be
mysql or apache or php fcgi doesn't matter. Just free some memory.
Although there are already 5GB free.

Is it a NUMA machine and _every_ node has enough free pages?

grep . /sys/devices/system/node/node*/vmstat

Thanks,
Fengguang

Hi Fengguang,

thanks for your fast reply.

Here is the data you requested:

root@server1015-han:~# grep . /sys/devices/system/node/node*/vmstat
/sys/devices/system/node/node0/vmstat:nr_written 5546561
/sys/devices/system/node/node0/vmstat:nr_dirtied 5572497
/sys/devices/system/node/node1/vmstat:nr_written 3936
/sys/devices/system/node/node1/vmstat:nr_dirtied 4190

modified it a little bit:
~# while [ true ]; do ps -eo user,pid,tid,class,rtprio,ni,pri,psr,pcpu,vsz,rss,pmem,stat,wchan:28,cmd | grep scp | grep -v grep; sleep 1; done

root 12409 12409 TS - 0 19 0 59.8 42136 1724 0.0 Ss poll_schedule_timeout scp -t /tmp/
root 12409 12409 TS - 0 19 0 64.0 42136 1724 0.0 Rs - scp -t /tmp/
root 12409 12409 TS - 0 19 0 67.7 42136 1724 0.0 Rs - scp -t /tmp/
root 12409 12409 TS - 0 19 8 70.6 42136 1724 0.0 Ss poll_schedule_timeout scp -t /tmp/
root 12409 12409 TS - 0 19 8 73.5 42136 1724 0.0 Rs - scp -t /tmp/
root 12409 12409 TS - 0 19 8 76.0 42136 1724 0.0 Rs - scp -t /tmp/
root 12409 12409 TS - 0 19 8 78.2 42136 1724 0.0 Rs - scp -t /tmp/
root 12409 12409 TS - 0 19 8 80.0 42136 1724 0.0 Rs - scp -t /tmp/
root 12409 12409 TS - 0 19 8 80.9 42136 1724 0.0 Ss poll_schedule_timeout scp -t /tmp/
root 12409 12409 TS - 0 19 2 76.7 42136 1724 0.0 Ss poll_schedule_timeout scp -t /tmp/
root 12409 12409 TS - 0 19 1 75.6 42136 1724 0.0 Ds pipe_read scp -t /tmp/
root 12409 12409 TS - 0 19 0 76.0 42136 1724 0.0 Rs - scp -t /tmp/
root 12409 12409 TS - 0 19 1 75.2 42136 1724 0.0 Rs - scp -t /tmp/
root 12409 12409 TS - 0 19 1 76.6 42136 1724 0.0 Rs - scp -t /tmp/
root 12409 12409 TS - 0 19 1 77.9 42136 1724 0.0 Rs - scp -t /tmp/
root 12409 12409 TS - 0 19 1 79.0 42136 1724 0.0 Ss poll_schedule_timeout scp -t /tmp/
root 12409 12409 TS - 0 19 1 72.8 42136 1724 0.0 Ss poll_schedule_timeout scp -t /tmp/
root 12409 12409 TS - 0 19 0 73.0 42136 1724 0.0 Ss poll_schedule_timeout scp -t /tmp/
root 12409 12409 TS - 0 19 0 73.8 42136 1724 0.0 Ss poll_schedule_timeout scp -t /tmp/
root 12409 12409 TS - 0 19 1 74.3 42136 1724 0.0 Ss poll_schedule_timeout scp -t /tmp/
root 12409 12409 TS - 0 19 1 73.4 42136 1724 0.0 Ss - scp -t /tmp/
root 12409 12409 TS - 0 19 1 71.3 42136 1724 0.0 Ss poll_schedule_timeout scp -t /tmp/
root 12409 12409 TS - 0 19 1 71.9 42136 1724 0.0 Rs - scp -t /tmp/
root 12409 12409 TS - 0 19 0 72.7 42136 1724 0.0 Ss poll_schedule_timeout scp -t /tmp/
root 12409 12409 TS - 0 19 3 73.5 42136 1724 0.0 Rs - scp -t /tmp/
root 12409 12409 TS - 0 19 3 74.4 42136 1724 0.0 Rs - scp -t /tmp/
root 12409 12409 TS - 0 19 3 75.2 42136 1724 0.0 Rs - scp -t /tmp/
root 12409 12409 TS - 0 19 0 76.0 42136 1724 0.0 Ss poll_schedule_timeout scp -t /tmp/
root 12409 12409 TS - 0 19 8 76.6 42136 1724 0.0 Ss poll_schedule_timeout scp -t /tmp/
root 12409 12409 TS - 0 19 1 74.8 42136 1724 0.0 Ss poll_schedule_timeout scp -t /tmp/
root 12409 12409 TS - 0 19 1 73.2 42136 1724 0.0 Ss poll_schedule_timeout scp -t /tmp/
root 12409 12409 TS - 0 19 1 73.9 42136 1724 0.0 Rs poll_schedule_timeout scp -t /tmp/
root 12409 12409 TS - 0 19 0 72.4 42136 1724 0.0 Ss poll_schedule_timeout scp -t /tmp/
root 12409 12409 TS - 0 19 8 72.0 42136 1724 0.0 Ss poll_schedule_timeout scp -t /tmp/
root 12409 12409 TS - 0 19 8 72.5 42136 1724 0.0 Ss poll_schedule_timeout scp -t /tmp/
root 12409 12409 TS - 0 19 8 72.9 42136 1724 0.0 Rs - scp -t /tmp/
root 12409 12409 TS - 0 19 8 73.5 42136 1724 0.0 Rs - scp -t /tmp/
root 12566 12566 TS - 0 19 1 0.0 42136 1728 0.0 Rs - scp -t /tmp/
root 12566 12566 TS - 0 19 1 23.0 42136 1728 0.0 Rs - scp -t /tmp/
root 12566 12566 TS - 0 19 1 49.5 42136 1728 0.0 Rs - scp -t /tmp/
root 12566 12566 TS - 0 19 2 63.3 42136 1728 0.0 Rs - scp -t /tmp/
root 12566 12566 TS - 0 19 1 71.5 42136 1728 0.0 Rs - scp -t /tmp/
root 12566 12566 TS - 0 19 1 77.4 42136 1728 0.0 Rs - scp -t /tmp/
root 12566 12566 TS - 0 19 1 70.3 42136 1728 0.0 Rs - scp -t /tmp/
root 12566 12566 TS - 0 19 1 73.1 42136 1728 0.0 Ss poll_schedule_timeout scp -t /tmp/
root 12566 12566 TS - 0 19 0 65.7 42136 1728 0.0 Ss poll_schedule_timeout scp -t /tmp/
root 12566 12566 TS - 0 19 1 61.2 42136 1728 0.0 Ss - scp -t /tmp/
root 12566 12566 TS - 0 19 1 63.7 42136 1728 0.0 Rs - scp -t /tmp/
root 12636 12636 TS - 0 19 8 0.0 42136 1728 0.0 Ss poll_schedule_timeout scp -t /tmp/

Stefan

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Next message: Tejun Heo: "Re: [PATCH 2/6] cgroup: improve old cgroup handling incgroup_attach_proc()"
Previous message: Benjamin Tissoires: "[PATCH v3] HID: Add full support for Logitech Unifying receivers"
In reply to: Wu Fengguang: "Re: slow performance on disk/network i/o full speed afterdrop_caches"
Next in thread: Wu Fengguang: "Re: slow performance on disk/network i/o full speed afterdrop_caches"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]