Re: Where is the performance bottleneck?

From: Holger Kiehl
Date: Wed Aug 31 2005 - 11:26:22 EST

Next message: Alan Cox: "Re: tty problem"
Previous message: Jens Axboe: "Re: Where is the performance bottleneck?"
In reply to: Nick Piggin: "Re: Where is the performance bottleneck?"
Next in thread: Nick Piggin: "Re: Where is the performance bottleneck?"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

On Wed, 31 Aug 2005, Nick Piggin wrote:

Holger Kiehl wrote:

3236497 total 1.4547
2507913 default_idle 52248.1875
158752 shrink_zone 43.3275
121584 copy_user_generic_c 3199.5789
34271 __wake_up_bit 713.9792
31131 __make_request 23.1629
22096 scsi_request_fn 18.4133
21915 rotate_reclaimable_page 80.5699

^^^^^^^^^

I don't think this function should be here. This indicates that
lots of writeout is happening due to pages falling off the end
of the LRU.

There was a bug recently causing memory estimates to be wrong
on Opterons that could cause this I think.

Can you send in 2 dumps of /proc/vmstat taken 10 seconds apart
while you're writing at full speed (with 2.6.13 or the latest
-git tree).

I took 2.6.13, there where no git snapshots at www.kernel.org when
I looked. With 2.6.13 I must load the Fusion MPT driver as module.
Compiling it in it does not detect the drive correctly, as module
there is no problem.

Here is what I did:

#!/bin/bash

time dd if=/dev/full of=/dev/sdc1 bs=4M count=4883 &
time dd if=/dev/full of=/dev/sdd1 bs=4M count=4883 &
time dd if=/dev/full of=/dev/sde1 bs=4M count=4883 &
time dd if=/dev/full of=/dev/sdf1 bs=4M count=4883 &
time dd if=/dev/full of=/dev/sdg1 bs=4M count=4883 &
time dd if=/dev/full of=/dev/sdh1 bs=4M count=4883 &
time dd if=/dev/full of=/dev/sdi1 bs=4M count=4883 &
time dd if=/dev/full of=/dev/sdj1 bs=4M count=4883 &

sleep 20

cat /proc/vmstat > /root/vmstat-1.dump

sleep 10

cat /proc/vmstat > /root/vmstat-2.dump
cat /proc/zoneinfo > /root/zoneinfo.dump
cat /proc/meminfo > /root/meminfo.dump

exit 0

vmstat-1.dump:

nr_dirty 787282
nr_writeback 44317
nr_unstable 0
nr_page_table_pages 633
nr_mapped 6373
nr_slab 53030
pgpgin 263362
pgpgout 5260352
pswpin 0
pswpout 0
pgalloc_high 0
pgalloc_normal 2448628
pgalloc_dma 1041
pgfree 2457343
pgactivate 5775
pgdeactivate 2113
pgfault 465679
pgmajfault 321
pgrefill_high 0
pgrefill_normal 5940
pgrefill_dma 33
pgsteal_high 0
pgsteal_normal 148759
pgsteal_dma 0
pgscan_kswapd_high 0
pgscan_kswapd_normal 153813
pgscan_kswapd_dma 1089
pgscan_direct_high 0
pgscan_direct_normal 0
pgscan_direct_dma 0
pginodesteal 0
slabs_scanned 0
kswapd_steal 148759
kswapd_inodesteal 0
pageoutrun 5304
allocstall 0
pgrotated 0
nr_bounce 0

vmstat-2.dump:

nr_dirty 786397
nr_writeback 44233
nr_unstable 0
nr_page_table_pages 640
nr_mapped 6406
nr_slab 53027
pgpgin 263382
pgpgout 7835732
pswpin 0
pswpout 0
pgalloc_high 0
pgalloc_normal 3091687
pgalloc_dma 2420
pgfree 3101327
pgactivate 5817
pgdeactivate 2918
pgfault 466269
pgmajfault 322
pgrefill_high 0
pgrefill_normal 28265
pgrefill_dma 150
pgsteal_high 0
pgsteal_normal 789909
pgsteal_dma 1388
pgscan_kswapd_high 0
pgscan_kswapd_normal 904101
pgscan_kswapd_dma 4950
pgscan_direct_high 0
pgscan_direct_normal 0
pgscan_direct_dma 0
pginodesteal 0
slabs_scanned 1152
kswapd_steal 791297
kswapd_inodesteal 0
pageoutrun 28299
allocstall 0
pgrotated 562
nr_bounce 0

zoneinfo.dump:

Node 3, zone Normal
pages free 899
min 726
low 907
high 1089
active 3996
inactive 490989
scanned 0 (a: 16 i: 0)
spanned 524287
present 524287
protection: (0, 0, 0)
pagesets
cpu: 0 pcp: 0
count: 2
low: 62
high: 186
batch: 31
cpu: 0 pcp: 1
count: 0
low: 0
high: 62
batch: 31
numa_hit: 10186
numa_miss: 3313
numa_foreign: 0
interleave_hit: 10136
local_node: 0
other_node: 13499
cpu: 1 pcp: 0
count: 13
low: 62
high: 186
batch: 31
cpu: 1 pcp: 1
count: 0
low: 0
high: 62
batch: 31
numa_hit: 6559
numa_miss: 1668
numa_foreign: 0
interleave_hit: 6559
local_node: 0
other_node: 8227
cpu: 2 pcp: 0
count: 84
low: 62
high: 186
batch: 31
cpu: 2 pcp: 1
count: 0
low: 0
high: 62
batch: 31
numa_hit: 5579
numa_miss: 12806
numa_foreign: 0
interleave_hit: 5579
local_node: 0
other_node: 18385
cpu: 3 pcp: 0
count: 93
low: 62
high: 186
batch: 31
cpu: 3 pcp: 1
count: 55
low: 0
high: 62
batch: 31
numa_hit: 834769
numa_miss: 1
numa_foreign: 940192
interleave_hit: 5563
local_node: 834770
other_node: 0
all_unreclaimable: 0
prev_priority: 12
temp_priority: 12
start_pfn: 1572864
Node 2, zone Normal
pages free 1036
min 726
low 907
high 1089
active 360
inactive 501700
scanned 0 (a: 26 i: 0)
spanned 524287
present 524287
protection: (0, 0, 0)
pagesets
cpu: 1 pcp: 0
count: 91
low: 62
high: 186
batch: 31
cpu: 1 pcp: 1
count: 0
low: 0
high: 62
batch: 31
numa_hit: 6002
numa_miss: 15490
numa_foreign: 0
interleave_hit: 6002
local_node: 0
other_node: 21492
cpu: 2 pcp: 0
count: 75
low: 62
high: 186
batch: 31
cpu: 2 pcp: 1
count: 56
low: 0
high: 62
batch: 31
numa_hit: 410692
numa_miss: 0
numa_foreign: 76064
interleave_hit: 5223
local_node: 410692
other_node: 0
cpu: 3 pcp: 0
count: 73
low: 62
high: 186
batch: 31
cpu: 3 pcp: 1
count: 0
low: 0
high: 62
batch: 31
numa_hit: 5163
numa_miss: 288909
numa_foreign: 1
interleave_hit: 5152
local_node: 0
other_node: 294072
all_unreclaimable: 0
prev_priority: 12
temp_priority: 12
start_pfn: 1048576
Node 1, zone Normal
pages free 859
min 703
low 878
high 1054
active 1224
inactive 485043
scanned 0 (a: 14 i: 0)
spanned 507903
present 507760
protection: (0, 0, 0)
pagesets
cpu: 0 pcp: 0
count: 1
low: 62
high: 186
batch: 31
cpu: 0 pcp: 1
count: 0
low: 0
high: 62
batch: 31
numa_hit: 9443
numa_miss: 15475
numa_foreign: 18446604437880297808
interleave_hit: 18446604437880307200
local_node: 1
other_node: 24917
cpu: 1 pcp: 0
count: 181
low: 62
high: 186
batch: 31
cpu: 1 pcp: 1
count: 38
low: 0
high: 62
batch: 31
numa_hit: 368191
numa_miss: 0
numa_foreign: 39046
interleave_hit: 5967
local_node: 368191
other_node: 0
cpu: 2 pcp: 0
count: 85
low: 62
high: 186
batch: 31
cpu: 2 pcp: 1
count: 0
low: 0
high: 62
batch: 31
numa_hit: 5139
numa_miss: 18963
numa_foreign: 0
interleave_hit: 5139
local_node: 0
other_node: 24102
cpu: 3 pcp: 0
count: 92
low: 62
high: 186
batch: 31
cpu: 3 pcp: 1
count: 0
low: 0
high: 62
batch: 31
numa_hit: 5124
numa_miss: 363472
numa_foreign: 0
interleave_hit: 5115
local_node: 0
other_node: 368596
all_unreclaimable: 0
prev_priority: 12
temp_priority: 12
start_pfn: 524288
Node 0, zone DMA
pages free 2045
min 5
low 6
high 7
active 0
inactive 992
scanned 0 (a: 2 i: 2)
spanned 4096
present 3994
protection: (0, 2031, 2031)
pagesets
cpu: 0 pcp: 0
count: 1
low: 2
high: 6
batch: 1
cpu: 0 pcp: 1
count: 1
low: 0
high: 2
batch: 1
numa_hit: 18446604437880298786
numa_miss: 18446604442220017848
numa_foreign: 0
interleave_hit: 0
local_node: 7567460
other_node: 0
all_unreclaimable: 0
prev_priority: 12
temp_priority: 12
start_pfn: 0
Node 0, zone Normal
pages free 1052
min 721
low 901
high 1081
active 845
inactive 480162
scanned 0 (a: 2 i: 0)
spanned 520191
present 520191
protection: (0, 0, 0)
pagesets
cpu: 0 pcp: 0
count: 96
low: 62
high: 186
batch: 31
cpu: 0 pcp: 1
count: 50
low: 0
high: 62
batch: 31
numa_hit: 18446604437880708763
numa_miss: 18446604439958819000
numa_foreign: 29590
interleave_hit: 9679
local_node: 7977309
other_node: 0
cpu: 1 pcp: 0
count: 88
low: 62
high: 186
batch: 31
cpu: 1 pcp: 1
count: 0
low: 0
high: 62
batch: 31
numa_hit: 6206
numa_miss: 21831
numa_foreign: 0
interleave_hit: 6206
local_node: 0
other_node: 28037
cpu: 2 pcp: 0
count: 65
low: 62
high: 186
batch: 31
cpu: 2 pcp: 1
count: 0
low: 0
high: 62
batch: 31
numa_hit: 5367
numa_miss: 44135
numa_foreign: 0
interleave_hit: 5365
local_node: 0
other_node: 49502
cpu: 3 pcp: 0
count: 92
low: 62
high: 186
batch: 31
cpu: 3 pcp: 1
count: 0
low: 0
high: 62
batch: 31
numa_hit: 5544
numa_miss: 286378
numa_foreign: 0
interleave_hit: 5507
local_node: 0
other_node: 291922
all_unreclaimable: 0
prev_priority: 12
temp_priority: 12
start_pfn: 4096

meminfo.dump:

MemTotal: 8124172 kB
MemFree: 23564 kB
Buffers: 7825944 kB
Cached: 19216 kB
SwapCached: 0 kB
Active: 25708 kB
Inactive: 7835548 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 8124172 kB
LowFree: 23564 kB
SwapTotal: 15631160 kB
SwapFree: 15631160 kB
Dirty: 3145604 kB
Writeback: 176452 kB
Mapped: 25624 kB
Slab: 212116 kB
CommitLimit: 19693244 kB
Committed_AS: 85112 kB
PageTables: 2560 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 16288 kB
VmallocChunk: 34359721635 kB

Thanks,
Holger

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Next message: Alan Cox: "Re: tty problem"
Previous message: Jens Axboe: "Re: Where is the performance bottleneck?"
In reply to: Nick Piggin: "Re: Where is the performance bottleneck?"
Next in thread: Nick Piggin: "Re: Where is the performance bottleneck?"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]