Re: [PATCH 0/13] Parallel struct page initialisation v4

[Waiman Long]

On 04/28/2015 09:16 PM, Waiman Long wrote:
> On 04/28/2015 10:36 AM, Mel Gorman wrote:
> > The bulk of the changes here are related to Andrew's feedback. 
> > Functionally
> > there is almost no difference.
> >
> > Changelog since v3
> > o Fix section-related warning
> > o Comments, clarifications, checkpatch
> > o Report the number of pages initialised
> >
> > Changelog since v2
> > o Reduce overhead of topology_init
> > o Remove boot-time kernel parameter to enable/disable
> > o Enable on UMA
> >
> > Changelog since v1
> > o Always initialise low zones
> > o Typo corrections
> > o Rename parallel mem init to parallel struct page init
> > o Rebase to 4.0
> >
> > Struct page initialisation had been identified as one of the reasons why
> > large machines take a long time to boot. Patches were posted a long 
> > time ago
> > to defer initialisation until they were first used.  This was 
> > rejected on
> > the grounds it should not be necessary to hurt the fast paths. This 
> > series
> > reuses much of the work from that time but defers the initialisation of
> > memory to kswapd so that one thread per node initialises memory local to
> > that node.
> >
> > After applying the series and setting the appropriate Kconfig variable I
> > see this in the boot log on a 64G machine
> >
> > [    7.383764] kswapd 0 initialised deferred memory in 188ms
> > [    7.404253] kswapd 1 initialised deferred memory in 208ms
> > [    7.411044] kswapd 3 initialised deferred memory in 216ms
> > [    7.411551] kswapd 2 initialised deferred memory in 216ms
> >
> > On a 1TB machine, I see
> >
> > [    8.406511] kswapd 3 initialised deferred memory in 1116ms
> > [    8.428518] kswapd 1 initialised deferred memory in 1140ms
> > [    8.435977] kswapd 0 initialised deferred memory in 1148ms
> > [    8.437416] kswapd 2 initialised deferred memory in 1148ms
> >
> > Once booted the machine appears to work as normal. Boot times were 
> > measured
> > from the time shutdown was called until ssh was available again.  In the
> > 64G case, the boot time savings are negligible. On the 1TB machine, the
> > savings were 16 seconds.
> >
> > It would be nice if the people that have access to really large machines
> > would test this series and report how much boot time is reduced.
>
> I ran a bootup timing test on a 12-TB 16-socket IvyBridge-EX system. 
> From grub menu to ssh login, the bootup time was 453s before the patch 
> and 265s after the patch - a saving of 188s (42%). I used a different 
> OS environment and config file with this test and so the timing data 
> weren't comparable with my previous testing data. The kswapd log 
> entries were
>
> [   45.973967] kswapd 4 initialised 197655470 pages in 4390ms
> [   45.974214] kswapd 7 initialised 197655470 pages in 4390ms
> [   45.976692] kswapd 15 initialised 197654299 pages in 4390ms
> [   45.993284] kswapd 0 initialised 197131131 pages in 4410ms
> [   46.032735] kswapd 9 initialised 197655470 pages in 4447ms
> [   46.065856] kswapd 8 initialised 197655470 pages in 4481ms
> [   46.066615] kswapd 1 initialised 197622702 pages in 4483ms
> [   46.077995] kswapd 2 initialised 197655470 pages in 4495ms
> [   46.219508] kswapd 13 initialised 197655470 pages in 4633ms
> [   46.224358] kswapd 3 initialised 197655470 pages in 4641ms
> [   46.228441] kswapd 11 initialised 197655470 pages in 4643ms
> [   46.232258] kswapd 12 initialised 197655470 pages in 4647ms
> [   46.239659] kswapd 10 initialised 197655470 pages in 4654ms
> [   46.243402] kswapd 14 initialised 197655470 pages in 4657ms
> [   46.250368] kswapd 5 initialised 197655470 pages in 4666ms
> [   46.254659] kswapd 6 initialised 197655470 pages in 4670ms
>
> Cheers,
> Longman

Bad news!

I tried your patch on a 24-TB DragonHawk and got an out of memory panic. 
The kernel log messages were:
  :
[   80.126186] CPU  474: hi:  186, btch:  31 usd:   0
[   80.131457] CPU  475: hi:  186, btch:  31 usd:   0
[   80.136726] CPU  476: hi:  186, btch:  31 usd:   0
[   80.141997] CPU  477: hi:  186, btch:  31 usd:   0
[   80.147267] CPU  478: hi:  186, btch:  31 usd:   0
[   80.152538] CPU  479: hi:  186, btch:  31 usd:   0
[   80.157813] active_anon:0 inactive_anon:0 isolated_anon:0
[   80.157813]  active_file:0 inactive_file:0 isolated_file:0
[   80.157813]  unevictable:0 dirty:0 writeback:0 unstable:0
[   80.157813]  free:209 slab_reclaimable:7 slab_unreclaimable:42986
[   80.157813]  mapped:0 shmem:0 pagetables:0 bounce:0
[   80.157813]  free_cma:0
[   80.190428] Node 0 DMA free:568kB min:0kB low:0kB high:0kB 
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15988kB 
managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB 
slab_reclaimable:0kB slab_unreclaimable:14928kB kernel_stack:400kB 
pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB 
pages_scanned:0 all_unreclaimable? yes
[   80.233475] lowmem_reserve[]: 0 0 0 0
[   80.237542] Node 0 DMA32 free:20kB min:0kB low:0kB high:0kB 
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1961924kB 
managed:1333604kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB 
shmem:0kB slab_reclaimable:12kB slab_unreclaimable:101664kB 
kernel_stack:50176kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB 
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[   80.281456] lowmem_reserve[]: 0 0 0 0
[   80.285527] Node 0 Normal free:0kB min:0kB low:0kB high:0kB 
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
unevictable:0kB isolated(anon):0kB isolated(file):0kB 
present:1608515580kB managed:2097148kB mlocked:0kB dirty:0kB 
writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:4kB 
slab_unreclaimable:948kB kernel_stack:0kB pagetables:0kB unstable:0kB 
bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 
all_unreclaimable? yes
[   80.328958] lowmem_reserve[]: 0 0 0 0
[   80.333031] Node 1 Normal free:248kB min:0kB low:0kB high:0kB 
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
unevictable:0kB isolated(anon):0kB isolated(file):0kB 
present:1610612732kB managed:2228220kB mlocked:0kB dirty:0kB 
writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:12kB 
slab_unreclaimable:46240kB kernel_stack:3232kB pagetables:0kB 
unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 
all_unreclaimable? yes
[   80.377256] lowmem_reserve[]: 0 0 0 0
[   80.381325] Node 2 Normal free:0kB min:0kB low:0kB high:0kB 
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
unevictable:0kB isolated(anon):0kB isolated(file):0kB 
present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB 
writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB 
slab_unreclaimable:612kB kernel_stack:0kB pagetables:0kB unstable:0kB 
bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 
all_unreclaimable? yes
[   80.424764] lowmem_reserve[]: 0 0 0 0
[   80.428842] Node 3 Normal free:0kB min:0kB low:0kB high:0kB 
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
unevictable:0kB isolated(anon):0kB isolated(file):0kB 
present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB 
writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB 
slab_unreclaimable:600kB kernel_stack:0kB pagetables:0kB unstable:0kB 
bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 
all_unreclaimable? yes
[   80.472293] lowmem_reserve[]: 0 0 0 0
[   80.476360] Node 4 Normal free:0kB min:0kB low:0kB high:0kB 
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
unevictable:0kB isolated(anon):0kB isolated(file):0kB 
present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB 
writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB 
slab_unreclaimable:620kB kernel_stack:0kB pagetables:0kB unstable:0kB 
bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 
all_unreclaimable? yes
[   80.519803] lowmem_reserve[]: 0 0 0 0
[   80.523875] Node 5 Normal free:0kB min:0kB low:0kB high:0kB 
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
unevictable:0kB isolated(anon):0kB isolated(file):0kB 
present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB 
writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB 
slab_unreclaimable:584kB kernel_stack:0kB pagetables:0kB unstable:0kB 
bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 
all_unreclaimable? yes
[   80.567312] lowmem_reserve[]: 0 0 0 0
[   80.571379] Node 6 Normal free:0kB min:0kB low:0kB high:0kB 
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
unevictable:0kB isolated(anon):0kB isolated(file):0kB 
present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB 
writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB 
slab_unreclaimable:556kB kernel_stack:0kB pagetables:0kB unstable:0kB 
bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 
all_unreclaimable? yes
[   80.614814] lowmem_reserve[]: 0 0 0 0
[   80.618881] Node 7 Normal free:0kB min:0kB low:0kB high:0kB 
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
unevictable:0kB isolated(anon):0kB isolated(file):0kB 
present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB 
writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB 
slab_unreclaimable:556kB kernel_stack:0kB pagetables:0kB unstable:0kB 
bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 
all_unreclaimable? yes
[   80.662316] lowmem_reserve[]: 0 0 0 0
[   80.666390] Node 8 Normal free:0kB min:0kB low:0kB high:0kB 
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
unevictable:0kB isolated(anon):0kB isolated(file):0kB 
present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB 
writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB 
slab_unreclaimable:572kB kernel_stack:0kB pagetables:0kB unstable:0kB 
bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 
all_unreclaimable? yes
[   80.709827] lowmem_reserve[]: 0 0 0 0
[   80.713898] Node 9 Normal free:0kB min:0kB low:0kB high:0kB 
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
unevictable:0kB isolated(anon):0kB isolated(file):0kB 
present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB 
writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB 
slab_unreclaimable:572kB kernel_stack:0kB pagetables:0kB unstable:0kB 
bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 
all_unreclaimable? yes
[   80.757336] lowmem_reserve[]: 0 0 0 0
[   80.761407] Node 10 Normal free:0kB min:0kB low:0kB high:0kB 
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
unevictable:0kB isolated(anon):0kB isolated(file):0kB 
present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB 
writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB 
slab_unreclaimable:564kB kernel_stack:0kB pagetables:0kB unstable:0kB 
bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 
all_unreclaimable? yes
[   80.804941] lowmem_reserve[]: 0 0 0 0
[   80.809015] Node 11 Normal free:0kB min:0kB low:0kB high:0kB 
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
unevictable:0kB isolated(anon):0kB isolated(file):0kB 
present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB 
writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB 
slab_unreclaimable:572kB kernel_stack:0kB pagetables:0kB unstable:0kB 
bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 
all_unreclaimable? yes
[   80.852548] lowmem_reserve[]: 0 0 0 0
[   80.856620] Node 12 Normal free:0kB min:0kB low:0kB high:0kB 
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
unevictable:0kB isolated(anon):0kB isolated(file):0kB 
present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB 
writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB 
slab_unreclaimable:616kB kernel_stack:0kB pagetables:0kB unstable:0kB 
bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 
all_unreclaimable? yes
[   80.900158] lowmem_reserve[]: 0 0 0 0
[   80.904236] Node 13 Normal free:0kB min:0kB low:0kB high:0kB 
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
unevictable:0kB isolated(anon):0kB isolated(file):0kB 
present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB 
writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB 
slab_unreclaimable:592kB kernel_stack:0kB pagetables:0kB unstable:0kB 
bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 
all_unreclaimable? yes
[   80.947765] lowmem_reserve[]: 0 0 0 0
[   80.951847] Node 14 Normal free:0kB min:0kB low:0kB high:0kB 
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
unevictable:0kB isolated(anon):0kB isolated(file):0kB 
present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB 
writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB 
slab_unreclaimable:600kB kernel_stack:0kB pagetables:0kB unstable:0kB 
bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 
all_unreclaimable? yes
[   80.995380] lowmem_reserve[]: 0 0 0 0
[   80.999448] Node 15 Normal free:0kB min:0kB low:0kB high:0kB 
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
unevictable:0kB isolated(anon):0kB isolated(file):0kB 
present:1610612736kB managed:2097152kB mlocked:0kB dirty:0kB 
writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB 
slab_unreclaimable:548kB kernel_stack:0kB pagetables:0kB unstable:0kB 
bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 
all_unreclaimable? yes
[   81.042974] lowmem_reserve[]: 0 0 0 0
[   81.047044] Node 0 DMA: 132*4kB (U) 5*8kB (U) 0*16kB 0*32kB 0*64kB 
0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 568kB
[   81.059632] Node 0 DMA32: 5*4kB (U) 0*8kB 0*16kB 0*32kB 0*64kB 
0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20kB
[   81.071733] Node 0 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 
0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
[   81.083443] Node 1 Normal: 52*4kB (U) 5*8kB (U) 0*16kB 0*32kB 0*64kB 
0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 248kB
[   81.096227] Node 2 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 
0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
[   81.107935] Node 3 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 
0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
[   81.119643] Node 4 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 
0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
[   81.131347] Node 5 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 
0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
[   81.143056] Node 6 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 
0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
[   81.154767] Node 7 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 
0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
[   81.166473] Node 8 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 
0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
[   81.178179] Node 9 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 
0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
[   81.189893] Node 10 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 
0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
[   81.201695] Node 11 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 
0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
[   81.213496] Node 12 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 
0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
[   81.225324] Node 13 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 
0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
[   81.237130] Node 14 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 
0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
[   81.248926] Node 15 Normal: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 
0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 0kB
[   81.260726] 0 total pagecache pages
[   81.264565] 0 pages in swap cache
[   81.268212] Swap cache stats: add 0, delete 0, find 0/0
[   81.273962] Free swap  = 0kB
[   81.277125] Total swap = 0kB
[   81.280341] 6442421132 pages RAM
[   81.283888] 0 pages HighMem/MovableOnly
[   81.288109] 6433662383 pages reserved
[   81.292135] 0 pages hwpoisoned
[   81.295491] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds 
swapents oom_score_adj name
[   81.305245] Kernel panic - not syncing: Out of memory and no killable 
processes...
[   81.305245]
[   81.315200] CPU: 240 PID: 1 Comm: swapper/0 Not tainted 
4.0.1-pmm-bigsmp #1
[   81.322856] Hardware name: HP Superdome2 16s x86, BIOS Bundle: 
006.000.042 SFW: 015.099.000 04/01/2015
[   81.333096]  0000000000000000 ffff8800044c79c8 ffffffff8151b0c9 
ffff8800044c7a48
[   81.341262]  ffffffff8151ae1e 0000000000000008 ffff8800044c7a58 
ffff8800044c79f8
[   81.349428]  ffffffff810785c3 ffffffff81a13480 0000000000000000 
ffff8800001001d0
[   81.357595] Call Trace:
[   81.360287]  [<ffffffff8151b0c9>] dump_stack+0x68/0x77
[   81.365942]  [<ffffffff8151ae1e>] panic+0xb9/0x219
[   81.371213]  [<ffffffff810785c3>] ? 
__blocking_notifier_call_chain+0x63/0x80
[   81.378971]  [<ffffffff811384ce>] __out_of_memory+0x34e/0x350
[   81.385292]  [<ffffffff811385ee>] out_of_memory+0x5e/0x90
[   81.391230]  [<ffffffff8113ce9e>] __alloc_pages_slowpath+0x6be/0x740
[   81.398219]  [<ffffffff8113d15c>] __alloc_pages_nodemask+0x23c/0x250
[   81.405212]  [<ffffffff81186346>] kmem_getpages+0x56/0x110
[   81.411246]  [<ffffffff81187f44>] fallback_alloc+0x164/0x200
[   81.417474]  [<ffffffff81187cfd>] ____cache_alloc_node+0x8d/0x170
[   81.424179]  [<ffffffff811887bb>] kmem_cache_alloc_trace+0x17b/0x240
[   81.431169]  [<ffffffff813d5f3a>] init_memory_block+0x3a/0x110
[   81.437586]  [<ffffffff81b5f687>] memory_dev_init+0xd7/0x13d
[   81.443810]  [<ffffffff81b5f2af>] driver_init+0x2f/0x37
[   81.449556]  [<ffffffff81b1599b>] do_basic_setup+0x29/0xd5
[   81.455597]  [<ffffffff81b372c4>] ? sched_init_smp+0x140/0x147
[   81.462015]  [<ffffffff81b15c55>] kernel_init_freeable+0x20e/0x297
[   81.468815]  [<ffffffff81512ea0>] ? rest_init+0x80/0x80
[   81.474565]  [<ffffffff81512ea9>] kernel_init+0x9/0xf0
[   81.480216]  [<ffffffff8151f788>] ret_from_fork+0x58/0x90
[   81.486156]  [<ffffffff81512ea0>] ? rest_init+0x80/0x80
[   81.492350] ---[ end Kernel panic - not syncing: Out of memory and no 
killable processes...
[   81.492350]

-Longman




--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/