Re: 2.6.32.3 x86_64 - XFS hangs, all I/O to D-state bug (again)

[Justin Piszcz]

Is this normal?  After the crash I just showed, the system was rebooted,
it is resyncing now but is this normal?

$ cat /proc/mdstat 
Personalities : [raid1] [raid6] [raid5] [raid4] 
md1 : active raid1 sdb2[1] sda2[0]
      136448 blocks [2/2] [UU]

md2 : active raid1 sdb3[1] sda3[0]
      129596288 blocks [2/2] [UU]

md3 : active raid5 sdj1[7] sdi1[6] sdh1[5] sdf1[3] sdg1[4] sde1[2] sdd1[1] sdc1[0]
      5128001536 blocks level 5, 1024k chunk, algorithm 2 [8/8] [UUUUUUUU]
      [==>..................]  resync = 12.3% (90186112/732571648) finish=1046.9min speed=10226K/sec

md0 : active raid1 sdb1[1] sda1[0]
      16787776 blocks [2/2] [UU]

unused devices: <none>

--

root        24  0.3  0.0      0     0 ?        S    07:37   0:23 [async/mgr]
root     15514  0.2  0.0      0     0 ?        S    08:04   0:11 [async/0]
root     15515  0.2  0.0      0     0 ?        S    08:04   0:09 [async/1]
root     15516  0.2  0.0      0     0 ?        S    08:04   0:10 [async/2]
root     15517  0.2  0.0      0     0 ?        S    08:04   0:09 [async/3]
root     15518  0.2  0.0      0     0 ?        S    08:04   0:10 [async/4]
root     15519  0.2  0.0      0     0 ?        S    08:04   0:10 [async/5]
root     15520  0.2  0.0      0     0 ?        S    08:04   0:11 [async/6]
root     15521  0.2  0.0      0     0 ?        S    08:04   0:10 [async/7]
root     15522  0.2  0.0      0     0 ?        S    08:04   0:10 [async/8]
root     15523  0.2  0.0      0     0 ?        S    08:04   0:10 [async/9]
root     15524  0.2  0.0      0     0 ?        S    08:04   0:10 [async/10]
root     15525  0.2  0.0      0     0 ?        S    08:04   0:10 [async/11]
root     15526  0.2  0.0      0     0 ?        S    08:04   0:11 [async/12]
root     15527  0.2  0.0      0     0 ?        S    08:04   0:10 [async/13]
root     15528  0.2  0.0      0     0 ?        S    08:04   0:11 [async/14]
root     15529  0.2  0.0      0     0 ?        S    08:04   0:10 [async/15]
root     15530  0.2  0.0      0     0 ?        S    08:04   0:10 [async/16]
root     15531  0.2  0.0      0     0 ?        S    08:04   0:11 [async/17]
root     15532  0.2  0.0      0     0 ?        S    08:04   0:11 [async/18]
root     15533  0.2  0.0      0     0 ?        S    08:04   0:11 [async/19]
root     15534  0.2  0.0      0     0 ?        S    08:04   0:11 [async/20]
root     15535  0.2  0.0      0     0 ?        S    08:04   0:11 [async/21]
root     15536  0.2  0.0      0     0 ?        S    08:04   0:10 [async/22]
root     15537  0.2  0.0      0     0 ?        S    08:04   0:10 [async/23]
root     15538  0.2  0.0      0     0 ?        S    08:04   0:11 [async/24]
root     15539  0.2  0.0      0     0 ?        S    08:04   0:11 [async/25]
root     15540  0.2  0.0      0     0 ?        S    08:04   0:11 [async/26]
root     15541  0.2  0.0      0     0 ?        S    08:04   0:11 [async/27]
root     15542  0.2  0.0      0     0 ?        S    08:04   0:10 [async/28]
root     15543  0.2  0.0      0     0 ?        S    08:04   0:12 [async/29]
root     15544  0.2  0.0      0     0 ?        S    08:04   0:10 [async/30]
root     15545  0.2  0.0      0     0 ?        S    08:04   0:11 [async/31]
root     15546  0.2  0.0      0     0 ?        R    08:04   0:11 [async/32]
root     15547  0.2  0.0      0     0 ?        S    08:04   0:11 [async/33]
root     15548  0.2  0.0      0     0 ?        S    08:04   0:11 [async/34]
root     15549  0.2  0.0      0     0 ?        R    08:04   0:11 [async/35]
root     15550  0.2  0.0      0     0 ?        S    08:04   0:11 [async/36]
root     15551  0.2  0.0      0     0 ?        S    08:04   0:11 [async/37]
root     15552  0.2  0.0      0     0 ?        S    08:04   0:11 [async/38]
root     15553  0.2  0.0      0     0 ?        S    08:04   0:10 [async/39]
root     15554  0.2  0.0      0     0 ?        S    08:04   0:11 [async/40]
root     15555  0.2  0.0      0     0 ?        S    08:04   0:11 [async/41]
root     15556  0.2  0.0      0     0 ?        S    08:04   0:11 [async/42]
root     15557  0.2  0.0      0     0 ?        S    08:04   0:10 [async/43]
root     15558  0.2  0.0      0     0 ?        S    08:04   0:10 [async/44]
root     15559  0.2  0.0      0     0 ?        R    08:04   0:11 [async/45]
root     15560  0.2  0.0      0     0 ?        S    08:04   0:10 [async/46]
root     15561  0.2  0.0      0     0 ?        S    08:04   0:11 [async/47]
root     15562  0.2  0.0      0     0 ?        S    08:04   0:10 [async/48]
root     15563  0.2  0.0      0     0 ?        S    08:04   0:11 [async/49]
root     15564  0.2  0.0      0     0 ?        S    08:04   0:11 [async/50]
root     15565  0.2  0.0      0     0 ?        R    08:04   0:11 [async/51]
root     15566  0.2  0.0      0     0 ?        S    08:04   0:11 [async/52]
root     15567  0.2  0.0      0     0 ?        S    08:04   0:11 [async/53]
root     15568  0.2  0.0      0     0 ?        S    08:04   0:11 [async/54]
root     15569  0.2  0.0      0     0 ?        S    08:04   0:10 [async/55]
root     15570  0.2  0.0      0     0 ?        S    08:04   0:11 [async/56]
root     15571  0.2  0.0      0     0 ?        S    08:04   0:10 [async/57]
root     15572  0.2  0.0      0     0 ?        S    08:04   0:12 [async/58]
root     15573  0.2  0.0      0     0 ?        S    08:04   0:10 [async/59]
root     15574  0.2  0.0      0     0 ?        S    08:04   0:12 [async/60]
root     15575  0.2  0.0      0     0 ?        S    08:04   0:11 [async/61]
root     15576  0.2  0.0      0     0 ?        S    08:04   0:10 [async/62]
root     15577  0.2  0.0      0     0 ?        R    08:04   0:12 [async/63]
root     15578  0.2  0.0      0     0 ?        S    08:04   0:11 [async/64]
root     15579  0.2  0.0      0     0 ?        S    08:04   0:11 [async/65]
root     15580  0.2  0.0      0     0 ?        S    08:04   0:11 [async/66]
root     15581  0.2  0.0      0     0 ?        S    08:04   0:10 [async/67]
root     15582  0.2  0.0      0     0 ?        S    08:04   0:12 [async/68]
root     15583  0.2  0.0      0     0 ?        S    08:04   0:10 [async/69]
root     15584  0.2  0.0      0     0 ?        R    08:04   0:11 [async/70]
root     15585  0.2  0.0      0     0 ?        S    08:04   0:10 [async/71]
root     15586  0.2  0.0      0     0 ?        S    08:04   0:11 [async/72]
root     15587  0.2  0.0      0     0 ?        S    08:04   0:11 [async/73]
root     15588  0.2  0.0      0     0 ?        S    08:04   0:11 [async/74]
root     15589  0.2  0.0      0     0 ?        S    08:04   0:11 [async/75]
root     15590  0.2  0.0      0     0 ?        S    08:04   0:11 [async/76]
root     15591  0.2  0.0      0     0 ?        S    08:04   0:11 [async/77]
root     15592  0.2  0.0      0     0 ?        S    08:04   0:11 [async/78]
root     15593  0.2  0.0      0     0 ?        S    08:04   0:11 [async/79]
root     15594  0.2  0.0      0     0 ?        S    08:04   0:11 [async/80]
root     15595  0.2  0.0      0     0 ?        S    08:04   0:10 [async/81]
root     15596  0.2  0.0      0     0 ?        S    08:04   0:11 [async/82]
root     15597  0.2  0.0      0     0 ?        S    08:04   0:12 [async/83]
root     15598  0.2  0.0      0     0 ?        S    08:04   0:11 [async/84]
root     15599  0.2  0.0      0     0 ?        S    08:04   0:11 [async/85]
root     15600  0.2  0.0      0     0 ?        S    08:04   0:11 [async/86]
root     15601  0.2  0.0      0     0 ?        S    08:04   0:10 [async/87]
root     15602  0.2  0.0      0     0 ?        S    08:04   0:12 [async/88]
root     15603  0.2  0.0      0     0 ?        S    08:04   0:11 [async/89]
root     15604  0.2  0.0      0     0 ?        S    08:04   0:11 [async/90]
root     15605  0.2  0.0      0     0 ?        S    08:04   0:11 [async/91]
root     15606  0.2  0.0      0     0 ?        S    08:04   0:11 [async/92]
root     15607  0.2  0.0      0     0 ?        S    08:04   0:13 [async/93]
root     15608  0.2  0.0      0     0 ?        S    08:04   0:11 [async/94]
root     15609  0.2  0.0      0     0 ?        S    08:04   0:11 [async/95]
root     15610  0.2  0.0      0     0 ?        S    08:04   0:11 [async/96]
root     15611  0.2  0.0      0     0 ?        S    08:04   0:10 [async/97]
root     15612  0.2  0.0      0     0 ?        S    08:04   0:12 [async/98]
root     15613  0.2  0.0      0     0 ?        S    08:04   0:11 [async/99]
root     15614  0.2  0.0      0     0 ?        S    08:04   0:11 [async/100]
root     15615  0.2  0.0      0     0 ?        S    08:04   0:11 [async/101]
root     15616  0.2  0.0      0     0 ?        S    08:04   0:11 [async/102]
root     15617  0.2  0.0      0     0 ?        S    08:04   0:12 [async/103]
root     15618  0.2  0.0      0     0 ?        S    08:04   0:11 [async/104]
root     15619  0.2  0.0      0     0 ?        R    08:04   0:12 [async/105]
root     15620  0.2  0.0      0     0 ?        S    08:04   0:11 [async/106]
root     15621  0.2  0.0      0     0 ?        S    08:04   0:10 [async/107]
root     15622  0.2  0.0      0     0 ?        S    08:04   0:12 [async/108]
root     15623  0.2  0.0      0     0 ?        S    08:04   0:12 [async/109]
root     15624  0.2  0.0      0     0 ?        S    08:04   0:11 [async/110]
root     15625  0.2  0.0      0     0 ?        S    08:04   0:11 [async/111]
root     15626  0.2  0.0      0     0 ?        S    08:04   0:11 [async/112]
root     15627  0.2  0.0      0     0 ?        S    08:04   0:12 [async/113]
root     15628  0.2  0.0      0     0 ?        S    08:04   0:12 [async/114]
root     15629  0.2  0.0      0     0 ?        S    08:04   0:12 [async/115]
root     15630  0.2  0.0      0     0 ?        S    08:04   0:12 [async/116]
root     15631  0.2  0.0      0     0 ?        S    08:04   0:10 [async/117]
root     15632  0.2  0.0      0     0 ?        S    08:04   0:12 [async/118]
root     15633  0.2  0.0      0     0 ?        S    08:04   0:11 [async/119]
root     15634  0.2  0.0      0     0 ?        S    08:04   0:11 [async/120]
root     15635  0.2  0.0      0     0 ?        S    08:04   0:11 [async/121]
root     15636  0.2  0.0      0     0 ?        S    08:04   0:11 [async/122]
root     15637  0.2  0.0      0     0 ?        S    08:04   0:12 [async/123]
root     15638  0.2  0.0      0     0 ?        R    08:04   0:11 [async/124]
root     15639  0.2  0.0      0     0 ?        S    08:04   0:11 [async/125]
root     15640  0.2  0.0      0     0 ?        S    08:04   0:12 [async/126]
root     15641  0.2  0.0      0     0 ?        S    08:04   0:10 [async/127]
root     15642  0.2  0.0      0     0 ?        S    08:04   0:12 [async/128]
root     15643  0.2  0.0      0     0 ?        S    08:04   0:12 [async/129]
root     15644  0.2  0.0      0     0 ?        S    08:04   0:11 [async/130]
root     15645  0.2  0.0      0     0 ?        S    08:04   0:11 [async/131]
root     15646  0.2  0.0      0     0 ?        S    08:04   0:11 [async/132]
root     15647  0.2  0.0      0     0 ?        S    08:04   0:12 [async/133]
root     15648  0.2  0.0      0     0 ?        S    08:04   0:12 [async/134]
root     15649  0.2  0.0      0     0 ?        S    08:04   0:11 [async/135]
root     15650  0.2  0.0      0     0 ?        S    08:04   0:11 [async/136]
root     15651  0.2  0.0      0     0 ?        S    08:04   0:11 [async/137]
root     15652  0.2  0.0      0     0 ?        S    08:04   0:12 [async/138]
root     15653  0.2  0.0      0     0 ?        S    08:04   0:11 [async/139]
root     15654  0.2  0.0      0     0 ?        S    08:04   0:11 [async/140]
root     15655  0.2  0.0      0     0 ?        S    08:04   0:11 [async/141]
root     15656  0.2  0.0      0     0 ?        S    08:04   0:11 [async/142]
root     15657  0.2  0.0      0     0 ?        S    08:04   0:12 [async/143]
root     15658  0.2  0.0      0     0 ?        S    08:04   0:11 [async/144]
root     15659  0.2  0.0      0     0 ?        S    08:04   0:11 [async/145]
root     15660  0.2  0.0      0     0 ?        S    08:04   0:11 [async/146]
root     15661  0.2  0.0      0     0 ?        S    08:04   0:11 [async/147]
root     15662  0.2  0.0      0     0 ?        S    08:04   0:12 [async/148]
root     15663  0.2  0.0      0     0 ?        S    08:04   0:11 [async/149]
root     15664  0.2  0.0      0     0 ?        S    08:04   0:11 [async/150]
root     15665  0.2  0.0      0     0 ?        S    08:04   0:11 [async/151]
root     15666  0.2  0.0      0     0 ?        S    08:04   0:11 [async/152]
root     15667  0.2  0.0      0     0 ?        S    08:04   0:12 [async/153]
root     15668  0.2  0.0      0     0 ?        S    08:04   0:11 [async/154]
root     15669  0.2  0.0      0     0 ?        S    08:04   0:11 [async/155]
root     15670  0.2  0.0      0     0 ?        S    08:04   0:11 [async/156]
root     15671  0.2  0.0      0     0 ?        S    08:04   0:11 [async/157]
root     15672  0.2  0.0      0     0 ?        S    08:04   0:13 [async/158]
root     15673  0.2  0.0      0     0 ?        S    08:04   0:11 [async/159]
root     15674  0.2  0.0      0     0 ?        S    08:04   0:11 [async/160]
root     15675  0.2  0.0      0     0 ?        S    08:04   0:11 [async/161]
root     15676  0.2  0.0      0     0 ?        S    08:04   0:10 [async/162]
root     15677  0.2  0.0      0     0 ?        S    08:04   0:12 [async/163]
root     15678  0.2  0.0      0     0 ?        S    08:04   0:11 [async/164]
root     15679  0.2  0.0      0     0 ?        S    08:04   0:11 [async/165]
root     15680  0.2  0.0      0     0 ?        S    08:04   0:11 [async/166]
root     15681  0.2  0.0      0     0 ?        S    08:04   0:11 [async/167]
root     15682  0.2  0.0      0     0 ?        S    08:04   0:12 [async/168]
root     15683  0.2  0.0      0     0 ?        S    08:04   0:11 [async/169]
root     15684  0.2  0.0      0     0 ?        R    08:04   0:11 [async/170]
root     15685  0.2  0.0      0     0 ?        S    08:04   0:11 [async/171]
root     15686  0.2  0.0      0     0 ?        S    08:04   0:10 [async/172]
root     15687  0.2  0.0      0     0 ?        S    08:04   0:12 [async/173]
root     15688  0.2  0.0      0     0 ?        S    08:04   0:11 [async/174]
root     15689  0.2  0.0      0     0 ?        S    08:04   0:11 [async/175]
root     15690  0.2  0.0      0     0 ?        S    08:04   0:11 [async/176]
root     15691  0.2  0.0      0     0 ?        S    08:04   0:11 [async/177]
root     15692  0.2  0.0      0     0 ?        S    08:04   0:12 [async/178]
root     15693  0.2  0.0      0     0 ?        S    08:04   0:10 [async/179]
root     15694  0.2  0.0      0     0 ?        S    08:04   0:12 [async/180]
root     15695  0.2  0.0      0     0 ?        S    08:04   0:11 [async/181]
root     15696  0.2  0.0      0     0 ?        S    08:04   0:10 [async/182]
root     15697  0.2  0.0      0     0 ?        S    08:04   0:12 [async/183]
root     15698  0.2  0.0      0     0 ?        R    08:04   0:11 [async/184]
root     15699  0.2  0.0      0     0 ?        S    08:04   0:11 [async/185]
root     15700  0.2  0.0      0     0 ?        S    08:04   0:11 [async/186]
root     15701  0.2  0.0      0     0 ?        S    08:04   0:10 [async/187]
root     15702  0.2  0.0      0     0 ?        R    08:04   0:12 [async/188]
root     15703  0.2  0.0      0     0 ?        S    08:04   0:10 [async/189]
root     15704  0.2  0.0      0     0 ?        R    08:04   0:11 [async/190]
root     15705  0.2  0.0      0     0 ?        S    08:04   0:11 [async/191]
root     15706  0.2  0.0      0     0 ?        R    08:04   0:11 [async/192]
root     15707  0.2  0.0      0     0 ?        S    08:04   0:11 [async/193]
root     15708  0.2  0.0      0     0 ?        R    08:04   0:11 [async/194]
root     15709  0.2  0.0      0     0 ?        S    08:04   0:11 [async/195]
root     15710  0.2  0.0      0     0 ?        S    08:04   0:11 [async/196]
root     15711  0.2  0.0      0     0 ?        R    08:04   0:10 [async/197]
root     15712  0.2  0.0      0     0 ?        R    08:04   0:12 [async/198]
root     15713  0.2  0.0      0     0 ?        S    08:04   0:11 [async/199]
root     15714  0.2  0.0      0     0 ?        S    08:04   0:11 [async/200]
root     15715  0.2  0.0      0     0 ?        R    08:04   0:11 [async/201]
root     15716  0.2  0.0      0     0 ?        S    08:04   0:11 [async/202]
root     15717  0.2  0.0      0     0 ?        R    08:04   0:12 [async/203]
root     15718  0.2  0.0      0     0 ?        R    08:04   0:11 [async/204]
root     15719  0.2  0.0      0     0 ?        S    08:04   0:11 [async/205]
root     15720  0.2  0.0      0     0 ?        R    08:04   0:11 [async/206]
root     15721  0.2  0.0      0     0 ?        R    08:04   0:11 [async/207]
root     15722  0.2  0.0      0     0 ?        S    08:04   0:12 [async/208]
root     15723  0.2  0.0      0     0 ?        R    08:04   0:11 [async/209]
root     15724  0.2  0.0      0     0 ?        S    08:04   0:11 [async/210]
root     15725  0.2  0.0      0     0 ?        R    08:04   0:11 [async/211]
root     15726  0.2  0.0      0     0 ?        S    08:04   0:11 [async/212]
root     15727  0.2  0.0      0     0 ?        R    08:04   0:12 [async/213]
root     15728  0.2  0.0      0     0 ?        S    08:04   0:11 [async/214]
root     15729  0.2  0.0      0     0 ?        R    08:04   0:12 [async/215]
root     15730  0.2  0.0      0     0 ?        S    08:04   0:11 [async/216]
root     15731  0.2  0.0      0     0 ?        R    08:04   0:11 [async/217]
root     15732  0.2  0.0      0     0 ?        S    08:04   0:12 [async/218]
root     15733  0.2  0.0      0     0 ?        R    08:04   0:11 [async/219]
root     15734  0.2  0.0      0     0 ?        S    08:04   0:11 [async/220]
root     15735  0.2  0.0      0     0 ?        R    08:04   0:11 [async/221]
root     15736  0.2  0.0      0     0 ?        R    08:04   0:11 [async/222]
root     15737  0.2  0.0      0     0 ?        R    08:04   0:12 [async/223]
root     15738  0.2  0.0      0     0 ?        S    08:04   0:11 [async/224]
root     15739  0.2  0.0      0     0 ?        R    08:04   0:11 [async/225]
root     15740  0.2  0.0      0     0 ?        R    08:04   0:12 [async/226]
root     15741  0.2  0.0      0     0 ?        S    08:04   0:11 [async/227]
root     15742  0.2  0.0      0     0 ?        R    08:04   0:12 [async/228]
root     15743  0.2  0.0      0     0 ?        S    08:04   0:11 [async/229]
root     15744  0.2  0.0      0     0 ?        R    08:04   0:11 [async/230]
root     15745  0.2  0.0      0     0 ?        R    08:04   0:11 [async/231]
root     15746  0.2  0.0      0     0 ?        S    08:04   0:11 [async/232]
root     15747  0.2  0.0      0     0 ?        S    08:04   0:12 [async/233]
root     15748  0.2  0.0      0     0 ?        S    08:04   0:11 [async/234]
root     15749  0.2  0.0      0     0 ?        S    08:04   0:12 [async/235]
root     15750  0.2  0.0      0     0 ?        S    08:04   0:12 [async/236]
root     15751  0.2  0.0      0     0 ?        S    08:04   0:11 [async/237]
root     15752  0.2  0.0      0     0 ?        S    08:04   0:13 [async/238]
root     15753  0.2  0.0      0     0 ?        S    08:04   0:12 [async/239]
root     15754  0.2  0.0      0     0 ?        R    08:04   0:12 [async/240]
root     15755  0.2  0.0      0     0 ?        S    08:04   0:13 [async/241]
root     15756  0.3  0.0      0     0 ?        R    08:04   0:15 [async/242]
root     15757  0.2  0.0      0     0 ?        R    08:04   0:13 [async/243]
root     15758  0.2  0.0      0     0 ?        R    08:04   0:13 [async/244]
root     15759  0.2  0.0      0     0 ?        R    08:04   0:13 [async/245]
root     15760  0.3  0.0      0     0 ?        R    08:04   0:14 [async/246]
root     15761  0.5  0.0      0     0 ?        S    08:04   0:24 [async/247]
root     15762  0.3  0.0      0     0 ?        R    08:04   0:16 [async/248]
root     15763  0.4  0.0      0     0 ?        R    08:04   0:18 [async/249]
root     15764  0.4  0.0      0     0 ?        R    08:04   0:20 [async/250]
root     15765  0.5  0.0      0     0 ?        R    08:04   0:25 [async/251]
root     15766  0.8  0.0      0     0 ?        R    08:04   0:37 [async/252]
root     15767  1.3  0.0      0     0 ?        R    08:04   1:01 [async/253]
root     15768  1.9  0.0      0     0 ?        S    08:04   1:30 [async/254]
root     15769  2.1  0.0      0     0 ?        R    08:04   1:40 [async/255]



On Thu, 11 Feb 2010, Justin Piszcz wrote:

> The errors in the e-mail had bad formatting, please see the following: 
> http://home.comcast.net/~jpiszcz/20100211/crash.txt
>
> In addition, the linux-raid list had a typo in it, fixed.
>
> On Thu, 11 Feb 2010, Justin Piszcz wrote:
>
> > Hello,
> >
> > While tarring and compressing (bzip2) a lot of files, the following error
> > occurred, note the output is not clean because this was taken from 
> > netconsole.
> >
> > When this occurs, the host cannot be rebooted with reboot/proceses cannot
> > be killed and the box locks up.  There are no apparent hardware issues.
> >
> > Before, asterisk would trigger this bug, since asterisk no longer runs on 
> > this host, it ran for ~2-3 months without any problems, until now.
> >
> > Please cc me as I am not on the lists, thanks.
> >
> > Is this a md raid issue or XFS? From the trace it appears to be an XFS bug?
> >
> > Feb 11 07:38:14 l1 [   20.270270] e100: eth1 NIC Link is Up 100 Mbps Full 
> > Duplex Feb 11 07:47:54 l1 [  600.432165] INFO: task scp:4871 blocked for 
> > more than 120 seconds. Feb 11 07:47:54 l1 [  600.432177] "echo 0 > 
> > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Feb 11 
> > 07:47:54 l1 [  600.432182] scp           D
> > Feb 11 07:47:54 l1  ffff8801eee6f14c Feb 11 07:47:54 l1     0  4871   4870 
> > 0x00000000 Feb 11 07:47:54 l1 [  600.432188]  ffff880220c87950
> > Feb 11 07:47:54 l1  0000000000000082
> > Feb 11 07:47:54 l1  0000000000200200
> > Feb 11 07:47:54 l1  ffff88023eb71400
> > Feb 11 07:47:54 l1 Feb 11 07:47:54 l1 [  600.432196]  000000000000c928
> > Feb 11 07:47:54 l1  ffff8801f7a55fd8
> > Feb 11 07:47:54 l1  ffff880205fc6150
> > Feb 11 07:47:54 l1  ffff880205fc63c8
> > Feb 11 07:47:54 l1 Feb 11 07:47:54 l1 [  600.432203]  0000000000001000
> > Feb 11 07:47:54 l1  ffffffff8108d02a
> > Feb 11 07:47:54 l1  ffff8801eee6f0c0
> > Feb 11 07:47:54 l1  ffff880205fc63c8
> > Feb 11 07:47:54 l1 Feb 11 07:47:54 l1 [  600.432209] Call Trace: Feb 11 
> > 07:47:54 l1 [  600.432230]  [<ffffffff8108d02a>] ? 
> > generic_file_buffered_write+0x1aa/0x290 Feb 11 07:47:54 l1 [  600.432236] 
> > [<ffffffff8145f02d>] ? __down_write_nested+0x7d/0xb0 Feb 11 07:47:54 l1 [ 
> > 600.432242]  [<ffffffff8120387d>] ? xfs_write+0x23d/0x950 Feb 11 07:47:54 
> > l1 [  600.432250]  [<ffffffff810ba913>] ? do_sync_write+0xe3/0x130 Feb 11 
> > 07:47:54 l1 [  600.432260]  [<ffffffff8106dd90>] ? 
> > autoremove_wake_function+0x0/0x30 Feb 11 07:47:54 l1 [  600.432266] 
> > [<ffffffff810e9654>] ? fsnotify+0x4/0x1a0 Feb 11 07:47:54 l1 [  600.432271] 
> > [<ffffffff8102f78e>] ? common_interrupt+0xe/0x13 Feb 11 07:47:54 l1 [ 
> > 600.432276]  [<ffffffff810bb45b>] ? vfs_write+0xcb/0x180 Feb 11 07:47:54 l1 
> > [ 600.432280]  [<ffffffff810bb613>] ? sys_write+0x53/0xa0 Feb 11 07:47:54 
> > l1 [ 600.432285]  [<ffffffff8102eeeb>] ? system_call_fastpath+0x16/0x1b Feb 
> > 11 07:47:54 l1 [  600.432291] INFO: task flush-9:3:4874 blocked for more 
> > than 120 seconds. Feb 11 07:47:54 l1 [  600.432294] "echo 0 > 
> > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Feb 11 
> > 07:47:54 l1 [  600.432297] flush-9:3     D
> > Feb 11 07:47:54 l1  ffff88021948d750 Feb 11 07:47:54 l1     0  4874      2 
> > 0x00000000 Feb 11 07:47:54 l1 [  600.432309]  ffff88023fa3c8c0
> > Feb 11 07:47:54 l1  0000000000000046
> > Feb 11 07:47:54 l1  000000001948d750
> > Feb 11 07:47:54 l1  ffffffff8163d788
> > Feb 11 07:47:54 l1 Feb 11 07:47:54 l1 [  600.432323]  000000000000c928
> > Feb 11 07:47:54 l1  ffff8801f79fbfd8
> > Feb 11 07:47:54 l1  ffff88021948d750
> > Feb 11 07:47:54 l1  ffff88021948d9c8
> > Feb 11 07:47:54 l1 Feb 11 07:47:54 l1 [  600.432329]  000000013f467bb0
> > Feb 11 07:47:54 l1  ffffffff8108c183
> > Feb 11 07:47:54 l1  0000000000000000
> > Feb 11 07:47:54 l1  ffff88021948d9c8
> > Feb 11 07:47:54 l1 Feb 11 07:47:54 l1 [  600.432337] Call Trace: Feb 11 
> > 07:47:54 l1 [  600.432341]  [<ffffffff8108c183>] ? find_lock_page+0x23/0x80 
> > Feb 11 07:47:54 l1 [  600.432344]  [<ffffffff8108cbb1>] ? 
> > find_or_create_page+0x41/0xc0 Feb 11 07:47:54 l1 [  600.432349] 
> > [<ffffffff8145dbc5>] ? schedule_timeout+0x195/0x1f0 Feb 11 07:47:54 l1 [ 
> > 600.432352]  [<ffffffff8145ea81>] ? __down+0x61/0xa0 Feb 11 07:47:54 l1 [ 
> > 600.432356]  [<ffffffff81072046>] ? down+0x46/0x50 Feb 11 07:47:54 l1 [ 
> > 600.432360]  [<ffffffff811ff4a4>] ? _xfs_buf_find+0x134/0x220 Feb 11 
> > 07:47:54 l1 [  600.432363]  [<ffffffff811ff5fe>] ? 
> > xfs_buf_get_flags+0x6e/0x190 Feb 11 07:47:54 l1 [  600.432366] 
> > [<ffffffff811ff732>] ? xfs_buf_read_flags+0x12/0xa0 Feb 11 07:47:54 l1 [ 
> > 600.432371] [<ffffffff811f5226>] ? xfs_trans_read_buf+0x1f6/0x350 Feb 11 
> > 07:47:54 l1 [ 600.432377]  [<ffffffff811b0268>] ? xfs_read_agf+0x68/0x190 
> > Feb 11 07:47:54 l1 [  600.432380]  [<ffffffff811b03c0>] ? 
> > xfs_alloc_read_agf+0x30/0xd0 Feb 11 07:47:54 l1 [  600.432383] 
> > [<ffffffff811b2519>] ? xfs_alloc_fix_freelist+0x379/0x450 Feb 11 07:47:54 
> > l1 [  600.432388] [<ffffffff811ddfe5>] ? xfs_iext_remove+0x35/0x80 Feb 11 
> > 07:47:54 l1 [ 600.432393]  [<ffffffff811bd83f>] ? 
> > xfs_bmap_add_extent_delay_real+0x5ef/0x11a0 Feb 11 07:47:54 l1 [ 
> > 600.432396] [<ffffffff811f4e73>] ? xfs_trans_log_buf+0x63/0xa0 Feb 11 
> > 07:47:54 l1 [ 600.432401]  [<ffffffff811e7410>] ? 
> > xlog_state_get_iclog_space+0x60/0x2c0 Feb 11 07:47:54 l1 [  600.432404] 
> > [<ffffffff8145f087>] ? __down_read+0x17/0xae Feb 11 07:47:54 l1 [ 
> > 600.432408]  [<ffffffff811b29c0>] ? xfs_alloc_vextent+0x310/0x4b0 Feb 11 
> > 07:47:54 l1 [  600.432412] [<ffffffff811bf148>] ? 
> > xfs_bmap_btalloc+0x598/0xa40 Feb 11 07:47:54 l1 [ 600.432417] 
> > [<ffffffff811c19a2>] ? xfs_bmapi+0x9e2/0x11a0 Feb 11 07:47:54 l1 [ 
> > 600.432422]  [<ffffffff811f3d4f>] ? xfs_trans_reserve+0x9f/0x210 Feb 11 
> > 07:47:54 l1 [  600.432425]  [<ffffffff811e2efe>] ? 
> > xfs_iomap_write_allocate+0x23e/0x3b0 Feb 11 07:47:54 l1 [  600.432429] 
> > [<ffffffff811e3aa0>] ? xfs_iomap+0x2c0/0x300 Feb 11 07:47:54 l1 [ 
> > 600.432433]  [<ffffffff811fb875>] ? xfs_map_blocks+0x25/0x30 Feb 11 
> > 07:47:54 l1 [  600.432437]  [<ffffffff811fc9a4>] ? 
> > xfs_page_state_convert+0x414/0x6c0 Feb 11 07:47:54 l1 [  600.432443] 
> > [<ffffffff81224463>] ? radix_tree_gang_lookup_tag_slot+0xc3/0xf0 Feb 11 
> > 07:47:54 l1 [  600.432447] [<ffffffff811fcf57>] ? 
> > xfs_vm_writepage+0x77/0x130 Feb 11 07:47:54 l1 [ 600.432453] 
> > [<ffffffff8109249a>] ? __writepage+0xa/0x40 Feb 11 07:47:54 l1 [ 
> > 600.432456]  [<ffffffff81092c6f>] ? write_cache_pages+0x1df/0x3c0 Feb 11 
> > 07:47:54 l1 [  600.432459]  [<ffffffff81092490>] ? __writepage+0x0/0x40 Feb 
> > 11 07:47:54 l1 [  600.432464]  [<ffffffff810d9092>] ? 
> > writeback_single_inode+0xd2/0x390 Feb 11 07:47:54 l1 [  600.432468] 
> > [<ffffffff810d9f3f>] ? writeback_inodes_wb+0x3ff/0x5e0 Feb 11 07:47:54 l1 [ 
> > 600.432473]  [<ffffffff810da23e>] ? wb_writeback+0x11e/0x1f0 Feb 11 
> > 07:47:54 l1 [  600.432479]  [<ffffffff810630ee>] ? 
> > try_to_del_timer_sync+0x5e/0x90 Feb 11 07:47:54 l1 [  600.432484] 
> > [<ffffffff810da56b>] ? wb_do_writeback+0x17b/0x180 Feb 11 07:47:54 l1 [ 
> > 600.432487] [<ffffffff810da5cd>] ? bdi_writeback_task+0x5d/0xa0 Feb 11 
> > 07:47:54 l1 [ 600.432492]  [<ffffffff8109e900>] ? bdi_start_fn+0x0/0xf0 Feb 
> > 11 07:47:54 l1 [  600.432496]  [<ffffffff8109e97e>] ? 
> > bdi_start_fn+0x7e/0xf0 Feb 11 07:47:54 l1 [  600.432499] 
> > [<ffffffff8109e900>] ? bdi_start_fn+0x0/0xf0 Feb 11 07:47:54 l1 [ 
> > 600.432503]  [<ffffffff8106dab6>] ? kthread+0x96/0xa0 Feb 11 07:47:54 l1 [ 
> > 600.432508]  [<ffffffff8102fdfa>] ? child_rip+0xa/0x20 Feb 11 07:47:54 l1 [ 
> > 600.432513]  [<ffffffff8106da20>] ? kthread+0x0/0xa0 Feb 11 07:47:54 l1 [ 
> > 600.432517]  [<ffffffff8102fdf0>] ? child_rip+0x0/0x20 Feb 11 07:49:54 l1 [ 
> > 720.432171] INFO: task scp:4871 blocked for more than 120 seconds. Feb 11 
> > 07:49:54 l1 [  720.432187] "echo 0 > 
> > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Feb 11 
> > 07:49:54 l1 [  720.432192] scp           D
> > Feb 11 07:49:54 l1  ffff8801eee6f14c Feb 11 07:49:54 l1     0  4871   4870 
> > 0x00000000 Feb 11 07:49:54 l1 [  720.432198]  ffff880220c87950
> > Feb 11 07:49:54 l1  0000000000000082
> > Feb 11 07:49:54 l1  0000000000200200
> > Feb 11 07:49:54 l1  ffff88023eb71400
> > Feb 11 07:49:54 l1 Feb 11 07:49:54 l1 [  720.432206]  000000000000c928
> > Feb 11 07:49:54 l1  ffff8801f7a55fd8
> > Feb 11 07:49:54 l1  ffff880205fc6150
> > Feb 11 07:49:54 l1  ffff880205fc63c8
> > Feb 11 07:49:54 l1 Feb 11 07:49:54 l1 [  720.432217]  0000000000001000
> > Feb 11 07:49:54 l1  ffffffff8108d02a
> > Feb 11 07:49:54 l1  ffff8801eee6f0c0
> > Feb 11 07:49:54 l1  ffff880205fc63c8
> > Feb 11 07:49:54 l1 Feb 11 07:49:54 l1 [  720.432224] Call Trace: Feb 11 
> > 07:49:54 l1 [  720.432238]  [<ffffffff8108d02a>] ? 
> > generic_file_buffered_write+0x1aa/0x290 Feb 11 07:49:54 l1 [  720.432243] 
> > [<ffffffff8145f02d>] ? __down_write_nested+0x7d/0xb0 Feb 11 07:49:54 l1 [ 
> > 720.432249]  [<ffffffff8120387d>] ? xfs_write+0x23d/0x950 Feb 11 07:49:54 
> > l1 [  720.432254]  [<ffffffff810ba913>] ? do_sync_write+0xe3/0x130 Feb 11 
> > 07:49:54 l1 [  720.432259]  [<ffffffff8106dd90>] ? 
> > autoremove_wake_function+0x0/0x30 Feb 11 07:49:54 l1 [  720.432265] 
> > [<ffffffff810e9654>] ? fsnotify+0x4/0x1a0 Feb 11 07:49:54 l1 [  720.432270] 
> > [<ffffffff8102f78e>] ? common_interrupt+0xe/0x13 Feb 11 07:49:54 l1 [ 
> > 720.432275]  [<ffffffff810bb45b>] ? vfs_write+0xcb/0x180 Feb 11 07:49:54 l1 
> > [ 720.432280]  [<ffffffff810bb613>] ? sys_write+0x53/0xa0 Feb 11 07:49:54 
> > l1 [ 720.432285]  [<ffffffff8102eeeb>] ? system_call_fastpath+0x16/0x1b Feb 
> > 11 07:49:54 l1 [  720.432291] INFO: task flush-9:3:4874 blocked for more 
> > than 120 seconds. Feb 11 07:49:54 l1 [  720.432295] "echo 0 > 
> > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Feb 11 
> > 07:49:54 l1 [  720.432298] flush-9:3     D
> > Feb 11 07:49:54 l1  ffff88021948d750 Feb 11 07:49:54 l1     0  4874      2 
> > 0x00000000 Feb 11 07:49:54 l1 [  720.432306]  ffff88023fa3c8c0
> > Feb 11 07:49:54 l1  0000000000000046
> > Feb 11 07:49:54 l1  000000001948d750
> > Feb 11 07:49:54 l1  ffffffff8163d788
> > Feb 11 07:49:54 l1 Feb 11 07:49:54 l1 [  720.432322]  000000000000c928
> > Feb 11 07:49:54 l1  ffff8801f79fbfd8
> > Feb 11 07:49:54 l1  ffff88021948d750
> > Feb 11 07:49:54 l1  ffff88021948d9c8
> > Feb 11 07:49:54 l1 Feb 11 07:49:54 l1 [  720.432329]  000000013f467bb0
> > Feb 11 07:49:54 l1  ffffffff8108c183
> > Feb 11 07:49:54 l1  0000000000000000
> > Feb 11 07:49:54 l1  ffff88021948d9c8
> > Feb 11 07:49:54 l1 Feb 11 07:49:54 l1 [  720.432336] Call Trace: Feb 11 
> > 07:49:54 l1 [  720.432340]  [<ffffffff8108c183>] ? find_lock_page+0x23/0x80 
> > Feb 11 07:49:54 l1 [  720.432343]  [<ffffffff8108cbb1>] ? 
> > find_or_create_page+0x41/0xc0 Feb 11 07:49:54 l1 [  720.432348] 
> > [<ffffffff8145dbc5>] ? schedule_timeout+0x195/0x1f0 Feb 11 07:49:54 l1 [ 
> > 720.432351]  [<ffffffff8145ea81>] ? __down+0x61/0xa0 Feb 11 07:49:54 l1 [ 
> > 720.432356]  [<ffffffff81072046>] ? down+0x46/0x50 Feb 11 07:49:54 l1 [ 
> > 720.432361]  [<ffffffff811ff4a4>] ? _xfs_buf_find+0x134/0x220 Feb 11 
> > 07:49:54 l1 [  720.432365]  [<ffffffff811ff5fe>] ? 
> > xfs_buf_get_flags+0x6e/0x190 Feb 11 07:49:54 l1 [  720.432368] 
> > [<ffffffff811ff732>] ? xfs_buf_read_flags+0x12/0xa0 Feb 11 07:49:54 l1 [ 
> > 720.432373] [<ffffffff811f5226>] ? xfs_trans_read_buf+0x1f6/0x350 Feb 11 
> > 07:49:54 l1 [ 720.432378]  [<ffffffff811b0268>] ? xfs_read_agf+0x68/0x190 
> > Feb 11 07:49:54 l1 [  720.432382]  [<ffffffff811b03c0>] ? 
> > xfs_alloc_read_agf+0x30/0xd0 Feb 11 07:49:54 l1 [  720.432386] 
> > [<ffffffff811b2519>] ? xfs_alloc_fix_freelist+0x379/0x450 Feb 11 07:49:54 
> > l1 [  720.432391] [<ffffffff811ddfe5>] ? xfs_iext_remove+0x35/0x80 Feb 11 
> > 07:49:54 l1 [ 720.432396]  [<ffffffff811bd83f>] ? 
> > xfs_bmap_add_extent_delay_real+0x5ef/0x11a0 Feb 11 07:49:54 l1 [ 
> > 720.432399] [<ffffffff811f4e73>] ? xfs_trans_log_buf+0x63/0xa0 Feb 11 
> > 07:49:54 l1 [ 720.432404]  [<ffffffff811e7410>] ? 
> > xlog_state_get_iclog_space+0x60/0x2c0 Feb 11 07:49:54 l1 [  720.432410] 
> > [<ffffffff8145f087>] ? __down_read+0x17/0xae Feb 11 07:49:54 l1 [ 
> > 720.432415]  [<ffffffff811b29c0>] ? xfs_alloc_vextent+0x310/0x4b0 Feb 11 
> > 07:49:54 l1 [  720.432420] [<ffffffff811bf148>] ? 
> > xfs_bmap_btalloc+0x598/0xa40 Feb 11 07:49:54 l1 [ 720.432426] 
> > [<ffffffff811c19a2>] ? xfs_bmapi+0x9e2/0x11a0 Feb 11 07:49:54 l1 [ 
> > 720.432430]  [<ffffffff811f3d4f>] ? xfs_trans_reserve+0x9f/0x210 Feb 11 
> > 07:49:54 l1 [  720.432434]  [<ffffffff811e2efe>] ? 
> > xfs_iomap_write_allocate+0x23e/0x3b0 Feb 11 07:49:54 l1 [  720.432437] 
> > [<ffffffff811e3aa0>] ? xfs_iomap+0x2c0/0x300 Feb 11 07:49:54 l1 [ 
> > 720.432441]  [<ffffffff811fb875>] ? xfs_map_blocks+0x25/0x30 Feb 11 
> > 07:49:54 l1 [  720.432445]  [<ffffffff811fc9a4>] ? 
> > xfs_page_state_convert+0x414/0x6c0 Feb 11 07:49:54 l1 [  720.432451] 
> > [<ffffffff81224463>] ? radix_tree_gang_lookup_tag_slot+0xc3/0xf0 Feb 11 
> > 07:49:54 l1 [  720.432455] [<ffffffff811fcf57>] ? 
> > xfs_vm_writepage+0x77/0x130 Feb 11 07:49:54 l1 [ 720.432459] 
> > [<ffffffff8109249a>] ? __writepage+0xa/0x40 Feb 11 07:49:54 l1 [ 
> > 720.432462]  [<ffffffff81092c6f>] ? write_cache_pages+0x1df/0x3c0 Feb 11 
> > 07:49:54 l1 [  720.432466]  [<ffffffff81092490>] ? __writepage+0x0/0x40 Feb 
> > 11 07:49:54 l1 [  720.432472]  [<ffffffff810d9092>] ? 
> > writeback_single_inode+0xd2/0x390 Feb 11 07:49:54 l1 [  720.432476] 
> > [<ffffffff810d9f3f>] ? writeback_inodes_wb+0x3ff/0x5e0 Feb 11 07:49:54 l1 [ 
> > 720.432480]  [<ffffffff810da23e>] ? wb_writeback+0x11e/0x1f0 Feb 11 
> > 07:49:54 l1 [  720.432485]  [<ffffffff810630ee>] ? 
> > try_to_del_timer_sync+0x5e/0x90 Feb 11 07:49:54 l1 [  720.432489] 
> > [<ffffffff810da56b>] ? wb_do_writeback+0x17b/0x180 Feb 11 07:49:54 l1 [ 
> > 720.432493] [<ffffffff810da5cd>] ? bdi_writeback_task+0x5d/0xa0 Feb 11 
> > 07:49:54 l1 [ 720.432497]  [<ffffffff8109e900>] ? bdi_start_fn+0x0/0xf0 Feb 
> > 11 07:49:54 l1 [  720.432500]  [<ffffffff8109e97e>] ? 
> > bdi_start_fn+0x7e/0xf0 Feb 11 07:49:54 l1 [  720.432503] 
> > [<ffffffff8109e900>] ? bdi_start_fn+0x0/0xf0 Feb 11 07:49:54 l1 [ 
> > 720.432506]  [<ffffffff8106dab6>] ? kthread+0x96/0xa0 Feb 11 07:49:54 l1 [ 
> > 720.432511]  [<ffffffff8102fdfa>] ? child_rip+0xa/0x20 Feb 11 07:49:54 l1 [ 
> > 720.432515]  [<ffffffff8106da20>] ? kthread+0x0/0xa0 Feb 11 07:49:54 l1 [ 
> > 720.432519]  [<ffffffff8102fdf0>] ? child_rip+0x0/0x20
> >
> > Justin.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/