Re: [3.6.6] panic on reboot / khungtaskd blocked? (WARNING: at arch/x86/kernel/smp.c:123 native_smp_send_reschedule)

From: PaweÅ Sikora
Date: Mon Nov 12 2012 - 05:22:45 EST


On Monday 12 of November 2012 15:40:31 Michael Wang wrote:
> On 11/12/2012 03:16 PM, PaweÅ Sikora wrote:
> > On Monday 12 of November 2012 11:04:12 Michael Wang wrote:
> >> On 11/09/2012 09:48 PM, PaweÅ Sikora wrote:
> >>> Hi,
> >>>
> >>> during playing with new ups i've caught an nice oops on reboot:
> >>>
> >>> http://imgbin.org/index.php?page=image&id=10253
> >>>
> >>> probably the upstream is also affected.
> >>
> >> Hi, PaweÅ
> >>
> >> Are you using a clean 3.6.6 without any modify?
> >
> > yes, pure 3.6.6 form git tree with modular config.
> >
> >> Looks like some threads has set itself to be UNINTERRUPTIBLE with out
> >> any design on switch itself back later(or the time is too long), are you
> >> accidentally using some bad designed module?
> >
> > hmm, hard to say. mostly all modules are loaded automatically by kernel.
>
> Could you please provide the whole dmesg in text? your picture lost the
> print info of the hung task.

i've grabbed the console via rs232 but there's no more info (see attached txt).
the dmesg (filesystem) is not synced during panic (leds on keyboard blink, sysrq doesn't work).
how can i grab more info?

INIT: Switching to runlevel: 6

INIT: Sending processes the TERM signal
Resource Manager: Entering runlevel number............................[ 6 ]
Stopping automounter service.......................................[ BUSY ][ DONE ]
Setting terminals to default values................................[ BUSY ][ DONE ]
Stopping mdadm service.............................................[ BUSY ][ DONE ]
Stopping Samba SMB/CIFS Server service.............................[ BUSY ][ DONE ]
Stopping Samba NetBIOS Name Server service.........................[ BUSY ][ DONE ]
Stopping cronie crond service......................................[ BUSY ][ DONE ]
Stopping NIS client service........................................[ BUSY ][ DONE ]
Stopping S.M.A.R.T. Daemon service.................................[ BUSY ][ DONE ]
Stopping block device availability:................................[ BUSY ][ DONE ]
Unmounting NFS filesystems.........................................[ BUSY ][ DONE ]
Unmounting NFS filesystems.........................................[ BUSY ][ DONE ]
Stopping Name Switch Cache Daemon service..........................[ BUSY ][ DONE ]
Saving random seed.................................................[ BUSY ][ DONE ]
Stopping NFS daemon service........................................[ BUSY ][ DONE ]
Stopping NFS mountd service........................................[ BUSY ][ DONE ]
Stopping NFS exportfs service......................................[ BUSY ][ DONE ]
Stopping RPC idmapd service........................................[ BUSY ][ DONE ]
Stopping syslog-ng service.........................................[ BUSY ][ DONE ]
Stopping D-BUS Message bus service.................................[ BUSY ][ DONE ]
Stopping rpcbind service...........................................[ BUSY ][ DONE ]
Stopping OpenSSH service...........................................[ DONE ]
Shutting down interface eth0.......................................[ BUSY ][ DONE ]
Shutting down interface eth1.......................................[ BUSY ][ DONE ]
Stopping UPS daemon service........................................Network UPS Tools upsd 2.6.4-Unversioned directory
Stopping UPS drivers...............................................[ BUSY ][ DONE ]
Stopping UPSmon service............................................[ BUSY ][ DONE ]
Flushing all chains................................................[ DONE ]
Removing user defined chains.......................................[ DONE ]
Resetting built-in chains to the default ACCEPT policy.............[ DONE ]
Flushing all chains................................................[ DONE ]
Removing user defined chains.......................................[ DONE ]
Resetting built-in chains to the default ACCEPT policy.............[ DONE ]
Stopping irqbalance service........................................[ BUSY ][ DONE ]
Sending all processes the TERM signal..............................[ BUSY ][ DONE ]
Sending all processes the KILL signal..............................[ BUSY ][ DONE ]
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 5.6 0.0 8468 772 ? Ss 10:47 0:15 init [6]
root 2 0.0 0.0 0 0 ? S 10:47 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S 10:47 0:00 [ksoftirqd/0]
root 4 0.1 0.0 0 0 ? S 10:47 0:00 [kworker/0:0]
root 5 0.0 0.0 0 0 ? S< 10:47 0:00 [kworker/0:0H]
root 6 0.0 0.0 0 0 ? S 10:47 0:00 [kworker/u:0]
root 7 0.0 0.0 0 0 ? S< 10:47 0:00 [kworker/u:0H]
root 8 0.0 0.0 0 0 ? S 10:47 0:00 [migration/0]
root 9 0.0 0.0 0 0 ? S 10:47 0:00 [watchdog/0]
root 10 0.0 0.0 0 0 ? S 10:47 0:00 [migration/1]
root 11 0.0 0.0 0 0 ? S 10:47 0:00 [kworker/1:0]
root 12 0.0 0.0 0 0 ? S< 10:47 0:00 [kworker/1:0H]
root 13 0.0 0.0 0 0 ? S 10:47 0:00 [ksoftirqd/1]
root 14 0.0 0.0 0 0 ? S 10:47 0:00 [watchdog/1]
root 15 0.0 0.0 0 0 ? S 10:47 0:00 [migration/2]
root 16 0.0 0.0 0 0 ? S 10:47 0:00 [kworker/2:0]
root 17 0.0 0.0 0 0 ? S< 10:47 0:00 [kworker/2:0H]
root 18 0.0 0.0 0 0 ? S 10:47 0:00 [ksoftirqd/2]
root 19 0.0 0.0 0 0 ? S 10:47 0:00 [watchdog/2]
root 20 0.0 0.0 0 0 ? S 10:47 0:00 [migration/3]
root 21 0.0 0.0 0 0 ? S 10:47 0:00 [kworker/3:0]
root 22 0.0 0.0 0 0 ? S< 10:47 0:00 [kworker/3:0H]
root 23 0.0 0.0 0 0 ? S 10:47 0:00 [ksoftirqd/3]
root 24 0.0 0.0 0 0 ? S 10:47 0:00 [watchdog/3]
root 25 0.0 0.0 0 0 ? S 10:47 0:00 [migration/4]
root 26 0.0 0.0 0 0 ? S 10:47 0:00 [kworker/4:0]
root 27 0.0 0.0 0 0 ? S< 10:47 0:00 [kworker/4:0H]
root 28 0.0 0.0 0 0 ? S 10:47 0:00 [ksoftirqd/4]
root 29 0.0 0.0 0 0 ? S 10:47 0:00 [watchdog/4]
root 30 0.0 0.0 0 0 ? S 10:47 0:00 [migration/5]
root 31 0.0 0.0 0 0 ? S 10:47 0:00 [kworker/5:0]
root 32 0.0 0.0 0 0 ? S< 10:47 0:00 [kworker/5:0H]
root 33 0.0 0.0 0 0 ? S 10:47 0:00 [ksoftirqd/5]
root 34 0.0 0.0 0 0 ? S 10:47 0:00 [watchdog/5]
root 35 0.0 0.0 0 0 ? S 10:47 0:00 [migration/6]
root 36 0.0 0.0 0 0 ? S 10:47 0:00 [kworker/6:0]
root 37 0.0 0.0 0 0 ? S< 10:47 0:00 [kworker/6:0H]
root 38 0.0 0.0 0 0 ? S 10:47 0:00 [ksoftirqd/6]
root 39 0.0 0.0 0 0 ? S 10:47 0:00 [watchdog/6]
root 40 0.0 0.0 0 0 ? S 10:47 0:00 [migration/7]
root 41 0.0 0.0 0 0 ? S 10:47 0:00 [kworker/7:0]
root 42 0.0 0.0 0 0 ? S< 10:47 0:00 [kworker/7:0H]
root 43 0.0 0.0 0 0 ? S 10:47 0:00 [ksoftirqd/7]
root 44 0.0 0.0 0 0 ? S 10:47 0:00 [watchdog/7]
root 45 0.0 0.0 0 0 ? S 10:47 0:00 [migration/8]
root 46 0.0 0.0 0 0 ? S 10:47 0:00 [kworker/8:0]
root 47 0.0 0.0 0 0 ? S< 10:47 0:00 [kworker/8:0H]
root 48 0.0 0.0 0 0 ? S 10:47 0:00 [ksoftirqd/8]
root 49 0.0 0.0 0 0 ? S 10:47 0:00 [watchdog/8]
root 50 0.0 0.0 0 0 ? S 10:47 0:00 [migration/9]
root 51 0.0 0.0 0 0 ? S 10:47 0:00 [kworker/9:0]
root 52 0.0 0.0 0 0 ? S< 10:47 0:00 [kworker/9:0H]
root 53 0.0 0.0 0 0 ? S 10:47 0:00 [ksoftirqd/9]
root 54 0.0 0.0 0 0 ? S 10:47 0:00 [watchdog/9]
root 55 0.0 0.0 0 0 ? S 10:47 0:00 [migration/10]
root 56 0.0 0.0 0 0 ? S 10:47 0:00 [kworker/10:0]
root 57 0.0 0.0 0 0 ? S< 10:47 0:00 [kworker/10:0H]
root 58 0.0 0.0 0 0 ? S 10:47 0:00 [ksoftirqd/10]
root 59 0.0 0.0 0 0 ? S 10:47 0:00 [watchdog/10]
root 60 0.0 0.0 0 0 ? S 10:47 0:00 [migration/11]
root 61 0.0 0.0 0 0 ? S 10:47 0:00 [kworker/11:0]
root 62 0.0 0.0 0 0 ? S< 10:47 0:00 [kworker/11:0H]
root 63 0.0 0.0 0 0 ? S 10:47 0:00 [ksoftirqd/11]
root 64 0.0 0.0 0 0 ? S 10:47 0:00 [watchdog/11]
root 65 0.0 0.0 0 0 ? S 10:47 0:00 [migration/12]
root 66 0.0 0.0 0 0 ? S 10:47 0:00 [kworker/12:0]
root 67 0.0 0.0 0 0 ? S< 10:47 0:00 [kworker/12:0H]
root 68 0.0 0.0 0 0 ? S 10:47 0:00 [ksoftirqd/12]
root 69 0.0 0.0 0 0 ? S 10:47 0:00 [watchdog/12]
root 70 0.0 0.0 0 0 ? S 10:47 0:00 [migration/13]
root 71 0.0 0.0 0 0 ? S 10:47 0:00 [kworker/13:0]
root 72 0.0 0.0 0 0 ? S< 10:47 0:00 [kworker/13:0H]
root 73 0.0 0.0 0 0 ? S 10:47 0:00 [ksoftirqd/13]
root 74 0.1 0.0 0 0 ? S 10:47 0:00 [watchdog/13]
root 75 0.0 0.0 0 0 ? S 10:47 0:00 [migration/14]
root 76 0.0 0.0 0 0 ? S 10:47 0:00 [kworker/14:0]
root 77 0.0 0.0 0 0 ? S< 10:47 0:00 [kworker/14:0H]
root 78 0.0 0.0 0 0 ? S 10:47 0:00 [ksoftirqd/14]
root 79 0.0 0.0 0 0 ? S 10:47 0:00 [watchdog/14]
root 80 0.0 0.0 0 0 ? S 10:47 0:00 [migration/15]
root 81 0.0 0.0 0 0 ? S 10:47 0:00 [kworker/15:0]
root 82 0.0 0.0 0 0 ? S< 10:47 0:00 [kworker/15:0H]
root 83 0.0 0.0 0 0 ? S 10:47 0:00 [ksoftirqd/15]
root 84 0.0 0.0 0 0 ? S 10:47 0:00 [watchdog/15]
root 85 0.0 0.0 0 0 ? S< 10:47 0:00 [cpuset]
root 86 0.0 0.0 0 0 ? S< 10:47 0:00 [khelper]
root 87 0.0 0.0 0 0 ? S 10:47 0:00 [kdevtmpfs]
root 88 0.0 0.0 0 0 ? S< 10:47 0:00 [netns]
root 89 0.0 0.0 0 0 ? S 10:47 0:00 [bdi-default]
root 90 0.0 0.0 0 0 ? S< 10:47 0:00 [kintegrityd]
root 91 0.0 0.0 0 0 ? S< 10:47 0:00 [kblockd]
root 92 0.0 0.0 0 0 ? S 10:48 0:00 [kworker/13:1]
root 93 0.0 0.0 0 0 ? S 10:48 0:00 [irq/40-AMD-Vi]
root 94 0.0 0.0 0 0 ? S 10:48 0:00 [khungtaskd]
root 95 0.0 0.0 0 0 ? S 10:48 0:00 [kswapd0]
root 96 0.0 0.0 0 0 ? S 10:48 0:00 [kswapd1]
root 97 0.0 0.0 0 0 ? S 10:48 0:00 [kswapd2]
root 98 0.0 0.0 0 0 ? S 10:48 0:00 [kswapd3]
root 99 0.0 0.0 0 0 ? SN 10:48 0:00 [ksmd]
root 100 0.0 0.0 0 0 ? SN 10:48 0:00 [khugepaged]
root 101 0.0 0.0 0 0 ? S 10:48 0:00 [fsnotify_mark]
root 102 0.0 0.0 0 0 ? S< 10:48 0:00 [crypto]
root 109 0.0 0.0 0 0 ? S< 10:48 0:00 [kthrotld]
root 110 2.7 0.0 0 0 ? S 10:48 0:07 [kworker/0:1]
root 111 0.0 0.0 0 0 ? S 10:48 0:00 [kworker/14:1]
root 112 0.0 0.0 0 0 ? S 10:48 0:00 [kworker/15:1]
root 113 0.0 0.0 0 0 ? S 10:48 0:00 [kworker/12:1]
root 114 0.0 0.0 0 0 ? S 10:48 0:00 [kworker/5:1]
root 115 0.0 0.0 0 0 ? S 10:48 0:00 [kworker/1:1]
root 116 0.0 0.0 0 0 ? S 10:48 0:00 [kworker/2:1]
root 117 0.0 0.0 0 0 ? S 10:48 0:00 [kworker/3:1]
root 118 0.0 0.0 0 0 ? S< 10:48 0:00 [deferwq]
root 119 0.0 0.0 0 0 ? S 10:48 0:00 [kworker/u:1]
root 120 0.0 0.0 0 0 ? S< 10:48 0:00 [charger_manage
root 121 0.0 0.0 0 0 ? S< 10:48 0:00 [devfreq_wq]
root 122 0.0 0.0 0 0 ? S 10:48 0:00 [kworker/4:1]
root 125 0.5 0.0 0 0 ? S 10:48 0:01 [kworker/9:1]
root 137 0.0 0.0 0 0 ? S 10:48 0:00 [kworker/6:1]
root 138 0.0 0.0 0 0 ? S 10:48 0:00 [kworker/10:1]
root 141 0.0 0.0 0 0 ? S 10:48 0:00 [kworker/11:1]
root 143 0.0 0.0 0 0 ? S 10:48 0:00 [khubd]
root 145 0.0 0.0 0 0 ? S 10:48 0:00 [kworker/0:2]
root 146 0.0 0.0 0 0 ? S 10:48 0:00 [kworker/7:1]
root 147 0.0 0.0 0 0 ? S 10:48 0:00 [kworker/8:1]
root 153 0.0 0.0 0 0 ? S< 10:48 0:00 [ata_sff]
root 156 0.1 0.0 0 0 ? S 10:48 0:00 [scsi_eh_0]
root 157 0.0 0.0 0 0 ? S 10:48 0:00 [scsi_eh_1]
root 158 0.0 0.0 0 0 ? S 10:48 0:00 [scsi_eh_2]
root 159 0.1 0.0 0 0 ? S 10:48 0:00 [scsi_eh_3]
root 160 0.0 0.0 0 0 ? S 10:48 0:00 [kworker/u:2]
root 161 0.6 0.0 0 0 ? S 10:48 0:01 [kworker/u:3]
root 162 0.0 0.0 0 0 ? S 10:48 0:00 [kworker/u:4]
root 163 0.0 0.0 0 0 ? S 10:48 0:00 [kworker/u:5]
root 165 0.0 0.0 0 0 ? S< 10:48 0:00 [md]
root 169 0.2 0.0 0 0 ? S< 10:48 0:00 [kworker/0:1H]
root 170 0.0 0.0 0 0 ? S< 10:48 0:00 [kworker/10:1H]
root 171 0.0 0.0 0 0 ? S< 10:48 0:00 [kworker/7:1H]
root 172 0.0 0.0 0 0 ? S< 10:48 0:00 [kworker/4:1H]
root 200 0.0 0.0 0 0 ? S 10:48 0:00 [kworker/1:2]
root 209 0.0 0.0 0 0 ? S< 10:48 0:00 [kworker/11:1H]
root 210 0.0 0.0 0 0 ? S< 10:48 0:00 [kworker/13:1H]
root 211 0.0 0.0 0 0 ? S< 10:48 0:00 [kworker/14:1H]
root 212 0.0 0.0 0 0 ? S< 10:48 0:00 [kworker/12:1H]
root 213 0.0 0.0 0 0 ? S< 10:48 0:00 [kworker/15:1H]
root 236 0.0 0.0 0 0 ? S< 10:48 0:00 [kworker/6:1H]
root 237 0.0 0.0 0 0 ? S< 10:48 0:00 [kworker/5:1H]
root 284 0.0 0.0 0 0 ? S 10:48 0:00 [md0_raid1]
root 308 0.0 0.0 0 0 ? S< 10:48 0:00 [kworker/9:1H]
root 309 0.0 0.0 0 0 ? S 10:48 0:00 [kjournald]
root 661 0.0 0.0 0 0 ? S< 10:48 0:00 [kworker/8:1H]
root 664 0.0 0.0 0 0 ? S< 10:48 0:00 [kworker/1:1H]
root 667 0.0 0.0 0 0 ? S< 10:48 0:00 [edac-poller]
root 672 0.0 0.0 0 0 ? S< 10:48 0:00 [kvm-irqfd-clea
root 674 0.0 0.0 0 0 ? S 10:48 0:00 [kworker/0:3]
root 675 0.0 0.0 0 0 ? S< 10:48 0:00 [ttm_swap]
root 677 0.0 0.0 0 0 ? S 10:48 0:00 [kworker/12:2]
root 678 0.0 0.0 0 0 ? S 10:48 0:00 [flush-9:0]
root 690 0.0 0.0 0 0 ? S 10:48 0:00 [kauditd]
root 816 0.0 0.0 0 0 ? S 10:48 0:00 [kworker/4:2]
root 819 0.0 0.0 0 0 ? S 10:48 0:00 [kworker/6:2]
root 827 0.0 0.0 0 0 ? S 10:48 0:00 [kworker/5:2]
root 828 0.0 0.0 0 0 ? S 10:48 0:00 [kworker/7:2]
root 866 0.7 0.0 0 0 ? S 10:48 0:01 [jbd2/md1-8]
root 867 0.0 0.0 0 0 ? S< 10:48 0:00 [ext4-dio-unwri
root 868 0.0 0.0 0 0 ? S 10:48 0:00 [jbd2/md2-8]
root 869 0.0 0.0 0 0 ? S< 10:48 0:00 [ext4-dio-unwri
root 1291 4.6 0.0 0 0 ? S 10:48 0:10 [flush-9:1]
root 1292 0.0 0.0 0 0 ? S 10:48 0:00 [flush-9:2]
root 1371 0.0 0.0 0 0 ? SN 10:48 0:00 [kipmi0]
root 1394 0.0 0.0 0 0 ? S< 10:48 0:00 [rpciod]
root 1460 0.0 0.0 0 0 ? S 10:48 0:00 [lockd]
root 1741 0.0 0.0 0 0 ? S< 10:48 0:00 [kworker/3:1H]
root 1983 0.0 0.0 0 0 ? S< 10:48 0:00 [nfsiod]
2000 1999 0.2 0.0 0 0 ? D 10:48 0:00 [Xvnc]
2000 2004 0.0 0.0 0 0 ? D 10:48 0:00 [vncconfig]
2000 2007 0.0 0.0 0 0 ? D 10:48 0:00 [sh]
2000 2013 0.0 0.0 0 0 ? D 10:48 0:00 [xfce4-session]
2000 2025 0.0 0.0 320208 11372 ? D 10:48 0:00 xfwm4
2000 2029 0.0 0.0 500000 13700 ? D 10:48 0:00 xfce4-panel
2000 2031 0.0 0.0 0 0 ? D 10:48 0:00 [Thunar]
2000 2033 0.0 0.0 495880 14220 ? D 10:48 0:00 xfdesktop
2000 2041 0.0 0.0 402676 7660 ? Ds 10:48 0:00 xfsettingsd
2000 2061 0.0 0.0 0 0 ? D 10:48 0:00 [panel-6-systra
2000 2063 0.0 0.0 0 0 ? D 10:48 0:00 [panel-2-action
root 2066 0.0 0.0 0 0 ? S< 10:48 0:00 [kworker/2:1H]
root 2081 0.0 0.0 0 0 ? S 10:48 0:00 [flush-0:29]
root 2194 0.0 0.0 0 0 ? S 10:48 0:00 [kworker/3:2]
root 2248 0.0 0.0 0 0 ? S 10:49 0:00 [nfsv4.0-svc]
997 2300 0.4 0.0 0 0 ? D 10:49 0:00 [wish]
997 3891 0.0 0.0 0 0 ? Z 10:50 0:00 [slave_odra_d_0
root 4156 0.0 0.0 0 0 ? S 10:51 0:00 [flush-0:32]
root 4159 0.1 0.0 4796 1128 ? Ss 10:51 0:00 /bin/sh /etc/rc
root 4524 0.0 0.0 0 0 ? S 10:52 0:00 [kworker/10:2]
root 4726 0.0 0.0 13084 1056 ? R 10:52 0:00 ps aux
+ halt -w
+ is_yes no
+ [ 1 = 0 ]
+ return 1
+ run_cmd Turning off swap swapoff -a
+ typeset force_err=0 exit_code=0 errors user
+ [ 3 -gt 0 ]
+ break
+ typeset message=Turning off swap
+ shift
+ show Turning off swap
+ typeset text len
+ is_no no
+ [ 1 = 0 ]
+ return 0
+ is_yes yes
+ [ 1 = 0 ]
+ return 0
+ nls -n Turning off swap
+ typeset msg_echo nls_domain text message
+ msg_echo=\n
+ nls_domain=
+ [ -n != n ]
+ msg_echo=
+ shift
+ [ Turning off swap != Turning off swap ]
+ message=Turning off swap
+ shift
+ [ -z Turning off swap ]
+ is_yes yes
+ [ 1 = 0 ]
+ return 0
+ gettext -e --domain=rc-scripts Turning off swap
+ TEXTDOMAINDIR=/etc/sysconfig/locale
+ message=Turning off swap
+ printf Turning off swap
+ echo -en
+ text=Turning off swap
+ len=16
+ [ 16 -lt 67 ]
+ text=Turning off swap.
+ [ 17 -lt 67 ]
+ text=Turning off swap..
+ [ 18 -lt 67 ]
+ text=Turning off swap...
(...)
+ text=Turning off swap..................................................
+ [ 66 -lt 67 ]
+ text=Turning off swap...................................................
+ [ 67 -lt 67 ]
+ [ -n ]
+ echo -n Turning off swap...................................................
Turning off swap...................................................+ busy
+ echo -n [ BUSY ]
[ BUSY ]+ cd /
+ export HOME=/tmp TMPDIR=/tmp
+ is_no no
+ [ 1 = 0 ]
+ return 0
+ swapoff -a
+ 2>&1
+ errors=
+ ok
+ echo [ DONE ]
[ DONE ]
+ log_success swapoff Turning off swap
+ :
+ [ -n ]
+ return 0
+ [ -x /etc/rc.d/rc.acct ]
+ [ -x /sbin/quotaoff ]
+ cat /sys/kernel/kexec_loaded
+ 2>/dev/null
+ kexec_loaded=0
+ sig=-15
+ retry=3
+ force=
+ [ -z ]
+ UMOUNT_IGNORE=/
+ awk -v ig=^(/)$ BEGIN { gsub(/[\t ]+/, "|", ig); } \
$3 !~ /^(proc|loopfs|devpts|devtmpfs|shm|iso9660|ramfs|tmpfs|sysfs|securityfs|squashfs)$/ \
&& $1 !~ /^(none|\/dev\/root)$/ \
&& $2 !~ ig {print $2} /proc/mounts
+ remaining=/home/atest
/home/users
/proc/fs/nfsd
/var/lib/nfs/rpc_pipefs
/remote/dragon
/remote/nexus
+ [ -n /home/atest
/home/users
/proc/fs/nfsd
/var/lib/nfs/rpc_pipefs
/remote/dragon
/remote/nexus -a 3 -gt 0 ]
+ show Unmounting file systems
+ typeset text len
+ is_no no
+ [ 1 = 0 ]
+ return 0
+ is_yes yes
+ [ 1 = 0 ]
+ return 0
+ nls -n Unmounting file systems
+ typeset msg_echo nls_domain text message
+ msg_echo=\n
+ nls_domain=
+ [ -n != n ]
+ msg_echo=
+ shift
+ [ Unmounting file systems != Unmounting file systems ]
+ message=Unmounting file systems
+ shift
+ [ -z Unmounting file systems ]
+ is_yes yes
+ [ 1 = 0 ]
+ return 0
+ gettext -e --domain=rc-scripts Unmounting file systems
+ TEXTDOMAINDIR=/etc/sysconfig/locale
+ message=Unmounting file systems
+ printf Unmounting file systems
+ echo -en
+ text=Unmounting file systems
+ len=23
+ [ 23 -lt 67 ]
+ text=Unmounting file systems.
+ [ 24 -lt 67 ]
+ text=Unmounting file systems..
(...)
+ text=Unmounting file systems............................................
+ [ 67 -lt 67 ]
+ [ -n ]
+ echo -n Unmounting file systems............................................
Unmounting file systems............................................+ busy
+ echo -n [ BUSY ]
[ BUSY ]+ umount -a -t noproc
+ 2>&1
+ ERRORS=umount: /home/atest: target is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
umount: /dev: target is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
+ rc=32
+ is_yes yes
+ [ 1 = 0 ]
+ return 0
+ TPUT=
+ rc_gettext_init
+ [ -z yes ]
+ [ -z ]
+ [ -d /usr/share/terminfo ]
+ [ -x /usr/bin/tput -o -x /bin/tput ]
+ TPUT=yes
+ tput longname
+ >/dev/null
+ 2>&1
+ [ 32 = 0 ]
+ fail
+ echo [ FAIL ]
[ FAIL ]
+ return 1
+ [ -n umount: /home/atest: target is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
umount: /dev: target is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1)) ]
+ echo umount: /home/atest: target is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
umount: /dev: target is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
umount: /home/atest: target is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
umount: /dev: target is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
+ sleep 2
+ awk -v ig=^(/)$ BEGIN { gsub(/[\t ]+/, "|", ig); } \
$3 !~ /^(proc|loopfs|devpts|devtmpfs|shm|iso9660|ramfs|tmpfs|sysfs|securityfs|squashfs)$/ \
&& $1 !~ /^(none|\/dev\/root)$/ \
&& $2 !~ ig {print $2} /proc/mounts
+ remaining=/home/atest
+ [ -z /home/atest ]
+ fuser -k -m -15 /home/atest
+ >/dev/null
Cannot stat file /proc/2025/fd/1: Connection timed out
Cannot stat file /proc/2025/fd/2: Connection timed out
Cannot stat file /proc/2029/fd/1: Connection timed out
Cannot stat file /proc/2029/fd/2: Connection timed out
Cannot stat file /proc/2033/fd/1: Connection timed out
Cannot stat file /proc/2033/fd/2: Connection timed out
Cannot stat file /proc/2041/fd/1: Connection timed out
Cannot stat file /proc/2041/fd/2: Connection timed out
+ sleep 5
+ retry=2
+ sig=-9
+ force=-f
+ [ -n /home/atest -a 2 -gt 0 ]
+ show Unmounting file systems
+ typeset text len
+ is_no no
+ [ 1 = 0 ]
+ return 0
+ is_yes yes
+ [ 1 = 0 ]
+ return 0
+ nls -n Unmounting file systems
+ typeset msg_echo nls_domain text message
+ msg_echo=\n
+ nls_domain=
+ [ -n != n ]
+ msg_echo=
+ shift
+ [ Unmounting file systems != Unmounting file systems ]
+ message=Unmounting file systems
+ shift
+ [ -z Unmounting file systems ]
+ is_yes yes
+ [ 1 = 0 ]
+ return 0
+ gettext -e --domain=rc-scripts Unmounting file systems
+ TEXTDOMAINDIR=/etc/sysconfig/locale
+ message=Unmounting file systems
+ printf Unmounting file systems
+ echo -en
+ text=Unmounting file systems
+ len=23
+ [ 23 -lt 67 ]
+ text=Unmounting file systems.
+ [ 24 -lt 67 ]
+ text=Unmounting file systems..
(...)
+ text=Unmounting file systems............................................
+ [ 67 -lt 67 ]
+ [ -n ]
+ echo -n Unmounting file systems............................................
Unmounting file systems............................................+ busy
+ echo -n [ BUSY ]
[ BUSY ]+ umount -a -f -t noproc
+ 2>&1
+ ERRORS=umount: /home/atest: target is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
umount: /dev: target is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
+ rc=32
+ is_yes yes
+ [ 1 = 0 ]
+ return 0
+ TPUT=
+ rc_gettext_init
+ [ -z yes ]
+ [ -z ]
+ [ -d /usr/share/terminfo ]
+ [ -x /usr/bin/tput -o -x /bin/tput ]
+ TPUT=yes
+ tput longname
+ >/dev/null
+ 2>&1
+ [ 32 = 0 ]
+ fail
+ echo [ FAIL ]
[ FAIL ]
+ return 1
+ [ -n umount: /home/atest: target is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
umount: /dev: target is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1)) ]
+ echo umount: /home/atest: target is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
umount: /dev: target is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
umount: /home/atest: target is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
umount: /dev: target is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
+ sleep 2
+ awk -v ig=^(/)$ BEGIN { gsub(/[\t ]+/, "|", ig); } \
$3 !~ /^(proc|loopfs|devpts|devtmpfs|shm|iso9660|ramfs|tmpfs|sysfs|securityfs|squashfs)$/ \
&& $1 !~ /^(none|\/dev\/root)$/ \
&& $2 !~ ig {print $2} /proc/mounts
+ remaining=/home/atest
+ [ -z /home/atest ]
+ fuser -k -m -9 /home/atest
+ >/dev/null
Cannot stat file /proc/2025/fd/1: Connection timed out
Cannot stat file /proc/2025/fd/2: Connection timed out
Cannot stat file /proc/2029/fd/1: Connection timed out
Cannot stat file /proc/2029/fd/2: Connection timed out
Cannot stat file /proc/2033/fd/1: Connection timed out
Cannot stat file /proc/2033/fd/2: Connection timed out
Cannot stat file /proc/2041/fd/1: Connection timed out
Cannot stat file /proc/2041/fd/2: Connection timed out
+ sleep 5
+ retry=1
+ sig=-9
+ force=-f
+ [ -n /home/atest -a 1 -gt 0 ]
+ show Unmounting file systems
+ typeset text len
+ is_no no
+ [ 1 = 0 ]
+ return 0
+ is_yes yes
+ [ 1 = 0 ]
+ return 0
+ nls -n Unmounting file systems
+ typeset msg_echo nls_domain text message
+ msg_echo=\n
+ nls_domain=
+ [ -n != n ]
+ msg_echo=
+ shift
+ [ Unmounting file systems != Unmounting file systems ]
+ message=Unmounting file systems
+ shift
+ [ -z Unmounting file systems ]
+ is_yes yes
+ [ 1 = 0 ]
+ return 0
+ gettext -e --domain=rc-scripts Unmounting file systems
+ TEXTDOMAINDIR=/etc/sysconfig/locale
+ message=Unmounting file systems
+ printf Unmounting file systems
+ echo -en
+ text=Unmounting file systems
+ len=23
+ [ 23 -lt 67 ]
+ text=Unmounting file systems.
+ [ 24 -lt 67 ]
+ text=Unmounting file systems..
(...)
+ [ 67 -lt 67 ]
+ [ -n ]
+ echo -n Unmounting file systems............................................
Unmounting file systems............................................+ busy
+ echo -n [ BUSY ]
[ BUSY ]+ umount -a -f -t noproc
+ 2>&1
+ ERRORS=umount: /home/atest: target is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
umount: /dev: target is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
+ rc=32
+ is_yes yes
+ [ 1 = 0 ]
+ return 0
+ TPUT=
+ rc_gettext_init
+ [ -z yes ]
+ [ -z ]
+ [ -d /usr/share/terminfo ]
+ [ -x /usr/bin/tput -o -x /bin/tput ]
+ TPUT=yes
+ tput longname
+ >/dev/null
+ 2>&1
+ [ 32 = 0 ]
+ fail
+ echo [ FAIL ]
[ FAIL ]
+ return 1
+ [ -n umount: /home/atest: target is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
umount: /dev: target is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1)) ]
+ echo umount: /home/atest: target is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
umount: /dev: target is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
umount: /home/atest: target is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
umount: /dev: target is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
+ sleep 2
+ awk -v ig=^(/)$ BEGIN { gsub(/[\t ]+/, "|", ig); } \
$3 !~ /^(proc|loopfs|devpts|devtmpfs|shm|iso9660|ramfs|tmpfs|sysfs|securityfs|squashfs)$/ \
&& $1 !~ /^(none|\/dev\/root)$/ \
&& $2 !~ ig {print $2} /proc/mounts
+ remaining=/home/atest
+ [ -z /home/atest ]
+ fuser -k -m -9 /home/atest
+ >/dev/null
Cannot stat file /proc/2025/fd/1: Connection timed out
Cannot stat file /proc/2025/fd/2: Connection timed out
Cannot stat file /proc/2029/fd/1: Connection timed out
Cannot stat file /proc/2029/fd/2: Connection timed out
Cannot stat file /proc/2033/fd/1: Connection timed out
Cannot stat file /proc/2033/fd/2: Connection timed out
+ sleep 5
+ retry=0
+ sig=-9
+ force=-f
+ [ -n /home/atest -a 0 -gt 0 ]
+ run_cmd Remounting root filesystem in ro mode mount -n -o remount,ro /
+ typeset force_err=0 exit_code=0 errors user
+ [ 6 -gt 0 ]
+ break
+ typeset message=Remounting root filesystem in ro mode
+ shift
+ show Remounting root filesystem in ro mode
+ typeset text len
+ is_no no
+ [ 1 = 0 ]
+ return 0
+ is_yes yes
+ [ 1 = 0 ]
+ return 0
+ nls -n Remounting root filesystem in ro mode
+ typeset msg_echo nls_domain text message
+ msg_echo=\n
+ nls_domain=
+ [ -n != n ]
+ msg_echo=
+ shift
+ [ Remounting root filesystem in ro mode != Remounting root filesystem in ro mode ]
+ message=Remounting root filesystem in ro mode
+ shift
+ [ -z Remounting root filesystem in ro mode ]
+ is_yes yes
+ [ 1 = 0 ]
+ return 0
+ gettext -e --domain=rc-scripts Remounting root filesystem in ro mode
+ TEXTDOMAINDIR=/etc/sysconfig/locale
+ message=Remounting root filesystem in ro mode
+ printf Remounting root filesystem in ro mode
+ echo -en
+ text=Remounting root filesystem in ro mode
+ len=37
+ [ 37 -lt 67 ]
+ text=Remounting root filesystem in ro mode.
+ [ 38 -lt 67 ]
+ text=Remounting root filesystem in ro mode..
(...)
+ text=Remounting root filesystem in ro mode..............................
+ [ 67 -lt 67 ]
+ [ -n ]
+ echo -n Remounting root filesystem in ro mode..............................
Remounting root filesystem in ro mode..............................+ busy
+ echo -n [ BUSY ]
[ BUSY ]+ cd /
+ export HOME=/tmp TMPDIR=/tmp
+ is_no no
+ [ 1 = 0 ]
+ return 0
+ mount -n -o remount,ro /
+ 2>&1
+ errors=mount: / is busy
+ fail
+ echo [ FAIL ]
[ FAIL ]
+ return 1
+ log_failed mount Remounting root filesystem in ro mode
+ :
+ exit_code=1
+ [ -n mount: / is busy ]
+ [ 1 -eq 1 -o 0 -eq 1 ]
+ echo mount: / is busy
mount: / is busy
+ return 1
+ [ -x /sbin/vgchange ]
+ /sbin/vgchange --version
+ 2>/dev/null
+ awk /LVM version:/{if ($3 >= 2) print "2"}
+ lvmversion=2
+ [ 2 = 2 ]
+ lvmsysinit=--sysinit
+ /sbin/vgchange -a n --sysinit
+ >/dev/null
+ 2>&1
+ goraidtab=1
+ [ -x /sbin/mdadm -a -f /etc/mdadm.conf ]
+ grep -qE ^([[:blank:]]|)ARRAY[[:blank:]] /etc/mdadm.conf
+ 2>/dev/null
+ /sbin/mdadm --stop --scan
+ >/dev/null
+ 2>&1
+ rc=1
+ [ 1 -eq 0 ]
+ [ -x /sbin/raidstop -a -f /etc/raidtab -a 1 -eq 1 ]
+ show Remounting remaining filesystems ro mode
+ typeset text len
+ is_no no
+ [ 1 = 0 ]
+ return 0
+ is_yes yes
+ [ 1 = 0 ]
+ return 0
+ nls -n Remounting remaining filesystems ro mode
+ typeset msg_echo nls_domain text message
+ msg_echo=\n
+ nls_domain=
+ [ -n != n ]
+ msg_echo=
+ shift
+ [ Remounting remaining filesystems ro mode != Remounting remaining filesystems ro mode ]
+ message=Remounting remaining filesystems ro mode
+ shift
+ [ -z Remounting remaining filesystems ro mode ]
+ is_yes yes
+ [ 1 = 0 ]
+ return 0
+ gettext -e --domain=rc-scripts Remounting remaining filesystems ro mode
+ TEXTDOMAINDIR=/etc/sysconfig/locale
+ message=Remounting remaining filesystems ro mode
+ printf Remounting remaining filesystems ro mode
+ echo -en
+ text=Remounting remaining filesystems ro mode
+ len=40
+ [ 40 -lt 67 ]
+ text=Remounting remaining filesystems ro mode.
(...)
+ text=Remounting remaining filesystems ro mode...........................
+ [ 67 -lt 67 ]
+ [ -n ]
+ echo -n Remounting remaining filesystems ro mode...........................
Remounting remaining filesystems ro mode...........................+ busy
+ echo -n [ BUSY ]
[ BUSY ]+ mount
+ awk /ext2|ext3|reiserfs|xfs|jfs/ { print $3 }
+ read line
+ mount -n -o ro,remount /
mount: / is busy
+ read line
+ fail
+ echo [ FAIL ]
[ FAIL ]
+ return 1
+ _rebootwhat=system
+ [ -f /etc/crypttab ]
+ is_empty_file /etc/crypttab
+ [ -s /etc/crypttab ]
+ grep -vqE ^(#|[[:blank:]]*$) /etc/crypttab
+ return 0
+ [ 6 = 0 ]
+ show Please stand by while rebooting the system
+ typeset text len
+ is_no no
+ [ 1 = 0 ]
+ return 0
+ is_yes yes
+ [ 1 = 0 ]
+ return 0
+ nls -n Please stand by while rebooting the system
+ typeset msg_echo nls_domain text message
+ msg_echo=\n
+ nls_domain=
+ [ -n != n ]
+ msg_echo=
+ shift
+ [ Please stand by while rebooting the system != Please stand by while rebooting the system ]
+ message=Please stand by while rebooting the system
+ shift
+ [ -z Please stand by while rebooting the system ]
+ is_yes yes
+ [ 1 = 0 ]
+ return 0
+ gettext -e --domain=rc-scripts Please stand by while rebooting the system
+ TEXTDOMAINDIR=/etc/sysconfig/locale
+ message=Please stand by while rebooting the system
+ printf Please stand by while rebooting the system
+ echo -en
+ text=Please stand by while rebooting the system
+ len=42
+ [ 42 -lt 67 ]
+ text=Please stand by while rebooting the system.
(...)
+ text=Please stand by while rebooting the system.........................
+ [ 67 -lt 67 ]
+ [ -n ]
+ echo -n Please stand by while rebooting the system.........................
Please stand by while rebooting the system.........................+ ok
+ echo [ DONE ]
[ DONE ]
+ [ -f /fastboot ]
+ emit stopped JOB=shutdown
+ use_upstart
+ is_yes no
+ [ 1 = 0 ]
+ return 1
+ is_no no
+ [ 1 = 0 ]
+ return 0
+ return 1
+ return 0
+ emit stopping JOB=shutdown
+ use_upstart
+ is_yes no
+ [ 1 = 0 ]
+ return 1
+ is_no no
+ [ 1 = 0 ]
+ return 0
+ return 1
+ return 0
+ [ -x /sbin/kexec ]
+ [ 3 != unknown ]
+ eval reboot -d -f
+ reboot -d -f
[ 493.548872] Kernel panic - not syncing: hung_task: blocked tasks
[ 493.620619] Pid: 94, comm: khungtaskd Tainted: G W 3.6.6 #15
[ 493.698572] Call Trace:
[ 493.727745] [<ffffffff8149d2a8>] panic+0xbc/0x1cb
[ 493.784938] [<ffffffff810c6eb5>] watchdog+0x245/0x250
[ 493.846286] [<ffffffff810c6c70>] ? hung_task_panic+0x20/0x20
[ 493.914898] [<ffffffff810727ce>] kthread+0x8e/0xa0
[ 493.973132] [<ffffffff814afa04>] kernel_thread_helper+0x4/0x10
[ 494.043821] [<ffffffff81072740>] ? kthread_freezable_should_stop+0x70/0x70
[ 494.126966] [<ffffffff814afa00>] ? gs_change+0x13/0x13
[ 494.189390] panic occurred, switching back to text console
[ 494.254971] ------------[ cut here ]------------
[ 494.310074] WARNING: at arch/x86/kernel/smp.c:123 native_smp_send_reschedule+0x53/0x60()
[ 494.406710] Hardware name: H8DGU
[ 494.445221] Modules linked in: nfsv4 fuse nfsv3 nfs fscache nfsd lockd nfs_acl auth_rpcgss sunrpc ipmi_si ipmi_devintf ipmi_msghandler sch_sfq iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter xt_TCPMSS xt_tcpudp iptable_mangle ip_tables ip6table_filter ip6_tables x_tables ext4 jbd2 crc16 raid0 dm_mod autofs4 dummy ide_cd_mod cdrom ata_generic pata_acpi sp5100_tco pata_atiixp ide_pci_generic mgag200 ttm drm_kms_helper drm igb powernow_k8 freq_table mperf joydev kvm_amd i2c_algo_bit sysimgblt sysfillrect syscopyarea dca ptp hid_generic kvm pps_core k10temp microcode i2c_piix4 evdev pcspkr hwmon amd64_edac_mod edac_core atiixp ide_core edac_mce_amd i2c_core processor button ext3 jbd mbcache sd_mod crc_t10dif raid1 md_mod ahci libahci libata scsi_mod usbhid hid ohci_hcd ehci_hcd usbcore usb_common
[ 495.326812] Pid: 94, comm: khungtaskd Tainted: G W 3.6.6 #15
[ 495.404768] Call Trace:
[ 495.433938] <IRQ> [<ffffffff810500da>] warn_slowpath_common+0x7a/0xb0
[ 495.513036] [<ffffffff81050125>] warn_slowpath_null+0x15/0x20
[ 495.582688] [<ffffffff81034d73>] native_smp_send_reschedule+0x53/0x60
[ 495.660644] [<ffffffff81089fcc>] trigger_load_balance+0x17c/0x220
[ 495.734446] [<ffffffff81081ae4>] scheduler_tick+0x104/0x140
[ 495.802022] [<ffffffff810611d7>] update_process_times+0x67/0x80
[ 495.873749] [<ffffffff8109e579>] tick_sched_timer+0x69/0xe0
[ 495.941324] [<ffffffff810769ce>] __run_hrtimer+0x6e/0x1c0
[ 496.006822] [<ffffffff8109e510>] ? tick_nohz_handler+0x100/0x100
[ 496.079589] [<ffffffff81058a15>] ? __do_softirq+0x115/0x1f0
[ 496.147163] [<ffffffff810772af>] hrtimer_interrupt+0xef/0x220
[ 496.216814] [<ffffffff81035b34>] smp_apic_timer_interrupt+0x64/0xa0
[ 496.292694] [<ffffffff814af40a>] apic_timer_interrupt+0x6a/0x70
[ 496.364420] <EOI> [<ffffffff8149d373>] ? panic+0x187/0x1cb
[ 496.432100] [<ffffffff810c6eb5>] watchdog+0x245/0x250
[ 496.493445] [<ffffffff810c6c70>] ? hung_task_panic+0x20/0x20
[ 496.562059] [<ffffffff810727ce>] kthread+0x8e/0xa0
[ 496.620293] [<ffffffff814afa04>] kernel_thread_helper+0x4/0x10
[ 496.690981] [<ffffffff81072740>] ? kthread_freezable_should_stop+0x70/0x70
[ 496.774126] [<ffffffff814afa00>] ? gs_change+0x13/0x13
[ 496.836511] ---[ end trace f42d1c6fc3b40f64 ]---