[5.15-rc1][PPC/PCI] IPR adapter reset threw EEH and fail to recover while booting mainline on powerpc

From: Abdul Haleem
Date: Mon Sep 20 2021 - 02:00:58 EST

Greeting's

IPR adapter reset triggers EEH errors and recovered 6 times and than task hung on today's mainline kernel on powerpc box with loads of hung task call traces

The problem is not seen with 5.14-rc7 (good)  and is last recreated on 5.15-rc1 (bad) kernel

Console logs
EEH: Recovering PHB#134-PE#800000
EEH: PE location: N/A, PHB location: N/A
EEH: Frozen PHB#134-PE#800000 detected
EEH: Call Trace:
EEH: [c000000000054f90] __eeh_send_failure_event+0x70/0x150
EEH: [c00000000004e148] eeh_dev_check_failure+0x2e8/0x680
EEH: [c00000000004e5a8] eeh_check_failure+0xc8/0x100
EEH: [c008000000501c08] eeh_readl+0x70/0xa0 [ipr]
EEH: [c00800000050e76c] ipr_reset_enable_ioa+0x474/0x5a0 [ipr]
EEH: [c0080000005049a0] ipr_reset_ioa_job+0x78/0x150 [ipr]
EEH: [c008000000508f04] ipr_pci_slot_reset+0xac/0x140 [ipr]
EEH: [c000000000052ce8] eeh_report_reset+0xb8/0xe0
EEH: [c000000000054834] eeh_pe_report+0x254/0x4b4
EEH: [c000000000053efc] eeh_handle_normal_event+0x9dc/0xa80
EEH: [c000000000054de8] eeh_event_handler+0x108/0x190
EEH: [c00000000018151c] kthread+0x18c/0x1a0
EEH: [c00000000000cfd4] ret_from_kernel_thread+0x5c/0x64
EEH: PHB#134-PE#800000 has failed 6 times in the last hour and has been permanently disabled.
EEH: Unable to recover from failure from PHB#134-PE#800000.
Please try reseating or replacing it
EEH: of node=0134:80:00.0
EEH: PCI device/vendor: 034a1014
EEH: PCI cmd/status register: 00100546
EEH: PCI-E capabilities and status follow:
EEH: PCI-E 00: 0002a410 00008fc2 0000585e 0003f503
EEH: PCI-E 10: 10830000 00000000 00000000 00000000
EEH: PCI-E 20: 00000000
EEH: PCI-E AER capability register set follows:
EEH: PCI-E AER 00: 18820001 00000000 00400000 00462030
EEH: PCI-E AER 10: 00000000 0000e000 000001e0 00000000
EEH: PCI-E AER 20: 00000000 00000000 00000000 00000000
EEH: PCI-E AER 30: 00000000 00000000
EEH: Beginning: 'error_detected(permanent failure)'
PCI 0134:80:00.0#800000: EEH: not actionable (1,1,1)
[FAILED] Failed to start udev Wait for Complete Device Initialization.
See 'systemctl status systemd-udev-settle.service' for details.
         Starting Device-Mapper Multipath Device Controller...
[  OK  ] Started Device-Mapper Multipath Device Controller.
[  OK  ] Reached target Local File Systems (Pre).
[  OK  ] Reached target Local File Systems.
         Starting Create Volatile Files and Directories...
[  OK  ] Started Create Volatile Files and Directories.
[  OK  ] Reached target System Initialization.
[  OK  ] Reached target Basic System.
EEH: Finished:'error_detected(permanent failure)'
device-mapper: multipath service-time: version 0.3.0 loaded
INFO: task eehd:137 blocked for more than 122 seconds.
      Not tainted 5.15.0-rc1-autotest #3
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:eehd            state:D stack:    0 pid:  137 ppid:     2 flags:0x00000800
Call Trace:
[c00000000d167650] [c0000000001c329c] __wake_up_common+0xcc/0x290 (unreliable)
[c00000000d167840] [c00000000001e6c8] __switch_to+0x278/0x490
[c00000000d1678a0] [c000000000c8b9bc] __schedule+0x31c/0xa20
[c00000000d167960] [c000000000c8c128] schedule+0x68/0x120
[c00000000d167990] [c00800000050a3c4] __ipr_remove+0x10c/0x4b0 [ipr]
[c00000000d167a60] [c00800000050a828] ipr_remove+0xc0/0x110 [ipr]
[c00000000d167ae0] [c000000000732bd4] pci_device_remove+0x64/0x120
[c00000000d167b20] [c000000000811318] device_release_driver_internal+0x168/0x2c0
[c00000000d167b60] [c000000000724618] pci_stop_bus_device+0xa8/0x100
[c00000000d167ba0] [c000000000724810] pci_stop_and_remove_bus_device+0x20/0x40
[c00000000d167bd0] [c000000000075e60] pci_hp_remove_devices+0x90/0x130
[c00000000d167c60] [c000000000053cd8] eeh_handle_normal_event+0x7b8/0xa80
[c00000000d167d40] [c000000000054de8] eeh_event_handler+0x108/0x190
[c00000000d167da0] [c00000000018151c] kthread+0x18c/0x1a0
[c00000000d167e10] [c00000000000cfd4] ret_from_kernel_thread+0x5c/0x64
INFO: task systemd-udevd:473 blocked for more than 122 seconds.
      Not tainted 5.15.0-rc1-autotest #3

--
Regard's

Abdul Haleem
IBM Linux Technology Center



OF stdout device is: /vdevice/vty@30000000
Preparing to boot Linux version 5.15.0-rc1-autotest (root@ltcd11) (gcc (GCC) 8.4.1 20200928 (Red Hat 8.4.1-1), GNU ld version 2.30-93.el8) #5 SMP Mon Sep 13 10:16:56 EDT 2021
Detected machine type: 0000000000000101
command line: BOOT_IMAGE=/vmlinuz-5.15.0-rc1-autotest root=/dev/mapper/rhel_ltcd11-root ro crashkernel=auto rd.lvm.lv=rhel_ltcden11/root rd.lvm.lv=rhel_ltcden11-lp3/swap biosdevname=0 ipr.debug=1
Max number of cores passed to firmware: 256 (NR_CPUS = 2048)
Calling ibm,client-architecture-support... done
memory layout at init:
memory_limit : 0000000000000000 (16 MB aligned)
alloc_bottom : 0000000010560000
alloc_top : 0000000020000000
alloc_top_hi : 0000000020000000
rmo_top : 0000000020000000
ram_top : 0000000020000000
instantiating rtas at 0x000000001ec30000... done
prom_hold_cpus: skipped
copying OF device tree...
Building dt strings...
Building dt structure...
Device tree strings 0x0000000010570000 -> 0x0000000010571aaa
Device tree struct 0x0000000010580000 -> 0x0000000010590000
Quiescing Open Firmware ...
Booting Linux via __start() @ 0x000000000a710000 ...
[ 0.000000] crashkernel: memory value expected
[ 0.000000] radix-mmu: Page sizes from device-tree:
[ 0.000000] radix-mmu: Page size shift = 12 AP=0x0
[ 0.000000] radix-mmu: Page size shift = 16 AP=0x5
[ 0.000000] radix-mmu: Page size shift = 21 AP=0x1
[ 0.000000] radix-mmu: Page size shift = 30 AP=0x2
[ 0.000000] Activating Kernel Userspace Execution Prevention
[ 0.000000] Activating Kernel Userspace Access Prevention
[ 0.000000] radix-mmu: Mapped 0x0000000000000000-0x0000000001600000 with 2.00 MiB pages (exec)
[ 0.000000] radix-mmu: Mapped 0x0000000001600000-0x0000000c80000000 with 2.00 MiB pages
[ 0.000000] lpar: Using radix MMU under hypervisor
[ 0.000000] Linux version 5.15.0-rc1-autotest (root@ltcd11) (gcc (GCC) 8.4.1 20200928 (Red Hat 8.4.1-1), GNU ld version 2.30-93.el8) #5 SMP Mon Sep 13 10:16:56 EDT 2021
[ 0.000000] Found initrd at 0xc00000000c500000:0xc000000010558bc4
[ 0.000000] Using pSeries machine description
[ 0.000000] printk: bootconsole [udbg0] enabled
[ 0.000000] Partition configured for 24 cpus.
[ 0.000000] CPU maps initialized for 8 threads per core
[ 0.000000] numa: Partition configured for 32 NUMA nodes.
[ 0.000000] -----------------------------------------------------
[ 0.000000] phys_mem_size = 0xc80000000
[ 0.000000] dcache_bsize = 0x80
[ 0.000000] icache_bsize = 0x80
[ 0.000000] cpu_features = 0x000c00eb8f5f9187
[ 0.000000] possible = 0x000ffbfbcf5fb187
[ 0.000000] always = 0x0000000380008181
[ 0.000000] cpu_user_features = 0xdc0065c2 0xaef60000
[ 0.000000] mmu_features = 0x3c007641
[ 0.000000] firmware_features = 0x0000019fc45bfc57
[ 0.000000] vmalloc start = 0xc008000000000000
[ 0.000000] IO start = 0xc00a000000000000
[ 0.000000] vmemmap start = 0xc00c000000000000
[ 0.000000] -----------------------------------------------------
[ 0.000000] numa: NODE_DATA [mem 0xc7fafb380-0xc7fb003ff]
[ 0.000000] rfi-flush: fallback displacement flush available
[ 0.000000] count-cache-flush: hardware flush enabled.
[ 0.000000] link-stack-flush: software flush enabled.
[ 0.000000] stf-barrier: eieio barrier available
[ 0.000000] lpar: H_BLOCK_REMOVE supports base psize:0 psize:0 block size:8
[ 0.000000] PPC64 nvram contains 15360 bytes
[ 0.000000] barrier-nospec: using ORI speculation barrier
[ 0.000000] Zone ranges:
[ 0.000000] Normal [mem 0x0000000000000000-0x0000000c7fffffff]
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000000000000-0x0000000c7fffffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000000c7fffffff]
[ 0.000000] percpu: Embedded 10 pages/cpu s590360 r0 d65000 u655360
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 818400
[ 0.000000] Policy zone: Normal
[ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-5.15.0-rc1-autotest root=/dev/mapper/rhel_ltcden11--lp3-root ro crashkernel=auto rd.lvm.lv=rhel_ltcden11-lp3/root rd.lvm.lv=rhel_ltcden11-lp3/swap biosdevname=0 ipr.debug=1
[ 0.000000] Unknown command line parameters: BOOT_IMAGE=/vmlinuz-5.15.0-rc1-autotest crashkernel=auto biosdevname=0
[ 0.000000] Dentry cache hash table entries: 8388608 (order: 10, 67108864 bytes, linear)
[ 0.000000] Inode-cache hash table entries: 4194304 (order: 9, 33554432 bytes, linear)
[ 0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[ 0.000000] Memory: 52058944K/52428800K available (12928K kernel code, 5760K rwdata, 4096K rodata, 3904K init, 2625K bss, 369856K reserved, 0K cma-reserved)
[ 0.000000] SLUB: HWalign=128, Order=0-3, MinObjects=0, CPUs=24, Nodes=32
[ 0.000000] trace event string verifier disabled
[ 0.000000] rcu: Hierarchical RCU implementation.
[ 0.000000] rcu: RCU event tracing is enabled.
[ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=2048 to nr_cpu_ids=24.
[ 0.000000] rcu: RCU debug extended QS entry/exit.
[ 0.000000] Trampoline variant of Tasks RCU enabled.
[ 0.000000] Rude variant of Tasks RCU enabled.
[ 0.000000] Tracing variant of Tasks RCU enabled.
[ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=24
[ 0.000000] NR_IRQS: 512, nr_irqs: 512, preallocated irqs: 16
[ 0.000000] xive: Using IRQ range [400000-400017]
[ 0.000000] xive: Interrupt handling initialized with spapr backend
[ 0.000000] xive: Using priority 7 for all interrupts
[ 0.000000] xive: Using 64kB queues
[ 0.000000] rcu: Offload RCU callbacks from CPUs: (none).
[ 0.000000] random: get_random_u64 called from start_kernel+0x65c/0x8bc with crng_init=0
[ 0.000001] time_init: 56 bit decrementer (max: 7fffffffffffff)
[ 0.000025] clocksource: timebase: mask: 0xffffffffffffffff max_cycles: 0x761537d007, max_idle_ns: 440795202126 ns
[ 0.000065] clocksource: timebase mult[1f40000] shift[24] registered
[ 0.000126] Console: colour dummy device 80x25
[ 0.000146] printk: console [hvc0] enabled
[ 0.000146] printk: console [hvc0] enabled
[ 0.000164] printk: bootconsole [udbg0] disabled
[ 0.000164] printk: bootconsole [udbg0] disabled
[ 0.000202] pid_max: default: 32768 minimum: 301
[ 0.000228] LSM: Security Framework initializing
[ 0.000238] Yama: becoming mindful.
[ 0.000245] SELinux: Initializing.
[ 0.000332] Mount-cache hash table entries: 131072 (order: 4, 1048576 bytes, linear)
[ 0.000397] Mountpoint-cache hash table entries: 131072 (order: 4, 1048576 bytes, linear)
[ 0.000993] POWER10 performance monitor hardware support registered
[ 0.001012] rcu: Hierarchical SRCU implementation.
[ 0.010106] Kprobe smoke test: started
[ 0.017789] Kprobe smoke test: passed successfully
[ 0.017976] smp: Bringing up secondary CPUs ...
[ 0.022951] smp: Brought up 1 node, 24 CPUs
[ 0.022959] numa: Node 0 CPUs: 0-23
[ 0.022963] Big cores detected but using small core scheduling
[ 0.023618] devtmpfs: initialized
[ 0.025671] PCI host bridge /pci@800000020000010 ranges:
[ 0.025681] MEM 0x0000040000000000..0x000004007effffff -> 0x0000000080000000
[ 0.025686] MEM 0x0000044000000000..0x0000047fffffffff -> 0x0006020000000000
[ 0.025704] PCI host bridge /pci@800000020000134 ranges:
[ 0.025709] MEM 0x0000040080000000..0x000004008fffffff -> 0x00000000e0000000
[ 0.025714] MEM 0x0000041000000000..0x0000041fffffffff -> 0x0006202000000000
[ 0.025751] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[ 0.025759] futex hash table entries: 8192 (order: 4, 1048576 bytes, linear)
[ 0.025968] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[ 0.026113] audit: initializing netlink subsys (disabled)
[ 0.026159] audit: type=2000 audit(1631801812.020:1): state=initialized audit_enabled=0 res=1
[ 0.026203] thermal_sys: Registered thermal governor 'fair_share'
[ 0.026204] thermal_sys: Registered thermal governor 'step_wise'
[ 0.026282] cpuidle: using governor menu
[ 0.026448] pstore: Registered nvram as persistent store backend
[ 0.026873] EEH: pSeries platform initialized
[ 0.029200] software IO TLB: tearing down default memory pool
[ 0.029262] PCI: Probing PCI hardware
[ 0.029321] PCI host bridge to bus 0010:01
[ 0.029326] pci_bus 0010:01: root bus resource [mem 0x40000000000-0x4007effffff] (bus address [0x80000000-0xfeffffff])
[ 0.029332] pci_bus 0010:01: root bus resource [mem 0x44000000000-0x47fffffffff 64bit] (bus address [0x6020000000000-0x6023fffffffff])
[ 0.029339] pci_bus 0010:01: root bus resource [bus 01-ff]
[ 0.029855] pci 0010:01:00.0: No hypervisor support for SR-IOV on this device, IOV BARs disabled.
[ 0.031328] pci 0010:01:00.0: PME# supported from D0 D3hot D3cold
[ 0.032699] pci 0010:01:00.0: 63.008 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x8 link at 0010:01:00.0 (capable of 126.016 Gb/s with 8.0 GT/s PCIe x16 link)
[ 0.037434] IOMMU table initialized, virtual merging enabled
[ 0.037490] PCI host bridge to bus 0134:80
[ 0.037493] pci_bus 0134:80: Unknown NUMA node; performance will be reduced
[ 0.037497] pci_bus 0134:80: root bus resource [mem 0x40080000000-0x4008fffffff] (bus address [0xe0000000-0xefffffff])
[ 0.037503] pci_bus 0134:80: root bus resource [mem 0x41000000000-0x41fffffffff 64bit] (bus address [0x6202000000000-0x6202fffffffff])
[ 0.037508] pci_bus 0134:80: root bus resource [bus 80-ff]
[ 0.038166] pci 0134:80:00.0: No hypervisor support for SR-IOV on this device, IOV BARs disabled.
[ 0.040026] pci 0134:80:00.0: PME# supported from D0 D3hot D3cold
[ 0.041666] pci 0134:80:00.0: 63.008 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x8 link at 0134:80:00.0 (capable of 126.016 Gb/s with 8.0 GT/s PCIe x16 link)
[ 0.046896] pci 0010:01:00.0: Adding to iommu group 0
[ 0.048009] pci 0134:80:00.0: Adding to iommu group 1
[ 0.049538] EEH: Capable adapter found: recovery enabled.
[ 0.049607] pseries-rng: Registering arch random hook.
[ 0.050488] Kprobes globally optimized
[ 0.050610] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[ 0.050615] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[ 0.079264] iommu: Default domain type: Translated
[ 0.079269] iommu: DMA domain TLB invalidation policy: strict mode
[ 0.079308] vgaarb: loaded
[ 0.079398] SCSI subsystem initialized
[ 0.079425] usbcore: registered new interface driver usbfs
[ 0.079433] usbcore: registered new interface driver hub
[ 0.079442] usbcore: registered new device driver usb
[ 0.079496] EDAC MC: Ver: 3.0.0
[ 0.079838] NetLabel: Initializing
[ 0.079841] NetLabel: domain hash size = 128
[ 0.079843] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO
[ 0.079856] NetLabel: unlabeled traffic allowed by default
[ 0.079978] clocksource: Switched to clocksource timebase
[ 0.086272] VFS: Disk quotas dquot_6.6.0
[ 0.086308] VFS: Dquot-cache hash table entries: 8192 (order 0, 65536 bytes)
[ 0.087857] NET: Registered PF_INET protocol family
[ 0.088002] IP idents hash table entries: 262144 (order: 5, 2097152 bytes, linear)
[ 0.090039] tcp_listen_portaddr_hash hash table entries: 32768 (order: 3, 524288 bytes, linear)
[ 0.090105] TCP established hash table entries: 524288 (order: 6, 4194304 bytes, linear)
[ 0.090810] TCP bind hash table entries: 65536 (order: 4, 1048576 bytes, linear)
[ 0.090887] TCP: Hash tables configured (established 524288 bind 65536)
[ 0.090934] UDP hash table entries: 32768 (order: 4, 1048576 bytes, linear)
[ 0.091014] UDP-Lite hash table entries: 32768 (order: 4, 1048576 bytes, linear)
[ 0.091153] NET: Registered PF_UNIX/PF_LOCAL protocol family
[ 0.091282] PCI: CLS 128 bytes, default 128
[ 0.091355] Trying to unpack rootfs image as initramfs...
[ 0.117522] vio_register_device_node: node lid missing 'reg'
[ 0.117633] vas: GZIP feature is available
[ 0.118526] hv-24x7: read 548 catalog entries, created 387 event attrs (0 failures), 387 descs
[ 0.120504] Initialise system trusted keyrings
[ 0.120560] workingset: timestamp_bits=38 max_order=20 bucket_order=0
[ 0.121816] zbud: loaded
[ 0.132036] NET: Registered PF_ALG protocol family
[ 0.132043] Key type asymmetric registered
[ 0.132047] Asymmetric key parser 'x509' registered
[ 0.132063] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250)
[ 0.132107] io scheduler mq-deadline registered
[ 0.132112] io scheduler kyber registered
[ 0.132382] atomic64_test: passed
[ 0.132699] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[ 0.132892] Non-volatile memory driver v1.3
[ 0.132916] Linux agpgart interface v0.103
[ 0.133081] rdac: device handler registered
[ 0.133119] hp_sw: device handler registered
[ 0.133123] emc: device handler registered
[ 0.133156] alua: device handler registered
[ 0.133239] libphy: Fixed MDIO Bus: probed
[ 0.133271] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 0.133281] ehci-pci: EHCI PCI platform driver
[ 0.133290] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[ 0.133299] ohci-pci: OHCI PCI platform driver
[ 0.133307] uhci_hcd: USB Universal Host Controller Interface driver
[ 0.133339] usbcore: registered new interface driver usbserial_generic
[ 0.133347] usbserial: USB Serial support registered for generic
[ 0.133402] mousedev: PS/2 mouse device common for all mice
[ 0.133506] rtc-generic rtc-generic: registered as rtc0
[ 0.133532] rtc-generic rtc-generic: setting system clock to 2021-09-16T14:16:53 UTC (1631801813)
[ 0.133611] xcede: xcede_record_size = 10
[ 0.133615] xcede: Record 0 : hint = 1, latency = 0x1400 tb ticks, Wake-on-irq = 1
[ 0.133620] xcede: Record 1 : hint = 2, latency = 0x3c00 tb ticks, Wake-on-irq = 0
[ 0.133625] cpuidle: Skipping the 2 Extended CEDE idle states
[ 0.133628] cpuidle: Fixed up CEDE exit latency to 10 us
[ 0.133938] nx_compress_pseries ibm,compression-v1: nx842_OF_upd: max_sync_size new:65536 old:0
[ 0.133945] nx_compress_pseries ibm,compression-v1: nx842_OF_upd: max_sync_sg new:510 old:0
[ 0.133950] nx_compress_pseries ibm,compression-v1: nx842_OF_upd: max_sg_len new:4080 old:0
[ 0.134062] alg: No test for 842 (842-nx)
[ 0.134137] hid: raw HID events driver (C) Jiri Kosina
[ 0.134250] usbcore: registered new interface driver usbhid
[ 0.134256] usbhid: USB HID core driver
[ 0.134271] drop_monitor: Initializing network drop monitor service
[ 0.134350] Initializing XFRM netlink socket
[ 0.134466] NET: Registered PF_INET6 protocol family
[ 0.134706] Segment Routing with IPv6
[ 0.134714] In-situ OAM (IOAM) with IPv6
[ 0.134732] NET: Registered PF_PACKET protocol family
[ 0.134786] Running instruction emulation self-tests ...
[ 0.134792] emulate_step_test: ld : PASS
[ 0.134797] emulate_step_test: pld : PASS
[ 0.134801] emulate_step_test: lwz : PASS
[ 0.134806] emulate_step_test: plwz : PASS
[ 0.134810] emulate_step_test: lwzx : PASS
[ 0.134814] emulate_step_test: std : PASS
[ 0.134818] emulate_step_test: pstd : PASS
[ 0.134823] emulate_step_test: ldarx / stdcx. : PASS
[ 0.134830] emulate_step_test: lfsx : PASS
[ 0.134834] emulate_step_test: stfsx : PASS
[ 0.134838] emulate_step_test: plfs : PASS
[ 0.134842] emulate_step_test: pstfs : PASS
[ 0.134846] emulate_step_test: lfdx : PASS
[ 0.134849] emulate_step_test: stfdx : PASS
[ 0.134853] emulate_step_test: plfd : PASS
[ 0.134856] emulate_step_test: pstfd : PASS
[ 0.134861] emulate_step_test: lvx : PASS
[ 0.134865] emulate_step_test: stvx : PASS
[ 0.134870] emulate_step_test: lxvd2x : PASS
[ 0.134875] emulate_step_test: stxvd2x : PASS
[ 0.134879] emulate_step_test: lxvp : PASS
[ 0.134884] emulate_step_test: stxvp : PASS
[ 0.134888] emulate_step_test: lxvpx : PASS
[ 0.134892] emulate_step_test: stxvpx : PASS
[ 0.134896] emulate_step_test: plxvp : PASS
[ 0.134900] emulate_step_test: pstxvp : PASS
[ 0.134904] emulate_step_test: nop : R0 = LONG_MAX PASS
[ 0.134910] emulate_step_test: setb : BFA = 1, CR = GT PASS
[ 0.134916] emulate_step_test: setb : BFA = 4, CR = LT PASS
[ 0.134921] emulate_step_test: setb : BFA = 5, CR = EQ PASS
[ 0.134926] emulate_step_test: add : RA = LONG_MIN, RB = LONG_MIN PASS
[ 0.134931] emulate_step_test: add : RA = LONG_MIN, RB = LONG_MAX PASS
[ 0.134936] emulate_step_test: add : RA = LONG_MAX, RB = LONG_MAX PASS
[ 0.134941] emulate_step_test: add : RA = ULONG_MAX, RB = ULONG_MAX PASS
[ 0.134946] emulate_step_test: add : RA = ULONG_MAX, RB = 0x1 PASS
[ 0.134951] emulate_step_test: add : RA = INT_MIN, RB = INT_MIN PASS
[ 0.134956] emulate_step_test: add : RA = INT_MIN, RB = INT_MAX PASS
[ 0.134961] emulate_step_test: add : RA = INT_MAX, RB = INT_MAX PASS
[ 0.134967] emulate_step_test: add : RA = UINT_MAX, RB = UINT_MAX PASS
[ 0.134972] emulate_step_test: add : RA = UINT_MAX, RB = 0x1 PASS
[ 0.134977] emulate_step_test: add. : RA = LONG_MIN, RB = LONG_MIN PASS
[ 0.134982] emulate_step_test: add. : RA = LONG_MIN, RB = LONG_MAX PASS
[ 0.134987] emulate_step_test: add. : RA = LONG_MAX, RB = LONG_MAX PASS
[ 0.134992] emulate_step_test: add. : RA = ULONG_MAX, RB = ULONG_MAX PASS
[ 0.134997] emulate_step_test: add. : RA = ULONG_MAX, RB = 0x1 PASS
[ 0.135002] emulate_step_test: add. : RA = INT_MIN, RB = INT_MIN PASS
[ 0.135007] emulate_step_test: add. : RA = INT_MIN, RB = INT_MAX PASS
[ 0.135013] emulate_step_test: add. : RA = INT_MAX, RB = INT_MAX PASS
[ 0.135018] emulate_step_test: add. : RA = UINT_MAX, RB = UINT_MAX PASS
[ 0.135022] emulate_step_test: add. : RA = UINT_MAX, RB = 0x1 PASS
[ 0.135028] emulate_step_test: addc : RA = LONG_MIN, RB = LONG_MIN PASS
[ 0.135033] emulate_step_test: addc : RA = LONG_MIN, RB = LONG_MAX PASS
[ 0.135038] emulate_step_test: addc : RA = LONG_MAX, RB = LONG_MAX PASS
[ 0.135043] emulate_step_test: addc : RA = ULONG_MAX, RB = ULONG_MAX PASS
[ 0.135048] emulate_step_test: addc : RA = ULONG_MAX, RB = 0x1 PASS
[ 0.135053] emulate_step_test: addc : RA = INT_MIN, RB = INT_MIN PASS
[ 0.135058] emulate_step_test: addc : RA = INT_MIN, RB = INT_MAX PASS
[ 0.135064] emulate_step_test: addc : RA = INT_MAX, RB = INT_MAX PASS
[ 0.135069] emulate_step_test: addc : RA = UINT_MAX, RB = UINT_MAX PASS
[ 0.135074] emulate_step_test: addc : RA = UINT_MAX, RB = 0x1 PASS
[ 0.135079] emulate_step_test: addc : RA = LONG_MIN | INT_MIN, RB = LONG_MIN | INT_MIN PASS
[ 0.135085] emulate_step_test: addc. : RA = LONG_MIN, RB = LONG_MIN PASS
[ 0.135090] emulate_step_test: addc. : RA = LONG_MIN, RB = LONG_MAX PASS
[ 0.135098] emulate_step_test: addc. : RA = LONG_MAX, RB = LONG_MAX PASS
[ 0.135103] emulate_step_test: addc. : RA = ULONG_MAX, RB = ULONG_MAX PASS
[ 0.135109] emulate_step_test: addc. : RA = ULONG_MAX, RB = 0x1 PASS
[ 0.135114] emulate_step_test: addc. : RA = INT_MIN, RB = INT_MIN PASS
[ 0.135119] emulate_step_test: addc. : RA = INT_MIN, RB = INT_MAX PASS
[ 0.135124] emulate_step_test: addc. : RA = INT_MAX, RB = INT_MAX PASS
[ 0.135130] emulate_step_test: addc. : RA = UINT_MAX, RB = UINT_MAX PASS
[ 0.135135] emulate_step_test: addc. : RA = UINT_MAX, RB = 0x1 PASS
[ 0.135140] emulate_step_test: addc. : RA = LONG_MIN | INT_MIN, RB = LONG_MIN | INT_MIN PASS
[ 0.135146] emulate_step_test: divde : RA = LONG_MIN, RB = LONG_MIN PASS
[ 0.135151] emulate_step_test: divde : RA = 1L, RB = 0 PASS
[ 0.135156] emulate_step_test: divde : RA = LONG_MIN, RB = LONG_MAX PASS
[ 0.135162] emulate_step_test: divde. : RA = LONG_MIN, RB = LONG_MIN PASS
[ 0.135167] emulate_step_test: divde. : RA = 1L, RB = 0 PASS
[ 0.135172] emulate_step_test: divde. : RA = LONG_MIN, RB = LONG_MAX PASS
[ 0.135177] emulate_step_test: divdeu : RA = LONG_MIN, RB = LONG_MIN PASS
[ 0.135183] emulate_step_test: divdeu : RA = 1L, RB = 0 PASS
[ 0.135188] emulate_step_test: divdeu : RA = LONG_MIN, RB = LONG_MAX PASS
[ 0.135193] emulate_step_test: divdeu : RA = LONG_MAX - 1, RB = LONG_MAX PASS
[ 0.135198] emulate_step_test: divdeu : RA = LONG_MIN + 1, RB = LONG_MIN PASS
[ 0.135203] emulate_step_test: divdeu. : RA = LONG_MIN, RB = LONG_MIN PASS
[ 0.135208] emulate_step_test: divdeu. : RA = 1L, RB = 0 PASS
[ 0.135213] emulate_step_test: divdeu. : RA = LONG_MIN, RB = LONG_MAX PASS
[ 0.135219] emulate_step_test: divdeu. : RA = LONG_MAX - 1, RB = LONG_MAX PASS
[ 0.135224] emulate_step_test: divdeu. : RA = LONG_MIN + 1, RB = LONG_MIN PASS
[ 0.135229] emulate_step_test: paddi : RA = LONG_MIN, SI = SI_MIN, R = 0 PASS
[ 0.135234] emulate_step_test: paddi : RA = LONG_MIN, SI = SI_MAX, R = 0 PASS
[ 0.135239] emulate_step_test: paddi : RA = LONG_MAX, SI = SI_MAX, R = 0 PASS
[ 0.135245] emulate_step_test: paddi : RA = ULONG_MAX, SI = SI_UMAX, R = 0 PASS
[ 0.135250] emulate_step_test: paddi : RA = ULONG_MAX, SI = 0x1, R = 0 PASS
[ 0.135255] emulate_step_test: paddi : RA = INT_MIN, SI = SI_MIN, R = 0 PASS
[ 0.135260] emulate_step_test: paddi : RA = INT_MIN, SI = SI_MAX, R = 0 PASS
[ 0.135265] emulate_step_test: paddi : RA = INT_MAX, SI = SI_MAX, R = 0 PASS
[ 0.135270] emulate_step_test: paddi : RA = UINT_MAX, SI = 0x1, R = 0 PASS
[ 0.135276] emulate_step_test: paddi : RA = UINT_MAX, SI = SI_MAX, R = 0 PASS
[ 0.135281] emulate_step_test: paddi : RA is r0, SI = SI_MIN, R = 0 PASS
[ 0.135286] emulate_step_test: paddi : RA = 0, SI = SI_MIN, R = 0 PASS
[ 0.135292] emulate_step_test: paddi : RA is r0, SI = 0, R = 1 PASS
[ 0.135297] emulate_step_test: paddi : RA is r0, SI = SI_MIN, R = 1 PASS
[ 0.135302] emulate_step_test: paddi : RA = R22(0), SI = 0, R = 1 PASS
[ 0.136064] registered taskstats version 1
[ 0.136315] Loading compiled-in X.509 certificates
[ 0.402845] random: fast init done
[ 0.863696] Freeing initrd memory: 65856K
[ 0.865419] alg: No test for pkcs1pad(rsa,sha1) (pkcs1pad(rsa-generic,sha1))
[ 0.865939] Loaded X.509 cert 'Build time autogenerated kernel key: a5f791480b3ed63caeaff187da537ab36b5500c1'
[ 0.866007] zswap: loaded using pool lzo/zbud
[ 0.867046] pstore: Using crash dump compression: deflate
[ 0.920244] Freeing unused kernel image (initmem) memory: 3904K
[ 0.920254] Kernel memory protection not selected by kernel config.
[ 0.920261] Run /init as init process
[ 0.926916] systemd[1]: systemd 239 (239-45.el8) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy)
[ 0.927020] systemd[1]: Detected architecture ppc64-le.
[ 0.927026] systemd[1]: Running in initial RAM disk.

Welcome to Red Hat Enterprise Linux 8.4 (Ootpa) dracut-049-135.git20210121.el8 (Initramfs)!

[ 0.960343] systemd[1]: Set hostname to <ltcden11-lp3.aus.stglabs.ibm.com>.
[ 0.993161] random: systemd: uninitialized urandom read (16 bytes read)
[ 0.993191] systemd[1]: Reached target Swap.
[ OK ] Reached target Swap.
[ 0.993296] random: systemd: uninitialized urandom read (16 bytes read)
[ 0.993304] systemd[1]: Reached target Timers.
[ OK ] Reached target Timers.
[ 0.993343] random: systemd: uninitialized urandom read (16 bytes read)
[ 0.993352] systemd[1]: Reached target Slices.
[ OK ] Reached target Slices.
[ 0.993449] systemd[1]: Listening on udev Control Socket.
[ OK ] Listening on udev Control Socket.
[ 0.993519] systemd[1]: Listening on udev Kernel Socket.
[ OK ] Listening on udev Kernel Socket.
[ 0.993601] systemd[1]: Listening on Journal Socket (/dev/log).
[ OK ] Listening on Journal Socket (/dev/log).
[ OK ] Listening on Journal Socket.
Starting Journal Service...
[ OK ] Started Memstrack Anylazing Service.
[ OK ] Reached target Sockets.
[ OK ] Started Hardware RNG Entropy Gatherer Daemon.
Starting Load Kernel Modules...
Starting Create list of required st…ce nodes for the current kernel...
Starting Setup Virtual Console...
[ OK ] Started Create list of required sta…vice nodes for the current kernel.
Starting Create Static Device Nodes in /dev...
[ OK ] Started Create Static Device Nodes in /dev.
[ 1.015760] fuse: init (API version 7.34)
[ OK ] Started Load Kernel Modules.
Starting Apply Kernel Variables...
[ OK ] Started Journal Service.
[ OK ] Started Apply Kernel Variables.
[ OK ] Started Setup Virtual Console.
Starting dracut ask for additional cmdline parameters...
[ OK ] Started dracut ask for additional cmdline parameters.
Starting dracut cmdline hook...
[ OK ] Started dracut cmdline hook.
Starting dracut pre-udev hook...
[ 1.150387] device-mapper: uevent: version 1.0.3
[ 1.150462] device-mapper: ioctl: 4.45.0-ioctl (2021-03-22) initialised: dm-devel@xxxxxxxxxx
[ OK ] Started dracut pre-udev hook.
Starting udev Kernel Device Manager...
[ OK ] Started udev Kernel Device Manager.
Starting udev Coldplug all Devices...
Mounting Kernel Configuration File System...
[ OK ] Mounted Kernel Configuration File System.
[ 1.234519] synth uevent: /devices/vio: failed to send uevent
[ 1.234534] vio vio: uevent: failed to send synthetic uevent
[ 1.234589] synth uevent: /devices/vio/4000: failed to send uevent
[ 1.234595] vio 4000: uevent: failed to send synthetic uevent
[ 1.234610] synth uevent: /devices/vio/4001: failed to send uevent
[ 1.234616] vio 4001: uevent: failed to send synthetic uevent
[ 1.234630] synth uevent: /devices/vio/4002: failed to send uevent
[ 1.234636] vio 4002: uevent: failed to send synthetic uevent
[ 1.234650] synth uevent: /devices/vio/4004: failed to send uevent
[ 1.234656] vio 4004: uevent: failed to send synthetic uevent
[ OK ] Started udev Coldplug all Devices.
Starting Show Plymouth Boot Screen...
Starting udev Wait for Complete Device Initialization...
Starting dracut initqueue hook...
[ 1.256579] ipr: IBM Power RAID SCSI Device Driver version: 2.6.4 (March 14, 2017)
[ 1.256627] ipr: Entering ipr_probe_ioa
[ 1.256636] ipr 0010:01:00.0: Found IOA with IRQ: 0
[ 1.256925] ipr 0010:01:00.0: enabling device (0140 -> 0142)
[ 1.257138] ipr 0010:01:00.0: ibm,query-pe-dma-windows(53) 10000 8000000 20000010 returned 0
[ 1.257893] ipr 0010:01:00.0: ibm,create-pe-dma-window(54) 10000 8000000 20000010 10 24 returned 0 (liobn = 0x70000010 starting addr = 8000000 0)
[ 1.260344] ibmveth 30000002 net0: renamed from eth0
[ 1.260696] ibmvfc: IBM Virtual Fibre Channel Driver version: 1.0.11 (April 12, 2013)
[ 1.261087] scsi host1: IBM POWER Virtual FC Adapter
[ OK ] Started Show Plymouth Boot Screen.
[ 1.261836] ibmvfc 300000cb: Partner initialization complete
[ OK ] Reached target Paths.
[ OK ] Started Forward Password Requests to Plymouth Directory Watch.
[ 1.270767] ibmvfc 300000cb: Host partition: ltcden11-vios1, device: vfchost2 U78D8.ND0.FGD004S-P0-C1-C0-T1 U9080.HEX.134C488-V100-C203 max sectors 2048
[ 1.284176] scsi 1:0:0:0: Direct-Access IBM 2145 0000 PQ: 0 ANSI: 6
[ 1.286246] scsi 1:0:0:0: alua: supports implicit TPGS
[ 1.286254] scsi 1:0:0:0: alua: device naa.60050768108001b3a8000000000000c5 port group 10 rel port 680
[ 1.286893] scsi 1:0:1:0: Direct-Access IBM 2145 0000 PQ: 0 ANSI: 6
[ 1.287436] scsi 1:0:1:0: alua: supports implicit TPGS
[ 1.287442] scsi 1:0:1:0: alua: device naa.60050768108001b3a8000000000000c5 port group 11 rel port e80
[ 1.290119] scsi 1:0:0:0: Attached scsi generic sg0 type 0
[ 1.290170] scsi 1:0:1:0: Attached scsi generic sg1 type 0
[ 1.293817] sd 1:0:1:0: Power-on or device reset occurred
[ 1.293972] sd 1:0:0:0: Power-on or device reset occurred
[ 1.341504] sd 1:0:1:0: alua: transition timeout set to 60 seconds
[ 1.341512] sd 1:0:1:0: alua: port group 11 state A non-preferred supports tolusna
[ 1.341631] sd 1:0:0:0: alua: transition timeout set to 60 seconds
[ 1.341641] sd 1:0:0:0: alua: port group 10 state N non-preferred supports tolusna
[ 1.341966] sd 1:0:1:0: [sdb] 209715200 512-byte logical blocks: (107 GB/100 GiB)
[ 1.342053] sd 1:0:1:0: [sdb] Write Protect is off
[ 1.342271] sd 1:0:1:0: [sdb] Write cache: disabled, read cache: enabled, supports DPO and FUA
[ 1.343010] sd 1:0:0:0: [sda] 209715200 512-byte logical blocks: (107 GB/100 GiB)
[ 1.343543] sd 1:0:0:0: [sda] Write Protect is off
[ 1.344294] sd 1:0:0:0: [sda] Write cache: disabled, read cache: enabled, supports DPO and FUA
[ 1.382396] ipr: Entering ipr_test_msi
[ 1.382482] ipr 0010:01:00.0: IRQ assigned: 60
[ 1.382493] ipr 0010:01:00.0: Received IRQ : 60
[ 1.389983] ipr 0010:01:00.0: MSI test succeeded.
[ 1.390005] ipr: Leaving ipr_test_msi
[ 1.390010] ipr 0010:01:00.0: Request for 16 MSI-Xs succeeded.
[ 1.390485] ipr: Entering ipr_alloc_mem
[ 1.390894] ipr: Leaving ipr_alloc_mem
[ 1.392712] ipr: Leaving ipr_probe_ioa
[ 1.392716] ipr: Entering ipr_probe_ioa_part2
[ 1.392721] ipr: Entering ipr_reset_enable_ioa
[ 1.392777] ipr: Entering ipr_ioafp_identify_hrrq
[ 1.392782] ipr 0010:01:00.0: Starting IOA initialization sequence.
[ 1.392787] ipr: Leaving ipr_ioafp_identify_hrrq
[ 1.392792] ipr: Leaving ipr_probe_ioa_part2
[ 1.392797] scsi host0: IBM 0 Storage Adapter
[ 1.392974] ipr: Entering ipr_worker_thread
[ 1.392978] ipr: Leaving ipr_worker_thread
[ 1.392982] ipr: Entering ipr_add_remove_thread
[ 1.393003] ipr: Entering ipr_probe_ioa
[ 1.393008] ipr 0134:80:00.0: Found IOA with IRQ: 0
[ 1.393228] ipr 0134:80:00.0: enabling device (0140 -> 0142)
[ 1.393424] ipr 0134:80:00.0: ibm,query-pe-dma-windows(53) 800000 8000000 20000134 returned 0
[ 1.394214] ipr 0134:80:00.0: ibm,create-pe-dma-window(54) 800000 8000000 20000134 10 24 returned 0 (liobn = 0x70000134 starting addr = 8000000 0)
[ 1.442260] ipr: Entering ipr_test_msi
[ 1.442381] ipr 0134:80:00.0: IRQ assigned: 76
[ 1.442388] ipr 0134:80:00.0: Received IRQ : 76
[ 1.442395] ipr 0134:80:00.0: MSI test succeeded.
[ 1.442407] ipr: Leaving ipr_test_msi
[ 1.442410] ipr 0134:80:00.0: Request for 16 MSI-Xs succeeded.
[ 1.442858] ipr: Entering ipr_alloc_mem
[ 1.443204] ipr: Leaving ipr_alloc_mem
[ 1.445029] ipr: Leaving ipr_probe_ioa
[ 1.445032] ipr: Entering ipr_probe_ioa_part2
[ 1.445035] ipr: Entering ipr_reset_enable_ioa
[ 1.445128] ipr: Entering ipr_ioafp_identify_hrrq
[ 1.445131] ipr 0134:80:00.0: Starting IOA initialization sequence.
[ 1.445134] ipr: Leaving ipr_ioafp_identify_hrrq
[ 1.445137] ipr: Leaving ipr_probe_ioa_part2
[ 1.445140] scsi host2: IBM 0 Storage Adapter
[ 1.445276] ipr: Entering ipr_worker_thread
[ 1.445279] ipr: Leaving ipr_worker_thread
[ 1.445282] ipr: Entering ipr_add_remove_thread
[ 1.463025] sdb: sdb1 sdb2 sdb3
[ 1.464074] sd 1:0:1:0: [sdb] Attached SCSI disk
[ 1.485033] sda: sda1 sda2 sda3
[ 1.488382] sd 1:0:0:0: [sda] Attached SCSI disk
[ 3.180796] random: crng init done
[ 3.180803] random: 7 urandom warning(s) missed due to ratelimiting
[ 31.839978] ipr: Entering ipr_timeout
[ 31.839978] ipr: Entering ipr_timeout
[ 31.839982] ipr 0010:01:00.0: Adapter being reset due to command timeout.
[ 31.839991] ipr 0134:80:00.0: Adapter being reset due to command timeout.
[ 31.839993] ipr: Entering ipr_reset_shutdown_ioa
[ 31.839997] ipr: Entering ipr_reset_shutdown_ioa
[ 31.839998] ipr: Leaving ipr_reset_shutdown_ioa
[ 31.840001] ipr: Leaving ipr_reset_shutdown_ioa
[ 31.840003] ipr: Entering ipr_reset_alert
[ 31.840005] ipr: Entering ipr_reset_alert
[ 31.840161] ipr: Entering ipr_reset_start_timer
[ 31.840165] ipr: Leaving ipr_reset_alert
[ 31.840167] EEH: Recovering PHB#10-PE#10000
[ 31.840169] ipr: Leaving ipr_timeout
[ 31.840177] EEH: PE location: N/A, PHB location: N/A
[ 31.840180] EEH: Frozen PHB#10-PE#10000 detected
[ 31.840184] EEH: Call Trace:
[ 31.840186] EEH: [c000000000054f90] __eeh_send_failure_event+0x70/0x150
[ 31.840195] EEH: [c00000000004e148] eeh_dev_check_failure+0x2e8/0x680
[ 31.840200] EEH: [c00000000004e5a8] eeh_check_failure+0xc8/0x100
[ 31.840204] EEH: [c0080000008cd4c0] ipr_mask_and_clear_interrupts+0x1d8/0x200 [ipr]
[ 31.840213] EEH: [c0080000008cd620] ipr_reset_alert+0x138/0x180 [ipr]
[ 31.840220] EEH: [c0080000008c49a0] ipr_reset_ioa_job+0x78/0x150 [ipr]
[ 31.840226] EEH: [c0080000008c7f00] ipr_timeout+0xd8/0x160 [ipr]
[ 31.840232] EEH: [c0000000002201f0] call_timer_fn+0x50/0x200
[ 31.840237] EEH: [c000000000221b90] run_timer_softirq+0x340/0x7c0
[ 31.840241] EEH: [c000000000c95c7c] __do_softirq+0x15c/0x3d0
[ 31.840246] EEH: [c000000000152058] irq_exit+0x168/0x1b0
[ 31.840251] EEH: [c000000000027458] timer_interrupt+0x178/0x3e0
[ 31.840256] EEH: [c000000000009a60] decrementer_common_virt+0x210/0x220
[ 31.840260] EEH: [c0000000019afd40] init_stack+0x3d40/0x4000
[ 31.840264] EEH: [c0000000009bc424] dedicated_cede_loop+0x94/0x1a0
[ 31.840270] EEH: [c0000000009b8da4] cpuidle_enter_state+0x2d4/0x4e0
[ 31.840274] EEH: [c0000000009b9048] cpuidle_enter+0x48/0x70
[ 31.840277] EEH: [c0000000001a0ad4] call_cpuidle+0x44/0x80
[ 31.840282] EEH: [c0000000001a10b0] do_idle+0x340/0x390
[ 31.840286] EEH: [c0000000001a1334] cpu_startup_entry+0x34/0x40
[ 31.840291] EEH: [c000000000012974] rest_init+0xf4/0x10c
[ 31.840294] EEH: [c0000000010b4220] start_kernel+0x868/0x8bc
[ 31.840299] ipr: Entering ipr_reset_start_timer
[ 31.840299] EEH: [c00000000000d49c] start_here_common+0x1c/0x600
[ 31.840303] ipr: Leaving ipr_reset_alert
[ 31.840305] EEH: This PCI device has failed 1 times in the last hour and will be permanently disabled after 5 failures.
[ 31.840307] EEH: Notify device drivers to shutdown
[ 31.840311] ipr: Leaving ipr_timeout
[ 31.840313] EEH: Beginning: 'error_detected(IO frozen)'
[ 31.840315] PCI 0010:01:00.0#10000: EEH: Invoking ipr->error_detected(IO frozen)
[ 31.840335] PCI 0010:01:00.0#10000: EEH: ipr driver reports: 'can recover'
[ 31.840341] EEH: Finished:'error_detected(IO frozen)' with aggregate recovery state:'can recover'
[ 31.840392] EEH: Collect temporary log
[ 31.840891] EEH: of node=0010:01:00.0
[ 31.840923] EEH: PCI device/vendor: 034a1014
[ 31.840954] EEH: PCI cmd/status register: 00100546
[ 31.840958] EEH: PCI-E capabilities and status follow:
[ 31.841102] EEH: PCI-E 00: 0002a410 00008fc2 0000585e 0003f503
[ 31.841218] EEH: PCI-E 10: 10830000 00000000 00000000 00000000
[ 31.841222] EEH: PCI-E 20: 00000000
[ 31.841224] EEH: PCI-E AER capability register set follows:
[ 31.841367] EEH: PCI-E AER 00: 18820001 00000000 00400000 00462030
[ 31.841483] EEH: PCI-E AER 10: 00000000 0000e000 000001e0 00000000
[ 31.841598] EEH: PCI-E AER 20: 00000000 00000000 00000000 00000000
[ 31.841630] EEH: PCI-E AER 30: 00000000 00000000
[ 31.842516] EEH: Enable I/O for affected devices
[ 31.842551] EEH: Notify device drivers to resume I/O
[ 31.842554] EEH: Beginning: 'mmio_enabled'
[ 31.842557] PCI 0010:01:00.0#10000: EEH: Invoking ipr->mmio_enabled()
[ 31.842559] PCI 0010:01:00.0#10000: EEH: ipr driver reports: 'need reset'
[ 31.842562] EEH: Finished:'mmio_enabled' with aggregate recovery state:'need reset'
[ 31.842568] EEH: Reset without hotplug activity
[ 31.949977] ipr: Entering ipr_reset_start_timer
[ 32.059975] ipr: Entering ipr_reset_start_timer
[ 32.169974] ipr: Entering ipr_reset_start_timer
[ 32.279975] ipr: Entering ipr_reset_start_timer
[ 32.389975] ipr: Entering ipr_reset_start_timer
[ 32.499975] ipr: Entering ipr_reset_start_timer
[ 32.609975] ipr: Entering ipr_reset_start_timer
[ 32.719976] ipr: Entering ipr_reset_start_timer
[ 32.829975] ipr: Entering ipr_reset_start_timer
[ 32.939975] ipr: Entering ipr_reset_start_timer
[ 33.049975] ipr: Entering ipr_reset_start_timer
[ 33.159975] ipr: Entering ipr_reset_start_timer
[ 33.269975] ipr: Entering ipr_reset_start_timer
[ 33.379976] ipr: Entering ipr_reset_start_timer
[ 33.489975] ipr: Entering ipr_reset_start_timer
[ 33.599975] ipr: Entering ipr_reset_start_timer
[ 33.709975] ipr: Entering ipr_reset_start_timer
[ 33.819975] ipr: Entering ipr_reset_start_timer
[ 33.929975] ipr: Entering ipr_reset_start_timer
[ 34.000356] ipr 0010:01:00.0: enabling device (0140 -> 0142)
[ 34.023909] EEH: Beginning: 'slot_reset'
[ 34.023913] PCI 0010:01:00.0#10000: EEH: Invoking ipr->slot_reset()
[ 34.023915] ipr: Entering ipr_reset_restore_cfg_space
[ 34.025673] ipr: Entering ipr_fail_all_ops
[ 34.025677] ipr: Leaving ipr_fail_all_ops
[ 34.025680] ipr: Leaving ipr_reset_restore_cfg_space
[ 34.025683] ipr: Entering ipr_reset_enable_ioa
[ 34.025697] ipr 0010:01:00.0: Initializing IOA.
[ 34.025897] ipr: IPL stage = 0xff000000, IPL stage time = 65535
[ 34.025901] PCI 0010:01:00.0#10000: EEH: ipr driver reports: 'recovered'
[ 34.025903] EEH: Finished:'slot_reset' with aggregate recovery state:'recovered'
[ 34.025908] EEH: Notify device driver to resume
[ 34.025911] EEH: Beginning: 'resume'
[ 34.025913] PCI 0010:01:00.0#10000: EEH: ipr driver reports: 'none'
[ 34.025915] EEH: Finished:'resume'
[ 34.025918] EEH: Recovery successful.
[ 34.025922] EEH: Recovering PHB#10-PE#10000
[ 34.025925] EEH: PE location: N/A, PHB location: N/A
[ 34.025927] EEH: Frozen PHB#10-PE#10000 detected
[ 34.025930] EEH: Call Trace:
[ 34.025932] EEH: [c000000000054f90] __eeh_send_failure_event+0x70/0x150
[ 34.025938] EEH: [c00000000004e148] eeh_dev_check_failure+0x2e8/0x680
[ 34.025942] EEH: [c00000000004e5a8] eeh_check_failure+0xc8/0x100
[ 34.025946] EEH: [c0080000008d1c70] ipr_reset_next_stage+0x198/0x370 [ipr]
[ 34.025953] EEH: [c0080000008c49a0] ipr_reset_ioa_job+0x78/0x150 [ipr]
[ 34.025960] EEH: [c0080000008c8f04] ipr_pci_slot_reset+0xac/0x140 [ipr]
[ 34.025966] EEH: [c000000000052ce8] eeh_report_reset+0xb8/0xe0
[ 34.025970] EEH: [c000000000054834] eeh_pe_report+0x254/0x4b4
[ 34.025974] EEH: [c000000000053efc] eeh_handle_normal_event+0x9dc/0xa80
[ 34.025979] EEH: [c000000000054de8] eeh_event_handler+0x108/0x190
[ 34.025983] EEH: [c00000000018151c] kthread+0x18c/0x1a0
[ 34.025987] EEH: [c00000000000cfd4] ret_from_kernel_thread+0x5c/0x64
[ 34.025991] EEH: This PCI device has failed 2 times in the last hour and will be permanently disabled after 5 failures.
[ 34.025995] EEH: Notify device drivers to shutdown
[ 34.025998] EEH: Beginning: 'error_detected(IO frozen)'
[ 34.026000] PCI 0010:01:00.0#10000: EEH: Invoking ipr->error_detected(IO frozen)
[ 34.026011] PCI 0010:01:00.0#10000: EEH: ipr driver reports: 'can recover'
[ 34.026016] EEH: Finished:'error_detected(IO frozen)' with aggregate recovery state:'can recover'
[ 34.026059] EEH: Collect temporary log
[ 34.026537] EEH: of node=0010:01:00.0
[ 34.026568] EEH: PCI device/vendor: 034a1014
[ 34.026600] EEH: PCI cmd/status register: 00100546
[ 34.026603] EEH: PCI-E capabilities and status follow:
[ 34.026745] EEH: PCI-E 00: 0002a410 00008fc2 0000585e 0003f503
[ 34.026860] EEH: PCI-E 10: 10830000 00000000 00000000 00000000
[ 34.026863] EEH: PCI-E 20: 00000000
[ 34.026866] EEH: PCI-E AER capability register set follows:
[ 34.027010] EEH: PCI-E AER 00: 18820001 00000000 00400000 00462030
[ 34.027125] EEH: PCI-E AER 10: 00000000 0000e000 000001e0 00000000
[ 34.027239] EEH: PCI-E AER 20: 00000000 00000000 00000000 00000000
[ 34.027270] EEH: PCI-E AER 30: 00000000 00000000
[ 34.028118] EEH: Enable I/O for affected devices
[ 34.028153] EEH: Notify device drivers to resume I/O
[ 34.028155] EEH: Beginning: 'mmio_enabled'
[ 34.028158] PCI 0010:01:00.0#10000: EEH: Invoking ipr->mmio_enabled()
[ 34.028159] PCI 0010:01:00.0#10000: EEH: ipr driver reports: 'need reset'
[ 34.028163] EEH: Finished:'mmio_enabled' with aggregate recovery state:'need reset'
[ 34.028168] EEH: Reset without hotplug activity
[ 34.039978] ipr: Entering ipr_reset_start_timer
[ 34.149976] ipr: Entering ipr_reset_start_bist
[ 34.149981] ipr: Entering ipr_reset_start_timer
[ 34.149985] ipr: Leaving ipr_reset_start_bist
[ 36.159976] ipr: Entering ipr_reset_bist_done
[ 36.159981] ipr: Leaving ipr_reset_bist_done
[ 36.159984] ipr: Entering ipr_reset_restore_cfg_space
[ 36.162229] ipr: Entering ipr_fail_all_ops
[ 36.162232] ipr: Leaving ipr_fail_all_ops
[ 36.162234] ipr: Leaving ipr_reset_restore_cfg_space
[ 36.162237] ipr: Entering ipr_reset_enable_ioa
[ 36.162251] ipr: Entering ipr_ioafp_identify_hrrq
[ 36.162254] ipr 0134:80:00.0: Starting IOA initialization sequence.
[ 36.162258] ipr: Leaving ipr_ioafp_identify_hrrq
[ 36.162637] ipr 0010:01:00.0: enabling device (0140 -> 0142)
[ 36.185512] EEH: Beginning: 'slot_reset'
[ 36.185516] PCI 0010:01:00.0#10000: EEH: Invoking ipr->slot_reset()
[ 36.185518] ipr: Entering ipr_reset_restore_cfg_space
[ 36.187270] ipr: Entering ipr_fail_all_ops
[ 36.187274] ipr: Leaving ipr_fail_all_ops
[ 36.187276] ipr: Leaving ipr_reset_restore_cfg_space
[ 36.187279] ipr: Entering ipr_reset_enable_ioa
[ 36.187286] ipr 0010:01:00.0: Initializing IOA.
[ 36.187480] ipr: IPL stage = 0xff000000, IPL stage time = 65535
[ 36.187484] PCI 0010:01:00.0#10000: EEH: ipr driver reports: 'recovered'
[ 36.187486] EEH: Finished:'slot_reset' with aggregate recovery state:'recovered'
[ 36.187491] EEH: Notify device driver to resume
[ 36.187494] EEH: Beginning: 'resume'
[ 36.187496] PCI 0010:01:00.0#10000: EEH: ipr driver reports: 'none'
[ 36.187497] EEH: Finished:'resume'
[ 36.187500] EEH: Recovery successful.
[ 36.187504] EEH: Recovering PHB#10-PE#10000
[ 36.187507] EEH: PE location: N/A, PHB location: N/A
[ 36.187509] EEH: Frozen PHB#10-PE#10000 detected
[ 36.187512] EEH: Call Trace:
[ 36.187514] EEH: [c000000000054f90] __eeh_send_failure_event+0x70/0x150
[ 36.187519] EEH: [c00000000004e148] eeh_dev_check_failure+0x2e8/0x680
[ 36.187523] EEH: [c00000000004e5a8] eeh_check_failure+0xc8/0x100
[ 36.187527] EEH: [c0080000008d1c70] ipr_reset_next_stage+0x198/0x370 [ipr]
[ 36.187534] EEH: [c0080000008c49a0] ipr_reset_ioa_job+0x78/0x150 [ipr]
[ 36.187541] EEH: [c0080000008c8f04] ipr_pci_slot_reset+0xac/0x140 [ipr]
[ 36.187547] EEH: [c000000000052ce8] eeh_report_reset+0xb8/0xe0
[ 36.187552] EEH: [c000000000054834] eeh_pe_report+0x254/0x4b4
[ 36.187556] EEH: [c000000000053efc] eeh_handle_normal_event+0x9dc/0xa80
[ 36.187560] EEH: [c000000000054de8] eeh_event_handler+0x108/0x190
[ 36.187564] EEH: [c00000000018151c] kthread+0x18c/0x1a0
[ 36.187568] EEH: [c00000000000cfd4] ret_from_kernel_thread+0x5c/0x64
[ 36.187572] EEH: This PCI device has failed 3 times in the last hour and will be permanently disabled after 5 failures.
[ 36.187576] EEH: Notify device drivers to shutdown
[ 36.187578] EEH: Beginning: 'error_detected(IO frozen)'
[ 36.187581] PCI 0010:01:00.0#10000: EEH: Invoking ipr->error_detected(IO frozen)
[ 36.187591] PCI 0010:01:00.0#10000: EEH: ipr driver reports: 'can recover'
[ 36.187595] EEH: Finished:'error_detected(IO frozen)' with aggregate recovery state:'can recover'
[ 36.187636] EEH: Collect temporary log
[ 36.188115] EEH: of node=0010:01:00.0
[ 36.188146] EEH: PCI device/vendor: 034a1014
[ 36.188177] EEH: PCI cmd/status register: 00100546
[ 36.188180] EEH: PCI-E capabilities and status follow:
[ 36.188324] EEH: PCI-E 00: 0002a410 00008fc2 0000585e 0003f503
[ 36.188439] EEH: PCI-E 10: 10830000 00000000 00000000 00000000
[ 36.188442] EEH: PCI-E 20: 00000000
[ 36.188444] EEH: PCI-E AER capability register set follows:
[ 36.188587] EEH: PCI-E AER 00: 18820001 00000000 00400000 00462030
[ 36.188702] EEH: PCI-E AER 10: 00000000 0000e000 000001e0 00000000
[ 36.188816] EEH: PCI-E AER 20: 00000000 00000000 00000000 00000000
[ 36.188847] EEH: PCI-E AER 30: 00000000 00000000
[ 36.189681] EEH: Enable I/O for affected devices
[ 36.189716] EEH: Notify device drivers to resume I/O
[ 36.189719] EEH: Beginning: 'mmio_enabled'
[ 36.189721] PCI 0010:01:00.0#10000: EEH: Invoking ipr->mmio_enabled()
[ 36.189723] PCI 0010:01:00.0#10000: EEH: ipr driver reports: 'need reset'
[ 36.189726] EEH: Finished:'mmio_enabled' with aggregate recovery state:'need reset'
[ 36.189732] EEH: Reset without hotplug activity
[ 38.320356] ipr 0010:01:00.0: enabling device (0140 -> 0142)
[ 38.343242] EEH: Beginning: 'slot_reset'
[ 38.343247] PCI 0010:01:00.0#10000: EEH: Invoking ipr->slot_reset()
[ 38.343249] ipr: Entering ipr_reset_restore_cfg_space
[ 38.345002] ipr: Entering ipr_fail_all_ops
[ 38.345005] ipr: Leaving ipr_fail_all_ops
[ 38.345008] ipr: Leaving ipr_reset_restore_cfg_space
[ 38.345011] ipr: Entering ipr_reset_enable_ioa
[ 38.345018] ipr 0010:01:00.0: Initializing IOA.
[ 38.345220] ipr: IPL stage = 0xff000000, IPL stage time = 65535
[ 38.345224] PCI 0010:01:00.0#10000: EEH: ipr driver reports: 'recovered'
[ 38.345226] EEH: Finished:'slot_reset' with aggregate recovery state:'recovered'
[ 38.345232] EEH: Notify device driver to resume
[ 38.345234] EEH: Beginning: 'resume'
[ 38.345236] PCI 0010:01:00.0#10000: EEH: ipr driver reports: 'none'
[ 38.345238] EEH: Finished:'resume'
[ 38.345241] EEH: Recovery successful.
[ 38.345245] EEH: Recovering PHB#10-PE#10000
[ 38.345247] EEH: PE location: N/A, PHB location: N/A
[ 38.345250] EEH: Frozen PHB#10-PE#10000 detected
[ 38.345253] EEH: Call Trace:
[ 38.345254] EEH: [c000000000054f90] __eeh_send_failure_event+0x70/0x150
[ 38.345259] EEH: [c00000000004e148] eeh_dev_check_failure+0x2e8/0x680
[ 38.345264] EEH: [c00000000004e5a8] eeh_check_failure+0xc8/0x100
[ 38.345268] EEH: [c0080000008d1c70] ipr_reset_next_stage+0x198/0x370 [ipr]
[ 38.345274] EEH: [c0080000008c49a0] ipr_reset_ioa_job+0x78/0x150 [ipr]
[ 38.345281] EEH: [c0080000008c8f04] ipr_pci_slot_reset+0xac/0x140 [ipr]
[ 38.345287] EEH: [c000000000052ce8] eeh_report_reset+0xb8/0xe0
[ 38.345291] EEH: [c000000000054834] eeh_pe_report+0x254/0x4b4
[ 38.345295] EEH: [c000000000053efc] eeh_handle_normal_event+0x9dc/0xa80
[ 38.345299] EEH: [c000000000054de8] eeh_event_handler+0x108/0x190
[ 38.345303] EEH: [c00000000018151c] kthread+0x18c/0x1a0
[ 38.345307] EEH: [c00000000000cfd4] ret_from_kernel_thread+0x5c/0x64
[ 38.345311] EEH: This PCI device has failed 4 times in the last hour and will be permanently disabled after 5 failures.
[ 38.345315] EEH: Notify device drivers to shutdown
[ 38.345317] EEH: Beginning: 'error_detected(IO frozen)'
[ 38.345319] PCI 0010:01:00.0#10000: EEH: Invoking ipr->error_detected(IO frozen)
[ 38.345330] PCI 0010:01:00.0#10000: EEH: ipr driver reports: 'can recover'
[ 38.345334] EEH: Finished:'error_detected(IO frozen)' with aggregate recovery state:'can recover'
[ 38.345376] EEH: Collect temporary log
[ 38.345854] EEH: of node=0010:01:00.0
[ 38.345885] EEH: PCI device/vendor: 034a1014
[ 38.345916] EEH: PCI cmd/status register: 00100546
[ 38.345919] EEH: PCI-E capabilities and status follow:
[ 38.346062] EEH: PCI-E 00: 0002a410 00008fc2 0000585e 0003f503
[ 38.346177] EEH: PCI-E 10: 10830000 00000000 00000000 00000000
[ 38.346180] EEH: PCI-E 20: 00000000
[ 38.346182] EEH: PCI-E AER capability register set follows:
[ 38.346325] EEH: PCI-E AER 00: 18820001 00000000 00400000 00462030
[ 38.346440] EEH: PCI-E AER 10: 00000000 0000e000 000001e0 00000000
[ 38.346556] EEH: PCI-E AER 20: 00000000 00000000 00000000 00000000
[ 38.346587] EEH: PCI-E AER 30: 00000000 00000000
[ 38.347423] EEH: Enable I/O for affected devices
[ 38.347458] EEH: Notify device drivers to resume I/O
[ 38.347461] EEH: Beginning: 'mmio_enabled'
[ 38.347463] PCI 0010:01:00.0#10000: EEH: Invoking ipr->mmio_enabled()
[ 38.347464] PCI 0010:01:00.0#10000: EEH: ipr driver reports: 'need reset'
[ 38.347468] EEH: Finished:'mmio_enabled' with aggregate recovery state:'need reset'
[ 38.347473] EEH: Reset without hotplug activity
[ 40.480353] ipr 0010:01:00.0: enabling device (0140 -> 0142)
[ 40.503235] EEH: Beginning: 'slot_reset'
[ 40.503239] PCI 0010:01:00.0#10000: EEH: Invoking ipr->slot_reset()
[ 40.503241] ipr: Entering ipr_reset_restore_cfg_space
[ 40.504991] ipr: Entering ipr_fail_all_ops
[ 40.504995] ipr: Leaving ipr_fail_all_ops
[ 40.504997] ipr: Leaving ipr_reset_restore_cfg_space
[ 40.505000] ipr: Entering ipr_reset_enable_ioa
[ 40.505007] ipr 0010:01:00.0: Initializing IOA.
[ 40.505201] ipr: IPL stage = 0xff000000, IPL stage time = 65535
[ 40.505205] PCI 0010:01:00.0#10000: EEH: ipr driver reports: 'recovered'
[ 40.505207] EEH: Finished:'slot_reset' with aggregate recovery state:'recovered'
[ 40.505212] EEH: Notify device driver to resume
[ 40.505215] EEH: Beginning: 'resume'
[ 40.505217] PCI 0010:01:00.0#10000: EEH: ipr driver reports: 'none'
[ 40.505218] EEH: Finished:'resume'
[ 40.505221] EEH: Recovery successful.
[ 40.505225] EEH: Recovering PHB#10-PE#10000
[ 40.505227] EEH: PE location: N/A, PHB location: N/A
[ 40.505230] EEH: Frozen PHB#10-PE#10000 detected
[ 40.505233] EEH: Call Trace:
[ 40.505234] EEH: [c000000000054f90] __eeh_send_failure_event+0x70/0x150
[ 40.505239] EEH: [c00000000004e148] eeh_dev_check_failure+0x2e8/0x680
[ 40.505244] EEH: [c00000000004e5a8] eeh_check_failure+0xc8/0x100
[ 40.505248] EEH: [c0080000008d1c70] ipr_reset_next_stage+0x198/0x370 [ipr]
[ 40.505254] EEH: [c0080000008c49a0] ipr_reset_ioa_job+0x78/0x150 [ipr]
[ 40.505261] EEH: [c0080000008c8f04] ipr_pci_slot_reset+0xac/0x140 [ipr]
[ 40.505267] EEH: [c000000000052ce8] eeh_report_reset+0xb8/0xe0
[ 40.505271] EEH: [c000000000054834] eeh_pe_report+0x254/0x4b4
[ 40.505275] EEH: [c000000000053efc] eeh_handle_normal_event+0x9dc/0xa80
[ 40.505279] EEH: [c000000000054de8] eeh_event_handler+0x108/0x190
[ 40.505284] EEH: [c00000000018151c] kthread+0x18c/0x1a0
[ 40.505287] EEH: [c00000000000cfd4] ret_from_kernel_thread+0x5c/0x64
[ 40.505291] EEH: This PCI device has failed 5 times in the last hour and will be permanently disabled after 5 failures.
[ 40.505295] EEH: Notify device drivers to shutdown
[ 40.505298] EEH: Beginning: 'error_detected(IO frozen)'
[ 40.505300] PCI 0010:01:00.0#10000: EEH: Invoking ipr->error_detected(IO frozen)
[ 40.505310] PCI 0010:01:00.0#10000: EEH: ipr driver reports: 'can recover'
[ 40.505314] EEH: Finished:'error_detected(IO frozen)' with aggregate recovery state:'can recover'
[ 40.505355] EEH: Collect temporary log
[ 40.505833] EEH: of node=0010:01:00.0
[ 40.505864] EEH: PCI device/vendor: 034a1014
[ 40.505895] EEH: PCI cmd/status register: 00100546
[ 40.505898] EEH: PCI-E capabilities and status follow:
[ 40.506041] EEH: PCI-E 00: 0002a410 00008fc2 0000585e 0003f503
[ 40.506157] EEH: PCI-E 10: 10830000 00000000 00000000 00000000
[ 40.506161] EEH: PCI-E 20: 00000000
[ 40.506163] EEH: PCI-E AER capability register set follows:
[ 40.506305] EEH: PCI-E AER 00: 18820001 00000000 00400000 00462030
[ 40.506420] EEH: PCI-E AER 10: 00000000 0000e000 000001e0 00000000
[ 40.506534] EEH: PCI-E AER 20: 00000000 00000000 00000000 00000000
[ 40.506565] EEH: PCI-E AER 30: 00000000 00000000
[ 40.507411] EEH: Enable I/O for affected devices
[ 40.507446] EEH: Notify device drivers to resume I/O
[ 40.507449] EEH: Beginning: 'mmio_enabled'
[ 40.507451] PCI 0010:01:00.0#10000: EEH: Invoking ipr->mmio_enabled()
[ 40.507452] PCI 0010:01:00.0#10000: EEH: ipr driver reports: 'need reset'
[ 40.507456] EEH: Finished:'mmio_enabled' with aggregate recovery state:'need reset'
[ 40.507461] EEH: Reset without hotplug activity
[ 42.640358] ipr 0010:01:00.0: enabling device (0140 -> 0142)
[ 42.663240] EEH: Beginning: 'slot_reset'
[ 42.663244] PCI 0010:01:00.0#10000: EEH: Invoking ipr->slot_reset()
[ 42.663246] ipr: Entering ipr_reset_restore_cfg_space
[ 42.664997] ipr: Entering ipr_fail_all_ops
[ 42.665001] ipr: Leaving ipr_fail_all_ops
[ 42.665003] ipr: Leaving ipr_reset_restore_cfg_space
[ 42.665006] ipr: Entering ipr_reset_enable_ioa
[ 42.665013] ipr 0010:01:00.0: Initializing IOA.
[ 42.665213] ipr: IPL stage = 0xff000000, IPL stage time = 65535
[ 42.665217] PCI 0010:01:00.0#10000: EEH: ipr driver reports: 'recovered'
[ 42.665219] EEH: Finished:'slot_reset' with aggregate recovery state:'recovered'
[ 42.665224] EEH: Notify device driver to resume
[ 42.665227] EEH: Beginning: 'resume'
[ 42.665229] PCI 0010:01:00.0#10000: EEH: ipr driver reports: 'none'
[ 42.665230] EEH: Finished:'resume'
[ 42.665233] EEH: Recovery successful.
[ 42.665237] EEH: Recovering PHB#10-PE#10000
[ 42.665240] EEH: PE location: N/A, PHB location: N/A
[ 42.665242] EEH: Frozen PHB#10-PE#10000 detected
[ 42.665245] EEH: Call Trace:
[ 42.665247] EEH: [c000000000054f90] __eeh_send_failure_event+0x70/0x150
[ 42.665252] EEH: [c00000000004e148] eeh_dev_check_failure+0x2e8/0x680
[ 42.665256] EEH: [c00000000004e5a8] eeh_check_failure+0xc8/0x100
[ 42.665260] EEH: [c0080000008d1c70] ipr_reset_next_stage+0x198/0x370 [ipr]
[ 42.665267] EEH: [c0080000008c49a0] ipr_reset_ioa_job+0x78/0x150 [ipr]
[ 42.665273] EEH: [c0080000008c8f04] ipr_pci_slot_reset+0xac/0x140 [ipr]
[ 42.665279] EEH: [c000000000052ce8] eeh_report_reset+0xb8/0xe0
[ 42.665284] EEH: [c000000000054834] eeh_pe_report+0x254/0x4b4
[ 42.665288] EEH: [c000000000053efc] eeh_handle_normal_event+0x9dc/0xa80
[ 42.665292] EEH: [c000000000054de8] eeh_event_handler+0x108/0x190
[ 42.665296] EEH: [c00000000018151c] kthread+0x18c/0x1a0
[ 42.665299] EEH: [c00000000000cfd4] ret_from_kernel_thread+0x5c/0x64
[ 42.665303] EEH: PHB#10-PE#10000 has failed 6 times in the last hour and has been permanently disabled.
[ 42.665307] EEH: Unable to recover from failure from PHB#10-PE#10000.
[ 42.665307] Please try reseating or replacing it
[ 42.665788] EEH: of node=0010:01:00.0
[ 42.665819] EEH: PCI device/vendor: 034a1014
[ 42.665851] EEH: PCI cmd/status register: 00100546
[ 42.665854] EEH: PCI-E capabilities and status follow:
[ 42.665996] EEH: PCI-E 00: 0002a410 00008fc2 0000585e 0003f503
[ 42.666112] EEH: PCI-E 10: 10830000 00000000 00000000 00000000
[ 42.666115] EEH: PCI-E 20: 00000000
[ 42.666117] EEH: PCI-E AER capability register set follows:
[ 42.666260] EEH: PCI-E AER 00: 18820001 00000000 00400000 00462030
[ 42.666375] EEH: PCI-E AER 10: 00000000 0000e000 000001e0 00000000
[ 42.666489] EEH: PCI-E AER 20: 00000000 00000000 00000000 00000000
[ 42.666520] EEH: PCI-E AER 30: 00000000 00000000
[ 42.667476] EEH: Beginning: 'error_detected(permanent failure)'
[ 42.667479] PCI 0010:01:00.0#10000: EEH: not actionable (1,1,1)
[ 42.667481] EEH: Finished:'error_detected(permanent failure)'
[ 42.667492] ipr: Entering ipr_remove
[ 42.667542] ipr: Entering __ipr_remove
[ 66.399976] ipr: Entering ipr_timeout
[ 66.399983] ipr 0134:80:00.0: Adapter being reset due to command timeout.
[ 66.399989] ipr: Entering ipr_reset_shutdown_ioa
[ 66.399992] ipr: Leaving ipr_reset_shutdown_ioa
[ 66.399995] ipr: Entering ipr_reset_alert
[ 66.400049] ipr: Entering ipr_reset_start_timer
[ 66.400054] ipr: Leaving ipr_reset_alert
[ 66.400057] ipr: Leaving ipr_timeout
[ 66.509975] ipr: Entering ipr_reset_start_timer
[ 66.619975] ipr: Entering ipr_reset_start_timer
[ 66.729975] ipr: Entering ipr_reset_start_timer
[ 66.839975] ipr: Entering ipr_reset_start_timer
[ 66.949975] ipr: Entering ipr_reset_start_timer
[ 67.059975] ipr: Entering ipr_reset_start_timer
[ 67.169975] ipr: Entering ipr_reset_start_timer
[ 67.279975] ipr: Entering ipr_reset_start_timer
[ 67.389975] ipr: Entering ipr_reset_start_timer
[ 67.499975] ipr: Entering ipr_reset_start_timer
[ 67.609975] ipr: Entering ipr_reset_start_timer
[ 67.719975] ipr: Entering ipr_reset_start_timer
[ 67.829975] ipr: Entering ipr_reset_start_timer
[ 67.939975] ipr: Entering ipr_reset_start_timer
[ 68.049976] ipr: Entering ipr_reset_start_timer
[ 68.159975] ipr: Entering ipr_reset_start_timer
[ 68.269975] ipr: Entering ipr_reset_start_timer
[ 68.379977] ipr: Entering ipr_reset_start_timer
[ 68.489975] ipr: Entering ipr_reset_start_timer
[ 68.599975] ipr: Entering ipr_reset_start_timer
[ 68.709975] ipr: Entering ipr_reset_start_bist
[ 68.709979] ipr: Entering ipr_reset_start_timer
[ 68.709983] ipr: Leaving ipr_reset_start_bist
[ 70.719975] ipr: Entering ipr_reset_bist_done
[ 70.719980] ipr: Leaving ipr_reset_bist_done
[ 70.719983] ipr: Entering ipr_reset_restore_cfg_space
[ 70.722256] ipr: Entering ipr_fail_all_ops
[ 70.722260] ipr: Leaving ipr_fail_all_ops
[ 70.722263] ipr: Leaving ipr_reset_restore_cfg_space
[ 70.722266] ipr: Entering ipr_reset_enable_ioa
[ 70.722280] ipr: Entering ipr_ioafp_identify_hrrq
[ 70.722283] ipr 0134:80:00.0: Starting IOA initialization sequence.
[ 70.722288] ipr: Leaving ipr_ioafp_identify_hrrq
[ 100.959975] ipr: Entering ipr_timeout
[ 100.959988] ipr 0134:80:00.0: Adapter being reset due to command timeout.
[ 100.959995] ipr: Entering ipr_reset_shutdown_ioa
[ 100.959998] ipr: Leaving ipr_reset_shutdown_ioa
[ 100.960000] ipr: Entering ipr_reset_alert
[ 100.960066] ipr: Entering ipr_reset_start_timer
[ 100.960070] ipr: Leaving ipr_reset_alert
[ 100.960072] ipr: Leaving ipr_timeout
[ 101.069976] ipr: Entering ipr_reset_start_timer
[ 101.179975] ipr: Entering ipr_reset_start_timer
[ 101.289975] ipr: Entering ipr_reset_start_timer
[ 101.399977] ipr: Entering ipr_reset_start_timer
[ 101.509976] ipr: Entering ipr_reset_start_timer
[ 101.619975] ipr: Entering ipr_reset_start_timer
[ 101.729975] ipr: Entering ipr_reset_start_timer
[ 101.839975] ipr: Entering ipr_reset_start_timer
[ 101.949975] ipr: Entering ipr_reset_start_timer
[ 102.059975] ipr: Entering ipr_reset_start_timer
[ 102.169975] ipr: Entering ipr_reset_start_timer
[ 102.279975] ipr: Entering ipr_reset_start_timer
[ 102.389976] ipr: Entering ipr_reset_start_timer
[ 102.499975] ipr: Entering ipr_reset_start_timer
[ 102.609975] ipr: Entering ipr_reset_start_timer
[ 102.719975] ipr: Entering ipr_reset_start_timer
[ 102.829975] ipr: Entering ipr_reset_start_timer
[ 102.939975] ipr: Entering ipr_reset_start_timer
[ 103.049975] ipr: Entering ipr_reset_start_timer
[ 103.159975] ipr: Entering ipr_reset_start_timer
[ 103.269976] ipr: Entering ipr_reset_start_bist
[ 103.269980] ipr: Entering ipr_reset_start_timer
[ 103.269984] ipr: Leaving ipr_reset_start_bist
[ 105.279975] ipr: Entering ipr_reset_bist_done
[ 105.279980] ipr: Leaving ipr_reset_bist_done
[ 105.279983] ipr: Entering ipr_reset_restore_cfg_space
[ 105.282241] ipr: Entering ipr_fail_all_ops
[ 105.282245] ipr: Leaving ipr_fail_all_ops
[ 105.282248] ipr: Leaving ipr_reset_restore_cfg_space
[ 105.282251] ipr: Entering ipr_reset_enable_ioa
[ 105.282265] ipr: Entering ipr_ioafp_identify_hrrq
[ 105.282268] ipr 0134:80:00.0: Starting IOA initialization sequence.
[ 105.282273] ipr: Leaving ipr_ioafp_identify_hrrq
[FAILED] Failed to start udev Wait for Complete Device Initialization.
See 'systemctl status systemd-udev-settle.service' for details.
Starting Device-Mapper Multipath Device Controller...
[ OK ] Started Device-Mapper Multipath Device Controller.
[ OK ] Reached target Local File Systems (Pre).
[ OK ] Reached target Local File Systems.
Starting Create Volatile Files and Directories...
[ OK ] Started Create Volatile Files and Directories.
[ OK ] Reached target System Initialization.
[ OK ] Reached target Basic System.
[ 121.492754] device-mapper: multipath service-time: version 0.3.0 loaded
[ 135.519977] ipr: Entering ipr_timeout
[ 135.519991] ipr 0134:80:00.0: Adapter being reset due to command timeout.
[ 135.519996] ipr 0134:80:00.0: IOA taken offline - error recovery failed
[ 135.520002] ipr: Entering ipr_reset_shutdown_ioa
[ 135.520005] ipr: Leaving ipr_reset_shutdown_ioa
[ 135.520008] ipr: Entering ipr_reset_alert
[ 135.520083] ipr: Entering ipr_reset_start_timer
[ 135.520088] ipr: Leaving ipr_reset_alert
[ 135.520092] ipr: Leaving ipr_timeout
[ 135.629975] ipr: Entering ipr_reset_start_timer
[ 135.739975] ipr: Entering ipr_reset_start_timer
[ 135.849975] ipr: Entering ipr_reset_start_timer
[ 135.959975] ipr: Entering ipr_reset_start_timer
[ 136.069977] ipr: Entering ipr_reset_start_timer
[ 136.179975] ipr: Entering ipr_reset_start_timer
[ 136.289975] ipr: Entering ipr_reset_start_timer
[ 136.399975] ipr: Entering ipr_reset_start_timer
[ 136.509975] ipr: Entering ipr_reset_start_timer
[ 136.619976] ipr: Entering ipr_reset_start_timer
[ 136.729976] ipr: Entering ipr_reset_start_timer
[ 136.839976] ipr: Entering ipr_reset_start_timer
[ 136.949975] ipr: Entering ipr_reset_start_timer
[ 137.059977] ipr: Entering ipr_reset_start_timer
[ 137.169975] ipr: Entering ipr_reset_start_timer
[ 137.279975] ipr: Entering ipr_reset_start_timer
[ 137.389975] ipr: Entering ipr_reset_start_timer
[ 137.499975] ipr: Entering ipr_reset_start_timer
[ 137.609975] ipr: Entering ipr_reset_start_timer
[ 137.719975] ipr: Entering ipr_reset_start_timer
[ 137.829976] ipr: Entering ipr_reset_start_bist
[ 137.829980] ipr: Entering ipr_reset_start_timer
[ 137.829984] ipr: Leaving ipr_reset_start_bist
[ 139.839975] ipr: Entering ipr_reset_bist_done
[ 139.839980] ipr: Leaving ipr_reset_bist_done
[ 139.839983] ipr: Entering ipr_reset_restore_cfg_space
[ 139.842243] ipr: Entering ipr_fail_all_ops
[ 139.842247] ipr: Leaving ipr_fail_all_ops
[ 139.842249] ipr: Leaving ipr_reset_restore_cfg_space
[ 139.842252] ipr: Entering ipr_ioa_bringdown_done
[ 139.842255] ipr: drivers/scsi/ipr.c: ipr_ioa_bringdown_done: Line: 7243
[ 139.842259] ipr: Leaving ipr_ioa_bringdown_done
[ 139.842266] ipr: Entering ipr_worker_thread
[ 139.842270] ipr: Leaving ipr_worker_thread
[ 139.842273] ipr: Entering ipr_add_remove_thread
[ 243.679987] INFO: task eehd:137 blocked for more than 122 seconds.
[ 243.680001] Not tainted 5.15.0-rc1-autotest #5
[ 243.680005] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 243.680009] task:eehd state:D stack: 0 pid: 137 ppid: 2 flags:0x00000800
[ 243.680016] Call Trace:
[ 243.680018] [c000000008167650] [c0000000001dd368] info_print_prefix+0xa8/0x130 (unreliable)
[ 243.680027] [c000000008167840] [c00000000001e6c8] __switch_to+0x278/0x490
[ 243.680033] [c0000000081678a0] [c000000000c8b9bc] __schedule+0x31c/0xa20
[ 243.680039] [c000000008167960] [c000000000c8c128] schedule+0x68/0x120
[ 243.680043] [c000000008167990] [c0080000008ca3c4] __ipr_remove+0x10c/0x4b0 [ipr]
[ 243.680053] [c000000008167a60] [c0080000008ca828] ipr_remove+0xc0/0x110 [ipr]
[ 243.680061] [c000000008167ae0] [c000000000732bd4] pci_device_remove+0x64/0x120
[ 243.680067] [c000000008167b20] [c000000000811318] device_release_driver_internal+0x168/0x2c0
[ 243.680073] [c000000008167b60] [c000000000724618] pci_stop_bus_device+0xa8/0x100
[ 243.680079] [c000000008167ba0] [c000000000724810] pci_stop_and_remove_bus_device+0x20/0x40
[ 243.680083] [c000000008167bd0] [c000000000075e60] pci_hp_remove_devices+0x90/0x130
[ 243.680089] [c000000008167c60] [c000000000053cd8] eeh_handle_normal_event+0x7b8/0xa80
[ 243.680095] [c000000008167d40] [c000000000054de8] eeh_event_handler+0x108/0x190
[ 243.680100] [c000000008167da0] [c00000000018151c] kthread+0x18c/0x1a0
[ 243.680105] [c000000008167e10] [c00000000000cfd4] ret_from_kernel_thread+0x5c/0x64
[ 243.680117] INFO: task systemd-udevd:476 blocked for more than 122 seconds.
[ 243.680120] Not tainted 5.15.0-rc1-autotest #5
[ 243.680123] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 243.680126] task:systemd-udevd state:D stack: 0 pid: 476 ppid: 455 flags:0x00040402
[ 243.680131] Call Trace:
[ 243.680133] [c00000000c4df620] [c0000000001c350c] __wake_up_common_lock+0xac/0x100 (unreliable)
[ 243.680139] [c00000000c4df810] [c00000000001e6c8] __switch_to+0x278/0x490
[ 243.680144] [c00000000c4df870] [c000000000c8b9bc] __schedule+0x31c/0xa20
[ 243.680148] [c00000000c4df930] [c000000000c8c128] schedule+0x68/0x120
[ 243.680152] [c00000000c4df960] [c000000000186cd4] async_synchronize_cookie_domain+0x144/0x1a0
[ 243.680158] [c00000000c4dfa10] [c00000000024f2e4] do_init_module+0x1a4/0x320
[ 243.680164] [c00000000c4dfa90] [c000000000252b88] load_module+0x2098/0x22f0
[ 243.680169] [c00000000c4dfc90] [c000000000253100] __do_sys_finit_module+0xd0/0x150
[ 243.680174] [c00000000c4dfdb0] [c00000000002ffe8] system_call_exception+0x158/0x2c0
[ 243.680178] [c00000000c4dfe10] [c00000000000c764] system_call_common+0xf4/0x258
[ 243.680184] --- interrupt: c00 at 0x7fff86a5a504
[ 243.680188] NIP: 00007fff86a5a504 LR: 00007fff86caea9c CTR: 0000000000000000
[ 243.680192] REGS: c00000000c4dfe80 TRAP: 0c00 Not tainted (5.15.0-rc1-autotest)
[ 243.680196] MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 28222244 XER: 00000000
[ 243.680204] IRQMASK: 0
[ 243.680204] GPR00: 0000000000000161 00007ffff1dcdfd0 00007fff86b47100 000000000000000e
[ 243.680204] GPR04: 0000000136b9b720 0000000000000000 000000000000000e 0000000000000000
[ 243.680204] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 243.680204] GPR12: 0000000000000000 00007fff856784a0 0000000000020000 00000001109cd5a0
[ 243.680204] GPR16: 0000000110a10130 00000001109d57d8 0000000000000000 0000000000000000
[ 243.680204] GPR20: 0000000000000000 0000000000000000 0000000136b70de0 0000000000000008
[ 243.680204] GPR24: 0000000000000007 0000000000000000 0000000000000000 0000000136b94860
[ 243.680204] GPR28: 0000000136b9b720 0000000000020000 0000000000000000 0000000136b70de0
[ 243.680244] NIP [00007fff86a5a504] 0x7fff86a5a504
[ 243.680247] LR [00007fff86caea9c] 0x7fff86caea9c
[ 243.680250] --- interrupt: c00
[ 243.680252] INFO: task systemd-udevd:478 blocked for more than 122 seconds.
[ 243.680255] Not tainted 5.15.0-rc1-autotest #5
[ 243.680258] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 243.680261] task:systemd-udevd state:D stack: 0 pid: 478 ppid: 455 flags:0x00040402
[ 243.680266] Call Trace:
[ 243.680268] [c00000000c4e7620] [c000000000000000] start_first_256B+0x0/0x38 (unreliable)
[ 243.680273] [c00000000c4e7810] [c00000000001e6c8] __switch_to+0x278/0x490
[ 243.680278] [c00000000c4e7870] [c000000000c8b9bc] __schedule+0x31c/0xa20
[ 243.680282] [c00000000c4e7930] [c000000000c8c128] schedule+0x68/0x120
[ 243.680287] [c00000000c4e7960] [c000000000186cd4] async_synchronize_cookie_domain+0x144/0x1a0
[ 243.680292] [c00000000c4e7a10] [c00000000024f2e4] do_init_module+0x1a4/0x320
[ 243.680297] [c00000000c4e7a90] [c000000000252b88] load_module+0x2098/0x22f0
[ 243.680302] [c00000000c4e7c90] [c000000000253100] __do_sys_finit_module+0xd0/0x150
[ 243.680307] [c00000000c4e7db0] [c00000000002ffe8] system_call_exception+0x158/0x2c0
[ 243.680311] [c00000000c4e7e10] [c00000000000c764] system_call_common+0xf4/0x258
[ 243.680317] --- interrupt: c00 at 0x7fff86a5a504
[ 243.680320] NIP: 00007fff86a5a504 LR: 00007fff86caea9c CTR: 0000000000000000
[ 243.680323] REGS: c00000000c4e7e80 TRAP: 0c00 Not tainted (5.15.0-rc1-autotest)
[ 243.680327] MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 28222244 XER: 00000000
[ 243.680335] IRQMASK: 0
[ 243.680335] GPR00: 0000000000000161 00007ffff1dcdfd0 00007fff86b47100 000000000000000e
[ 243.680335] GPR04: 00007fff86cbad30 0000000000000000 000000000000000e 0000000000000000
[ 243.680335] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 243.680335] GPR12: 0000000000000000 00007fff856784a0 0000000000020000 00000001109cd5a0
[ 243.680335] GPR16: 0000000110a10130 00000001109d57d8 0000000000000000 0000000000000000
[ 243.680335] GPR20: 0000000000000000 00000001109d5810 0000000136b926b0 00007ffff1dce1f0
[ 243.680335] GPR24: 0000000110a10150 0000000000020000 0000000000000000 0000000136b730e0
[ 243.680335] GPR28: 00007fff86cbad30 0000000000020000 0000000000000000 0000000136b926b0
[ 243.680374] NIP [00007fff86a5a504] 0x7fff86a5a504
[ 243.680377] LR [00007fff86caea9c] 0x7fff86caea9c
[ 243.680380] --- interrupt: c00
[ 366.559988] INFO: task eehd:137 blocked for more than 245 seconds.
[ 366.560004] Not tainted 5.15.0-rc1-autotest #5
[ 366.560007] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 366.560012] task:eehd state:D stack: 0 pid: 137 ppid: 2 flags:0x00000800
[ 366.560018] Call Trace:
[ 366.560020] [c000000008167650] [c0000000001dd368] info_print_prefix+0xa8/0x130 (unreliable)
[ 366.560029] [c000000008167840] [c00000000001e6c8] __switch_to+0x278/0x490
[ 366.560034] [c0000000081678a0] [c000000000c8b9bc] __schedule+0x31c/0xa20
[ 366.560040] [c000000008167960] [c000000000c8c128] schedule+0x68/0x120
[ 366.560045] [c000000008167990] [c0080000008ca3c4] __ipr_remove+0x10c/0x4b0 [ipr]
[ 366.560054] [c000000008167a60] [c0080000008ca828] ipr_remove+0xc0/0x110 [ipr]
[ 366.560061] [c000000008167ae0] [c000000000732bd4] pci_device_remove+0x64/0x120
[ 366.560067] [c000000008167b20] [c000000000811318] device_release_driver_internal+0x168/0x2c0
[ 366.560074] [c000000008167b60] [c000000000724618] pci_stop_bus_device+0xa8/0x100
[ 366.560079] [c000000008167ba0] [c000000000724810] pci_stop_and_remove_bus_device+0x20/0x40
[ 366.560084] [c000000008167bd0] [c000000000075e60] pci_hp_remove_devices+0x90/0x130
[ 366.560090] [c000000008167c60] [c000000000053cd8] eeh_handle_normal_event+0x7b8/0xa80
[ 366.560095] [c000000008167d40] [c000000000054de8] eeh_event_handler+0x108/0x190
[ 366.560101] [c000000008167da0] [c00000000018151c] kthread+0x18c/0x1a0
[ 366.560105] [c000000008167e10] [c00000000000cfd4] ret_from_kernel_thread+0x5c/0x64
[ 366.560112] INFO: task kworker/u48:2:207 blocked for more than 122 seconds.
[ 366.560115] Not tainted 5.15.0-rc1-autotest #5
[ 366.560118] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 366.560121] task:kworker/u48:2 state:D stack: 0 pid: 207 ppid: 2 flags:0x00000800
[ 366.560127] Workqueue: events_unbound async_run_entry_fn
[ 366.560131] Call Trace:
[ 366.560133] [c000000008bd36c0] [c000000008bd3700] 0xc000000008bd3700 (unreliable)
[ 366.560137] [c000000008bd38b0] [c00000000001e6c8] __switch_to+0x278/0x490
[ 366.560142] [c000000008bd3910] [c000000000c8b9bc] __schedule+0x31c/0xa20
[ 366.560147] [c000000008bd39d0] [c000000000c8c128] schedule+0x68/0x120
[ 366.560151] [c000000008bd3a00] [c000000000c93ee8] schedule_timeout+0x348/0x3f0
[ 366.560156] [c000000008bd3ae0] [c000000000c8e2cc] wait_for_completion+0xbc/0x270
[ 366.560161] [c000000008bd3b50] [c000000000872670] do_scan_async+0x60/0x2b0
[ 366.560167] [c000000008bd3bd0] [c000000000186a2c] async_run_entry_fn+0x5c/0x1c0
[ 366.560171] [c000000008bd3c60] [c000000000175468] process_one_work+0x288/0x570
[ 366.560177] [c000000008bd3d00] [c0000000001757c8] worker_thread+0x78/0x660
[ 366.560182] [c000000008bd3da0] [c00000000018151c] kthread+0x18c/0x1a0
[ 366.560187] [c000000008bd3e10] [c00000000000cfd4] ret_from_kernel_thread+0x5c/0x64
[ 366.560196] INFO: task systemd-udevd:476 blocked for more than 245 seconds.
[ 366.560199] Not tainted 5.15.0-rc1-autotest #5
[ 366.560202] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 366.560205] task:systemd-udevd state:D stack: 0 pid: 476 ppid: 455 flags:0x00040402
[ 366.560210] Call Trace:
[ 366.560212] [c00000000c4df620] [c0000000001c350c] __wake_up_common_lock+0xac/0x100 (unreliable)
[ 366.560218] [c00000000c4df810] [c00000000001e6c8] __switch_to+0x278/0x490
[ 366.560224] [c00000000c4df870] [c000000000c8b9bc] __schedule+0x31c/0xa20
[ 366.560228] [c00000000c4df930] [c000000000c8c128] schedule+0x68/0x120
[ 366.560232] [c00000000c4df960] [c000000000186cd4] async_synchronize_cookie_domain+0x144/0x1a0
[ 366.560238] [c00000000c4dfa10] [c00000000024f2e4] do_init_module+0x1a4/0x320
[ 366.560243] [c00000000c4dfa90] [c000000000252b88] load_module+0x2098/0x22f0
[ 366.560249] [c00000000c4dfc90] [c000000000253100] __do_sys_finit_module+0xd0/0x150
[ 366.560254] [c00000000c4dfdb0] [c00000000002ffe8] system_call_exception+0x158/0x2c0
[ 366.560258] [c00000000c4dfe10] [c00000000000c764] system_call_common+0xf4/0x258
[ 366.560264] --- interrupt: c00 at 0x7fff86a5a504
[ 366.560268] NIP: 00007fff86a5a504 LR: 00007fff86caea9c CTR: 0000000000000000
[ 366.560271] REGS: c00000000c4dfe80 TRAP: 0c00 Not tainted (5.15.0-rc1-autotest)
[ 366.560275] MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 28222244 XER: 00000000
[ 366.560284] IRQMASK: 0
[ 366.560284] GPR00: 0000000000000161 00007ffff1dcdfd0 00007fff86b47100 000000000000000e
[ 366.560284] GPR04: 0000000136b9b720 0000000000000000 000000000000000e 0000000000000000
[ 366.560284] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 366.560284] GPR12: 0000000000000000 00007fff856784a0 0000000000020000 00000001109cd5a0
[ 366.560284] GPR16: 0000000110a10130 00000001109d57d8 0000000000000000 0000000000000000
[ 366.560284] GPR20: 0000000000000000 0000000000000000 0000000136b70de0 0000000000000008
[ 366.560284] GPR24: 0000000000000007 0000000000000000 0000000000000000 0000000136b94860
[ 366.560284] GPR28: 0000000136b9b720 0000000000020000 0000000000000000 0000000136b70de0
[ 366.560324] NIP [00007fff86a5a504] 0x7fff86a5a504
[ 366.560327] LR [00007fff86caea9c] 0x7fff86caea9c
[ 366.560329] --- interrupt: c00
[ 366.560332] INFO: task systemd-udevd:478 blocked for more than 245 seconds.
[ 366.560335] Not tainted 5.15.0-rc1-autotest #5
[ 366.560338] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 366.560341] task:systemd-udevd state:D stack: 0 pid: 478 ppid: 455 flags:0x00040402
[ 366.560345] Call Trace:
[ 366.560347] [c00000000c4e7620] [c000000000000000] start_first_256B+0x0/0x38 (unreliable)
[ 366.560353] [c00000000c4e7810] [c00000000001e6c8] __switch_to+0x278/0x490
[ 366.560357] [c00000000c4e7870] [c000000000c8b9bc] __schedule+0x31c/0xa20
[ 366.560362] [c00000000c4e7930] [c000000000c8c128] schedule+0x68/0x120
[ 366.560366] [c00000000c4e7960] [c000000000186cd4] async_synchronize_cookie_domain+0x144/0x1a0
[ 366.560372] [c00000000c4e7a10] [c00000000024f2e4] do_init_module+0x1a4/0x320
[ 366.560376] [c00000000c4e7a90] [c000000000252b88] load_module+0x2098/0x22f0
[ 366.560381] [c00000000c4e7c90] [c000000000253100] __do_sys_finit_module+0xd0/0x150
[ 366.560386] [c00000000c4e7db0] [c00000000002ffe8] system_call_exception+0x158/0x2c0
[ 366.560391] [c00000000c4e7e10] [c00000000000c764] system_call_common+0xf4/0x258
[ 366.560396] --- interrupt: c00 at 0x7fff86a5a504
[ 366.560399] NIP: 00007fff86a5a504 LR: 00007fff86caea9c CTR: 0000000000000000
[ 366.560403] REGS: c00000000c4e7e80 TRAP: 0c00 Not tainted (5.15.0-rc1-autotest)
[ 366.560407] MSR: 800000000000d033 <SF,EE,PR,ME,IR,DR,RI,LE> CR: 28222244 XER: 00000000
[ 366.560414] IRQMASK: 0
[ 366.560414] GPR00: 0000000000000161 00007ffff1dcdfd0 00007fff86b47100 000000000000000e
[ 366.560414] GPR04: 00007fff86cbad30 0000000000000000 000000000000000e 0000000000000000
[ 366.560414] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 366.560414] GPR12: 0000000000000000 00007fff856784a0 0000000000020000 00000001109cd5a0
[ 366.560414] GPR16: 0000000110a10130 00000001109d57d8 0000000000000000 0000000000000000
[ 366.560414] GPR20: 0000000000000000 00000001109d5810 0000000136b926b0 00007ffff1dce1f0
[ 366.560414] GPR24: 0000000110a10150 0000000000020000 0000000000000000 0000000136b730e0
[ 366.560414] GPR28: 00007fff86cbad30 0000000000020000 0000000000000000 0000000136b926b0
[ 366.560453] NIP [00007fff86a5a504] 0x7fff86a5a504
[ 366.560456] LR [00007fff86caea9c] 0x7fff86caea9c
[ 366.560459] --- interrupt: c00