User Stack Tracer Causes Crash

[He Zhe]

Hi,

User stack tracer causes crash and hang since the following commit till now(5.1-rc1).

c3bc8fd637a9 ("tracing: Centralize preemptirq tracepoints and unify their usage")

echo 1 > /proc/sys/kernel/stack_tracer_enabled
echo userstacktrace > /sys/kernel/debug/tracing/trace_options
echo 1 > /sys/kernel/debug/tracing/events/enable
dmesg

login[269]: segfault at 80 ip 00007f7e847edc19 sp 00007ffcc8cefdc0 error 7 in libc-2.29.so[7f7e8478e000+142000]
Code: ff ff 0f 1f 80 00 00 00 00 4a 8d 0c e0 48 8b 51 40 48 85 d2 0f 84 2a ff ff ff 48 81 fb ff 03 00 00 0f 87 ba 01 00 00 48 8b 32 <48> 89 71 40 42 80 2c 20 01 48 c7 42 08 00 00 00 00 48 83 c4 08 48
systemd[1]: segfault at b ip 00007ff15b8a8420 sp 00007ffc6eaab890 error 7 in libc-2.29.so[7ff15b7a1000+142000]
Code: b6 8f 08 00 ff 25 a0 71 08 00 48 83 ec 08 be 01 00 00 00 31 c0 83 3d 1e de 08 00 00 74 0c f0 0f b1 35 1c df 08 00 75 0b eb 23 <0f> b1 35 11 df 08 00 74 1a 48 8d 3d 08 df 08 00 48 81 ec 80 00 00
systemd[1]: segfault at 0 ip 00007ff15bab40db sp 00007ffc6eaaa7f0 error 7 in libsystemd-shared-241.so[7ff15ba2c000+12f000]
Code: cb d4 f7 ff 48 83 c4 20 44 8b 54 24 0c eb b3 41 57 41 56 4d 89 ce 41 55 4d 89 c5 41 54 55 89 f5 53 89 fb 48 81 ec 38 08 00 00 <48> 89 54 24 08 4c 8b bc 24 70 08 00 00 89 4c 24 18 64 48 8b 04 25
printk: systemd: 30 output lines suppressed due to ratelimiting
Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
CPU: 5 PID: 1 Comm: systemd Not tainted 5.1.0-rc1-yocto-standard+ #2
Hardware name: Intel Corporation Broadwell Client platform/Basking Ridge, BIOS BDW-E2R1.86C.0118.R01.1503110618 03/11/2015
Call Trace:
Âdump_stack+0x67/0x95
Âpanic+0xfd/0x282
Â? do_exit+0xe34/0xf30
Âdo_exit+0xf24/0xf30
Â? do_exit+0x5/0xf30
Âdo_group_exit+0x5c/0xd0
Âget_signal+0x18e/0xa40
Âdo_signal+0x37/0x830
Âexit_to_usermode_loop+0x78/0xf0
Âprepare_exit_to_usermode+0xa0/0x100
Â? page_fault+0x8/0x30
Âretint_user+0x8/0x18
RIP: 0033:0x7ff15bab40db
Code: Bad RIP value.
RSP: 002b:00007ffc6eaaa7f0 EFLAGS: 00010206
RAX: 00007ffc6eaab070 RBX: 0000000000000000 RCX: 00000000000000d8
RDX: 0000559989da74d5 RSI: 0000000000000000 RDI: 0000000000000000
RBP: 0000000000000000 R08: 0000559989daa4b5 R09: 0000559989da8150
R10: 0000000000000004 R11: 0000000000000246 R12: 000000000000000b
R13: 0000559989daa4b5 R14: 0000559989da8150 R15: 000000000000000b
Kernel Offset: 0x33200000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]---
------------[ cut here ]------------
sched: Unexpected reschedule of offline CPU#1!
WARNING: CPU: 5 PID: 1 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x95/0xc0
Modules linked in:
CPU: 5 PID: 1 Comm: systemd Not tainted 5.1.0-rc1-yocto-standard+ #2
Hardware name: Intel Corporation Broadwell Client platform/Basking Ridge, BIOS BDW-E2R1.86C.0118.R01.1503110618 03/11/2015
RIP: 0010:native_smp_send_reschedule+0x95/0xc0
Code: 5d 5d c3 b9 01 00 00 00 31 d2 be 01 00 00 00 48 c7 c7 b8 bc b6 b5 e8 0a 85 13 00 44 89 e6 48 c7 c7 c8 7f 78 b5 e8 eb c5 02 00 <0f> 0b b9 01 00 00 00 31 d2 be 01 00 00 00 48 c7 c7 88 bc b6 b5 e8
RSP: 0018:ffff9434f8d43c38 EFLAGS: 00010082
RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffffb5b83e98
RBP: ffff9434f8d43c50 R08: 0000000000010004 R09: 00000000000002bf
R10: ffff9434f8d43b70 R11: 00000000000002be R12: 0000000000000001
R13: 0000000000000001 R14: ffff9434f8d43d28 R15: ffff9434f0570000
FS:Â 00007ff15b6b6840(0000) GS:ffff9434f8d40000(0000) knlGS:0000000000000000
CS:Â 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff15bab40b1 CR3: 000000007220e005 CR4: 00000000003606e0
Call Trace:
Â<IRQ>
Âresched_curr+0xac/0x180
Âcheck_preempt_curr+0x56/0xb0
Âttwu_do_wakeup.isra.17+0x1e/0x270
Âttwu_do_activate+0x78/0x90
Âtry_to_wake_up+0x243/0x5c0
Â? default_wake_function+0x5/0x20
Âdefault_wake_function+0x12/0x20
Âautoremove_wake_function+0x12/0x40
Â__wake_up_common+0x8c/0x130
Â__wake_up_common_lock+0x80/0xc0
Â__wake_up+0x13/0x20
Âwake_up_klogd_work_func+0x4c/0x80
Âirq_work_run_list+0x6c/0x90
Â? tick_sched_handle.isra.5+0x50/0x50
Âirq_work_tick+0x55/0x60
Âupdate_process_times+0x42/0x60
Âtick_sched_handle.isra.5+0x34/0x50
Âtick_sched_timer+0x40/0xa0
Â__hrtimer_run_queues+0x175/0x450
Âhrtimer_interrupt+0x141/0x290
Âsmp_apic_timer_interrupt+0x8f/0x260
Âapic_timer_interrupt+0xf/0x20
Â</IRQ>
RIP: 0010:panic+0x242/0x282
Code: b0 83 3d 8a e4 bf 01 00 74 05 e8 4b c6 02 00 48 c7 c6 00 91 e7 b5 48 c7 c7 68 f7 78 b5 e8 ab 74 07 00 e8 e3 60 10 00 fb 31 db <4c> 39 eb 7c 1d 41 83 f4 01 48 8b 05 30 e4 bf 01 44 89 e7 e8 78 64
RSP: 0018:ffffb306c038fc58 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffffffffb42815d4 RDI: ffffffffb427ac5d
RBP: ffffb306c038fcd0 R08: 0000000080000000 R09: 000000000000b7cc
R10: 0000000000000944 R11: 00000000000002bc R12: 0000000000000000
R13: 0000000000000000 R14: ffff9434f5f80070 R15: ffff9434f5f80000
Â? do_exit+0xf24/0xf30
Â? panic+0x23f/0x282
Â? panic+0x23f/0x282
Â? do_exit+0xe34/0xf30
Âdo_exit+0xf24/0xf30
Â? do_exit+0x5/0xf30
Âdo_group_exit+0x5c/0xd0
Âget_signal+0x18e/0xa40
Âdo_signal+0x37/0x830
Âexit_to_usermode_loop+0x78/0xf0
Âprepare_exit_to_usermode+0xa0/0x100
Â? page_fault+0x8/0x30
Âretint_user+0x8/0x18
RIP: 0033:0x7ff15bab40db
Code: Bad RIP value.
RSP: 002b:00007ffc6eaaa7f0 EFLAGS: 00010206
RAX: 00007ffc6eaab070 RBX: 0000000000000000 RCX: 00000000000000d8
RDX: 0000559989da74d5 RSI: 0000000000000000 RDI: 0000000000000000
RBP: 0000000000000000 R08: 0000559989daa4b5 R09: 0000559989da8150
R10: 0000000000000004 R11: 0000000000000246 R12: 000000000000000b
R13: 0000559989daa4b5 R14: 0000559989da8150 R15: 000000000000000b
---[ end trace 3a4eec2e1252dd20 ]---
------------[ cut here ]------------
sched: Unexpected reschedule of offline CPU#6!
WARNING: CPU: 5 PID: 1 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x95/0xc0
Modules linked in:
CPU: 5 PID: 1 Comm: systemd Tainted: GÂÂÂÂÂÂÂ WÂÂÂÂÂÂÂÂ 5.1.0-rc1-yocto-standard+ #2
Hardware name: Intel Corporation Broadwell Client platform/Basking Ridge, BIOS BDW-E2R1.86C.0118.R01.1503110618 03/11/2015
RIP: 0010:native_smp_send_reschedule+0x95/0xc0
Code: 5d 5d c3 b9 01 00 00 00 31 d2 be 01 00 00 00 48 c7 c7 b8 bc b6 b5 e8 0a 85 13 00 44 89 e6 48 c7 c7 c8 7f 78 b5 e8 eb c5 02 00 <0f> 0b b9 01 00 00 00 31 d2 be 01 00 00 00 48 c7 c7 88 bc b6 b5 e8
RSP: 0018:ffff9434f8d43b20 EFLAGS: 00010082
RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffffb5b83e98
RBP: ffff9434f8d43b38 R08: 0000000000010006 R09: 0000000000000307
R10: ffff9434f8d43a58 R11: 0000000000000306 R12: 0000000000000006
R13: 0000000000000001 R14: ffff9434f8d43c10 R15: ffff9434f5138000
FS:Â 00007ff15b6b6840(0000) GS:ffff9434f8d40000(0000) knlGS:0000000000000000
CS:Â 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff15bab40b1 CR3: 000000007220e005 CR4: 00000000003606e0
Call Trace:
Â<IRQ>
Âresched_curr+0xac/0x180
Âcheck_preempt_curr+0x56/0xb0
Âttwu_do_wakeup.isra.17+0x1e/0x270
Âttwu_do_activate+0x78/0x90
Âtry_to_wake_up+0x243/0x5c0
Â? default_wake_function+0x5/0x20
Âdefault_wake_function+0x12/0x20
Â__wake_up_common+0x8c/0x130
Â__wake_up_common_lock+0x80/0xc0
Â__wake_up+0x13/0x20
Âep_poll_callback+0x1bb/0x350
Â__wake_up_common+0x8c/0x130
Â__wake_up_common_lock+0x80/0xc0
Â__wake_up+0x13/0x20
Âwake_up_klogd_work_func+0x4c/0x80
Âirq_work_run_list+0x6c/0x90
Â? tick_sched_handle.isra.5+0x50/0x50
Âirq_work_tick+0x55/0x60
Âupdate_process_times+0x42/0x60
Âtick_sched_handle.isra.5+0x34/0x50
Âtick_sched_timer+0x40/0xa0
Â__hrtimer_run_queues+0x175/0x450
Âhrtimer_interrupt+0x141/0x290
Âsmp_apic_timer_interrupt+0x8f/0x260
Âapic_timer_interrupt+0xf/0x20
Â</IRQ>
RIP: 0010:panic+0x242/0x282
Code: b0 83 3d 8a e4 bf 01 00 74 05 e8 4b c6 02 00 48 c7 c6 00 91 e7 b5 48 c7 c7 68 f7 78 b5 e8 ab 74 07 00 e8 e3 60 10 00 fb 31 db <4c> 39 eb 7c 1d 41 83 f4 01 48 8b 05 30 e4 bf 01 44 89 e7 e8 78 64
RSP: 0018:ffffb306c038fc58 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffffffffb42815d4 RDI: ffffffffb427ac5d
RBP: ffffb306c038fcd0 R08: 0000000080000000 R09: 000000000000b7cc
R10: 0000000000000944 R11: 00000000000002bc R12: 0000000000000000
R13: 0000000000000000 R14: ffff9434f5f80070 R15: ffff9434f5f80000
Â? do_exit+0xf24/0xf30
Â? panic+0x23f/0x282
Â? panic+0x23f/0x282
Â? do_exit+0xe34/0xf30
Âdo_exit+0xf24/0xf30
Â? do_exit+0x5/0xf30
Âdo_group_exit+0x5c/0xd0
Âget_signal+0x18e/0xa40
Âdo_signal+0x37/0x830
Âexit_to_usermode_loop+0x78/0xf0
Âprepare_exit_to_usermode+0xa0/0x100
Â? page_fault+0x8/0x30
Âretint_user+0x8/0x18
RIP: 0033:0x7ff15bab40db
Code: Bad RIP value.
RSP: 002b:00007ffc6eaaa7f0 EFLAGS: 00010206
RAX: 00007ffc6eaab070 RBX: 0000000000000000 RCX: 00000000000000d8
RDX: 0000559989da74d5 RSI: 0000000000000000 RDI: 0000000000000000
RBP: 0000000000000000 R08: 0000559989daa4b5 R09: 0000559989da8150
R10: 0000000000000004 R11: 0000000000000246 R12: 000000000000000b
R13: 0000559989daa4b5 R14: 0000559989da8150 R15: 000000000000000b
---[ end trace 3a4eec2e1252dd21 ]---
------------[ cut here ]------------
sched: Unexpected reschedule of offline CPU#2!
WARNING: CPU: 5 PID: 1 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x95/0xc0
Modules linked in:
CPU: 5 PID: 1 Comm: systemd Tainted: GÂÂÂÂÂÂÂ WÂÂÂÂÂÂÂÂ 5.1.0-rc1-yocto-standard+ #2
Hardware name: Intel Corporation Broadwell Client platform/Basking Ridge, BIOS BDW-E2R1.86C.0118.R01.1503110618 03/11/2015
RIP: 0010:native_smp_send_reschedule+0x95/0xc0
Code: 5d 5d c3 b9 01 00 00 00 31 d2 be 01 00 00 00 48 c7 c7 b8 bc b6 b5 e8 0a 85 13 00 44 89 e6 48 c7 c7 c8 7f 78 b5 e8 eb c5 02 00 <0f> 0b b9 01 00 00 00 31 d2 be 01 00 00 00 48 c7 c7 88 bc b6 b5 e8
RSP: 0018:ffff9434f8d43df8 EFLAGS: 00010082
RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffffb5b83e98
RBP: ffff9434f8d43e10 R08: 0000000000010001 R09: 0000000000000352
R10: 0000000000000000 R11: 0000000000000351 R12: 0000000000000002
R13: 0000000000000001 R14: 0000000000000002 R15: 0000000000000000
FS:Â 00007ff15b6b6840(0000) GS:ffff9434f8d40000(0000) knlGS:0000000000000000
CS:Â 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff15bab40b1 CR3: 000000007220e005 CR4: 00000000003606e0
Call Trace:
Â<IRQ>
Âkick_ilb+0xe2/0x110
Âtrigger_load_balance+0x113/0x200
Âscheduler_tick+0xa9/0xd0
Â? tick_sched_handle.isra.5+0x50/0x50
Âupdate_process_times+0x47/0x60
Âtick_sched_handle.isra.5+0x34/0x50
Âtick_sched_timer+0x40/0xa0
Â__hrtimer_run_queues+0x175/0x450
Âhrtimer_interrupt+0x141/0x290
Âsmp_apic_timer_interrupt+0x8f/0x260
Âapic_timer_interrupt+0xf/0x20
Â</IRQ>
RIP: 0010:panic+0x242/0x282
Code: b0 83 3d 8a e4 bf 01 00 74 05 e8 4b c6 02 00 48 c7 c6 00 91 e7 b5 48 c7 c7 68 f7 78 b5 e8 ab 74 07 00 e8 e3 60 10 00 fb 31 db <4c> 39 eb 7c 1d 41 83 f4 01 48 8b 05 30 e4 bf 01 44 89 e7 e8 78 64
RSP: 0018:ffffb306c038fc58 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffffffffb42815d4 RDI: ffffffffb427ac5d
RBP: ffffb306c038fcd0 R08: 0000000080000000 R09: 000000000000b7cc
R10: 0000000000000944 R11: 00000000000002bc R12: 0000000000000000
R13: 0000000000000000 R14: ffff9434f5f80070 R15: ffff9434f5f80000
Â? do_exit+0xf24/0xf30
Â? panic+0x23f/0x282
Â? panic+0x23f/0x282
Â? do_exit+0xe34/0xf30
Âdo_exit+0xf24/0xf30
Â? do_exit+0x5/0xf30
Âdo_group_exit+0x5c/0xd0
Âget_signal+0x18e/0xa40
Âdo_signal+0x37/0x830
Âexit_to_usermode_loop+0x78/0xf0
Âprepare_exit_to_usermode+0xa0/0x100
Â? page_fault+0x8/0x30
Âretint_user+0x8/0x18
RIP: 0033:0x7ff15bab40db
Code: Bad RIP value.
RSP: 002b:00007ffc6eaaa7f0 EFLAGS: 00010206
RAX: 00007ffc6eaab070 RBX: 0000000000000000 RCX: 00000000000000d8
RDX: 0000559989da74d5 RSI: 0000000000000000 RDI: 0000000000000000
RBP: 0000000000000000 R08: 0000559989daa4b5 R09: 0000559989da8150
R10: 0000000000000004 R11: 0000000000000246 R12: 000000000000000b
R13: 0000559989daa4b5 R14: 0000559989da8150 R15: 000000000000000b
---[ end trace 3a4eec2e1252dd22 ]---



Zhe