Re: [Query]: trace-cmd crashing with '-e syscalls' on 3.13-rc1

[Steven Rostedt]

On Tue, 24 Dec 2013 16:58:51 +0530
Viresh Kumar <viresh.kumar@xxxxxxxxxx> wrote:

> Hi Steven,
> 
> I am trying to trace some stuff on Samsung's Arndale Board (ARM)
> with 3.13-rc1 and seeing these crashes/segmentation faults
> whenever I try to trace syscalls.
> 
> Is this a know issue? Or something new?
> 
> trace-cmd version 2.2.0

Can you also produce this with just 
"echo 1 > /sys/kernel/debug/tracing/events/syscalls/enable"?


> 
> 
> [  299.998874] Unable to handle kernel NULL pointer dereference at
> virtual address 0000001b
> [  300.005554] pgd = ed9fc000
> [  300.008201] [0000001b] *pgd=6eb1e831, *pte=00000000, *ppte=00000000
> [  300.014428] Internal error: Oops: 17 [#1] PREEMPT SMP ARM
> [  300.019812] Modules linked in:
> [  300.022847] CPU: 0 PID: 2172 Comm: is-cpu-isolated Not tainted
> 3.13.0-rc1-00090-gd6eaf98-dirty #141

Are you running some isolated CPU test? Do you have NO_HZ_FULL enabled
too?

I Cc'd Frederic too.

Note, I'm on PTO for two weeks (coming back Jan 6th), so I wont be
debugging this any time soon. My wife has me debugging house issues
(putting in a new floor for laundry room, painting, re-arranging
furniture, etc). (I have to sneak to the computer in order to goof off
and respond to this email ;-)

-- Steve

> [  300.031880] task: ee9a9680 ti: eddfc000 task.ti: eddfc000
> [  300.037263] PC is at ftrace_syscall_enter+0x44/0x17c
> [  300.042208] LR is at syscall_trace_enter+0xf8/0x138
> [  300.047066] pc : [<c00ad654>]    lr : [<c0010eb0>]    psr: a0000013
> [  300.047066] sp : eddfdf40  ip : eddfdf80  fp : eddfdf7c
> [  300.058529] r10: 00001400  r9 : eddfc000  r8 : c000eea4
> [  300.063731] r7 : eddfdfb0  r6 : eddfc000  r5 : ffffffff  r4 : 000f0005
> [  300.070242] r3 : eddfc000  r2 : 000f001d  r1 : eddfdfb0  r0 : c06cb2f8
> [  300.076754] Flags: NzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
> [  300.083872] Control: 10c5387d  Table: 6d9fc06a  DAC: 00000015
> [  300.089599] Process is-cpu-isolated (pid: 2172, stack limit = 0xeddfc240)
> [  300.096370] Stack: (0xeddfdf40 to 0xeddfe000)
> [  300.100711] df40: ef1c5910 00000000 eddfdf8c 00000022 00000003
> ede2c490 000f0005 eddfc000
> [  300.108872] df60: eddfdfb0 c000eea4 eddfc000 00001400 eddfdfa4
> eddfdf80 c0010eb0 c00ad61c
> [  300.117034] df80: b6f824c0 b6f83c90 00000001 00000000 000f0005
> c000eea4 00000000 eddfdfa8
> [  300.125194] dfa0: c000ee4c c0010dc4 b6f83c90 00000001 b6f824c0
> b6f88050 b6f82b98 b6f824c0
> [  300.133352] dfc0: b6f83c90 00000001 00000000 000f0005 00000001
> 00000000 0009688c b6f88958
> [  300.141511] dfe0: 000f0005 bef66460 b6f698fb b6f79966 80000030
> b6f824c0 00000000 00000000
> [  300.149678] [<c00ad654>] (ftrace_syscall_enter+0x44/0x17c) from
> [<c0010eb0>] (syscall_trace_enter+0xf8/0x138)
> [  300.159573] [<c0010eb0>] (syscall_trace_enter+0xf8/0x138) from
> [<c000ee4c>] (__sys_trace+0xc/0x38)
> [  300.168510] Code: e2842018 e7905102 e3550000 0a000038 (e595201c)
> [  300.174595] ---[ end trace f8b3633d964e9e60 ]---
> [  300.179189] note: is-cpu-isolated[2172] exited with preempt_count 1
> Segmentation fault
> [  300.956933] Unable to handle kernel NULL pointer dereference at
> virtual address 0000001b
> [  300.963587] pgd = ed9fc000
> [  300.966234] [0000001b] *pgd=6ebd3831, *pte=00000000, *ppte=00000000
> [  300.972483] Internal error: Oops: 17 [#2] PREEMPT SMP ARM
> [  300.977867] Modules linked in:
> [  300.980903] CPU: 0 PID: 2173 Comm: trace-cmd Tainted: G      D
> 3.13.0-rc1-00090-gd6eaf98-dirty #141
> [  300.990372] task: ee9aa1c0 ti: eddfe000 task.ti: eddfe000
> [  300.995752] PC is at ftrace_syscall_enter+0x44/0x17c
> [  301.000700] LR is at syscall_trace_enter+0xf8/0x138
> [  301.005557] pc : [<c00ad654>]    lr : [<c0010eb0>]    psr: a0000013
> [  301.005557] sp : eddfff40  ip : eddfff80  fp : eddfff7c
> [  301.017022] r10: 00001400  r9 : eddfe000  r8 : c000eea4
> [  301.022222] r7 : eddfffb0  r6 : eddfe000  r5 : ffffffff  r4 : 000f0005
> [  301.028733] r3 : eddfe000  r2 : 000f001d  r1 : eddfffb0  r0 : c06cb2f8
> [  301.035244] Flags: NzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
> [  301.042362] Control: 10c5387d  Table: 6d9fc06a  DAC: 00000015
> [  301.048089] Process trace-cmd (pid: 2173, stack limit = 0xeddfe240)
> [  301.054340] Stack: (0xeddfff40 to 0xede00000)
> [  301.058679] ff40: eddfffb0 ef09a280 eddfff8c eddfff58 c00ad2d0
> ede2c490 000f0005 eddfe000
> [  301.066843] ff60: eddfffb0 c000eea4 eddfe000 00001400 eddfffa4
> eddfff80 c0010eb0 c00ad61c
> [  301.075002] ff80: 000bc320 000bc318 00000008 00000010 000f0005
> c000eea4 00000000 eddfffa8
> [  301.083162] ffa0: c000ee4c c0010dc4 000bc318 00000008 000be4c0
> 000b2018 00000000 000bc320
> [  301.091321] ffc0: 000bc318 00000008 00000010 000f0005 00000038
> 000be4c0 00000004 00000001
> [  301.099480] ffe0: 000f0005 be9cc6d8 0002ed43 0002ef66 40000030
> 000be4c0 6f7fd821 6f7fdc21
> [  301.107645] [<c00ad654>] (ftrace_syscall_enter+0x44/0x17c) from
> [<c0010eb0>] (syscall_trace_enter+0xf8/0x138)
> [  301.117540] [<c0010eb0>] (syscall_trace_enter+0xf8/0x138) from
> [<c000ee4c>] (__sys_trace+0xc/0x38)
> [  301.126477] Code: e2842018 e7905102 e3550000 0a000038 (e595201c)
> [  301.132565] ---[ end trace f8b3633d964e9e61 ]---
> [  301.137158] note: trace-cmd[2173] exited with preempt_count 1
> Segmentation fault
> [  301.922253] Unable to handle kernel NULL pointer dereference at
> virtual address 0000001b
> [  301.928899] pgd = eda00000
> [  301.931556] [0000001b] *pgd=6d9f2831, *pte=00000000, *ppte=00000000
> [  301.937803] Internal error: Oops: 17 [#3] PREEMPT SMP ARM
> [  301.943187] Modules linked in:
> [  301.946223] CPU: 0 PID: 2174 Comm: cat Tainted: G      D
> 3.13.0-rc1-00090-gd6eaf98-dirty #141
> [  301.955172] task: ee9aa580 ti: ed9f8000 task.ti: ed9f8000
> [  301.960552] PC is at ftrace_syscall_enter+0x44/0x17c
> [  301.965500] LR is at syscall_trace_enter+0xf8/0x138
> [  301.970356] pc : [<c00ad654>]    lr : [<c0010eb0>]    psr: a0000013
> [  301.970356] sp : ed9f9f40  ip : ed9f9f80  fp : ed9f9f7c
> [  301.981820] r10: 00001400  r9 : ed9f8000  r8 : c000eea4
> [  301.987021] r7 : ed9f9fb0  r6 : ed9f8000  r5 : ffffffff  r4 : 000f0005
> [  301.993532] r3 : ed9f8000  r2 : 000f001d  r1 : ed9f9fb0  r0 : c06cb2f8
> [  302.000044] Flags: NzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
> [  302.007161] Control: 10c5387d  Table: 6da0006a  DAC: 00000015
> [  302.012888] Process cat (pid: 2174, stack limit = 0xed9f8240)
> [  302.018617] Stack: (0xed9f9f40 to 0xed9fa000)
> [  302.022958] 9f40: ed9f9fb0 00000000 ed9f9f8c 00000022 00000003
> ede2c490 000f0005 ed9f8000
> [  302.031122] 9f60: ed9f9fb0 c000eea4 ed9f8000 00001400 ed9f9fa4
> ed9f9f80 c0010eb0 c00ad61c
> [  302.039281] 9f80: b6f6d840 b6f6d178 00000001 00000000 000f0005
> c000eea4 00000000 ed9f9fa8
> [  302.047440] 9fa0: c000ee4c c0010dc4 b6f6d178 00000001 b6f6d840
> b6f72050 b6f6df18 b6f6d840
> [  302.055599] 9fc0: b6f6d178 00000001 00000000 000f0005 00000001
> 00000000 0000e67c b6f72958
> [  302.063758] 9fe0: 000f0005 beaf6480 b6f538fb b6f63966 80000030
> b6f6d840 00000000 00000000
> [  302.071922] [<c00ad654>] (ftrace_syscall_enter+0x44/0x17c) from
> [<c0010eb0>] (syscall_trace_enter+0xf8/0x138)
> [  302.081819] [<c0010eb0>] (syscall_trace_enter+0xf8/0x138) from
> [<c000ee4c>] (__sys_trace+0xc/0x38)
> [  302.090756] Code: e2842018 e7905102 e3550000 0a000038 (e595201c)
> [  302.096845] ---[ end trace f8b3633d964e9e62 ]---
> [  302.101436] note: cat[2174] exited with preempt_count 1
> Segmentation fault
> [  302.881881] Unable to handle kernel NULL pointer dereference at
> virtual address 0000001b
> [  302.888532] pgd = ed9fc000
> [  302.891180] [0000001b] *pgd=6eb1e831, *pte=00000000, *ppte=00000000
> [  302.897429] Internal error: Oops: 17 [#4] PREEMPT SMP ARM
> [  302.902813] Modules linked in:
> [  302.905849] CPU: 0 PID: 2175 Comm: trace-cmd Tainted: G      D
> 3.13.0-rc1-00090-gd6eaf98-dirty #141
> [  302.915319] task: ee9aa1c0 ti: ee492000 task.ti: ee492000
> [  302.920698] PC is at ftrace_syscall_enter+0x44/0x17c
> [  302.925645] LR is at syscall_trace_enter+0xf8/0x138
> [  302.930503] pc : [<c00ad654>]    lr : [<c0010eb0>]    psr: a0000013
> [  302.930503] sp : ee493f40  ip : ee493f80  fp : ee493f7c
> [  302.941967] r10: 00001400  r9 : ee492000  r8 : c000eea4
> [  302.947169] r7 : ee493fb0  r6 : ee492000  r5 : ffffffff  r4 : 000f0005
> [  302.953679] r3 : ee492000  r2 : 000f001d  r1 : ee493fb0  r0 : c06cb2f8
> [  302.960190] Flags: NzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
> [  302.967307] Control: 10c5387d  Table: 6d9fc06a  DAC: 00000015
> [  302.973035] Process trace-cmd (pid: 2175, stack limit = 0xee492240)
> [  302.979286] Stack: (0xee493f40 to 0xee494000)
> [  302.983625] 3f40: ee493fb0 ef09a280 ee493f8c ee493f58 c00ad2d0
> ede2c490 000f0005 ee492000
> [  302.991789] 3f60: ee493fb0 c000eea4 ee492000 00001400 ee493fa4
> ee493f80 c0010eb0 c00ad61c
> [  302.999948] 3f80: 000bc320 000bc318 00000008 00000010 000f0005
> c000eea4 00000000 ee493fa8
> [  303.008108] 3fa0: c000ee4c c0010dc4 000bc318 00000008 000be4c0
> 000b2018 00000000 000bc320
> [  303.016267] 3fc0: 000bc318 00000008 00000010 000f0005 00000038
> 000be4c0 00000004 00000001
> [  303.024426] 3fe0: 000f0005 be8c76c8 0002ed43 0002ef66 40000030
> 000be4c0 00000000 00000000
> [  303.032591] [<c00ad654>] (ftrace_syscall_enter+0x44/0x17c) from
> [<c0010eb0>] (syscall_trace_enter+0xf8/0x138)
> [  303.042487] [<c0010eb0>] (syscall_trace_enter+0xf8/0x138) from
> [<c000ee4c>] (__sys_trace+0xc/0x38)
> [  303.051423] Code: e2842018 e7905102 e3550000 0a000038 (e595201c)
> [  303.057512] ---[ end trace f8b3633d964e9e63 ]---
> [  303.062103] note: trace-cmd[2175] exited with preempt_count 1
> Segmentation fault
> [  303.842447] Unable to handle kernel NULL pointer dereference at
> virtual address 0000001b
> [  303.849096] pgd = eddfc000
> [  303.851752] [0000001b] *pgd=6d9f1831, *pte=00000000, *ppte=00000000
> [  303.858002] Internal error: Oops: 17 [#5] PREEMPT SMP ARM
> [  303.863386] Modules linked in:
> [  303.866422] CPU: 0 PID: 2176 Comm: is-cpu-isolated Tainted: G
> D      3.13.0-rc1-00090-gd6eaf98-dirty #141
> [  303.876412] task: ee9a9680 ti: ed9fa000 task.ti: ed9fa000
> [  303.881791] PC is at ftrace_syscall_enter+0x44/0x17c
> [  303.886738] LR is at syscall_trace_enter+0xf8/0x138
> [  303.891596] pc : [<c00ad654>]    lr : [<c0010eb0>]    psr: a0000013
> [  303.891596] sp : ed9fbf40  ip : ed9fbf80  fp : ed9fbf7c
> [  303.903059] r10: 00001400  r9 : ed9fa000  r8 : c000eea4
> [  303.908261] r7 : ed9fbfb0  r6 : ed9fa000  r5 : ffffffff  r4 : 000f0005
> [  303.914772] r3 : ed9fa000  r2 : 000f001d  r1 : ed9fbfb0  r0 : c06cb2f8
> [  303.921283] Flags: NzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
> [  303.928400] Control: 10c5387d  Table: 6ddfc06a  DAC: 00000015
> [  303.934128] Process is-cpu-isolated (pid: 2176, stack limit = 0xed9fa240)
> [  303.940900] Stack: (0xed9fbf40 to 0xed9fc000)
> [  303.945239] bf40: ed9fbfb0 00000000 ed9fbf8c 00000022 00000003
> ede2c490 000f0005 ed9fa000
> [  303.953403] bf60: ed9fbfb0 c000eea4 ed9fa000 00001400 ed9fbfa4
> ed9fbf80 c0010eb0 c00ad61c
> [  303.961562] bf80: b6f174c0 b6f18c90 00000001 00000000 000f0005
> c000eea4 00000000 ed9fbfa8
> [  303.969721] bfa0: c000ee4c c0010dc4 b6f18c90 00000001 b6f174c0
> b6f1d050 b6f17b98 b6f174c0
> [  303.977880] bfc0: b6f18c90 00000001 00000000 000f0005 00000001
> 00000000 0009688c b6f1d958
> [  303.986039] bfe0: 000f0005 be90c460 b6efe8fb b6f0e966 80000030
> b6f174c0 6f7fd821 6f7fdc21
> [  303.994204] [<c00ad654>] (ftrace_syscall_enter+0x44/0x17c) from
> [<c0010eb0>] (syscall_trace_enter+0xf8/0x138)
> [  304.004101] [<c0010eb0>] (syscall_trace_enter+0xf8/0x138) from
> [<c000ee4c>] (__sys_trace+0xc/0x38)
> [  304.013037] Code: e2842018 e7905102 e3550000 0a000038 (e595201c)
> [  304.019125] ---[ end trace f8b3633d964e9e64 ]---
> [  304.023717] note: is-cpu-isolated[2176] exited with preempt_count 1
> Segmentation fault
> Saved trace in: dft
> [  304.807095] Unable to handle kernel NULL pointer dereference at
> virtual address 0000001b
> [  304.813737] pgd = ed9fc000
> [  304.816395] [0000001b] *pgd=6eb1e831, *pte=00000000, *ppte=00000000
> [  304.822644] Internal error: Oops: 17 [#6] PREEMPT SMP ARM
> [  304.828029] Modules linked in:
> [  304.831065] CPU: 0 PID: 2177 Comm: sync Tainted: G      D
> 3.13.0-rc1-00090-gd6eaf98-dirty #141
> [  304.840100] task: ee9aa1c0 ti: ee492000 task.ti: ee492000
> [  304.845479] PC is at ftrace_syscall_enter+0x44/0x17c
> [  304.850426] LR is at syscall_trace_enter+0xf8/0x138
> [  304.855284] pc : [<c00ad654>]    lr : [<c0010eb0>]    psr: a0000013
> [  304.855284] sp : ee493f40  ip : ee493f80  fp : ee493f7c
> [  304.866749] r10: 00001400  r9 : ee492000  r8 : c000eea4
> [  304.871950] r7 : ee493fb0  r6 : ee492000  r5 : ffffffff  r4 : 000f0005
> [  304.878461] r3 : ee492000  r2 : 000f001d  r1 : ee493fb0  r0 : c06cb2f8
> [  304.884972] Flags: NzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
> [  304.892089] Control: 10c5387d  Table: 6d9fc06a  DAC: 00000015
> [  304.897816] Process sync (pid: 2177, stack limit = 0xee492240)
> [  304.903632] Stack: (0xee493f40 to 0xee494000)
> [  304.907973] 3f40: ee493fb0 00000000 ee493f8c 00000022 00000003
> ede2c490 000f0005 ee492000
> [  304.916137] 3f60: ee493fb0 c000eea4 ee492000 00001400 ee493fa4
> ee493f80 c0010eb0 c00ad61c
> [  304.924296] 3f80: b6fa8840 b6fa8178 00000001 00000000 000f0005
> c000eea4 00000000 ee493fa8
> [  304.932455] 3fa0: c000ee4c c0010dc4 b6fa8178 00000001 b6fa8840
> b6fad050 b6fa8f18 b6fa8840
> [  304.940615] 3fc0: b6fa8178 00000001 00000000 000f0005 00000001
> 00000000 0000b0d8 b6fad958
> [  304.948773] 3fe0: 000f0005 bebbb4a0 b6f8e8fb b6f9e966 80000030
> b6fa8840 00000000 00000000
> [  304.956937] [<c00ad654>] (ftrace_syscall_enter+0x44/0x17c) from
> [<c0010eb0>] (syscall_trace_enter+0xf8/0x138)
> [  304.966834] [<c0010eb0>] (syscall_trace_enter+0xf8/0x138) from
> [<c000ee4c>] (__sys_trace+0xc/0x38)
> [  304.975771] Code: e2842018 e7905102 e3550000 0a000038 (e595201c)
> [  304.981859] ---[ end trace f8b3633d964e9e65 ]---
> [  304.986451] note: sync[2177] exited with preempt_count 1
> Segmentation fault
> 
> --
> viresh

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/