Re: [3.4-rc1 crash]: NULL pointer deref infs/sysfs/group.c:create_files -- sysctl related?

From: Bruno PrÃmont
Date: Mon Apr 02 2012 - 15:34:44 EST

[adding a few perf people to CC as might originate from perf]

On Mon, 02 April 2012 Eric W. Biederman wrote:
> Bruno PrÃmont writes:
> > On Mon, 2 Apr 2012 16:27:16 Bruno PrÃmont wrote:
> >> Trying to boot a freshly built 3.4-rc1 (x86_64) kernel I'm getting the following
> >> trace (server is HP Proliant G4):
> >>
> >> [ 0.986317] BUG: unable to handle kernel NULL pointer dereference at (null)
> >> [ 0.990542] IP: [<ffffffff81152673>] internal_create_group+0x83/0x1a0
> >> [ 0.993693] PGD 0
> >> [ 0.994682] Oops: 0000 [#1] SMP
> >> [ 0.996198] CPU 0
> >> [ 0.996198] Modules linked in:
> >> [ 0.996198]
> >> [ 0.996198] Pid: 1, comm: swapper/0 Not tainted 3.4.0-rc1-x86_64 #3 HP ProLiant DL360 G4
> >> [ 0.996198] RIP: 0010:[<ffffffff81152673>] [<ffffffff81152673>] internal_create_group+0x83/0x1a0
> >> [ 0.996198] RSP: 0018:ffff88019485fd70 EFLAGS: 00010202
> >> [ 0.996198] RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000001
> >> [ 0.996198] RDX: ffff880192e99908 RSI: ffff880192e99630 RDI: ffffffff81a26c60
> >> [ 0.996198] RBP: ffff88019485fdc0 R08: 0000000000000000 R09: 0000000000000000
> >> [ 0.996198] R10: ffff880192e99908 R11: 0000000000000000 R12: ffffffff81a16a00
> >> [ 0.996198] R13: ffff880192e99908 R14: ffffffff81a16900 R15: 0000000000000000
> >> [ 0.996198] FS: 0000000000000000(0000) GS:ffff88019bc00000(0000) knlGS:0000000000000000
> >> [ 0.996198] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> >> [ 0.996198] CR2: 0000000000000000 CR3: 0000000001a0c000 CR4: 00000000000007f0
> >> [ 0.996198] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> >> [ 0.996198] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> >> [ 0.996198] Process swapper/0 (pid: 1, threadinfo ffff88019485e000, task ffff880194878000)
> >> [ 0.996198] Stack:
> >> [ 0.996198] ffff88019485fdd0 ffff880192da9d60 0000000000000000 ffff880192e99908
> >> [ 0.996198] ffff880192e995d8 0000000000000001 ffffffff81a16a00 ffff880192da9d60
> >> [ 0.996198] 0000000000000000 0000000000000000 ffff88019485fdd0 ffffffff811527be
> >> [ 0.996198] Call Trace:
> >> [ 0.996198] [<ffffffff811527be>] sysfs_create_group+0xe/0x10
> >> [ 0.996198] [<ffffffff81376ca6>] device_add_groups+0x46/0x80
> >> [ 0.996198] [<ffffffff81377d3d>] device_add+0x46d/0x6a0
> >> [ 0.996198] [<ffffffff81377891>] ? device_private_init+0x51/0x90
> >> [ 0.996198] [<ffffffff81a98975>] ? utsname_sysctl_init+0x14/0x14
> >> [ 0.996198] [<ffffffff810a7228>] pmu_dev_alloc+0x98/0xe0
> >> [ 0.996198] [<ffffffff81a98975>] ? utsname_sysctl_init+0x14/0x14
> >> [ 0.996198] [<ffffffff81a989c0>] perf_event_sysfs_init+0x4b/0x9a
> >> [ 0.996198] [<ffffffff810002ad>] do_one_initcall+0x3d/0x170
> >> [ 0.996198] [<ffffffff81a85cbd>] kernel_init+0x12d/0x1be
> >> [ 0.996198] [<ffffffff81a85505>] ? rdinit_setup+0x28/0x28
> >> [ 0.996198] [<ffffffff815f3714>] kernel_thread_helper+0x4/0x10
> >> [ 0.996198] [<ffffffff81a85b90>] ? start_kernel+0x373/0x373
> >> [ 0.996198] [<ffffffff815f3710>] ? gs_change+0xb/0xb
> >> [ 0.996198] Code: ff 85 c0 0f 85 bc 00 00 00 4c 8b 6d c8 4d 85 ed 74 15 41 8b 45 00 85 c0 0f 84 0b 01 00 00 f0 41 ff 45 00 4c 8b 6d c8 49 8b 5e 10 <48> 8b 03 48 85 c0 74 71 45 31 e4 eb 44 49 8b 46 08 48 85 c0 74
> >> [ 0.996198] RIP [<ffffffff81152673>] internal_create_group+0x83/0x1a0
> >> [ 0.996198] RSP <ffff88019485fd70>
> >> [ 0.996198] CR2: 0000000000000000
> >> [ 1.131357] ---[ end trace 319c95c486d7d9cd ]---
> >> [ 1.133676] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000009
> >> [ 1.133677]
> >
> > The patch below works around it and leaves exactly one trace for WARN_ON() matching
> > above BUG.
> > With it, system boots to userspace.
> >
> > Thanks,
> > Bruno
> >
> > ---
> > diff --git a/fs/sysfs/group.c b/fs/sysfs/group.c
> > index dd1701c..0040ff2 100644
> > --- a/fs/sysfs/group.c
> > +++ b/fs/sysfs/group.c
> > @@ -32,7 +32,8 @@ static int create_files(struct sysfs_dirent *dir_sd, struct kobject *kobj,
> > struct attribute *const* attr;
> > int error = 0, i;
> >
> > - for (i = 0, attr = grp->attrs; *attr && !error; i++, attr++) {
> > + WARN_ON(!grp->attrs);
> > + for (i = 0, attr = grp->attrs; attr && *attr && !error; i++, attr++) {
> > umode_t mode = 0;
> >
> > /* in update mode, we're changing the permissions or
>
> Sysfs has not changed in this area from 3.3.
>
> The sysctl in your backtrace looks like left over addresses on the stack.
>
> The backtrack indicates this is something perf related going wonky.
>
> I would suggest you try disabling your perf related options one by one
> until the broken one shows up. Or possibly just initially disable perf.

Well, I didn't enable perf, all perf-related options that are enabled
are selected by X86!

Symbol: HAVE_PERF_EVENTS [=y]
Type : boolean
Selected by: X86 [=y]

Symbol: PERF_EVENTS [=y]
Type : boolean
Prompt: Kernel performance events and counters
Defined at init/Kconfig:1157
Depends on: HAVE_PERF_EVENTS [=y]
Location:
-> General setup
-> Kernel Performance Events And Counters
Selects: ANON_INODES [=y] && IRQ_WORK [=y]
Selected by: X86 [=y] || KVM [=n] && VIRTUALIZATION [=n] && HAVE_KVM [=y] && PCI [=y] && NET [=y]

Symbol: HAVE_PERF_EVENTS_NMI [=y]
Type : boolean
Selected by: X86 [=y]

> This looks like one of those crazy things where something registers
> with the perf subsystem, and then perf later registers it with sysfs,
> and whatever was registered did not have set the needed group attrs.


>From quick-reading kernel/events/core.c which contains perf_event_sysfs_init()
and pmu_dev_alloc() and commits from v3.3..v3.4 for that file
commit 0c9d42ed4cee2aa1dfc3a260b741baae8615744f (perf, x86: Provide means
for disabling userspace RDPMC) by Peter looks like a possible
candidate or at least startpoint:

diff --git a/kernel/events/core.c b/kernel/events/core.c
index 05affc3..dcd4049 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -5505,6 +5505,7 @@ static int pmu_dev_alloc(struct pmu *pmu)
if (!pmu->dev)
goto out;

+ pmu->dev->groups = pmu->attr_groups;
device_initialize(pmu->dev);
ret = dev_set_name(pmu->dev, "%s", pmu->name);
if (ret)

Will try bisecting corresponding merge tomorrow when I have full access to affected
system.

Thanks,
Bruno
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/