Re: crashme fault

From: Randy Dunlap
Date: Sat Sep 15 2007 - 15:55:52 EST


Linus Torvalds wrote:

On Sat, 15 Sep 2007, Randy Dunlap wrote:
Had another on recent last night (probably not helpful):

At least the original "crashme" would write its random number seeds to a logfile each time (and I made it fsync it in some versions), which meant that once a crash happened, you could re-produce it immediately (if it was reproducible at all, of course).

Does your crashme have something like that?

I tell it the "random" seed to use. I can also sets its debug level,
but when I did that yesterday, it never faulted, so I lowered it again,
them boom. Could be coincidence.


All your crashes look basically identical - I don't think there is anything new in this one, they're all the same issue. What CPU do you have - vendor, stepping, version etc - and has something else than the kernel changed in your setup lately?

Just kernel changes. CPU is dual Pentium Xeon + HT:

processor : 0
vendor_id : GenuineIntel
cpu family : 15
model : 3
model name : Intel(R) Xeon(TM) CPU 3.40GHz
stepping : 4
cpu MHz : 3400.227
cache size : 1024 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 c
lflush dts acpi mmx fxsr sse sse2 ss ht tm syscall lm constant_tsc pni monitor ds_cpl est t
m2 cid xtpr
bogomips : 6805.96
clflush size : 64
cache_alignment : 128
address sizes : 36 bits physical, 48 bits virtual
power management:

processor : 1
vendor_id : GenuineIntel
cpu family : 15
model : 3
model name : Intel(R) Xeon(TM) CPU 3.40GHz
stepping : 4
cpu MHz : 3400.227
cache size : 1024 KB
physical id : 3
siblings : 2
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 c
lflush dts acpi mmx fxsr sse sse2 ss ht tm syscall lm constant_tsc pni monitor ds_cpl est t
m2 cid xtpr
bogomips : 6800.28
clflush size : 64
cache_alignment : 128
address sizes : 36 bits physical, 48 bits virtual
power management:

processor : 2
vendor_id : GenuineIntel
cpu family : 15
model : 3
model name : Intel(R) Xeon(TM) CPU 3.40GHz
stepping : 4
cpu MHz : 3400.227
cache size : 1024 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 c
lflush dts acpi mmx fxsr sse sse2 ss ht tm syscall lm constant_tsc pni monitor ds_cpl est t
m2 cid xtpr
bogomips : 6800.72
clflush size : 64
cache_alignment : 128
address sizes : 36 bits physical, 48 bits virtual
power management:

processor : 3
vendor_id : GenuineIntel
cpu family : 15
model : 3
model name : Intel(R) Xeon(TM) CPU 3.40GHz
stepping : 4
cpu MHz : 3400.227
cache size : 1024 KB
physical id : 3
siblings : 2
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 c
lflush dts acpi mmx fxsr sse sse2 ss ht tm syscall lm constant_tsc pni monitor ds_cpl est t
m2 cid xtpr
bogomips : 6800.57
clflush size : 64
cache_alignment : 128
address sizes : 36 bits physical, 48 bits virtual
power management:



As mentioned, the crash does look like a user-level crash got reported as a kernel page fault, and while a CPU bug sounds incredibly unlikely, this does have the smell of something strange like a fault in the middle of an "iretq" or "sysretq", where part of the CPU state has already been restored - which would explain why rip/cs is user space - but some part of the CPU is still in kernel mode - which would explain the incorrect page fault error code.

Here's a really *stupid* patch (and untested too, btw) to see if it gets easier to debug when you don't oops, just print the register state instead.

Will add this patch.

(It might be interesting to also do something like

force_sig_specific(SIGSTOP, current);

to then be able to more easily attach to the process that had problems, and debug it in user space to see what was going on..)

Linus
---
diff --git a/arch/x86_64/mm/fault.c b/arch/x86_64/mm/fault.c
index 327c9f2..1b81392 100644
--- a/arch/x86_64/mm/fault.c
+++ b/arch/x86_64/mm/fault.c
@@ -320,6 +320,11 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs *regs,
info.si_code = SEGV_MAPERR;
+ if (!(error_code & PF_USER) && user_mode(regs)) {
+ printk("kernel mode page fault from user space? Huh?\n");
+ __show_regs(regs);
+ error_code |= PF_USER;
+ }
/*
* We fault-in kernel-space virtual memory on-demand. The

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/