Re: "hwcap 0 nosegneg" doesnt work with paravirt_ops xen as of 2.6.23.9

[Andres Freund]

Hi Jeremy,

On Wednesday 28 November 2007, Jeremy Fitzhardinge wrote in "Re: "hwcap 0 
nosegneg" doesnt work with paravirt_ops xen as of 2.6.23.9":
> Andres Freund wrote:
> > after converting a host from using a ubuntu patched 2.6.22 domU to using
> > 2.6.23 xen via paravirt_ops (no configuration change except devicename
> > change in fstab, and another getty running) ldconfig no longer picks up
> > the nosegneg libc leading to segfaults for some programs.
> It shouldn't ever lead to segfaults.  At worst, you should get poor
> performance because the hypervisor emulates the -ve segment offset.
strace -ff 
perl /usr/bin/debsums --generate=nocheck -sp /var/cache/apt/archives
...
open(".", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 4
fstat64(4, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
fcntl64(4, F_SETFD, FD_CLOEXEC)         = 0
getdents64(4, /* 3 entries */, 4096)    = 72
getdents64(4, /* 0 entries */, 4096)    = 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
+++ killed by SIGSEGV +++
Process 3238 detached

root@test:~# gdb perl5.8.8                 
GNU gdb 6.6-debian
...
Starting 
program: /usr/bin/perl5.8.8 /usr/bin/debsums --generate=nocheck -sp /var/cache/apt/archives
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
[Thread debugging using libthread_db enabled]
[New Thread -1209940304 (LWP 3356)]
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1209940304 (LWP 3356)]
0xb7eda3d7 in readdir64_r () from /lib/tls/i686/cmov/libc.so.6
(gdb) bt
#0  0xb7eda3d7 in readdir64_r () from /lib/tls/i686/cmov/libc.so.6
#1  0x080fdca5 in Perl_pp_readdir ()
#2  0x080c0d29 in Perl_runops_standard ()
#3  0x0806727a in perl_run ()
#4  0x08063732 in main ()

This was before running an ldconfig. After running, it still selects the wrong 
libraries, but doesnt crash anymore (hm?).


> > /etc/ld.so.conf.d/xen.conf:
> > hwcap 0 nosegneg
> This looks OK.  If this doesn't work, then it suggests there's something
> wrong with the kernel vdso which is not exporting the nonegseg
> (pseudo-)hardware capability.  But we've gone over this several times
> now, and I was fairly sure we'd finally got it right - works for me, at
> least.

> The alternative is that there's some bug on the userspace side in your
> version of Ubuntu...
The thing is, that the same image, aside from the neccessary config changes 
(sda1->xvda1, xvc0->hvc0), recognizes the right libraries. So I dont see an 
obvious problem there.

> > I now discover, that after some playing around (during discovering what
> > the problem is) i must have changed something, because ldd still shows
> > the use of the i686/cmov libraries, but applications reliably causing
> > crashes earlier, dont do so anymore.
> As I said, you shouldn't be seeing crashes either way.  If you are, it
> suggests a hypervisor bug.  Does "xm dmesg" show anything?
(XEN) traps.c:1698:d16 Domain attempted WRMSR 0000008b from 00000056:00000000 
to 00000000:00000000.
(XEN) traps.c:1698:d17 Domain attempted WRMSR 00000404 from 00000000:00000001 
to ffffffff:ffffffff.
And some more of it. But why does a ldconfig, which doesnt change the 
libraries loaded, change that?

For all debug data of the crashes see above.

> > PS: I did this conversion mainly, because I was getting quite random
> > crashes inside the domU. I guess, as this was a vendor kernel + xensource
> > xen kernel, this isnt appropriate for lkml? As well as the crashes of the
> > Dom0 (ubuntu again)?
> Yeah, anything which isn't 2.6.18-xen isn't officially "Xen project",
> but its worth reporting the bugs to xen-devel (and Ubuntu, of course).
> And to me, for that matter.  I won't be able to help directly, but the
> crashes may highlight things we need to be careful about in the pvops
> Xen support.
Ok, will do so in the next days and will CC you at that.

Btw, is there a roadmap how in-kernel Xen is planned to develop?

Greetings, 

Andres

Attachment: signature.asc
Description: This is a digitally signed message part.