source line numbers with x86_64 modules? [Was: Re: [patch] measurements, numbers about CONFIG_OPTIMIZE_INLINING=y impact]

[Mike Snitzer]

On Sat, Jan 10, 2009 at 1:44 AM, Nicholas Miell <nmiell@xxxxxxxxxxx> wrote:
> On Fri, 2009-01-09 at 20:05 -0800, Linus Torvalds wrote:
>>
>> On Fri, 9 Jan 2009, Nicholas Miell wrote:
>> >
>> > It's only too big if you always keep it in memory, and I wasn't
>> > suggesting that.
>>
>> Umm. We're talking kernel panics here. If it's not in memory, it doesn't
>> exist as far as the kernel is concerned.
>>
>> If it doesn't exist, it cannot be reported.
>
> The idea was that the kernel would generate a crash dump and then after
> the reboot, a post-processing tool would do something with it. (e.g. run
> the dump through crash to get a stack trace using separate debug info or
> ship the entire dump off to a collection server or something).
>
>> > And this is where we disagree. I believe that crash dumps should be the
>> > norm and all the reasons you have against crash dumps in general are in
>> > fact reasons against Linux's sub-par implementation of crash dumps in
>> > specific.
>>
>> Good luck with that. Go ahead and try it.  You'll find it wasn't so easy
>> after all.
>>
>> > So, here I am, a non-enterprise end user with a non-stale kernel who'd
>> > love to be able to give you a crash dump (or, more likely, a stack trace
>> > created from that crash dump), but I can't because Linux crash dumps are
>> > stuck in the enterprise ghetto.
>>
>> No, you're stuck because you apparently have your mind stuck on a
>> crash-dump, and aren't willing to look at alternatives.
>>
>> You could use a network console. Trust me - if you can't set up a network
>> console, you have no business mucking around with crash dumps.
>
> netconsole requires a second computer. Feel free to mail me one. :)
>
>> And if the crash is hard enough that you can't any output from that,
>> again, a crash dump wouldn't exactly help, would it?
>>
>> > Hell, I'd be happy if I could get the the normal panic text written to
>> > disk, but since the hard part is the actual writing to disk, there's no
>> > reason not to do the full crash dump if you can.
>>
>> Umm. And why do you think the two have anything to do with each other?
>>
>> Only insane people want the kernel to write to disk when it has problems.
>> Sane people try to write to something that doesn't potentially overwrite
>> their data. Like the network.
>>
>> Which is there. Try it. Trust me, it's a _hell_ of a lot more likely to
>> wotk than a crash dump.
>
> Well, yes, but that has everything to do with how terrible kdump is and
> nothing to do with the idea of crash dumps in general.
>
> Anyway, we've strayed off topic long enough, I'm sure everyone in the Cc
> list would be happy to stop getting an earful about the merits of crash
> dumps.

Yes, especially from someone who lacks the ability to properly
configure kdump.  I'm fairly surprised others are giving you a free
pass when you keep asserting how broken kdump is with such hollow
criticism.  I rely heavily on kdump and it works quite well (kvm
integration was lacking but has improved).

Now that said, I too value crash dumps and will stray a bit off-topic
(subject changed), I do have one significant debugability issue when
using crash dumps: on x86_64 I'm unable to get line number information
from symbols that reside in a module.

I tried to get some insight on this from Dave Anderson (crash utility
developer/maintainer) here:
http://www.mail-archive.com/crash-utility@xxxxxxxxxx/msg01101.html

Dave was helpful but ultimately couldn't explain why I'm unable to get
line numbers with my x86_64 kernel.org kernels (and he/redhat with
RHEL4 kernels).

I strongly respect those cc'd and have to believe someone can help me
cut through this.

I've updated scripts/package/mkspec so that the 'make rpm' target
produces RedHat-style kernel rpms (models redhat's kernel.spec).  This
includes creating debuginfo rpms.  I've attached a patch that works on
2.6.28; but again the resulting debuginfo doesn't provide line numbers
for the kernel modules under crash!?

If anyone has some insight on what I might be missing (as part of the
kernel build) I'd _really_ appreciate it.  I provided the mkspec patch
just as a reference, if the RPM stuff is too opaque please just help
me understand the bare mechanics of what is needed without mkspec (in
the meantime I'll try the debuginfo generation patch Sam Ravnborg
recently posted too).

regards,
Mike
diff --git a/scripts/package/mkspec b/scripts/package/mkspec
index 2500886..a71b4f8 100755
--- a/scripts/package/mkspec
+++ b/scripts/package/mkspec
@@ -24,33 +24,53 @@ fi
 PROVIDES="$PROVIDES kernel-$KERNELRELEASE"
 __KERNELRELEASE=`echo $KERNELRELEASE | sed -e "s/-//g"`
 
+echo "%define _enable_debug_packages 1"
+echo ""
+
+# for compatibility with RedHat-based kernel specfile-isms
+echo "%define KVERREL $KERNELRELEASE"
+echo ""
+
 echo "Name: kernel"
 echo "Summary: The Linux Kernel"
 echo "Version: $__KERNELRELEASE"
 # we need to determine the NEXT version number so that uname and
 # rpm -q will agree
-echo "Release: `. $srctree/scripts/mkversion`"
+echo "Release: `echo $KERNELRELEASE | cut -d'-' -f2-`"
 echo "License: GPL"
 echo "Group: System Environment/Kernel"
 echo "Vendor: The Linux Community"
 echo "URL: http://www.kernel.org";
+echo "Prereq: fileutils, module-init-tools, initscripts, mkinitrd"
 
 if ! $PREBUILT; then
-echo "Source: kernel-$__KERNELRELEASE.tar.gz"
+echo "Source: kernel-%{KVERREL}.tar.gz"
 fi
 
-echo "BuildRoot: /var/tmp/%{name}-%{PACKAGE_VERSION}-root"
+echo "BuildRoot: /var/tmp/%{name}-%{KVERREL}-root"
 echo "Provides: $PROVIDES"
-echo "%define __spec_install_post /usr/lib/rpm/brp-compress || :"
-echo "%define debug_package %{nil}"
+
 echo ""
 echo "%description"
 echo "The Linux Kernel, the operating system core itself"
 echo ""
 
+# devel package (used to allow modules to be built against this kernel)
+echo "%package devel"
+echo "Summary: Development package for building kernel modules to match the kernel."
+echo "Group: System Environment/Kernel"
+echo "AutoReqProv: no"
+echo "Provides: kernel-devel-%{_target_cpu} = %{version}-%{release}"
+echo "Prereq: /usr/bin/find"
+echo ""
+echo "%description devel"
+echo "This package provides kernel headers and makefiles sufficient to build modules"
+echo "against the kernel package."
+echo ""
+
 if ! $PREBUILT; then
 echo "%prep"
-echo "%setup -q"
+echo "%setup -q -n kernel-%{KVERREL}"
 echo ""
 fi
 
@@ -59,41 +79,210 @@ echo "%build"
 if ! $PREBUILT; then
 echo "make clean && make %{?_smp_mflags}"
 echo ""
+
+echo "###"
+echo "### Special hacks for debuginfo subpackages."
+echo "###"
+
+echo "# This macro is used by %%install, so we must redefine it before that."
+echo "%define debug_package %{nil}"
+
+echo '%if "%{_enable_debug_packages}" == "1"'
+echo '%ifnarch noarch'
+echo '%global __debug_package 1'
+echo "%endif"
+echo "%endif"
 fi
 
 echo "%install"
-echo "%ifarch ia64"
-echo 'mkdir -p $RPM_BUILD_ROOT/boot/efi $RPM_BUILD_ROOT/lib/modules'
-echo 'mkdir -p $RPM_BUILD_ROOT/lib/firmware'
-echo "%else"
 echo 'mkdir -p $RPM_BUILD_ROOT/boot $RPM_BUILD_ROOT/lib/modules'
 echo 'mkdir -p $RPM_BUILD_ROOT/lib/firmware'
-echo "%endif"
 
 echo 'INSTALL_MOD_PATH=$RPM_BUILD_ROOT make %{_smp_mflags} modules_install'
-echo "%ifarch ia64"
-echo 'cp $KBUILD_IMAGE $RPM_BUILD_ROOT'"/boot/efi/vmlinuz-$KERNELRELEASE"
-echo 'ln -s '"efi/vmlinuz-$KERNELRELEASE" '$RPM_BUILD_ROOT'"/boot/"
-echo "%else"
-echo "%ifarch ppc64"
-echo "cp vmlinux arch/powerpc/boot"
-echo "cp arch/powerpc/boot/"'$KBUILD_IMAGE $RPM_BUILD_ROOT'"/boot/vmlinuz-$KERNELRELEASE"
-echo "%else"
-echo 'cp $KBUILD_IMAGE $RPM_BUILD_ROOT'"/boot/vmlinuz-$KERNELRELEASE"
-echo "%endif"
-echo "%endif"
+echo 'cp arch/%{_arch}/boot/bzImage $RPM_BUILD_ROOT'"/boot/vmlinuz-%{KVERREL}"
+
+# symvers needed for %post
+echo 'gzip -c9 < Module.symvers > $RPM_BUILD_ROOT/boot/symvers-%{KVERREL}.gz'
 
-echo 'cp System.map $RPM_BUILD_ROOT'"/boot/System.map-$KERNELRELEASE"
+echo 'cp System.map $RPM_BUILD_ROOT'"/boot/System.map-%{KVERREL}"
+
+echo 'cp .config $RPM_BUILD_ROOT'"/boot/config-%{KVERREL}"
+
+echo 'touch $RPM_BUILD_ROOT/boot/initrd-%{KVERREL}.img'
+
+echo ""
+
+# devel package install
+echo "KernelVer=%{KVERREL}"
+
+echo "# And save the headers/makefiles etc for building modules against"
+echo "#"
+echo "# This all looks scary, but the end result is supposed to be:"
+echo "# * all arch relevant include/ files"
+echo "# * all Makefile/Kconfig files"
+echo "# * all script/ files"
+echo ""
+echo 'rm -f $RPM_BUILD_ROOT/lib/modules/$KernelVer/build'
+echo 'rm -f $RPM_BUILD_ROOT/lib/modules/$KernelVer/source'
+echo 'mkdir -p $RPM_BUILD_ROOT/lib/modules/$KernelVer/build'
+echo '(cd $RPM_BUILD_ROOT/lib/modules/$KernelVer ; ln -s build source)'
+echo "# dirs for additional modules per module-init-tools, kbuild/modules.txt"
+echo 'mkdir -p $RPM_BUILD_ROOT/lib/modules/$KernelVer/extra'
+echo 'mkdir -p $RPM_BUILD_ROOT/lib/modules/$KernelVer/updates'
+echo 'mkdir -p $RPM_BUILD_ROOT/lib/modules/$KernelVer/weak-updates'
+echo '# first copy everything'
+echo 'cp --parents `find  -type f -name "Makefile*" -o -name "Kconfig*"` $RPM_BUILD_ROOT/lib/modules/$KernelVer/build'
+echo 'cp Module.symvers $RPM_BUILD_ROOT/lib/modules/$KernelVer/build'
+echo "# then drop all but the needed Makefiles/Kconfig files"
+echo 'rm -rf $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/Documentation'
+echo 'rm -rf $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/scripts'
+echo 'rm -rf $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/include'
+echo 'cp .config $RPM_BUILD_ROOT/lib/modules/$KernelVer/build'
+echo 'cp -a scripts $RPM_BUILD_ROOT/lib/modules/$KernelVer/build'
+echo "if [ -d arch/%{_arch}/scripts ]; then"
+echo '  cp -a arch/%{_arch}/scripts $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/arch/%{_arch} || :'
+echo "fi"
+echo 'if [ -f arch/%{_arch}/*lds ]; then'
+echo '  cp -a arch/%{_arch}/*lds $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/arch/%{_arch}/ || :'
+echo "fi"
+echo 'rm -f $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/scripts/*.o'
+echo 'rm -f $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/scripts/*/*.o'
+echo 'mkdir -p $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/include'
+echo 'cd include'
+echo 'cp -a acpi config keys linux math-emu media mtd net pcmcia rdma rxrpc scsi sound video asm asm-generic $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/include'
+echo 'cp -a `readlink asm` $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/include'
+#echo 'if [ "%{_arch}" = "x86_64" ]; then'
+#echo '  cp -a asm-i386 $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/include'
+#echo "fi"
+echo ""
+
+echo "# Make sure the Makefile and version.h have a matching timestamp so that"
+echo "# external modules can be built"
+echo 'touch -r $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/Makefile $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/include/linux/version.h'
+echo 'touch -r $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/.config $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/include/linux/autoconf.h'
+echo '# Copy .config to include/config/auto.conf so "make prepare" is unnecessary.'
+echo 'cp $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/.config $RPM_BUILD_ROOT/lib/modules/$KernelVer/build/include/config/auto.conf'
+echo "cd .."
+echo ""
+
+echo 'find $RPM_BUILD_ROOT/lib/modules/$KernelVer -name "*.ko" -type f >modnames'
+echo "# mark modules executable so that strip-to-file can strip them"
+echo "cat modnames | xargs chmod u+x"
+echo "rm -f modnames"
+echo "# remove files that will be auto generated by depmod at rpm -i time"
+echo 'rm -f $RPM_BUILD_ROOT/lib/modules/$KernelVer/modules.*'
+echo ""
+echo "# Move the devel headers out of the root file system"
+echo "DevelDir=/usr/src/kernels/%{KVERREL}-%{_target_cpu}"
+echo 'mkdir -p $RPM_BUILD_ROOT/usr/src/kernels'
+echo 'mv $RPM_BUILD_ROOT/lib/modules/$KernelVer/build $RPM_BUILD_ROOT/$DevelDir'
+echo 'ln -sf ../../..$DevelDir $RPM_BUILD_ROOT/lib/modules/$KernelVer/build'
+echo ""
+
+# debuginfo package install
+echo '%if "%{_enable_debug_packages}" == "1"'
+echo 'mkdir -p $RPM_BUILD_ROOT/usr/lib/debug/boot'
+echo 'mkdir -p $RPM_BUILD_ROOT/usr/lib/debug/lib/modules/$KernelVer'
+echo "#"
+echo "# save the vmlinux file for kernel debugging into the kernel-debuginfo rpm"
+echo "#"
+echo 'cp vmlinux $RPM_BUILD_ROOT/usr/lib/debug/lib/modules/$KernelVer'
+echo ""
+# help crash find the Linux source
+echo 'mkdir -p $RPM_BUILD_ROOT/usr/src/debug'
+echo '(cd $RPM_BUILD_ROOT/usr/src/debug ; ln -s %{name}-%{KVERREL} %{KVERREL})'
+echo "%endif"
 
-echo 'cp .config $RPM_BUILD_ROOT'"/boot/config-$KERNELRELEASE"
 echo ""
 echo "%clean"
-echo '#echo -rf $RPM_BUILD_ROOT'
+echo "%{__rm} -rf %{buildroot}"
+echo ""
+
+#
+# scripts
+#
+
+# install: update grub.conf et al
+echo "%post"
+echo 'if [ `uname -i` == "x86_64" -o `uname -i` == "i386" ]; then'
+echo "  if [ -f /etc/sysconfig/kernel ]; then"
+echo "    /bin/sed -i -e 's/^DEFAULTKERNEL=kernel-smp$/DEFAULTKERNEL=kernel/' /etc/sysconfig/kernel || "'exit $?'
+echo "  fi"
+echo "fi"
+echo '/sbin/new-kernel-pkg --package kernel --mkinitrd --depmod --install %{KVERREL} || exit $?'
+echo "if [ -x /sbin/weak-modules ]"
+echo "then"
+echo '    /sbin/weak-modules --add-kernel %{KVERREL} || exit $?'
+echo "fi"
 echo ""
+
+# uninstall
+echo "%preun"
+echo '/sbin/new-kernel-pkg --rminitrd --rmmoddep --remove %{KVERREL} || exit $?'
+echo "if [ -x /sbin/weak-modules ]"
+echo "then"
+echo '    /sbin/weak-modules --remove-kernel %{KVERREL} || exit $?'
+echo "fi"
+echo ""
+
 echo "%files"
 echo '%defattr (-, root, root)'
-echo "%dir /lib/modules"
-echo "/lib/modules/$KERNELRELEASE"
+echo "/boot/vmlinuz-%{KVERREL}"
+echo "/boot/System.map-%{KVERREL}"
+echo "/boot/symvers-%{KVERREL}.gz"
+echo "/boot/config-%{KVERREL}"
+echo "%dir /lib/modules/%{KVERREL}"
 echo "/lib/firmware"
-echo "/boot/*"
+echo "/lib/modules/%{KVERREL}/kernel"
+echo "/lib/modules/%{KVERREL}/build"
+echo "/lib/modules/%{KVERREL}/source"
+echo "/lib/modules/%{KVERREL}/extra"
+echo "/lib/modules/%{KVERREL}/updates"
+echo "/lib/modules/%{KVERREL}/weak-updates"
+echo "%ghost /boot/initrd-%{KVERREL}.img"
 echo ""
+
+echo "%files devel"
+echo "%defattr(-,root,root)"
+echo "%verify(not mtime) /usr/src/kernels/%{KVERREL}-%{_target_cpu}"
+echo ""
+
+# debuginfo packages
+echo '%if "%{_enable_debug_packages}" == "1"'
+echo '%ifnarch noarch'
+
+echo "%package debuginfo-common"
+echo "Summary: Kernel source files used by %{name}-debuginfo packages"
+echo "Group: Development/Debug"
+echo "Provides: %{name}-debuginfo-common-%{_target_cpu} = %{KVERREL}"
+echo ""
+echo "%description debuginfo-common"
+echo "This package is required by %{name}-debuginfo subpackages."
+echo "It provides the kernel source files common to all builds."
+echo ""
+echo "%files debuginfo-common"
+echo "%defattr(-,root,root)"
+echo "/usr/src/debug/%{name}-%{KVERREL}"
+echo "/usr/src/debug/%{KVERREL}"
+echo "%dir /usr/src/debug"
+echo "%dir /usr/lib/debug"
+echo "%dir /usr/lib/debug/boot"
+echo "%dir /usr/lib/debug/lib"
+echo "%dir /usr/lib/debug/lib/modules"
+echo "%dir /usr/lib/debug/usr/src/kernels"
+
+echo "%package debuginfo"
+echo "Summary: Debug information for package %{name}"
+echo "Group: Development/Debug"
+echo "Requires: %{name}-debuginfo-common-%{_target_cpu} = %{KVERREL}"
+echo "Provides: %{name}-debuginfo-%{_target_cpu} = %{KVERREL}"
+echo "%description debuginfo"
+echo "This package provides debug information for package %{name}"
+echo "This is required to use SystemTap with %{name}-%{KVERREL}."
+echo "%files debuginfo"
+echo "%defattr(-,root,root)"
+echo "/usr/lib/debug/lib/modules/%{KVERREL}"
+echo "/usr/lib/debug/usr/src/kernels/%{KVERREL}-%{_target_cpu}"
+
+echo "%endif"
+echo "%endif"