SystemTap 2.4 release

From: Jonathan Lebon
Date: Wed Nov 06 2013 - 11:57:42 EST

The SystemTap team announces release 2.4, "virtual world"!

smarter error reporting, better suggestions, virtual machine probing,
.gnu_debugdata support, improved memory allocation, and more!

= Where to get it - our project page
git tag release-2.4 (commit aa7b1e0)

There have been over 247 commits since the last release.
There have been over 35 bugs fixed / features added since the last

= How to build it

See the README and NEWS files at;a=tree

Further information at

= SystemTap frontend (stap) changes

- Better suggestions are given in many of the semantic errors in which
alternatives are provided. Additionally, suggestions are now provided
when plt and trace probes could not be resolved. For example,
kernel.trace("sched_siwtch") will suggest 'sched_switch'.

- SystemTap is now smarter about error reporting. Errors from the same
source are considered duplicates and suppressed. A message is
displayed on exit if any errors/warnings were suppressed.

- SystemTap now supports probing inside virtual machines using the
libvirt and unix schemes, e.g.

stap -ve 'probe timer.s(1) { printf("hello!\n") }' \

Virtual machines managed by libvirt can be prepared using stapvirt.
See stapvirt(1) and the --remote option in stap(1) for more details.

- SystemTap now checks for and uses (when available) the .gnu_debugdata
section which contains a subset of debuginfo, useful for backtraces
and function probing

= SystemTap script language changes

- For systems with in-kernel utrace, 'process.end' and 'thread.end'
probes will hit before the target's parent process is notified of the
target's death. This matches the behavior of newer kernels without
in-kernel utrace.

= SystemTap runtime changes

- The stap-server client & server code now enable all SSL/TLS ciphers
rather than just the "export" subset.

- Statistics aggregate typed objects are now implemented locklessly, if
the translator finds that they are only ever read (using the foreach /
@count / etc. constructs) in a probe-begin/end/error.

- SystemTap map variables are now allocated with vmalloc() instead of
with kmalloc(), which should cause memory to be less fragmented.

- Although SystemTap itself requires elfutils 0.148+, staprun only
requires elfutils 0.142+, which could be useful with the
'--disable-translator' configure switch.

- Under FIPS mode (/proc/sys/crypto/fips_enabled=1), staprun will refuse
to load SystemTap modules (since these are not normally signed with
the kernel's build-time keys). This protection may be suppressed with
the $STAP_FIPS_OVERRIDE environment variable.

= SystemTap tapset changes

macros.stpm new '@MAXSTRINGLEN' macro
context-caller.stp use the new '@MAXSTRINGLEN' macro instead of
%{ MAXSTRINGLEN %} to avoid problens for
unprivileged users
context-symbols.stp ditto
ucontext-symbols.stp ditto
ucontext.stp ditto
uconversions.stp ditto and new function
context.stp fix cmdline_arg() processing
dentry.stp new inode_path() function which tries to
locate the vfsmount containing an inode
nd_syscalls.stp (1) fix syscall.{execve,compat_execve} for
older kernels
(2) break the 'mmap2' probe alias down into
subprobes to better convert 'pgoff'
syscalls.stp (1) & (2) as above and
(3) fix syscall.accept probe alias
memory.stp correct VM_FAULT_MINOR conditional handling
and protect __pa() call with
nfs_proc.stp update probes to handle removal of NFSv2
signal.stp update signal.pending[.return] probe aliases
registers.stp on s390, new _stp_get_kernel_stack_param()
function to get stack parameters
aux_syscalls.stp decode SIGTRAP|0x80 from

- Several syscall tapsets which read user buffers of known length will
now use user_string_n2_quoted rather than user_string_n_quoted().

- The syscall/nd_syscall tapsets had multiple probes split into
subprobes to avoid using symname(addr()).

- The s390 syscalls.stp tapset had multiple probes updated to handle
newer kernels with renamed parameters.

= SystemTap sample scripts

- New samples:

last_100_frees.stp log recent free(3) libc calls along
with a userspace backtrace at those
futexes2.stp watches just shared futex syscalls on
the system and prints contention info
on exit
semop-watch.stp prints a timed trace of
semop(2)/semtimedop(2) syscalls
strace.stp loosely emulates strace, can be applied
to individual processes (via -c/-x) or
entire system

- Changed samples:

badname.stp update for newer kernels
eventcount.stp add -G lines=NNN option
tcpdumplike.stp demonstrates recently added UDP tapset
by also printing out UDP packets
deviceseeks.stp handle older kernel's names for sector
size field
numa_faults.stp update to tolerate exception now thrown
by addr_to_node() tapset function
threadstacks.stp updated for glibc 2.17 changes
regarding pthread stack allocation
periodic.stp tweak to operate without kernel

= Examples of tested kernel versions

2.6.9 (x86)
2.6.18 (x86)
2.6.32 (x86)
3.10.0 (x86_64)
3.11.6 (x86_64)

= Known issues with this release

- Some kernel crashes continue to be reported when a script probes
broad kernel function wildcards. (PR2725)

- 32-on-64 bit userspace unwinding is truncated on older kernels, such
as 2.6.32 (PR15757)

- The dyninst backend is still very much a prototype, with a number
of issues, limitations, and general teething woes. For instance:
+ lack of support for multiarch/cross-instrumentation
+ tapset functions are still incomplete relative to what is supported
when the kernel backend is active
+ exception handling becomes completely broken in programs
instrumented by the current version of dyninst (PR14702)
+ command line interrupts are slightly mishandled (PR15049)
+ not all registers are made available on 32-bit x86 (PR15136)

See dyninst/README and the systemtap/dyninst Bugzilla component
( if you want all the gory
details about the state of the feature.

= Contributors for this release

Dave Brolley, David Smith, Frank Ch. Eigler, Jonathan Lebon, Josh
Stone, Lukas Berk, Mark Wielaard, Martin Cermak*, Masanari Iida,
Miroslav Franc*, Naresh Kamboju*, Paddie O'Brien*, Serguei Makarov,
Stan Cox, Timo Juhani Lindfors, Vincent Bernat*, William Cohen

Special thanks to new contributors, marked with '*' above.
Special thanks to Jonathan Lebon for compiling these notes.

= Bugs fixed for this release <>

4331 systemtap.stress/current.stp need to be updated for s390x
6764 Execution of probefunc.exp causes system hang on s390 system
9822 reduce timer load
10088 on s390x, the translator gets an assertion
11751 on s390x, externalvar.exp gets libdwfl failures
13078 investigate qemu virtio-serial channel for talking to stap-sh
13974 sdt.h is incompatible with clang
14960 frame size too large for systemtap.examples/process/psig
15625 unprivileged_embedded_C test confused by stub registers.stp
15663 Failing to remove the stap kernel modules on kernel 3.9.4
15782 nss usage bugs - PR_Read returning short results
15788 tarballs should be PGP signed
15789 ERROR: couldn't initialize gettimeofday
15793 Kernel panic when running systemtap's test suite with kernel 3.10.1
15803 systemtap 2.3 testsuite crashes linux 3.10 sometimes at netdev/enter_netfilter_probe_0
15805 maps should not be allocated with k*alloc
15865 nfs_proc.stp needs to be handle NFSv2 support being removed
15866 on rawhide, the 'signal.pending' and 'signal.pending.return' probe aliases are broken
15872 on rawhide, we're getting odd client.exp test failures
15874 the systemtap.base/onoffprobe.exp testcase times out
15875 the ptrace.exp testcase will fail if 'strace' isn't present on the system
15876 The ptridx.exp testcase has timeout errors
15881 unprivileged_myproc.exp test fails on rawhide
15900 syntax errors in server_locale.exp
15906 don't try to run modules in kernel-fips mode
15913 on s390x, nd_syscall testsuite failures when accessing arguments 1 and 6
15928 the memory1.exp testcase is accessing user memory as kernel memory
15937 On arm fc19 unable to find $whence argment for kernel.function("sys_llseek").call
15961 nd_syscall.exp failure on i686
15980 uprobes MAXSKIPPED not overruled by --suppress-handler-errors
15982 process.end probes broken on RHEL6
16019 limit number of similar semantic errors output
16057 task_dentry_path fails to cross mount points
16058 syscall.mmap/nd_syscall.mmap probe alias problem
16096 on i686, the nd_syscall tapset is broken

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at
Please read the FAQ at