Hi,
API and core updates for jump labels. (I also included changes for mips,
and sparc, (compile tested only), so hopefully the respective arch gurus,
can ACK those bits). Applies on top of latest -tip tree.
Introduce:
static __always_inline bool static_branch(struct jump_label_key *key);
instead of the old JUMP_LABEL(key, label) macro.
In this way, jump labels become really easy to use:
Define:
struct jump_label_key jump_key;
Can be used as:
if (static_branch(&jump_key))
do unlikely code
enable/disale via:
jump_label_enable(&jump_key);
jump_label_disable(&jump_key);
that's it!
For the jump labels disabled case, the static_branch() becomes an
atomic_read(), and jump_label_enable()/disable() are simply atomic_inc(),
atomic_dec() operations. We show testing results for this change below.
Thanks to H. Peter Anvin for suggesting the 'static_branch()' construct.
Since we now require a 'struct jump_label_key *key', we can store a pointer into
the jump table addresses. In this way, we can enable/disable jump labels, in
basically constant time. This change allows us to completely remove the previous
hashtable scheme. Thanks to Peter Zijlstra for this re-write.
Testing:
I ran a series of 'tbench 20' runs 5 times (with reboots) for 3
configurations, where tracepoints were disabled.
jump label configured in
avg: 815.6
jump label *not* configured in (using atomic reads)
avg: 800.1
jump label *not* configured in (regular reads)
avg: 803.4
thanks,
-Jason
Signed-off-by: Peter Zijlstra<a.p.zijlstra@xxxxxxxxx>
Signed-off-by: Jason Baron<jbaron@xxxxxxxxxx>
Suggested-by: H. Peter Anvin<hpa@xxxxxxxxxxxxxxx>
---[...]
arch/mips/include/asm/jump_label.h | 22 +-
arch/mips/kernel/jump_label.c | 2 +-
arch/sparc/include/asm/jump_label.h | 25 +-
arch/x86/include/asm/alternative.h | 3 +-
arch/x86/include/asm/jump_label.h | 26 +-
arch/x86/kernel/alternative.c | 2 +-
arch/x86/kernel/jump_label.c | 2 +-
arch/x86/kernel/module.c | 1 +
include/asm-generic/vmlinux.lds.h | 14 +-
include/linux/dynamic_debug.h | 35 ++-
include/linux/jump_label.h | 82 ++++---
include/linux/jump_label_ref.h | 44 ---
include/linux/perf_event.h | 26 +-
include/linux/tracepoint.h | 8 +-
kernel/jump_label.c | 532 ++++++++++++++---------------------
kernel/perf_event.c | 4 +-
kernel/tracepoint.c | 23 +-
lib/dynamic_debug.c | 13 +-
18 files changed, 368 insertions(+), 496 deletions(-)
delete mode 100644 include/linux/jump_label_ref.h