Re: [PATCH v6 0/6] arm64: Add kernel probes (kprobes) support

[William Cohen]

On 05/13/2015 05:22 AM, Masami Hiramatsu wrote:
> On 2015/05/12 21:48, William Cohen wrote:

>> Hi Dave,
>>
>> In some of the previous diagnostic output it looked like things would go wrong
>> in the entry.S when the D bit was cleared and the debug interrupts were 
>> unmasksed.  I wonder if some of the issue might be due to the starting the 
>> kprobe for the trampoline, but leaving things in an odd state when another
>> set of krpobe/kretporbes are hit when the trampoline is running.
> 
> Hmm, does this mean we have a trouble if a user kprobe handler calls the
> function which is probed by other kprobe? Or, is this just a problem
> only for kretprobes?

Hi Masami,

I wrote an example based off of sample/kprobes/kprobes_sample.c to force the reentry issue for kprobes (the attached kprobe_rentry_example.c). That seemed to run fine.  I think the reason that the trampoline handler got into trouble is because of the reset_current_kprobe() before the possible call to kfree, but I haven't verified it. It seems like that should be at the end of trampoline handler just before the return.  Other architectures have similar trampoline handlers, so I am surprised that the other architectures haven't encountered this issue with kretprobes.  Maybe this is due to specific of arm64 exception handling.

# modprobe kprobe_reentry_example
[  909.617295] Planted kprobe at fffffe00000b7b34
[  909.623873] Planted kprobe at fffffe000032d34c
# rmmod kprobe_reentry_example
[ 1482.647504] kprobe at fffffe00000b7b34 unregistered
[ 1482.687506] kprobe at fffffe000032d34c unregistered
[ 1482.692361] y = 42
[ 1482.694361] z = 0
# grep \ int_sqrt$ /proc/kallsyms 
fffffe000032d34c T int_sqrt
# grep \ do_fork$ /proc/kallsyms 
fffffe00000b7b34 T do_fork

> 
>>  As Dave
>> mentioned the proposed trampoline patch avoids using a kprobe in the
>> trampoline and directly calls the trampoline handler.  Attached is the
>> current version of the patch which was able to run the systemtap testsuite.
>>  Systemtap does exercise the kprobe/kretprobe infrastructure, but it would
>> be good to have additional raw kprobe tests to check that kprobe reentry
>> works as expected.
> 
> Actually, Will's patch looks like the same thing what I did on x86,
> as the kretprobe-booster. So I'm OK for that. But if the above problem
> is not solved, we need to fix that, since kprobes can be used from
> different sources.

The patch should look similar to the x86 code. The x86 code was used as a model.

-Will

/*
 * NOTE: This example is designed to check that the kprobe reentry work.
 *
 * For more information on theory of operation of kprobes, see
 * Documentation/kprobes.txt
 *
 */

#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/kprobes.h>

/* For each probe you need to allocate a kprobe structure */
static struct kprobe kp = {
	.symbol_name	= "do_fork",
};

static struct kprobe kp_re = {
	.symbol_name	= "int_sqrt",
};

static unsigned long y=0;
static unsigned long z=0;

/* kprobe pre_handler: called just before the probed instruction is executed */
static int handler_pre(struct kprobe *p, struct pt_regs *regs)
{
  /* call another function that is instrumented with a kprobe to
     ensure that reentry works */
  unsigned long x=1764;
  y = int_sqrt(x);
  return 0;
}

/* kprobe post_handler: called after the probed instruction is executed */
static void handler_post(struct kprobe *p, struct pt_regs *regs,
				unsigned long flags)
{
  return;
}

/* kprobe pre_handler: called just before the probed instruction is executed */
static int handler_pre_re(struct kprobe *p, struct pt_regs *regs)
{
  /* if reentry is working as expected this code may not be executed */
  z = 0xdeadbeef;
  return 0;
}

/* kprobe post_handler: called after the probed instruction is executed */
static void handler_post_re(struct kprobe *p, struct pt_regs *regs,
				unsigned long flags)
{
  return;
}

/*
 * fault_handler: this is called if an exception is generated for any
 * instruction within the pre- or post-handler, or when Kprobes
 * single-steps the probed instruction.
 */
static int handler_fault(struct kprobe *p, struct pt_regs *regs, int trapnr)
{
	printk(KERN_INFO "fault_handler: p->addr = 0x%p, trap #%dn",
		p->addr, trapnr);
	/* Return 0 because we don't handle the fault. */
	return 0;
}

static int __init kprobe_init(void)
{
	int ret;
	kp.pre_handler = handler_pre;
	kp.post_handler = handler_post;
	kp.fault_handler = handler_fault;

	ret = register_kprobe(&kp);
	if (ret < 0) {
		printk(KERN_INFO "register_kprobe failed, returned %d\n", ret);
		return ret;
	}
	printk(KERN_INFO "Planted kprobe at %p\n", kp.addr);

	kp_re.pre_handler = handler_pre_re;
	kp_re.post_handler = handler_post_re;
	kp_re.fault_handler = handler_fault;

	ret = register_kprobe(&kp_re);
	if (ret < 0) {
		printk(KERN_INFO "register_kprobe failed, returned %d\n", ret);
		return ret;
	}
	printk(KERN_INFO "Planted kprobe at %p\n", kp_re.addr);
	return 0;
}

static void __exit kprobe_exit(void)
{
	unregister_kprobe(&kp);
	printk(KERN_INFO "kprobe at %p unregistered\n", kp.addr);
	unregister_kprobe(&kp_re);
	printk(KERN_INFO "kprobe at %p unregistered\n", kp_re.addr);
	printk(KERN_INFO "y = %ld\n", y);
	printk(KERN_INFO "z = %lx\n", z);
}

module_init(kprobe_init)
module_exit(kprobe_exit)
MODULE_LICENSE("GPL");