Re: [PATCH] lib/sys_info: add a simple timer based memory corruption detector

[Feng Tang]

Hi Steven,

On Thu, Jun 11, 2026 at 05:23:36PM +0800, Feng Tang wrote:
> Add more reviewers as Petr suggested.
> 
> On Wed, Jun 10, 2026 at 12:50:13PM -0400, Steven Rostedt wrote:
> > On Wed, 27 May 2026 11:43:24 +0800
> > Feng Tang <feng.tang@xxxxxxxxxxxxxxxxx> wrote:
> > 
> > > During debugging some bios/hardware related nasty memory corruption
> > > issues, we found using periodic timer to monitor specific dram/mmio
> > > physical address is very useful for debugging, which acts like
> > > a basic software watchpoint.
> > > 
> > > For those bugs,  who (and when) change(corrupt) those dram or mmio
> > > register is hard to trace, and sometimes even hardware jtag debugger
> > > can't help (say the physical address watchpoint doesn't work).
> > > 
> > > The biggest shortcoming is it can never capture the exact point like
> > > a hardware watchpoint, no matter how small the timer interval is set,
> > > the idea is trying to approach the point, hoping the caught context
> > > have enough debug info (which did help us in solving bios/hardware
> > > bugs)
> > 
> > Instead of using a timer, can't you use the function tracer? That is,
> > have the value checked at *every* function call. You can easily add a
> > custom callback that gets called when every function is executed.
> > 
> > See https://docs.kernel.org/trace/ftrace-uses.html
> 
> Many thanks for the suggestion and detailed sample codes !
> 
> For the 2 bugs I mentioned in https://lore.kernel.org/lkml/aiaRLiaNs9M9c2q-@U-2FWC9VHC-2323.local/,
> this approach should make the software watchpoint hit much more
> precisely, though the culprits were not kernel itself.
> 
> One thought is, adding this software hook to every function call could
> be a high cost for production kernel. For the kernel zero page
> corruption case, we already added a 3-seconds interval timer to detect
> similar issue, and also ftrace could be disabled for some kernel.
> 
> So I'm thinking to use your ftrace method as the base solution in
> next version, and in future provide an possible light weight option
> of using timer method. How do you think?    
> 
> I will go test this method and sent the v1 soon, thanks

I tried the ftrace_ops way, and it worked well also to hit the
watchpoint. But the system is obviously slower comparing to the 100ms
periodic timer way, for the same QEMU test setup, the
'systemd-analyze' cmd show the difference of boot time:

	17.555s (kernel) + 3min 7.559s (userspace)
vs
	350ms (kernel) + 3.646s (userspace)

So I tend to still use the 'timer' way as the base and add an option
to switch to ftrace_ops way. 

For the nasty memory corruption issues I've worked on, many of them
are triggered by device (like NIC), BIOS runtime service or silicon,
and not by kernel itself. 

And for kernel triggered ones, as inspired by your ftrace idea, I
tried the 'fgraph_ops' in a hacked way, which has the 'retfunc'
feature to insert the watchpoint check before exitting the function,
and did help on catching the exact culprit function. 

Thanks,
Feng