RE: [Question] softlockup in run_timer_softirq

[liujian (CE)]

> -----Original Message-----
> From: liujian (CE)
> Sent: Wednesday, February 15, 2023 4:34 PM
> To: 'John Stultz' <jstultz@xxxxxxxxxx>
> Cc: tglx@xxxxxxxxxxxxx; sboyd@xxxxxxxxxx; linux-kernel@xxxxxxxxxxxxxxx;
> peterz@xxxxxxxxxxxxx; Paul E. McKenney <paulmck@xxxxxxxxxx>
> Subject: RE: [Question] softlockup in run_timer_softirq
> 
> 
> 
> > -----Original Message-----
> > From: John Stultz [mailto:jstultz@xxxxxxxxxx]
> > Sent: Tuesday, February 14, 2023 4:01 AM
> > To: liujian (CE) <liujian56@xxxxxxxxxx>
> > Cc: tglx@xxxxxxxxxxxxx; sboyd@xxxxxxxxxx;
> > linux-kernel@xxxxxxxxxxxxxxx; peterz@xxxxxxxxxxxxx; Paul E. McKenney
> > <paulmck@xxxxxxxxxx>
> > Subject: Re: [Question] softlockup in run_timer_softirq
> >
> > On Fri, Feb 10, 2023 at 1:51 AM liujian (CE) <liujian56@xxxxxxxxxx> wrote:
> > >
> > > During the syz test, we encountered many problems with various timer
> > > handler functions softlockup.
> > >
> > > We analyze __run_timers() and find the following problem.
> > >
> > > In the while loop of __run_timers(), because there are too many
> > > timers or improper timer handler functions, if the processing time
> > > of the expired timers is always greater than the time wheel's
> > > next_expiry, the function will loop infinitely.
> > >
> > > The following extreme test case can be used to reproduce the problem.
> > > An extreme test case[1] is constructed to reproduce the problem.
> >
> > Thanks for reporting and sending out this data:
> >
> > First, any chance you might submit this as a in-kernel-stress test?
> > Maybe utilizing the kernel/torture.c framework?
> >
> Okay,   I'll learn this framework and do this thing.
> > (Though the test may need to occasionally take a break so the system
> > can eventually catch up)
> >
> > > Is this a problem or an unreasonable use?
> > >
> > > Can we limit the running time of __run_timers() [2]?
> > >
> > > Does anyone have a good idea to solve this problem?
> >
> > So your patch reminds me of Peter's softirq_needs_break() logic:
> >
> > https://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git/log/?
> > h=co
> > re/softirq
> >
> > Maybe it could extend that series for the timer softirq as well?
> >
> Thank you. Yes.
> Base on the patchset and the extended patch for timer [1], the soft lockup
> problem does not occur.
> 
> By the way, I see this is a very old patchset?  Will this patchset push the main
> line? @John @Peter
> 
Hi, peter,
Do you have an upstream plan for this patchset? Or other ideas.
I want to use softirq_needs_break() to limit the runtime of timer soft interrupt handler function, wonder if this is appropriate?
Thank you~
> 
>  [1]
> Author: Liu Jian <liujian56@xxxxxxxxxx>
> Date:   Tue Feb 14 09:53:46 2023 +0800
> 
>     softirq, timer: Use softirq_needs_break()
> 
>     In the while loop of __run_timers(), because there are too many timers or
>     improper timer handler functions, if the processing time of the expired
>     timers is always greater than the time wheel's next_expiry, the function
>     will loop infinitely.
> 
>     To prevent this, use the timeout/break logic provided by SoftIRQs.If the
>     running time exceeds the limit, break the loop and an additional
>     TIMER_SOFTIRQ is triggered.
> 
>     Signed-off-by: Liu Jian <liujian56@xxxxxxxxxx>
> 
> diff --git a/kernel/time/timer.c b/kernel/time/timer.c index
> 63a8ce7177dd..70744a469a39 100644
> --- a/kernel/time/timer.c
> +++ b/kernel/time/timer.c
> @@ -1992,7 +1992,7 @@ void timer_clear_idle(void)
>   * __run_timers - run all expired timers (if any) on this CPU.
>   * @base: the timer vector to be processed.
>   */
> -static inline void __run_timers(struct timer_base *base)
> +static inline void __run_timers(struct timer_base *base, struct
> +softirq_action *h)
>  {
>         struct hlist_head heads[LVL_DEPTH];
>         int levels;
> @@ -2020,6 +2020,12 @@ static inline void __run_timers(struct timer_base
> *base)
> 
>                 while (levels--)
>                         expire_timers(base, heads + levels);
> +
> +               if (softirq_needs_break(h)) {
> +                       if (time_after_eq(jiffies, base->next_expiry))
> +                               __raise_softirq_irqoff(TIMER_SOFTIRQ);
> +                       break;
> +               }
>         }
>         raw_spin_unlock_irq(&base->lock);
>         timer_base_unlock_expiry(base);
> @@ -2032,9 +2038,9 @@ static __latent_entropy void
> run_timer_softirq(struct softirq_action *h)  {
>         struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]);
> 
> -       __run_timers(base);
> +       __run_timers(base, h);
>         if (IS_ENABLED(CONFIG_NO_HZ_COMMON))
> -               __run_timers(this_cpu_ptr(&timer_bases[BASE_DEF]));
> +               __run_timers(this_cpu_ptr(&timer_bases[BASE_DEF]), h);
>  }
> 
>  /*
> > thanks
> > -john