Re: [patch 2/3] timers: do not raise softirq unconditionally (spinlockless version)

[Marcelo Tosatti]

On Wed, May 29, 2019 at 04:53:26PM +0200, Anna-Maria Gleixner wrote:
> On Mon, 15 Apr 2019, Marcelo Tosatti wrote:
> 
> > Check base->pending_map locklessly and skip raising timer softirq 
> > if empty.
> > 
> > What allows the lockless (and potentially racy against mod_timer) 
> > check is that mod_timer will raise another timer softirq after
> > modifying base->pending_map.
> 
> The raise of the timer softirq after adding the timer is done
> unconditionally - so there are timer softirqs raised which are not required
> at all, as mentioned before.

Yes. However i can't see a way to avoid that: its not possible to know
if the race described earlier happened or not. 

Do you have a suggestion on how to avoid this or a way to avoid
the IPI+raise softirq ?

> This check is for !CONFIG_PREEMPT_RT_FULL only implemented. The commit
> message totally igonres that you are implementing something
> CONFIG_PREEMPT_RT_FULL dependent as well.
> 
> > Signed-off-by: Marcelo Tosatti <mtosatti@xxxxxxxxxx>
> > 
> > ---
> >  kernel/time/timer.c |   18 ++++++++++++++++++
> >  1 file changed, 18 insertions(+)
> > 
> > Index: linux-rt-devel/kernel/time/timer.c
> > ===================================================================
> > --- linux-rt-devel.orig/kernel/time/timer.c	2019-04-15 14:21:02.788704354 -0300
> > +++ linux-rt-devel/kernel/time/timer.c	2019-04-15 14:22:56.755047354 -0300
> > @@ -1776,6 +1776,24 @@
> >  		if (time_before(jiffies, base->clk))
> >  			return;
> >  	}
> > +
> > +#ifdef CONFIG_PREEMPT_RT_FULL
> > +/* On RT, irq work runs from softirq */
> > +	if (irq_work_needs_cpu())
> > +		goto raise;
> 
> So with this patch and the change you made in the patch before, timers on
> RT are expired only when there is pending irq work or after modifying a
> timer on a non housekeeping cpu?

Well, run_timer_softirq execute only if pending_map contains a bit set.

> With your patches I could create the following problematic situation on RT
> (if I understood everything properly): I add a timer which should expire in
> 50 jiffies to the wheel of a non housekeeping cpu. So it ends up 50 buckets
> away form now in the first wheel. This timer is the only timer in the wheel
> and the next timer softirq raise is required in 50 jiffies. After adding
> the timer, the timer interrupt is raised, and no timer has to be expired,
> because there is no timer pending.

But the softirq will be raised, because pending_map will be set:

+               if (!bitmap_empty(base->pending_map, WHEEL_SIZE))
+                       goto raise;

No?

>  If there is no irq work required during
> the next 51 jiffies and also no timer changed, the timer I added, will not
> expire in time. The timer_base will come out of idle but will not forward
> the base clk. 

> This makes it even worse: When then adding a timer, the timer
> base is forwarded - but without checking for the next pending timer, so the
> first added timer will be delayed even more.
> 
> So your implementation lacks forwarding the timer_base->clk when timer_base
> comes out of idle with respect to the next pending timer.

> > +#endif
> > +	base = this_cpu_ptr(&timer_bases[BASE_STD]);
> > +	if (!housekeeping_cpu(base->cpu, HK_FLAG_TIMER)) {
> > +		if (!bitmap_empty(base->pending_map, WHEEL_SIZE))
> > +			goto raise;
> > +		base++;
> > +		if (!bitmap_empty(base->pending_map, WHEEL_SIZE))
> > +			goto raise;
> > +
> > +		return;
> > +	}
> > +
> > +raise:
> >  	raise_softirq(TIMER_SOFTIRQ);
> >  }
> >  
> >
> 
> Thanks,
> 
> 	Anna-Maria