Re: [PATCH 0/3] rcu: Add RCU stall diagnosis information
From: Leizhen (ThunderTown)
Date: Mon Oct 24 2022 - 05:23:52 EST
On 2022/10/22 3:15, Paul E. McKenney wrote:
> On Fri, Oct 21, 2022 at 04:07:43PM +0800, Leizhen (ThunderTown) wrote:
>>
>>
>> On 2022/10/21 7:13, Paul E. McKenney wrote:
>>> On Mon, Oct 17, 2022 at 06:01:05PM +0800, Zhen Lei wrote:
>>>> In some extreme cases, such as the I/O pressure test, the CPU usage may
>>>> be 100%, causing RCU stall. In this case, the printed information about
>>>> current is not useful. Displays the number and usage of hard interrupts,
>>>> soft interrupts, and context switches that are generated within half of
>>>> the CPU stall timeout, can help us make a general judgment. In other
>>>> cases, we can preliminarily determine whether an infinite loop occurs
>>>> when local_irq, local_bh or preempt is disabled.
>>>>
>>>> Zhen Lei (3):
>>>> sched: Add helper kstat_cpu_softirqs_sum()
>>>> sched: Add helper nr_context_switches_cpu()
>>>> rcu: Add RCU stall diagnosis information
>>>
>>> Interesting approach, thank you!
>>>
>>> I have pulled this in for testing and review, having rescued it from my
>>> spam folder.
>>
>> Thanks. My company's mail system has been having some problems lately.
>>
>> Also, I need to apologize that yesterday I found out there was a mistake
>> in patch 3/3. Yesterday, I finally got to print_other_cpu_stall() by forcing
>> a stub.
>
> OK, I done dropped your three patches for the time being.
>
> Please feel free to submit v2 whenever you are ready to do so.
>
>> diff --git a/kernel/rcu/tree_stall.h b/kernel/rcu/tree_stall.h
>> index 08cfcf526f7d245..caaee5f4ee091df 100644
>> --- a/kernel/rcu/tree_stall.h
>> +++ b/kernel/rcu/tree_stall.h
>> @@ -451,7 +451,7 @@ static void print_cpu_stat_info(int cpu)
>> if (r->gp_seq != rdp->gp_seq)
>> return;
>>
>> - cpustat = kcpustat_this_cpu->cpustat;
>> + cpustat = kcpustat_cpu(cpu).cpustat;
>> half_timeout = rcu_jiffies_till_stall_check() / 2;
>>
>> pr_err(" hardirqs softirqs csw/system\n");
>>
>>>
>>> Some questions that might come up include: (1) Can the addition of
>>> things like cond_resched() make RCU happier with the I/O pressure test?
>>> (2) Should there be a way to turn this off for environments with slow
>>> consoles? (3) If this information shows heavy CPU usage, what debug
>>> and fix approach should be used?
>>
>> If the CPU usage is high due to busy services, I think it is excusable
>> to report RCU stall warning.
>
> Not so much. RCU CPU stall warning usually means that there is some
> overly long loop in the kernel.
Yes, so the added statistics are helpful. Low count means long loop may exist.
>
>> When users see RCU stall, they are most
>> worried about whether there are unrecoverable errors, such as dead loop.
>> If the cause is known to be the CPU usage, the I/O performance has
>> reached its peak, this is probably what people want to see.
>
> You have a workload that can be carried out entirely in interrupt
> handlers? If not, the kernel code really should be updated to avoid
> the RCU CPU stall warnings.
>
>> (1) This needs to be considered by the business task itself. As far as I
>> know some drivers' data processing is done in an interrupt context.
>
> You are seeing an entire CPU being consumed by interrupt handlers?
> If so, is the CPU being carefully placed in the idle loop beforehand?
> If not, how do you keep the long delays in the interrupted processing
> from causing problems?
Some chips with poor implementation may respond to interrupts on only a
few cores. However, I didn't actually see. After all, these process
information, if not specifically recorded, will not be seen again.
>
>> (2) Do you mean to suppress such new debugging information that I added?
>> or the whole RCU stall information?
>
> Only the new debugging information. I already get complaints about RCU
> CPU stall warnings producing more output than people like.
OK, I got it.
Perhaps we can consider recording the RCU stall information in the buffer
and printing the following brief information.
Warning: RCU stall id=%d
Then, the user-mode process accesses the new added RCU attribute file and
queries detailed information based on the ID.
Or delay printing until the new GP or panic.
>
>> (3) The statistics can be accurate to a single hard interrupt, software
>> interrupt, or task. However, the price will be higher. Users can
>> recall what they did at the time, then reproduce it. Maybe we can get
>> this code ready, add a new debugging option, and turn it on when needed.
>
> Let me ask this a different way. What combination of numbers would lead
> you to believe that a given RCU CPU stall warning can safely be ignored?
>
>>> For an example of #1, if a CPU is flooded with softirq activity, one
>>> might hope that the call to rcu_softirq_qs() would prevent the RCU CPU
>>> stall warning, at least for kernels built with CONFIG_PREEMPT_RT=n.
>>> Similarly, if there are huge numbers of context switches, one might hope
>>> that the rcu_note_context_switch() would report a quiescent state sooner
>>> rather than later.
>>
>> Good idea. I'm going to dig deeper.
>>
>> How about dynamically extending the stall timeout if the CPU usage is too high?
>
> Let's first work out why the RCU CPU stalls are happening. After all, if
OK
> there is a particular case that is truly harmless, it would be better to
> arrange that stalls not be printed at all in that case. It is better to
> just not have false positives, right?
Yes.
>
> Thanx, Paul
>
>>> Thoughts?
>>>
>>> Thanx, Paul
>>>
>>>> include/linux/kernel_stat.h | 12 +++++++++++
>>>> kernel/rcu/tree.h | 11 ++++++++++
>>>> kernel/rcu/tree_stall.h | 40 +++++++++++++++++++++++++++++++++++++
>>>> kernel/sched/core.c | 5 +++++
>>>> 4 files changed, 68 insertions(+)
>>>>
>>>> --
>>>> 2.25.1
>>>>
>>> .
>>>
>>
>> --
>> Regards,
>> Zhen Lei
> .
>
--
Regards,
Zhen Lei