Re: [PATCH v2] sched/debug, sys_info: Introduce SYS_INFO_CPU_RUNQUEUES

[Aaron Tomlin]

On Tue, Sep 22, 2026 at 04:40:27PM +0200, Peter Zijlstra wrote:
> On Fri, Sep 11, 2026 at 09:32:40PM -0400, Aaron Tomlin wrote:
> > When investigating kernel panics, inspectability of per-CPU runqueues
> > and runnable task states is valuable for diagnosing CPU starvation
> > priority inversion, etc.
> > 
> > While debugfs (/sys/kernel/debug/sched/debug) exposes runqueue metrics
> > to userspace, these details are not captured during an automated kernel
> > panic or crash dump. Capturing per-CPU runqueue state directly into
> > log_buf fills this diagnostic gap for post-mortem crash analysis.
> 
> Uh, crash-dump preserves everything.

Hi Peter,

Yes, fair point. The intention here is to preserve this state directly
within the kernel log buffer, particularly for scenarios where kdump is not
configured, the vmcore was truncated or fails to complete.
In such circumstances, dmesg, whether captured via pstore or serial, is
frequently the sole surviving diagnostic artefact.

> 
> 
> >  	rcu_read_lock();
> >  	for_each_process_thread(g, p) {
> >  		if (task_cpu(p) != rq_cpu)
> >  			continue;
> >  
> > -		print_task(m, rq, p);
> > +		if (queued_only && !task_current(rq, p) && !task_on_rq_queued(p))
> > +			continue;
> > +
> > +		print_task(m, rq, p, show_cgroup_path);
> >  	}
> >  	rcu_read_unlock();
> >  }
> > @@ -1234,7 +1242,7 @@ do {									\
> >  	print_rt_stats(m, cpu);
> >  	print_dl_stats(m, cpu);
> >  
> > -	print_rq(m, rq, cpu);
> > +	print_rq(m, rq, cpu, true, false);
> >  	SEQ_printf(m, "\n");
> >  }
> >  
> > @@ -1322,6 +1330,48 @@ void sysrq_sched_debug_show(void)
> >  	}
> >  }
> >  
> > +void sched_show_runqueues(void)
> > +{
> > +	int cpu;
> > +
> > +	pr_info("CPU Runqueues:\n");
> > +	for_each_online_cpu(cpu) {
> > +		struct rq *rq = cpu_rq(cpu);
> > +		struct task_struct *curr;
> > +		unsigned int nr_running;
> > +		u64 nr_switches;
> > +		unsigned long flags;
> > +		bool locked;
> > +
> > +		touch_nmi_watchdog();
> > +		touch_all_softlockup_watchdogs();
> > +
> > +		rcu_read_lock();
> > +		local_irq_save(flags);
> > +		locked = raw_spin_rq_trylock(rq);
> > +		if (locked) {
> > +			nr_running = rq->nr_running;
> > +			nr_switches = rq->nr_switches;
> > +			curr = rcu_dereference(rq->curr);
> > +			raw_spin_rq_unlock(rq);
> > +		} else {
> > +			nr_running = READ_ONCE(rq->nr_running);
> > +			nr_switches = READ_ONCE(rq->nr_switches);
> > +			curr = rcu_dereference(rq->curr);
> > +		}
> > +		local_irq_restore(flags);
> 
> This seems to want to avoid deadlocking on rq->lock, but then
> print_rq()->print_cfs_stats() will unconditionally take rq->lock again.
> 
> So meh.

No, print_rq() does not call print_cfs_stats(). It is print_cpu() (i.e.
used by debugfs and SysRq) that calls print_cfs_stats() and unconditionally
takes rq->lock via print_cfs_rq().

With sched_show_runqueues(), it specifically avoids print_cpu() and only
calls print_rq(NULL, rq, cpu, false, true). Inside print_rq(), it merely
iterates threads under rcu_read_lock() and calls print_task(), neither of
which acquires rq->lock. Hence, rq->lock is never taken again, and the
non-blocking guarantee of the trylock remains intact throughout the entire
dump.

> 
> > +
> > +		pr_info("cpu#%d: nr_running:%u switches:%llu curr:%s[%d]%s\n",
> > +			cpu, nr_running, nr_switches,
> > +			curr ? curr->comm : "<none>",
> > +			curr ? task_pid_nr(curr) : -1,
> > +			locked ? "" : " (contended)");
> > +
> > +		print_rq(NULL, rq, cpu, false, true);
> > +		rcu_read_unlock();
> > +	}
> > +}
> > +
> >  /*
> >   * This iterator needs some explanation.
> >   * It returns 1 for the header position.
> > diff --git a/lib/sys_info.c b/lib/sys_info.c
> > index f32a06ec9ed4..fc5bfcc121de 100644
> > --- a/lib/sys_info.c
> > +++ b/lib/sys_info.c
> > @@ -22,6 +22,7 @@ static const char * const si_names[] = {
> >  	[ilog2(SYS_INFO_PANIC_CONSOLE_REPLAY)]	= "",
> >  	[ilog2(SYS_INFO_ALL_BT)]		= "all_bt",
> >  	[ilog2(SYS_INFO_BLOCKED_TASKS)]		= "blocked_tasks",
> > +	[ilog2(SYS_INFO_CPU_RUNQUEUES)]		= "cpu_runqueues",
> >  };
> >  
> >  /*
> > @@ -158,6 +159,9 @@ static void __sys_info(unsigned long si_mask)
> >  
> >  	if (si_mask & SYS_INFO_BLOCKED_TASKS)
> >  		show_state_filter(TASK_UNINTERRUPTIBLE);
> > +
> > +	if (si_mask & SYS_INFO_CPU_RUNQUEUES)
> > +		sched_show_runqueues();
> >  }
> 
> I really don't know if this is worth the trouble. I have *never* needed
> this.

I understand. However, the requirement primarily arises in production
support. In environments where kdump is not configured or the resulting
vmcore is truncated, dmesg is frequently the sole surviving diagnostic
artefact.

During an NMI-induced panic, capturing per-CPU runqueue depth in the log is
invaluable for identifying CPU starvation, etc. As with the remainder of
sys_info, this functionality is strictly opt-in and disabled by default,
imposing no overhead on systems that do not require it while providing
vital visibility where full memory dumps are unavailable.

Kind regards,
-- 
Aaron Tomlin