Re: [PATCH v3] scheduler: enhancement to show_state_filter and SysRq

[Peter Zijlstra]

On Wed, Aug 09, 2017 at 05:26:14PM +0800, Yafang Shao wrote:
> 2017-08-09 17:09 GMT+08:00 Peter Zijlstra <peterz@xxxxxxxxxxxxx>:
> > On Wed, Aug 09, 2017 at 04:01:49PM +0800, Yafang Shao wrote:
> >> 2017-08-09 15:43 GMT+08:00 Peter Zijlstra <peterz@xxxxxxxxxxxxx>:
> >> > On Wed, Aug 09, 2017 at 06:31:28PM +0800, Yafang Shao wrote:
> >> >> Sometimes we want to get tasks in TASK_RUNNING sepcifically,
> >> >> instead of dump all tasks.
> >> >>
> >> >> For example, when the loadavg are high, we want to dump
> >> >> tasks in TASK_RUNNING and TASK_UNINTERRUPTIBLE, which contribute
> >> >> to system load. But mostly there're lots of tasks in Sleep state,
> >> >> which occupies almost all of the kernel log buffer, even overflows
> >> >> it, that causes the useful messages get lost. Although we can
> >> >> enlarge the kernel log buffer, but that's not a good idea.
> >> >
> >> > That's what you have serial consoles for...
> >> >
> >> mostly we don't even have one console because we alwayas login the
> >> servers via ssh. And manage the servers with console is not so convenient.
> >
> > I find IPMI SOL very useful. Serial console (esp. earlyprintk) keeps on
> > working long after most other things have died.
> >
> > In any case, you can easily dump the printk output into your ssh session
> > if you want, use something like:
> >
> >         cat /dev/kmsg | tee logfile & echo t > /proc/sysrq-trigger
> 
> that's what I'm doing it currently :)
> Then I thought deeply why not do it more smartly?
> Introducing a new key(here I just modified the key 'w') only dump
> tasks in running and blocked should be more smarter.

Since you're strictly ssh based, you could maybe do a sysctl that allows
changing the 'default' filter of sysctl-t, dunno if that makes sense
though.

Also, since you're not actually debugging a dead machine, maybe you can
do a custom kernel module / systemtap / ebpf thing that collects
precisely the information you want.

sysrq is typically a last ditch debug mostly dead machine thing, which
you're very much not having.

> >
> > I really see no problem here. Then you can run a bit of awk or whatever
> > your favourite tool is to filter out the stuff you don't want.
> >
> Another question, if we could filter with scritpts in userland, why
> did we introduced the key 'w' to dump only blocked state
> as we already have a key 't' to dump all tasks ?

No idea that is long before my time, I expect because 'w' (blocked) is
typically a small number of tasks.