Re: excessive kworker activity when idle. (was Re: vma corruptionin today's -git)
From: Tejun Heo
Date: Thu Mar 31 2011 - 02:56:52 EST
On Wed, Mar 30, 2011 at 08:37:21PM -0700, Linus Torvalds wrote:
> On Wed, Mar 30, 2011 at 8:09 PM, Dave Jones <davej@xxxxxxxxxx> wrote:
> > But rerunning the same tests on current head (6aba74f2791287ec407e0f92487a725a25908067)
> > I can still reproduce the problem where kworker threads go nutso
> > when the machine should be completely idle.
> > top shows two kworker threads constantly at >80% cpu.
> Ok, I've seen that "tons of cpu by kworker" triggered by a few
> different issues. One was the intel graphics driver doing monitor
> detection constantly, and spending all its time in one of the worker
> threads doing __udelay() for the stupid i2c driver.
> The other case I've seen is a wireless thing that falls back to GPIO,
> and spends a lot of CPU time on that.
> I'm not saying yours is either of those cases, but one of the problems
> with that behavior is that it's actually fairly hard to figure out
> what the hell is happening. You don't see some nice thread description
> in 'top' any more (like you used to when everybody created their own
> threads and didn't do the common worker thread thing), and the best
> approach literally seems to be something like
Yes, not having dedicated workers decrease immediate visibility via
ps/top but I don't think the actual vsibility is worse. There are two
types for kworker looping - something being scheduled in rapid
succession or a single work consuming lots of CPU cycles.
The first one can be tracked using tracing.
$ echo workqueue:workqueue_queue_work > /sys/kernel/debug/tracing/set_event
$ cat /sys/kernel/debug/tracing/trace_pipe > out.txt
(wait a few secs)
If something is busy looping on work queueing, it would be dominating
the output and the offender can be determined with the work item
For the second type, "cat /proc/THE_OFFENDING_KWORKER/stack" is the
easiest. The work item function will be trivially visible in the
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/