Re: Debugging system freezes on filesystem writes

[Marcus Sundman]

On 13.11.2012 15:51, Jan Kara wrote:
> On Fri 09-11-12 15:12:43, Marcus Sundman wrote:
> > On 09.11.2012 01:41, Marcus Sundman wrote:
> > > On 07.11.2012 18:17, Jan Kara wrote:
> > > > On Fri 02-11-12 04:19:24, Marcus Sundman wrote:
> > > > > Also, and this might be important, according to iotop there is
> > > > > almost no disk writing going on during the freeze. (Occasionally
> > > > > there are a few MB/s, but mostly it's 0-200 kB/s.) Well, at least
> > > > > when an iotop running on nice -20 hasn't frozen completely, which it
> > > > > does during the more severe freezes.
> > > >    OK, it seems as if your machine has some problems with memory
> > > > allocations. Can you capture /proc/vmstat before the freeze and
> > > > after the
> > > > freeze and send them for comparison. Maybe it will show us what is the
> > > > system doing.
> > > t=01:06 http://sundman.iki.fi/vmstat.pre-freeze.txt
> > > t=01:08 http://sundman.iki.fi/vmstat.during-freeze.txt
> > > t=01:12 http://sundman.iki.fi/vmstat.post-freeze.txt
> > Here are some more vmstats:
> > http://sundman.iki.fi/vmstats.tar.gz
> >
> > They are from running this:
> > while true; do cat /proc/vmstat > "vmstat.$(date +%FT%X).txt"; sleep
> > 10; done
> >
> > There were lots and lots of freezes for almost 20 mins from 14:37:45
> > onwards, pretty much constantly, but at 14:56:50 the freezes
> > suddenly stopped and everything went back to how it should be.
>    I was looking into the data but they didn't show anything problematic.
> The machine seems to be writing a lot but there's always some free memory,
> even direct reclaim isn't ever entered. Hum, actually you wrote iotop isn't
> showing much IO going on but vmstats show there is about 1 GB written
> during the freeze. It is not a huge amount given the time span but it
> certainly gives a few MB/s of write load.

I didn't watch iotop during this particular freeze. I'll try to keep an 
eye on iotop in the future. Is there some particular options I should 
run iotop with, or is a "nice -n -20 iotop -od3" fine?

> There's surprisingly high number of allocations going on but that may be
> due to the IO activity. So let's try something else: Can you switch to
> console and when the hang happens press Alt-Sysrq-w (or you can just do
> "echo w >/proc/sysrq-trigger" if the machine is live enough to do that).
> Then send me the output from dmesg.  Thanks!

Sure! Here are two:
http://sundman.iki.fi/dmesg-1.txt
http://sundman.iki.fi/dmesg-2.txt


Best regards,
Marcus

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/