Re: backing up ext4 fs, system unresponsive, thrashing like crazyeven though swap is unused

[Roland Eggner]

On 2012-11-25 Sun 23:59:55 +0100 Roland Eggner wrote:
>On 2012-11-25 Sunday at 21:30 +0200 Dimitrios Apostolou wrote:
> > On Sun, 2012-11-25 at 15:55 +0200, Dimitrios Apostolou wrote:
> > > on an old PIII-500MHz laptop, 128MB RAM, kernel 3.6.6, I started a
> > > backup process (tar|xz -4, nice'd and ionice'd -c3) from ext4 on local
> > > ATA disk to ext3 on external USB disk (USB-2.0 port on PCMCIA card).
> > > Even though earlier system load was minimal, free memory was plenty, the
> > > system now is unresponsive and is thrashing the disk, but the swapfile
> > > is rarely touched.
> >
> > I'm now having the same experience even though I replaced xz (which
> > needed ~50MB RAM) with gzip. Even though I feel the realtime root shell
> > is a bit more responsive than before, the OOM killer is out killing
> > small processes like syslog-ng and systemd-logind... The
> > ext4_inode_cache slab is taking almost all my memory (117MB). Please
> > advise!
> 
> Hello Dimitrios,
> 
> I would try a 2.6.27.* kernel, for following reasons:
> 
> (1)  Kernel development since 2.6.27 achieved significant performance
> improvements at the cost of exploding memory consumption by the kernel for
> _internal_  data structures.  I am currently using a 3.2.34 kernel on a Notebook
> with 4 G RAM.  0,5 â 1 G RAM is usually occupied just by kernel slab [1];  this
> memory cannot be swapped, it cannot be released by other means than rebooting,
> and there seems to be  _no_  adjustment to memory pressure.  I am surprised,
> that you have managed to boot a 3.6.* kernel at all with only 128 M RAM.
> 
> (2)  At the time, when I used a PIII-Notebook, kernels 2.6.27 to 2.6.29 where
> current.  Thus chances are good, that a 2.6.27.* kernel will support chipset,
> PCI bus and devices of your notebook.  2.6.27 got longterm maintainance, the
> latest release in the linux-stable git repository is 2.6.27.62.
> So Q @ LKML community:
> Does anybody know a x86 distribution or live-CD using a 2.6.27.* kernel?
> 
> 
> [1]  Picture described in my LKML message
> Date:  Fri, 20 Jan 2012 01:08:00 +0100
> Subject:  Re: [kmemleak report 1/2] kernel 3.1.6, x86_64: mm, xfs ?, vfs ?
> remained the same with  _every_  3.1.* and 3.2.* kernel tried so far.


On 2012-12-02 Sunday at 14:44 +0200 Dimitrios Apostolou wrote:
> Hi, the problem in the quoted message still happens, shouldn't all of
> ext4_inode_cache slab be emptied after "echo 3
> > /proc/sys/vm/drop_caches"? In my case slab uses too much memory even
> after all processes finish and system is in bad shape due to lack of
> physical RAM. CC'ing tytso and Catalin Marinas since I've not been able
> to track any leak with kmemleak.
> 
> Kernel is booted with slub_debug=,ext4_inode_cache, as this is the only
> way to avoid for some time the following message. Nevertheless it has
> not been able to show any leak.
> 
> kmemleak: Cannot allocate a kmemleak_object structure
> kmemleak: Automatic memory scanning thread ended
> kmemleak: Kernel memory leak detector disabled
> 
> Please have a look at the log (archived at [1]).
> 
> [1] http://lkml.indiana.edu/hypermail/linux/kernel/1211.3/00183.html


Hello Dimitrios!


Which part of â

â0,5 â 1 G RAM is usually occupied just by kernel slab [1];  this
memory cannot be swapped, it cannot be released by other means than rebooting,
and there seems to be  _no_  adjustment to memory pressure.â 

â should I explain?


When tar or gzip writes to ext4 filesystem on your external disk, the kernel 
keeps all inode data in slab memory  _by design_ , not by memory leaks.
Several 100 M slab data cannot be stored within 128 M total RAM.
This cannot by solved by usage of /proc/sys/vm/drop_caches.
This cannot by solved by oom-killer-actions.
This cannot be solved by zram tricks.
Unmounting the external disk would release part of slab memory, but then you
cannot backup.
Reformatting the ext4 filesystem on your external disk with 128 byte inode size, 
largest possible blocksize and with extents enabled might mitigate the memory 
pressure slightly â probably not sufficient to get a working system with 3.6 
kernel.


One advantage of Linux compared to other OS is much more support for old 
hardware, if a  _proper_  kernel version is selected.  Many years ago I used
a notebook with 40 M total RAM, with a 2.4 kernel, Blackbox window manager and
Opera web browser â it worked flawlessly â just rather slowly due to swapping.  
Your notebook has much more RAM than 40 M, thus there is surely a Linux solution 
for you.  Try a 2.6.27.62 kernel, it supports ext4 (âext4devâ), probably it 
supports all devices of your notebook, and with a slim window manager e.g.  
WindowMaker or OpenBox your notebook will probably âflyâ.


PS:  Please type your reply below the text you are replying to.

-- 
Roland

Attachment: pgp00000.pgp
Description: PGP signature