Re: backing up ext4 fs, system unresponsive, thrashing like crazyeven though swap is unused

[Dimitrios Apostolou]

Dear Ronald,

sorry for not replying at your first message but I didn't consider 
changing kernel as a resolution to my problem.

On Sun, 2 Dec 2012, Roland Eggner wrote:
> Hello Dimitrios!
>
>
> Which part of â
>
> â0,5 â 1 G RAM is usually occupied just by kernel slab [1];  this
> memory cannot be swapped, it cannot be released by other means than rebooting,
> and there seems to be  _no_  adjustment to memory pressure.â
>
> â should I explain?

Check my initial message. If you see the /proc/meminfo attached you'll see 
that from all slab memory most is reclaimable, however it's not being 
reclaimed despite the memory pressure:

Slab:              64880 kB
SReclaimable:      60496 kB
SUnreclaim:         4384 kB

Also I mentioned that a nice'd -20 (Realtime!) process was unresponsive 
because of two nice'd + ionice'd processes. This is a bug no matter the 
memory pressure, IMHO. If not I'd rather hear it from someone more 
authoritative.

> When tar or gzip writes to ext4 filesystem on your external disk, the kernel
> keeps all inode data in slab memory  _by design_ , not by memory leaks.

ext4_inode_cache is a *cache*. So the kernel does not keep /all/ inode 
data there, but only what was needed recently. And drops it afterwards, 
which didn't happen after the backup ended.

> Several 100 M slab data cannot be stored within 128 M total RAM.
> This cannot by solved by usage of /proc/sys/vm/drop_caches.

Please take a look at [2], it's what is called in that case. Via this call 
all slabs that have registered a shrinker, get actually shrinked. 
Other fs do, but I can't find whether ext4 actually registers a shrinker 
for its slabs.

[2] http://lxr.linux.no/#linux+v3.6.8/mm/vmscan.c#L207


> This cannot by solved by oom-killer-actions.
> This cannot be solved by zram tricks.
> Unmounting the external disk would release part of slab memory, but then you
> cannot backup.

Unmounting the back'ed up fs didn't really help in my case. Of course all 
my filesystems are ext[234], mounted by ext4 code, so they use the same 
slab for caches. But anyway, the slab didn't shrink significantly as I was 
expecting.

> Reformatting the ext4 filesystem on your external disk with 128 byte inode size,
> largest possible blocksize and with extents enabled might mitigate the memory
> pressure slightly â probably not sufficient to get a working system with 3.6
> kernel.
>
>
> One advantage of Linux compared to other OS is much more support for old
> hardware, if a  _proper_  kernel version is selected.  Many years ago I used
> a notebook with 40 M total RAM, with a 2.4 kernel, Blackbox window manager and
> Opera web browser â it worked flawlessly â just rather slowly due to swapping.
> Your notebook has much more RAM than 40 M, thus there is surely a Linux solution
> for you.  Try a 2.6.27.62 kernel, it supports ext4 (âext4devâ), probably it
> supports all devices of your notebook, and with a slim window manager e.g.
> WindowMaker or OpenBox your notebook will probably âflyâ.

I appreciate your advice. I remember when we were running 2.4 much less 
memory was needed, but still I consider the kernel /fairly/ lean 
considering the time passed and the bloat in userspace applications. I 
choose to run latest kernels on old hardware, hopefully this will continue 
to be the case, and if things deteriorate much then let's hope we'll have 
enough time to help and fix them! :-)


Thanks,
Dimitris