Re: Resource starvation on a 2.2.19 web server

From: Alan Cox (alan@lxorguk.ukuu.org.uk)
Date: Tue Sep 11 2001 - 09:48:03 EST


>
>
> I'm not on the linux-kernel list; please specifically include me in
> any reply. Thanks...
>
> I'm seeing a problem with what appears to be running out of file (or
> inode) handles on an apache web server. I've done everything I have
> been able to find to fix this but have run out of things to try.
> The summary below is as concise and complete as I can make it; if
> additional information is needed I'll happily provide it. I've run
> linux since 1.57, but I've never had to dig into tuning at this
> level until now.
>
> This is a 1GB dual PIII-650 box. I'm using the Mandrake 2.2.19-4.3
> source tree, their latest, as a base for the kernel. The server is
> running a couple of software raid partitions and a 26 GB EATA SCSI
> partition. Most of the file systems are ext2, with a Reiser fs on
> the SCSI partition.
>
> Based on logged errors, some time ago I started increasing
> /proc/sys/fs/{file,inode}-max to 98304 and 524300 respectively.
> The large counts are pushed by rsync in syncing the large web tree
> onto another box.
>
> >From what I have found on the 'net on kernel tuning, I have made a
> custom 2.2.19 kernel with the following changes in the include
> files. I am not seeing logged errors on any sort of resource
> starvation on this kernel; before upping the limits I was seeing
> both system and apache errors about no sockets and file handles.
>
> limits.h #define NR_OPEN 2048 (overridded by file-nr, I think)
> limits.h #define OPEN_MAX 2048 (max open files per process)
> posix_types.h: #define __FD_SETSIZE 2048
> socket.h: #define SOMAXCONN 16384
>
> The symptom is that after the box runs for a while, nslookup cannot
> resolve a name; it complains that it cannot find the configured DNS
> servers. If I USR1 warm start apache, nslookup immediately works
> again. Some system resource is being freed up when apache is
> restarted, but I've been unable to isolate what it is or fix it.
>
> The two snapshots below are of /proc/sys/fs/* before and after an
> apache warm start. There is some strange info in file-nr and
> inode-nr. Per linux/Documentation/proc.txt, the middle number of
> file-nr is "the number of used file handles", and the middle number
> of inode-nr is "the number of free inodes".
>
> After reset, the number of free inodes goes up. What seems very
> strange is that the number of free files goes from 4 to 57034,
> almost as if beyond a certain limit the file-nr counter starts
> acting as an underflow counter (just my own observation).
> Particularly inconsistent is that the number of allocated file
> handles before warm start, 58783, is well below the permitted
> maximum of 98304.
>
> Has anyone seen anything like this to help me understand where to go
> next? Thanks much in advance,
>
> -- Michael
>
>
> # before apache warm start
> 0 421083 45 0 0 0
> 1024
> 0 0
> 98304
> 58783 4 98304 <=== file-nr: note the middle number
> 524288
> 524300 43622 <=== inode-nr: note the middle number
> 524300 43622 0 0 0 0 0
> 256
> 12
>
> # after apache warm start
> 0 421625 45 0 0 0
> 1024
> 0 0
> 98304
> 58797 57034 98304 <=== was 4 before warm start
> 524288
> 524300 99808 <=== was 43622 before warm start
> 524300 99808 0 0 0 0 0
> 256
> 12
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/



This archive was generated by hypermail 2b29 : Sat Sep 15 2001 - 21:00:33 EST