> I've been been having a rather obnoxious problem with "cannot spawn
> child process" errors in my apache logs (1.2.4 and 1.2.0) (around 20,000
> - 50,000 a day). Since I have MaxClients, MinSpareServers and
> MaxSpareServers set at rediculously high values, I'm pretty sure the
> kernel is to blame.
But are you sure that it's because of filedescriptors?
What kernel are you running? If it's 2.0.30 you may be running into the
problems with the swap system.
> echo 4096 > /proc/sys/kernel/file-max
> echo 12288 > /proc/sys/kernel/inode-max
Scary thing is that I went through ALL of the code edits you did :)
Eventually Michael did a much nicer patch. It (along with header hacks
I did) is available at 'http://www.linux.org.za/filehandle.patch.linux'.
Before you go wild and apply this patch please read the below though :)
> echo 300 400 500 > /proc/sys/vm/freepages
> which the kernel HOWTO suggested, although it was unclear what exactly
> this is supposed to do.
Alleviate the swap system problems, I guess
> If anyone has any ideas why apache might still be misbehaving or if I'm
> overlooking something else I need to do to increase the effective
> maximum number of file descriptors, I'd be much obliged.. the Apache
> documentation said that the 'cannot spawn child process' is generally
> related to a shortages of file descriptors, so I assumed that was my
> problem. If anyone has any other ideas, please let me know. The server
> is doing primarily CGI (perl), if that any help.
>
> Also, if someone could shed some light on what OPEN_MAX and NR_OPEN
> control and how they differ, that would be helpful as well.
Sure. It's been a while, but I hope that this helps:
There are basically 2 limits to filedescriptors:
1) Per process filedescriptors
2) Overall system filedescriptors
With a threaded program (or a 'huge select loop' like squid) you quite often
hit the first limit. If you have 50 established connections to your program
you use 50 FD's... plus for files you read etc... I made my initial patch
for squid... and Michael made a better one for an even heavier loaded squid.
This is normally set at 256.
With apache, though, you don't have one process... you have lots, and they
don't all have all the FD's open at the same time.... if you have 256
processes running at once (all using a few FD's) then you could hit the
overall system limit.
To change the per-process limit you normally have to recompile with the
new settings. (otherwise you are stuck with the old limits still). But the
overall system FD's used aren't known by any process, so you can change those
without a problem.
To see how many FD's your system has used overall type this:
[root@kasmilos kernel]# cat /proc/sys/kernel/file-nr
1680
(this is on our FTP server - lots of processes with a few FD's each...)
Note that the number never goes down, so if you have a burst of usage then
you can't tell when it disappears....
To see what your overall limit is, type this:
[root@kasmilos kernel]# cat /proc/sys/kernel/file-max
4096
This means that between all of my processes I can open 4096 FD's.
If my 'file-nr' value was close to 'file-max' I could increase it by putting
something like this in my 'rc.local':
#Increase systemwide filehandle max
/bin/echo 4096 >/proc/sys/kernel/file-max
/bin/echo 12288 > /proc/sys/kernel/inode-max
Note that if you increase the 'file-max' value you MUST increase the
'inode-max' value, and normally in the same ratio (double one, double the
other)
To find out how many FD's a running process is using, do this:
[root@kasmilos kernel]# cat /usr/local/lib/httpd/logs/httpd.pid
5616
[root@kasmilos kernel]# ls -l /proc/5616/fd/ | wc -l
15
means it's using 15 FD's.... if this was close to 256 (in the default config
or 1024 in your new kernel) then you would have a problem. (You may also have
to check the child processes to see if they are the possible cause).
Hope that this helps...
Oskar
-- "Haven't slept at all. I don't see why people insist on sleeping. You feel so much better if you don't. And how can anyone want to lose a minute - a single minute of being alive?" -- Think Twice