>What would be the optimal paradigm for this?
[....]
>I think a HOWTO on this subject might be in order :)
Ok. Here is a mini-mini-howto.
-------------------------------------------------------------------------
This is the many-fds-mini-howto.
-------------------------------
Some applications require many, many file descriptors on a
server. This mini howto explains how to get your Linux system capable
of handling this.
The server program.
------------------
The server programs where this problem happens most often is a program
having lots of persistent "client" programs that connect to the server
using sockets. Other applications may need lots of disk-based
filedescriptors, but those are generally less common.
The easiest way to handle persistant client connections is to fork a
new process, each process handling one client. This scales until you
have a few hundred servers running. Then RAM on your server becomes
a scarce resource, and maybe you'll hit the max amount of processes.
The server then gets rewritten to have a select loop. This scales a
bit further, but not all that far. RAM usage is usually much
better. However once you exhaust the max number of filedescriptors
per process, you're stuck once again.
I recommend writing the server to fork off a worker process every time
when you get near the filedescriptors limit. In the server that I've
written, I've just chosen the "fork moment" to be 100 clients. That
means that I can handle 100 times more clients than when I would fork
for every client.
It however complicates the server a bit. You have both the fork code
and the select loop. The advantage of being able to handle thousands
of clients is however worth it if you expect that useage pattern.
The kernel.
----------
Newer Kernels can support more file descriptors per process. You need
kernel ..... for this to be in the standard kernel.
There is no kernel-wide max-fd limit. ??? <- Someone correct me on this.
I'm not sure....
Roger Wolff.
-- ** R.E.Wolff@BitWizard.nl ** http://www.BitWizard.nl/ ** +31-15-2137555 ** *-- BitWizard writes Linux device drivers for any device you may have! --* ------ Microsoft SELLS you Windows, Linux GIVES you the whole house ------
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/