Ports 59873 - 60000 in use. Not sure by what.

[Jeremy Freeman]

I have exhausted all other avenues to solve this. So in a last ditch 
effort I am posting to KML.

For some reason on one of my servers ports 59873 through 60000 are bound 
to some mystery process.

netstat -nap shows nothing using them.
lsof shows nothing using them.

However they are most definitely in use.

I'll use nc for an example:

# nc -l 59872
.. works and listens ...

# nc -l 59873
nc: Address already in use

... < all ports in between> ...

# nc -l 60000
nc: Address already in use

nc -l 60001
.. works and listens ...

stracing nc shows:
bind(3, {sa_family=AF_INET, sin_port=htons(60000), 
sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EADDRINUSE (Address already in use)

So the obvious culprit is a rootkit of some-sort. I checked the system 
using rkhunter and chkrootkit and they found nothing. Ran lsof, tcpdump 
and netstat from clean binaries on write-protected media.. also nothing. 
Further, this system has not been "on-net".. so although I am not 
disqualifying this as the issue, I cannot find any evidence.

I tried to run kstat but there does not seem to be a version for 2.6.

I tried changing my ip_local_port_range to 32768 - 55000 and the issue 
persists.

Even in run-level 1 those ports cannot be bound to.

BOX is: Red Hat Enterprise Linux Server release 5.1 (Tikanga)
Kernel: 2.6.18-53.1.14.el5 #1 SMP Tue Feb 19 07:18:46 EST 2008 x86_64 
x86_64 x86_64 GNU/Linux

All I can think is the kernel is somehow reserving these ports for 
outgoing use or something? Which I am not even sure about because I 
changed my ip_local_port_range to not include those ports and they are 
still held.

So.. now I am out of ideas.. perhaps someone out there can help me or 
give me some other ideas to try.

Thank you.

Please CC me if possible as I am not subscribed.

--
Jeremy
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/