BUG: soft lockup on all kernels after 2.6.3x

[Alexey Vlasov]

Hello.

I used 2.6.2x kernel for a long time on my shared hosting and I didn't
have any problems. Kernels worked well and server uptime was about 2-3
years.

But investigating some strange hangings of my clients' sites I came to
this:
http://bugs.mysql.com/bug.php?id=50399
from this bug it is clear that on kernels younger than 2.6.32 (
unfortunately I can't remember if it is true with 2.6.30-31) happens
mysql client hanging.

It is not clear whether it is a bug of kernel or libc or mysql-client, I
didn't manage to find it out. I decided to do simpler (as it seemed to
me at that moment) to start using 2.6.3x kernels. And that caused
greater problems. By trying to use new kernels on my working servers
with peak load I got an uptime from an hour to 1-3 months.

I even got some statistics for how long can every kernel from version
2.6.32 work in peak load situations. It sounds funny but my clients are
not happy with all these reboots.

>From all the variety of servers from 2.6.32 to 3.7.4 I can say that
2.6.35 is the most stabil, I got about 30 servers on it. But they hang
usually once in 1-3 months.

Returning to the problem of kernels >= 2.6.32, as I have noticed they
hang totally alike, giving the console:

...
Feb  8 10:27:45 10.2.0.7 [470393.417168] BUG: soft lockup - CPU#2 stuck for 61s! [vsftpd:29013]
...
[see the attachment]

it doesn't happen on an empty server, only on loaded ones. Unfortunately
I don't know how to provoke such hanging artificially.

I' ve given a trace attached. In fact I don't know what to do with all
these bugs, I can't use 2.6.2x because of MySQL hanging and = >2.6.3
start hanging themselves.

-- 
BRGDS. Alexey Vlasov.

Attachment: bug_spinlock.txt.gz
Description: Binary data