On 10/17/02 12:08, GrandMasterLee wrote:
> On Wed, 2002-10-16 at 22:54, Michael Clark wrote:
>
>>On 10/17/02 11:12, GrandMasterLee wrote:
>>
>>>On Wed, 2002-10-16 at 11:49, Michael Clark wrote:
>>>
>>>>Seems to be the correlation so far. qlogic driver without lvm works okay.
>>>>qlogic driver with lvm, oopsorama.
>>>
>>>
>>>Michael, what exactly do your servers do? Are they DB servers with ~1Tb
>>>connected, or file-servers with hundreds of gigs, etc?
>>
>>My customer currently has about 400Gb on this particular 4 node Application
>>cluster (actually 2 x 2 node clusters using kimberlite HA software).
>>
>>It has 11 logical hosts (services) spread over the 4 nodes with services such
>>as Oracle 8.1.7, Oracle Financials (11i), a busy openldap server, and busy
>>netatalk AppleShare Servers, Cyrus IMAP server. All are on ext3 partitions
>>and were previously using LVM to slice up the storage.
>
>
> On each of the Nodes, correct?
We had originally planned to split up the storage in the RAID head
using individual luns for each cluster logical host - so we could use SCSI
reservations - but we encountered problems with the RAID heads device queue.
The RAID head has a global queue depth of 64 and to alleviate
queue problems with the RAID heading locking up, we needed to minimise
the number of luns, so late in the piece we added LVM to split up the storage.
We are using LVM in a clustered fashion. ie. we export most of the array
as one big lun, and slice it into lvs, each one associated with a logical host.
All lvs are accessible from all 4 physical hosts in the cluster. Care and
application locking ensures only 1 physical host mounts any lv/partition at
the same time (except for cluster quorum partitions which need to be accessed
concurrently from 2 nodes - and for these we have seperate quorum disks in
the array).
lvm metadata changes are made from one node while the others are down
(or just have volumes deactivated, unmounted, then lvm-mod removed)
to avoid screwing our metadata because lvm is not cluster aware.
We are not using mutlipath put have the cluster arranged in a topology
such that the HA RAID Head has 2 controllers with each side of the cluster
hanging of a different one ie. L and R. If we have a path failure, we will
just loose CPU capacity (25-50% depending). The logical hosts will automatically
move onto a physical node which still has connectivity to the RAID head
(by the cluster software checking of connectivity to the quorum paritions).
This gives us a good level of redundancy without the added cost of 2 paths
from each host. ie. after a path failure we run with degraded performance only.
We are using vanilla 2300's.
~mc
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
This archive was generated by hypermail 2b29 : Wed Oct 23 2002 - 22:00:32 EST