Thanks Jeff for the data. I went back and re-read the code.
It looks like the template cmd_per_lun value is recalculated before the
scsi-register call is made by the driver. The number is 255 / number of
units (which you said from your previous post is one). This then matches
your proc info. From the numbers these look like pretty nice cards :-).
-Mike
Jeff V. Merkey [jmerkey@vger.timpanogas.org] wrote:
>
>
> Mike,
>
> Here are some numbers from the running system. This system is
> running at 120 MB/S on a single 3Ware adapter. Stats attached. You
> will note that the max commands hitting the adapter are way above
> 15. I can only presume this is due to caching behavior on the card.
> I do have these cards enabled with caching. I have had these numbers
> as high as 319 MB/S with multiple cards in separate buses. The system
> this test is running on has 3 PCI buses. 2 x 33 Mhz and 1 x 66 Mhz.
> with Serverwork HE Chipset.
>
> We are averaging complete saturation of the inbound command queue
> on the 3Ware adapter and are posting 64K writes (128 * 512) * 3 on
> aeverage with each command. I have no idea why the 15 queue depth
> limit is being exceeded here, possibly due to the fact the adapter
> is caching, but the stats report up to 255 requests being posted
> per driver pass.
>
> cat /proc/scsi/3w-xxxx/3
>
> scsi3: 3ware Storage Controller
> Driver version: 1.02.00.016
> Current commands posted: 205
> Max commands posted: 255
> Current pending commands: 0
> Max pending commands: 0
> Last sgl length: 0
> Max sgl length: 3
> Last sector count: 128
> Max sector count: 256
> Resets: 0
> Aborts: 0
> AEN's: 0
>
>
> cat /proc/scsi/3w-xxxx/3
>
> scsi3: 3ware Storage Controller
> Driver version: 1.02.00.016
> Current commands posted: 7
> Max commands posted: 255
> Current pending commands: 0
> Max pending commands: 0
> Last sgl length: 0
> Max sgl length: 3
> Last sector count: 128
> Max sector count: 256
> Resets: 0
> Aborts: 0
> AEN's: 0
>
>
> This test is posting a max of 54,000 4K buffer heads to a single 3Ware
> adapter. See below:
>
>
> trxinfo -a
>
> ioctl TRXDRV_QUERY_AIO ret 0
> TRXDRV-AIO STATS
> aio_submitted-3 aio_completed-25601331 aio_error-0
> disk_aio_submitted-25601539 disk_aio_completed-25601331
> sync_active [ x x x x x x x x ]
> async_active [ x x x x x x x x ]
> cb_active-0 aio_sequence-0
> bh_count-65536 bh_inuse-32208 bh_max_inuse-54593 bh_waiters-0
> hash_hits-0 hash_misses-0 hash_fill-3 hash_total-3
> probe_avg-0 probe_max-1
> total_read_req-0 total_write_req-25601536 total_fill_req-0
> total_complete-25739847
> req_sec-0 seconds-12925
>
> trxinfo -a
>
> ioctl TRXDRV_QUERY_AIO ret 0
> TRXDRV-AIO STATS
> aio_submitted-3 aio_completed-25605732 aio_error-0
> disk_aio_submitted-25605891 disk_aio_completed-25605732
> sync_active [ x x x x x x x x ]
> async_active [ x x x x x x x x ]
> cb_active-0 aio_sequence-0
> bh_count-65536 bh_inuse-31440 bh_max_inuse-54593 bh_waiters-0
> hash_hits-0 hash_misses-0 hash_fill-3 hash_total-3
> probe_avg-0 probe_max-1
> total_read_req-0 total_write_req-25605888 total_fill_req-0
> total_complete-25744271
> req_sec-0 seconds-12927
>
> This test is posting a max of 54,000 4K buffer heads to a single 3Ware
> adapter.
>
>
> Jeff
>
>
>
>
> On Fri, Mar 01, 2002 at 09:39:08PM -0700, Jeff V. Merkey wrote:
> >
> >
> > We are going to sleep a lot in __get_request_wait(). This
> > means the write queue has no free request blocks. We are mostly writing
> > to the adapter in this test case, and the data we are writing
> > is already in order when it's posted.
> >
> > We are also posting via submit_bh() so you should trace
> > that path. I am seeing 22,000+ buffer heads posted concurrently
> > on each 3Ware card of 4K each with this application
> > on the patch for 2.4.19-pre2. I will post the actual data
> > for you. Stand by.
> >
> > These 3Ware cards are incredible.
> >
> > Jeff
> >
> >
> > On Fri, Mar 01, 2002 at 04:51:04PM -0800, Mike Anderson wrote:
> > > Jeff V. Merkey [jmerkey@vger.timpanogas.org] wrote:
> > > >
> > > > ..snip..
> > > >
> > > > What is really needed here is to allow queue_nr_requests to be
> > > > configurable on a per adapter/device basis for these high end
> > > > raid cards like 3Ware since in a RAID 0 configuration, 8 drives
> > > > are in essence a terabyte (1.3 terrabytes in our configuration)
> > > > and each adapter is showing up as a 1.3 TB device. 64/128
> > > > requests are simply not enough to get the full spectrum of
> > > > performance atainable with these cards.
> > > >
> > > Not having direct experience on this card it appears that increasing the
> > > queue_nr_requests number will not allow you to have more ios in flight.
> > >
> > > Unless I am reading the driver wrong you will be limited to
> > > TW_MAX_CMDS_PER_LUN (15). This value is used by scsi_build_commandblocks
> > > to allocate scsi commands for your scsi_device. This driver does not provide
> > > a select_queue_depths function which allows for increase to the default
> > > template value.
> > >
> > > Could it be that the experimentation of increasing this number has
> > > allowed for better merging.
> > >
> > > -Mike
> > > --
> > > Michael Anderson
> > > andmike@us.ibm.com
> > >
> > > -
> > > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at http://vger.kernel.org/majordomo-info.html
> > > Please read the FAQ at http://www.tux.org/lkml/
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
-- Michael Anderson andmike@us.ibm.com- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
This archive was generated by hypermail 2b29 : Thu Mar 07 2002 - 21:00:31 EST