Re: [PATCH 00/26] AIO performance improvements/cleanups, v2

[Kent Overstreet]

On Fri, Dec 14, 2012 at 08:35:53AM +0100, Jens Axboe wrote:
> On 2012-12-14 03:26, Jack Wang wrote:
> > 2012/12/14 Jens Axboe <jaxboe@xxxxxxxxxxxx>:
> >> On Mon, Dec 03 2012, Kent Overstreet wrote:
> >>> Last posting: http://thread.gmane.org/gmane.linux.kernel.aio.general/3169
> >>>
> >>> Changes since the last posting should all be noted in the individual
> >>> patch descriptions.
> >>>
> >>>  * Zach pointed out the aio_read_evt() patch was calling functions that
> >>>    could sleep in TASK_INTERRUPTIBLE state, that patch is rewritten.
> >>>  * Ben pointed out some synchronize_rcu() usage was problematic,
> >>>    converted it to call_rcu()
> >>>  * The flush_dcache_page() patch is new
> >>>  * Changed the "use cancellation list lazily" patch so as to remove
> >>>    ki_flags from struct kiocb.
> >>
> >> Kent, I ran a few tests, and the below patches still don't seem as fast
> >> as the approach I took. To keep it fair, I used your aio branch and
> >> applied by dio speedups too. As a sanity check, I ran with your branch
> >> alone as well. The quick results below - kaio is kent-aio, just your
> >> branch. kaio-dio is with the direct IO speedups too. jaio is my branch,
> >> which already has the dio changes too.
> >>
> >> Devices         Branch          IOPS
> >> 1               kaio            ~915K
> >> 1               kaio-dio        ~930K
> >> 1               jaio           ~1220K
> >> 6               kaio           ~3050K
> >> 6               kaio-dio       ~3080K
> >> 6               jaio            3500K
> >>
> >> The box runs out of CPU driving power, which is why it doesn't scale
> >> linearly, otherwise I know that jaio at least does. It's basically
> >> completion limited for the 6 device test at the moment.
> >>
> >> I'll run some profiling tomorrow morning and get you some better
> >> results. Just thought I'd share these at least.
> >>
> >> --
> >> Jens Axboe
> >>
> > 
> > A really good performance, woo.
> > 
> > I think the device tested is really fast PCIe SSD builded by fusionio
> > with fusionio in house block driver?
> 
> It is pci-e flash storage, but it is not fusion-io.
> 
> > any compare number with current mainline?
> 
> Sure, I should have included that. Here's the table again, this time
> with mainline as well.
> 
> Devices         Branch          IOPS
> 1               mainline        ~870K
> 1               kaio            ~915K
> 1               kaio-dio        ~930K
> 1               jaio           ~1220K
> 6               kaio           ~3050K
> 6               kaio-dio       ~3080K
> 6               jaio           ~3500K
> 6               mainline       ~2850K

Cool, thanks for the numbers!

I suspect the difference is due to contention on the ringbuffer,
completion side. You didn't enable my batched completion stuff, did you?

I suspect the numbers would look quite a bit different with that,
based on my own profiling. If the driver for the device you're testing
on is open source, I'd be happy to do the conversion (it's a 5 minute
job).

Also, I don't think our approaches really conflict - it's been awhile
since I looked at your patch but you're getting rid of the aio
ringbuffer and using a linked list instead, right? My batched completion
stuff should still benefit that case.

Though - hrm, I'd have expected getting rid of the cancellation linked
list to make a bigger difference and both our patchsets do that.

What device are you testing on, and what's your fio script? I may just
have to buy some hardware so I can test this myself.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/