Re: [Lsf-pc] [LSF/MM TOPIC] really large storage sectors - goingbeyond 4096 bytes

From: Ric Wheeler
Date: Wed Jan 22 2014 - 13:40:43 EST

On 01/22/2014 01:35 PM, James Bottomley wrote:
On Wed, 2014-01-22 at 13:17 -0500, Ric Wheeler wrote:
On 01/22/2014 01:13 PM, James Bottomley wrote:
On Wed, 2014-01-22 at 18:02 +0000, Chris Mason wrote:
On Wed, 2014-01-22 at 09:21 -0800, James Bottomley wrote:
On Wed, 2014-01-22 at 17:02 +0000, Chris Mason wrote:
[ I like big sectors and I cannot lie ]
I think I might be sceptical, but I don't think that's showing in my
concerns ...

I really think that if we want to make progress on this one, we need
code and someone that owns it. Nick's work was impressive, but it was
mostly there for getting rid of buffer heads. If we have a device that
needs it and someone working to enable that device, we'll go forward
much faster.
Do we even need to do that (eliminate buffer heads)? We cope with 4k
sector only devices just fine today because the bh mechanisms now
operate on top of the page cache and can do the RMW necessary to update
a bh in the page cache itself which allows us to do only 4k chunked
writes, so we could keep the bh system and just alter the granularity of
the page cache.

We're likely to have people mixing 4K drives and <fill in some other
size here> on the same box. We could just go with the biggest size and
use the existing bh code for the sub-pagesized blocks, but I really
hesitate to change VM fundamentals for this.
If the page cache had a variable granularity per device, that would cope
with this. It's the variable granularity that's the VM problem.

From a pure code point of view, it may be less work to change it once in
the VM. But from an overall system impact point of view, it's a big
change in how the system behaves just for filesystem metadata.
Agreed, but only if we don't do RMW in the buffer cache ... which may be
a good reason to keep it.

The other question is if the drive does RMW between 4k and whatever its
physical sector size, do we need to do anything to take advantage of
it ... as in what would altering the granularity of the page cache buy
us?
The real benefit is when and how the reads get scheduled. We're able to
do a much better job pipelining the reads, controlling our caches and
reducing write latency by having the reads done up in the OS instead of
the drive.
I agree with all of that, but my question is still can we do this by
propagating alignment and chunk size information (i.e. the physical
sector size) like we do today. If the FS knows the optimal I/O patterns
and tries to follow them, the odd cockup won't impact performance
dramatically. The real question is can the FS make use of this layout
information *without* changing the page cache granularity? Only if you
answer me "no" to this do I think we need to worry about changing page
cache granularity.

Realistically, if you look at what the I/O schedulers output on a
standard (spinning rust) workload, it's mostly large transfers.
Obviously these are misalgned at the ends, but we can fix some of that
in the scheduler. Particularly if the FS helps us with layout. My
instinct tells me that we can fix 99% of this with layout on the FS + io
schedulers ... the remaining 1% goes to the drive as needing to do RMW
in the device, but the net impact to our throughput shouldn't be that
great.

James

I think that the key to having the file system work with larger
sectors is to
create them properly aligned and use the actual, native sector size as
their FS
block size. Which is pretty much back the original challenge.
Only if you think laying out stuff requires block size changes. If a 4k
block filesystem's allocation algorithm tried to allocate on a 16k
boundary for instance, that gets us a lot of the performance without
needing a lot of alteration.

The key here is that we cannot assume that writes happen only during allocation/append mode.

Unless the block size enforces it, we will have non-aligned, small block IO done to allocated regions that won't get coalesced.

It's not even obvious that an ignorant 4k layout is going to be so
bad ... the RMW occurs only at the ends of the transfers, not in the
middle. If we say 16k physical block and average 128k transfers,
probabalistically we misalign on 6 out of 31 sectors (or 19% of the
time). We can make that better by increasing the transfer size (it
comes down to 10% for 256k transfers.

This really depends on the nature of the device. Some devices could produce very erratic performance or even (not today, but some day) reject the IO.


Teaching each and every file system to be aligned at the storage
granularity/minimum IO size when that is larger than the physical
sector size is
harder I think.
But you're making assumptions about needing larger block sizes. I'm
asking what can we do with what we currently have? Increasing the
transfer size is a way of mitigating the problem with no FS support
whatever. Adding alignment to the FS layout algorithm is another. When
you've done both of those, I think you're already at the 99% aligned
case, which is "do we need to bother any more" territory for me.


I would say no, we will eventually need larger file system block sizes.

Tuning and getting 95% (98%?) of the way there with alignment and IO scheduler does help a lot. That is what we do today and it is important when looking for high performance.

However, this is more of a short term work around for a lack of a fundamental ability to do the right sized file system block for a specific class of device. As such, not a crisis that must be solved today, but rather something that I think is definitely worth looking at so we can figure this out over the next year or so.

Ric

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/