Re: ATA 4 KiB sector issues.

[Greg Freemyer]

cc'ing Martin Petersen since I believe he is one of the most
knowledgeable kernel hackers on this topic and has been working the
issue for the last year.

On Sun, Mar 7, 2010 at 10:48 PM, Tejun Heo <tj@xxxxxxxxxx> wrote:
> Hello, guys.
>
> It looks like transition to ATA 4k drives will be quite painful and we
> aren't really ready although these drives are already selling widely.
> I've written up a summary document on the issue to clarify stuff as
> it's getting more and more confusing and develop some consensus.  It's
> also on the linux ata wiki.
>
>  http://ata.wiki.kernel.org/index.php/ATA_4_KiB_sector_issues
>
> I've cc'd people whom I can think of off the top of my head but I
> surely have missed some people who would have been interested.  Please
> feel free to add cc's or forward the message to other MLs.
> Especially, I don't know much about partitioners so the details there
> are pretty shallow and could be plain wrong.  It would be great if
> someone who knows more about this stuff can chime in.
>
> Thanks.
>
> === Document follows ===
>
> ATA 4 KiB sector issues
>
> Background
> ==========
>
> Up until recently, all ATA hard drives have been organized in 512 byte
> sectors.  For example, my 500 GB or 477 GiB hard drive is organized of
> 976773168 512 byte sectors numbered from 0 to 976773167.  This is how
> a drive communicates with the driver.  When the operating system wants
> to read 32 KiB of data at 1 MiB position, the driver asks the drive to
> read 64 sectors from LBA (Logical block address, sector number) 2048.
>
> Because each sector should be addressable, readable and writable
> individually, the physical medium also is organized in the same sized
> sectors.  In addition to the area to store the actual data, each
> sector requires extra space for book keeping - inter-sector space to
> enable locating and addressing each sector and ECC data to detect and
> correct inevitable raw data errors.
>
> As the densities and capacities of hard drives keep growing, stronger
> ECC becomes necessary to guarantee acceptable level of data integrity
> increasing the space overhead.  In addition, in most applications,
> hard drives are now accessed in units of at least 8 sectors or 4096
> bytes and maintaining 512 byte granularity has become somewhat
> meaningless.
>
> This reached a point where enlarging the sector size to 4096 bytes
> would yield measurably more usable space given the same raw data
> storage size and hard drive manufacturers are transitioning to 4 KiB
> sectors.
>
> Anandtech has a good article which illustrates the background and
> issues with pretty diagrams[1].
>
>
> Physical vs. Logical
> ====================
>
> Because the 512 byte sector size has been around for a very long time
> and upto ATA/ATAPI-7 the sector size was fixed at 512 bytes, the
> sector size assumption is scattered across all the layers -
> controllers or bridge chips snooping commands, BIOSs, boot codes,
> drivers, partitioners and system utilities, which makes it very
> difficult to change the sector size from 512 byte without breaking
> backward compatibility massively.
>
> As a workaround, the concept of logical sector size was introduced.
> The physical medium is organized in 4 KiB sectors but the firmware on
> the drive will present it as if the drive is composed of 512 byte
> sectors thus making the drive behave as before, so if the driver asks
> the hard drive to read 64 sectors from LBA 2048, the firmware will
> translate it and read 8 4 KiB sectors from hardware sector 256.  As a
> result, the hard drive now has two sector sizes - the physical one
> which the physical media is actually organized in, and the logical one
> which the firmware presents to the outside world.
>
> A straight forward example mapping between physical sector and LBA
> would be
>
>  LBA = 8 * phys_sect
>
>
> Alignment problem on 4 KiB physical / 512 logical drives
> =======================================================
>
> This workaround keeps older hardware and software working while
> allowing the drive to use larger sector size internally.  However, the
> discrepancy between physical and logical sector sizes creates an
> alignment issue.  For example, if the driver wants to read 7 sectors
> from LBA 2047, the firmware has to read hardware sector 255 and 256
> and trim leading 7*512 bytes and tailing 512 bytes.
>
> For reads, this isn't an issue as drives read in larger chunks anyway
> but for writes, the drive has to do read-modify-write to achieve the
> requested action.  It has to first read hardware sector 255 and 256,
> update requested parts and then write back those sectors which can
> cause significant performance degradation[2].
>
> The problem is aggravated by the way DOS partitions[3] have been laid
> out traditionally.  For reasons dating back more than two decades,
> they are laid out considering something called disk geometry which
> nowadays are arbitrary values with a number of restrictions for
> backward compatibility accumulated over the years.  The end result is
> that until recently (most Linux variants and upto Windows XP) the
> first partition ends up on sector 63 and later ones on cylinder
> boundaries where each cylinder usually is composed of 255 * 63
> sectors.
>
> Most modern filesystems generate 4 KiB aligned accesses from the
> partition it is in.  If a drive maps 4 KiB physical sectors to 512
> byte logical sectors from LBA0, the filesystem in the first partition
> will always be misaligned and filesystems in later partitions are
> likely to be misaligned too.
>
>
> Solving the alignment problem on 4 KiB physical / 512 logical drives
> ====================================================================
>
> There are multiple ways which attempt to solve the problem.
>
> S-1. Yet another workaround from the firmware - offset-by-one.
>
>  Yet another workaround which can be done by the firmware is to
>  offset physical to logical mapping by one logical sector such that
>  LBA 63 ends up on physical sector boundary, which aligns the first
>  partition to physical sectors without requiring any software update.
>  The example mapping between phys_sector and LBA becomes
>
>    LBA = 8 * phys_sect - 1
>
>  The leading 512 bytes from phys_sect 0 is not used and LBA 0 starts
>  from after that point.  phys_sect 1 maps to LBA 7 and phys_sect 8 to
>  63, making LBA 63 aligned on hardware sector.
>
>  Although this aligns only the first partition, for many use cases,
>  especially the ones involving older software, this workaround was
>  deemed useful and some recent drives with 4 KiB physical sectors are
>  equipped with a dip switch to turn on or off offset-by-one mapping.
>
> S-2. The proper solution.
>
>  Correct alignments for all partitions can't be achieved by the
>  firmware alone.  The system utilities should be informed about the
>  alignment requirements and align partitions accordingly.
>
>  The above firmware workaround complicates the situation because the
>  two different configurations require different offsets to achieve
>  the correct alignments.  ATA/ATAPI-8 specifies a way for a drive to
>  export the physical and logical sector sizes and the LBA offset
>  which is aligned to the physical sectors.
>
>  In Linux, these parameters are exported via the following sysfs
>  nodes.
>
>    physical sector size        : /sys/block/sdX/queue/physical_block_size
>    logical sector size         : /sys/block/sdX/queue/logical_block_size
>    alignment offset            : /sys/block/sdX/alignment_offset
>
>  Let the physical sector size be PSS, logical sector size LSS and
>  alignment offset AOFF.  The system software should place partitions
>  such that the starting LBAs of all partitions are aligned on
>
>    (n * PSS + AOFF) / LSS
>
>  For 4 KiB physical sector offset-by-one drives, PSS is 4096, LSS 512
>  and AOFF 3584 and with n of 7 the above becomes,
>
>    (7 * 4096 + 3584) / 512 == 63
>
>  making sector 63 an aligned LBA where the first partition can be
>  put, but without the offset-by-one mapping, AOFF is zero and LBA 63
>  is not aligned.
>
>  With the above new alignment requirement in place, it becomes
>  difficult to honor the legacy one - first partition on sector 63 and
>  all other partitions on cylinder boundary (255 * 63 sectors) - as
>  the two alignment requirements contradict each other.  This might be
>  worked around by adjusting how LBA and CHS addresses are mapped but
>  the disk geometry parameters are hard coded everywhere and there is
>  no reliable way to communicate custom geometry parameters.
>
>
> Complications
> =============
>
> Unfortunately, there are complications.
>
> C-1. The standard is not and won't be followed as-is.
>
>  Some of the existing BIOSs and/or drivers can't cope with drives
>  which report 4 KiB physical sector size.  To work around this, some
>  drive models lie that its physical sector size is 512 bytes when the
>  actual configuration is 4 KiB without offsetting.
>
>  This nullifies the provisions for alignment in the ATA standard but
>  results in the correct alignment for Windows Vista and 7.  OS
>  behaviors will be described further later.
>
>  For these drives, which are likely to continue to be shipped for the
>  foreseeable future, traditional LBA 63 and cylinder based aligning
>  results in misalignment.
>
> C-2. Windows XP depends on the traditional partition layout.
>
>  Windows XP makes use of the CHS start/end addresses in the partition
>  table and gets confused if partitions are not laid out
>  traditionally.  This means that XP can't be installed into a
>  partition prepared by later versions of Windows[4].  This isn't a
>  big problem for Windows because in most cases the later version is
>  replacing the older one, not the other way around.
>
>  Unfortunately, the situation is more complex for Linux because Linux
>  is often co-installed with various versions of Windows and XP is
>  still quite popular.  This means that when a Linux partitioner is
>  used to prepare a partition which may be used by Windows, the
>  partitioner might have to consider which version of Windows is going
>  to be used and whether to align the partitions for the correct
>  alignment or compatibility with older versions of Windows.
>
> C-3. The 2 TiB barrier and the possibility for 4 KiB logical sector size.
>
>  The DOS partition format uses 32 bit for the starting LBA and the
>  number of sectors and, reportedly, 32 bit Windows XP shares the
>  limitation.  With 32 bit addressing and 512 byte logical sector
>  size, the maximum addressable sector + 1 is at
>
>    2^32 * 2^9 == 2^41 == 2 TiB
>
>  The DOS partition format allows a partition to reach beyond 2 TiB as
>  long as the starting LBA is under 2 TiB; however, both Windows XP
>  and and the Linux kernel (at least upto v2.6.33) refuse such
>  partition configurations.
>
>  With the right combination of host controller, BIOS and driver, this
>  barrier can be overcome by enlarging the logical sector size to 4
>  KiB, which will push the barrier out to 16 TiB.  On the right
>  configuration, Windows XP is reportedly able to address beyond the 2
>  TiB barrier with a DOS partition and 4 KiB logical sector size.
>  Linux kernel upto v2.6.33 doesn't work under such configurations but
>  a patch to make it work is pending[5].
>
>  This might also be beneficial for operating systems which don't
>  suffer from this limitation.  A different partition format - GPT[6]
>  - should be used beyond 2^32 sectors, which could harm compatibility
>  with older BIOSs or other operating systems which don't recognize
>  the new format.
>
>  As mentioned previously, 512 byte sector assumption has been there
>  for a very long time and changing it is likely to cause various
>  compatibility problems at many different layers from hardware up to
>  the system utilities.
>
>
> Windows
> =======
>
> As hard drive vendors aim for performance and compatibility in modern
> Windows environments, it is worthwhile to investigate how Windows
> partitions with different alignment requirements.  Up until Windows
> XP, it followed the traditional layout - the first partition on LBA 63
> and the others on cylinder boundaries where a cylinder is defined as
> 255 tracks with 63 sectors each.
>
> Windows Vista and 7 align partitions differently.  As the two behave
> similarly, only 7's behavior is shown here.  These partition tables
> are created by Windows 7 RC installer on blank disks.
>
> W-1. 512 byte physical and logical sector drive.
>
>  ST FIRST  T  LAST   LBA      NBLKS
>  80 202100 07 df130c 00080000 00200300
>  00 df140c 07 feffff 00280300 00689e12
>  00 000000 00 000000 00000000 00000000
>  00 000000 00 000000 00000000 00000000
>
>  Part0:        FIRST   C    0  H   32  S   33  : 2048          (63 sec/trk)
>                LAST    C   12  H  223  S   19  : 206847        (255 heads/cyl)
>                LBA     2048 + 204800 = 206848
>
>  Part1:        FIRST   C   12  H  223  S   20  : 206848
>                LAST    C 1023  H  254  S   63  : E
>                LBA     206848 + 312371200 = 312578048
>
>  Both aligned at (2048 * n).  Part 1 not aligned to cylinder.
>
> W-2. 4 KiB physical and 512 byte logical sector drive without offset-by-one.
>
>  ST FIRST  T  LAST   LBA      NBLKS
>  80 202100 07 df130c 00080000 00200300
>  00 df140c 07 feffff 00280300 00b83f25
>  00 000000 00 000000 00000000 00000000
>  00 000000 00 000000 00000000 00000000
>
>  Part0:        FIRST   C    0  H   32  S   33  : 2048          (63 sec/trk)
>                LAST    C   12  H  223  S   19  : 206847        (255 heads/cyl)
>                LBA     2048 + 204800 = 206848
>
>  Part1:        FIRST   C   12  H  223  S   20  : 206848
>                LAST    C 1023  H  254  S   63  : E
>                LBA     206848 + 624932864 = 625139712
>
>  Both aligned at (2048 * n).  Part 1 not aligned to cylinder.
>
> W-3. 4 KiB physical and 512 byte logical sector drive with offset-by-one.
>
>  ST FIRST  T  LAST   LBA      NBLKS
>  80 202800 07 df130c 07080000 f91f0300
>  00 df1b0c 07 feffff 07280300 f9376d74
>  00 000000 00 000000 00000000 00000000
>  00 000000 00 000000 00000000 00000000
>
>  Part0:        FIRST   C    0  H   32  S   40  : 2055          (63 sec/trk)
>                LAST    C   12  H  223  S   19  : 206847        (255 heads/cyl)
>                LBA     2055 + 204793 = 206848
>
>  Part1:        FIRST   C   12  H  223  S   27  : 206855
>                LAST    C 1023  H  254  S   63  : E
>                LBA     206855 + 1953314809 = 1953521664
>
>  Both aligned at (2048 * n + 7).  Part 1 not aligned to cylinder.
>
> The partitioner seems to be using 1M as the basic alignment unit and
> offsetting from there if explicitly requested by the drive and there
> is no difference between handling of 512 byte and 4 KiB drives, which
> explains why C-1 works for hard drive vendors.
>
> In all cases, the partitioner ignores both the first partition on LBA
> 63 and the others on cylinder boundary requirements while still using
> the same 255*63 cylinder size.  Also, note that in W-3, both part 0
> and 1 end up with odd number of sectors.  It seems that they simply
> decided to completely break away from the traditional layout, which is
> understandable given that there really isn't one good solution which
> can cover all the cases and that the default larger alignment benefits
> earlier SSDs.
>
> Windows Vista basically shows the same behavior.  Vista was tested by
> creating two partitions using the management tool.  Test data is
> available at [7].
>
>  *-alignment_offset    : alignment_offset reported by Linux kernel
>  *-fdisk               : fdisk -l output
>  *-fdisk-u             : fdisk -lu output
>  *-hdparm              : hdparm -I output
>  *-mbr                 : dump of mbr
>  *-part                : decoded partition table from mbr
>
> Please note that hdparm is misreporting the alignment offset.  It
> should be reporting 512 instead of 256 for offset-by-one drives.
>
>
> So, what now for Linux?
> =======================
>
> The situation is not easy.  Considering all the factors, the only
> workable solution looks like doing what Windows is doing.  Hard drive
> and SSD vendors are focusing on compatibility and performance on
> recent Windows releases and are happy to do things which break the
> standard defined mechanism as shown by C-1, so parting away from what
> Windows does would be unnecessarily painful.
>
> Unfortunately, while Windows can assume that newer releases won't
> share the hard drive with older releases including Windows XP, Linux
> distros can't do that.  There will be many installations where a
> modern Linux distros share a hard drive with older releases of
> Windows.  At this point, I can't see a silver bullet solution.
>
> Partitioners maybe should only align partitions which will be used by
> Linux and default to the traditional layout for others while allowing
> explicit override.  I think Windows XP wouldn't have problem with
> differently aligned partitions as long as it doesn't actually use them
> but haven't tested it.
>
> Reportedly, commonly used partitioners aren't ready to handle drives
> larger than 2 TiB in any configuration and alignment isn't done
> properly for drives with 4 KiB physical sectors.  4 KiB logical sector
> support is broken in both the kernel and partitioners.  (need more
> details and probably a whole section on partitioner behaviors)
>
> Unfortunately, the transition to 4 KiB sector size, physical only or
> logical too, is looking fairly ugly.  Hopefully, a reasonable solution
> can be reached in not too distant future but even with all the
> software side updated, it looks like it's gonna cause significant
> amount of confusion and frustration.
>
>
> [1] http://www.anandtech.com/storage/showdoc.aspx?i=3691
> [2] http://www.osnews.com/story/22872/Linux_Not_Fully_Prepared_for_4096-Byte_Sector_Hard_Drives
> [3] http://en.wikipedia.org/wiki/Master_boot_record
> [4] http://support.microsoft.com/kb/931760
> [5] http://thread.gmane.org/gmane.linux.kernel/953981
> [6] http://en.wikipedia.org/wiki/GUID_Partition_Table
> [7] http://userweb.kernel.org/~tj/partalign/
>
> * Mar 04 2009
>        Initial draft, Tejun Heo <tj@xxxxxxxxxx>
> * Mar 08 2009
>        Updated according to comments from Daniel Taylor
>        <Daniel.Taylor@xxxxxxx>.  Other minor updates.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ide" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
Greg Freemyer
Head of EDD Tape Extraction and Processing team
Litigation Triage Solutions Specialist
http://www.linkedin.com/in/gregfreemyer
Preservation and Forensic processing of Exchange Repositories White Paper -
<http://www.norcrossgroup.com/forms/whitepapers/tng_whitepaper_fpe.html>

The Norcross Group
The Intersection of Evidence & Technology
http://www.norcrossgroup.com
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/