Re: [RFC] udf: 2.01 interoperability issues with Windows 10

From: Steve Magnani
Date: Tue Jul 09 2019 - 16:14:51 EST

On 7/9/19 1:56 PM, Pali RohÃr wrote:
Steve, can you describe what you mean by Advanced Format 4K sector size?

It is hard disk with 512 bytes logical sector size and 4096 bytes
physical sector size? Or do you mean hard disk which has both logical
and physical sector size equal to 4096 bytes?

Sorry, forgot that the term Advanced Format introduces ambiguity.
As far as the OSes are concerned the drive is 4Kn.

On Tuesday 09 July 2019 13:27:58 Steve Magnani wrote:


Recently I have been exploring Advanced Format (4K sector size)
and high capacity aspects of UDF 2.01 support in Linux and
Windows 10. I thought it might be helpful to summarize my findings.

The good news is that I did not see any bugs in the Linux
ecosystem (kernel driver + mkudffs).

The not-so-good news is that Windows has some issues that affect
interoperability. One of my goals in posting this is to open a
discussion on whether changes should be made in the Linux UDF
ecosystem to accommodate these quirks.

My test setup includes the following software components:

* mkudffs 1.3 and 2.0
Can you do tests also with last version of mkudffs 2.1?

A very quick smoke test of a 16-ish TiB 4Kn partition seemed OK.

* kernel 4.15.0-43 and 4.15.0-52
* Windows 10 1803 17134.829
* chkdsk 10.0.17134.1
* udfs.sys 10.0.17134.648

ISSUE 1: Inability of the Linux UDF driver to mount 4K-sector
media formatted by Windows.
Can you check if udfinfo (from udftools) can recognize such disk?

It cannot:

$ ./udfinfo /dev/sdb1
udfinfo: Error: UDF Volume Recognition Sequence found but not Anchor Volume Descriptor Pointer, maybe wrong --blocksize?
udfinfo: Error: Cannot process device '/dev/sdb1' as UDF disk

$ ./udfinfo --blocksize=4096 /dev/sdb1
udfinfo: Error: UDF Volume Recognition Sequence not found
udfinfo: Error: Cannot process device '/dev/sdb1' as UDF disk

$ ./udfinfo
udfinfo from udftools 2.1

And can blkid (from util-linux) recognize such disk as UDF with reading
all properties?


$ blkid --info /dev/sdb1
/dev/sdb1: MINIMUM_IO_SIZE="4096"

$ blkid --probe /dev/sdb1
/dev/sdb1: VOLUME_ID="UDF Volume"
VOLUME_SET_ID="0E131B3B UDF Volume Set"

$ blkid --version
blkid from util-linux 2.31.1 (libblkid 2.31.1, 19-Dec-2017)

Can grub2 recognize such disks?

I'm not sure what you're asking here. The physical interface to this drive is USB,
and it's not designed for general-purpose storage (or booting). That said, if you
have some grub2 commands you want me to run against this drive/partition let me know.

Also can you check if libparted from git master branch can recognize
such disk? In following commit I added support for recognizing UDF
filesystem in libparted, it is only in git master branch, not released:

Build failed:
In file included from pt-tools.c:114:0:
pt-tools.c: In function 'pt_limit_lookup':
pt-limit.gperf:78:1: error: function might be candidate for attribute 'pure' [-Werror=suggest-attribute=pure]

If you send me some other SHA to try I can attempt a rebuild.

ISSUE 2: Inability of Windows chkdsk to analyze 4K-sector media
formatted by mkudffs.
This is really bad :-(

It would be possible to work around this by tweaking mkudffs to
insert dummy BOOT2 components in between the BEA/NSR/TEA:

0000: 00 42 45 41 30 31 01 00 00 00 00 00 00 00 00 00 .BEA01..........
0800: 00 42 4f 4f 54 32 01 00 00 00 00 00 00 00 00 00 .BOOT2..........
1000: 00 4e 53 52 30 33 01 00 00 00 00 00 00 00 00 00 .NSR03..........
1800: 00 42 4f 4f 54 32 01 00 00 00 00 00 00 00 00 00 .BOOT2..........
2000: 00 54 45 41 30 31 01 00 00 00 00 00 00 00 00 00 .TEA01..........

That would introduce a slight ECMA-167 nonconformity, but Linux
does not object and Windows actually performs better. I would
have to tweak udffsck though since I believe this could confuse
its automatic detection of medium block size.
I would like to avoid this hack. If chkdsk is unable to detect such
filesystem, it is really a good idea to let it do try doing filesystem
checks and recovery? You are saying that udfs.sys can recognize such
disk and mount it. I think this should be enough.

Fair enough, but it's also reasonable to assume the bugginess is
limited to the VRS corner case. AFAIK that's the only place in ECMA-167
where there is a difference in layout specific to 4K sectors.
With the BOOT2 band-aid chkdsk is able to analyze filesystems on 4Kn media.

I use chkdsk frequently to double-check UDF generation firmware
I am writing, and also udffsck work-in-progress.

Steven J. Magnani "I claim this network for MARS! Earthling, return my space modulator!"

#include <standard.disclaimer>