Re: Unification of filesystem encoding options

[Pali RohÃr]

On Tuesday 07 January 2020 15:03:01 Theodore Y. Ts'o wrote:
> On Tue, Jan 07, 2020 at 06:38:42PM +0100, Pali RohÃr wrote:
> > Adding support for case-insensitivity into UTF-8 NLS encoding would mean
> > to create completely new kernel NLS API (which would support variable
> > length encodings) and rewrite all NLS filesystems to use this new API.
> > Also all existing NLS encodings would be needed to port into this new
> > API.
> > 
> > It is really something which have a value? Just because of UTF-8?
> > 
> > For me it looks like better option would be to remove UTF-8 NLS encoding
> > as it is broken. Some filesystems already do not use NLS API for their
> > UTF-8 support (e.g. vfat, udf or newly prepared exfat). And others could
> > be modified/extended/fixed in similar way.
> 
> You didn't mention ext4 and f2fs, which is using the Unicode code in
> fs/unicode for its case-folding and normalization support.

Hi! I have not mentioned because I took only filesystems which use NLS.
And also I forgot that ext4 now has unicode flag which basically put
this filesystem into group where FS "enforce" encoding of on disk
structure.

> Ext4 and
> f2fs only supports utf-8, so using the NLS API would have added no
> value --- and it as you pointed out, the NLS API doesn't support
> variable length encoding anyway.

Theoretically using NLS API could have a value if userspace is
configured to work in e.g. Latin1 encoding and you want to use ext4/f2fs
with unicode flag in this userspace (NLS API in this case could convert
3byte UTF-8 to Latin1). But it is very theoretical and limited use case.

> In contrast the fs/unicode functions
> have support for full Unicode case folding and normalization, and
> currently has the latest Unicode 12.1 tables (released May 2019).

That is great!

But for example even this is not enough for exfat. exfat has stored
upcase table directly in on-disk FS, so ensure that every implementation
of exfat driver would have same rules how to convert character (code
point) to upper case or lower case (case folding). Upcase table is
stored to FS itself when formatting. And in MS decided that for exfat
would not be used any Unicode normalization. So this whole fs/unicode
code is not usable for exfat.

> What I'd suggest is to create a new API, enhancing the functions in
> fs/unicode, to support those file systems that need to deal with
> UTF-16 and UTF-32 for their on-disk directory format, and that we
> assume that for the most part, userspace *will* be using a UTF-8
> encoding for the user<->kernel interface.

I do not see a use-case for such a new API. Kernel has already API
functions:

    int utf8_to_utf32(const u8 *s, int len, unicode_t *pu);
    int utf32_to_utf8(unicode_t u, u8 *s, int maxlen);
    int utf8s_to_utf16s(const u8 *s, int len, enum utf16_endian endian, wchar_t *pwcs, int maxlen);
    int utf16s_to_utf8s(const wchar_t *pwcs, int len, enum utf16_endian endian, u8 *s, int maxlen);

which are basically enough for all mentioned filesystems. Maybe in for
some cases would be useful function utf16 to utf32 (and vice-versa), but
that is all. fs/unicode does not bring a new value or simplification.

Mentioned filesystems are in most cases either case-sensitive (UDF),
having own case-folding (exfat), using own special normalization
incompatible with anything (hfsplus) or do not enforce any normalization
(cifs, vfat, ntfs, isofs+joliet). So result is that simple UTF-8 to
UTF-16LE/BE conversion function is enough and then filesystem module
implements own specific rules (special upcase table, incompatible
normalization).

And I do not thing that it make sense to extending fs/unicode for every
one stupid functionality which those filesystems have and needs to
handle. I see this as a unique filesystem specific code.

> We can keep the existing
> NLS interface and mount options for legacy support, but in my opinion
> it's not worth the effort to try to do anything else.

NLS interface is crucial part of VFAT. Reason is that in VFAT can be
filenames stored either as UTF-16LE or as 8bit in some CP encoding.
Linux kernel stores new non-7-bit-ASCII filenames as UTF-16LE, but it
has to able to read 8-bit filenames which were not stored as UTF-16LE,
but rather as 8bit in CP encoding. And therefore mount option codepage=
which specify it is required needs to be implemented. It says how
vfat.ko should handle on-disk structure, not which encoding is exported
to userspace (those are two different things).

And current vfat implementation uses NLS API for it. Via CONFIG_* is
specified default codepage= mount option (CP473 or what it is -- if you
do not specify one explicitly at mount time). And because FAT is
required part of UEFI, Linux kernel would have to support this stuff
forever (or at least until it support UEFI). I think this cannot be
marked as "legacy". It is pity, but truth.

-- 
Pali RohÃr
pali.rohar@xxxxxxxxx

Attachment: signature.asc
Description: PGP signature