NLS: utf8 conversions

From: Alan Stern
Date: Fri Apr 24 2009 - 16:03:09 EST


Although nobody seems to have made a big deal about it, the conversions
between utf8 and utf16 done by fs/nls/nls_base.c are wrong in a couple
of important respects:

They don't handle Unicode code points larger than U+FFFF.

They don't detect invalid values, in particular, surrogate
code points.

The problems stem from the fact the characters at issue can't be
represented by a single 16-bit wchar_t. But that's no excuse for
performing an incorrect conversion to or from utf16.

Are there any definite thoughts on how this should be handled? I don't
see any way for the single-character conversion routines (utf8_mbtowc
and utf8_wctomb) to come to grips with these issues, except perhaps for
returning an error when a character would be invalid or too big to fit
in 16 bits.

The string-oriented routines (utf8_mbstowcs and utf8_wcstombs) could be
adapted to deal with these issues properly.

Any comments or suggestions for other approaches?

Alan Stern

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/