Re: OT: character encodings
From: Tilman Schmidt
Date: Sun Jan 07 2007 - 14:25:51 EST
Am 07.01.2007 18:06 schrieb Russell King:
>
> $ git log | head -n 1000 | tail -n 200 > o
> $ file -i o
> o: text/plain; charset=us-ascii
> $ git log | head -n 1000 | tail -n 300 > o
> $ file -i o
> o: text/plain; charset=us-ascii
> $ git log | head -n 1000 | tail -n 400 > o
> $ file -i o
> o: text/plain; charset=utf-8
>
> (and you know what charset the file is thought to have with all 1000
> lines in it.)
What the "file" command thinks is hardly relevant here. "file" just
attempts to guess what the contents of a file might be, by applying
a simple set of heuristics. Your results only highlight the actual
problem: "git" is apparently unable to handle character sets properly
and instead produces a mix of encodings as output.
> All on a system with LANG set to en_GB (iow ISO-8859-1).
For software with proper multilingual support, that should have been
enough to make sure that all its output would be in iso-8859-1, too.
Obviously "git" doesn't fall into that category.
>> Yes. When you stored it on disk, the character set information was lost.
>
> The same thing actually happens when I look at it via:
>
> $ git log | head -n 1000 | less
The loss has happened long before you run that command, when the
data was committed into "git".
> So, I think you'll find that the contents of git _is_ an ad-hoc collection
> of character sets which people happen to have in use on their machines.
Exactly.
>> A mixed charset environment was _already_ a pain in the butt, because
>> almost nobody got labelling right. It's wrong to blame that on UTF-8.
>
> I'm not talking about a mixed charset environment. I'm talking about
> non-UTF-8 single charset environments being broken by programs which
> universally think the universe is UTF-8 only.
The problem is not programs thinking the universe is UTF-8 only; it's
people mixing different charsets, in conjunction with programs not
caring about charsets at all.
Specifically, your non-UTF-8 single charset environment was not broken
by git thinking everything was UTF-8, but to the contrary by some data
in the git repository actually being UTF-8 and git *not* thinking that.
And that problem is, I repeat, much older than UTF-8.
HTH
Tilman
--
Tilman Schmidt E-Mail: tilman@xxxxxxx
Bonn, Germany
Diese Nachricht besteht zu 100% aus wiederverwerteten Bits.
Ungeöffnet mindestens haltbar bis: (siehe Rückseite)
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/