The kernel and character set encodings

These encodings are mostly useless in a true multi-user system. Why ? Because they are all incompatible. So there is no way for a user that uses encoding A to read stuff (including filenames) made by another user using encoding B. And this is true even for close stuff (KOI8-U and KOI8-R for example). Not to speak of the poor users that may want to quote another langage (French + Russian, Welsh + Greek etc).

The only thing all those encodings are compatible with is english, which restricts second language to english and english only.

One could argue userspace would have just to use Greek encoding for Greek filenames, Russian for Russian ones and so on. But the crux of the problem is userspace have no way to request or guess what encoding was used to write a filename, since the kernel does not enforce any particular encoding nor provides encoding info to userspace.

One additionnal problem is some byte strings can result in invalid UTF-8 and cause applications to barf if they try to decode them.

> These strings result in exactly the same visual appearance on screen, yet they can't be compared with a byte comparison.

You do not need even Unicode normalization for that. In most fonts the following two lines would have exactly the same visual presentation (you have to view the page with UTF-8 encoding as LWN does not allow to enter РОТ in HTML comments due to bugs in recognition of &code; escapes):
POT
РОТ
yet the first uses pure ASCII and the second uses only Cyrillic characters and means mouth in Russian.

IMHO such examples supports the notion that kernel should not impose any policy on file names encoding as in practice there are always more then one way to encode the same visual presentation and UTF-8 with Unicode does not help here.

  If Unicode is a horde of zombies with flaming dung sticks, 
  the hideous intricacies of JIS, Chinese Big-5, Chinese 
  Traditional, KOI-8, et cetera are at least an army of ogres 
  with salt and flensing knives.        -- Eric S. Raymond, python-dev

Unfortunately there are hordes of people out there who think dumb ideas
like case-insensitivity should be applied at low levels to stuff that
really is binary data. This kind of thinking is what causes complexity,
and complexity causes bugs and security holes.

Any program that takes a string it thinks is UTF-8 and does
<i>ANYTHING</i> other than pass the exact bytes unchanged to another
interface that wants UTF-8 is by definition broken. This simple rule will
completely eliminate all ambiguity about UTF-8.

I'm not sure Unicode needs to live in the kernel as long as there is good library support, but it's better for library or kernel maintainers to solve these problems once rather than have different buggy implementations in every application.

The specific IIS issues were related to Microsoft's non-standard %uNNNN encoding of 16-bit UCS-2 (Unicode) characters, so I don't think this is a reason to abandon Unicode.

Well, that should get their attention. :-) The exact wording was ``Unicode is just too complex to ever be secure.''

In his July 2000 Crypto-gram article on Unicode, Schneier points up the failures we've had dealing with ASCII control characters, escape sequences, different semantics at different levels of the application (think writing a bash command to grep for a particular grep regular expression), and concludes that with Unicode it's not merely hard, it's effectively impossible.

I don't know enough about Unicode to argue the details, but it certainly made me sit up and take notice.

"../" in a UTF-8 filename means the *BYTES* for '.', '.', and '/' appear
next to each other. It is entirely irrelevant if the UTF-8 string is
legal or if it contains a byte sequence that some broken software by
Microsoft will turn into a slash.

I don't know how many times this has to be stated. But if your program is
looking at a UTF-8 string and is doing anything other than drawing the
characters on the screen, YOU DO NOT NEED TO DECODE IT! Just look at the
bytes!

[Strawman proposal---please point me toward discussions where it's all been hashed out, shot down, &c. Or, just flame direct.]

How about changing filename semantics (and, of course, every filesystem known to Linux): make filenames a three-element struct: a fixed-length specification of the name's character-set encoding, a fixed-length count of the bytes in the name, and a variable-length string holding said name:

    struct filename {
        enum encoding enc;
        int cb;
        byte *rgb;
    };

Now, the kernel doesn't give a fig what the encoding is, or what it might mean---it's all bytes, with no chance (hah!) of filename buffer overflows and their attendant dangers to root. The libraries just use the struct for calls to fopen(), remove(), rename(), & friends, with the caller allowed to specify that

• an exact match (on all elements of the struct) is needed for equality comparisons,
• a bytewise match on the byte *s, regardless of the encoding, is sufficient, or
• its own comparator function (supplied) be run on pairs of the structs.

The kernel code is encoding-agnostic, and the rest of the work (emphatically including sorting) is in userland.