<stephen(a)xemacs.org&gt; writes: I open a file with UTF-8 coding-system. I touch that file outside of XEmacs and then do M-x revert-buffer RET. The non-ASCII characters get mangled. From the looks of it, upon re-reading the file is treated as Latin-1 (i.e. multibyte UTF-8 encodings get turned into the characters represented by their component bytes), and the result is then translated to UTF-8. (The modeline still says "UTF-8", though.) For example, "Anfänger" gets turned into "Anfänger" upon revert. (I hope Gnus hasn't screwed this up on send.) -- Cheers =8-} Mike Friede, Völkerverständigung und überhaupt blabla _______________________________________________ XEmacs-Beta mailing list XEmacs-Beta(a)xemacs.org http://calypso.tux.org/cgi-bin/mailman/listinfo/xemacs-beta

Another one: When I open a new file with an explicit coding system (say, UTF-8), I sometimes get "UTF8" in the modeline, and sometimes "Auto", which then seems to save in Latin-1. -- Cheers =8-} Mike Friede, Völkerverständigung und überhaupt blabla _______________________________________________ XEmacs-Beta mailing list XEmacs-Beta(a)xemacs.org http://calypso.tux.org/cgi-bin/mailman/listinfo/xemacs-beta

<stephen(a)xemacs.org&gt; writes: No, it's totally spurious, AFAICT. "Auto" on one open, "UTF8" the next. No. -- Cheers =8-} Mike Friede, Völkerverständigung und überhaupt blabla _______________________________________________ XEmacs-Beta mailing list XEmacs-Beta(a)xemacs.org http://calypso.tux.org/cgi-bin/mailman/listinfo/xemacs-beta

Ar an tríú lá is fiche de mí Deireadh Fómhair, scríobh Michael Sperber: Umm, it’s not a problem on non-Mule, right? I can’t reproduce your non-deterministic buffer-file-coding-system when doing C-u C-x C-f random-coding-system RET ; can you with xemacs -vanilla? -- Santa Maradona, priez pour moi! _______________________________________________ XEmacs-Beta mailing list XEmacs-Beta(a)xemacs.org http://calypso.tux.org/cgi-bin/mailman/listinfo/xemacs-beta

<stephen(a)xemacs.org&gt; writes: No, unfortunately. -- Cheers =8-} Mike Friede, Völkerverständigung und überhaupt blabla _______________________________________________ XEmacs-Beta mailing list XEmacs-Beta(a)xemacs.org http://calypso.tux.org/cgi-bin/mailman/listinfo/xemacs-beta

Nix writes: Article buffers are constructed, not read from files. I was referring to something read from outside. Gnus basically has the stance that Mule doesn't work so it's going to do it all itself; it doesn't bother to get things Mule-correct most of the time AFAICT. Could you send a build report the next time you build? M-x build-report RET. That way I'll have some idea what exactly your configuration is. If you're seeing ugly warnings, read the docs and set up the build-report Custom group so you can send a build log (ie, the output of make, with boring echoes of commands filtered out). As you surmise that should have nothing to do with it. Is there a reason to have "utf-8-bom" following "iso-8-1"? Basically, if the utf-8 detector doesn't catch it, and the iso-8-1 detector doesn't catch it, then the BOM detector won't either. OTOH, it's possible for there to be a file that looks like iso-8-1 based on character ranges but starts with the BOM (all of the octets in the BOM are valid ISO-8859 characters, at least in all the full-up sets like ISO-8859-1), which probably should be detected as UTF-8 *if* the other UTF-8 constraints are satisfied. I mean, how many text/plain files do you see that start with "" ?? (I suppose in Arabic or Cyrillic it might become something sensible. Do you read any of the non-Latin scripts?) Which is a long-winded way of saying, "if you have an empirical reason for this, it's probably an XEmacs bug!" _______________________________________________ XEmacs-Beta mailing list XEmacs-Beta(a)xemacs.org http://calypso.tux.org/cgi-bin/mailman/listinfo/xemacs-beta

Nix writes: Actually, you can get quite far with display without MULE. IIRC, historically that's why Gnus does things this way. Consider that X11 doesn't know any differences among ISO Latin-1, ISO Latin-9 (hardly surprising to humans), and ISO Arabic (astonishing to most humans). Just change the registry of the font for the default face from iso8859-1 to iso8859-15 to iso8859-6 as you have need for the ancient currency symbol (aka the "horny bullet"), the Euro, or the language of the Koran. It's multilingual input and handling files in different character sets *simultaneously* where MULE (or something like it) becomes essential. Day job etc keeping you busy, eh? :-) Yes, please do. _______________________________________________ XEmacs-Beta mailing list XEmacs-Beta(a)xemacs.org http://calypso.tux.org/cgi-bin/mailman/listinfo/xemacs-beta

Michael> I touch that file outside of XEmacs and then do M-x Michael> revert-buffer RET. Don't know if this is related but I have this situation in a folder: ---------------------------------------------------------------------- /tmp/TEST: totalt 28 -rw-r--r-- 1 matsl users 0 20 okt 23.28 ÄÄÄ -rw-r--r-- 1 matsl users 0 20 okt 23.28 ÖÖÖ ---------------------------------------------------------------------- Now I open the file 'ÄÄÄ' for editing, (modeline says it i ÄÄÄ) insert some chars and save it. This is what I get in dired: ---------------------------------------------------------------------- /tmp/TEST: totalt 32 -rw-r--r-- 1 matsl users 4 23 okt 12.40 ÄÄÄ -rw-r--r-- 1 matsl users 0 20 okt 23.28 ÄÄÄ -rw-r--r-- 1 matsl users 0 20 okt 23.28 ÖÖÖ ---------------------------------------------------------------------- In a shell, outside XEmacs, I get: ---------------------------------------------------------------------- spencer:/tmp/TEST% ls -l totalt 4 -rw-r--r-- 1 matsl users 4 23 okt 12.40 ??? -rw-r--r-- 1 matsl users 0 20 okt 23.28 ÄÄÄ -rw-r--r-- 1 matsl users 0 20 okt 23.28 ÖÖÖ ---------------------------------------------------------------------- The file I saved from XEmacs, and is shown as ÄÄÄ in dired is in my shell shown as ???. Both XEmacs and the shell uses the same locale sv_SE.UTF-8. If I remove the file ??? in the shell, and update dired, XEmacs is smart and gets the names back readable again. Is this a bug or should it be like this? Yours -- %% Mats _______________________________________________ XEmacs-Beta mailing list XEmacs-Beta(a)xemacs.org http://calypso.tux.org/cgi-bin/mailman/listinfo/xemacs-beta

Michael Sperber writes: What are you trying to do? I forget what is done in 21.4, but in 21.5 `file-name-coding-system' controls conversion of file names. This is a Unicode issue, I'm pretty sure. I suspect you're on a Mac with HFS+, and what you see is not (a umlaut, a umlaut, a umlaut) but rather (a, combining umlaut, a, combining umlaut, a, combining umlaut) because the Mac puts file names into NFD or NFKD (canonical decomposed normalization form). XEmacs doesn't know about the various Unicode normalizations, and it will take some effort to incorporate them. Note that it can't possibly be the UTF-8 encoding of the file name because all accented characters have high-bit set representations in UTF-8. _______________________________________________ XEmacs-Beta mailing list XEmacs-Beta(a)xemacs.org http://calypso.tux.org/cgi-bin/mailman/listinfo/xemacs-beta

Mats Lidell <matsl(a)xemacs.org&gt; writes: I can reproduce the situation on a MULE XEmacs in an UTF-8 locale. (It also breaks on a non-MULE XEmacs, but in a different way. Fixing it there is harder.) The reason is that `default-process-coding-system' is 'undecided in the read direction, and UTF-8 fails to get detected. My suggestion is to align the coding system in `insert-directory' with the locale's coding system. Works for me. Maybe the MULE experts could review? I'd appreciate that! 2008-01-13 Michael Sperber <mike(a)xemacs.org&gt; * files.el (insert-directory): Bind `coding-system-for-read' according to the current locale where available. (Previously, the default ended up being undecided, which doesn't work well for UTF-8-based locales, for example.) -- Cheers =8-} Mike Friede, Völkerverständigung und überhaupt blabla diff --git a/lisp/files.el b/lisp/files.el --- a/lisp/files.el +++ b/lisp/files.el ＠＠ -4177,6 +4177,12 ＠＠ If WILDCARD, it also runs the shell spec file switches wildcard full-directory-p))) (t (let* ((beg (point)) + ;; on Unix, assume that ls will output in what the + ;; current locale specifies + (coding-system-for-read + (if (fboundp 'get-coding-system-from-locale) + (declare-fboundp (get-coding-system-from-locale (current-locale))) + coding-system-from-read)) (result (if wildcard ;; Run ls in the directory of the file pattern we asked for. _______________________________________________ XEmacs-Beta mailing list XEmacs-Beta(a)xemacs.org http://calypso.tux.org/cgi-bin/mailman/listinfo/xemacs-beta

"Stephen J. Turnbull" <stephen(a)xemacs.org&gt; writes: Yes. It only applies to local "ls" on Unix systems. Right. However, the sanest assumption is that the user has set a locale matching what the filenames are. (I was afraid of tackling the general issue---that seems like a much taller order.) Could you give a hint about detecting UTF-8? (I know what UTF-8 looks like, but enough about the other coding systems to be able to say what distinguishes them.) -- Cheers =8-} Mike Friede, Völkerverständigung und überhaupt blabla _______________________________________________ XEmacs-Beta mailing list XEmacs-Beta(a)xemacs.org http://calypso.tux.org/cgi-bin/mailman/listinfo/xemacs-beta

"Stephen J. Turnbull" <stephen(a)xemacs.org&gt; writes: I'm not sure I understand: These are conditions which must hold true for UTF-8. Is the presence of a valid UTF-8 3-byte encoding in a byte sequence enough to be able to say that it is UTF-8? What about typical Latin-1 text, whose UTF-8 encodings will include only 2-byte encodings? -- Cheers =8-} Mike Friede, Völkerverständigung und überhaupt blabla _______________________________________________ XEmacs-Beta mailing list XEmacs-Beta(a)xemacs.org http://calypso.tux.org/cgi-bin/mailman/listinfo/xemacs-beta

Michael Sperber writes: No, it's not. For one thing, in a shell buffer you might be accessing several remote systems or cat'ing files saved from MIME mail in the specified encoding. Let's not worry about those, though. Consider the popular Western European languages (English, Spanish, French, and German). Suppose the text has three non-ASCII characters you want to encode. The accented letters (acute, grave, tilde, umlaut) almost always occur in isolation: that can't be UTF-8 where high-bit-set bytes occur only in groups of two or more. If they do occur in groups, what is the chance that the first of the group has high bits that encode the byte count of the UTF-8-like group it is in? Note that the probability that an ISO-8859 encoding of random characters will put a UTF-8 trailing byte in the text is fairly low by itself, since only the range 0xA0-0xBF (1/8) satisfies both ISO-8859's avoidance of C1 controls and the UTF-8 trailing byte restriction. How likely it is given the contraints of natural language vocabulary, I don't know. Spanish, however, has those inverted punctuation characters used at the beginning of a sentence, and it does have words that begin with an accented character. Do we need to worry? No! What will precede the punctuation character? Most likely an ASCII whitespace character, possibly NO-BREAK SPACE (NBSP). But what precedes NBSP? Probably whitespace or ASCII punctuation. NBSP is encoded as a UTF-8 trailing byte, so that can't be a UTF-8 character: no leading byte. How about the inverted punctuation? We got lucky: they're both trailing bytes, the argument holds. I believe the same analysis holds for French guillemots, currency symbols, etc. So once you've got 3 non-ASCII characters encoded in ISO-8859 Latin, the chances that some UTF-8 restriction isn't violated seem rather slim. It's all very hand-wavy and heuristic, but seems pretty solid to me. The only reason we even care about this, by the way, is that Mule decoding is a lossy transformation and we don't provide any way for the user to recover the original, and try again. _______________________________________________ XEmacs-Beta mailing list XEmacs-Beta(a)xemacs.org http://calypso.tux.org/cgi-bin/mailman/listinfo/xemacs-beta

Ar an cúigiú lá déag de mí Eanair, scríobh Aidan Kehoe: “Truncated.” And yay, I’m wrong, it doesn’t happen. -- ¿Dónde estará ahora mi sobrino Yoghurtu Nghé, que tuvo que huir precipitadamente de la aldea por culpa de la escasez de rinocerontes? _______________________________________________ XEmacs-Beta mailing list XEmacs-Beta(a)xemacs.org http://calypso.tux.org/cgi-bin/mailman/listinfo/xemacs-beta