Hi, This is a followup to your commentary on my initial patch of the eighteenth of October. It’s far, far, later than I expected it would be, but ah well, minimal civilian casualties this time. More work and mail later this month. Ar an seachtú lá is fiche de mí Deireadh Fómhair, scríobh Stephen J. Turnbull: Okay, noted. Certainly, the general aesthetic changes shouldn’t be part of those patches. But as to the big deletions, when they’re made obsolete by what I’m doing, I think taking them out makes sense. Well, the intention was to move away from the current ISO-2022-friendly structure, to several alternative Unicode-friendly encodings. And my feeling is that the UTF-8 compatible encoding _should not_ be the default, given today’s RAM sizes and that every other internationalised app in the world save Perl has moved to using fixed-width or UTF-16 internally. So calling UTF-8 “FORMAT_DEFAULT” would just be confusing, in that context. I didn’t judge changing those names as gratuitous. I still don’t, to be honest. If you remember, what was this in reference to? I don’t think I did remove any code that would eventually be needed again, in any format close to what was there already. Hmm, but how to compute a canonical Ichar from the text representation, and how to compute a text representation from an Ichar, has always been dependent on the FORMAT_* defines. It’s just that FORMAT_DEFAULT was always implicit whenever MULE was #defined. If MULE goes away, as I would like it to do, and, from what I read, as does Ben, that’ll become that bit more evident. --> and charset encodes bytecount To choose a single buffer text representation at compile time. I think multiple, equally capable buffer text representations in a single binary are overkill; and I think where one representation is more capable than another, the less capable one should be taken out. The developers can experiment for six months, and pick whichever has the best features for VM folders or whatever. I don’t want to knowingly give anyone code that gives inaccurate character counts.:-) A single backspace should only eliminate a single “ S,”this LATIN CAPITAL LETTER SHARP S should succeed in a string equality comparison with (format "%c%c" ?S ?S) and implementing redisplay for composite characters is on no-one’s timetable. Dealing with external data, even if we could determine that a given “SS” string loaded from disk or pasted in _is_ a LATIN CAPITAL LETTER SHARP S according to (and all these things must be taken into account, when working it out): - German’s standardised spelling. - Whether or not our user chooses to honour the recent Rechtschreibreform. - Whether or not our user wants to stick to seven bit characters because they’re posting to a Unix-heavy newsgroup. - Whether our user is in Switzerland and will thus, normally, ignore the es-zed (ß) entirely. I think downcasing this LATIN CAPITAL LETTER SHARP S to es-zed will surprise our users--disagree with me if appropriate, those of you reading with native-speaker German--and to be able to downcase it is the only reason to work out where to have it at all. So what then? We now have a character for which the lowercase mapping _must be equivalent to two things, which are themselves distinct, at the same time_. So: (assert (equal "SS" (upcase "ß"))) (assert (equal "ss" (downcase (upcase "ß")))) Sure, that’s workable. It doesn’t solve the case for the sharp s or for downcasing Turkish LATIN CAPITAL LETTER I WITH DOT ABOVE. Or any of those crazy ligatures around U+FB00, just to cite a some examples from many at CaseFolding.txt on Unicode.org . Yeah; I think wrote that comment when I thought moving to a non-ASCII compatible internal string representation (as opposed to the buffer representation) was practical in the medium term. [And verifying alignment by checking whether addresses are divisible by the number of bytes of alignment required is emphatically not standard C, but it seems to have worked out beautifully.] Yes, normally the callers have done the work beforehand. But there remains, for the functions themselves, the possibility that the byte count passed across doesn’t map to an integer number of characters; my changes prevent them moving beyond the end of the source string, in that event. It’s a bit theoretical, and extraneous to the bulk of these changes, but I don’t think people read that code often enough that anyone else is likely to pick up on the potential for it happening. Okay. A lot of the code I was reading didn’t explicitly comment on the constant folding it was doing, so I didn’t put the energy into commenting myself. I will, though. Will do. That’s only used, and is only intended to be used in once place, and there’s an error signalled just above it if the surrogates are wrong. There are several text checking assertions on valid UTF-16 in the changes I made to text.h; from looking at the code beyond that, it didn’t seem to me as appropriate to make them for the coding systems. There are no assertions at all in mule-coding.c, from my examination, so perhaps treating coding system code as subject to text checking assertions is the right thing to do. And there always have been. I'm using one now. ? With Ichars as UTF-32, it is pretty complex. So what algorithm should be used there for determining the width of a character? Check the coding system being used for the console in question, work out what character set that (Unicode) code point maps to in that coding system, check the columns of that character set? Alternative; have several available versions of wcwidth at the C level, one with Japanese semantics (--> treating everything in JISX 0212 as two-column, doing pretty much what it does at the moment) , two with Chinese semantics--GB vs. Big5--and one with Markus' Unicode semantics[1]. Switch between them based on language environment. But there's nothing to stop someone in a Japanese language environment using a UTF-8 terminal coding system--indeed, I would expect them to be doing so on OS X, where that's the coding system used for file names--and at that point we can't predict with certainty what character width is the right one to use. It may even vary based on the font used in the terminal, and that's not information we can access via curses. (After having used the wrong character width for the double directed quotation marks for several weeks, I have to say that it’s something that’s profoundly annoying in real life. I was about to argue here that we can’t fix the thing in the general case, and, and we gain the most by endorsing an emerging standard, but after that experience, I won’t. So, the multiple-c-wcwidths thing seems the most attractive option; working things out from the console coding system is probably inviting disaster--initialising it based on language environment is probably the way to go, if that can be organised.) I�m sure you’ve an idea of it alread, but the existing algorithm is broken, as it is; try the below in a *scratch* buffer with a Unicode terminal coding system, and observe the thing wrapping at forty columns. (let ((i 0) (str "")) (while (< i 80) ;; EM DASH is two columns in our internal encoding, one in MS Mincho ;; when treated as Unicode. (setq str (format "%s%c" str (make-char 'chinese-cns11643-1 33 55)) i (+ 1 i))) str) Oooh, here's another thing that's broken; (equal ;; LATIN CAPITAL LETTER C WITH CARON (encode-coding-string (make-char 'latin-iso8859-2 #xc8) 'utf-8) ;; Idem. (encode-coding-string (make-char 'japanese-jisx0212 42 45) 'utf-8)) => t (char-width (make-char 'latin-iso8859-2 #xc8)) => 1 (char-width (make-char 'japanese-jisx0212 42 45)) => 2 (and ;; [2] (check-can-encode (make-char 'latin-iso8859-2 #xc8) 'iso-2022-jp) (check-can-encode (char-width (make-char 'japanese-jisx0212 42 45)) 'iso-2022-jp)) => t So we can't just look for the first supported character set that can be used in the console's coding system; we have to make sure that the coding system will use it. And in the abstract it's possible that something will alter the order of preferred character sets between the checking and the encoding. Best regards, - Aidan [1] Koreans probably aren't using Han characters to the extent that this sort of thing will annoy them. [2] (defun check-can-encode (string codesys) "Check if coding system CODESYS will preserve the information in STRING." (if (characterp string) (setq string (format "%c" string))) (coerce string 'string) (equal (decode-coding-string (encode-coding-string string codesys) codesys) string)) -- “Ah come on now Ted, a Volkswagen with a mind of its own, driving all over the place and going mad, if that’s not scary I don’t know what is.”