Stephen J. Turnbull wrote: it's the other way around. the current situation is incompatible with the X11 fonts, so we have to hack the values using the bogus `graphic' characteristic. note that in the new world, charsets can have values > 127 in any case. cf. big5, shift-jis, etc. so when i'm creating a new charset like `latin-windows-1252', which is compatible with iso-8859-1 but has extra chars in the range 128-159, do i do the right thing and have its chars in the range 128-255 be indexed as 128-255 (and hence be inconsistent with the `latin-iso8859-1' charset), or do i do the wrong thing and move its range down to 0-127? and then it appears to have ascii control chars in the range 0-31, but they aren't control chars, value 10 is not linefeed, value 13 is not cr, etc.? ben

Aidan Kehoe wrote: i don't want two different kinds of charsets. but we can create all charsets with the proper indices and have `split-char' and `char-octet' generate the old, wrong indices and a new function do it right. suggestions for a new api to replace `make-char' and `split-char'? btw unicode-internal now compiles (and crashes at startup, naturally). i still need to add the translation tables for koi8-r and friends, implement surrogates and (the biggest current issue) redo font handling to eliminate the concept of one-font-per-charset. also, add a concept of "language" and introduce it appropriately in the unicode/charset conversion functions. (neither of these last two will make it into the first version of unicode-internal to be integrated into the mainline.) currently only about 64 ifdef UNICODE_INTERNAL's, and almost all localized to text.h, text.c and charset.h. (these are the only files that know anything about the actual encoding of characters. chartab.c, for example, knows only that its hashing function must be different. mule-coding.c knows only that the bogus split big5 charsets don't exist under UNICODE_INTERNAL.) charset.h is totally rewritten and might go away entirely. chartab.c is drastically changed and now uses the same basic format as the unicode translation tables. also, we unfortunately can only implement 30-bit chars, even though Unicode theoretically allows 31 bits. ben

Stephen J. Turnbull wrote: yeah, i forgot about its iso-2022 connection. i really don't understand your sarcastic attitude, or what point you're trying to make. "charset" as i have defined it is a set of characters, indexed by one or two bytes. the indices are as defined in the unicode translation tables. you can certainly see my code if you want. the main purpose of charsets in the new world is to interface with external encodings, and maybe secondarily for font indexing under X. i see no purpose in creating a totally new concept rather than extending the current `charset' concept. in the new world, you can have an arbitrary number of charsets, with any characters you want in them; the only way that the code knows what's in the charset is by the appropriate unicode translations that have been provided. i've created new charsets `japanese-shift-jis' and `latin-windows-1252' and various others; `cyrillic-koi8-r' and such will be coming soon. there will be a new `mbcs' coding system type that just encodes one or more charsets using their indices, in the obvious fashion; this will replace the need for ccl. ben

Aidan Kehoe wrote: right. these functions take an extra precedence-list argument (a list of charsets), which when nil does something reasonable as a default. essentially i removed unicode-to-char and integrated it into make-char; but i'm open to other suggestions. well, basically i want some api that says "convert a charset and codepoints into a Lisp char" and vice-versa "get a charset and codepoints from a Lisp char". keep in mind that we will be running both unicode-internal and old-mule for awhile, so the api's must work with both. for this reason, i also introduced other levels of conversion, similar but a bit different; e.g. charset-codepoint-to-unicode and unicode-to-charset-codepoint unilaterally convert between a unicode codepoint and a charset codepoint, regardless of the representation of a char. on the other hand, int-to-char and char-to-int always give you the actual int that makes up the character; not very portable. if you want unicode, char-to-unicode. if you want charset codepoints, it's currently `split-char'. but i suppose i could create `char-to-charset-codepoint', which would follow the others, and `charset-codepoint-to-char', and reintroduce `unicode-to-char', and deprecate `make-char' and `split-char'. then you'd have a very symmetrical api. suggestions for less verbose names are welcome. no, neither the redone font handling nor the concept of a language introduced into the unicode conversion functions will make it in. the font handling will still bogusly be in terms of charsets, with the same bogus hack currently there (under windows at least) to look harder through various fonts to find a font that can display a char, when necessary. i know what needs to be done to change this but it will be a pervasive change and i don't want to bite off too much at a time right now. similarly, the idea of introducing a language and tracking the language of text using extent properties will take some doing. first step would be to create a language object and set up properties on it, such as the charset precedence list for unicode translation. then there is a buffer-local `current-language' variable (maybe the language environments can be made to work without too much effort). so we need to pass around some sort of object from which the charset precedence list can be derived -- the list itself, a language object (maybe just a symbol, who knows), a buffer, etc. currently most functions don't bother with this, so it requires a fair amount of refactoring. which file? chartab.c got radically reworked to support 32-bit characters using a page-table-style lookup, as for unicode translation tables. same system is used for old-mule as well. mule-coding.c did not need so much rewriting, but it now works in terms of generic functions. e.g. when iso2022 encodes to external, it uses macros to accumulate a whole character, then calls itext_to_charset_codepoint() -- an inline function -- to get an appropriate charset and codepoint. similarly, once it's gotten a charset and codepoints, it calls charset_codepoint_to_dynarr() -- another inline function -- to write it out. these inliners work in the old and new system, conditionalized appropriately. similar stuff goes everywhere else; i abstracted out "leading bytes" and all specific knowledge of our internal representation whenever possible. yeah, i've heard, but UCS-4 is still "theoretically" 31 bits, but all private use ... oh well. we'll just say the high half of the space is "private and reserved", heh heh :) ben

Ben> but we can create all charsets with the proper indices and Ben> have `split-char' and `char-octet' generate the old, wrong Ben> indices and a new function do it right. suggestions for a Ben> new api to replace `make-char' and `split-char'? Why do we want this interface at all, aside from backward compatibility with legacy Mule? With Unicode inside, a `get-char-components' function must be set-valued or nondeterministic (it depends on some global precedence list), and (eq (list cs o1 o2) (get-char-components (make-char-from-components cs o1 o2))) is mostly false. `charset-of-char' and `octet-of-char' don't make sense as separate APIs, they must be generated together. As far as I can see, this is not something that can be done right. There are only different ways to do it wrong, none of which are very attractive to me. -- School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.

Ben> well, basically i want some api that says "convert a charset Ben> and codepoints into a Lisp char" and vice-versa "get a Ben> charset and codepoints from a Lisp char". keep in mind that Ben> we will be running both unicode-internal and old-mule for Ben> awhile, FWIW, IMO nobody's going to be running with unicode-internal if neither font-handling nor language support is in, except for the sake of testing unicode-internal. So, we only need to support legacy Mule stuff for applications testers are using, on request from the testers. That will give us a much better idea of how much backward compatibility we should support, and how. Ben> on the other hand, int-to-char and char-to-int always give Ben> you the actual int that makes up the character; not very Ben> portable. But with Unicode inside, we simply don't care, because characters have no internal structure. Sure, users will find uses for inspecting the internal representation (eg, pure curiosity), so we may as well provide int-to-char and char-to-int. (And of course Unicodes are perfectly portable.) I cannot think of a single use case (excluding legacy Mule compatibility) for the charset-oriented functions. Start with the core code. There is no use of `split-char' and exactly one use of `make-char' in 21.5/lisp. There are a few of each in 21.5/lisp/mule. The majority of uses are those in about.el, european.el, greek.el, and hebrew.el. Those reflect what is just a long-standing bug: the lack of a proper facility for reading non-ASCII characters in LISP code, which will obviously be fixed by standardizing on Unicode (presumably UTF-8) for source code in XEmacs. Of the rest, I doubt that any of the language-specific uses will remain in unicode-internal. Although I haven't checked carefully, there is reason for confidence. Since they are language-specific files, they all refer to a single character set, so only the octets are of interest; the charset can be left implicit. Then surely we can use tables. That leaves 5 uses. Of those 5, the two in mule-category.el can be trivially and portably replaced by int-to-char. The use in latin.el can and should be replaced by reference to the Unicode case conversion tables. Which leaves two from mule-charset.el. Deprecate the notion of charset, as I propose, and we have *no* uses of split-char or make-char left in core. lisp/about.el: (let ((muenchen (format "M%cnchen" (make-char 'latin-iso8859-1 252)))) 21.5/lisp/mule/cyrillic.el: (split (split-char ch))) 21.5/lisp/mule/cyrillic.el: (split (split-char ch))) 21.5/lisp/mule/devan-util.el: (let ((charcodes (split-char char))) 21.5/lisp/mule/devan-util.el: (make-char 'indian-2-column ?\x21 (nth 1 charcodes)) 21.5/lisp/mule/devan-util.el: (let ((charcodes (split-char char))) 21.5/lisp/mule/devan-util.el: (make-char 'indian-is13194 (nth 2 charcodes)) 21.5/lisp/mule/ethio-util.el: (make-char 21.5/lisp/mule/ethio-util.el: (let ((char-components (split-char ch))) 21.5/lisp/mule/european.el: do (modify-syntax-entry (make-char 'latin-iso8859-14 c) "w")) 21.5/lisp/mule/european.el: (modify-syntax-entry (make-char 'latin-iso8859-14 code) "_")) 21.5/lisp/mule/european.el: do (modify-syntax-entry (make-char 'latin-iso8859-16 c) "w")) 21.5/lisp/mule/european.el: (modify-syntax-entry (make-char 'latin-iso8859-16 code) "_")) 21.5/lisp/mule/european.el:(modify-syntax-entry (make-char 'latin-iso8859-16 #xa5) "\"") 21.5/lisp/mule/european.el:(modify-syntax-entry (make-char 'latin-iso8859-16 #xb5) "\"") 21.5/lisp/mule/european.el:(modify-syntax-entry (make-char 'latin-iso8859-16 #xab) 21.5/lisp/mule/european.el: (format "(%c" (make-char 'latin-iso8859-16 #xbb))) 21.5/lisp/mule/european.el:(modify-syntax-entry (make-char 'latin-iso8859-16 #xbb) 21.5/lisp/mule/european.el: (format ")%c" (make-char 'latin-iso8859-16 #xab))) 21.5/lisp/mule/european.el: (make-char 'latin-iso8859-15 c) 21.5/lisp/mule/european.el: (string (char-syntax (make-char 'latin-iso8859-1 c))))) 21.5/lisp/mule/european.el:(modify-syntax-entry (make-char 'latin-iso8859-16 #xab) 21.5/lisp/mule/european.el: (format "(%c" (make-char 'latin-iso8859-16 #xbb))) 21.5/lisp/mule/european.el:(modify-syntax-entry (make-char 'latin-iso8859-16 #xbb) 21.5/lisp/mule/european.el: (format ")%c" (make-char 'latin-iso8859-16 #xab))) 21.5/lisp/mule/european.el: do (modify-syntax-entry (make-char 'latin-iso8859-2 c) "w")) 21.5/lisp/mule/european.el:(modify-syntax-entry (make-char 'latin-iso8859-2 32) "w") ; no-break space 21.5/lisp/mule/european.el: do (modify-syntax-entry (make-char 'latin-iso8859-3 c) "w")) 21.5/lisp/mule/european.el:(modify-syntax-entry (make-char 'latin-iso8859-3 32) "w") ; no-break space 21.5/lisp/mule/european.el: do (modify-syntax-entry (make-char 'latin-iso8859-4 c) "w")) 21.5/lisp/mule/european.el:(modify-syntax-entry (make-char 'latin-iso8859-4 32) "w") ; no-break space 21.5/lisp/mule/greek.el: do (modify-syntax-entry (make-char 'greek-iso8859-7 c) "w")) 21.5/lisp/mule/hebrew.el: do (modify-syntax-entry (make-char 'hebrew-iso8859-8 c) "w")) 21.5/lisp/mule/hebrew.el:(modify-syntax-entry (make-char 'hebrew-iso8859-8 32) "w") ; no-break space 21.5/lisp/mule/latin.el: (put-case-table-pair (make-char charset uc) (make-char charset lc) tbl)))) 21.5/lisp/mule/mule-category.el: (setq found (make-char 'ascii a))) 21.5/lisp/mule/mule-category.el: (setq list (cons (make-char 'ascii a) list))) 21.5/lisp/mule/mule-charset.el: (from-rev (nreverse (split-char from))) 21.5/lisp/mule/mule-charset.el: (to-rev (nreverse (split-char to)))) 21.5/lisp/mule/thai-xtis.el: to (nth 1 (split-char (aref chars i)))) 21.5/lisp/mule/thai-xtis.el: (setq ch (nth 1 (split-char (aref chars i))) 21.5/lisp/mule/vietnamese.el: (split-char (aref viet-viscii-decode-table i))) 21.5/lisp/mule/vietnamese.el: (split-char (aref viet-vscii-decode-table i))) How about mule-base? Not much here, either. char-table.el is a utility for describing a Mule charset. The language-specific uses can presumably be replaced by simple tables. mule-keyboard.el is an interface to terminals whose keyboards send ISO 2022 encodings. I don't think we can justify charsets on the basis of mule-base, either. mule-base/char-table.el: (make-char charset plane (+ (* r 16) l)) mule-base/char-table.el: (make-char charset (+ (* r 16) l)) mule-base/ethio-util.el: (make-char mule-base/ethio-util.el: (let ((char-components (split-char ch))) mule-base/mule-keyboard.el: (keyboard-self-insert (make-char charset last-command-char)) mule-base/mule-keyboard.el: (keyboard-self-insert (make-char charset _keyboard-first-byte_ mule-base/mule-keyboard.el: (keyboard-self-insert (make-char 'japanese-jisx0201-kana last-command-char))) mule-base/thai-xtis-util.el: (make-char 'thai-xtis))) How about charset precedence for Unicode translation? Ben> first step would be to create a language object and set up Ben> properties on it, such as the charset precedence list for Ben> unicode translation. Display doesn't need it. Windows, Quartz, and Xft all support Unicode font indices. Just give the font engine the preferred font (which will vary by language in a natural way) and the Unicode code point.[1] Text output doesn't need it all, because the coding system will be specified, and it will specify character sets. Charsets are not used for text input encoding detection as implemented in current Mule. We detect _coding categories_ based on presence or absence of ISO 2022 control functions, plus looking at the high bits of octets to detect UTF-8, Shift-JIS, and Big5. Coding systems (and thus charsets) are not detected; they are configured as properties of the coding categories. Maybe we _should_ use character set information, but we currently don't, and I don't see why `charset' needs to be exposed to LISP at this point. Anyway, for both input and output, specifying a coding system achieves everything (and more) that specifying charsets could, and is no more confusing (especially since users in general confound coding systems and character sets). What am I missing? Footnotes: [1] Sure, there are legacy fonts, especially on X11, where we quite likely will have to supply a registry in order to locate a usable font. But in those cases it will almost always be the case that given a font family all registries that contain a given character will map it to the same glyph. So just take the first one. Anything left can probably be handled ad hoc as a per-font user customization in the face. (I have xfontsel in mind as the interface: get the registries using XListFonts, then give the user a menu for the given font. Add an XEmacs-specific alist mapping X registries to descriptions that will be more useful to typical users.) -- School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.

Ben> why is it a waste of time? took me half an hour or so. Because other people (eg, me) could do it in 45 minutes the first time, and the same half-hour it takes you for the next one. Ben> mbcs is used for implementing things like the koi8-r coding Ben> system under unicode-internal. The KOI8 coded character sets are unibyte and have no mode shifts, and at most 256 characters. Why not simply have one table? Ben> it can also replace the hand-coded big5 and shift-jis coding Ben> systems, in unicode-internal. Sure. But as you say, we've already got implementations of those, and they're going to go away over time. Ben> any ideas? (if this doesn't work, i'm sure there are gpl-ed Ben> utf-7 implementations available.) I doubt the implementation in the Unicode book is efficient or robust, and there's no error handling in it. I'm sure there's one in Emacs and another in gconv (glibc's implememntation of iconv). Python has one. Surely Perl and Ruby do. Ben> i can implement this if you can tell me the names and Ben> encodings that are typically used in these segments. the x Ben> standard only defines the general format of extended segments Ben> and doesn't say what is actually encoded in them. According to the standard, anything with an agreed name that isn't in the list (ie, iso8859-14 and iso8859-15 violate XF86's own standard, and UTF-8 should be in there). The elegant way to implement it would be to treat it as a buffer and translate it using a new lstream, parsing the name out of the extended segment header and using that to determine the coding system. Ben> we already have a gzip coding system. we also have base64 Ben> functions but not yet converted to a coding system (not too Ben> hard to do, though). internally, i already generalized Ben> coding systems (some time ago, in fact) to be typed for Ben> either bytes or characters at either end; there's also a Ben> `chain' coding system for stringing multiple coding systems Ben> together. Yeah, I'm aware of all that, but again it's mostly stuff that somebody else can do, except that it would be really nice if the lstreams and chain coding systems were exposed to LISP somehow. Ben> keep in mind that i've already done most of the work you're Ben> describing here. i think we're talking past each other; at Ben> any rate, you seem to think i'm more confused than i am. I'm sure you know exactly what you're doing, in the small. I will look at the code asap, but your verbal descriptions do not inspire confidence that what I will find is going to be a GNU-beater in practice. GNU has more than one person seriously working on their Mule implementation, and any of the senior developers is reasonably comfortable trying to diagnose and even fix bugs. That's simply not true for current XEmacs, and you're emphasizing backward compatibility. Call it back-seat driving if you like, but somebody needs to tell you about that tree looming in front of the windshield. Ben> actually, what would really help is if you could take a look Ben> at emacs-unicode-2, figure out what their api is, and Ben> summarize it. this would be extremely useful to me. I was afraid you'd say something like that. Maybe somebody left some notes in Japanese.... -- School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.

Ben> actually, in python the situation is worse, since we at least Ben> have call-with-condition-handler, which allows continuing Ben> from a thrown error; i don't think there's any equivalent in Ben> python. Hm? Any exception can be caught in Python. It's true that you cannot handle the error and restart processing from where you started, which could be painful if you error on byte 49,999,999 of a 50MB file, but because coding systems are written in Python it would be fairly easy to return the current state. I don't know if this is done in current codecs, but it wouldn't be hard. Current Mule, it's hard to even pass a sane value for where an error happened because of the layers on layers of buffering in coding streams. Ben> i think that's more or less what handa does. but it seems Ben> low-priority to me. It was a FAQ for a while. I suspect it's no longer a FAQ because people who use ISO 8859-15 have stopped using XEmacs. :-( Ben> when you want lstreams exposed, is this so people can write Ben> their own coding systems? No. I want the standard coding systems in LISP so that ordinary hackers can do more maintenance on them. Ben> well, take a look. my verbal descriptions are often elliptic Ben> because it's difficult for me to type long, detailed emails. Ben> but i keep feeling like you are worried about issues i'm Ben> already dealing with. Could be. But it's quite clear you're more worried about maintaining Mule compatibility and efficiency for 50MB buffers than about getting the cleanest possible design in front of the reviewers and testers as quickly as possible. -- School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.

Ben> Stephen J. Turnbull wrote: Ben> This has nothing to do with whether it's written in C or Ben> Python and everything to do with the buffering, which would Ben> be the same in either case. I didn't say passing error codes was hard because it was in C vs. Python; I was pointing out that the Python design doesn't do much buffering, so sane error-recovery is possible, and that because it's in Python, people who are not Python C internals geeks can work on it safely. I'd like the next generation of XEmacs to be informed by those design principles. Ben> all right, stephen, i've had enough; you're really starting Ben> to piss me off. You asked for help, but you obviously don't really want the help that's available. So just commit it when you're ready, and we'll either like it or not. I don't really care; I have no doubt that the 18000 line patch you sent me is far better for my purposes than what we currently have. But I am not going to be able to reverse engineer architecture from that in a reasonable amount of time, so I'm not even going to try. (That doesn't mean I'm not going to study the code, it just means it will probably be 6 months to a year before I'm ready to give you the kind of help you want.) -- School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.

Ben> yet your response is just "sorry, don't have time", along Ben> with further sarcasm. do you not see the persistent sarcasm Ben> in comments like "you obviously don't really want the help Ben> that's available"? No sarcasm, just my reading based on some unpleasant facts. You keep saying I should shut up until I read the code. Currently that's an 18,000 line patch, which is only going to get longer. Given the time I'm willing to put in, we're talking months. Will you wait? Now, about emacs-unicode, what is it you want to know? It looks like most of the relevant API stuff is in src/character.h (sort of similar to 21.5's text.h), charset.h, and coding.h. The DEFUNs are in the corresponding C files. A diff against current CVS head didn't show much interesting differences in a few spot checks (eg, subr.el and simple.el), but the raw diff is about 350000 lines. Pruning that to 35000 lines will be easy, but it would help to have some idea what I'm looking for. A quick skim suggests that they're using a strategy similar to Morioka's UTF-2000, changing the internal representation in the buffer, and converting to standard Mule charset and octets representation when doing display and stream I/O. For example, the interface to fonts is still fontsets, which as far as I know are only slightly abstracted from XFontSets; I saw nothing that suggests that Unicode is going to get special treatment in accessing fonts. They do now have an abstract character type. I can't say much more than that yet, but they do have a whole set of characterp etc. APIs. Although the internal representation is basically UTF-8, there are weirdities. As you know, there are a number of "holes" in UTF-8, because legal UTF-8 must be the shortest representation. Eg, although algorithmically UTF-8 0xC0 0xA4 converts to 0x24, or "$", that's not a legal UTF-8 representation, it must be simply 0x24. So the first "hole" is the two-byte sequences 0xC0 0xNN and 0xC1 0xNN, which exactly cover the 8-bit-set octets in a natural way. Emacs is going to use them, but I'm not sure what those "8-bit characters" are for. I'm guessing binary, but I'll let you know more when I've looked closer. They're also using the 47 "high planes" from U+00110000 to U+00FFFFFF for something. I suspect they're taking another page from Morioka's book and stuffing all the national standard (ie, Mule) character sets in there. I'm not sure why they're restricted to 24 bits; it may have something to do with their Lisp_Object layout. I haven't looked at regexps yet. Superficially, except for the use of denormalized UTF-8, it looks like what you've described for XEmacs, except adapted to GNU Emacs. I suppose you've seen that much for yourself, but it's useful to me to write it down. -- School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.

Stephen J. Turnbull wrote: in time. it's a coded character set, no? it's treated that way in other applications. if i want to create an mbcs coding system for shift-jis, i need to combine the shift-jis coded charset with half-width katakana and ascii. either i hard-code it into C or i do it from lisp, at which point you need to shift-jis charset visible. there is currently no other method for declaring a coded charset than using the `charset' object (that's its point, in fact). also, are you *sure* that shift-jis is *identical* to jisx0208-anything? might it not have extra characters in it? well, i did pick such a value, 0x0001 i think. i'll see about implementing the inverse lookup trick. right. in general, every coding system needs its own detector (mbcs or not). the current system is extremely bogus. well, that comment was left over from the previous implementation of koi8 as ccl; it used to say "this should really be in the ccl category but xemacs doesn't have one". of course, a ccl category is totally bogus, as bogus as an mbcs category. it would indeed be possible to make a no-c0-controls category. actually, there's already code to do checks something like this in some of the detectors, to distinguish them from binary. in general, there needs to be a separation of detectors from categories; and in fact categories should disappear entirely. instead we should just have one "category" per coding system, and each detector assigns some likelihood to all the coding systems that it knows how to detect. a simple mbcs detector, for example, would check the characters that go by, for each existing mbcs coding system, to see whether they are defined in that coding system, and based on this, assign some likelihood to all mbcs coding systems. we could then break ties based on some preferred list of coding systems. well, it doesn't hurt to use dynarrs. there won't be near enough mbcs coding systems for the space used by them to make any difference, and i'd rather not build in arbitrary limits if there's no need to. will do. l2r is the default; likewise the default for the columns is the dimension of the charset. i'm not nuking these properties but just eliminating unnecessary arguments to simplify the code. you're confusing Ichars with the indices on the to-charset tables. the to-charset tables use unsigned shorts to represent the two octets in a charset codepoint. Ichars represent actual characters (unicode or old-mule), and are ints. since all valid character values are non-negative, a negative number is a good choice for "no such character", but 0xFFFF or any other positive number isn't, even if 0xFFFF is theoretically "not a character" under Unicode. right, i did this last night. sure, collating order varies, but regexps are about searching, not sorting. a range over some part of a national charset specifies a well-defined set of characters. a range over different charsets makes no sense since charsets themselves are unordered wrt each other. a range over any subset of unicode is also well-defined. there's no reason to deprecate ranges. not until i rewrite the font-handling code. that won't happen till after i get the current code working and checked in. ben