OG> Note: The file is detected as MSW-UTF8, whatever that means. I think that's just using the UTF-8 signature. Wordpad likes this. I think it's insane. You're on your own for now though -- Ben's gone to Thailand for a month. Look in unicode.c for the priority variables, then tweak them. -- Institute of Policy and Planning Sciences http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.

Frank> That's what's pissing me off. He writes an extremely buggy Frank> new mule which no one else understands and which lacks Frank> essential features and then doesn't care about it anymore. That's not true. What is true is that XEmacs 21.5 is a development series, and the developers reserve the right to work on what they think is important in their own time. Ben has probably produced about 200,000 lines of patches since the Mule workspace merge. It just happens that few of them are directly Mule-related. That's the way he works: he concentrates on one big change per workspace. Admittedly, this has pretty much the same effect on users as "not caring." But nobody has claimed that 21.5 is "suitable for daily use" yet, let alone promised support. For many purposes it's fine for daily use; several of the developers use it pretty much exclusively. For you, it's not. I'm sorry about that---I'd like to see all of the features being developed at about five times the current rate, and most especially Mule, which I use heavily---but I don't see what you're "pissed off" about. Frank> Katsumi Yamaoka reports several times that 2.5 is Frank> absolutely broken and unusable for Japanese users, no Frank> reaction. That's an exaggeration. It may be unusable for Katsumi and similar users, but I don't find it unusable, and I do use Japanese on a daily basis with 21.5. It's clear from patch submissions that there are other Japanese users actively working with XEmacs 21.5. Frank> Well, Gnu Emacs can now display images under MS Windows and Frank> works generally faster and more stable, so I'll probably Frank> switch over Christmas. If GNU's image support is sufficient for you, please do use it. I think I speak for all the developers when I say that we really appreciate your bug reports, and we plan to address them in appropriate sequence with other work. And I hope that you will check out XEmacs 21.5 again in, say, mid-March, by which time I expect another wave of Mule work to have been committed. But you should use the tool that gets the job done for you. -- Institute of Policy and Planning Sciences http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.

KY> The autodetection is one of the most important Mule feature. Yes, and it is _terribly_ broken in 21.4 from the point of view of everyone in the world except Japanese and ISO-2022 multibyte users in general. It's the attempt to make autodetection _safe_, let alone useful, for the 95% of our users who aren't Japanese that is a major motivation for rewriting Mule. We get patches to return things to Japan-centric normal from you guys, and they re-break things for the Europeans. That's not acceptable. KY> There are other Mule-related problems as I reported in the KY> past. You have the source, you have the data that uncovers the problem. Feel free to fix it, just don't break things for other users. The best I can personally offer you (I am not up to speed on 21.5 yet) is that if you write a test for tests/automated/mule-tests.el, I'll make sure it gets committed, and it will be an eternal irritant to Ben (who hates test failures), until he fixes them. And I know Ben plans to get back to Mule soon after vacation, but he's got a couple more work spaces to commit first. -- Institute of Policy and Planning Sciences http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.

KY> Only Japanese people might be receiving a benefit from the KY> autodetection. It's worse than that. There are several verified ways that 21.4-style autodetection can destroy data. GNU Emacs fixed many of those problems with "safe-charsets", but by the time that happened Ben's rewrite was well underway. And also the GNU Emacs safe-charset implementation is not very satisfactory in some ways. KY> I cannot bear the thought of being lost the feature. Well, as I suppose you know, what Ben has is mind is a statistical detector that (eg) can distinguish EUC-JP from EUC-TW or EUC-KR (although the really important case is the ISO-8859-X mess and the various non-conforming sets like KOI8 and the Windows 12xx sets). But the design is completely new, so we need to retune it. Also there seem to be some bugs in coding priorities. KY> I heard it also before. I can believe all the other work of KY> him is extremely important for the future of XEmacs. But I KY> wish he don't leave Japanese XEmacs fan in the lurch. Well, what we really need to do is find a way to get more participation from others. Right now it's very hard because "new Mule" is hardly documented and very incomplete, but we appreciate efforts from you and Ueno-san and Ikiuo-san. Also there are some discussions with Morioka-san; I don't think we can "unify" XEmacs with UTF-2000 yet[1], but perhaps we can bring them a lot closer. Pretty soon XEmacs 21.4 will be stable, Vin will take care of it, and I'll be able to work more on 21.5 and on "infrastructure" (== helping people who don't have time to be "core team" to contribute), too. Footnotes: [1] And also I think Morioka-san still hopes for UTF-2000 for GNU Emacs. -- Institute of Policy and Planning Sciences http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.

Alexey> I can say that there is an extremely good scheme for Alexey> statistical detection of various Russian (really Russian, Alexey> not Cyrillic) encodings, done by S. V. Znamensky. I tried Alexey> it, and it works really wonderful, allowing even Alexey> "twice-encoded" text which is seen occasionally. That would be a very nice example from my point of view, even though it is limited to Russian. If it also happened to be able to reject (say) KOI8 Ukrainian and ISO 8859-7 Greek, that would be a wonderful showcase for the feature. Alexey> I thought of adding something like this to XEmacs. Now if Alexey> there is a common infrastructure for this, I'd be glad to Alexey> help in that area. Well, AFAIK Ben is monoscriptal in ISO-8859-1 for practical purposes. So I don't know if the current infrastructure will necessarily support existing statistical detectors. But I'll take a close look and try to come up with some docs. I'm pretty sure Ben is interested enough to be responsive to requests for enhancement of the mechanism. Alexey> I'm now playing with current XEmacs-beta. It recognizes Alexey> my ~/.xemacs/init.el as UTF-16, This is probably a priority bug. Alexey> and does not let me to change the encoding with "C-x RET f Alexey> koi8-r RET" This is probably not. Alexey> (but "C-x RET c koi8-r C-x C-f" works). The file itself Alexey> is mostly ASCII, with two strings in Russian inside (near Alexey> the end of the file). Alexey> Are you interested in such bug reports, and if yes, should Alexey> I send the file or what? Other files it at least detects Alexey> as "Raw". Most definitely. Especially from Cyrillic and Japanese users, who are the roughest tests on autodetection (except for maybe Buddhist scholars) because of the multiple encodings in daily use, plus the need to handle ASCII, ISO-8859-1, and ISO-8859-15 for programming etc. The report alone is probably enough for priority bugs. However, if you have a file you can send, that would be very nice. As usual, the shorter the better. The very best would be a test library in test-harness.el format (see tests/automated/test-harness.el and the "Regression Testing" node in the Internals Info manual). Alexey> set-language-enviroment Cyrillic-KOI8 does not help at Alexey> all. This doesn't entirely surprise me as I know Ben started a synch of the language environment stuff to GNU 21.x, but in the process broke stuff for Japanese at least. I wouldn't be surprised if something similar happened in Cyrillic. -- Institute of Policy and Planning Sciences http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.

Alexey> and I don's see anything except for UTF*, UCS, iso2022, Alexey> big5 and shift-jis. Well, that covers almost everything except KOI8 and Devanagari. :-) Alexey> No wonder it does not know about koi8-r :) What's the Alexey> status of additional detectors now? Will Ben write Alexey> missing ones himself, or is it expected that someone takes Alexey> this job? :) Or they aren't really missing? I'm sure they're ... ahh ... "unimplemented". :-) The problem is that both writing and testing detectors, especially the latter, requires familiarity with the language and access to a variety of natural texts. Somebody (Ben or me or ...) will eventually get around to writing them. However, the most effective method would be for you or another native speaker to dive in. Ben is normally eager to work with people who are doing this kind of work, although you do have to keep after him (eg, write him directly, and so on; his medical condition makes it difficult for him to follow the mailing lists etc much of the time). Note that Ben will not be back from Thailand until Jan 18, and probably will be catching up with school and so on for the first two weeks. Adjust to fit your own schedule, but from the point of view of getting his attention I'd say writing him around Jan 25, especially if you have some code in hand by then, is optimal timing. Feel free to post to the -beta or -mule lists any time. I know that the Japanese are feeling very constrained by the current state of Mule, too, so there may be people willing to help each other even though the language specifics are different. I'll help, too, if I can, but at this point all I know is that the detection stuff has changed. :-) -- Institute of Policy and Planning Sciences http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.

SJT> Well, that covers almost everything except KOI8 and SJT> Devanagari. :-) Alexey> Huh? What about ISO8859-x? What about CP1251/CP866/etc.? ISO 8859-X are all versions of ISO 2022. The IBM/Microsoft code pages are quite close to their ISO 8859 counterparts in practice, except for the damn dumbquotes. The problem for KOI8 as I recall is that unlike the code pages its map is substantially different from the ISO version, and maybe there are letters in the C1 range (0x80--0x9F)? So traditionally Mule has hacked it in using CCL, but in XEmacs 21.5 that should not be necessary. I'll try to look at this and give you some more information, but at the moment I'm all promised up :-( SJT> However, the most effective method would be for you or another SJT> native speaker to dive in. Ok. I understand. I'll try to take a look at this... --alexm -- Institute of Policy and Planning Sciences http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.