2:42 p.m.
Yes, really! In XEmacs MULE 21.5 (and 21.4 too).
Cyrillic letters are being displayed correctly in XEmacs under MULE, that's
true. BUT! many files, written in Russian, espesially in news groups contain
non-cyrillic graphic characters. Examples are: "em dash" and "en
dash", single
and double quotes [1] in cp1251, box drawing in koi8-r.
XEmacs maps codepoints that correspond to the mentioned chars into chinese and
japanese charsets. I don't have ISO-2022 on my hands, but I am pretty sure
that the standard positions those chars in Far East charsets.
This leads to a funny result -- when opening a text file in RUSSIAN, XEmacs
warns that it can not instantiate fonts for JAPANESE (CHINESE, KOREAN)!
Notice: Unable to instantiate font for charset korean-ksc5601, face default
Notice: Unable to instantiate font for charset chinese-cns11643-1, face default
Notice: Unable to instantiate font for charset japanese-jisx0208, face default
That is not a problem, but an annoying inconvenience.
I see severals ways to fix it.
1. Add entries to display table that map graphic characters to reasonable
substitutions (e.g. "en dash" to "-", "em dash" to
"--").
2. Introduce a new non-standard charset that contains such characters and
specify a particular display policy for it.
3. Wait until full Unicodizing internals and hope the problem will disappear.
Current implementation of Unicode support maps box drawing and friends to
Far East charsets.
Any comments/opinions?
Nick.
Footnotes:
[1] MS Word loves to insert them into saved plain text
2:25 p.m.
>>>> "Nickolay" == Nickolay Pakoulin
<npak(a)ispras.ru> writes:
Nickolay> XEmacs maps codepoints that correspond to the mentioned
Nickolay> chars into chinese and japanese charsets. I don't have
Nickolay> ISO-2022 on my hands, but I am pretty sure that the
Nickolay> standard positions those chars in Far East charsets.
Ugly, ain't it?
This has always been true. Nobody in a position to help with testing
has ever pushed it before (I think the usual practice is to simply use
an 8-bit XEmacs with a KOI8-R font, which gives pleasant results with
no fuss).
First, check to see if GNU Emacs has this right. We should be able to
synch to their implementation for 21.4.
If not, I think the right way to deal with it in 21.4 is your #2,
assuming you have KOI8-R fonts. (Caveat, as I said before I don't
know how to do this on Windows.)
Create a private 96 charset for each non-ISO encoding that uses C1 to
represent graphic characters. Eg, KOI8-R, create symbol-koi8-r (as
you did in the recent post). The registry should be "koi8-r" (if
there are variants, regexpify that to handle them all). Change the
registry for the cyrillic-iso8859-5 charset to "koi8-r". Change the
current KOI8-R CCL codecs to simply map graphic C1 characters to
symbol-koi8-r code point + 32 - 128, and back. Use these codecs both
for fileio and redisplay.
That's the theory, I don't know if there's anybody left who has done
this in practice.
For 21.5, you'll have to ask Ben.
Nickolay> 3. Wait until full Unicodizing internals and hope the
Nickolay> problem will disappear.
Nope. This will require work, and help from you (or some Cyrillic
users, anyway) for testing and sanity checking on X (where we can't
yet count on Unicode fonts) and maybe on Windows (I suspect we
probably need a KOI8 detector).
Nickolay> Current implementation of Unicode support maps box
Nickolay> drawing and friends to Far East charsets.
This can and should be fixed in 21.5 by changing charset precedence.
I don't know if 21.5's mechanism is flexible enough to do it right
yet; again, Ben's the guy to ask.
In 21.4, Unicode is not supported. Yes, we do some autodetection, and
we provide the Mule-UCS package. But it's orphaned by upstream, and
nobody understands it. In principle you should be able to change
Mule-UCS mappings to use KOI8-R or whatever.
--
Institute of Policy and Planning Sciences http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
Ask not how you can "do" free software business;
ask what your business can "do for" free software.
5:07 p.m.
intro: "SJT" == Stephen J Turnbull <stephen(a)xemacs.org> writes:
>>>>> "Nickolay" == Nickolay Pakoulin
<npak(a)ispras.ru> writes:
Nickolay> XEmacs maps codepoints that correspond
to the mentioned chars into
Nickolay> chinese and japanese charsets. I don't have ISO-2022 on my hands,
Nickolay> but I am pretty sure that the standard positions those chars in Far
Nickolay> East charsets.
SJT> Ugly, ain't it?
Yes, it is.
SJT> (I think the usual practice is to simply use
SJT> an 8-bit XEmacs with a KOI8-R font, which gives pleasant results with no
SJT> fuss).
Yes, but not under Windows, and not MULE (because decoding / encoding tables
are broken in non-cyrillic part of the table, see below).
SJT> First, check to see if GNU Emacs has this right. We should be able to
SJT> synch to their implementation for 21.4.
GNU Emacs 21.2 maps codepoints for box drawing to characters with the same
code. That is, when GNU Emacs comes across 130 ("BOX DRAWINGS LIGHT DOWN AND
RIGHT") in koi8-r input file then it inserts \202 in the buffer. When GNU
Emacs encodes buffer, it converts \202 back to 130. This is not the best
solution, because character \202 belongs to control-1 and it is NOT "BOX
DRAWINGS LIGHT DOWN AND RIGHT".
SJT> If not, I think the right way to deal with it in 21.4 is your #2,
SJT> assuming you have KOI8-R fonts. (Caveat, as I said before I don't know
SJT> how to do this on Windows.)
SJT> Create a private 96 charset ...
I have already finished the first draft. See
http://www.ispras.ru/~npak/koi8-pgraph.el
TODO is to make a ccl program for the charset. But makes no sense under
Windows because Windows XEmacs does not use ccl programs (imho) for charsets.
It is not tested under X. But still is helps to work with koi8-r files in
Windows.
SJT> That's the theory, I don't know if there's anybody left who has
done
SJT> this in practice.
Tables for koi8-r are broken. There are two bugs, imho.
Firstly, `cyrillic-koi8-r-decode-table' maps some codepoints to 32 (space) thus
ultimately loosing information and damging the data. Just opening and saving a
file (without editing) alters file. The effect is non-recoverable.
Secondly, ccl program `ccl-encode-koi8' takes care about cyrillic characters
only. Still, `cyrillic-koi8-r-decode-table' maps certain codepoints into Far
East characters. When `ccl-encode-koi8' comes across bytes that represent such
characters it just dumps them 'as it is'! That is, in place a single byte
there appear three bytes. This effect is recoverable, but recovery is tricky.
koi8-pgraph.el mentioned above does fix both bugs for koi8-r. I volunteer to
make similar fix for alternativny coding in both 21.4 and 21.5 and windows-1251
in 21.4. In 21.5 (at least under Windows) decoding / encoding for windows-1251
is done internally and requires further investigations.
Comments / suggestions ?
Nick.
1:19 p.m.
>>>> "Nickolay" == Nickolay Pakoulin
<npak(a)ispras.ru> writes:
Nickolay> Yes, but not under Windows, and not MULE (because
Nickolay> decoding / encoding tables are broken in non-cyrillic
Nickolay> part of the table, see below).
Oh, my. :-(
Nickolay> Comments / suggestions ?
I really, really appreciate you going to the trouble of doing all this
background study. We seem to have a moderately active contingent of
Russian users, but they mostly have been either silent in the devel
channels, or more interested in "mainstream" issues rather than
specifically Cyrillic support.
I'll look at your stuff ASAP, which means sometime next week, sorry.
At that point I'll try to specifically drag Ben into the discussion.
--
Institute of Policy and Planning Sciences http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
Ask not how you can "do" free software business;
ask what your business can "do for" free software.
7714
days inactive
7716
days old
3 comments
2 participants
participants (2)
-
Nickolay Pakoulin -
Stephen J. Turnbull