12:01 p.m.
I noticed a problem with some Croatian characters when I set up XKB to
a Croatian keyboard layout. The "hr_US" layout is intended for
programming: it uses the standard US layout with the addition of
Croatian keys when AltGr and the appropriate character where the
Croatian keys are expected to be found on Croatian keyboards.
I tested it in XEmacs and it seemed to work nicely. Then I noticed
that the "š" character was misbehaving: M-b and M-f were treating it
as a word separator and M-u wouldn't capitalize it. WTF? I was
pretty sure that character properties had worked correctly in Mule for
at least several years; was this a regression?
It turned out that it wasn't: the š character produced using any of
the Quail input methods behaved correctly. Pressing the scaron keysym
simply causes the insertion of a /different character/. For š it
inserts a character with the mule code 0x14a8 and for ž it inserts the
character with the mule code 0x14b8. The Latin-2 š and ž have Mule
codes 0x139 and 0x13e, respectively. The impostors have identical
glyphs, so they are not visually distinguishable from the real ones.
char-charset reports those chars to belong to the `latin-iso8859-15'
charset, which comes as a surprise. I didn't know ISO 8859-15 even
*contained* š and ž! But they do, and XEmacs prefers that charset to
ISO 8859-2 when inserting keysyms "scaron" and "zcaron". For
example:
(char-charset (get 'scaron 'ascii-character))
-> latin-iso8859-15
(char-charset (get 'ccaron 'ascii-character))
-> latin-iso8859-2
I know I could use latin-unity to convert the Latin 9 chars to Latin
2, but it seems reasonable to attempt to insert the correct chars in
the first place. If the user's environment is Latin 2 based, wouldn't
it make sense to prefer Latin 2 characters when converting keysyms to
characters?
9:40 a.m.
Ar an séú lá déag de mí Méan Fómhair, scríobh Hrvoje Nikšić:
[...] I know I could use latin-unity to convert the Latin 9 chars to
Latin 2, but it seems reasonable to attempt to insert the correct chars
in the first place. If the user's environment is Latin 2 based, wouldn't
it make sense to prefer Latin 2 characters when converting keysyms to
characters?
It makes even more sense not to differentiate between identical characters
at all ...
Stephen’s Latin Unity package assigns the Latin 9 characters as the
corresponding keysyms’ mappings, if no mapping exists when it’s loaded. To
avoid this, and to have your Latin 2 characters inserted by default, start
XEmacs with the hr_US layout already loaded, and the X11 platform code will
look at the keysym on the keyboard, note that it’s in the Latin 2 range, and
assign it the Latin 2 character.
--
„Frauen achten mehr aufs Herz und weniger auf Dummheiten. Darum leben sie
länger.“ (C.R. Zafón -- Übertragung von Peter Schwaar.)
11:24 a.m.
Aidan Kehoe <kehoea(a)parhasard.net> writes:
It makes even more sense not to differentiate between identical
characters at all ...
No argument here.
Stephen-F˘s Latin Unity package assigns the Latin 9 characters as
the-B
corresponding keysyms-F˘ mappings, if no mapping exists when it˘s-B
loaded. To avoid this, and to have your Latin 2 characters inserted
by default, start XEmacs with the hr_US layout already loaded, and
the X11 platform code will look at the keysym on the keyboard, note
that it-F˘s in the Latin 2 range, and assign it the Latin 2 character.-B
I'm not sure what you mean. For one, I'm not even using latin unity,
so it can't be held responsible for this problem. Then, I don't think
starting XEmacs with hr_US already loaded helps any; XEmacs is capable
of locating "new" keysyms, looking up their character values, and
binding them to self-insert-command.
Also, why do you think the keysym will prove to be "in the Latin 2
range"? What if the keysym turns out to be in the Latin 9 range? š
and ž *are* also present in Latin 9.
11:57 a.m.
Ar an seachtú lá déag de mí Méan Fómhair, scríobh Hrvoje Nikšić:
> Stephen’s Latin Unity package assigns the Latin 9 characters as
the
> corresponding keysyms’ mappings, if no mapping exists when it’s
> loaded. To avoid this, and to have your Latin 2 characters inserted
> by default, start XEmacs with the hr_US layout already loaded, and
> the X11 platform code will look at the keysym on the keyboard, note
> that it’s in the Latin 2 range, and assign it the Latin 2 character.
I'm not sure what you mean. For one, I'm not even using latin unity,
so it can't be held responsible for this problem. Then, I don't think
starting XEmacs with hr_US already loaded helps any;
[Have you tried?]
XEmacs is capable of locating "new" keysyms, looking up
their character
values, and binding them to self-insert-command.
Yeah; and if they’re already bound, XEmacs won’t do that.
Cf. this, which used to be in latin-unity, but was factored out to
latin-euro-latin9.el to get Mule-UCS to stop trashing data from some
relevant character sets;
;;;###autoload
(flet ((define-keysym-as-char (keysym character)
(unless (lookup-key global-map (vector keysym))
(define-key global-map (vector keysym) #'self-insert-command))
;; Belt and braces ...
(unless (get-character-of-keysym keysym)
(set-character-of-keysym keysym character)))
(foo (k o)
(define-keysym-as-char k (make-char 'latin-iso8859-15 o))))
(foo 'EuroSign #x24)
(foo 'Scaron #x26)
(foo 'scaron #x28)
(foo 'Zcaron #x34)
(foo 'zcaron #x38)
(foo 'OE #x3C)
(foo 'oe #x3D)
(foo 'Ydiaeresis #x3E))
If (featurep 'latin-euro-latin9) or (featurep 'latin-unity-latin9) is true,
and XEmacs didn’t know about an scaron or zcaron keysym existing on the
server’s keyboards when those files were loaded, then scaron and zcaron will
be bound to the Latin 9 characters by that autoload code. Changing to a
keyboard with scaron or zcaron available won’t change their bindings, which
is the right thing to do in the general case, but bit you in the arse here.
The general solution to this problem is to have a state for X11 keysyms that
says “this keysym is on the keyboard but we haven’t found anything to bind
it to, even though it should bind to a character” As it is, we don’t
differentiate between this case and the “this keysym is on the keyboard
and isn’t bound.” Implementing this general case does not make sense given
the benefit involved and the work necessary. “Start XEmacs with hr_US
already loaded” is an acceptable workaround, given that that’s what you’re
doing anyway.
Also, why do you think the keysym will prove to be "in the Latin
2
range"? What if the keysym turns out to be in the Latin 9 range?
Then the X server is broken, because that character already existed in the
Latin 2 range before the Latin 9 range was introduced, and the specs say
“use the old keysym” in that case.
Of course, lots of X servers _are_ broken, and they may well be broken in
this way. That’s what x-win-init-xfree86 and x-win-init-sun are for, and we
can stick in code there to handle this case if it happens.
š and ž *are* also present in Latin 9.
--
„Frauen achten mehr aufs Herz und weniger auf Dummheiten. Darum leben sie
länger.“ (C.R. Zafón -- Übertragung von Peter Schwaar.)
1:24 p.m.
Aidan Kehoe <kehoea(a)parhasard.net> writes:
> I'm not sure what you mean. For one, I'm not even
using latin unity,
> so it can't be held responsible for this problem. Then, I don't think
> starting XEmacs with hr_US already loaded helps any;
[Have you tried?]
Of course, and more than once. Since I now use hr_US all the time,
any time I start XEmacs, it is with hr_US already "loaded".
Cf. this, which used to be in latin-unity, but was factored out to
latin-euro-latin9.el to get Mule-UCS to stop trashing data from some
relevant character sets;
;;;###autoload
(flet ((define-keysym-as-char (keysym character)
(unless (lookup-key global-map (vector keysym))
(define-key global-map (vector keysym) #'self-insert-command))
;; Belt and braces ...
(unless (get-character-of-keysym keysym)
(set-character-of-keysym keysym character)))
(foo (k o)
(define-keysym-as-char k (make-char 'latin-iso8859-15 o))))
(foo 'EuroSign #x24)
(foo 'Scaron #x26)
(foo 'scaron #x28)
(foo 'Zcaron #x34)
(foo 'zcaron #x38)
Eek. That is amazingly evil. I don't know why they added š and ž to
Latin 9 in the first place, but hardcoding them to that charset in
XEmacs definitely makes no sense. I vote to remove scaron and zcaron
from that code.
If (featurep 'latin-euro-latin9) or (featurep
'latin-unity-latin9)
is true,
The former is t.
I've elided your text which refers to a workaround ("starting XEmacs
with hr_US already loaded") that doesn't work. I still think it makes
sense not to force scaron and zcaron into Latin 9, where they clearly
don't belong.
Even if it worked, it wouldn't be a good workaround. People *do*
switch keyboards, and switching keyboards can't be expected to break
things.
1:46 p.m.
Ar an seachtú lá déag de mí Méan Fómhair, scríobh Hrvoje Nikšić:
Of course, and more than once. Since I now use hr_US all the time,
any time I start XEmacs, it is with hr_US already "loaded".
Then I can’t reproduce your problem.
setxkbmap hr_US
mv ~/.xemacs/init.el ~/.xemacs/init.el-
echo "(require 'latin-euro-latin9)" > ~/.xemacs/init.el
rm ~/.xemacs/init.elc
xemacs-21.4.17 &
gives me an XEmacs where typing AltGr with the left square bracket produces
a Latin 2 character.
[...] Eek. That is amazingly evil.
Evil, sure. Not amazingly so; it’s a reasonable thing to do for the Euro
sign, where older XEmacs versions won’t know anything about the X11 keysym,
and the letters with carons make sense in a non-Mule XEmacs with a
-iso8859-15 font chosen.
I don't know why they added š and ž to Latin 9 in the first
place, but
hardcoding them to that charset in XEmacs definitely makes no sense. I
vote to remove scaron and zcaron from that code.
Seconded. Of course, that package is my baby so it’s one more thing I’m not
getting to right now. Have at it, if you have the free time.
> If (featurep 'latin-euro-latin9) or (featurep
'latin-unity-latin9)
> is true,
The former is t.
I've elided your text which refers to a workaround ("starting XEmacs
with hr_US already loaded") that doesn't work. I still think it makes
sense not to force scaron and zcaron into Latin 9, where they clearly
don't belong.
I think coding for both Mule and non-Mule involves jumping through too many
hoops to be remotely sure you’re doing it right, and this is just a symptom
of that.
Even if it worked, it wouldn't be a good workaround. People
*do*
switch keyboards, and switching keyboards can't be expected to break
things.
People switch keyboards, but this isn’t Windows, creating a personalised
keymapping so you don’t have to is trivial. Once you’ve done that, there’s
really no further need to.
--
„Frauen achten mehr aufs Herz und weniger auf Dummheiten. Darum leben sie
länger.“ (C.R. Zafón -- Übertragung von Peter Schwaar.)
10 a.m.
>>>> "Hrvoje" == Hrvoje Niksic
<hniksic(a)xemacs.org> writes:
(foo 'EuroSign #x24)
(foo 'Scaron #x26)
(foo 'scaron #x28)
(foo 'Zcaron #x34)
(foo 'zcaron #x38)
Hrvoje> Eek. That is amazingly evil.
Hrvoje, calm down. It is _not_ evil. That file is loaded only on
explicit user request (at least, that's why I put it in the packages)
and if those keysyms have associated characters, they will not be
overridden. The bug is that they don't get assigned to Latin 2
characters in a Latin 2 locale; let's fix that bug. What more could
you reasonably ask for?
I would guess that what needs to be done in principle is to extend the
code in keymap.c to do all the ISO 8859 keysyms, but that's probably
not in the cards for 21.4 and should be done very differently for 21.5
(which is not going to have Mule charsets internally in the near
future if I read the runes correctly). So the thing to do is to have
a similar file for Latin 2 (etc), and have the language environment
code invoke it.
This isn't ideal in this case (it does seem unlikely that very many
people are using direct entry of Scaron and friends in Latin 9) but I
would guess that there are other cases where hardcoding at dump time
would cause annoying shadows.
> If (featurep 'latin-euro-latin9) or (featurep
> 'latin-unity-latin9) is true,
Hrvoje> The former is t.
How did that happen? If we did that behind your back, it's an XEmacs
bug; that support was designed to be loaded on request by people who
need Latin 9 support as their primary environment---little thought has
been put into making it loadable by default, and none in combination
with non-Latin-9 environments. If you did it, I'd like to know why so
I can figure out how to ensure that you can fixup keysym translations
before the latin9 support gets loaded.
Hrvoje> I still think it makes sense not to force scaron and
Hrvoje> zcaron into Latin 9, where they clearly don't belong.
The ISO disagrees. Please, let's find a solution that allows us to
support _both_ coded character sets as defined in the standard.
Hrvoje> Even if it worked, it wouldn't be a good workaround.
Hrvoje> People *do* switch keyboards, and switching keyboards
Hrvoje> can't be expected to break things.
"The X Window System. Live the horror." Switching keyboards has
always broken things for me. The most recent Debian upgrades to X
broke my keyboard out from under me. And of course XKB in XFree86 and
possibly other X11 distros is deliberately backward incompatible with
XEmacs and other apps that handle legacy keyboards in a sophisticated
way.
This stuff is a nightmare, and it will only get worse if we
deliberately write code that doesn't conform to the existing standards.
--
School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
Ask not how you can "do" free software business;
ask what your business can "do for" free software.
10:40 a.m.
"Stephen J. Turnbull" <stephen(a)xemacs.org> writes:
That file is loaded only on explicit user request (at least,
that's
why I put it in the packages)
But I never requested it.
The bug is that they don't get assigned to Latin 2 characters in
a
Latin 2 locale; let's fix that bug. What more could you reasonably
ask for?
That's entirely fine. I could further argue that those two characters
shouldn't be assigned to Latin 9 at all because they're almost never
used by Latin 9 users (they were missing in Latin 1 and I suspect they
were added for compatibility with Windows-1252). But there's no
reason to insist on that if the above goal is met.
>> If (featurep 'latin-euro-latin9) or (featurep
>> 'latin-unity-latin9) is true,
Hrvoje> The former is t.
How did that happen?
I've just tracked it down using `load-history': it got loaded when I
loaded `un-define' to deal with a UTF-8 news article in Gnus (those
are getting more and more common). It loaded `latin-euro-standards',
which in turn loaded `latin-euro-latin9'.
If we did that behind your back, it's an XEmacs bug; that
support
was designed to be loaded on request by people who need Latin 9
support as their primary environment---little thought has been put
into making it loadable by default, and none in combination with
non-Latin-9 environments.
Yup. That's the bug, then. I certainly didn't load
`latin-euro-latin9' for the kicks of it. :-)
"The X Window System. Live the horror." Switching
keyboards has
always broken things for me.
Maybe it's normal in Japan, but people don't expect such breakage
here. Maybe we've been spoiled by Microsoft or something.
This stuff is a nightmare, and it will only get worse if we
deliberately write code that doesn't conform to the existing
standards.
I never suggested that. š and ž belong to (at least) *two*
standard-compliant charsets, and given Mule's design, we must choose
one. I was proposing to choose one whose users actually need those
characters. I still think it's a reasonable proposal.
4:50 p.m.
Ar an naoú lá déag de mí Méan Fómhair, scríobh Hrvoje Nikšić:
"Stephen J. Turnbull" <stephen(a)xemacs.org> writes:
> That file is loaded only on explicit user request (at least, that's
> why I put it in the packages)
But I never requested it.
Mule-UCS requests it, because otherwise there’s no way for it to know about
the Euro sign from the beginning, and because reworking it to take account
of new character sets added to the running XEmacs as they are loaded would
involve much more debugging of nested macros and undocumented CCL programs
than I had time or stomach for.
--
„Frauen achten mehr aufs Herz und weniger auf Dummheiten. Darum leben sie
länger.“ (C.R. Zafón -- Übertragung von Peter Schwaar.)
5:09 p.m.
>>>> "Hrvoje" == Hrvoje Niksic
<hniksic(a)xemacs.org> writes:
Hrvoje> I've just tracked it down using `load-history': it got
Hrvoje> loaded when I loaded `un-define' to deal with a UTF-8 news
Hrvoje> article in Gnus (those are getting more and more common).
Hrvoje> It loaded `latin-euro-standards', which in turn loaded
Hrvoje> `latin-euro-latin9'.
Ah, that makes some sense. I suspect that's the only way to get a
Euro sign currently. So we should be able to defeat the bad side
effect by assigning those characters to Latin 2 at startup time.
I guess the hack would be to have a latin-input-priority variable.
This would contain a list of Latin charset symbols. The default would
be <x11/keysymdef.h> order, giving the result that makes sense in the
case in point. Users with different needs could change it.
But I'm confused. Suppose you want to use a SHARP S or some other
letter, which is common to ISO 8859/1 and ISO 8859/2. Surely that's a
reasonable thing for you to do. But it gets assigned the charset
latin-iso8859-1 (which is wrong for you), doesn't it? Don't you
perceive that as breakage too?
> "The X Window System. Live the horror." Switching
keyboards
> has always broken things for me.
Hrvoje> Maybe it's normal in Japan, but people don't expect such
Hrvoje> breakage here. Maybe we've been spoiled by Microsoft or
Hrvoje> something.
I didn't exactly say I expect it, just that I've always gotten it. Be
that as it may, keyboard misbehavior is a FAQ on XEmacs Beta, and
often it is from Europeans.
Hrvoje> š and ž belong to (at least) *two* standard-compliant
Hrvoje> charsets, and given Mule's design, we must choose one. I
Hrvoje> was proposing to choose one whose users actually need
Hrvoje> those characters.
We have to choose at some point, but not at library-writing time! At
the very least, let's wait until initialization. :-)
--
School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
Ask not how you can "do" free software business;
ask what your business can "do for" free software.
5:21 p.m.
"Stephen J. Turnbull" <stephen(a)xemacs.org> writes:
But I'm confused. Suppose you want to use a SHARP S or some
other
letter, which is common to ISO 8859/1 and ISO 8859/2. Surely that's
a reasonable thing for you to do. But it gets assigned the charset
latin-iso8859-1 (which is wrong for you), doesn't it? Don't you
perceive that as breakage too?
That depends on the outlook. I completely understand if es-tzet gets
unconditionally assigned to Latin 1, and I wouldn't call it wrong. It
might be "wrong" for a very specific set of expectations (mixing
Croatian and German and expecting all characters to be in the Latin 2
range) which are very rare in the real world.
On the other hand, Croatian XEmacs users, including myself, *do*
expect š and ž (which they have on their keyboards and use daily) to
be part of Latin 2, so that they can safely enter all Croatian chars
and save the buffer as Latin 2.
4:18 a.m.
>>>> "Hrvoje" == Hrvoje Niksic
<hniksic(a)xemacs.org> writes:
Hrvoje> That depends on the outlook. I completely understand if
Hrvoje> es-tzet gets unconditionally assigned to Latin 1, and I
Hrvoje> wouldn't call it wrong.
Bad example, I guess. Are there _no_ characters used in Croation (or
other Latin-2-supported languages) that get assigned to Latin 1 by the
"use the first ISO 8859 charset that contains the character"
algorithm? For the extreme case consider Latin 9, where everything,
except the Euro, gets assigned to another Latin character set.
Hrvoje> On the other hand, Croatian XEmacs users, including
Hrvoje> myself, *do* expect š and ž (which they have on their
Hrvoje> keyboards and use daily) to be part of Latin 2, so that
Hrvoje> they can safely enter all Croatian chars and save the
Hrvoje> buffer as Latin 2.
*sigh* It's such a shame the Mule designers had to be Japanese. ISO
8859 very clearly[1] thinks of the various "parts" (ie, Latin 1, Latin
2, ISO Greek, etc) as _subsets_, and you should be able to save a
buffer in any of the subsets which contains all the buffer characters.
Only Japanese would have such a strong bias toward implementing the
ISO 8859 character sets as disjoint.
We really, really need to go to Unicode internally to force this mess
to change.
Footnotes:
[1] In hindsight. I realized this thanks to your rants.
--
School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
Ask not how you can "do" free software business;
ask what your business can "do for" free software.
9:55 a.m.
"Stephen J. Turnbull" <stephen(a)xemacs.org> writes:
>>>>> "Hrvoje" == Hrvoje Niksic
<hniksic(a)xemacs.org> writes:
Hrvoje> That depends on the outlook. I completely understand if
Hrvoje> es-tzet gets unconditionally assigned to Latin 1, and I
Hrvoje> wouldn't call it wrong.
Bad example, I guess. Are there _no_ characters used in Croation
(or other Latin-2-supported languages) that get assigned to Latin 1
by the "use the first ISO 8859 charset that contains the character"
algorithm?
I don't think so. The Croatian script is simple; in addition to ASCII
it uses "special" characters čćđšž which, with their uppercase
versions, amounts to ten characters, all present in Latin 2. Sorting
is a bit more work because it requires correct collation of those
characters and possibly support for digraphs dž, lj, and nj (which
have Unicode code points assigned that no one uses), but that's not an
immediate concern for a text editor, and can be kludged in later by
locally written applications that need that kind of thing.
Note that XEmacs is pretty close to meeting Croatian users' needs in
this; GNU Emacs doesn't recognize the `scaron' etc. keysyms *at all*.
7003
days inactive
7007
days old
12 comments
3 participants
participants (3)
-
Aidan Kehoe -
Hrvoje Niksic -
Stephen J. Turnbull