6:44 a.m.
it now builds and runs when compiled without UNICODE_INTERNAL; this is a
sign that the new char tables work, the new char-conversion system
basically works, etc.
with UNICODE_INTERNAL i've encountered the dreaded
"iso2022-preservation" problem -- i.e. byte-compiling an iso2022-encoded
lisp file is unlikely to work without further hacking. converting them
to utf-8 is not an option since in many cases the lisp code depends on
having the proper charset encoded in the actual chars in the file, at
least in an old-Mule world. so i'm [a] trying to remove this
assumption, [b] putting in hacks to make byte-compilation
iso2022-preserving, using private unicode chars.
ben
1:07 p.m.
Ar an ceathrú lá déag de mí na Samhain, scríobh Ben Wing:
it now builds and runs when compiled without UNICODE_INTERNAL; this
is a
sign that the new char tables work, the new char-conversion system
basically works, etc.
with UNICODE_INTERNAL i've encountered the dreaded
"iso2022-preservation" problem -- i.e. byte-compiling an iso2022-encoded
lisp file is unlikely to work without further hacking. converting them
to utf-8 is not an option since in many cases the lisp code depends on
having the proper charset encoded in the actual chars in the file, at
least in an old-Mule world. so i'm [a] trying to remove this
assumption,
It’s not _that_ widespread, since non-Mule XEmacs wouldn’t honour it, as I
understand it.
[b] putting in hacks to make byte-compilation iso2022-preserving,
using
private unicode chars.
There also arises the problem of what to do when you have Unicode-encoded
source code with characters that don’t have a mapping in Mule. For example,
I’m studying a language of the former Soviet Union which uses more Cyrillic
characters than are available in ISO 8859-5, and my local UTF-code-points
preserving code[1] will not allow my hacked input method to be reliably
byte-compiled. The byte-compiled code uses escape-quoted as its coding
system, which, although it is in general a universal coding system for Mule,
breaks down if you have just-in-time allocation like the implementation
linked below does, as different invocations will have different internal
Unicode code point mappings.
I suppose there are three cases we care about --
-- The source code sticks to ASCII. In this case, don’t specify a coding
system in the compiled file, and we don’t have a problem.
-- The source code uses ISO 2022 escape sequences. In this case, we should
specify escape-quoted as the coding system in the compiled file, because ISO
2022 encodes more characters than Unicode does, because we want the
assumptions about character sets mentioned by you above to be preserved, and
because it’s more compatbile with old implementations of Mule.
I consider files with the existing eight-bit coding systems a subset of
this case, because that behaviour is more compatible with old Mule.
-- The source code uses a Unicode coding system. In this case, specify utf-8
as the coding system in the compiled file. This will be incompatible with
older implementations of Mule; but hopefully with the advantage that they
will choke obviously, and that they will preserve non-Mule characters stored
in the source file.
[1] http://list-archive.xemacs.org/xemacs-patches/200506/msg00040.html
--
“I, alone, perhaps, in this city of nearly two million, view it with sadness,
sympathy, and respect, seeing in the millions of Russian youngsters who laid
down their lives in that war a tragedy rising above all the political
emotions of that time [...]” -- George Kennan on Vienna's „Erbsenkönig“
1:40 a.m.
On 11/15/05, Aidan Kehoe <kehoea(a)parhasard.net> wrote:
Ar an ceathrú lá déag de mí na Samhain, scríobh Ben Wing:
> it now builds and runs when compiled without UNICODE_INTERNAL; this is a
> sign that the new char tables work, the new char-conversion system
> basically works, etc.
>
> with UNICODE_INTERNAL i've encountered the dreaded
> "iso2022-preservation" problem -- i.e. byte-compiling an iso2022-encoded
> lisp file is unlikely to work without further hacking. converting them
> to utf-8 is not an option since in many cases the lisp code depends on
> having the proper charset encoded in the actual chars in the file, at
> least in an old-Mule world. so i'm [a] trying to remove this
> assumption,
It's not _that_ widespread, since non-Mule XEmacs wouldn't honour it, as I
understand it.
btw "remove this assumption" means using calls to (make-char ...) instead of
literal characters.
[b] putting in hacks to make byte-compilation iso2022-preserving,
using
> private unicode chars.
There also arises the problem of what to do when you have Unicode-encoded
source code with characters that don't have a mapping in Mule. For
example,
I'm studying a language of the former Soviet Union which uses more
Cyrillic
characters than are available in ISO 8859-5, and my local UTF-code-points
preserving code[1] will not allow my hacked input method to be reliably
byte-compiled. The byte-compiled code uses escape-quoted as its coding
system, which, although it is in general a universal coding system for
Mule,
breaks down if you have just-in-time allocation like the implementation
linked below does, as different invocations will have different internal
Unicode code point mappings.
I suppose there are three cases we care about --
-- The source code sticks to ASCII. In this case, don't specify a coding
system in the compiled file, and we don't have a problem.
-- The source code uses ISO 2022 escape sequences. In this case, we should
specify escape-quoted as the coding system in the compiled file, because
ISO
2022 encodes more characters than Unicode does, because we want the
assumptions about character sets mentioned by you above to be preserved,
and
because it's more compatbile with old implementations of Mule.
I consider files with the existing eight-bit coding systems a subset of
this case, because that behaviour is more compatible with old Mule.
-- The source code uses a Unicode coding system. In this case, specify
utf-8
as the coding system in the compiled file. This will be incompatible with
older implementations of Mule; but hopefully with the advantage that they
will choke obviously, and that they will preserve non-Mule characters
stored
in the source file.
[1] http://list-archive.xemacs.org/xemacs-patches/200506/msg00040.html
so i've also created some unicode charsets for old-mule; these cover
everything up through 0x2FFFF and should take care of your problem to some
extent. (i rearranged the handling of leading bytes so any private leading
byte can be for charsets of any size, so less issues with running out of
charset space.) if i assign appropriate ISO2022 final bytes, or implement
the ISO2022 extension mechanism, these can be reliably stored in
byte-compiled code. (but this problem only arises if the source .el file is
in utf-8, *and* we use escape-quoted for the .elc file, *and* you have
literal characters embedded in your .el file. are these all true?) the use
of the extension mechanism is more standard and less hackish but with the
former method, current 21.5 versions of xemacs can at least load the files
without puking, since they create unknown iso charsets on the fly. probably
we don't really care about such compatibility in any case.
9:29 p.m.
Ar an cúigiú lá déag de mí na Samhain, scríobh Ben Wing:
> It's not _that_ widespread, since non-Mule XEmacs
wouldn't honour it, as I
> understand it.
btw "remove this assumption" means using calls to (make-char ...) instead of
literal characters.
Note that doing that to, say, the Quail input methods (which are, of course,
not in core) will make them much less maintainable. Not that they’re
especially maintained as is.
[...] so i've also created some unicode charsets for old-mule;
these cover
everything up through 0x2FFFF and should take care of your problem to some
extent. (i rearranged the handling of leading bytes so any private leading
byte can be for charsets of any size, so less issues with running out of
charset space.) if i assign appropriate ISO2022 final bytes, or implement
the ISO2022 extension mechanism, these can be reliably stored in
byte-compiled code.
Thanks!
(but this problem only arises if the source .el file is in utf-8,
*and*
we use escape-quoted for the .elc file, *and* you have literal characters
embedded in your .el file. are these all true?)
Yes. (It’s an input method, it makes it a lot more maintainable when you can
see what a sequence of characters maps to.)
Auto-save files use escape-quoted for much the same reason as does the byte
compiler--because it was a universal coding system for Mule--and will have
encountered the same problem.
Lisp hackers; how do I evaluate some code only at byte-compile time? The
below assertion fails both at compile time and run time.
(eval-when-compile
(assert (not t)
t
"This file should not be compiled;
The UTF-8 to escape-quoted mapping is not stable from one XEmacs invocation
to the next, in my JIT-UTF-8 implementation. Since byte-compiled files use
escape-quoted, this will break."))
the use of the extension mechanism is more standard and less hackish
but
with the former method, current 21.5 versions of xemacs can at least load
the files without puking, since they create unknown iso charsets on the
fly. probably we don't really care about such compatibility in any case.
--
“I, alone, perhaps, in this city of nearly two million, view it with sadness,
sympathy, and respect, seeing in the millions of Russian youngsters who laid
down their lives in that war a tragedy rising above all the political
emotions of that time [...]” -- George Kennan on Vienna's „Erbsenkönig“
9:54 a.m.
Aidan Kehoe wrote:
Ar an cúigiú lá déag de mí na Samhain, scríobh Ben Wing:
> > It's not _that_ widespread, since non-Mule XEmacs wouldn't honour it,
as I
> > understand it.
>
> btw "remove this assumption" means using calls to (make-char ...) instead
of
> literal characters.
Note that doing that to, say, the Quail input methods (which are, of course,
not in core) will make them much less maintainable. Not that they’re
especially maintained as is.
well, what i've actually done when i changed things is to leave the
original characters inside of comments.
btw in some cases switching to make-char makes things more maintainable
[at least for me] ... e.g. i have no visual clue that such-and-such
character is meant to be latin-3 rather than latin-1. also, in the
weirder charsets, e.g. indian-1-column, all i see are ~'s anyway.
> [...] so i've also created some unicode charsets for
old-mule; these cover
> everything up through 0x2FFFF and should take care of your problem to some
> extent. (i rearranged the handling of leading bytes so any private leading
> byte can be for charsets of any size, so less issues with running out of
> charset space.) if i assign appropriate ISO2022 final bytes, or implement
> the ISO2022 extension mechanism, these can be reliably stored in
> byte-compiled code.
Thanks!
> (but this problem only arises if the source .el file is in utf-8, *and*
> we use escape-quoted for the .elc file, *and* you have literal characters
> embedded in your .el file. are these all true?)
Yes. (It’s an input method, it makes it a lot more maintainable when you can
see what a sequence of characters maps to.)
Auto-save files use escape-quoted for much the same reason as does the byte
compiler--because it was a universal coding system for Mule--and will have
encountered the same problem.
hmm. i actually invented escape-quoted and put things in for the same
reason you're citing, but i'd forgotten about that. i think i'll go
ahead and use the extension mechanism.
Lisp hackers; how do I evaluate some code only at byte-compile time?
The
below assertion fails both at compile time and run time.
(eval-when-compile
(assert (not t)
t
"This file should not be compiled;
The UTF-8 to escape-quoted mapping is not stable from one XEmacs invocation
to the next, in my JIT-UTF-8 implementation. Since byte-compiled files use
escape-quoted, this will break."))
> the use of the extension mechanism is more standard and less hackish but
> with the former method, current 21.5 versions of xemacs can at least load
> the files without puking, since they create unknown iso charsets on the
> fly. probably we don't really care about such compatibility in any case.
6945
days inactive
6948
days old
4 comments
3 participants
participants (3)
-
Aidan Kehoe -
Ben Wing -
Ben Wing