2:27 p.m.
Ar an ceathrú lá is fiche de mí na Nollaig, scríobh Stephen J. Turnbull:
So after dealing with the test all characters issue[1] I noticed
that
charsets-in-region and charsets-in-string were failing. I suspect the
reason is that in the Unicode build the precedence lists are not tuned
right. I guess a real fix would need to be some intelligent approach
to modifying the precedence list according to the charsets used at
read time or input (although that won't work if the input is
Unicode). The other problem is that "intelligent" really means
constructing some kind of precedence graph, and it could easily be
impossible (eg if both Chinese and Japanese were used in the same
file, you'd need a language tag to disambiguate).
I guess the first thing I'd try is ensuring that ISO charsets come
before the windows-xxx and IBM CPxxx versions.
Any other suggestions?
We could disable the charsets-in-region tests on the Unicode builds. Or have
a defined order we expect depending on the current language environment; but
to be honest, with unicode-internal, the output of charsets-in-region isn’t
something the user is going to care about, I would lean more towards the
former.
Footnotes:
[1] By the way, why does the Unicode build have a 2^30 repertoire?
ISO 10646 has a 2^31 repertoire IIRC (maybe 2^32?), but Unicode only
has 2^21 (precisely, 2^20+2^16).
Ben’s choice. We do need more than 2^21, for our invalid sequence
characters, but we certainly don’t need the full 2^30.
--
‘As I sat looking up at the Guinness ad, I could never figure out /
How your man stayed up on the surfboard after forty pints of stout’
(C. Moore)
4:05 p.m.
New subject: Failures in "test charset-in-* functions" in mule-tests.el
Aidan Kehoe writes:
We could disable the charsets-in-region tests on the Unicode builds.
Or have
a defined order we expect depending on the current language environment; but
to be honest, with unicode-internal, the output of charsets-in-region isn’t
something the user is going to care about, I would lean more towards the
former.
Users like me who use diff on ISO-2022 (or windows-xxxx for that
matter) care about getting the same charsets out that they put in. If
we get that right, charsets-in-* will follow.
Ben’s choice. We do need more than 2^21, for our invalid sequence
characters, but we certainly don’t need the full 2^30.
Um, 2^21 leaves us with 15 non-Unicode planes (17-31). Surely that's
enough? Am I missing something? There's also the Python PEP 283
strategy (use singleton low surrogates), which allows you to live
within the 17 Unicode planes.
Steve
11:05 a.m.
New subject: Failures in "test charset-in-* functions" in mule-tests.el
Ar an cúigiú lá is fiche de mí na Nollaig, scríobh Stephen J. Turnbull:
Aidan Kehoe writes:
> We could disable the charsets-in-region tests on the Unicode builds.
> Or have a defined order we expect depending on the current language
> environment; but to be honest, with unicode-internal, the output of
> charsets-in-region isn’t something the user is going to care about, I
> would lean more towards the former.
Users like me who use diff on ISO-2022 (or windows-xxxx for that
matter) care about getting the same charsets out that they put in. If
we get that right, charsets-in-* will follow.
Well, then, we should test the charsets assigned by the iso-2022 coding
systems.
The coding systems should pass through extent information. GB2312, ISO-2022,
SHIFT_JIS and so forth should tag their non-ASCII as zh_CN, ja, zh_TW and so
forth, using extents, and then use that information when re-encoding. This
would be also helpful for the ECMA-48 support that GNU provides with
ansi-color.el, which doesn’t work on XEmacs currently. I’m not volunteering
for this anytime soon.
I was never interested in a proliferation of internal character sets, I
needed to use latin-unity for too long for that. My fixed-width coding
systems decoded from external data to existing internal characters, or the
jit-unicode character sets; Ben decided to replace them by creating internal
character sets for all the corresponding coding systems, including the
Windows code pages. I note he added a `charset-iso-2022-p’ predicate; I hope
the ISO-2022 coding systems take note of this and don’t use the Windows code
pages, but I haven’t checked.
> Ben’s choice. We do need more than 2^21, for our invalid
sequence
> characters, but we certainly don’t need the full 2^30.
Um, 2^21 leaves us with 15 non-Unicode planes (17-31). Surely that's
enough? Am I missing something? There's also the Python PEP 283
strategy (use singleton low surrogates), which allows you to live
within the 17 Unicode planes.
We need internal characters that have no standardly-defined external
representation. ‘Standardly-defined external representation with no
associated semantic definition’ is no use, as you and I see on the Mac; it’s
routine to encounter code points in the Unicode private use area in macOS
data, and if they conflict with characters we have assigned, that will give
bugs. ?\uF8FF should not suddenly be encoded as, e.g. a single-byte ÿ in a
UTF-8 file.
--
‘As I sat looking up at the Guinness ad, I could never figure out /
How your man stayed up on the surfboard after forty pints of stout’
(C. Moore)
8:22 a.m.
New subject: Failures in "test charset-in-* functions" in mule-tests.el
Aidan Kehoe writes:
I was never interested in a proliferation of internal character
sets, needed to use latin-unity for too long for that. My
fixed-width coding systems decoded from external data to existing
internal characters, or the jit-unicode character sets; Ben decided
to replace them by creating internal
I missed that. That must have been in his Unicode branch?
I'm pretty sure Ben was thinking about the reproducibility problem:
garbage in, *identical* garbage out. More generally, if the editor
doesn't touch a range in the buffer, it should be encoded exactly as
in the input file unless the user has requested a transcoding. This
is pretty important in stuff like mail, where the ISO 2022 encodings
are still in pretty common use in Japan, and I think in Korea as
well. I think I understand your frustration with latin-unity enough
to say this is just as important to me.
I don't think it's worth dealing with stuff like designation sequences
immediately followed by another, etc. -- it will never be perfect
unless you compare to a copy of the original. So I'm up for just
fiddling with the algorithm used by the charsets-in-* functions until
the tests pass with HELLO, and deal with any reproducibility issues
that still remain as they show up.
I assume that is OK with you, but tell me I'm wrong if not. Do you
see any problem likely to arise with that kind of algorithm fiddling?
Any suggestions would be appreciated for reference while fiddling
(granted, I don't see any that aren't just as bad in the current
situation).
We need internal characters that have no standardly-defined external
representation.
My question is do we need more than 983040 such? Unicode uses all
0xyyyy codepoints, but of the 1xyyyy code points it only uses 10yyyy.
So we're free to use 110000-1fffff and still stay in 21 bits, in which
case the #x200000 constraint on mule-tests.el's #'test covers the
entire internal repertoire. And the test for Unicode code points vs
such "internal characters" is cheap: `ch < 110000h'.
2523
days inactive
2525
days old
3 comments
2 participants
participants (2)
-
Aidan Kehoe -
Stephen J. Turnbull