10:37 a.m.
Glynn Clements <glynn(a)sensei.co.uk> writes:
I installed eterm-1.04, and noticed that the CR characters are being
eaten without eterm actually doing a carriage return, resulting in the
`staircase effect'.
Verified. Note that the newest version of eterm has ANSI color
support (color ls works, although the effect is lost in the ugliness
of the staircase).
Using (setq process-coding-system-alist '((".*" .
binary))) fixed the
problem,
Does it? What happens if you cat a Japanese text file? I see something
highly ugly and definitely not Japanese.
but it smells as if eterm is assuming that the coding system is
appropriate rather than making it so.
Yup.
Is this MULE-related?
I'm pretty sure. As far as I know, eterm has never been MULE-ized.
Could someone knowledgeable please do up an appropriate patch?
Did anyone actually test eterm-1.04 on a MULE XEmacs?
Is this a rhetorical question? :-)
2:35 p.m.
New subject: coding-system lossage with eterm
>>>> Glynn Clements <glynn(a)sensei.co.uk> writes:
> but it smells as if eterm is assuming that the coding system is
> appropriate rather than making it so.
Define "appropriate." For example, today trying to figure out what
was going on in my undergraduate college's Web site, I saw English,
German, EUC, Shift-JIS, ISO-2022-JP, and Big5 (which is Chinese, for
those who tuned in late). I just verified that there is a sequence of
4 links which hits all of those last four in turn. DEC OSF/1's less
in a kterm got the English and the three flavors of Japanese right,
but didn't get the German or Chinese. At least with file input, you
can assume that the character coding will be consistent throughout,
and the buffer is all safely Mule internal encoding.
You can't make that assumption here. This is going to require user
intervention, even if LANG is set properly. Even if you can be pretty
sure of the OS messages, who knows what cat is going to spew?
It would be nice if you could screw up the coding system first time
through and go back and fix what's in the buffer, too. I've never
figured out how to do that in Mule, though. It should be possible, I
think, although you may have to use some of the "private" functions.
> Is this MULE-related?
Or the recent surgical separation of file-encoding from Mule may have
exposed the underlying problem.
4:43 p.m.
New subject: coding-system lossage with eterm
SL Baur wrote:
> I installed eterm-1.04, and noticed that the CR characters are
being
> eaten without eterm actually doing a carriage return, resulting in the
> `staircase effect'.
Verified. Note that the newest version of eterm has ANSI color
support (color ls works, although the effect is lost in the ugliness
of the staircase).
Yep; that was the main reason for upgrading. BTW, I tried reverting to
the previous version (1.02), but it has the same problem, so it's due
something that's changed in XEmacs rather than than in eterm.
> Using (setq process-coding-system-alist '((".*" .
binary))) fixed the
> problem,
Does it? What happens if you cat a Japanese text file? I see something
highly ugly and definitely not Japanese.
I see pretty much the same as if I cat a Japanese text file in an
xterm.
> but it smells as if eterm is assuming that the coding system is
> appropriate rather than making it so.
Yup.
> Is this MULE-related?
I'm pretty sure. As far as I know, eterm has never been MULE-ized.
Could someone knowledgeable please do up an appropriate patch?
Actually, I suspect that it's more likely coding-system related. I've
built with MULE for about as long as I've been running beta versions
(although my knowledge of languages other than English is roughly
nil), and eterm used to work OK.
Any suggestions as to what coding system it should set? Or should it
be a defcustom? In which case what would the default be?
--
Glynn Clements <glynn(a)sensei.co.uk>
7:12 p.m.
New subject: coding-system lossage with eterm
>>>> "Glynn" == Glynn Clements
<glynn(a)sensei.co.uk> writes:
Glynn> SL Baur wrote:
> > I installed eterm-1.04, and noticed that the CR characters
> are being > eaten without eterm actually doing a carriage
> return, resulting in the > `staircase effect'.
>
> Verified. Note that the newest version of eterm has ANSI color
> support (color ls works, although the effect is lost in the
> ugliness of the staircase).
I get the same result in eterm 1.01.
A little bit of experimentation shows that setting the process coding
system to 'iso-2022-jp gives nice Japanese, with the same ol' ugly
staircase. 'undecided-unix gives nice left-justified moji-bake, and
'iso-2022-unix gives left-justified Japanese. 'undecided-dos gives
staircase.
Glynn> Actually, I suspect that it's more likely coding-system
Glynn> related.
It's hard to see how Mule could guess in advance that it's going to be
seeing EUC-JP vs some other 8-bit ISO-2022-compatible encoding. So
it's not surprising it gets the Japanese wrong. What seems to have
happened is that for the process coding system we're defaulting to
*-dos, while the file coding systems are defaulted (usually correctly)
by the build environment. I would guess somebody just overlooked the
process coding system in the coding system overhaul.
Glynn> Any suggestions as to what coding system it should set? Or
Glynn> should it be a defcustom? In which case what would the
Glynn> default be?
The default should be a defcustom, probably defaulted to ISO-8859-1.
When possible, the coding system should be set according to LANG.
(Steve, were you running with LANG=ja or not? I gotta get some sleep,
so I'm not going to experiment further....)
8:56 p.m.
New subject: coding-system lossage with eterm
Stephen J. Turnbull wrote:
A little bit of experimentation shows that setting the process
coding
system to 'iso-2022-jp gives nice Japanese, with the same ol' ugly
staircase. 'undecided-unix gives nice left-justified moji-bake, and
'iso-2022-unix gives left-justified Japanese. 'undecided-dos gives
staircase.
Previously, I got the same results for undecided-unix (left-justified
mojibake), which I though was odd. However, I've tried it again and
now undecided-unix gives me left-justified Japanese, which is what I
would expect.
Glynn> Actually, I suspect that it's more likely
coding-system
Glynn> related.
It's hard to see how Mule could guess in advance that it's going to be
seeing EUC-JP vs some other 8-bit ISO-2022-compatible encoding. So
it's not surprising it gets the Japanese wrong. What seems to have
happened is that for the process coding system we're defaulting to
*-dos, while the file coding systems are defaulted (usually correctly)
by the build environment. I would guess somebody just overlooked the
process coding system in the coding system overhaul.
Starting eterm and evaluating:
(process-coding-system (get-buffer-process (current-buffer)))
produces the result:
(#<coding_system undecided> . #<coding_system binary>)
Presumably the ^M results in the coding system being assumed to be
<something>-dos.
Glynn> Any suggestions as to what coding system it should set?
Or
Glynn> should it be a defcustom? In which case what would the
Glynn> default be?
The default should be a defcustom, probably defaulted to ISO-8859-1.
When possible, the coding system should be set according to LANG.
(Steve, were you running with LANG=ja or not? I gotta get some sleep,
so I'm not going to experiment further....)
I would guess that undecided-unix would be the right thing, _if_ it
worked consistently.
Could people report what results they get from the following:
M-: (setq process-coding-system-alist '((".*" . undecided-unix))) RET
M-x term RET RET
cat /usr/local/lib/xemacs/packages/etc/mule-doc/ISO2022.jp
C-c C-j
M-: (process-coding-system (get-buffer-process (current-buffer))) RET
--
Glynn Clements <glynn(a)sensei.co.uk>
10:44 p.m.
New subject: coding-system lossage with eterm
Some time ago, Glynn Clements wrote...
|+
| Starting eterm and evaluating:
|
| (process-coding-system (get-buffer-process (current-buffer)))
|
| produces the result:
|
| (#<coding_system undecided> . #<coding_system binary>)
|
| Presumably the ^M results in the coding system being assumed to be
| <something>-dos.
Absolutely correct. I think the right tests would be
1. Start term and cat ISO2022.jp
2. Start term, first do ls, then cat ISO2022.jp
I suspect that detection may fail after it sees any English before Japanese.
Kirill
1:01 a.m.
New subject: coding-system lossage with eterm
>>>> "Glynn" == Glynn Clements
<glynn(a)sensei.co.uk> writes:
Glynn> Previously, I got the same results for undecided-unix
Glynn> (left-justified mojibake), which I though was odd. However,
Glynn> I've tried it again and now undecided-unix gives me
Glynn> left-justified Japanese, which is what I would expect.
Mule is not deterministic.... Was it the same file? I was cat'ing
the same file every time, which just happened to be HTML code. So
Mule saw several lines of ASCII before seeing any Japanese. I bet
that is enough to defeat the heuristic. No that's wrong... see below.
Glynn> Presumably the ^M results in the coding system being
Glynn> assumed to be <something>-dos.
*smack* Of course. I'm not used to thinking about the CR there....
Glynn> I would guess that undecided-unix would be the right thing,
Glynn> _if_ it worked consistently.
I don't think it can; it depends on how smart Mule is about making
decisions.
_If_ the HTML file is cat'ted first into a new terminal buffer, I get
Japanese staircase. _If_ I do say `pwd' first, I get mojibake
staircase. With 'undecided-dos as default I get left-justified, same
results.
If it were possible to get Mule to make decisions incrementally, eg
undecided -> (see ^M) undecided-dos (continue several commands) ->
(see ESC $ B) -> ISO-2022-7, it could be pretty reliable. But it
looks like Mule makes its decision on the first command. So it's
going to depend on usage patterns, and if we want users to be happy
we're going to need to come up with some smarter heuristics.
10:53 a.m.
New subject: coding-system lossage with eterm
Stephen J. Turnbull wrote:
Glynn> I would guess that undecided-unix would be the right
thing,
Glynn> _if_ it worked consistently.
I don't think it can; it depends on how smart Mule is about making
decisions.
_If_ the HTML file is cat'ted first into a new terminal buffer, I get
Japanese staircase. _If_ I do say `pwd' first, I get mojibake
staircase. With 'undecided-dos as default I get left-justified, same
results.
If it were possible to get Mule to make decisions incrementally, eg
undecided -> (see ^M) undecided-dos (continue several commands) ->
(see ESC $ B) -> ISO-2022-7, it could be pretty reliable. But it
looks like Mule makes its decision on the first command. So it's
going to depend on usage patterns, and if we want users to be happy
we're going to need to come up with some smarter heuristics.
Hmm, it seems to be even less deterministic than that. Today I just
get mojibake regardless of whether I cat ISO2022.jp straight away.
So, I guess the only solution is to have an alist that maps LANG
values to coding systems, and set the coding system according to LANG.
--
Glynn Clements <glynn(a)sensei.co.uk>
7:41 p.m.
New subject: coding-system lossage with eterm
Some time ago, Glynn Clements wrote...
|+
| Any suggestions as to what coding system it should set? Or should it
| be a defcustom? In which case what would the default be?
|-
AFA I understand codind systems, there are coding part and eol converison
part. Perhaps coding must be set to 'undecided-unix, or to one of a specific
type, but with unix eol style forces (so ^M^N does not get converted to ^N
alone), for exmaple
(make-coding-system 'iso2022-lf 'undecided
"iso2022 coding with forced lf line endings"
'(eol-type lf))
I'm just guessing.
Kirill
9709
days inactive
9710
days old
8 comments
4 participants
participants (4)
-
Glynn Clements -
Kirill M. Katsnelson -
SL Baur -
Stephen J. Turnbull