3:49 p.m.
Hi - long-term XEmacs/VM user here. I've been seeing this problem for
a while but am reporting it for the first time now (sorry if this is a
FAQ, I did a quick web search but did not find an answer to this one).
Every so often, email message text is cut off with a message like this:
iconv: illegal input sequence at position 161
Looking in the raw INBOX file the offending character is a Unicode
"bullet", composed of bytes 226, 128, 162.
(It's coming from the string "sent from iphone")
The bullet displays correctly in a terminal, but if I copy/paste it to
an XEmacs window, it becomes a double-quote
Version info:
ELISP> (vm-version)
"8.0.12-devo-585"
ELISP> (emacs-version)
"XEmacs 21.5 (beta31) \"ginger\" [Lucid] (x86_64-pc-linux) of Wed Jan
25 2012 on x201"
cgw＠x201~$ iconv --version
iconv (GNU libc) 2.14.1
_______________________________________________
XEmacs-Beta mailing list
XEmacs-Beta(a)xemacs.org
http://lists.xemacs.org/mailman/listinfo/xemacs-beta
12:35 a.m.
On Wed, Mar 21, 2012 at 4:49 PM, Charles G Waldman <cgw(a)alum.mit.edu> wrote:
Hi - long-term XEmacs/VM user here. I've been seeing this problem for
a while but am reporting it for the first time now (sorry if this is a
FAQ, I did a quick web search but did not find an answer to this one).
Every so often, email message text is cut off with a message like this:
iconv: illegal input sequence at position 161
This would appear to be a VM issue, as XEmacs itself doesn't use iconv
anywhere that I know of, although if you just happen to be on AIX 3.2.2
it links to the iconv library for the benefit of X11R5. (Yes, "5". :-)
The bullet displays correctly in a terminal, but if I copy/paste it
to
an XEmacs window, it becomes a double-quote
I don't know what's going on here, but I assume this is an XEmacs
bug. I'll look into it.
_______________________________________________
XEmacs-Beta mailing list
XEmacs-Beta(a)xemacs.org
http://lists.xemacs.org/mailman/listinfo/xemacs-beta
7:27 a.m.
On 2012-03-23, Stephen J. Turnbull <stephen(a)xemacs.org> wrote:
On Wed, Mar 21, 2012 at 4:49 PM, Charles G Waldman
<cgw(a)alum.mit.edu> wrote:
> iconv: illegal input sequence at position 161
This would appear to be a VM issue, as XEmacs itself doesn't use iconv
VM doesn't use iconv, unless you tell it to.
Charles - check with no .vm or .emacs .
_______________________________________________
XEmacs-Beta mailing list
XEmacs-Beta(a)xemacs.org
http://lists.xemacs.org/mailman/listinfo/xemacs-beta
4:59 p.m.
Julian Bradfield writes:
VM doesn't use iconv, unless you tell it to.
Charles - check with no .vm or .emacs .
Thanks Julian!
I am now eligible for one "Big Dummy" award :-)
I've got some vm-related stuff in my custom.el which I don't
completely understand ... cargo-culted from somewhere ... I
don't understand Unicode as well as I ought to, perhaps this
is an opportunity to learn a few things.
I've got vm-mime-charset-converter-alist set to
(("utf-8" "iso-8859-1" "iconv -f utf-8 -t iso-8859-1"))
and vm-mime-default-face-charsets set to
("us-ascii" "iso-8859-1" "iso-8859-2"
"iso-8859-15" "windows-1252" "cp932"
"ISO-8859-7")
Setting vm-mime-charset-converter-alist to nil (the default),
the messages in question now show up as attachments: in VM I
get a line like this:
(icon) plain text, utf-8 [Click mouse-2 to save to a file]
If I right-click and do 'display as text in default face',
with vm-mime-default-face-charsets set to ("us-ascii", "iso-8859-1")
(the default), the attachment displays without the "iconv" warning,
but the Unicode chars do not display correctly, as I wrote earlier to
Stephen offlist - it shows up like this:
â¢sent from iphoneâ¢
In case that gets mangled in transmission: the "bullet" char is
showing up as lower-case-curcumflex-a, backslash-200, and the "cents"
sign. If I copy/paste that string to a terminal (with LANG=en_US.UTF-8)
the backslash-200 turns into a Euro symbol.
So, I guess my questions are:
1) Should the bullet char (composed of bytes 226, 128, 162) display
in XEmacs? Is this font-dependent?
2) What -fu do I need to put into my startup files to get VM to
display my messages inline, rather than attachments?
Thanks again for any and all help/suggestions!
- Charles
_______________________________________________
XEmacs-Beta mailing list
XEmacs-Beta(a)xemacs.org
http://lists.xemacs.org/mailman/listinfo/xemacs-beta
5:42 p.m.
New subject: PS: Re: "iconv: illegal input sequence" in VM
One more detail:
feeding my INBOX file to iconv in a terminal like this:
$ iconv -f utf-8 -t iso-8859-1 ~/Mail/INBOX
also results in the:
"iconv: illegal input sequence at position xxxx"
so there's really some invalid characters in the input, I'll have to
try to understand how they are getting in there (again, this seems to
happen with mail that people are sending from IPhones). The actual
bytes are: e2 80 a2
I can add the "-c" flag to iconv, which just throws the offending
characters away (I see "sent from iconv" in the message now, just
without the bullet chars).
Some more stuff that would probably make sense if I understood iconv
better: If I create a test file with just the offending sequence:
$ hexdump -C /tmp/test
00000000 e2 80 a2 74 65 73 74 0a 0a |...test..|
and feed this to iconv in a terminal, I get the following:
cgw＠x201~$ iconv /tmp/test
"test
cgw＠x201~$ iconv -f utf8 /tmp/test
"test
cgw＠x201~$ iconv -f utf8 -t iso8859-1 /tmp/test
iconv: illegal input sequence at position 0
(the double-quote before "test" is displaying correclty as a bullet in
the terminal,but turns into a double-quote when I copy/paste to
XEmacs).
What's going on here? So many questions :-) Why does copy/paste
turn a bullet into a double-quote, and why does iconv fail when I
specify iso8859-1 as target codeset?
Thanks again,
-Charles
Julian Bradfield writes:
On 2012-03-23, Stephen J. Turnbull <stephen(a)xemacs.org> wrote:
> On Wed, Mar 21, 2012 at 4:49 PM, Charles G Waldman <cgw(a)alum.mit.edu> wrote:
>> iconv: illegal input sequence at position 161
>
> This would appear to be a VM issue, as XEmacs itself doesn't use iconv
VM doesn't use iconv, unless you tell it to.
Charles - check with no .vm or .emacs .
_______________________________________________
XEmacs-Beta mailing list
XEmacs-Beta(a)xemacs.org
http://lists.xemacs.org/mailman/listinfo/xemacs-beta
6:08 a.m.
New subject: PS: Re: "iconv: illegal input sequence" in VM
On Sat, Mar 24, 2012 at 2:42 AM, Charles G Waldman <cgw(a)alum.mit.edu> wrote:
$ iconv -f utf-8 -t iso-8859-1 ~/Mail/INBOX
also results in the:
"iconv: illegal input sequence at position xxxx"
so there's really some invalid characters in the input, I'll have to
try to understand how they are getting in there (again, this seems to
happen with mail that people are sending from IPhones). The actual
bytes are: e2 80 a2
Sure. That's U+2022 (if I've done the math right, but it's close
enough:-), obviously out of the ISO 8859/1 range. That is,
the problem is not that it's illegal UTF-8. It's that it is illegal
to convert it to ISO 8859/1.
(the double-quote before "test" is displaying correclty as
a bullet in
the terminal,but turns into a double-quote when I copy/paste to
XEmacs).
This I don't yet understand, I'll try to look at it in the next
couple of days.
_______________________________________________
XEmacs-Beta mailing list
XEmacs-Beta(a)xemacs.org
http://lists.xemacs.org/mailman/listinfo/xemacs-beta
6:12 a.m.
On Sat, Mar 24, 2012 at 1:59 AM, Charles G Waldman <cgw(a)alum.mit.edu> wrote:
1) Should the bullet char (composed of bytes 226, 128, 162) display
in XEmacs? Is this font-dependent?
It should, but AFAIK it won't. Aidan did some work to ensure
that Unicode characters get into the buffer, but they don't
display correctly on X11 or in a terminal AFAICT. Maybe
that's just me. :-)
_______________________________________________
XEmacs-Beta mailing list
XEmacs-Beta(a)xemacs.org
http://lists.xemacs.org/mailman/listinfo/xemacs-beta
6:58 p.m.
Ar an tríú lá is fiche de mí Márta, scríobh Charles G Waldman:
[...]
So, I guess my questions are:
1) Should the bullet char (composed of bytes 226, 128, 162) display
in XEmacs? Is this font-dependent?
It should, absolutely, and will do, once VM is able to handle the encoding.
2) What -fu do I need to put into my startup files to get VM to
display my messages inline, rather than attachments?
Thanks again for any and all help/suggestions!
Here are the relevant lines from my .vm. The problem is, essentially, that
vm-mime-mule-charset-to-coding-alist needs a more useful initial value; this
hasn’t yet been fixed in trunk VM.
(pushnew '("windows-1252" windows-1252)
vm-mime-mule-charset-to-coding-alist
:test 'equal)
(pushnew '("windows-1251" windows-1251)
vm-mime-mule-charset-to-coding-alist
:test 'equal)
(pushnew '("windows-1250" windows-1250)
vm-mime-mule-charset-to-coding-alist
:test 'equal)
(pushnew '("gb2312" cn-gb-2312) vm-mime-mule-charset-to-coding-alist
:test 'equal)
(pushnew '("utf-8" utf-8) vm-mime-mule-charset-to-coding-alist
:test 'equal)
(pushnew '("shift_jis" shift_jis) vm-mime-mule-charset-to-coding-alist
:test 'equal)
(pushnew '("shift-jis" shift_jis) vm-mime-mule-charset-to-coding-alist
:test 'equal)
(pushnew '("ascii" no-conversion) vm-mime-mule-charset-to-coding-alist
:test 'equal)
(pushnew '("iso8859-15" iso-8859-15) vm-mime-mule-charset-to-coding-alist
:test 'equal)
(pushnew '("cp1252" windows-1252) vm-mime-mule-charset-to-coding-alist
:test 'equal)
(unless (assoc "utf8" vm-mime-mule-charset-to-coding-alist)
(nconc
vm-mime-mule-charset-to-coding-alist
;; Needs to be at the end, otherwise we get the wrong MIME name when
;; sending.
'(("utf8" utf-8))))
--
‘Iodine deficiency was endemic in parts of the UK until, through what has been
described as “an unplanned and accidental public health triumph”, iodine was
added to cattle feed to improve milk production in the 1930s.’
(EN Pearce, Lancet, June 2011)
_______________________________________________
XEmacs-Beta mailing list
XEmacs-Beta(a)xemacs.org
http://lists.xemacs.org/mailman/listinfo/xemacs-beta
12:06 p.m.
Ar an seachtú lá is fiche de mí Márta, scríobh Stephen J. Turnbull:
On Sat, Mar 24, 2012 at 1:59 AM, Charles G Waldman
<cgw(a)alum.mit.edu> wrote:
> 1) Should the bullet char (composed of bytes 226, 128, 162) display
> in XEmacs? Is this font-dependent?
It should, but AFAIK it won't. Aidan did some work to ensure
that Unicode characters get into the buffer, but they don't
display correctly on X11 or in a terminal AFAICT. Maybe
that's just me. :-)
Many of them (in particular, very common characters like this one) display
fine on X11 and in a terminal (make sure your language environment reflects
the encoding of the terminal, or set the terminal encoding explicitly).
They will have trouble under XFT, which Charles appears to be using, and the
long-term solution to this is to merge Jeff Sparke’s GTK2 support, and for
me to fix some poor architectural decisions about what level textual runs
should be split when it comes to character coverage. Here’s some code to
test:
(decode-coding-string (map 'string #'int-char '(226 128 162)) 'utf-8)
=> "•"
--
‘Iodine deficiency was endemic in parts of the UK until, through what has been
described as “an unplanned and accidental public health triumph”, iodine was
added to cattle feed to improve milk production in the 1930s.’
(EN Pearce, Lancet, June 2011)
_______________________________________________
XEmacs-Beta mailing list
XEmacs-Beta(a)xemacs.org
http://lists.xemacs.org/mailman/listinfo/xemacs-beta
4410
days inactive
4419
days old
8 comments
5 participants
participants (5)
-
Aidan Kehoe -
Charles Waldman -
Julian Bradfield -
Stephen J. Turnbull -
Stephen J. Turnbull