1:25 a.m.
================================================================
Dear Bug Team!
I get lots of mail with RFC2047-encoded From: fields (I use
Gnus). Mostly ISO-8859-2. Now, when mail-extr.el
(mail-extract-address-components) gets to those headers, they are
already in decoded form, which causes mail-extr to fail. I get chopped
names, annoying BBDB behavior, and all sorts of breakage. As an example,
if sender's name contains an 'ł', mail-extract-address-components will
return his name only up to (not containing) this character.
People who get ISO-8859-1 E-mail probably don't see this, because
mail-extr.el is "extended" to handle ISO-8859-1 in a particularly ugly
way.
I've reported this on the Gnus mailing list and got this reply:
From: Dave Love <d.love(a)dl.ac.uk>
Subject: Re: something decodes my headers...
To: Jan Rychter <jan(a)rychter.com>
Cc: ding(a)gnus.org
Date: Thu, 12 Jun 2003 23:38:19 +0100
Jan Rychter <jan(a)rychter.com> writes:
People who get
ISO-8859-1 E-mail don't see this, because mail-extr.el is "extended" to
handle ISO-8859-1 in a particularly ugly way.
That must be XEmacs-specific. mail-extr in Emacs (at least Emacs 21)
works with multilingual text.
I've tried to trace what it is that decodes the headers to 8bit
before
mail-extr sees them -- but I have no idea.
I think that's the wrong thing to look at. mail-extr should just cope
with multilingual names. [If it's really `8-bit' I assume it's a
non-Mule XEmacs, so you can't deal with multilingual text anyhow.]
And indeed, the Emacs version of mail-extr.el seems to use [:alnum:] and
[:alpha:]:
(defconst mail-extr-all-letters-but-separators
(purecopy "][[:alnum:]{|}'~`"))
instead of (in the XEmacs version):
(defconst mail-extr-all-letters-but-separators
(purecopy "][A-Za-z{|}'~0-9`\200-\376")) ;; XEmacs
Now, opinions are divided on whether mail-extract-address-components
should really get multilingual text or not. In any case, the current
solution is rather broken.
How can this be fixed?
--J.
================================================================
System Info to help track down your bug:
---------------------------------------
uname -a: Linux tnuctip.rychter.com 2.4.21-rc1 #5 Sun Apr 27 20:45:38 CEST 2003 i686 i686
i386 GNU/Linux
./configure '--with-mule' '--error-checking=none'
'--with-menubars=lucid' '--with-scrollbars=lucid'
'--with-dialogs=athena' '--with-widgets=athena' '--with-athena=3d'
'--with-sound=no' '--pdump'
XEmacs 21.4.12 "Portable Code" configured for `i686-pc-linux'.
Compilation / Installation:
Source code location: /usr/src/xemacs/xemacs-21.4.12
Installation prefix: /usr/local
Operating system description file: `s/linux.h'
Machine description file: `m/intel386.h'
Compiler: gcc -g -O3 -Wall -Wno-switch -Winline
-Wmissing-prototypes -Wsign-compare -Wshadow
Relocating allocator for buffers: no
GNU version of malloc: yes
- Using Doug Lea's new malloc from the GNU C Library.
Window System:
Compiling in support for the X window system:
- X Windows headers location: /usr/X11R6/include
- X Windows libraries location: /usr/X11R6/lib
- Handling WM_COMMAND properly.
Compiling in support for the Athena widget set:
- Athena headers location: X11/Xaw3d
- Athena library to link: Xaw3d
Using Lucid menubars.
Using Lucid scrollbars.
Using Athena dialog boxes.
Using Athena native widgets.
TTY:
Compiling in support for ncurses.
Compiling in support for GPM (General Purpose Mouse).
Images:
Compiling in support for GIF images (builtin).
Compiling in support for XPM images.
Compiling in support for PNG images.
Compiling in support for JPEG images.
Compiling in support for TIFF images.
Sound:
Databases:
Compiling in support for Berkeley database.
Compiling in support for LDAP.
Compiling in support for PostgreSQL.
- Using PostgreSQL header file: libpq-fe.h
- Using PostgreSQL V7 bindings.
Internationalization:
Compiling in support for Mule (multi-lingual Emacs).
Compiling in support for XIM (X11R5+ I18N input method).
- Using Motif to provide XIM support.
Compiling in support for Canna on Mule.
Compiling in support for the WNN input method on Mule.
Mail:
Compiling in support for "dot-locking" mail spool file locking method.
Other Features:
Inhibiting IPv6 canonicalization at startup.
Compiling in support for dynamic shared object modules.
Using the new portable dumper.
4:07 p.m.
>>>> "Jan" == Jan Rychter
<jan(a)rychter.com> writes:
Jan> And indeed, the Emacs version of mail-extr.el seems to use
Jan> [:alnum:] and [:alpha:]:
Which is the right thing to do. But our regexp engine doesn't
currently support those. It's not clear to me what the right way to
do that is, given the brokenness of the POSIX standard with respect to
multilingual text. I guess the best bet is simply to use Unicode's
idea of whether something is a word component or not.
Volunteers? Note that the implementation will be very different in
21.4 (which has no good access to Unicode tables) and 21.5.
Jan> Now, opinions are divided on whether
Jan> mail-extract-address-components should really get
Jan> multilingual text or not.
Jan> In any case, the current solution is rather broken.
mail-extr.el is basically a collection of hacks and kludges anyway;
you may as well just add the relevant characters to those regexps.
That will work in 21.4 as well as 21.5. Or (as a heuristic), just use
[^\000-\255] (you'd have to do that as an alternative rather than a
character class). I bet that will work well enough in practice.
--
Institute of Policy and Planning Sciences http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
Ask not how you can "do" free software business;
ask what your business can "do for" free software.
9:03 p.m.
>>>> "Stephen" == Stephen J Turnbull
<stephen(a)xemacs.org> writes:
>>>> "Jan" == Jan Rychter <jan(a)rychter.com> writes:
Jan> And indeed, the Emacs version of mail-extr.el seems to use
Jan> [:alnum:] and [:alpha:]:
Stephen> Which is the right thing to do. But our regexp engine doesn't
Stephen> currently support those. It's not clear to me what the right
Stephen> way to do that is, given the brokenness of the POSIX standard
Stephen> with respect to multilingual text. I guess the best bet is
Stephen> simply to use Unicode's idea of whether something is a word
Stephen> component or not.
Stephen> Volunteers? Note that the implementation will be very
Stephen> different in
Stephen> 21.4 (which has no good access to Unicode tables) and 21.5.
Well, it seems there were no volunteers :-( (and as I wrote, I am not
capable to fix it myself)
Jan> Now, opinions are divided on whether
Jan> mail-extract-address-components should really get multilingual
Jan> text or not. In any case, the current solution is rather broken.
Stephen> mail-extr.el is basically a collection of hacks and kludges
Stephen> anyway; you may as well just add the relevant characters to
Stephen> those regexps. That will work in 21.4 as well as 21.5. Or
Stephen> (as a heuristic), just use [^\000-\255] (you'd have to do that
Stephen> as an alternative rather than a character class). I bet that
Stephen> will work well enough in practice.
Well, I can continue to kludge around this adding relevant characters
myself, but...
The current situation basically means that mail-extr is broken for
anyone who doesn't use ASCII + ISO-8859-1. It breaks BBDB and Supercite
for me. I'd consider it quite serious breakage.
--J.
5:37 a.m.
>>>> "Jan" == Jan Rychter
<jan(a)rychter.com> writes:
Jan> The current situation basically means that mail-extr is
Jan> broken for anyone who doesn't use ASCII + ISO-8859-1. It
Jan> breaks BBDB and Supercite for me. I'd consider it quite
Jan> serious breakage.
So? Fixing XEmacs is hard; I hope to do this for 21.5 but you'd have
to pay me to do it for 21.4 (it would require extensive testing to
make sure it doesn't break other parts of the regexp engine _before_
committing it to the repository for the stable branch).
Fixing mail-extr is easy: add the necessary characters to the
character class. I don't know what they are; you do. Tell us!
(Preferably in the form of a patch.)
--
Institute of Policy and Planning Sciences http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
Ask not how you can "do" free software business;
ask what your business can "do for" free software.
2:18 p.m.
>>>> "Stephen" == Stephen J Turnbull
<stephen(a)xemacs.org>:
>>>> "Jan" == Jan Rychter <jan(a)rychter.com> writes:
Jan> The current situation basically means that mail-extr is broken for
Jan> anyone who doesn't use ASCII + ISO-8859-1. It breaks BBDB and
Jan> Supercite for me. I'd consider it quite serious breakage.
Stephen> So? Fixing XEmacs is hard; I hope to do this for 21.5 but
Stephen> you'd have to pay me to do it for 21.4 (it would require
Stephen> extensive testing to make sure it doesn't break other parts of
Stephen> the regexp engine _before_ committing it to the repository for
Stephen> the stable branch).
Stephen,
Sorry -- if I knew how to fix it, a patch would already have been in
your mail. The reason I'm bringing this up (repeatedly) is because I
don't know how to properly fix it. And I think that the fact that
XEmacs+Gnus+BBDB is broken for many people is rather important.
Stephen> Fixing mail-extr is easy: add the necessary characters to the
Stephen> character class. I don't know what they are; you do. Tell
Stephen> us! (Preferably in the form of a patch.)
I don't think it's possible to add all the necessary characters. I know
what they are for the Polish language, but there are probably many other
languages affected by this, not just Polish.
Following your advice, I'll just fix my copy as best I can, send you the
changes I've made (even though I know they are not general and are just
useless for most people) and shut up about the issue. I still think it's
a good idea to remember about the problem...
--J.
diff -u /usr/src/xemacs/packages/xemacs-packages/mail-lib/mail-extr.el
/local/lib/xemacs/xemacs-packages/lisp/mail-lib/mail-extr.el
--- /usr/src/xemacs/packages/xemacs-packages/mail-lib/mail-extr.el 2003-06-26
13:34:19.000000000 -0700
+++ /local/lib/xemacs/xemacs-packages/lisp/mail-lib/mail-extr.el 2003-06-26
13:34:19.000000000 -0700
＠＠ -302,8 +302,19 ＠＠
;; You will also notice the consideration for the
;; Swedish/Finnish/Norwegian character set.
;; XEmacs: go to \376 instead of \377 to work around bug in search.c...
+
+;; The Polish part of ISO-8859-2 additionally requires:
+;; (161 198 202 163 209 211 166 172 175 177 230 234 179 241 243 182 188 191)
+;; (?A ?C ?E ?L ?N ?O ?S ?Z ?Z ?a ?c ?e ?l ?n ?o ?s ?z ?z)
+;; Aogonek Cacute Eogonek Lslash Nacute Oacute Sacute Zacute Zdotaccent
+;; aogonek cacute eogonek lslash nacute oacute sacute zacute zdotaccent
+(defconst mail-extr-pl-chars (mapcar #'(lambda (a)
+ (make-char 'latin-iso8859-2 a))
+ '(161 198 202 163 209 211 166 172 175
+ 177 230 234 179 241 243 182 188 191)))
+
(defconst mail-extr-all-letters-but-separators
- (purecopy "][A-Za-z{|}'~0-9`\200-\376")) ;; XEmacs
+ (purecopy (concat "][A-Za-z{|}'~0-9`\200-\376" mail-extr-pl-chars))) ;;
XEmacs
;; Any character that can occur in a name in an RFC822 address including
;; the separator (hyphen and possibly period) for multipart names.
＠＠ -313,11 +324,11 ＠＠
;; Any character that can start a name.
;; Keep this set as minimal as possible.
-(defconst mail-extr-first-letters (purecopy "A-Za-z")) ;; XEmacs
+(defconst mail-extr-first-letters (purecopy (concat "A-Za-z"
mail-extr-pl-chars))) ;; XEmacs
;; Any character that can end a name.
;; Keep this set as minimal as possible.
-(defconst mail-extr-last-letters (purecopy "[A-Za-z`'.")) ;; XEmacs
+(defconst mail-extr-last-letters (purecopy (concat "[A-Za-z`'."
mail-extr-pl-chars))) ;; XEmacs
(defconst mail-extr-leading-garbage "\\W+")
8:41 a.m.
>>>> "Jan" == Jan Rychter
<jan(a)rychter.com> writes:
Jan> I don't think it's possible to add all the necessary
Jan> characters. I know what they are for the Polish language, but
Jan> there are probably many other languages affected by this, not
Jan> just Polish.
Actually, I know how to do this in general (the same way that
[:alpha:] works), but I don't have time, and it will only work in
21.5. But we can fix Polish _now_ with your help. That's all I
wanted.
Thanks for the patch, I'll push it to XEmacs Patches later.
--
Institute of Policy and Planning Sciences http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
Ask not how you can "do" free software business;
ask what your business can "do for" free software.
7820
days inactive
7833
days old
5 comments
2 participants
participants (2)
-
Jan Rychter -
Stephen J. Turnbull