On Tue, 2002-11-19 at 02:08, Martin Schwenke wrote: [...] Looks good to me Martin. Could you send a patch? -- \/ille Skyttä scop at xemacs.org

On Fri, 22 Nov 2002 16:24:28 +1100, Martin Schwenke said: You're going to make Lisp code decide between throwing its hands up in disgust and collapsing in paroxysms of giggles? ;) But seriously - the idea of trying to support X.400 addresses (especially non-prettyfied ones) makes me think about rms's refusal to accept code that supports non-free software. Sometimes it's time to take a moral stand and refuse to encourage bad ideas.... (OK, it's been a too-long week already. I'll quit ranting. ;) (string-match "/[GS]=\\([-a-zA-Z0-9._]+\\)[^!＠%]+\\([-a-zA-Z0-9!(a)%._]+\\) (1) (2) (2) (1) If we're going to try to support X.400 addresses, is there a reason to restrict it to some Motorola subset? I'm not *positive*, but I suspect that extending the string-match from "/[GS]=\\" to "/[A-Z]+=\\" will catch all the c= and prmd= and all that stuff too. At least it looks like this code only deals with the X.400 variants that smash ' ' to '_' - if we had to deal with quoted whitespace, the code would be a lot worse. (2) To support more general X.400, we'd have to accept something like /c=us/prmd=foo_mail/o=bar/ou=baz/cn=joe_random(a)some.x-400.gate - but that would require having '/' in the regexp as well... (3) I'm also not sure about the _ here - this looks like it will accept a _ in a hostname, which is a no-no. Of course, the _ *would* be OK after a ! or % since it would still be in the rfc822 local-part. I guess it's no sin accepting a broken hostname, as long as we correctly focus on a purported address and not a comment field. And lest I be accused of X.400 bashing, I'll leave you all with this gem of an address, straight from RFC822, section 3.1.4: So, for example, the folded body of an address field ":sysmail"＠ Some-Group. Some-Org, Muhammed.(I am the greatest) Ali ＠(the)Vegas.WBA is analyzed into the following lexical symbols and types: :sysmail quoted string ＠ special Some-Group atom . special Some-Org atom , special Muhammed atom . special (I am the greatest) comment Ali atom ＠ atom (the) comment Vegas atom . special WBA atom The canonical representations for the data in these addresses are the following strings: ":sysmail"＠Some-Group.Some-Org and Muhammed.Ali(a)Vegas.WBA Yes, it's valid. Gaak. ;) http://public.yahoo.com/~jfriedl/regex/code.html if you're masochistic. ;) /Valdis

[Added Joe to CC: for mail-extra.el discussion below...] VS> On Fri, 2002-11-22 at 08:41, Stephen J. Turnbull wrote: VS> This would be very much welcome. Whadda you say, Martin? Hmmm, my plot to take over the world takes its next step... :-) I'm already maintaining eiffel-mode.el in prog-modes (so most of the access is already there), so why not. supercite.el can't be that hard...:-) * It looks like X.400 addresses are supported in mail-extr.el so, when I clean up supercite.el, I might make things depend much more directly on mail-extract-address-components rather than trying to roll my own. * I'm seeing this too: ELISP> (mail-extract-address-components "Ville Skyttä <scop(a)xemacs.org&gt;&quot;) ("Ville Skytt" &quot;scop(a)xemacs.org&quot;) ELISP> (setq yip "Skyttä") "Skyttä" ELISP> (if (string-match mail-extr-name-pattern yip) (match-string 0 yip)) "Skytt" ELISP> (defconst mail-extr-last-letters (purecopy "[A-Za-z\200-\376`'.")) mail-extr-last-letters ELISP> (defconst mail-extr-name-pattern (purecopy (format "\\b[%s][%s]*[%s]" mail-extr-first-letters mail-extr-all-letters mail-extr-last-letters))) mail-extr-name-pattern ELISP> (if (string-match mail-extr-name-pattern yip) (match-string 0 yip)) "Skyttä" Ville, you should make sure this one gets fixed: XEmacs is discriminating against you! :-) There's probably a similar fix for mail-extr-first-letters. Joe, does this sound reasonable? GNU Emacs 21 uses character classes like [:alpha:] and seems to handle this stuff just fine. XEmacs doesn't have these character classes, does it? peace & happiness, martin

On Mon, 2002-11-25 at 02:46, Martin Schwenke wrote: Cool, CVS access stuff taken care of a couple of minutes ago. Thanks! [...] Yep, noticed that too, and a fix would be welcome :) -- \/ille Skyttä scop at xemacs.org

Simon> Doing substantial work on mail-extr.el does not seem like a Simon> good idea to me, but I guess it could be added if someone Simon> do spend the time working on it... Um, what is the canonical way to do mail-extr.el-like stuff then? -- Institute of Policy and Planning Sciences http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.

Simon> Do you mean how to use it? Like I said, call it with ASCII Simon> only strings. No, I mean "if we're going to let mail-extr.el bitrot, what are we going to use instead?" and it seems the answer is, there's nothing. So we can't just let mail-extr.el bitrot, and I conclude it is necessary to do some work on it. Simon> In fact, emacs doesn't have a correct RFC 2822 parser Simon> anywhere, which is a shame.) Thank larsi, who resisted all attempts to even use the relatively rigorous functions available in rfc822.el, let alone a real parser. (According to Steve Baur.) -- Institute of Policy and Planning Sciences http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.

[OK, I've finished annoying Joe Wells... By the way, is there an easy way of finding out who the current XEmacs maintainer for something is?] Simon> Do you mean how to use it? Like I said, call it with ASCII Simon> only strings. mail-extr.el seems to have been designed for Simon> parsing name and addressess from RFC 822 headers (see the Simon> first line of the file, for instance), and mail headers are Simon> ASCII only. I guess the problems comes when it is used in Simon> non-RFC822 situations, where the full mail address includes Simon> non-ASCII, but mail-extr.el was not designed to be used for Simon> that from what I can tell, and it does a poor job of it in Simon> practice too (demonstrated by the buggy snippet in this Simon> thread). OK, I have a really naive question (since I've always taken the non-ASCII stuff for granted, since I only have an ASCII name :-): In general, is there are simple way of getting the contents of the original 7-bit mail header without rendering it? Surely that's the right way of handling it... In the case of VM, it looks like the headers need to be taken from the INBOX buffer instead of the "INBOX Presentation" buffer (substitute appropriate folder names where necessary). That looks to be an easy way of getting the ASCII version (and see comment below). However, mail-extr.el still doesn't work adequately on the unrendered version: ELISP> (mail-extract-address-components "Ville =?iso-8859-1?q?Skytt=E4?= <scop(a)xemacs.org&gt;&quot;) ("Ville" &quot;scop(a)xemacs.org&quot;) :-( [Suddenly Ville joins people such as Prince and Madonna... :-] Simon> As for implementing this properly, I think it would be nice Simon> to have a higher-level API that handled mail addresses with Simon> non-ASCII content. Perhaps this new API should even Simon> replace the current `mail-extract-address-components' entry Simon> point. Of course, that effect could be simulated by Simon> enhancing the current mail-extr.el but looking at the code Simon> I doubt it will be a pleasant experience. OK, as you said, mail headers are ASCII. Once they are rendered using whatever encoding they might specify, it is probably too late to try to do something clever to try and handle them as ASCII, because the rendering isn't necessarily reversible (I guess). Hmmm... maybe we're saying the same thing? Simon> (Another rant is that mail-extr.el doesn't implement a Simon> correct RFC(2)822 parser, but rather a heuristic parser Simon> improved over time. In fact, emacs doesn't have a correct Simon> RFC 2822 parser anywhere, which is a shame.) I think that would be nice, but there is (at least) one interesting problem with using a proper RFC2822 parser. My reading of it tells me that in Simon Josefsson <jas(a)extundo.com&gt; the "Simon Josefsson" part is a display-name, which is useful for our purposes. However, in jas(a)extundo.com (Simon Josefsson) the "Simon Josefsson" part is a comment that isn't a separate synactic (and, therefore, semantic) entity. Writing a formal RFC2822 parser might actually gives us less than what we need... Another problem (separate from RFC2822 issues) is that things like BBDB store the rendered version, so if we start pulling our unrendered chunks of headers, we'd have to render them before comparing... Maybe it is a case of incrementally fixing the glitches... or coming up with something really clever. I think the really clever idea is to always deal with (and store) unrendered ASCII chunks and be careful when comparing them... but that sounds like a lot of work and involves getting a lot of people to agree... peace & happiness, martin

Simon> Looking at the Makefile in the CVS directory for the Simon> package [is the best/only way to find maintainers]? Yes, I think that's true. We should do something about it. One thing that can be done is to publish the PACKAGE-maintainer(a)xemacs.org list. Also, Ville or Rendhalver might want to work up a list from the Makefiles. Maybe what should be done is to have a Maintainers.values make include file as a sibling of XEmacs.rules, which would contain a list of assignments like EDICT_MAINTAINER = Stephen J. Turnbull <stephen(a)xemacs.org&gt; mule-ucs_MAINTAINER = Stephen J. Turnbull <stephen(a)xemacs.org&gt; and XEmacs.rules could do include Maintainers.values MAINTAINER = $($(PACKAGE)_MAINTAINER) (Dealing with whether "mule-ucs_MAINTAINER" is a valid make variable name is an exercise for the implementer, you could always run it through tr to upcase and transform ?- to ?_.) Simon> I don't think so -- having a real parser would allow you to Simon> say that you want to look at the first comment after an Simon> addr-spec. A function could try all the standard ways to Simon> put the full name in a header line and return it to the Simon> caller. The problem is that there aren't (as of 1123, I'm not really up on the RFC2822 clarifications) standardized ways, and identifying them is actually more a lexical problem than an RFC2?822 syntactic issue. A correct parser would at least allow us to separate the net address from the rest of the goop, though, which none of the current applications I use (Gnus, VM, BBDB, Supercite) do with 100% accuracy. Simon> This is a problem in BBDB currently too. Most if not all Simon> non-ASCII names have multiple entries in my .bbdb because Simon> they are encoded differently. OTOH using raw mail headers Simon> would be just as bad. A proper Simon> canonicalize-then-compare-for-equality function is needed Simon> to fix this. "Proper" probably has to wait for full Unicode support, as people can always confuse you with [U+00E4] <-> [U+0061 U+0308] in UTF-8, or even [a "] in pidgen composition. But as long as the BBDB database is stored in a UCS (which it currently is not), a quite high degree of canonicalization is automatic in Mule if rendered versions are always used, as things like Unicode composing characters are rare. Which is already solved (differently) by each of the major MUAs, plus TM and SEMI. Unicode and Mule really are the answer here. Simon> I dunno, there doesn't seem to be a simple solution. Simon> Dealing with and storing unrendered ASCII chunks has the Simon> same problems, the same name can be expressed in many ways. Exactly. Even at the level of transport encoding, Chinese spammers use 8-bit, QP, and Base64 for their headers. I don't know if things have improved recently, but I have often seen 8-bit in headers from human beings, too. (Naive ports of ASCII-oriented MUAs just extract a full name from a file and copy it verbatim to the header. You're supposed to use a utility to translate to MIME, but "power users" often don't bother---and this works fine for Japanese MUAs, which all are quite lenient about such things.) Nor are Base64 or QP actually uniquely defined! Naive implementations of QP may QP all characters, others the minimal set, and others will go to some trouble to pick a set of ASCII characters that could cause trouble in headers. E.g., the period, which naive software often assumes means "part of a host spec". We really need to get this stuff unified and ripped _out_ of the MUAs and into a separate library so that fixes and improvements to VM, Gnus, Mew, Wanderlust, TM, SEMI, mh-e, etc etc all go into the same library. (Yeah, I know, that was the idea behind TM and SEMI and look how far _that_ got. :-( ) -- Institute of Policy and Planning Sciences http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.

viteno> And perhaps a target viteno> maintainer: viteno> ＠echo $(MAINTAINER) maintainer-email: ＠echo $(PACKAGE)-maintainer(a)xemacs.org is supposed to work, although the aliases database is probably now quite out of date. -- Institute of Policy and Planning Sciences http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.

Looked, grepped (mule|xemacs)-packages/*/Makefile and browsed CVSROOT/avail and CVSROOT/ChangeLog, results attached. Please send me corrections if I've forgotten/missed someone (sorry!) or if other bugs or outdated info crept in. I'm planning to commit this into packages/MAINTAINERS tomorrow. -- \/ille Skyttä scop at xemacs.org

Simon> Looking at the Makefile in the CVS directory for the Simon> package [is the best/only way to find maintainers]? Is there anything wrong with rendering the list of known packages and their maintainers *somewhere* on the xemacs.org web-page?

Simon> Looking at the Makefile in the CVS directory for the Simon> package [is the best/only way to find maintainers]? David> Is there anything wrong with rendering the list of known David> packages and their maintainers *somewhere* on the David> xemacs.org web-page? In one sense, nothing, because it's already done: the known packages are in the Packages node of the XEmacs Users' Guide, and the maintainer address is $PACKAGE-maintainer(a)xemacs.org. Also, that's exactly what I suggested in the first line you cut, no? The administrative holdups on getting it done are 1) privacy/spam: we don't want private addresses published on the web page without permission of the person. So we need to filter personal addresses to ＠xemacs.org addresses. 2) We probably do want real names (or net aliases for the dogs among us) 3) (Adrian's point) Since this information is already available in many places, we want to eliminate redundancy so there will be half a chance of keeping it up to date and accurate. Perhaps we can do package-lists-maintainer: dmaster(a)synopsys.com hm? -- Institute of Policy and Planning Sciences http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.

David> All good reasons. My suggestion was a mild one that I David> thought might _eventually_ make it easier for people to see David> what packages XEmacs has available and what levels they are David> at. Right. I've been in a bad mood lately, my intent was not entirely nice, and I'm sorry about that. But I really do want people to understand what kind of process is going on here (not least, the developers themselves!!) I hope that will encourage both better bug reports and suggestions, and participation by people who can see a niche in the process where they can contribute without unwarranted sacrifice of time. David> The "synopsys" address is my work address and I wouldn't David> want something like this going through there. Exactly! ;-) David> I may be a 20 year programmer of Unix, but I feel like a David> novice when it comes to making a system visible on the David> Internet and keeping it safe from spammers and crackers. connection, (2) cvs to cvs.xemacs.org, and (3) the ability to ssh out to cvs.xemacs.org and perhaps ftp.xemacs.org. (4) rsync to cvs.xemacs.org is an optional optimization. Working on documentation, you probably should allow (5) outgoing connections to www.xemacs.org. You can shut down everything else. (Maybe including (2) CVS, I forget whether commit privilege in a subrepository implies read everywhere via:ext over ssh.) David> I feel, though, that it's better to get it (close to) right David> the first time than secure it later. Definitely. A word of advice: you shouldn't count on being able to shut crackers out completely. Make sure you have an early warning system checking logs etc for unusual activity, as well as locking the door. -- Institute of Policy and Planning Sciences http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.

Martin> allowed by RFC2882, Uhm, I have archives going back to 1983. I don't look at them often, but I doubt they conform to RFC2822! Martin> If I start maintaining any of this code I'm going to take Martin> the novel approach of putting together some regression Martin> tests... :-) Consider using the test-harness.el facility. I plan to package it, but for the moment you can get it from tests/automated/test-harness.el in the source directory. Although it's poorly documented, it is easy to use, and there are lots of examples in that directory. There are a few lines of documentation on regression testing in the Internals manual. Martin> if I get a rainy weekend, I might try to write a full Martin> RFC2882 address parser... :-) How about letting Eric do it for you (aka xemacs -batch -l semantic rfc2822.txt -f bovinate :-) ? -- Institute of Policy and Planning Sciences http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.

Martin> [OK, I've finished annoying Joe Wells... By the way, is Martin> there an easy Martin> way of finding out who the current XEmacs maintainer for Martin> something is?] Hi Martin, on a package level to get the info by XEmacs <= 21.1 Options->Manage Packages->Add Download Site->xemacs.org Options->Manage Packages->List & Install XEmacs >= 21.4 Tools->Packages->Add Download Site->xemacs.org Tools->Packages->List & Install then typing i for information on the package under point. Hope this helps, Adrian Simon> Do you mean how to use it? Like I said, call it with ASCII Simon> only strings. mail-extr.el seems to have been designed for Simon> parsing name and addressess from RFC 822 headers (see the Simon> first line of the file, for instance), and mail headers are Simon> ASCII only. I guess the problems comes when it is used in Simon> non-RFC822 situations, where the full mail address includes Simon> non-ASCII, but mail-extr.el was not designed to be used for Simon> that from what I can tell, and it does a poor job of it in Simon> practice too (demonstrated by the buggy snippet in this Simon> thread). Martin> OK, I have a really naive question (since I've always taken the Martin> non-ASCII stuff for granted, since I only have an ASCII name :-): Martin> In general, is there are simple way of getting the contents of the Martin> original 7-bit mail header without rendering it? Martin> Surely that's the right way of handling it... Martin> In the case of VM, it looks like the headers need to be taken from the Martin> INBOX buffer instead of the "INBOX Presentation" buffer (substitute Martin> appropriate folder names where necessary). That looks to be an easy Martin> way of getting the ASCII version (and see comment below). Martin> However, mail-extr.el still doesn't work adequately on the unrendered Martin> version: ELISP> (mail-extract-address-components "Ville =?iso-8859-1?q?Skytt=E4?= <scop(a)xemacs.org&gt;&quot;) Martin> ("Ville" &quot;scop(a)xemacs.org&quot;) Martin> :-( Martin> [Suddenly Ville joins people such as Prince and Madonna... :-] Simon> As for implementing this properly, I think it would be nice Simon> to have a higher-level API that handled mail addresses with Simon> non-ASCII content. Perhaps this new API should even Simon> replace the current `mail-extract-address-components' entry Simon> point. Of course, that effect could be simulated by Simon> enhancing the current mail-extr.el but looking at the code Simon> I doubt it will be a pleasant experience. Martin> OK, as you said, mail headers are ASCII. Once they are rendered using Martin> whatever encoding they might specify, it is probably too late to try Martin> to do something clever to try and handle them as ASCII, because the Martin> rendering isn't necessarily reversible (I guess). Martin> Hmmm... maybe we're saying the same thing? Simon> (Another rant is that mail-extr.el doesn't implement a Simon> correct RFC(2)822 parser, but rather a heuristic parser Simon> improved over time. In fact, emacs doesn't have a correct Simon> RFC 2822 parser anywhere, which is a shame.) Martin> I think that would be nice, but there is (at least) one interesting Martin> problem with using a proper RFC2822 parser. My reading of it tells me Martin> that in Martin> Simon Josefsson <jas(a)extundo.com&gt; Martin> the "Simon Josefsson" part is a display-name, which is useful for our Martin> purposes. However, in Martin> jas(a)extundo.com (Simon Josefsson) Martin> the "Simon Josefsson" part is a comment that isn't a separate synactic Martin> (and, therefore, semantic) entity. Martin> Writing a formal RFC2822 parser might actually gives us less than what Martin> we need... Martin> Another problem (separate from RFC2822 issues) is that things like Martin> BBDB store the rendered version, so if we start pulling our unrendered Martin> chunks of headers, we'd have to render them before comparing... Martin> Maybe it is a case of incrementally fixing the glitches... or coming Martin> up with something really clever. I think the really clever idea is to Martin> always deal with (and store) unrendered ASCII chunks and be careful Martin> when comparing them... but that sounds like a lot of work and involves Martin> getting a lot of people to agree... Martin> peace & happiness, Martin> martin -- Adrian Aichner mailto:adrian＠xemacs.org http://www.xemacs.org/