Stephen writes: I think it's becoming less so as Uday fits bits of refactoring into the frenetic feature addition. More importantly, it's less buggy. The version currently in our repository still has mail-corrupting coding system bugs. (If you think vm is bletcherous, you should see my add-ons;-) _______________________________________________ XEmacs-Beta mailing list XEmacs-Beta(a)xemacs.org http://lists.xemacs.org/mailman/listinfo/xemacs-beta

Stephen J. Turnbull writes: That is a bit harsh! As a theorist of software, I always make a crucial distinction between internal structure and external functionality. VM may not have the world's most beautiful code, but it definitely wins in its functionality. Some other tools that I won't name have pretty-looking code, but their usability is, well..., bletcherous. Moreover, I think VM has a certain logical coherence which makes it quite solid. Despite the fact it is going through a second maintainer now, almost nothing we do breaks it. That is quite phenomenal if you ask me. In the last couple of years, we added IMAP support (which was already there, but sort of half-done) and external messages (which was a fundamental architectural extension), without getting into a single major issue. Nobody lost an email message. Nobody had a mail folder corrupted. VM just continues to run without batting an eye lid. This is a far cry from what we read on our competitors' newsgroups. The beauty of code isn't skin-deep. It is in the logical soundness of the architecture which can stand the stresses of the times. VM definitely has it, and certainly wins my admiration for it. Cheers, Uday _______________________________________________ XEmacs-Beta mailing list XEmacs-Beta(a)xemacs.org http://lists.xemacs.org/mailman/listinfo/xemacs-beta

Stephen writes: It seems to me there is quite a bit of refactoring going on. I have mixed feelings about this: firstly, it breaks my add-ons that rely on internal structure (well, there was no documented api, was there?), secondly Uday likes Common Lisp, which is an alien idiom to me, so makes everything hard to read. _______________________________________________ XEmacs-Beta mailing list XEmacs-Beta(a)xemacs.org http://lists.xemacs.org/mailman/listinfo/xemacs-beta

Stephen J. Turnbull writes: That is good to know. We get demands from users for new functionality, but never complaints about excess functionality. So, if you can give me some samples of what you would consider excessive in the recent VM, that would be helpful. I think there is a more subtle point I am making. There is a distinction I can make along various dimensions, such as: internal vs external textual vs behavioral syntactic vs semantic static vs dynamic representation vs computation Traditional software engineering seems too focused on the first column in these dichotomies (what I called "skin-deep" in my earler message), whereas I think what matters more is the second column. You can see the syntactic structure as soon as you look at a piece of code. To understand its semantic structure, you need to understand its behavior and see what happens when you tweak it. Pretty looking pieces of code can fall apart at the simplest of tweaks. Having good syntactic structure doesn't mean that the code is logically coherent. VM, on the other hand, has logical coherence that continues to stand up when we tweak it. In the long run, that is what makes the code easy to maintain and modify. (Of course, nobody would complain if it had good syntactic structure too. But that is not the key criterion, as I see things now.) Cheers, Uday _______________________________________________ XEmacs-Beta mailing list XEmacs-Beta(a)xemacs.org http://lists.xemacs.org/mailman/listinfo/xemacs-beta

On 2012-02-02, Stephen J. Turnbull <stephen(a)xemacs.org&gt; wrote: I had no idea what you were talking about, but then I found (setq vm-mime-thumbnail-max-geometry nil) in my .vm. Is that what you're looking for? I'd never heard of it, never seen it, and there's nothing in my .vm to disable it. So why do I not see it, and you do? (Not really sure why this is on xemacs-beta, rather than the vm list!) _______________________________________________ XEmacs-Beta mailing list XEmacs-Beta(a)xemacs.org http://lists.xemacs.org/mailman/listinfo/xemacs-beta

On 2012-02-02, Stephen J. Turnbull <turnbull(a)sk.tsukuba.ac.jp&gt; wrote: But definitely. If you don't annoy the VM developers, the annoyances will get into a release, and then the other XEmacs VM users (if there is indeed more than one other) will have to complain about them. Using VM trunk under XEmacs is courageous, in the Yes, Minister sense, I think...How often do you update? _______________________________________________ XEmacs-Beta mailing list XEmacs-Beta(a)xemacs.org http://lists.xemacs.org/mailman/listinfo/xemacs-beta

On 2012-02-03, Stephen J. Turnbull <stephen(a)xemacs.org&gt; wrote: Ah. Yes. In fact, 21.5's inability to combine with VM was essentially the reason I stopped using 21.5, and therefore also the only reason I spent months unicodifying 21.4! A bit disproportionate, perhaps... Mike did post a patch here for one of the major problems five years ago, but it never got into the vm upstream source. I mentioned it in a bug track recently, but I'll check and see whether it's been included now. "Yes, Minister" was a 1980s British comedy series. Well worth getting if you enjoy satire of the body politic. "Courageous" is the word used by the civil servant to the minister to mean "will lose you the next election". It's now an established idiom in Britain. _______________________________________________ XEmacs-Beta mailing list XEmacs-Beta(a)xemacs.org http://lists.xemacs.org/mailman/listinfo/xemacs-beta

Stephen J. Turnbull writes: Yeah, both good examples! Some of the mess is because, when Kyle Jones became inactive, there was a bit of a free-for-all in the VM community. Rob Fenk had the attitude that the more the better. Recall that Rob had a separate distribution of VM, called "vmrf", which eventually became VM when Kyle handed control over to Rob. So, there are a lot of these "contributed" features that I wouldn't have put into VM if I had anything to do with it at the time. After I took over, I established the defining principle that "VM" means what is described in the VM manual and all the other stuff is a collection of "add-ons". The current codebase describes each file in the VM distribution as either being "part of VM" or "add-on for VM". The message-history stuff is an add-on, and you have to explicitly plug it in by adding invocations in your .emacs or .vm file, which I suppose you did! The thumbnails are a part of core VM, but I have added a way to disable them. (Set `vm-mime-thumbnail-max-geometry' to nil.) In any case, I am aware of these kind of issues and do some cleaning up as and when I get around to them. I think I am using "syntactic" more broadly, as everything to do with the program text. That includes the division of the code into functions and modules etc. So, all modularity is "syntactic" in this sense. However, any information hiding that is obeyed at run-time is "semantic", irrespective of whether it is represented in the program structure. What I mean by "semantic" is the trace of all the primitive operations that get done during an execution. The call graph is close to this, but it is still dependent on the program structure because the functions that get "called" are syntactic. If the functions were badly chosen, then the call graph doesn't tell us much useful. A piece of software is good semantically if it is doing the right sequence of operations in the right order. It is modular if the information flow in the trace can be easily discerned and is natural. If the system is good in this way, we can refactor the code for better modularity, and everything will continue to work fine. But if the software had bad semantics to start with, I believe that no amount of refactoring will fix it. If you will indulge me, I can suggest some visual imagery. Think of all (the instances of) primitive operations as grains of sand. The code is the mason that assembles these grains into a nice castle that the user will enjoy using and the maintainers will enjoy tweaking. The beauty of the castle is an entirely separate issue from the beauty of the mason. If the mason is good-looking, I wouldn't complain, but if the mason is unkempt, it wouldn't bother me much either. I believe that the traditional software engineering confuses the mason for the castle. Whether it has any idea of the castle at all is questionable. (The Distributed Systems people seem to understand this though. They often talk about the trace semantics, and totally ignore the syntax. The Programming Methodology people are quite the opposite.) I am not saying you should tweak it to see what happens. But the acid test of your design is whether it can be tweaked in the future without making it fall apart. "Seeing the code do what you designed it to do" is a short-term goal, but that is not good enough, is it? Long-lived software needs to be maintainable. Yes, "likely" is the operative word here. It is not guaranteed. Neither is the opposite guaranteed. In other words, even if there is no clear structure in the code, it is possible to have clear structure in the semantics. Of course, you average Joe Programmer may not be able to achieve it. But Kyle Jones could. vm-thread.el builds the threads database for folder, which can then be used by all other modules. For instance, vm-motion.el uses it to go to a parent message. vm-delete.el uses it to delete a message along with its thread-subtree and so on. But the internal structure of the threads database is exclusively handled inside vm-thread.el, which is the way it should be. Oh, that is good to know. The threading code had lots of subtle bugs, which took me ages to fix. In fact, I now regard the threads database as a challenge problem for programming theorists like me. I hope you now have at least a reasonably correct code base to work with. Cheers, Uday _______________________________________________ XEmacs-Beta mailing list XEmacs-Beta(a)xemacs.org http://lists.xemacs.org/mailman/listinfo/xemacs-beta

Stephen J. Turnbull writes: Agreed. Please feel free to report such issues on viewmail-bugs or viewmail-info mailing lists! I think VM was designed to be just free software. "Open source" is a newer concept. I would agree that VM doesn't meet the standards of open source. Perhaps, one day it will, if we keep rewriting things. That is a good goal! The trunk version of VM you are using does not treat threading as a form of sorting any more. I didn't know about RFC 5256, but I ran into Jamie Zawinski's web page on the threading algorithm and used its ideas. But, VM's algorithm was pretty similar to it already. VM predates Netscape, does it not? And, Jamie was a VM user. I don't know if he still is. The current status of threading in VM is that it is treated as a form of grouping, and all VM operations (deleting, saving, printing and what have you) can operate on threads. `vm-enable-thread-operations' enables it. So, the threads database is maintained at all times, and it is updated consistently during additions and deletions. There is also a host of "cached" information (the visible lineage of each message, the subtree rooted at each message etc.) which needs to be updated consistently. All kinds of problems cropped up in the corner cases (dealing with duplicate copies of messages, or subject line changing in the middle of a thread etc.) and each of them took me ages to fix. I came to the point of throwing away the old code and rewriting it from scratch several times. But I was hopeful that the problems were minor and I was converging on the right code. I think I did. But, still, I hope to rewrite it from scratch some day and think about why it proved to be so hard. Cheers, Uday _______________________________________________ XEmacs-Beta mailing list XEmacs-Beta(a)xemacs.org http://lists.xemacs.org/mailman/listinfo/xemacs-beta

Stephen J. Turnbull writes: Ok, it looks like we have agreement on that issue :-) Quoting from the current VM Manual, Sec. 10.1: "Unlike in previous versions of VM, threading is not a form of sorting. You can sort threads by the usual sort keys and the sort order will apply to at least the root messages of threads. Sorting threads by subject, for instance, can be a quick way to find threads with similar subject lines. Sorting them by date would sort them chronologically according to when the threads were initiated. Sorting them by activity is a variant of the chronological order where the dates of latest activity are given prominence instead of the dates of the initial messages." How the threads affect the sort order is of course handled in the sorting function. There are some mathematical compositions for sort-orders, such as the lexicographic combination etc. Once you plug in the right sort-order into the sorting function, it will produce threaded sort combined with various other sort keys. But building and maintaining the thread-tree is entirely separate to sorting. That is done in the vm-thread.el module. I am not sure that the data structures are entirely similar, but the principles certainly are. When I first looked at VM's code, it was quite impenetrable. So, I went around googling, found Jamie's page, understood what the issues were, and then came back and read Kyle's code. It had pretty much the same ideas. The details differ. Jamie is only concerned with building the thread tree. He doesn't care to maintain it. On the other hand, in VM, we maintain it and do insertions/deletions as messages arrive or leave. For instance, Jamie says, if there is an empty container, you can "nuke it". We can't. A message that needs to go into the empty container might arrive at any time. Instead, what we do is to calculate the visible lineage of each message separately (called "thread-list") and cache it. If a new message arrives to fill a hole, then the cached info is erased and recalculated. (There is a section on threading in the "VM Internals" chapter of the VM Manual. Please see that for full details.) One of the reasons I took pains to fix the problems with the threading code is that I would like to extend it as per the modern needs: merge broken threads, detach subthreads that go off-topic, and worry about identifying duplicate copies that come with different message id's (thanks to our Mailman, gmane and gnu.org friends). I don't know if you will be able to do all these things if you use Jamie's build-and-discard approach to the threads database. I do buy the fact that we have a lot of cycles available these days and can afford to be lazy in coding. Oh, if I do understand why it proved to be so hard, it will probably go into a journal or perhaps it will turn into a bigger project! Cheers, Uday _______________________________________________ XEmacs-Beta mailing list XEmacs-Beta(a)xemacs.org http://lists.xemacs.org/mailman/listinfo/xemacs-beta

Stephen J. Turnbull writes: No, I don't think so. You can turn on threading and sort the messages by various sorting orders. Unless there is a bug in the implementation, they should all work. The following bug-report provides some background as to why this was important: https://bugs.launchpad.net/vm/+bug/498323 In Kyle Jones's VM, there was one built-in sort order for threads (what we call 'activity' order these days), and that is all you got. People tried to persuade him to make it flexible but he didn't have any interest in it. I myself found activity order confusing when dealing with large threads and wished for the 'date' order. Some people prefer to use the 'reversed-date' order and presumably 'reversed-activity' order. So, these are the 4 important orders people want to use in practice. In future, we might have 'delivery-date' and its reverse, and God knows what else. So, you would sort the roots of all threads by the requested sort order, then sort the children of each root, and then their children and so on, recursively? That was in fact the first idea that occurred to me. But then I figured out a way to incorporate everything in just one sort applied to all messages (I still do remember how people used to sort punched cards in the old days!) and it became straightforward. The trick is based on the following property: The sort order of two messages (m1, m2) in a threaded sort reduces to the sort order of their oldest different ancestors (p1, p2). That is, m1 precedes m2 iff if p1 precedes p2. If m1 and m2 belong to different threads, then their oldest different ancestors would be their thread roots. Then the property is clear. If they belong to the same thread, then you can take the thread minus its root as a message list, and the same reasoning applies within the pared down message list. So, vm-sort-compare-thread, given m1 and m2, returns the corresponding p1 and p2, and the remaining sort-keys will order p1 and p2 in an appropriate way. I think I would call this clean and elegant, rather than "complex". If I was working in a higher-order programming language, I would make its type to be vm-sort-compare-thread : sort-order -> sort-order which captures more clearly what is going on. I don't recall all the details either. As I said, I used Jamie's web page to understand what VM was doing. But I made no attempt to modify VM's algorithm to correspond to Jamie's. One difference I remember vividly is that Kyle Jones was keeping cycles in the thread-tree and detecting them when traversing the tree. This made things unstable. One time the cycle might get resolved one way and another time another way. So, I used Jamie's recommendation not to keep cycles in the thread tree at all, and that made things cleaner and clearer. I am not sure I understand how the Internet comes into the picture. VM's thread trees are indeed folder-local. Virtual folders have their own thread trees. I see that it would be feasible to keep a single thread tree for an entire VM session. That would eliminate the issues of having to update the thread trees of virtual folders when an underlying folder changes. However, you would need to address the problem of reclaiming the thread tree nodes of the folders that have been quit, basically rolling out your own memory manager for the thread trees. So, that doesn't come for free! Indeed. But, as you observed earlier, building the thread-tree is not all that expensive. So, doing it as part of virtual folder creation is quite acceptable. What VM does avoid is the rebuilding of the thread tree when you get new mail or expunge a folder or just re-sort the messages. It would be feasible to rebuild the thread trees in those cases. I don't have strong objections to that if the thread-tree building is fast enough. One worry is that the cycles in the thread tree might get resolved differently each time, and that might confuse the user. Yes, indeed, multiple instances are definitely a serious issue. I referred to them as "duplicate copies" of messages earlier. They did give me significant trouble because I didn't worry about them initially. But, later, I discovered that it was important to keep track of which copy among the duplicate copies is regarded as the canonical copy in the thread tree. The duplicate copies share the same message-id, but we cannot assume that they are identical. In the simplest case, one of them might have the 'deleted' attribute set and another one not. The duplicate copies have different UID's on the IMAP server. Sometimes, they even have different References headers because the mailing-list admins munge them! Coming up with the algorithms wasn't hard, especially, because they were there already from Kyle Jones. But smoothing out the problems and fixing the bugs was hard. I do have some idea why that was. When I wrote out the invariants for the data structure, they turned out to be longer than a screenful. So, at each point in the (internal) code, I had to know which of the invariants were true and which were not. And, that is quite a lot of intellectual labor! So, the research question for me is how to layer the data structure so that the invariants are modularized and the intellectual labor in working with the code is reduced. I am also interested in how much the existing verification tools (such as Code Contracts from Microsoft) can help in checking these invariants, and whether new verification technology would be needed. Cheers, Uday _______________________________________________ XEmacs-Beta mailing list XEmacs-Beta(a)xemacs.org http://lists.xemacs.org/mailman/listinfo/xemacs-beta

Stephen J. Turnbull writes: I do add the thread "key" first if threading is turned on, but there no reordering. In typical use of VM, threading would not be among the global sort keys. And, as you know, I don't lose sleep over calling functions. (I believe in structured programming and information hiding, but the rest of software engineering, I regard as a bunch of dogma to be taken with a large grain of salt. But, still, you have evoked my interest in coupling issues and I will continue to think about them.) In typical use of VM, we are using a bunch of current folders, and occasionally we need to go search for something in the archived mail. If the thread database grows as soon as you load archived folders, never to be reclaimed, then that might degrade performance. As you might have seen, memory leakage is a serious concern in the FSF Emacs world right now. Yes, that seems like a useful extension. I look forward to playing with it when you are done! I don't know how common they are either. Here is an example from my test suite (debbugs.gnu.org being the culript):