Loadable modules

All, Well, its been a hectic 9 months, but I am now able to dedicate more of my time to the two projects I care most about, XEmacs and XFree86. I am glad to say that I now have official sanction to work on these projects (as well as a few others) and that I can play a more active role than I have been. I also promise to read the mailing list every day, not every six months :-) As some of you know, I am the persopn responsible for inflicting the modules stuff on you all, and I understand there have been some problems with it, especially on some of the less common platforms. I would like to hear from anyone that is having a problem with the dynamic modules. Please make sure they are enabled by configuring with --with-modules. So, aside from beefing up platform support, I would like to start talking about how we can best use the module technology. Remember that the main aim is to provide the extensibility that users are accustomed to in Lisp, but at the C level (for speed more than anything else). Aside from moving some features from Lisp into C modules, I would like to know if we should consider extending the autoload mechanism to load in modules on demand, whether or not modules impose a security risk (I am guessing it is possible to do things with the system API that are not possible from within the Lisp reader), and how we will deal with platforms that dont support any form of dynamic module loading. This last issue (how to deal with systems that dont support modules) is the toughest issue that I can see. If modules are used properly, they can provide an incredible performance boost to the users. Lets be honest, NO Emacs wins great awards for speed, even though they are almost all "acceptably fast". So there is a real benefit to using modules, but it is unrealistic to expect package authors to maintain two sets of code (C for the module folks, Lisp for the rest). So, before we realistically start debating what to move into modules, we need to establish exactly which systems are afflicted with no module disease, and how important that is. I suspect that if it is limited to very old, very crufty versions of OSes that are relatively unpopular, that we can probably live with the limitation. If, on the other hand, popular platforms are affected, we are of course dead in the water, and modules will remain a relatively uncommon form of Emacs extension. So, given the list of platforms that are currently supported, how many of them would be adversely affected by a move to modules? If the maintainers of each different OS support code can tell me whether or not some form of dynamic loading technology exists on their platforms, I can draw up a matrix of what would be affected, and we can go from there. Having said all of that, I would like to throw out a list of potential cnadidates for migration to modules. I am hoping that at least some of the authors of these packages are also on this list, and can comment on the feasability (and desirability) of moving to modules. o SGML mode. The mode itself could easily remain in Lisp, but is it desirable to have some kind of SGML validation tool loaded as a module? Could SP be loaded in to the editor, and would this make it easier to have on-the-fly validation to make XEmacs a really hot SGML/XML editing environment? o W3. This is the most obvious candidate, and if memory serves, the author expressed interest at some stage in doing a module version. Its big, its slow, but its damn useful and the performance win would make a huge difference to the package. It wont be Mozilla, but then who wants that anyway? :-) o Several of the edit-utils packages, such as scroll-in-place, bookmark, etc, that are used frequently. mode-motion+ is another potential candidate. o Some of the base packages, such as comint, the byte compiler and optimizer, common lisp, font-lock, simple.el (why isnt this in C already?). Thew font-lock would be the biggest win, especially if it was possible to use a different scanner, rather than zillions of regexps, which are much slower than actual tokenization. o Start moving portions of the current C code into modules, so that they are only loaded on demand. For example, TTY support, X11 support, the LDAP and base64 stuff (which are already sample modules), possibly compile many differnt menu / dialog types so the user can select one at load time, etc. There are many ways we can use modules on existing C code. Basically, any package which is executed frequently, especially those at every keystroke or on motion outside of a line, would benefit greatly from being implemented in C. If other folks have other candidates, lets talk about them. Well, thanks for your patience with this long message. I hope we can start dialogue on this topic again, and actually move forward with it. It is a very useful addition to XEmacs (and I believe the FSF has picked up, or is going to pick up, the same stuff), so maybe modules will start playing more of a role in an Emacs's life. Peace and happy hacking to you all :-) Kean.