SY> I just think that we should be using our extents api whenever SY> possible instead of overlays. Now there's a nice little job SY> for some conscientious hacker, getting rid of overlays from SY> all the XEmacs packages.:-) Well, I did a little looking around in the process of trying to figure out what was going on here, and my conclusion is that it is not going to happen. We can do just a little more work, and get rid of all use of even the term "overlay" in the core. But not packages. There are at least a dozen packages that use the overlay interface. It turns out that in most cases (defalias 'make-overlay make-extent) is sufficient. It also turns out that nobody _ever_ does (defalias 'make-extent make-overlay) to make GNU XEmacs-compatible. In fact, what normally happens is (if (featurep 'xemacs) (defalias 'package-make-overlay make-extent) (defalias 'package-make-overlay make-overlay)) which is incredibly ugly. I mean, when Kyle Jones does that, we can just fold our hand and wait for a new deal right there. I think that what we ought to do is restrict the overlay interface even more than we've already done in overlay.el, signal on "inappropriate" usage (ie, _anything_ not compatible with the intersection of XEmacs extent and GNU Emacs overlay APIs). Then packages that need bug-compatibility with overlays can write their ugly compatibility layers. For the rest, we let people do what they're gonna do anyway, except pretty. Of course, we could write "extents.el" for GNU Emacs and see if we can get it past rms and EZ.... It has to be somebody who's signed their papers, of course. Comments? -- Institute of Policy and Planning Sciences http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Don't ask how you can "do" free software business; ask what your business can "do for" free software.

|--==> "SJT" == Stephen J Turnbull <stephen(a)xemacs.org&gt; writes: SY> I just think that we should be using our extents api whenever SY> possible instead of overlays. Now there's a nice little job SY> for some conscientious hacker, getting rid of overlays from SY> all the XEmacs packages.:-) SJT> Well, I did a little looking around in the process of trying to figure SJT> out what was going on here, and my conclusion is that it is not going SJT> to happen. Isn't that a bit of a defeatist attitude? It's not like it would be impossible to do and in most cases not hard either, just time consuming. SJT> We can do just a little more work, and get rid of all use of SJT> even the term "overlay" in the core. Yes, this should happen. SJT> But not packages. There are at least a dozen packages that use the SJT> overlay interface. It turns out that in most cases SJT> (defalias 'make-overlay make-extent) SJT> is sufficient. To me, this is a good argument for doing it. SJT> I think that what we ought to do is restrict the overlay interface SJT> even more than we've already done in overlay.el, signal on SJT> "inappropriate" usage (ie, _anything_ not compatible with the SJT> intersection of XEmacs extent and GNU Emacs overlay APIs). You mean sort of like the obsolete function thingy? Sounds good as part of the solution/fight. :-) SJT> Of course, we could write "extents.el" for GNU Emacs and see if we can SJT> get it past rms and EZ.... It has to be somebody who's signed their SJT> papers, of course. How would this work? What would 'extents.el' do for the FSF folk? -- |---<Steve Youngs>---------------<GnuPG KeyID: 10D5C9C5>---| | XEmacs - It's not just an editor. | | It's a way of life. | |------------------------------------&lt;youngs(a)xemacs.org&gt;---|

|--==> "SJT" == Stephen J Turnbull <stephen(a)xemacs.org&gt; writes: SY> Isn't that a bit of a defeatist attitude? It's not like it SY> would be impossible to do and in most cases not hard either, SY> just time consuming. SJT> Exactly. And just whose time do you think it's going to consume? Not SJT> yours. Something being time consuming is hardly a reason not to do it, unless of course you (general "you", not you specifically) are lazy. SJT> That is negligible compared to the waste in time for the 3rd SJT> party developers who want to support both GNU Emacsen. I would hardly call it a waste in time. SY> You mean sort of like the obsolete function thingy? Sounds SY> good as part of the solution/fight. :-) SJT> It is not a "fight," for Heaven's sake. The 3rd party developers are SJT> not our enemy, GNU Emacs is not our enemy. Whoa! You've got the wrong end of the stick. The solution/fight I am referring to is the one to get people who develop for XEmacs to use our APIs which we consider superior. And that includes us. If we won't use our stuff, why should anyone else? [1] A good starting point is core, which is almost devoid of overlays but might need a little more work, (I just did a basic grep of the source and didn't look too closely). The next step after that is packages. SY> What would 'extents.el' do for the FSF folk? SJT> What it would do for the 3rd party developers is give them access to SJT> what we consider the superior interface and implementation. Then lets do it. Sounds like a fantastic idea to me. SJT> If it's really so much better, then at least some of them (think SJT> "Kyle Jones") would possibly switch to using $package-extent-* SJT> calls instead of $package-overlay-* calls. Has anyone bothered to ask Kyle about it? Kyle, are you still reading xemacs-beta? Why don't you use extents instead of overlays? Do you find extents lacking in some way? Do we need to improve them somehow? Footnotes: [1] BTW, when was the last time anyone wrote an XEmacs module? They sound like a nifty idea to me. I think we should be encouraging development there too. -- |---<Steve Youngs>---------------<GnuPG KeyID: 10D5C9C5>---| | XEmacs - It's not just an editor. | | It's a way of life. | |------------------------------------&lt;youngs(a)xemacs.org&gt;---|

|--==> "SJT" == Stephen J Turnbull <stephen(a)xemacs.org&gt; writes: SY> I would hardly call it a waste in time. SJT> Core GNU Emacs people certainly would, and people who are closer to SJT> the GNU project than to XEmacs probably would. But many of our SJT> packages come from just those people. All the more reason to encourage the use of the nifty things that XEmacs can do. Once they get used to it, you never know, some of them might switch to XEmacs completely. :-) SY> If we won't use our stuff, why should anyone else? [1] SJT> We do, in core, and in packages where the "upstream" maintainer is an SJT> XEmacs person. We do believe that extents are better than overlays don't we? In that case shouldn't we be aiming for 100% use? SJT> The packages are not "us", they're mostly third-party. In many cases SJT> originally developed and still maintained by GNU Emacs people for GNU SJT> Emacs. My mature response: So. SJT> VM does use extents on XEmacs. OK, now I don't know what you meant when you referred to Kyle. What was your point? Seems to me like you've shot your own argument. Or am I missing something? SJT> But the compatibility API uses the _name_ "overlay", and of course on SJT> GNU Emacs the VM functionality is implemented using overlays. Which is fair enough, isn't it? SY> [1] BTW, when was the last time anyone wrote an XEmacs module? SY> They sound like a nifty idea to me. I think we should be SY> encouraging development there too. SJT> Andrew Begel at UC Berkeley is doing very active development of SJT> "something" with modules, as is Jerry James ("something else") at U SJT> Kansas. Cool! -- |---<Steve Youngs>---------------<GnuPG KeyID: 10D5C9C5>---| | XEmacs - It's not just an editor. | | It's a way of life. | |------------------------------------&lt;youngs(a)xemacs.org&gt;---|

Steve Youngs writes: [snip] if they are like apache does their modules its a brilliant idea look what it did for apache its the most used and capable web server around -- XEmacs Advocate | Do not try to bend the spoon, that's impossible, FreeBSD Devote | but try to realise the truth... There is no spoon. Perl Hacker | - Child Prodigy, The Matrix. Apache God | <mailto:rendhalver＠users.sf.net>

On 14 Dec 2001 at 19:06:25 +0900, Yoshiki Hayashi <yoshiki(a)xemacs.org&gt; wrote: <*waves arms*> I'm still here! I've been reading xemacs-beta faithfully, although I admit that I haven't exactly been talkative. Let me tell you what I've been up to, and what my plans are. Then you can shoot them down. :-) The obvious API would completely isolate the XEmacs internals from the modules, forcing modules to (for example) call functions to get access to variables and functions not defined in the module itself, even if those variables and functions are defined at the C level. The problem is that this API doesn't quite cut it for my pet module, the ViaVoice interface. There are two hassles: (1) Not everything a module might need to do is accessible in this way. For example, the ViaVoice module needs to hook into the event stream. I copied BillP's approach to doing that, and actually modified event_stream->event_pending_p and event_stream->next_event_cb to call my functions, which in turn call whatever used to be there. With the obvious API, event_stream would not be accessible from the module. We could make an exception, just for event_stream, except that I have this gut feeling that the number of exceptions we would need would grow without bound, which would render the obvious API kind of pointless. To solve my particular problem (event_stream), I intend to write hook_event_stream and unhook_event_stream functions. This is not entirely trivial, since existing code will have to be modified to use the new functions . Other such modifications may be needed for future modules. (2) Performance, performance, performance. Did I mention performance? My ViaVoice module is already too slow. Forcing it to go through Elisp every time, even when the Elisp is just going to call back into C, isn't going to speed it up any. My initial thought on this one was to use a language-level solution. Have the module writer identify functions to be called and variables to be accessed in some idiomatic way, and then have ellcc do something to compile down to C function calls and C variable accesses whenever possible, using the Lisp interface otherwise. **Note that this means that the binary version of a module will be specific to the version of XEmacs corresponding to the ellcc that produced it.** I proposed something along those lines awhile back, only I got too ambitious and proposed a similar approach for the XEmacs core as well. Martin (rightly!) shot me down. :-) Would something like that be too onerous to impose on module writers? If not, I'll get to work on figuring out how to do it. (But not until after I grade all my finals, which effectively means not until after Christmas.) My plan is to approach the tasks in this order: 1. Spec out the "obvious" API described above, and announce it as module API version 0.1, so that potential module writers can get started. 2. Change event hooking and unhooking so that it will be accessible to module writers. (In fact, *un*hooking isn't even possible right now, so that will be a bonus.) Document this in API version 0.2. 3. Build a tool which will assemble a list of C functions and variables that are visible from Lisp and record their [Lisp name -> C name] mappings in a place accessible to ellcc. 4. Come up with an idiom for identifying Lisp accesses in a module, and make ellcc compare with the list built by 3 in order to optimize access in the current module. Also, give ellcc a switch to turn this optimization off, in order to generate XEmacs version independent modules. Document this in API version 0.3. Several. Identify Lisp functions and variables which will always and forever be implemented in C. Either: (a) let module writers have direct access to those; or (b) make ellcc always compile them to direct C accesses, even when the optimization switch is off. Identify them in the official API and release version 1.0. Many. Get to work on figuring out how to write an Elisp -> module converter, followed by optimizing intermodule function/variable accesses. There. Now you can tell me all about my insanity. Cool! Is your module publically available anywhere? In other module news, let me tell you about some ViaVoice developments. After writing the C code, I couldn't seem to find time to finish the module off properly by writing all the Elisp to use it. So when the graduate level Software Engineering class here asked for projects, I offered that one up. Some students in that class did a really nice job of developing it further. It still isn't what I would consider done, but it is a lot further along. After I do a little cleanup on it, I'll release it. -- Jerry James, who notes the lack of a serial comma in the following quote "Taurus: The stars predict tomorrow you'll wake up, do a bunch of stuff and then go back to sleep." -- "Weird Al" Yankovic, "Your Horoscope for Today"

Not that I should really stick my 2 cents in, but as someone who also uses the XEmacs module feature, I don't see what the big deal is about XEmacs modules having access to the XEmacs internals. Sure, it would have been nice, back when Emacs/XEmacs was young and small to have thought about externally written plug-ins, but I suspect that any good solution might require a fair bit of rearchitecting, and at low levels too. The current solution is that you need to know how to find the functions you're looking for inside XEmacs, and release your module to be tied to a specific XEmacs major/minor release. I know that my module needed to be modified between 21.1 and 21.4 because of some API changes, and that was fine; I updated it and my module's source code now works with both versions. Sure, it's a pain in the butt to always have to check my module's compatiblity with each new release of XEmacs, but that's the price to pay for an active ongoing project (in my case, a research project at a university). BTW, you can't force modules to be completely in Lisp when attaching legacy code written in another language (C, C++) to XEmacs. XEmacs ability to call into C is good enough. It's not a bad thing to write plug-ins in C or C++ -- just a different source language. If you want to write a foreign function interface to ELisp, that wouldn't be a bad idea; it'd be hard, but not a horrible idea. Perhaps the SWIG project team could be persuaded to implement it... Andrew Begel On Friday, December 14, 2001, at 10:31 PM, Stephen J. Turnbull wrote:

Jerry James <james(a)xemacs.org&gt; writes: I still agree with Martin and I think we should not put too much effort to imporove perfermance. This does not prevent you from working on those code but IMHO defining API and writing documentation is much more important for majority of users. I'm concentrating on this part so I'll send my proposal about how we will define API to this list. This is a pretty interesting project. However, I doubt how much that buys us with performance. It's too hard for simply try and see the results. Again, I wouldn't do that myself, but if you really want to do it, just go ahead. Bigloo project might be interesting to you. It compiles Scheme to executable code or Java byte code. http://kaolin.unice.fr/~serrano/bigloo/bigloo.html It's at http://www.sodan.org/~penny/XEmacs/xemacs-svn-0.0.2.tar.gz but it doesn't compile with the latest Subversion due to the interface change of Subversion. I'll release a new version in a few days but my development environment is not quite ready. I've been setting up my new laptop lately. Very cool. "Stephen J. Turnbull" <stephen(a)xemacs.org&gt; writes: My primary motivation to write module is to provide glue code between libraries written in C and Emacs Lisp. To make glue code as XEmacs version independent as possible, we need standard API and it must be free from internal details. As an added note, I'm thinking about adding configure option to compile database.c, postgres.c, ldap.c, mule-wnnfns.c and mule-canna.c as external module. With this change, Debian doesn't maintain two binaries for mule, with and without wnn and canna. Also it partially solves conflicting library problem. For example, right now you cannot use ldap and db3 support at the same time because openldap pulls in db2. Andrew Begel <abegel(a)eecs.berkeley.edu&gt; writes: I think it's worth the effort because XEmacs will be in more good shape. I'm ready to spend some time on it. It's not too hard for you and me to follow the API change in XEmacs core but I don't expect Sunday hacker would be willing to do the same. It's true that current situation is not too bad but I don't expect external module will be promoted to supported feature unless we have good API. Fully agreed. I looked at SWIG code a few weeks ago. It's not easy to write new language module but it's not too hard either. I estimate it to be one programmer week job. Any volunteers to write it? You need to familiar with both SWIG and XEmacs internals. Also, it requires minimal amount of C++ knowledge. I'll eventually do it myself if no one else does but it'd be nice if any lurker steps up and implement it. -- Yoshiki Hayashi