This sounds like a really fun project! Although I don't have experience with XEmacs internals, about three years ago I worked on integrating the GNU Objective-C runtime system with Boehm's garbage collector. Objective-C is one of the, until recently, three languages supported by GCC, and is a C based, object oriented language, very dynamic in nature, originally modeled after Smalltalk. Based on this work, I modified two rather large libraries I worked on at that time, to use the new memory management system. What I used to describe the layout of the memory was the so-called typed memory, which allows one to describe exactly what is the memory layout of a class or structure. Essentially this tell the GC where are the pointers in your data structures, and gives the GC a very fast way to skip over unwanted data. It turns out however that describing memory layout however is not an easy task. Different machines have different memory layouts and getting this right is very machine dependent. Currently the piece of code that determines the memory layout (which in turn is used to describe the typed memory for Boehm's GC) is the most problematic and causes the most headaches in the GNU Objective-C runtime library. Just a word of caution if you decide to use typed memory. You may want to take a look at the GCC code in the Objective-C runtime library, to see how things are done. Checkout gc.c and encoding.c in the libobjc directory: http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/libobjc/ Regards, -- Ovidiu Predescu <ovidiu(a)cup.hp.com&gt; http://orion.nsr.hp.com/ (inside HP's firewall only) http://www.geocities.com/SiliconValley/Monitor/7464/ (GNU, Emacs, other stuff) On 21 May 2001 22:30:52 +0100, Nix <nix(a)esperi.demon.co.uk&gt; wrote:

Nix> On Mon, 21 May 2001, Ovidiu Predescu said: Nix> Absolutely. I like cleaning up ugly cruft, and the XEmacs GC Nix> layer is really rather in need of a good clean, encrufted by Nix> sheer age. Well, it would be nice if you could be compatible with Richard Reingrub's basic outline. Then we could choose among similarly implemented GCs with different basic strategies, and avoid the question of "is it the GC or the way it's hooked into XEmacs?" BTW, I can't think of a good way to see what has been proposed, except by reading tons of old xemacs-beta posts. Ben's Architecting XEmacs page (link from www.xemacs.org) is the most complete (although dated by now). But you can see some of what's in the process of implementation by doing "cvs status -v version.sh" (in particular, grep for "gc" or "GC" to find Richard Reingrub's branch). I checked fairly recently with our host, and cvs.xemacs.org has sufficient bandwidth and disk space to add a more branches like that. So once you've got things in shape where you're willing to let other people look at it, talk to martin(a)xemacs.org about getting CVS commit privileges, create a branch, and let us know about it. :-) Nix> Myself, I'm still slightly worried about what this will do to Nix> the weird old architectures XEmacs presumably still builds Nix> on. Don't worry; for released XEmacsen your collector will surely go in as a configure option at first. If they still are in use but can't use your collector, so be it---the configure option won't work for them. -- University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Institute of Policy and Planning Sciences Tel/fax: +81 (298) 53-5091 _________________ _________________ _________________ _________________ What are those straight lines for? "XEmacs rules."

Nix> On Mon, 21 May 2001, Ovidiu Predescu said: Nix> Absolutely. I like cleaning up ugly cruft, and the XEmacs GC layer is Nix> really rather in need of a good clean, encrufted by sheer age. Nix> [typed memory] Actually, with Richard's changes, this particular part is trivial: there are only two sorts of objects, those containing only consecutive descriptors, and those containing only bitmap data not relevant to the GC. No more type-specific mark functions, and trivial tracing. -- Cheers =8-} Mike Friede, Völkerverständigung und überhaupt blabla

On Mon, 21 May 2001, Andy Piper gibbered: Any idea who? (This probably won't stop me from doing it again; this is at least partially a `learning project' for me, so I'll gain from it even if nobody but me uses it.) -- `LARTing lusers is supposed to be satisfying. This is just tedious. The silly shite I'm doing now is like trying to toothpick to death a Black Knight made of jelly.' --- RDD

Nix <nix(a)esperi.demon.co.uk&gt; writes: the other downside is its quite pessimal data locality characteristics (it groups objects according to their size -- great for lists, sucks for everything else). whether it matters for XEmacs -- dunno, perhaps not. I've seen one system where it made quite a huge difference, when compared with a custom copying collector. um, not really. it does help, but the delays are still there, especially if your heap is fragmented (and it will become so, as people tend to run one Emacs instance for weeks). the problem with walking the stack is not the unportability, it's the conservatism. you might want to tinker with the code to switch off the data segment scanning, BTW (I don't think it's a run-time option). er. if you consider living in gdb fun, yes. ;) last I heard, generational copying collectors were doing quite well. I believe all the major commercial (and most of the free) Common Lisp and *ML systems use such collectors. good luck, --mike -- When your hammer is C++, everything begins to look like a thumb. -- Steve Hoflich on comp.lang.c++

Nix> I'm currently in the middle of a hack of crazy size, jumping in at the Nix> deep end of XEmacs development, in order to familiarize myself with all Nix> the code. Richard Reingruber is working on a new GC. It can be found in the CVS branch NewGC-21-2. Richard is quiet. Search for his few postings on xemacs-beta. Richard is a student of Michael Sperber. Yoshiki Hayashi has written a copying gc for xemacs recently. It is not yet ready for prime time. His is precise, i.e. he has to maintain all the gcpros, and actually add a few, since the copying involves finding and updating ALL the places on the stack where Lisp_Objects live. Most of the recent maintenance of the existing gc has been by Olivier Galibert and myself. I've taken a look at the Boehm gc in the past, considering its possible use in XEmacs. I rejected using it because: - it's non-portable (reading its portability layer is frightening) - the concurrent or incremental gc feature isn't even intended to be portable currently, but incremental gc is a key attraction. - The big advantage of boehm gc is ditching the gcpros. But this is only a long-term advantage. Probably xemacs is fairly gcpro-clean these days. Incorporating boehm gc is going to be extreme short-term pain for long-term gain. However, the gcc project has recently started using boehm gc. Since gcc is very portable, they must be making any necessary portability improvements to boehm gc as they go. Nix> I'm doing what should have been done a *long* time ago; ditching the Nix> garbage collector and replacing it with a nicer one. I think everyone Nix> can agree that XEmacs's garbage collector currently sucks really rather Nix> hard; I don't think there are any ways to worsen its VM behaviour or Nix> intrusiveness if we tried. Everyone thinks the current gc sux, yes. Nix> The one I've picked is the Boehm collector; it's not terribly portable, Nix> but that can be fixed fairly easily --- and it lets us dump bloody GCPRO Nix> and all the uglinesses it has spawned over the years (blocking string Nix> compaction, Lispification in redisplay and many other things). Nix> The Boehm collector only has one downside that I can see; unportability Nix> (it's not portable to some of the more obscure platforms that XEmacs Nix> runs on). Upsides include: Nix> - better VM behaviour (mark bits kept in a bitmap, and GC_malloc_atomic() Nix> to state that certain kinds of objects cannot contain pointers). As Nix> it is most of XEmacs is forced to stay memory-resident all the time Nix> by mark bit setting; that proportion will reduce to just those Nix> parts that contain pointers that must be traced. This is my primary Nix> motivation for this because I run XEmacs on some fairly memory-poor Nix> machines and I'm fed up of waiting for bloody GC to finish. Hmmm. It seems to me mark() has to examine most of the heap anyways. Unless you do a lot of work, boehm gc might end up examining even more memory than the existing mark. Most of the guts of existing Lisp_Objects are other Lisp_Objects, so there isn't a whole lot of scope for GC_malloc_atomic() to be a big win. Nix> - mark bit sanity; we can reclaim at least one bit from many objects, Nix> and might even get lightweight cons cells back (I'm not sure if one Nix> bit saved is enough for that though). Certainly we can junk the myriad Nix> different ways we have to mark things; this is very much improved Nix> recently but the boehm-gc can fix it completely we can also put the mark-bits into a bitmap without complete boehmification. Understanding and extracting that code from boehm gc would be a much smaller and still very instructive project. Nix> - and last but not least, it's written and actively maintained by Nix> someone who's very accessible and who's one of the best GC hackers Nix> there is, and it is itself acknowledged as probably the best Nix> general-purpose C garbage collector in existence. I share your admiration of Hans. Nix> I'll be paying *no* attention to unexec() and friends in this, at least Nix> at first; I'll keep the portable dumper working, but if unexec() is Nix> totally broken by some interaction with the boehm gc, I will not mourn Nix> unduly. Agreed. Nix> I'm doing the changes to 21.4.3 at first, because I know it's pretty Nix> stable otherwise, so that any crashes I may encounter are my fault. Once Nix> I've got it working fairly stably in there, I'll post the patches (and Nix> yes, I'll split the patches up into pieces; I don't want the Nix> GCPRO-removal to drown out the interesting stuff) and let everyone rip Nix> into them... and then I guess I'll forward-port it to 2.5, and everyone Nix> can let GCPRO fade into the mists of memory (maybe with the aid of Nix> a psychiatrist). I encourage you on your adventure. However, some advice: - communicate with Richard and Yoshiki. - consider implementing mark bitmaps only in the existing gc. - investigate the version of boehm gc from the gcc project. - of course, you do have the blue book called "Garbage Collection", right? Good Luck.

Nix> I'm currently in the middle of a hack of crazy size, jumping in at the Nix> deep end of XEmacs development, in order to familiarize myself with all Nix> the code. Martin> Richard Reingruber is working on a new GC. It can be found in the Martin> CVS branch NewGC-21-2. Richard is quiet. Search for his few postings Martin> on xemacs-beta. Richard is a student of Michael Sperber. Martin> Yoshiki Hayashi has written a copying gc for xemacs recently. It is Martin> not yet ready for prime time. His is precise, i.e. he has to maintain Martin> all the gcpros, and actually add a few, since the copying involves Martin> finding and updating ALL the places on the stack where Lisp_Objects Martin> live. Let me try to clarify matters: - Robert Pluim <rpluim(a)bigfoot.com&gt; at one time did hook up the Boehm collector to XEmacs in 1999 with rather disappointing results. - Richard has been working on making the current GC replaceable. This is a somewhat different goal from implementing a new GC (which is what Yoshiki Hayashi did). Hopefully, there'll be enough time left from his thesis project (which is finished) to actually plug in a different GC, albeit one he didn't write himself. (Current plans are for the RScheme incremental collector.) The interface is mostly finished; the only thing currently missing is a write barrier. - I rather doubt that Richard will have enough time to integrate the changes of the NewGC branch into the trunk. This would be a very worthwhile goal for anyone working on the GC, no matter which one. - You generally might want to look at http://www-pu.informatik.uni-tuebingen.de/users/sperber/xemacs/next-gener... to read up on some of the issues involved. It's quite dated by now, but most of the stuff there still applies. -- Cheers =8-} Mike Friede, Völkerverständigung und überhaupt blabla

On Tue, 22 May 2001, Robert Pluim spake: Was ALL_INTERIOR_POINTERS turned off? (If it wasn't, I'd expect to see lots of leakage). Too late :) the integration doesn't seem to be very difficult (not *that* much of XEmacs has changed in the interim); before the cold struck I was about 30% of the way through it. I expect to finish it off this weekend. -- `LARTing lusers is supposed to be satisfying. This is just tedious. The silly shite I'm doing now is like trying to toothpick to death a Black Knight made of jelly.' --- RDD

On Tue, 22 May 2001, Martin Buchholz stated: It seems that it needs a bit of a forward port, at least :) The real problem with GCPROs is that they are nasty to maintain; the type-accuracy they provide is good. It occurs to me that I could make GCPROs maintainable by making all the alloc_*() routines do an automatic GCPRO (well, do the same thing to the gcpro roots that GCPRO now does), and could write a fairly simple preprocessor that takes (one file of) the XEmacs C code and spits out a (transient) source file with UNGCPRO-alikes inserted at the end of each block where they are allocated. That would let us dump visible GCPROs in the source yet keep type-accuracy... That's my biggest worry, too. I don't really mind which GC I go to, really; my primary goals are zapping the GCPRO construct as it currently exists (although keeping its type-accuracy would be nice) and getting something with better VM behaviour. Both of these can be done without actually replacing the GC; it just seemed simpler to do that. Not yet. There are many architectures that GCC works on that libjava and parts of libobjc (the primary users of that gc) do not work on, and the boehm-gc in the GCC tree is a rather aged implementation. When version 6 comes out Hans has said he will move to making the GCC tree's copy of the collector the master copy; if I'm still using the boehm-gc in XEmacs by then, I'll migrate it to that version. This isn't true of anything except for cons cells and other sequences, and things like hash tables, is it? For `leaf' objects (string data &c), a conservative collector like Boehm's would wind up tracing them for pointers they couldn't possibly contain pointers to anything. (It's true that the gains from this bit over the *existing* allocator would be marginal; it reduces the negative effects of using a conservative collector substantially, though. It's a tradeoff; GCPRO versus the occasional malloc_atomic()... but the more I think about it the more it seems to me that a preprocessor that automatically inserts UNGCPROs is the right way to go; type-accuracy *and* maintainability, wow ;) ) True. I think, for the first implementation, I'll go with -- Richard's infrastructure (general modifications), forward-ported to 21.4; they look praiseworthy, and the forward-porting job doesn't look too enormous -- A bitmapped version of the current GC -- and, if I can get it to work --- and I damned well will ;) --- a GCPROizing preprocessor, so we can dump *visible* evidence of GCPRO (that being what does the harm; we can forget it exists completely if it is automatically maintained!) (I might even be able to make it so that only one GCPRO/UNGCPRO operation needs to be done for an entire stack frame, because the preprocessor knows exactly what Lisp_Objects have been allocated in a given frame, so it can insert a single operation that does all the work...) Agreed and agreed (see above). Absolutely. Of course! Wonderful book it is, too. (I originally bought it many moons ago with the explicit intention of using it to improve XEmacs's GC, but it's taken me this long to assimilate it properly.) -- `LARTing lusers is supposed to be satisfying. This is just tedious. The silly shite I'm doing now is like trying to toothpick to death a Black Knight made of jelly.' --- RDD

Nix> The real problem with GCPROs is that they are nasty to maintain; the Nix> type-accuracy they provide is good. It occurs to me that I could make Nix> GCPROs maintainable by making all the alloc_*() routines do an automatic Nix> GCPRO (well, do the same thing to the gcpro roots that GCPRO now does), The alloc routines know what they're allocating, but they don't know about the references to the memory being allocated. Unless you do something very radical, you have to gcpro the address of the Lisp_Object on the stack that is about to receive the pointer to the allocated memory. This means that Fcons can't gcpro anything. Perhaps I'm missing something. The standard tricky thing about implementing a preprocessor that adds gcpros automatically is that sometimes the Lisp_Objects that have to be protected aren't even declared - they're temporaries. For example return Fcons_user1 (Fcons_user2 (Fcons (Qnil, Qnil))); This has to be expanded into { Lisp_Object x = Qnil, y = Qnil, z = Qnil; struct gcpro gcpro1, gcpro2, gcpro3; GCPRO3 (x, y, z); x = Fcons (Qnil, Qnil); y = Fcons_user2 (x); z = Fcons_user3 (y); UNGCPRO; return z; } (this example could of course be optimized - but this is the obvious translation) (this example also shows how incredibly ugly gcpros are) The current source code does a lot fewer GCPROs than the above, because of detailed knowledge of various functions (can they gc, or could they return a fresh object). It is possible to determine this kind of information from a global analysis of the source code, but it won't be easy. In particular, the C preprocessor will tend to make the source more resistant to automated understanding by your gcpro preprocessor. E.g. #ifdef MULE #define FROB(x) Fcons_user1 (Fcons_user2 (Fcons (x, x))) #else #define FROB(x) Fcons_user3 (Fcons_user4 (Fcons (x, x))) #endif ... return FROB(Qnil); Nix> and could write a fairly simple preprocessor that takes (one file of) Nix> the XEmacs C code and spits out a (transient) source file with Nix> UNGCPRO-alikes inserted at the end of each block where they are Nix> allocated. That would let us dump visible GCPROs in the source yet keep Nix> type-accuracy... Nix> That's my biggest worry, too. I don't really mind which GC I go to, Nix> really; my primary goals are zapping the GCPRO construct as it currently Nix> exists (although keeping its type-accuracy would be nice) and getting Nix> something with better VM behaviour. Both of these can be done without Nix> actually replacing the GC; it just seemed simpler to do that. Nix> Not yet. There are many architectures that GCC works on that libjava and Nix> parts of libobjc (the primary users of that gc) do not work on, and the Nix> boehm-gc in the GCC tree is a rather aged implementation. Bad news. Nix> When version 6 comes out Hans has said he will move to making the GCC Nix> tree's copy of the collector the master copy; if I'm still using the Nix> boehm-gc in XEmacs by then, I'll migrate it to that version. Very good news. Nix> This isn't true of anything except for cons cells and other sequences, Nix> and things like hash tables, is it? For `leaf' objects (string data &c), Nix> a conservative collector like Boehm's would wind up tracing them for Nix> pointers they couldn't possibly contain pointers to anything. Nix> (It's true that the gains from this bit over the *existing* allocator Nix> would be marginal; it reduces the negative effects of using a Nix> conservative collector substantially, though. It's a tradeoff; GCPRO Nix> versus the occasional malloc_atomic()... but the more I think about it Nix> the more it seems to me that a preprocessor that automatically inserts Nix> UNGCPROs is the right way to go; type-accuracy *and* maintainability, Nix> wow ;) ) Sorry, I meant there's little scope for a Boehm mark() to be better than the existing mark() because the existing one, via lisp object type mark methods, already knows exactly which object components might contain other Lisp_Objects. The mark bitmap won't help you much during the mark phase, in fact it ought to make things worse. The place where the Boehm gc shines performance wise is not during gc at all, but in the fact that gcpros can be dispensed with entirely while the mutator runs. Normally the advantage of mark&sweep over reference counting is: not slowing down the mutator. But gcproing does slow down the mutator. But I really don't know how much overhead we have from GCPROing, and it's hard to measure. My random guess is that 10% of the runtime of lisp function calls that do no work (i.e. return immediately) is taken up with gcproing. Nix> True. I think, for the first implementation, I'll go with Nix> -- Richard's infrastructure (general modifications), forward-ported to Nix> 21.4; they look praiseworthy, and the forward-porting job doesn't Nix> look too enormous Nix> -- A bitmapped version of the current GC Nix> -- and, if I can get it to work --- and I damned well will ;) --- a Nix> GCPROizing preprocessor, so we can dump *visible* evidence of GCPRO Nix> (that being what does the harm; we can forget it exists completely Nix> if it is automatically maintained!) Harder than it looks, as I try to point out above. I'll be very impressed if you can produce a reliable GCPROizing preprocessor. I think you'll have to run the source through the C preprocessor first, which means your gcproizer has to run every build. Using the standard Unix utilities, this will be hard (i.e. you'll want to use something like perl or python). Most free software projects don't have such dependencies for non-maintainer-mode, but I think the time might be right to introduce such a dependency for xemacs. Nix> (I might even be able to make it so that only one GCPRO/UNGCPRO Nix> operation needs to be done for an entire stack frame, because the Nix> preprocessor knows exactly what Lisp_Objects have been allocated in Nix> a given frame, so it can insert a single operation that does all the Nix> work...) I like your plan very much. ObAdvice: Have you read the internals manual section that deals with gc?

Martin> I think you'll have to run the source through the C preprocessor Martin> first, which means your gcproizer has to run every build. Using the Martin> standard Unix utilities, this will be hard (i.e. you'll want to use Martin> something like perl or python). Most free software projects don't Martin> have such dependencies for non-maintainer-mode, But we have plenty of library dependencies which are actually harder to satisfy in a lot of setups. Martin> but I think the time might be right to introduce such a Martin> dependency for xemacs. Yes, but preferably not on Perl or Python. You want an actual C frontend for this. I already have most of one lying around here for the job. -- Cheers =8-} Mike Friede, Völkerverständigung und überhaupt blabla

[Sorry for the delay; a cold. I didn't feel I could respond to this until I was capable of thinking again...] On Tue, 22 May 2001, Martin Buchholz said: Gaah. You're right, of course. (Stupid of me.) No, of course not; but this doesn't add much complexity to what I'm planning --- which probably indicates that it's hugely overcomplicated already :) Absolutely. This was almost exactly the transformation I was expecting to have to apply, although I intend that the GCPRO and UNGCPRO macros don't survive in their current form with this preprocessor; the limits on the numbers of objects that can be GCPROed in one block *will* go away, for instance; it irks me. (Of course their replacements will do the same thing, and will probably be the same code, just emitted already expanded.) ... and probably the one I'd apply. A good reason to automate away their creation then, so no humans have to look at them :) This is a very bad idea, I think. It means that much of the XEmacs source code has detailed knowledge of the internal implementation details of other parts of the source code; this reduces stability ('cos some *will* be missed) and drastically increases the difficulty of changing anything. I think this should be handled in a similar way to the way GCC works out what registers are going to be used in subfunctions; assume they'll all be clobbered. The expenditure of time is minimal, in any case. (If the time wastage gets extreme in some hot spots, we can optimize the hot spots; but in general I think that every stack slot that holds a Lisp_Object should be protected. XEmacs isn't GCPRO-bound, it's GC-bound ;) ) I'm not even going to *try* to do that. It's probably impossible in the general case, anyway (I'd need to think about it, but I'd be surprised if that kind of detail of global analysis didn't reduce to the halting problem.) I'm going to stitch this in after the preprocessor has run on a given source file. (I'd have to preprocess it anyway, so it seems sensible to only preprocess it once!) Oh, FWIW the preprocessor should be able to handle just about anything, 'cos I'm cheating. I'm taking c-parse.y and cpplex.c from GCC, and tearing them down into something that can unambiguously spot C blocks (`stmts_and_decls'), assignments (`expr_no_commas'), Lisp_Object variable declarations (`decl' nodes), and temporaries. Everything else that I can throw away I will; we don't need a complete C parser, let alone the Objective C parts (sorry, Ovidiu ;) ). In one respect it'll be more complicated than GCC's parser; it needs to maintain knowledge of where in the pre-lexed sources a given lexical component comes from, so we can rewrite the temporaries; but it shouldn't be terribly hard. (I think GCC also contains some temporary-rewriting stuff I can pinch from; building a temporary is an RTL-level equivalent of what this preprocessor would have to do on the source code level. Fun. (But if this were C++ it would not be fun, it would be torment...) Hmm. Why is that? The current garbage collector has appalling VM behaviour; every Lisp_Object is sucked back into memory, even if they cannot contain other Lisp_Objects. If a mark bitmap is used, a Lisp_Object will only be accessed for pointer tracing; if it cannot contain another Lisp_Object, it won't be touched. I'll admit that I haven't instrumented the existing gc to find out what percentage of objects are accessed only to set the mark bit; I'll do so as soon as this cold has worn off. (I'll feel a right idiot if the answer is something like 1%, too...) However, a (not huge) general slowdown like that is hard for humans to spot. What humans definitely *do* spot (well, I know I've spotted it, and so have friends of mine, new XEmacs hands and old) is the total stoppage we get whenever GCing runs and the world freezes except for the hammering of the disk :( if XEmacs is totally in swap, it can be frozen for a minute or more while GC runs. GCPROs suck, but the only ways we can eliminate them are conservative collection (not very portable and can leave cruft around, as you said), some really hefty global analysis (I'll leave that for a braver soul), or a compiler that knows how to emit typing information (and while we could probably get an option added to GCC that would emit such information, I don't think we can decree that XEmacs can only be built by GCC-3.1 and above...) Given Moore's Law and the fact that machines with small amounts of memory are getting steadily harder to find and steadily harder to run late versions of XEmacs on (and probably don't see many upgrades anyway), I am happy to *increase* the number of GCPROs substantially, to ensure that we can freely modify arbitrary functions without triggering GC hell. After all, GCPROs are really quite efficiently implemented; it's not as if they call malloc() or something like that. (As a *structure*, the gcprolist is praiseworthy. It's just the manual way it's kept updated that's disgusting and restrictive.) We have the advantage of a pre-existing free software project that is quite capable of understanding C code (GCC), and which does far more elaborate transformations than this preprocessor will have to; and I'm happy to nick code from it. (I'll probably mention such large-scale borrowing on the GCC list if I do it, if Ovidiu doesn't beat me to it...) The GCC C parser is actually quite neat, as such things go... Yes. Not a chance; this'll be in C. Parsing is what yacc is good for, and as usual we'd ship the results of running yacc on the preprocessor's .y files, so the builders won't need yacc. (Plus, GCC is already written in C, and while I'm willing to attack c-parse.y and cpplex with a stone axe to form the core of this preprocessor, I'm really not willing to translate them into Perl; I have more taste than that ;) ) No need. (I wouldn't want that dependency for purely personal reasons; one of the sites I run xemacs on has no perl or python, and I can't install either because I just barely have the space for xemacs. I know where *my* priorities lie ;) ) Yes, of course, long before I mentioned this plan here, when I harboured the secret desire that the fix to XEmacs's GC performance would be simple. (Then I read of GCPRO and felt quite ill...) -- `LARTing lusers is supposed to be satisfying. This is just tedious. The silly shite I'm doing now is like trying to toothpick to death a Black Knight made of jelly.' --- RDD

On Wed, 23 May 2001, Mike Alexander stipulated: The CodeSourcery folks are rewriting the C++ parser into something that's actually maintainable; when that's done I could produce a C++ variant of the preprocessor (let's give it a name: gcpp. The C++ variant would presumably be gcpppp...) That's irrelevant; this preprocessor's not trying to compile the code, just to pick out declarations and function calls and do a little simple rewriting. As long as the C used in the MSVC build (and headers) isn't gratutiously nonstandard, the parser should be happy. (And if it isn't I can rejig it until it is; MS's changes to C can't be *that* bad, can they? Unless they've broken block structure or declarations...) (Aside: I'm getting a little worried by the length of this cc list...) -- `LARTing lusers is supposed to be satisfying. This is just tedious. The silly shite I'm doing now is like trying to toothpick to death a Black Knight made of jelly.' --- RDD

Martin Buchholz <martin(a)xemacs.org&gt; writes: It's OK to make XEmacs a little bit slower but eliminate disruptive pause is the reason behind adopting incremental GC. -- Yoshiki Hayashi

On Sat, 26 May 2001, Ben Wing uttered the following: Not strictly true; in some places GCPRO can contribute more than constant time; e.g. inside recursive functions, or functions that are indirectly called recursively. (There aren't many such yet that I can see; in one place the comment says this is because it was too hard to keep the GCPROs straight... :( But, yes, fixing the algorithms is always a better move. I will, of course, restrain myself from replacing GCPRO with something that is insanely slow (that, say, does dynamic allocation or something)... (One viewpoint is that GCPROs are extra complexity of a totally mechanical nature and so they should be dealt with by computers. That's what they're good at.) -- `Technology is meaningless. What matters is how people _think_ of it.' --- Linus Torvalds

On Sat, 26 May 2001, Ben Wing yowled: Thank you! Hmm. Interesting. I agree with most of what that says, and it's nice to see I'm not going down an untrodden thought-path. I disagree that a self-hosting preprocessor (i.e. a preprocessor whose language is Emacs Lisp) would be terribly difficult. The Lisp reader and string manipulation code is fairly self-contained as it goes; rendering it completely so ought not to be too hard. All that would need to be done after that is to produce a bootstrap elisp interpreter by basically taking lread.c and eval.c and tying them together with a framework that simply doesn't bother to GC at all (so the lack of GCPROs in the interpreter is irrelevant). Then the makefile uses that to generate a `real' preprocessor by running it over the Lisp reader, runs *that* preprocessor over the Lisp reader and diffs the two. If there are no differences, we know the preprocessor works, and the makefile proceeds to use it to build all the rest of XEmacs. Self-hosting has a single really dull bit of slogwork (the initial construction of the bootstrap Lisp interpreter), but that only needs to be done occasionally, and it doesn't matter if it gets out of synch with the real Lisp interpreter, as long as it can understand enough of the subset of the language used by the preprocessor to preprocess itself. I also disagree that replacing GCPROs with a conservative collector is necessarily a good idea; I thought so until recently, but if it's already been tried and it was really inefficient; if incorrect identification of possible pointers on the stack was leaking a lot of memory, we can't do that and we need GCPRO. (Or rather, we need automatically inserted GCPROs.) I also don't think we can say anything much about how much time automatically inserted `GCPROs everywhere' can take until we've tried it; updating a GCPRO list is not a very expensive operation, after all. I'm doing *that* now, if you mean merging the gc branch back to the head. (At least, I'm merging it to 2.4, and that's nearly the head as volume-of-changes goes.) For me, the *really* fun part will be when I can run XEmacs without either weird GCPRO-related crashes (rare, but how can we ever be sure all of them are gone without automated insertion?) or massive disk churning due to the garbage collector's mega-paging :) Wrong order. I do the docs *first*; that way I find design faults before the code is written, and don't have to write the docs later. (I just have to revise them.) (In this case it's complicated by needing to understand the system before I write the docs; I'm counting forward-porting Richard's changes as `writing the docs'; see below.) What's an `important configuration'? I can test on i586-pc-linux-gnu, sparc-sun-solaris2.5.1, hppa2.0-hp-hpux10.20, alphaev56-dec-osf4.0d, i586-pc-cygwin32, and I think I can manage an NT-native Visual C++ build too. I'll be frequently testing on i586-pc-linux-gnu, as it's my development box and I have easy access to it; the others will get big periodic tests. That happens first :) and any changes I make should I think go into the garbage collection and memory allocation nodes in the internals manual. I *hate* rough edges :) In practice I think configuring with code that rips out GCPRO would have to mean that the GCPROs stay and gets sedded out by configure, or #defined away, and of course that removes all the benefit of removing the GCPROs (said benefit being maintainability, not anything the user sees). GCPRO-removal is one of those things that there's little point configuring out. (Richard's changes, again, probably can't be configured out. Too big, too pervasive. But I'm not sure about this, and I'll *try* to make them configurable out...) The GC changes with functional effect themselves; yes, I plan to make that configurable --- although if it causes functional changes I'll count that as a bug. 3? You're lucky. I normally multiply by 5 ;) This is just allowable paranoia :) `Anything that can go wrong, will'; yes, sure, but then lots of things you didn't think could go wrong will go wrong *too*. Forward-porting Richard's changes is pass 1. (And testing it, and documenting it... it appears to be completely undocumented; not even the changelogs are updated. Fixing *that* latter should be fun; I'd like to get real changelog entries if they exist, but... the CVS changelog entries don't really count, they're not detailed enough.) The forward-porting /per se/ shouldn't be too hard. Just a lot of slogwork. Bringing the docs up to date will probably take at *least* as long :) I'm thinking of the forward-port as a documentation job as much as anything else. Thank you! I may well take you up on that :) I'm perenially disorganized unless I force myself to be organized. As a result, on anything I care about I organize *first* so I can forget about it later on. I already have a plan of attack for the merge, and I know what order I'm doing the rest in and why. (It might change, as is usual with such things...) I think of this as `rewarding', not `difficult'. I *like* cleaning up horrible messes, because I hate the messes so much (GCPRO sounds like a good candidate ;) ) I may well lob my plan of attack in your direction so you can laugh at it for being hopelessly impractical then :) -- `Technology is meaningless. What matters is how people _think_ of it.' --- Linus Torvalds