Re: Forthcoming revisions to the garbage collector; crazy huge hack

Nix> I'm currently in the middle of a hack of crazy size, jumping in at the Nix> deep end of XEmacs development, in order to familiarize myself with all Nix> the code. Richard Reingruber is working on a new GC. It can be found in the CVS branch NewGC-21-2. Richard is quiet. Search for his few postings on xemacs-beta. Richard is a student of Michael Sperber. Yoshiki Hayashi has written a copying gc for xemacs recently. It is not yet ready for prime time. His is precise, i.e. he has to maintain all the gcpros, and actually add a few, since the copying involves finding and updating ALL the places on the stack where Lisp_Objects live. Most of the recent maintenance of the existing gc has been by Olivier Galibert and myself. I've taken a look at the Boehm gc in the past, considering its possible use in XEmacs. I rejected using it because: - it's non-portable (reading its portability layer is frightening) - the concurrent or incremental gc feature isn't even intended to be portable currently, but incremental gc is a key attraction. - The big advantage of boehm gc is ditching the gcpros. But this is only a long-term advantage. Probably xemacs is fairly gcpro-clean these days. Incorporating boehm gc is going to be extreme short-term pain for long-term gain. However, the gcc project has recently started using boehm gc. Since gcc is very portable, they must be making any necessary portability improvements to boehm gc as they go. Nix> I'm doing what should have been done a *long* time ago; ditching the Nix> garbage collector and replacing it with a nicer one. I think everyone Nix> can agree that XEmacs's garbage collector currently sucks really rather Nix> hard; I don't think there are any ways to worsen its VM behaviour or Nix> intrusiveness if we tried. Everyone thinks the current gc sux, yes. Nix> The one I've picked is the Boehm collector; it's not terribly portable, Nix> but that can be fixed fairly easily --- and it lets us dump bloody GCPRO Nix> and all the uglinesses it has spawned over the years (blocking string Nix> compaction, Lispification in redisplay and many other things). Nix> The Boehm collector only has one downside that I can see; unportability Nix> (it's not portable to some of the more obscure platforms that XEmacs Nix> runs on). Upsides include: Nix> - better VM behaviour (mark bits kept in a bitmap, and GC_malloc_atomic() Nix> to state that certain kinds of objects cannot contain pointers). As Nix> it is most of XEmacs is forced to stay memory-resident all the time Nix> by mark bit setting; that proportion will reduce to just those Nix> parts that contain pointers that must be traced. This is my primary Nix> motivation for this because I run XEmacs on some fairly memory-poor Nix> machines and I'm fed up of waiting for bloody GC to finish. Hmmm. It seems to me mark() has to examine most of the heap anyways. Unless you do a lot of work, boehm gc might end up examining even more memory than the existing mark. Most of the guts of existing Lisp_Objects are other Lisp_Objects, so there isn't a whole lot of scope for GC_malloc_atomic() to be a big win. Nix> - mark bit sanity; we can reclaim at least one bit from many objects, Nix> and might even get lightweight cons cells back (I'm not sure if one Nix> bit saved is enough for that though). Certainly we can junk the myriad Nix> different ways we have to mark things; this is very much improved Nix> recently but the boehm-gc can fix it completely we can also put the mark-bits into a bitmap without complete boehmification. Understanding and extracting that code from boehm gc would be a much smaller and still very instructive project. Nix> - and last but not least, it's written and actively maintained by Nix> someone who's very accessible and who's one of the best GC hackers Nix> there is, and it is itself acknowledged as probably the best Nix> general-purpose C garbage collector in existence. I share your admiration of Hans. Nix> I'll be paying *no* attention to unexec() and friends in this, at least Nix> at first; I'll keep the portable dumper working, but if unexec() is Nix> totally broken by some interaction with the boehm gc, I will not mourn Nix> unduly. Agreed. Nix> I'm doing the changes to 21.4.3 at first, because I know it's pretty Nix> stable otherwise, so that any crashes I may encounter are my fault. Once Nix> I've got it working fairly stably in there, I'll post the patches (and Nix> yes, I'll split the patches up into pieces; I don't want the Nix> GCPRO-removal to drown out the interesting stuff) and let everyone rip Nix> into them... and then I guess I'll forward-port it to 2.5, and everyone Nix> can let GCPRO fade into the mists of memory (maybe with the aid of Nix> a psychiatrist). I encourage you on your adventure. However, some advice: - communicate with Richard and Yoshiki. - consider implementing mark bitmaps only in the existing gc. - investigate the version of boehm gc from the gcc project. - of course, you do have the blue book called "Garbage Collection", right? Good Luck.