I too have thought about the idea of a Lispifying preprocessor for C, to avoid all the kludges necessitated by CPP and the broken C language. I've been tempted to make changes similar to the ones you propose. James, you have looked at other implementations that have done this, right? I believe one of the free lisps (CLISP?) does something like this. In my opinion, the disadvantages outweigh the advantages, at least for the changes you are planning. When the changes are done, no end user will see any advantage from the changes, except that build times will be a little slower. And the advantages for the developer are smaller than you might think. Mostly cosmetic. I doubt that the changes will succeed in making the module API that much more stable. With or without the changes, the stability of the module API depends on the will of the future xemacs maintainers. People often underestimate the impact of huge changes like this. There will be a ton of documentation that needs to be aligned with the new world. Will you remember to fix up the "make TAGS" target in the top-level Makefile? Will xemacs still be debuggable? Will .gdbinit keep working? Will xemacs know to use cc-mode for editing those funky .xc files? Will cc-mode understand the DEFSPECIAL foo ... as a function definition? Will func-menu? Will etags? Etc, etc. And some things will break that you cannot fix, because they're not part of xemacs. Like some developer's function cgrep { find . -name '*.[ch]' -print | xargs grep "$＠" ; } Many of these things are trivial, but for many of them, _others_ are paying the price. Everybody wants to change the world. But it's a lot harder to deal with _all_ the consequences. Now I didn't say we shouldn't have a preprocessor for C. But we better have a damn good reason. Getting rid of GCPROs seems to me a lot better reason than the "better API" reason. Curmudgeonly yours, Martin

Hrvoje> Martin Buchholz <martin(a)xemacs.org&gt; writes: Hrvoje> The *huge* disadvantage of using a custom preprocessor is that in that Hrvoje> case your code is no longer C code, which means that it cannot be Hrvoje> analyzed with the usual C tools such as lint or cflow, debugged with Hrvoje> GDB, etc. Well, gdb can be taught (perhaps) about the source code through #line directives in the generated source. To solve the problem of printing lisp variables like bell-volume, when C only understands bell_volume, we would have to teach gdb how to turn lisp variable names into C variable names, but that strains the capabilities of gdb's command language. Anyways, yes, the resulting program will be much harder to debug. And this preprocessor is supposed to make the developer's life easier.

Hrvoje Niksic writes: You fail to mention the potentially very great difficulty that would be felt by other important C tools named Hrvoje, Martin, James, and others!!! Seriously, I had to support code written by a fellow that wrote his own preprocessor (for an assembly language, but that doesn't matter). Basically, I inherited like 10,000 lines of code that was essentially in its own language. There were great plusses of his preprocessor -- a single statement did all the work for a procedure call -- but it was his language, and so debugging it was impossible (no text books available!) A better way to change the world might be to write elisp skeleton macros (using skeleton.el) that makes it real easy to insert C procedure definitions frameworks with all the interface stuff for elisp calling there already. In my mode verilog-mode.el, I have a AUTO feature that does local analysis of verilog code and then updates declarations based on other data that is in the procedure definition. Perhaps something could be written in elisp that would look at a C procedure, that is written according to a defined structure (I.E., has comments inserted at key points that say /* AUTOARG */ ) and update the routine definition to make it conform to what is needed for the c - elisp interface. Also a good idea. -- //' Michael McNamara <mac(a)verisity.com&gt; _ // Sr VP Technology 650-934-6888 \ // Verisity Design 650-934-6801 FAX \// <http://www.verisity.com> 408-930-6875 Cell -------------------------------------------------------------- Get my verilog emacs mode from <http://www.verilog.com>

Raymond Toy writes: Several years ago there was a project over at one of the German universities to write a Common Lisp to C Compiler (CLiCC). The last version (v0.6.5) claimed that it conformed to the Common Lisp specification in CLtL1 plus CLOS. The web site for the compiler is still active: http://www.informatik.uni-kiel.de/~wg/clicc.html A couple of years ago I used it to compile several small stand alone apps. I think that if someone were to pick up CLiCC and finish it (bring it up to ANSI Common Lisp compliance), it would probably make a good mechanism for developing something like XEmacs almost completely in Common Lisp. I intend to do something like this eventually. I would have already started, but all the inline comments are in German and I don't have the time to learn German just so that I can read the comments. Craig

Craig> Raymond Toy writes: Jerry> Somewhat ironically, I actually considered this preprocessor a step Jerry> towards an Elisp-to-C translator, using the source to be preprocessed as Jerry> the target of that translator. I'll rethink that, too. Craig> http://www.informatik.uni-kiel.de/~wg/clicc.html Craig> A couple of years ago I used it to compile several small stand alone Craig> apps. I think that if someone were to pick up CLiCC and finish it (bring Craig> it up to ANSI Common Lisp compliance), it would probably make a good Craig> mechanism for developing something like XEmacs almost completely in Craig> Common Lisp. I intend to do something like this eventually. Craig> I would have already started, but all the inline comments are in Craig> German and I don't have the time to learn German just so that I can Craig> read the comments. Then why not use gcl then? It's still being actively worked on (probably in the context of maxima, though). And comments are in English. (This is what keeps me from hacking on clisp: German comments.) Making it fit with xemacs might not be so easy.... Ray

Craig> The last time I tried gcl it claimed CLtL1 compatibility which doesn't Craig> include CLOS. Has it moved closer to ANSI Common Lisp? I already I don't follow gcl but I don't think it's any closer. It's certainly still missing lots of ANSI stuff like defpackage, loop, etc. CLOS is still implemented with PCL (but then CMUCL uses PCL too). However as a backend for xemacs it's probably doesn't matter since xemacs code probably wouldn't use any of missing ANSI things. :-) Ray

Raymond Toy <toy(a)rtp.ericsson.se&gt; writes: Because it's highly questionable how useful Gcl is. To be useful in short-term future, an elisp-to-C translator should not try to reimplement XEmacs from scratch or to turn Elisp into Common Lisp (or, for that matter, CLtL1). My favorite strategy for elisp-to-C is the one (I think) originally proposed by Kyle Jones some time ago: look at the opcodes that our current compiler generates. Now look at the code in bytecode.c that interprets it. A simple translator would simply take byte-compiled code and translate it into C that looks like bytecode.c, but inlined to directly *do* the stuff that P-code wants to do instead of interpreting the P-code. Although this looks simple-minded, I believe it would be a *huge* speed win over the current virtual machine. First, the cost of interpretation would be removed, which is by itself a big win -- the huge `case' statement would finally be eliminated. Secondly, such "unwound" C code would be processed by the C optimizer which would remove the many many redundant checks and thus make the code run really fast. The advantage of this strategy is that it opts for huge speedup in a *realistic* environment, i.e. it doesn't require rebuilding 100% of XEmacs to work. Furthermore, it builds on the work already invested in the byte-code compiler.

Craig> Raymond Toy writes: >> >> Craig> The last time I tried gcl it claimed CLtL1 compatibility which doesn't Craig> include CLOS. Has it moved closer to ANSI Common Lisp? I already Craig> Then I would have a difficult time calling it "actively maintained". Well, Clisp claims CLtL2 and ANSI compatibility but I think there are some things that are specified by ANSI that will never be done because they think it's wrong. So is clisp not actively maintained? Even though there have been 2 releases in the last couple of months? Craig> If those things were available in the development environment then Craig> XEmacs might use them :-) Craig> Modes are much easier to implement with an object system. Craig> Zmacs implements each mode (both major and minor) as Craig> flavors (Zetalisp object system). It makes it easier to Craig> inherit things that are common to all modes. But if modes are required to use the object system who's going to port over all the old code? Not me. Craig> Packages would be helpful in segregating the different lisp packages. People already do this by prefixing stuff. Having packages won't change existing code. That's why CLtL1 isn't so bad. There's so much old stuff that no one really wants to port it all over, and no one wants to maintain a version for XEmacs and Emacs. I think that's hard enough already. Besides, Hvroje's message about Kyle's approach is the right solution to me. No user-visible changes to the lisp code, but potentially major changes in speed. (That's why I stopped using VM. Too slow to load up my 100 MB mail files. If VM becomes fast, I might switch back.) Ray

Hrvoje> My favorite strategy for elisp-to-C is the one (I think) originally Hrvoje> proposed by Kyle Jones some time ago: look at the opcodes that our Hrvoje> current compiler generates. Now look at the code in bytecode.c that Hrvoje> interprets it. A simple translator would simply take byte-compiled Hrvoje> code and translate it into C that looks like bytecode.c, but inlined Hrvoje> to directly *do* the stuff that P-code wants to do instead of Hrvoje> interpreting the P-code. Hrvoje> Although this looks simple-minded, I believe it would be a *huge* Hrvoje> speed win over the current virtual machine. First, the cost of Hrvoje> interpretation would be removed, which is by itself a big win -- Hrvoje> the huge `case' statement would finally be eliminated. Secondly, such Hrvoje> "unwound" C code would be processed by the C optimizer which would Hrvoje> remove the many many redundant checks and thus make the code run Hrvoje> really fast. Hrvoje> The advantage of this strategy is that it opts for huge speedup in a Hrvoje> *realistic* environment, i.e. it doesn't require rebuilding 100% of Hrvoje> XEmacs to work. Furthermore, it builds on the work already invested Hrvoje> in the byte-code compiler. I've done some thinking about this in the past. This is a very EASY translator to write compared to some of the other projects being considered. The hard part is integrating this into xemacs. Like figuring out the shared object loading mechanism across platforms. Should all the package lisp be compiled to .so at installation time, or should there be a JIT that optimizes hotspots at run-time (it would have to call the compiler transparently to do so, however). Don't expect it to be a huge win, however. All the optimized machine code will take up more memory than the bytecodes. And many elisp operations like (setq i (1+ i)) will not be compilable to C i++, but instead will still require expensive frobbing of the lisp symbol `i'. A function call like (foo) will still require the notoriously slow Ffuncall, even if foo is a subr, due to the dynamic nature of lisp. But this is a great hack.

Martin Buchholz <martin(a)xemacs.org&gt; writes: Yes. I'm not convinced that integration is harder than writing the translator, but anyway... The former. I hate the JIT idea, especially in the context of Emacs! I never expect such an optimization because we don't have lexical scoping, declarations, type inference, etc. But I still believe that even without that optimization the translator would be a huge win.

Hrvoje> My favorite strategy for elisp-to-C [...] Yet another strategy is writing a metacircular compiler. A student of mine has one. Preliminary benchmarks indicate it's pretty fast even running *on top* of a byte-code implementation of Scheme. We intend to move to a faster Scheme implementation within the next few weeks. -- Cheers =8-} Mike Friede, Völkerverständigung und überhaupt blabla