Call for discussion: DLL handling

Introduction ------------ In order to do the right thing with dynamic loading technology in XEmacs, we need to get concensus on the best way of implementing this. There are several ways in which we can go about it, and all have their merits and drawbacks. It is my intention that this document will discuss all of the mechanisms availible, to give you all of the facts, so that we can choose between the different implementation mechanisms. Yes, this is rather long-winded at times, but I hope that more information will help more than less, so please bear with me :-) Before going into the actual mechanisms, it is priudent to examine WHY we think we need or want DSOs in XEmacs. It is already possible to extend the editor using Lisp, and it is also possible to extend it in C when you are compiling the editor. What makes DSOs important is the fact that compiled C code can be selectively loaded into the editor if and when it is needed. This will allow us to (potentially) move parts of the editor into DSOs, thus making the core editor smaller and faster to load, while retaining the ability to extend its functionality. It is important to be able to write some extensions in C, rather than Lisp, for speed purposes. While the Lisp engine is fairly quick, there are cases (such as the HTML parser) where a great benefit can be gained from writing it in C. But not everyone wants to use their editor for a browser, so not everyone should have their editor weighted down with an HTML parser. Hence the need for DSOs. William Perry has already started the work for DSO support by providing the sysdll.c functionality, which abstracts the actual mechanics of loading a DSO. This work is pretty much complete as it currently stands in XEmacs, but it is not really used. The code that is currently in dll.c doesnt really DO anything yet, and it doesnt provide a clean interface for module developers. This is the works that I have done, which now needs to be finalized and put into the source base. We also need to work with Richard Stallman and the GNU Emacs team to try and make sure that the work we do is compatible with that version of Emacs, although there will be some technical difficulties in that regard (more on this later). What I have done is to flesh out dll.c to provide an actual working implementation of DSO loading, using a variety of different models. We need to choose a model and stick with it. To make the right choice we need to understand all of the issues that affect that choice. One of my first goals is to make it easy for developers and contributors to use DSOs, and to provide a platform-independant mechanism for a developer to compile, install and test their modules. Secondly, we need to document the mechanisms we chose and provide as much information to the developer as possible. This will probably take the form of a new document which references Ben's internals document, although we could simply extend the internals document to cover DSOs. I prefer the former approach, as it will make the document less dependant on the actual Emacs implementation. Third, we need to try and keep things as similar to the current internal modules as possible, so that those people who are familiar with the Emacs internals can work in a familiar environment. If it ever comes to a trade-off between keeping existing internals developers happy and atracting new developers, it is my opinion that we err on the side of attracting new developers, so that the number of people extending Emacs can grow. Basic Architecture ------------------ Before getting into the low-down details, a word on the basic DSO architecture. Some of this is modelled after the DSO implementation for XFree86 Version 4, and some of my own thoughts and ideas. Each DSO is presented to Emacs as a single .so or .dll that is to be loaded. The actual file extension doesn't matter (for now). This module will contain the code required to implement the functionality the module intends to provide. When the moduel is loaded, either by the user interactively loading a module, or by some Lisp code being executed to load the module, the first thing the loader does is locate the file (more on this later, too), and then try to determine whether or not this is a propper Emacs DSO. The first thing it does is look for three special C symbols, which are all normal strings of type ``CONST char *''. These symbols are: CONST char *cemacs_mod_name = "whatever"; This variable is used to give the module a short tag or name, which is used later in resolving inter-module dependancies. This should be limited to 8 characters or less, and contain no spaces. CONST char *cemacs_mod_version = "1.0.0"; This is used to define the version of the module, in the form MAJOR.MINOR.PATCH, and is also used in dtermining inter-module dependancies. CONST char *cemacs_mod_title = "A sample module"; This is used to describe the module more fully, and should be a short string (less that 40 characters) and its primary use is in the display of the currently loaded modules. If any of these three strings is not present, the loader will reject the module as invalid. The last check for module validity is a C function, which is declared as: int cemacs_init (int task, char *arg) This function must be present for the module to be considered valid by the loader. If all 4 of these symbols are present, the loader will continue loading and initializing the module. All of the loading and initialization work is done through the cemacs_init() function, which is passed along an initialization task code (and optional argument pointer), during which time the function is expected to perform certain initialization tasks. The following basic tasks codes have been identified and defined: Task code 0 When cemacs_init() is called with task code 0, it means that the module is being loaded and initialized for the first time. The argument passed is the fully qualified path name fo the module that is being loaded. During this task code, the module is expected to load in any dependant modules, by calling cemacs_load_module(). Each of those modules can in turn load other modules, etc. If this task code does not return 0, then all of the modules loaded after it are discarded, and the module load fails. Task code 1 This is the moral equivalent of the syms_of_XXX() function, and this is where the code should announce new SUBRs. This is done by calling DEFSUBR() (or CDEFSUBR(), see below) for all those SUBRs the module defines. The return code from this phase is ignored. Task code 2 This is similar in intent to task code 1, but is the internal equivalent of the vars_of_XXX() function. During this task code, all new variables are defined and initialized to default values. Given the above, each module would contain code similar to the following skeleton code. This is not well commented, as I just want to get the concepts across: /* Emacs dynamic module foo */ #include "some_stuff.h" CONST char *cemacs_mod_name = "sample"; CONST char *cemacs_mod_version = "1.0.1"; CONST char *cemacs_mod_title = "Sample DSO module"; int cemacs_init (int task, char *arg) { int ret; switch (task) { case 0: /* Load in other modules and do internal inits */ ret = cemacs_load_module ("sample2.so", "sample2", "1.0.1"); if (ret != 1) return 1; return 0; break; case 1: /* Declare SUBRs */ DEFSUBR(Fsample_function); return 0; break; case 2: /* Delcare variables */ DEFVAR_LISP("sample-string", &Vsample_string); DEFVAR_BOOL("sample-boolean", &sample_bool); return 0; break; } return 0; } This architecture allows us to keep the same basic model, and extend it as time goes by by adding new task codes or changing the semantic interpretation of each task code. It is very open-ended and very eacy to implement. The names of the variables and function are, of course, open to debate, but the concepts are the most important part. Lets not start debating the actual names now (that can come in round two of the debate). Declaring new SUBRs ------------------- One of the main reasons people will write modules is to implement new functionality in the editor, or provide a faster mechanism for implementing existing functionality. This will involve creating new internal subrs and variables, and this is where one of the biggest debate points lies. In the existing code, the developer uses the DEFUN() macro to define a new C level subroutine. This results in a static structure being set up and populated with static values. This model does not lend itself well to supporting DSOs, due to the way PIC addresses are stored in shared objects. (If you need more details on why this is an issue, please feel free to contact me outside the scope of this discussion. For now, please just accept my word that there is a problem here). I saw two ways around this problem, and implemented both. On a recommendation from Hrvoje Niksic I dropped one of them, but I would like to open this up to debate again, to get input from others. The first mechanism I implemented was to define a new set of macros for defining SUBRs and variables. This used the same macros as the internal source did, except that the macro names were prepended with a 'C'. This was inspired, in part, by my stand-alone efforts to add DLL support to XEMacs way before I knew that work was already being done in this area, and I was writing a package I called "CE-Emacs", for C Extensible Emacs. In this model, when you declare a new subr, you would use the following syntax: CDEFUN("sample-function", Fsample_function, 0, 0, "\ This is the documentation for the subr, as a C string.\n\ \n\ Note that the way the documentation is done is radically\n\ different to the way it is currently done with DEFUN()\n\, as we have a static C string, not a C comment acting as\n\ the documentation.\n" ()) { message ("Eureka! It worked!"); } Compare this to how the same function would be done as an internal module: DEFUN("sample-function", Fsample_function, 0, 0, /* This is the documentation for the subr, as a C comment. This documentation would be snarfed by make-docfile and added to the DOC file. We cant do this dynamically because the DOC file is a static entity. More in this later. */ ()) { message ("Eureka! It worked!"); } Aside from the fact that the macro name has changed from DEFUN to CDEFUN, the biggest change is in how the documentation is handled. We need to be able to get the documentation for a subr or a variable into the Lisp engine dynamically, so we obviously can't use C comments. We need a machanism for getting the actual documentation as a string into memory. Using the above mechanism, I was able to take the string and add it correctly to the structures and have the subr correctly documented. The second implementation I developed was to keep the macro names and syntax the same, and have the subr or variable's documentation during the load phase of the DSO. This would involve calls to a special function that would document the subr or variable during the appropriate load phase. To support this, I added Task Code 3, during which time the module is expected to ducment the SUBRs and variables it defined during task codes 1 and 2. For example, the following code would be appended to the end of the switch in the skeleton code above: case 3: /* Document everything */ CDOCSUBR(Fsample_function, "\ This is the documentation for sample-function.\n\ \n\ It is implemented as a large C string, and added during\n\ task code 3 of the module load life-cycle.\n"); CDOCSYM("sample-string", "\ This is a sample string.\n\ \n\ It is loaded form a DSO and documented during task code\n\ 3 of the module load life-cycle.\n"); CDOCSYM("sample-boolean", "\ *Sample boolean variable.\n\ \n\ Defined in a DSO and documented during task code 3.\n"); return 0; distinct phases of module initialization, and we can potentially make come of them optional. For example, we COULD make it optional to document variables and functions, as not documenting them can reduce the memory requirements. The down-side of this mechanism is that it increases the distance between a function or symbols definition and its documentation. This in turn makes it more likely that documentation and implementation semantics get out of sync. Yes, this is just a matter of programmer discipline, but it is a factor to be considered. A third alternative (and one I have not implemented) is to have variables and functions defined exactly the way they currently are, using DEFUN(), DEFVAR_LISP() etc, with their comments in C as they currently are, and then to extend the module build process to include calls to make-docfile or some other suitable documentation snarfer that can write out a special init.c file, which can convert the documentation into C strings which we can then use to implement the actual documentation when the module is loaded. This could be extended to automatically creating the cemacs_init() function, or simply to creating special variables which we handle correctly during module load time. This has the benefit of being the least impact change, without breaking the basic DSO architecture. There are some deeper technical issues which would need to be resolved for this technology to be unleashed upon the Emacs world, such as avoiding having to xstrdup() documentation strings, but these are relatively minor implementation details. Right now, I have all of the basic infrastructure in place. I have functions for loading the modules, and I have sample modules which load and run exactly as expected. I have written documentation around the module creation process, but it is incomplete (and it cannot be made complete until such time as other issues are solved). One of the biggest issues facing us is how we can co-exist at a source code level with GNU Emacs. In the documentation I have written, and the shell scripts used to compile modules, I define XEMACS, so module writers can always surround XEmacs specific code with suitable #ifdef XEMACS lines, and similarly for GNU Emacs (whose define is simply EMACS). However, due to the internal differences between the two editors, this may be more complicated than it needs to be. The fact that XEmacs has different data types, different data structures and different ways of implementing certain basic functions will make life miserable for the module developer. Now more than ever we need to try once again to bring about a better syncing up of XEmacs and GNU Emacs. It will be up to Stallman to decide what and how much he wants to accept from this work, and to try and implement ways of abstracting the differences between the two editors. But, this is a larger issue than I think we need to worry about right now. Right now, I would like to reach some concensus on which of the three implementions I have outlined above should be the one moving forward. Please vote for and debate a choice between the three, or come up with alternatives I have not thought of. To summarise, the three choices are: (A) Use DSO specific macros such as CDEFUN(), CDEFVAR_LISP() etc. (B) Use the existing DEFUN() and DEFVAR_() macros and have documentation handled during task code 3, or (C) Keep everything exactly as it is and extend the DSO build scripts to snarf the documentation strings correctly and construct a .c file. One last thing. Do we want shared objects loaded from the normal Lisp load path, or do we want to set up a new variable to contain the shared objects, perhaps having the default be in the OS dependant directory? If anyone has not followed the previous threads between Hrjove and myself, and you have not seen the documentation or dll.c implementation and other patches, please let me know and I will put them up somewhere for people to FTP and peruse. I would like to get a decision on this as soon as possible, so that I can continue with the documention, samples and build scripts. Is it realistic to set a dead-line of, say, Friday October 30 for a decision to be made? Should I send this stuff on to Stallman to look at too, and get him in on the comment loop, or do we just make a choice and present everything to him for integration into GNU Emacs feit acompli? I am "FSF-friendly", in that my copyright assignments are on file at the FSF, so I can appease that fear at the very least by assigning copyright to them. I also have disclaimers of interest from SCO on file at the FSF, so there is no conflict there either. Comments, suggestions and questions welcome. -- J. Kean Johnston | "Every day, computers are making people easier Engineer, SPG | to use" -- Source unknown. Santa Cruz, CA +---------------------------------------------------------- Tel: 831-427-7569 Fax: 831-429-1887 E-mail: jkj(a)sco.com