Introduction
------------
In order to do the right thing with dynamic loading technology in XEmacs,
we need to get concensus on the best way of implementing this. There are
several ways in which we can go about it, and all have their merits and
drawbacks. It is my intention that this document will discuss all of the
mechanisms availible, to give you all of the facts, so that we can choose
between the different implementation mechanisms. Yes, this is rather
long-winded at times, but I hope that more information will help more than
less, so please bear with me :-)
Before going into the actual mechanisms, it is priudent to examine WHY we
think we need or want DSOs in XEmacs. It is already possible to extend the
editor using Lisp, and it is also possible to extend it in C when you are
compiling the editor. What makes DSOs important is the fact that compiled
C code can be selectively loaded into the editor if and when it is needed.
This will allow us to (potentially) move parts of the editor into DSOs,
thus making the core editor smaller and faster to load, while retaining the
ability to extend its functionality. It is important to be able to write
some extensions in C, rather than Lisp, for speed purposes. While the Lisp
engine is fairly quick, there are cases (such as the HTML parser) where a
great benefit can be gained from writing it in C. But not everyone wants
to use their editor for a browser, so not everyone should have their editor
weighted down with an HTML parser. Hence the need for DSOs.
William Perry has already started the work for DSO support by providing the
sysdll.c functionality, which abstracts the actual mechanics of loading a
DSO. This work is pretty much complete as it currently stands in XEmacs,
but it is not really used. The code that is currently in dll.c doesnt
really DO anything yet, and it doesnt provide a clean interface for module
developers. This is the works that I have done, which now needs to be
finalized and put into the source base. We also need to work with Richard
Stallman and the GNU Emacs team to try and make sure that the work we do
is compatible with that version of Emacs, although there will be some
technical difficulties in that regard (more on this later). What I have
done is to flesh out dll.c to provide an actual working implementation of
DSO loading, using a variety of different models. We need to choose a
model and stick with it. To make the right choice we need to understand
all of the issues that affect that choice.
One of my first goals is to make it easy for developers and contributors to
use DSOs, and to provide a platform-independant mechanism for a developer
to compile, install and test their modules. Secondly, we need to document
the mechanisms we chose and provide as much information to the developer as
possible. This will probably take the form of a new document which
references Ben's internals document, although we could simply extend the
internals document to cover DSOs. I prefer the former approach, as it will
make the document less dependant on the actual Emacs implementation.
Third, we need to try and keep things as similar to the current internal
modules as possible, so that those people who are familiar with the Emacs
internals can work in a familiar environment. If it ever comes to a
trade-off between keeping existing internals developers happy and atracting
new developers, it is my opinion that we err on the side of attracting new
developers, so that the number of people extending Emacs can grow.
Basic Architecture
------------------
Before getting into the low-down details, a word on the basic DSO
architecture. Some of this is modelled after the DSO implementation for
XFree86 Version 4, and some of my own thoughts and ideas.
Each DSO is presented to Emacs as a single .so or .dll that is to be
loaded. The actual file extension doesn't matter (for now). This module
will contain the code required to implement the functionality the module
intends to provide. When the moduel is loaded, either by the user
interactively loading a module, or by some Lisp code being executed to load
the module, the first thing the loader does is locate the file (more on
this later, too), and then try to determine whether or not this is a
propper Emacs DSO. The first thing it does is look for three special C
symbols, which are all normal strings of type ``CONST char *''. These
symbols are:
CONST char *cemacs_mod_name = "whatever";
This variable is used to give the module a short tag or name, which is
used later in resolving inter-module dependancies. This should be
limited to 8 characters or less, and contain no spaces.
CONST char *cemacs_mod_version = "1.0.0";
This is used to define the version of the module, in the form
MAJOR.MINOR.PATCH, and is also used in dtermining inter-module
dependancies.
CONST char *cemacs_mod_title = "A sample module";
This is used to describe the module more fully, and should be a short
string (less that 40 characters) and its primary use is in the display
of the currently loaded modules.
If any of these three strings is not present, the loader will reject the
module as invalid. The last check for module validity is a C function,
which is declared as:
int cemacs_init (int task, char *arg)
This function must be present for the module to be considered valid by the
loader. If all 4 of these symbols are present, the loader will continue
loading and initializing the module. All of the loading and initialization
work is done through the cemacs_init() function, which is passed along an
initialization task code (and optional argument pointer), during which time
the function is expected to perform certain initialization tasks. The
following basic tasks codes have been identified and defined:
Task code 0
When cemacs_init() is called with task code 0, it means that the module
is being loaded and initialized for the first time. The argument passed
is the fully qualified path name fo the module that is being loaded.
During this task code, the module is expected to load in any dependant
modules, by calling cemacs_load_module(). Each of those modules can in
turn load other modules, etc. If this task code does not return 0, then
all of the modules loaded after it are discarded, and the module load
fails.
Task code 1
This is the moral equivalent of the syms_of_XXX() function, and this is
where the code should announce new SUBRs. This is done by calling
DEFSUBR() (or CDEFSUBR(), see below) for all those SUBRs the module
defines. The return code from this phase is ignored.
Task code 2
This is similar in intent to task code 1, but is the internal equivalent
of the vars_of_XXX() function. During this task code, all new variables
are defined and initialized to default values.
Given the above, each module would contain code similar to the following
skeleton code. This is not well commented, as I just want to get the
concepts across:
/* Emacs dynamic module foo */
#include "some_stuff.h"
CONST char *cemacs_mod_name = "sample";
CONST char *cemacs_mod_version = "1.0.1";
CONST char *cemacs_mod_title = "Sample DSO module";
int cemacs_init (int task, char *arg)
{
int ret;
switch (task)
{
case 0: /* Load in other modules and do internal inits */
ret = cemacs_load_module ("sample2.so", "sample2", "1.0.1");
if (ret != 1)
return 1;
return 0;
break;
case 1: /* Declare SUBRs */
DEFSUBR(Fsample_function);
return 0;
break;
case 2: /* Delcare variables */
DEFVAR_LISP("sample-string", &Vsample_string);
DEFVAR_BOOL("sample-boolean", &sample_bool);
return 0;
break;
}
return 0;
}
This architecture allows us to keep the same basic model, and extend it as
time goes by by adding new task codes or changing the semantic
interpretation of each task code. It is very open-ended and very eacy to
implement. The names of the variables and function are, of course, open to
debate, but the concepts are the most important part. Lets not start
debating the actual names now (that can come in round two of the debate).
Declaring new SUBRs
-------------------
One of the main reasons people will write modules is to implement new
functionality in the editor, or provide a faster mechanism for implementing
existing functionality. This will involve creating new internal subrs and
variables, and this is where one of the biggest debate points lies. In the
existing code, the developer uses the DEFUN() macro to define a new C
level subroutine. This results in a static structure being set up and
populated with static values. This model does not lend itself well to
supporting DSOs, due to the way PIC addresses are stored in shared
objects. (If you need more details on why this is an issue, please feel
free to contact me outside the scope of this discussion. For now, please
just accept my word that there is a problem here).
I saw two ways around this problem, and implemented both. On a
recommendation from Hrvoje Niksic I dropped one of them, but I would like
to open this up to debate again, to get input from others.
The first mechanism I implemented was to define a new set of macros for
defining SUBRs and variables. This used the same macros as the internal
source did, except that the macro names were prepended with a 'C'. This
was inspired, in part, by my stand-alone efforts to add DLL support to
XEMacs way before I knew that work was already being done in this area, and
I was writing a package I called "CE-Emacs", for C Extensible Emacs. In
this model, when you declare a new subr, you would use the following
syntax:
CDEFUN("sample-function", Fsample_function, 0, 0, "\
This is the documentation for the subr, as a C string.\n\
\n\
Note that the way the documentation is done is radically\n\
different to the way it is currently done with DEFUN()\n\,
as we have a static C string, not a C comment acting as\n\
the documentation.\n"
())
{
message ("Eureka! It worked!");
}
Compare this to how the same function would be done as an internal module:
DEFUN("sample-function", Fsample_function, 0, 0, /*
This is the documentation for the subr, as a C comment.
This documentation would be snarfed by make-docfile and
added to the DOC file. We cant do this dynamically because
the DOC file is a static entity. More in this later.
*/
())
{
message ("Eureka! It worked!");
}
Aside from the fact that the macro name has changed from DEFUN to CDEFUN,
the biggest change is in how the documentation is handled. We need to be
able to get the documentation for a subr or a variable into the Lisp engine
dynamically, so we obviously can't use C comments. We need a machanism for
getting the actual documentation as a string into memory. Using the above
mechanism, I was able to take the string and add it correctly to the
structures and have the subr correctly documented.
The second implementation I developed was to keep the macro names and
syntax the same, and have the subr or variable's documentation during the
load phase of the DSO. This would involve calls to a special function that
would document the subr or variable during the appropriate load phase. To
support this, I added Task Code 3, during which time the module is expected
to ducment the SUBRs and variables it defined during task codes 1 and 2.
For example, the following code would be appended to the end of the switch
in the skeleton code above:
case 3: /* Document everything */
CDOCSUBR(Fsample_function, "\
This is the documentation for sample-function.\n\
\n\
It is implemented as a large C string, and added during\n\
task code 3 of the module load life-cycle.\n");
CDOCSYM("sample-string", "\
This is a sample string.\n\
\n\
It is loaded form a DSO and documented during task code\n\
3 of the module load life-cycle.\n");
CDOCSYM("sample-boolean", "\
*Sample boolean variable.\n\
\n\
Defined in a DSO and documented during task code 3.\n");
return 0;
>From an architectural point of view, this is fairly clean, as we have very
distinct phases of module initialization, and we can potentially make come
of them optional. For example, we COULD make it optional to document
variables and functions, as not documenting them can reduce the memory
requirements. The down-side of this mechanism is that it increases the
distance between a function or symbols definition and its documentation.
This in turn makes it more likely that documentation and implementation
semantics get out of sync. Yes, this is just a matter of programmer
discipline, but it is a factor to be considered.
A third alternative (and one I have not implemented) is to have variables
and functions defined exactly the way they currently are, using DEFUN(),
DEFVAR_LISP() etc, with their comments in C as they currently are, and then
to extend the module build process to include calls to make-docfile or some
other suitable documentation snarfer that can write out a special init.c
file, which can convert the documentation into C strings which we can then
use to implement the actual documentation when the module is loaded. This
could be extended to automatically creating the cemacs_init() function, or
simply to creating special variables which we handle correctly during
module load time. This has the benefit of being the least impact change,
without breaking the basic DSO architecture.
There are some deeper technical issues which would need to be resolved for
this technology to be unleashed upon the Emacs world, such as avoiding
having to xstrdup() documentation strings, but these are relatively minor
implementation details. Right now, I have all of the basic infrastructure
in place. I have functions for loading the modules, and I have sample
modules which load and run exactly as expected. I have written
documentation around the module creation process, but it is incomplete (and
it cannot be made complete until such time as other issues are solved).
One of the biggest issues facing us is how we can co-exist at a source code
level with GNU Emacs. In the documentation I have written, and the shell
scripts used to compile modules, I define XEMACS, so module writers can
always surround XEmacs specific code with suitable #ifdef XEMACS lines, and
similarly for GNU Emacs (whose define is simply EMACS). However, due to
the internal differences between the two editors, this may be more
complicated than it needs to be. The fact that XEmacs has different data
types, different data structures and different ways of implementing certain
basic functions will make life miserable for the module developer. Now
more than ever we need to try once again to bring about a better syncing up
of XEmacs and GNU Emacs. It will be up to Stallman to decide what and how
much he wants to accept from this work, and to try and implement ways of
abstracting the differences between the two editors. But, this is a larger
issue than I think we need to worry about right now. Right now, I would
like to reach some concensus on which of the three implementions I have
outlined above should be the one moving forward. Please vote for and
debate a choice between the three, or come up with alternatives I have not
thought of.
To summarise, the three choices are:
(A) Use DSO specific macros such as CDEFUN(), CDEFVAR_LISP() etc.
(B) Use the existing DEFUN() and DEFVAR_() macros and have documentation
handled during task code 3, or
(C) Keep everything exactly as it is and extend the DSO build scripts to
snarf the documentation strings correctly and construct a .c file.
One last thing. Do we want shared objects loaded from the normal Lisp load
path, or do we want to set up a new variable to contain the shared objects,
perhaps having the default be in the OS dependant directory?
If anyone has not followed the previous threads between Hrjove and myself,
and you have not seen the documentation or dll.c implementation and other
patches, please let me know and I will put them up somewhere for people to
FTP and peruse. I would like to get a decision on this as soon as
possible, so that I can continue with the documention, samples and build
scripts. Is it realistic to set a dead-line of, say, Friday October 30 for
a decision to be made?
Should I send this stuff on to Stallman to look at too, and get him in on
the comment loop, or do we just make a choice and present everything to him
for integration into GNU Emacs feit acompli? I am "FSF-friendly", in that
my copyright assignments are on file at the FSF, so I can appease that fear
at the very least by assigning copyright to them. I also have disclaimers
of interest from SCO on file at the FSF, so there is no conflict there
either.
Comments, suggestions and questions welcome.
--
J. Kean Johnston | "Every day, computers are making people easier
Engineer, SPG | to use" -- Source unknown.
Santa Cruz, CA +----------------------------------------------------------
Tel: 831-427-7569 Fax: 831-429-1887 E-mail: jkj(a)sco.com
