Call for discussion: DLL handling
27 years
Kean Johnston
Kean Johnston
Introduction
------------
In order to do the right thing with dynamic loading technology in XEmacs,
we need to get concensus on the best way of implementing this. There are
several ways in which we can go about it, and all have their merits and
drawbacks. It is my intention that this document will discuss all of the
mechanisms availible, to give you all of the facts, so that we can choose
between the different implementation mechanisms. Yes, this is rather
long-winded at times, but I hope that more information will help more than
less, so please bear with me :-)
Before going into the actual mechanisms, it is priudent to examine WHY we
think we need or want DSOs in XEmacs. It is already possible to extend the
editor using Lisp, and it is also possible to extend it in C when you are
compiling the editor. What makes DSOs important is the fact that compiled
C code can be selectively loaded into the editor if and when it is needed.
This will allow us to (potentially) move parts of the editor into DSOs,
thus making the core editor smaller and faster to load, while retaining the
ability to extend its functionality. It is important to be able to write
some extensions in C, rather than Lisp, for speed purposes. While the Lisp
engine is fairly quick, there are cases (such as the HTML parser) where a
great benefit can be gained from writing it in C. But not everyone wants
to use their editor for a browser, so not everyone should have their editor
weighted down with an HTML parser. Hence the need for DSOs.
William Perry has already started the work for DSO support by providing the
sysdll.c functionality, which abstracts the actual mechanics of loading a
DSO. This work is pretty much complete as it currently stands in XEmacs,
but it is not really used. The code that is currently in dll.c doesnt
really DO anything yet, and it doesnt provide a clean interface for module
developers. This is the works that I have done, which now needs to be
finalized and put into the source base. We also need to work with Richard
Stallman and the GNU Emacs team to try and make sure that the work we do
is compatible with that version of Emacs, although there will be some
technical difficulties in that regard (more on this later). What I have
done is to flesh out dll.c to provide an actual working implementation of
DSO loading, using a variety of different models. We need to choose a
model and stick with it. To make the right choice we need to understand
all of the issues that affect that choice.
One of my first goals is to make it easy for developers and contributors to
use DSOs, and to provide a platform-independant mechanism for a developer
to compile, install and test their modules. Secondly, we need to document
the mechanisms we chose and provide as much information to the developer as
possible. This will probably take the form of a new document which
references Ben's internals document, although we could simply extend the
internals document to cover DSOs. I prefer the former approach, as it will
make the document less dependant on the actual Emacs implementation.
Third, we need to try and keep things as similar to the current internal
modules as possible, so that those people who are familiar with the Emacs
internals can work in a familiar environment. If it ever comes to a
trade-off between keeping existing internals developers happy and atracting
new developers, it is my opinion that we err on the side of attracting new
developers, so that the number of people extending Emacs can grow.
Basic Architecture
------------------
Before getting into the low-down details, a word on the basic DSO
architecture. Some of this is modelled after the DSO implementation for
XFree86 Version 4, and some of my own thoughts and ideas.
Each DSO is presented to Emacs as a single .so or .dll that is to be
loaded. The actual file extension doesn't matter (for now). This module
will contain the code required to implement the functionality the module
intends to provide. When the moduel is loaded, either by the user
interactively loading a module, or by some Lisp code being executed to load
the module, the first thing the loader does is locate the file (more on
this later, too), and then try to determine whether or not this is a
propper Emacs DSO. The first thing it does is look for three special C
symbols, which are all normal strings of type ``CONST char *''. These
symbols are:
CONST char *cemacs_mod_name = "whatever";
This variable is used to give the module a short tag or name, which is
used later in resolving inter-module dependancies. This should be
limited to 8 characters or less, and contain no spaces.
CONST char *cemacs_mod_version = "1.0.0";
This is used to define the version of the module, in the form
MAJOR.MINOR.PATCH, and is also used in dtermining inter-module
dependancies.
CONST char *cemacs_mod_title = "A sample module";
This is used to describe the module more fully, and should be a short
string (less that 40 characters) and its primary use is in the display
of the currently loaded modules.
If any of these three strings is not present, the loader will reject the
module as invalid. The last check for module validity is a C function,
which is declared as:
int cemacs_init (int task, char *arg)
This function must be present for the module to be considered valid by the
loader. If all 4 of these symbols are present, the loader will continue
loading and initializing the module. All of the loading and initialization
work is done through the cemacs_init() function, which is passed along an
initialization task code (and optional argument pointer), during which time
the function is expected to perform certain initialization tasks. The
following basic tasks codes have been identified and defined:
Task code 0
When cemacs_init() is called with task code 0, it means that the module
is being loaded and initialized for the first time. The argument passed
is the fully qualified path name fo the module that is being loaded.
During this task code, the module is expected to load in any dependant
modules, by calling cemacs_load_module(). Each of those modules can in
turn load other modules, etc. If this task code does not return 0, then
all of the modules loaded after it are discarded, and the module load
fails.
Task code 1
This is the moral equivalent of the syms_of_XXX() function, and this is
where the code should announce new SUBRs. This is done by calling
DEFSUBR() (or CDEFSUBR(), see below) for all those SUBRs the module
defines. The return code from this phase is ignored.
Task code 2
This is similar in intent to task code 1, but is the internal equivalent
of the vars_of_XXX() function. During this task code, all new variables
are defined and initialized to default values.
Given the above, each module would contain code similar to the following
skeleton code. This is not well commented, as I just want to get the
concepts across:
/* Emacs dynamic module foo */
#include "some_stuff.h"
CONST char *cemacs_mod_name = "sample";
CONST char *cemacs_mod_version = "1.0.1";
CONST char *cemacs_mod_title = "Sample DSO module";
int cemacs_init (int task, char *arg)
{
int ret;
switch (task)
{
case 0: /* Load in other modules and do internal inits */
ret = cemacs_load_module ("sample2.so", "sample2", "1.0.1");
if (ret != 1)
return 1;
return 0;
break;
case 1: /* Declare SUBRs */
DEFSUBR(Fsample_function);
return 0;
break;
case 2: /* Delcare variables */
DEFVAR_LISP("sample-string", &Vsample_string);
DEFVAR_BOOL("sample-boolean", &sample_bool);
return 0;
break;
}
return 0;
}
This architecture allows us to keep the same basic model, and extend it as
time goes by by adding new task codes or changing the semantic
interpretation of each task code. It is very open-ended and very eacy to
implement. The names of the variables and function are, of course, open to
debate, but the concepts are the most important part. Lets not start
debating the actual names now (that can come in round two of the debate).
Declaring new SUBRs
-------------------
One of the main reasons people will write modules is to implement new
functionality in the editor, or provide a faster mechanism for implementing
existing functionality. This will involve creating new internal subrs and
variables, and this is where one of the biggest debate points lies. In the
existing code, the developer uses the DEFUN() macro to define a new C
level subroutine. This results in a static structure being set up and
populated with static values. This model does not lend itself well to
supporting DSOs, due to the way PIC addresses are stored in shared
objects. (If you need more details on why this is an issue, please feel
free to contact me outside the scope of this discussion. For now, please
just accept my word that there is a problem here).
I saw two ways around this problem, and implemented both. On a
recommendation from Hrvoje Niksic I dropped one of them, but I would like
to open this up to debate again, to get input from others.
The first mechanism I implemented was to define a new set of macros for
defining SUBRs and variables. This used the same macros as the internal
source did, except that the macro names were prepended with a 'C'. This
was inspired, in part, by my stand-alone efforts to add DLL support to
XEMacs way before I knew that work was already being done in this area, and
I was writing a package I called "CE-Emacs", for C Extensible Emacs. In
this model, when you declare a new subr, you would use the following
syntax:
CDEFUN("sample-function", Fsample_function, 0, 0, "\
This is the documentation for the subr, as a C string.\n\
\n\
Note that the way the documentation is done is radically\n\
different to the way it is currently done with DEFUN()\n\,
as we have a static C string, not a C comment acting as\n\
the documentation.\n"
())
{
message ("Eureka! It worked!");
}
Compare this to how the same function would be done as an internal module:
DEFUN("sample-function", Fsample_function, 0, 0, /*
This is the documentation for the subr, as a C comment.
This documentation would be snarfed by make-docfile and
added to the DOC file. We cant do this dynamically because
the DOC file is a static entity. More in this later.
*/
())
{
message ("Eureka! It worked!");
}
Aside from the fact that the macro name has changed from DEFUN to CDEFUN,
the biggest change is in how the documentation is handled. We need to be
able to get the documentation for a subr or a variable into the Lisp engine
dynamically, so we obviously can't use C comments. We need a machanism for
getting the actual documentation as a string into memory. Using the above
mechanism, I was able to take the string and add it correctly to the
structures and have the subr correctly documented.
The second implementation I developed was to keep the macro names and
syntax the same, and have the subr or variable's documentation during the
load phase of the DSO. This would involve calls to a special function that
would document the subr or variable during the appropriate load phase. To
support this, I added Task Code 3, during which time the module is expected
to ducment the SUBRs and variables it defined during task codes 1 and 2.
For example, the following code would be appended to the end of the switch
in the skeleton code above:
case 3: /* Document everything */
CDOCSUBR(Fsample_function, "\
This is the documentation for sample-function.\n\
\n\
It is implemented as a large C string, and added during\n\
task code 3 of the module load life-cycle.\n");
CDOCSYM("sample-string", "\
This is a sample string.\n\
\n\
It is loaded form a DSO and documented during task code\n\
3 of the module load life-cycle.\n");
CDOCSYM("sample-boolean", "\
*Sample boolean variable.\n\
\n\
Defined in a DSO and documented during task code 3.\n");
return 0;
>From an architectural point of view, this is fairly clean, as we have very
distinct phases of module initialization, and we can potentially make come
of them optional. For example, we COULD make it optional to document
variables and functions, as not documenting them can reduce the memory
requirements. The down-side of this mechanism is that it increases the
distance between a function or symbols definition and its documentation.
This in turn makes it more likely that documentation and implementation
semantics get out of sync. Yes, this is just a matter of programmer
discipline, but it is a factor to be considered.
A third alternative (and one I have not implemented) is to have variables
and functions defined exactly the way they currently are, using DEFUN(),
DEFVAR_LISP() etc, with their comments in C as they currently are, and then
to extend the module build process to include calls to make-docfile or some
other suitable documentation snarfer that can write out a special init.c
file, which can convert the documentation into C strings which we can then
use to implement the actual documentation when the module is loaded. This
could be extended to automatically creating the cemacs_init() function, or
simply to creating special variables which we handle correctly during
module load time. This has the benefit of being the least impact change,
without breaking the basic DSO architecture.
There are some deeper technical issues which would need to be resolved for
this technology to be unleashed upon the Emacs world, such as avoiding
having to xstrdup() documentation strings, but these are relatively minor
implementation details. Right now, I have all of the basic infrastructure
in place. I have functions for loading the modules, and I have sample
modules which load and run exactly as expected. I have written
documentation around the module creation process, but it is incomplete (and
it cannot be made complete until such time as other issues are solved).
One of the biggest issues facing us is how we can co-exist at a source code
level with GNU Emacs. In the documentation I have written, and the shell
scripts used to compile modules, I define XEMACS, so module writers can
always surround XEmacs specific code with suitable #ifdef XEMACS lines, and
similarly for GNU Emacs (whose define is simply EMACS). However, due to
the internal differences between the two editors, this may be more
complicated than it needs to be. The fact that XEmacs has different data
types, different data structures and different ways of implementing certain
basic functions will make life miserable for the module developer. Now
more than ever we need to try once again to bring about a better syncing up
of XEmacs and GNU Emacs. It will be up to Stallman to decide what and how
much he wants to accept from this work, and to try and implement ways of
abstracting the differences between the two editors. But, this is a larger
issue than I think we need to worry about right now. Right now, I would
like to reach some concensus on which of the three implementions I have
outlined above should be the one moving forward. Please vote for and
debate a choice between the three, or come up with alternatives I have not
thought of.
To summarise, the three choices are:
(A) Use DSO specific macros such as CDEFUN(), CDEFVAR_LISP() etc.
(B) Use the existing DEFUN() and DEFVAR_() macros and have documentation
handled during task code 3, or
(C) Keep everything exactly as it is and extend the DSO build scripts to
snarf the documentation strings correctly and construct a .c file.
One last thing. Do we want shared objects loaded from the normal Lisp load
path, or do we want to set up a new variable to contain the shared objects,
perhaps having the default be in the OS dependant directory?
If anyone has not followed the previous threads between Hrjove and myself,
and you have not seen the documentation or dll.c implementation and other
patches, please let me know and I will put them up somewhere for people to
FTP and peruse. I would like to get a decision on this as soon as
possible, so that I can continue with the documention, samples and build
scripts. Is it realistic to set a dead-line of, say, Friday October 30 for
a decision to be made?
Should I send this stuff on to Stallman to look at too, and get him in on
the comment loop, or do we just make a choice and present everything to him
for integration into GNU Emacs feit acompli? I am "FSF-friendly", in that
my copyright assignments are on file at the FSF, so I can appease that fear
at the very least by assigning copyright to them. I also have disclaimers
of interest from SCO on file at the FSF, so there is no conflict there
either.
Comments, suggestions and questions welcome.
--
J. Kean Johnston | "Every day, computers are making people easier
Engineer, SPG | to use" -- Source unknown.
Santa Cruz, CA +----------------------------------------------------------
Tel: 831-427-7569 Fax: 831-429-1887 E-mail: jkj(a)sco.com
- 2 participants
- 2 comments
Re: Call for discussion: DLL handling
27 years
Hrvoje Niksic
Hrvoje Niksic
Pete Ware <ware(a)cis.ohio-state.edu> writes:
> > (A) Use DSO specific macros such as CDEFUN(), CDEFVAR_LISP() etc.
>
> To move code into XEmacs at compile time, one can merely define
> CDEFXXX() to be DEFXXX() with elimination of the document string.
> Modify make-docfile to understand both a string and a comment as
> arguments as args to CDEFXXX() and DEFXXX().
I still prefer (C), because it makes even less difference between
module code and "normal" code, and is not much harder to implement.
--
Hrvoje Niksic <hniksic(a)srce.hr> | Student at FER Zagreb, Croatia
--------------------------------+--------------------------------
Old numerical analysts never die, they just get disarrayed.
- 1 participants
- 0 comments
Re: Debian bug tracking system contains patch for GPM C-z hang
27 years
Hrvoje Niksic
Hrvoje Niksic
Jan Vroonhof <vroonhof(a)math.ethz.ch> writes:
> Hrvoje Niksic <hniksic(a)srce.hr> writes:
>
> > I've never seen it mailed to xemacs-patches.
>
> Is xemacs-patches announced to the outside world anyway?
I'm not sure. A good question.
> > I had problems with them regarding Wget.
>
> Yes I have seen that discussion I know why they have pissed you of.
> However I think there is a fundamental problem here and that is not
> solved by "telling other developers not to cooperate".
I'm not telling them not to cooperate; I am warning them. I've
received such a warning myself, but only after the unfortunate
incident. :-(
Problem #2: a while ago, they "froze" Debian with Wget 1.4.4.
However, I soon released a bugfix 1.4.5, which fixed two or perhaps
three 1.4.5 bugs. They never bothered to add 1.4.5 to the
distribution, because the only updates were [sic!] bugfix-only. And
*I* had to cope with the questions of "why doesn't Wget 1.4.4 work",
because `wget --help' still printed *my* address for bug-fixes.
But the coup was overflown when now they went off and *stole* my stuff
and wrote *my* name under it, again. The miserable bastards. That
was the one time I regretted using the GNU GPL for Wget.
The sad thing is that I had a good opinion about Debian before all
that. Hell, I even use Debian on the PC here.
> Does it bother you that all XEmacsen here installed in their own
> tree of /usr/local/app/{version} although that is not the official
> way to share two different Emacs versions?
It doesn't bother me in the least -- I never looked at what we do
now as in any way being official.
> Does it bother anyone, apart from me, that that here psfrag 3.0 is
> disabled in favor of 2.0 in teTeX although the psfrag authors would
> like very much to see it die, etc etc?
If I were psfrag author, it would bother the hell out of me.
> Author
> Distribution maintainer
> Sysadmin
> user
>
> may work very well in case whether the original authorship is unclear,
> unresponsive etc but is surely not the most optimal solution in the
> case of an active "Author".
Yup. I believe there should be a closer relation between author and
distribution maintainer of a program, especially when the author is
willing to cooperate.
--
Hrvoje Niksic <hniksic(a)srce.hr> | Student at FER Zagreb, Croatia
--------------------------------+--------------------------------
Which is worse: ignorance or apathy? Who knows? Who cares?
- 1 participants
- 0 comments
RE: XEmacs splash screen
27 years
WJCarpenter
WJCarpenter
malcolm> The crucial section is the "To get out of XEmacs" bit. I
malcolm> know we don't ever want anyone to exit XEmacs, but a new user
malcolm> faced with that is going to be mighty confused. What is
malcolm> "Type M-x save-buffers-kill-emacs" when it's at home? We
malcolm> explain the meaning of C-, but not of M-x. I think it should
malcolm> read as follows:
Oh, story time! A thousand years ago, I was upgrading emacs on a
system where you couldn't remove a binary that was "busy". So, I
renamed the old executable to "junk", figuring I would remove it the
next day and if I forgot someone else would eventually see it and the
name would make them not be too worried about just getting rid of it.
(It was a big piece of disk space in those days.)
I got a frantic visit from another developer who had found himself in
emacs through unknown means (to him) and he had no idea how to get
out. (This was pre-GUI days, and C-x C-c is only intuitive to long
term emacs users who have had their neurons regrafted.) That
developer had done two different dumb things. (1) He called his
little throwaway test programs, in his own directory, "junk". (2) He
relied on having "." on the end of his path to run his "junk" from the
shell, instead of doing "./junk".
You'd think he would thank me for helping him stop doing those two
dumb things, but instead all he did was tell me that emacs was dumb
because he couldn't figure out how to exit from it. Oh well.
^[:q
:q
:q!
^[^[^[^[q!
--
bill(a)carpenter.ORG (WJCarpenter) PGP
bill(a)bubblegum.net 0x91865119
38 95 1B 69 C9 C6 3D 25 73 46 32 04 69 D6 ED F3
- 1 participants
- 0 comments
Build a package yourself
27 years
Jan Vroonhof
Jan Vroonhof
[I'm CC'ing XEmacs beta as this seems relevant]
> Hm, my tm scripts are in xemacs-21.0-b57/lib-src/, not somewhere in
> xemacs-packages/... Anyway.
With the latest packages and the latest beta they are in
xemacs-packages/lib-src
Note that both the package layout and the package tools changed quite
a lot in the lat weeks.
> c. No split. Put fonts into xemacs-packages/etc/x-symbol/pcf/
> or xemacs-packages/pcf/x-symbol/ ?
xemacs-packages/etc/x-symbol/pcf
> (2) Are there copyright problems with the xbm files in
> XEmacs-20.4/etc/frame-icon/? (They are not in XEmacs-21.0
> anymore.) Or can I move this into my distribution? I guess, dir
> etc/ would be the right choice.
They are in the frame-icon package now. Search them using locate-data-*
> (3) I would be nice to have a default directory name for executables
> which is at least somewhat consistent with the package system.
> What do you think of using
> lib/xemacs/xemacs-packages/lib/lib-src/CONFIGURATION-NAME/x-symbol/
> Any better idea?
I guess you'll have to contact Steve about this. Currently I think it
would be illegal for a packge to ave architecture dependent files.
> Hm. Aren't these infos availables directly?
No unless you count
http://cvs.xemacs.org/cgi-bin/cvswebp/XEmacs.rules?rev=1.12
For a simple example with data files:
http://cvs.xemacs.org/cgi-bin/cvswebp/oa/frame-icon/Makefile?rev=1.6
For a more complicated example
http://cvs.xemacs.org/cgi-bin/cvswebp/comm/gnus/Makefile?rev=1.35
http://cvs.xemacs.org/cgi-bin/cvswebp/wp/auctex/Makefile?rev=1.15
> I guess I'll wait that
> they will make their way to the normal XEmacs (21.0-pre) distribution
> before I install a new cvs here, get used to it and XEmacs cvs, and
> actually find the info...
I am afraid the CVS examples are the only documentation there is
currently.
Jan
- 1 participants
- 0 comments
Re: Autoload error: flyspell-mode-map
27 years
Jan Vroonhof
Jan Vroonhof
Marcus Thiessel <marcus(a)xemacs.org> writes:
> Yep, but that would help solve the problem since the autoload info
> still exists and breaks the following statement. As a result some
> symbol def's are void as image-decode-jpeg.
Simply add a '(defvar flyspell-map (make-sparse-keymap))' at the top
of the auto-autoloads lsip/text-modes.
Jan
- 2 participants
- 1 comments
Re: What is live-screen-p ??
27 years
Colin Rafferty
Colin Rafferty
Hrvoje Niksic writes:
> Marcus Thiessel <marcus(a)xemacs.org> writes:
>> Hrvoje Niksic writes:
>> | live-screen-p is a long-obsolete function, presumably removed from
>> | 21.2. I don't think Gnus uses live-screen-p, but an extension package
>> | might. Do you use bbdb or anything like that?
>>
>> Yes, I do use bbdb which is installed as a package (latest).
> That's it, then.
No it's not.
There is no reference to live-screen-p in bbdb.
--
Colin (co-maintainer of BBDB) Rafferty
- 2 participants
- 1 comments
Re: problems with flyspell mode
27 years
Greg Klanderman
Greg Klanderman
>>>>> "Manuel" == Manuel Serrano <Manuel.Serrano(a)unice.fr> writes:
Manuel>
You should definitely send us (XEmacs) versions of flyspell with the
version information. Are you announcing releases to xemacs-beta?
I think Jan clarified the first problem I had - the variable
flyspell-mode-map must be autoloaded so it is defined before the
add-minor-mode call which is autoloaded and uses flyspell-mode-map.
Please let me know if further clarification is needed.
Manuel> One more time, thanks for your report,
Thank you! Flyspell is excellent!
greg
- 1 participants
- 0 comments
Re: XEmacs splash screen
27 years
Hrvoje Niksic
Hrvoje Niksic
Malcolm Box <malcolm(a)brownale.demon.co.uk> writes:
> > To get out of XEmacs, type M-x save-buffers-kill-emacs.
This should say `C-x C-c'. Are you running -vanilla?
--
Hrvoje Niksic <hniksic(a)srce.hr> | Student at FER Zagreb, Croatia
--------------------------------+--------------------------------
By any chance, have you seen a summoned 9th order fire elemental
wandering around? No? Oh.. Tell me if you do.
- 2 participants
- 1 comments
Re: XEmacs splash screen
27 years
William M. Perry
William M. Perry
Malcolm Box <malcolm(a)brownale.demon.co.uk> writes:
> In the spirit of helping new users, and checking how things look for
> them, I read the splash screen today. This is how it reads for XEmacs
> 21.0 "Pyrenean-pre6" (win32).
>
> > Copyright (C) 1985-1998 Free Software Foundation, Inc.
> > Copyright (C) 1990-1994 Lucid, Inc.
> > Copyright (C) 1993-1997 Sun Microsystems, Inc. All Rights Reserved.
> > Copyright (C) 1994-1996 Board of Trustees, University of Illinois
> > Copyright (C) 1995-1996 Ben Wing
> >
> > XEmacs comes with ABSOLUTELY NO WARRANTY; type C-h C-w for full details.
> > You may give out copies of XEmacs; type C-h C-l to see the conditions.
> > Type C-h C-d for information on getting the latest version.
> >
> > Type f1 or use the Help menu to get help.
> > Type C-x u to undo changes (`C-' means use the Control key).
> > To get out of XEmacs, type M-x save-buffers-kill-emacs.
> > Type C-h t for a tutorial on using XEmacs.
> > Type C-h i to enter Info, which you can use to read online documentation.
> > For tips and answers to frequently asked questions, see the XEmacs FAQ.
> > (It's on the Help menu, or type C-h F [a capital F!].)
>
> The crucial section is the "To get out of XEmacs" bit. I know we don't
> ever want anyone to exit XEmacs, but a new user faced with that is going
> to be mighty confused. What is "Type M-x save-buffers-kill-emacs" when
> it's at home? We explain the meaning of C-, but not of M-x.
something is very hokey in your environment. Do you not have
save-buffers-kill-emacs bound to any key? By default, this would read:
To get out of XEmacs, type C-x C-c.
The splash screen uses where-is-internal and some magic to figure out
what keys will _really_ work for the user.
-Bill P.
- 1 participants
- 0 comments