Dynamic Modules under NT

I've successfully modified XEmacs 21.2.34 to load dynamic modules under Windows NT. My efforts point to some changes that need to be made to XEmacs to make this a real workable solution. There are three ways to make XEmacs compile on Windows: gcc with cygwin, gcc no cygwin, and MSVC++. I've gotten all three to work, though I prefer gcc no cygwin and MSVC++, since that allows one to use DLLs compiled by any Windows C compiler to be used with Emacs (if you use the cygnus cygwin library in any piece of the application, you break any code that depends on MSVCRT.dll that pretty much everything not compiled with gcc/cygwin depends on.) Anyway, here is a list of my changes: 1. for gcc no cygwin, I changed the config.guess script to return ${UNAME_MACHINE}-pc-mingw32 when it finds cygwin. I'm sure there's some way to force the configure script to assume this, but this was the easiest way for me. 2. added "-I/usr/local/mingw/include -mno-cygwin" to xe_cppflags in configure. added "-L/usr/local/mingw/lib -mno-cygwin" to xe_ldflags in configure. These two lines convince gcc that it's compiling for no-cygwin. It may just be voodoo to put them here, but it can't hurt to have the configure script also realize it's not supposed to use cygwin. 3. Changed nt/config.h to have the line #define HAVE_SHLIB 4. Added $(SRC)\emodules.c to DOC_SRC2, and $(SRC)\sysdll.c to DOC_SRC5 in nt/xemacs.mak. Added $(OUTDIR)\emodules.obj and $(OUTDIR)\sysdll.obj to TEMACS_OBJS in nt/xemacs.mak. 5. Added #include <windows.h> in the _WINDOWS/WIN32 section of sysdll.c. It was missing. ---- Now, the more annoying bits. In order to compile DLLs for Windows, you need to resolve all symbols at static link time. This means if a DLL needs symbols from the application which is loading it, those symbols must be munged into an import library for the DLL to link with. In addition, those same symbols must be munged into an export library for the application to link with. Yes, that's right, temacs needs to be linked with an export library that must "predict" what symbols any DLL might use. The easiest way to make this work is to construct a .def file with a list of all functions and data symbols from the XEmacs sources that may be used by a DLL. That's pretty much all of them. One way to do this simply is to build temacs normally, then use nm to spit out all of the symbols found inside, sed this output to produce a temacs.def file that looks like this: EXPORTS Qt DATA Qnil DATA message defsubr dumpopaque staticpro defvar_magic assert_failed make_uninit_string These are actually the symbols needed by modules/sample.c. Once this file is constructed, the way to make an export library for MSVC++ is: lib /def:temacs.def which produces two files, temacs.exp and temacs.lib. temacs.exp should be linked in with temacs.exe. temacs.lib is actually useless it turns out. (But you said that it was needed to link with the DLL!) Well, Windows has another gotcha, which is that it looks for the symbols in a particular file name. That name is the primary name of the .def file. In this case, I named it temacs.def so that temacs.exe could link with its temacs.exp successfully. However, since dynamic modules are loaded into xemacs.exe, we need to modify the .def file a bit: LIBRARY xemacs.exe EXPORTS Qt DATA Qnil DATA message defsubr dumpopaque staticpro defvar_magic assert_failed make_uninit_string This is the same file (now named xemacs.def), but with a LIBRARY directive at the top to tell the DLL where to find these symbols. We do the same lib command and we'll get the appropriate xemacs.lib file to link with the DLL. For gcc, you can use dlltool -e temacs.exp -d temacs.def and dlltool -l libxemacs.a -d xemacs.def. -------- Next: This export/import stuff almost completely works, but Windows states that anything that's available as a DLL symbol has to be declared so. So, the prototypes for emodule_compiler, emodule_name, emodule_version, emodule_title, docs_of_<module>, syms_of_<module>, vars_of_<module> and modules_of_<module> must be prefaced by __declspec(dllexport). Fine, just modify ellcc a bit. Well, ellcc is totally wrong for MSVC++. I rewrote the Makefile for module/sample.c for MSVC++: ---- CC=cl CFLAGS=-I. -I../../src -c -nologo -W3 -Od -Zi -ML \ -I"c:\xemacs\auxiliary\xpm-3.4k" \ -I"c:\ xemacs\auxiliary\xpm-3.4k\lib" \ -I"c:\xemacs\auxiliary\libpng-1.0.5" \ -I"c:\xemacs\auxiliary\zlib" \ -I"c:\xemacs\auxiliary\tiff-v3.4beta035\libtiff" \ -I"c:\xemacs\a uxiliary\jpeg-6b" -I..\nt\inc -I..\src \ -I..\lwlib -DHAVE_MS_WINDOWS -DHAVE_SCROLLBARS \ -DHAVE_MENUBARS -DHAVE_MSW_C_DIRED -DHAVE_XPM -DFOR_MSW \ -DHAVE_GIF -DHAVE_PNG -DHAVE_TIFF -DHAVE_JPEG -DHAVE_TOOLBARS \ -DHAVE_DIALOGS -DHAVE_WIDGETS -DHAVE_NATIVE_SOUND \ -DGNU_MALLOC -DWIN32 -D_WIN32 -DWIN32_LEAN_AND_MEAN \ -DWINDOWSNT -Demacs -DHAVE_CONFIG_H -DPATH_VERSION=\"21.2-b33\" \ -DPATH_PROGNAME=\"xemacs\" -DEMACS_VERSION=\"21.2-b33\" \ -DEMACS_PROGNAME=\"xemacs\" -DPATH_PREFIX=\"..\" \ -DDEBUG_XEMACS -D_DEBUG -DEMACS_MAJOR_VERSION=21 \ -DEMACS_MINOR_VERSION=2 -DEMACS_BETA_VERSION=33 \ -DXEMACS_CODENAME=\""Melpomene"\" \ -DEMACS_CONFIGURATION=\"i586-pc-win32\" \ -DCOMPILING_MODULE LD=$(CC) --mode=link LDFLAGS=/dll /incremental:no /subsystem:windows /machine:I386 SRCS=sample.c sample-module.c WINOBJS=$(SRCS:.c=.obj) %.obj: %.c $(CC) $(CFLAGS) -Fd$*.pdb $< sample.dll: $(WINOBJS) link.exe $(LDFLAGS) xemacs.lib /out:sample.dll $? all: sample.dll ---------- (That xemacs.lib is the import library I generated from xemacs.def). I don't know if I need all those defines, but well, they can't hurt. I also generated the module description file by hand and hand-annotated the function prototypes with __declspec(dllexport). ellcc is also wrong for GCC for Windows. No need for the pesky -fPIC or -shared flags: Here's a sample of the one I used to compile my application's xemacs modules. /usr/bin/g++ -MMD -c -pipe -DGC_USE_DLL -fno-rtti -fno-exceptions -fkeep-inline-functions -fno-gnu-keywords -fvtable-thunks -fguiding-decls -Woverloaded-virtual -Wchar-subscripts -Wcomment -Wformat -Wimplicit -Wmain -Wmultichar -Wparentheses -Wreturn-type -Wswitch -W -Wpointer-arith -Wbad-function-cast -Wwrite-strings -Wconversion -Wsign-compare -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations -Wnested-externs -g -DDEBUG -DTRACE -DLK_USE_DLL -DCOMPILING_MODULE -I/usr/misc/include -I/usr/local/mingw/include -I../../ -IGENSRC/ -I/xemacs/xemacs-21.2/src -mno-cygwin -fnative-struct -o OBJ.x86-pc-win32/node.o node.cc You need to have a dllwrap command to compile dlls from object files: dllwrap --export-all --driver-name=/usr/bin/g++ -mno-cygwin --target i386-mingw32 --output-lib OBJ.x86-pc-win32/libharmonia_xemacsimp.a -o OBJ.x86-pc-win32/harmonia_xemacs.dll OBJ.x86-pc-win32/harmonia-init.xemacs.o OBJ.x86-pc-win32/init.o OBJ.x86-pc-win32/node.o OBJ.x86-pc-win32/language.o OBJ.x86-pc-win32/document.o OBJ.x86-pc-win32/util.o OBJ.x86-pc-win32/diag.o ../../common/OBJ.x86-pc-win32/libcommon.a ../common/OBJ.x86-pc-win32/PointerTable.o -L../../lib/OBJ.x86-pc-win32 -llkimp -L/usr/misc/lib -lgcimp -LOBJ.x86-pc-win32 -lxemacs -L/usr/local/mingw/lib --------- After all this, you'd think I'd be done. Not quite. While the export file was good enough to link into temacs.exe, we have to change the prototypes for all *data* symbols imported by the DLL (found by #including <emodules.h>). So, I had to go into lisp.h, and added in the following: #ifndef COMPILING_MODULE extern Lisp_Object Qt; #else extern __declspec(dllimport) Lisp_Object Qt; #endif I made up this COMPILING_MODULE flag so that a data member like Qt would only be declared to be imported when we compile modules, not when we compile temacs. This has to be done in the XEmacs sources, since you can't redefine a symbol later on. It's going to require a large amount of tedious work to make essentially a conditionally imported type for all data objects (i.e. #ifndef COMPILING_MODULE #define Lisp_Object Real_Lisp_Object #else #define Lisp_Object __declspec(dllimport) Real_Lisp_Object #endif ) -------- Finally, after all this. We're done. You can make a dll and load it into emacs with M-x load-module. -------- This story points to a few changes to be made to xemacs. The first couple are trivial, and the last few are long and annoying and probably require touching all of the files in the source tree. But it is worth it and modules work just fine. Whoever set up the code to initially support modules, I thank you. Andrew Begel UC Berkeley