Ville> In the future, please split patches so that one patch Ville> changes one thing. Do you use diff-mode? You'll probably have to hand-apply ChangeLog patches unless people are very careful about making whole entires, but I find that applying, reviewing in place, and deleting (from a copy of the patch, of course) hunk by hunk is qute effective, and diff-mode keeps all that synched for you. Ville> Stephen, any objections? No. Ville> [Just Doing It] Works for me, Not in 21.5, it won't work in-place. XEmacs requires being told where the package root is in that case. This is a philosophical difference between me and Mike Sperber, at least: I say packages are a separate entity from the core, and XEmacs should (1) find the XEmacs binary (Mike hasn't changed this) (2) look for a _build_-specific package hierarchies relative to the binary in a _very_ small number of standard places (preferably just one, ../xemacs-packages and similar for mule-, site-), (3) look for them in --package-path places, and (4) finally default to ${datadir}/xemacs-packages etc. Effectively I would default "package_path=::${datadir}::". Where Mike and I part ways is he says (4') give up. ;-) Correct me if I'm wrong, Mike. Ville> But before that, I think we should roll new Sumos out of Ville> the way. Agreed, always checkpoint. -- Institute of Policy and Planning Sciences http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.

Ville> I have this setup ("default staging" means the place where Ville> "make install" dumps the packages by default): ~/cvs/xemacs/packages (contains the packages working dir) ~/cvs/xemacs/xemacs-packages (default staging for non-mule packages) ~/cvs/xemacs/mule-packages (default staging for mule packages) Ville> ...and then symlinks: ~/.xemacs/xemacs-packages -> ~/cvs/xemacs/xemacs-packages ~/.xemacs/mule-packages -> ~/cvs/xemacs/mule-packages This is more or less what Michael recommends. (He actually has them in the build root for the core.) It is not something that we can or should enforce, IMHO. Ville> Anyway, the point is that it looks to me that one doesn't Ville> *have* to use --package-path for the "personal Ville> installation" case. Of course. The problem is that if (like me, mostly) you have a site-wide installation of the packages and a personal installation of XEmacs, 21.5 doesn't work out of the box, while 21.4 does. -- Institute of Policy and Planning Sciences http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.

Stephen> This is more or less what Michael recommends. (He Stephen> actually has them in the build root for the core.) It is Stephen> not something that we can or should enforce, IMHO. ms> Ah, no, I've never recommended that. I've advocated symlinks ms> in the build directory for running *in place*. That's what I meant. ms> You can also run side-by-side, but it's something I'm less and ms> less happy with. What do you mean by "side-by-side"? ms> --package-path is almost always a bad idea, "Almost always"?! In theory, maybe, but in practice --package-path generally does exactly what the user wants: points to a specific Emacs root _where it already is and always has been_. ms> because it disables crucial parts of the XEmacs logic for ms> constructing the package path. Given that the package path is ms> a complex beast, this is asking for trouble. So are symlinks managed by hand. ms> Could you (re-)describe what precisely you're asking for. I'm ms> thoroughly confused. I just want XEmacs to conform to your Info documentation: Such a directory containing a hierarchy is called a "root". Whenever this section refers to a directory using the shorthand `<root>', it means that XEmacs searches for it under all hierarchies XEmacs was able to scrounge up. Note the "all hierarchies." [...] Moreover, XEmacs expects late hierarchies in the subdirectories `site-packages', `mule-packages', and `xemacs-packages' (in that order) of the `<root>/lib/xemacs' subdirectory of one of the installation hierarchies. In other words, I want cd $random-directory tar xzf xemacs-21.5.10.tar.gz ./configure make src/xemacs -vanilla to find packages installed in $datadir/xemacs/xemacs-packages, unless xemacs is configured --with-prefix=no. This is what testers expect. This is what 21.4 does. If you don't provide this, then testers will use --package-path. I did (I now use EMACSPACKAGEPATH). And even with that fallback, you've still got the problem that Linux distros now generally conform to the FHS, which puts the package hierarchies under /usr/share/xemacs. They still need --package-path. Admittedly, the documentation continues (If you run in-place, [the package hierarchies] are direct subdirectories of the build directory.) But I can't imagine anybody deducing that from the rest of the description. Nor that running in-place implies that the in-place xemacs _needs_ to have a whole set of private package hierarchies, and that running in-place "disables crucial parts of the XEmacs logic for constructing the package path." We know what that leads to. :-) -- Institute of Policy and Planning Sciences http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.

ms> You can have this directory structure ms> xemacs/ <XEmacs in-place build tree> ms> xemacs-packages/ ms> mule-packages/ ms> which XEmacs will recognize. Ugh. Just Say No. If you could truly run packages in-place, that might make some sense, but you can't (and anyway if you're using a standard CVS checkout, the package hierarchies are under packages!) Stephen> So are symlinks managed by hand [== "trouble"]. ms> Ermh, no, they don't disable any part of the XEmacs logic for ms> constructing the package path. Even with full XEmacs logic you can have trouble with symlinks. In fact, even vi users :-) have trouble with symlinks. Stephen> Note the "all hierarchies." ms> Note "[that] XEmacs was able to scrounge up." :-) If --with-prefix=yes, it had better be able to find $(prefix). It doesn't, for packages. I suspect it's looking for (eg) $prefix/lib/xemacs-21.5.9 and giving up when that's not there. (This is the package_root != executable_root case.) Stephen> In other words, I want Stephen> cd $random-directory Stephen> tar xzf xemacs-21.5.10.tar.gz Stephen> ./configure Stephen> make Stephen> src/xemacs -vanilla Stephen> to find packages installed in Stephen> $datadir/xemacs/xemacs-packages ms> Erh, and this doesn't work now? That'd be a bug. No. And it hasn't worked since "the change that Stephen requested". This is the bug the Michael Harnois was so upset about. ms> Can you be more specific about the problem scenario? In my case, prefix=/usr/local (the default), package hierarchies are installed in /usr/local/lib/xemacs, and the above script[1] with random-directory=/playpen/Projects/XEmacs/Builds/21.5 and --srcdir=/coda/Software/XEmacs/21.5 doesn't find the packages without setting EMACSPACKAGEPATH. Specifically: emacs-roots ("/playpen/Projects/XEmacs/Builds/21.5/") configure-prefix-directory "/usr/local/" If I then do "make install" and reinvoke xemacs I get: emacs-roots ("/playpen/Projects/XEmacs/Builds/21.5/" "/usr/local/") configure-prefix-directory "/usr/local/" and packages are found. Stephen> Linux distros now generally conform to the FHS, which Stephen> puts the package hierarchies under /usr/share/xemacs. Stephen> They still need --package-path. ms> Any scenario which "needs" --package-path is just plain ms> broken. I have every intention of fixing that: what exactly ms> is that scenario? According to the Info node quoted, package hierarchies are expected under ${prefix}/lib/xemacs. The File Hierarchy Standard says Lisp (Perl, Java, etc) libraries belong under datadir=/usr/share, and thus the package hierarchies should be under /usr/share/xemacs (or /usr/share/xemacs21 for Debian). GNU Emacs has long defaulted datadir to ${prefix}/share (since 19.29 according to configure.in). Footnotes: [1] The precise configure command is: /coda/Software/XEmacs/21.5/configure '--with-mule' '--with-xim=xlib' \ '--with-xfs' '--with-widgets=athena' '--with-athena=3d' \ '--with-dialogs=athena' '--with-sound=noesd,nonas' '--external-widget' \ '--with-modules' '--pdump' '--debug' '--error-checking=all' \ '--use-union-type' -- Institute of Policy and Planning Sciences http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.

ms> Ah, OK, I see. So, I'm now even more confused: Are you saying ms> we should revert "the change that Stephen requested"? No, we should just make it right. See elsewhere in the thread for what I requested vs what I got, and the misconception I had that meant they were different. ms> You can't have both, i.e. ms> - prevent XEmacs from finding an old, rotten package root ms> sitting where you might install your XEmacs in the future symlinks can do that, too. That's what burned me, and got me to rm all the symlinks in my blddirs and substitute --package-path. (And nowadays I typically do rm -rf * in any blddir that hasn't been built in more than a week, which has the same effect.) ms> - get XEmacs to find a current package root sitting where you ms> might install your XEmacs in the future The common case with testers and at least this one developer is that XEmacs is in fact installed there, but you're running in place because it's a new build you want to test. In that case you want XEmacs to present you with the expected environment. Yes, I can achieve that effect now with symlinks, but I have to remember to do it, and nothing guarantees they'll be updated. I think XEmacs should default to doing the right thing by simplifying common cases. Requiring ad hoc changes in directory structure is not my idea of a righteous default. Stephen> According to the Info node quoted, package hierarchies Stephen> are expected under ${prefix}/lib/xemacs [but FHS says Stephen> they go under ${prefix}/share/xemacs]. ms> So what happens if you set --datadir instead of ms> --package-path? XEmacs happily fails to find the packages. -- Institute of Policy and Planning Sciences http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.

ok. i find packages extremely confusing. almost all documentation i've seen has been a total mess. i've spent some time trying to clean the docs up to be clear. this discussion seems to be typical of it. michael, i don't want to be overcritical of you but you need to spend more time thinking of how to make clear, concise, introductory documentation. this may be why so many people seem confused about how packages work -- much of your docs goes into lots of detail very quickly, but the overviews presenting what you really need to know are not there or very confusing. [btw many many apologies in advance if you didn't write the docs i'm complaining about!!!] currently some of the intro files [e.g. README.packages] are clear, but only because i rewrote them! Beforehand the files were just dismal. The documentation in the .el files is unfortunately even worse, e.g.: (defun paths-construct-info-path (roots early-packages late-packages last-packages) "Construct the info path." ...) (defun paths-find-doc-directory (roots) "Find the documentation directory." ...) (defun paths-find-exec-directory (roots) "Find the binary directory." ...) (defun paths-construct-exec-path (roots exec-directory early-packages late-packages last-packages) "Find the binary path." ...) some examples of problems in the docs: [1] introductory docs need to have clear cross-references to where to find more info. in this case, this might mean a pointer from README.packages to the node in xemacs/startup.texi; but no such pointers exist. [2] startup.texi talks generically about "roots" but its definition of "root" is over-general and very confusing. You need to put in examples -- for example, the fact that <root> = /usr/local [typically] appears nowhere in the document. in fact, there's almost no mention of /usr/local. in general, examples are sorely lacking. [3] it says: By default, XEmacs expects an early package hierarchy in the subdirectory ＠file{.xemacs/xemacs-packages} of the user's home directory. but what goes there? do i put subdirectories xemacs-packages, mule-packages, and site-packages? why do we look only for xemacs-packages here, but elsewhere also mule-packages? what's the purpose of mule-packages? what should the layout under xemacs-packages look like, if i'm new to packages and confused about the whole thing? does xemacs require a specific layout or is it fairly flexible? [i know the answer here [i think!] but i'm indicating areas of problem] in general the docs suffer from a serious case of assumptionitis -- assuming the reader already knows various basic facts, and thereby not specifying them, when in fact these are exactly what is most important to specify. Now: In my case, i have xemacs in-place in /src/xemacs/someworkspace/src/xemacs.exe, and my packages typically get installed into /src/xemacs/xemacs-packages, /src/xemacs/mule-packages, etc. i have to manually set a package-path like PACKAGE_PREFIX=\src\xemacs PACKAGE_PATH=~\.xemacs;;$(PACKAGE_PREFIX)\site-packages;$(PACKAGE_PREFIX)\mule-p ackages;$(PACKAGE_PREFIX)\xemacs-packages for native Windows, or configure ... --package-path=~/.xemacs::/src/xemacs/site-packages:/src/xemacs/xemacs-packa ges:/src/xemacs/mule-packages for Cygwin. Note that symlinks DO NOT work under native Windows. Having to set package-path like this is way bogus. i should just be able to say something like --package-prefix=/src/xemacs and have it construct everything else automatically. can i? i don't know, and the documentation doesn't say. since you think setting package-patch is bad, what should i be doing instead? ben ----- Original Message ----- From: "Michael Sperber [Mr. Preprocessor]" <sperber(a)informatik.uni-tuebingen.de&gt; To: "Stephen J. Turnbull" <stephen(a)xemacs.org&gt; Cc: "Ville Skytt" <scop(a)xemacs.org>; "Ben Wing" <ben(a)xemacs.org>; <xemacs-beta(a)xemacs.org>; <mike(a)xemacs.org>; "Peter Brown" <rendhalver(a)xemacs.org&gt; Sent: Friday, December 06, 2002 5:15 AM Subject: Re: [Q] Re: [PATCH] all my package fixes, mostly build-related packages)

ms> Sorry, I'm not getting it. This discussion is about ms> *atypical* cases, namely about running in-place. (As have ms> most others been.) Things seem to work fine after "make ms> install." I am not sure this will be true on Windows, where the {bin,lib,share} hierarchies are nowhere near as ingrained as on Unixoid systems. Cf. Ben's installation, which (IMHO) _should not work_ without --package-path, since the parent of his generic "xemacs" directory is \src, not \lib or \share or Windows equivalent. (On Cygwin netinstaller installations, the root is "/Program Files/XEmacs", and it's otherwise conventional. I wouldn't be surprised if --with-prefix=no works there!) We regularly see Windows people reporting "I unpacked the SUMO like you said, but XEmacs _still_ doesn't find my packages." I suspect that's because they expect tar to work like setup.exe, and move the component into place. (And on a personal workstation, there's no particular reason why it shouldn't.) ms> I don't see how the introductory documentation can or should ms> address this issue. Then somebody else can write the docs. But definitely it's a FAQ, and the docs _should_ address it. And we need your help (cf my confusion about definition of "hierarchy" below). ms> I could add: "Construct the info path. This is the path where XEmacs will look for info files. ROOTS is a list of XEmacs roots. EARLY-PACKAGES is a list of early package hierarchies. LATE-PACKAGES is a list of late package hierarchies. LAST-PACKAGES is a list of last package hierarchies." ms> But this adds verbosity, not information. You say you're ms> confused, but you don't state a question that you think isn't ms> answered. It _does_ add information. The facts that ROOTS are "XEmacs roots" and that *-PACKAGES are "hierarchies" (rather than, say, already computed search paths) helps. Not much, but it's better. But what would probably make that usable to "people-who-aren't-Mike" is "Construct the info path. This is the path where XEmacs will look for info files. ROOTS is a list of XEmacs roots, eg, as returned by `paths-find-emacs-roots'. EARLY-PACKAGES is a list of early package hierarchies. LATE-PACKAGES is a list of late package hierarchies. LAST-PACKAGES is a list of last package hierarchies. A \"package hierarchy\" is the top directory of a collection of installed XEmacs packages. It will be named \"xemacs-packages\", \"mule-packages\", or \"site-packages\", and will typically have subdirectories \"info\", \"etc\", \"lisp\", and \"pkginfo\". Package Info files are usually named for the package (eg, \"gnus.info\") or library (\"message.info\") and installed directly into the \"info\" subdirectory." For the load-path variant, you'd substitute "Package Lisp files are installed into a package-specific subdirectory of the \"lisp\" subdirectory (eg, \"lisp/xemacs-base\")." for the last sentence. Etc, etc. Or maybe a "package hierarchy" is a hierarchy whose root _contains_ "xemacs-packages" et al. Then you'd need to adjust the above for correct semantics of hierarchy. (This is why I haven't long since done something about this. Resolving questions like "what is a hierarchy?" requires analyzing the sources, as there is no clear documentation. So usually I just cut-and-try, and (locally) adopt what works, postponing public spirit for another day.... ms> I don't think there's a "typical value" for a root. ms> Specifically, the problems people are having crop up when it's ms> not /usr/local. The ones you're worried about, maybe. But I _am_ running with the package root (as far as the docs ever told me) in /usr/local, and I have (what I consider to be) problems. In man/xemacs/startup.texi, we should say something like "Typically, where XEmacs is installed by a distribution (eg, Red Hat Linux), the root for both the core XEmacs and the packages is /usr. Typically on Windows it is [whatever]. The default for locally built XEmacs is /usr/local, ie, $prefix. If the root is $ROOT, then the XEmacs executable and utilities like etags are in $ROOT/bin, version specific Lisp and other files are under $ROOT/lib/xemacs-$VERSION, and packaged Lisp is in $ROOT/lib/xemacs/xemacs-packages. (Mule packages go in $ROOT/lib/xemacs/mule-packages.) There are a number of variations and extensions to support specific XEmacs features and historical practice. However, if you can lay out your installation under a single ROOT as above, we recommend that. You are very unlikely to have any problems with that configuration. (For developers and testers: note that the XEmacs executable must be _installed_. The run-in-place case is much more delicate.)" The parenthetical remark will be unnecessary if we adopt the multiple roots implementation discussed elsewhere. (But that doesn't help Ben, because his /src won't be considered a root according to either criterion.) Ben> In my case, i have xemacs in-place in Ben> /src/xemacs/someworkspace/src/xemacs.exe, and my packages Ben> typically get installed into /src/xemacs/xemacs-packages, Ben> /src/xemacs/mule-packages, etc. AFAIK simply "--package-path=/src/xemacs" should work, at least it did for me. You might need "--package-path=~/.xemacs::/src/xemacs", I forget whether a null component overrides the defaults.[1] (*) I wonder if the package roots specified in --package-path should be considered before the computed roots, not substituted for them. (*) I don't see why --package-path=/src/xemacs/xemacs-packages should be allowed. (Maybe the option should be named --packages-roots or suchlike.) The *-packages names (plus site-modules and the deprecated site-lisp) are completely standardized. We shouldn't allow people to shoot themselves in the foot by omitting some of them, nor (as Ben points out) require them to be entered, since they're redundant. Footnotes: [1] To be precise, at the moment I have nothing in ~/.xemacs, and my packages are in /usr/local/lib/xemacs/{xemacs,mule,site}-packages. "EMACSPACKAGEPATH=/usr/local/lib/xemacs src/xemacs" does pick up the packages for me in 21.5. ISTR I used to set --package-path that way. -- Institute of Policy and Planning Sciences http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.

Excessive CC list trimmed. ms> Note I said "introductory documentation." I'll be happy to ms> document it somewhere else. Once it's settled down, that is. OK. ms> But then this part of the documentation would be duplicated ms> many, many times throughout the code. Besides, the knowledge ms> here is encoded in some other, common abstraction. (This ms> would be a case of comment duplication which is about as bad ms> as the corresponding code duplication.) Agreed. I didn't mean that as a patch. ms> If [--package-path splices into the computed hierarchies ms> rather than replacing them], it should be configurable where ms> the system paths occur. Which means more syntax, more ms> documentation, more complication. I'd rather punt on this ms> one. OK, I see your point. --package-path freezes the hierarchies, and therefore is a last resort. Stephen> (*) I don't see why Stephen> --package-path=/src/xemacs/xemacs-packages should be Stephen> allowed. (Maybe the option should be named Stephen> --packages-roots or suchlike.) The *-packages names Stephen> (plus site-modules and the deprecated site-lisp) are Stephen> completely standardized. We shouldn't allow people to Stephen> shoot themselves in the foot by omitting some of them, Stephen> nor (as Ben points out) require them to be entered, since Stephen> they're redundant. ms> Are you suggesting we should do a sanity check based purely on ms> the name of the directory? I think this would make things ms> more opaque. We could check the contents, but they might not ms> be there yet :-( I don't think there should be _any_ sanity check (except for the colon syntax) on --package-path. Use of it is an admission on the admin's part that his system is insane and he can't or won't reconfigure it. How are we supposed to guess what kind of insanity XEmacs is supposed to conform to? If the contents aren't there yet, then any package-dependent build operations (there are some in "make check") will fail. Currently those are all benign; let's keep it that way. And it's the admin's duty to avoid user/runtime problems by installing packages where he claims they will be. However, the above all depends on having some reasonable default for a package root. Otherwise people will question _our_ sanity, not the nonconformist admin's. On Linux, Cygwin, Depot systems, /opt/$package systems, and probably *BSD and native Windows, at least, we have such a beast: $datadir/xemacs. -- Institute of Policy and Planning Sciences http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.