9:14 p.m.
The speed difference between XEmacs and GNU Emacs is amazing, in that
GNU Emacs is significantly faster, at least for "heavy" usage such as
Gnus. Entering a large summary buffer takes time proportional to the
*square* of the number of messages in the folder. (This is not a wild
guess, a friend has actually measured.) Time to enter a folder of
more than 3000 articles quickly degrades to minutes.
For the longest time I believed this to be a result of XEmacs's
implementation of text properties on top of extents. I planned to
convert Gnus to use extents natively, in the hope that it would speed
up summary buffer generation. But according to profiling, most of the
time is spent in `insert', at least once you discount excessive GC by
increasing gc-cons-theshold. I'm not sure what XEmacs is doing wrong
here. I suspect Mule is to blame, but I can't prove it.
Using FSF Emacs gave me an order-of-magnitude speedup when entering
large newsgroups and large mailing list folders. It's uncanny.
10:17 p.m.
On Thu, Oct 13 2005, Hrvoje Niksic wrote:
The speed difference between XEmacs and GNU Emacs is amazing, in
that
GNU Emacs is significantly faster, at least for "heavy" usage such as
Gnus.
Interesting. Which versions of (X)Emacs and Gnus did you test?
Entering a large summary buffer takes time proportional to the
*square* of the number of messages in the folder. (This is not a
wild guess, a friend has actually measured.) Time to enter a folder
of more than 3000 articles quickly degrades to minutes.
For the longest time I believed this to be a result of XEmacs's
implementation of text properties on top of extents. I planned to
convert Gnus to use extents natively, in the hope that it would speed
up summary buffer generation.
Then `M-RET' should be significantly faster, shouldn't it?
,----[ <f1> f gnus-group-quick-select-group RET ]
| gnus-group-quick-select-group is an interactive compiled Lisp
| function in `gnus-group.el'.
| (gnus-group-quick-select-group &optional ALL)
|
| Select the current group "quickly".
| This means that no highlighting or scoring will be performed.
| If ALL (the prefix argument) is 0, don't even generate the summary
| buffer.
|
| This might be useful if you want to toggle threading
| before entering the group.
`----
But according to profiling, most of the time is spent in
`insert',
at least once you discount excessive GC by increasing
gc-cons-theshold. I'm not sure what XEmacs is doing wrong here. I
suspect Mule is to blame, but I can't prove it.
`C-u 0 M-RET' probably doesn't use insert so often. Is it comparable
to `C-u 0 M-RET' in Emacs then?
Using FSF Emacs gave me an order-of-magnitude speedup when entering
large newsgroups and large mailing list folders. It's uncanny.
Bye, Reiner.
PS: Cc-ing ding. This might be interesting for Gnus developers. See
http://thread.gmane.org/gmane.emacs.xemacs.beta/20690 for the
whole thread.
--
,,,
(o o)
---ooO-(_)-Ooo--- | PGP key available | http://rsteib.home.pages.de/
10:52 p.m.
Reiner Steib <reinersteib+gmane(a)imap.cc> writes:
Then `M-RET' should be significantly faster, shouldn't it?
I don't think so. The majority of time is not lost in either
highlighting or scoring.
I even profiled the `insert's to see if Gnus was doing something
nasty with inserting to temporary buffers, but it wasn't, it was just
inserting to (what will become) the Summary buffer.
`C-u 0 M-RET' probably doesn't use insert so often.
It has to because it still has to create the buffer. The insert
function inserts string into the buffer.
PS: Cc-ing ding. This might be interesting for Gnus developers.
Not really. This is primarily a problem with XEmacs.
1:35 a.m.
Hrvoje Niksic wrote:
The speed difference between XEmacs and GNU Emacs is amazing, in that
GNU Emacs is significantly faster, at least for "heavy" usage such as
Gnus. Entering a large summary buffer takes time proportional to the
*square* of the number of messages in the folder. (This is not a wild
guess, a friend has actually measured.) Time to enter a folder of
more than 3000 articles quickly degrades to minutes.
For the longest time I believed this to be a result of XEmacs's
implementation of text properties on top of extents. I planned to
convert Gnus to use extents natively, in the hope that it would speed
up summary buffer generation. But according to profiling, most of the
time is spent in `insert', at least once you discount excessive GC by
increasing gc-cons-theshold. I'm not sure what XEmacs is doing wrong
here. I suspect Mule is to blame, but I can't prove it.
Using FSF Emacs gave me an order-of-magnitude speedup when entering
large newsgroups and large mailing list folders. It's uncanny.
first of all, are you sure you're running without error-checking? with
error-checking, there's some extent sledgehammer checks that could
easily give you the quadratic behavior.
as for mule, i don't think that's the case; or at least, you'd see a lot
of time going to `in-char-byte-conversion' in the profiling table.
it's also still possible it's the text property implementation. i know
that the extent code works very badly in the presence of a single extent
that covers the entire buffer, which is a common occurrence given the
way that text properties currently work. i have a new implementation of
text properties sitting in one of my workspaces, which redoes them the
way they are implemented in fsf -- i.e. as completely non-overlapping
intervals. it's all written, but as usual it needs some
testing/debugging work ... sigh ...
but maybe you're right about the textprops not being the culprit; the
only way they could be inside of insert is through `adjust_extents()' or
friends ...
you might try adding some profiling sections to cover certain parts of
the `insert' call. take a look at e.g. charbpos_to_bytebpos_func() and
you'll see how to add a profiling section to cover a certain part of
code; it's pretty easy at this point.
ben
2:45 a.m.
Ben Wing <ben(a)xemacs.org> writes:
first of all, are you sure you're running without error-checking?
Yes. I'm comparing Debian's XEmacs 21.4.17 with Debian's GNU Emacs
21.4.1. I've known for a few years that Summary buffers exhibited
quadratic complexity, I just never imagined it could be really linear
under FSF.
it's also still possible it's the text property
implementation.
[...]
but maybe you're right about the textprops not being the
culprit;
the only way they could be inside of insert is through
`adjust_extents()' or friends ...
And those insertions are pretty normal -- I tracked them using an
"advice"-style logger. They simply insert stuff into the buffer, I
don't think there's even much jumping around.
5:48 a.m.
Hrvoje Niksic wrote:
Ben Wing <ben(a)xemacs.org> writes:
>first of all, are you sure you're running without error-checking?
>
>
Yes. I'm comparing Debian's XEmacs 21.4.17 with Debian's GNU Emacs
21.4.1. I've known for a few years that Summary buffers exhibited
quadratic complexity, I just never imagined it could be really linear
under FSF.
hmm.
can you try this under an optimized, no-error recent CVS?
if you're using 21.4, the bytepos<->charpos conversion could well be the
culprit; the old algorithm of mine really sucked. and you won't see the
time spent in these routines in your profiling data in 21.4. a year or
so ago i rewrote these routines to use the fsf algorithm, and that
dramatically sped up some operations for me; but that code is not in 21.4.
ben
10:23 a.m.
The problem is that people who tried it claim that XEmacs 21.5 is much
*slower* than 21.4, especially for Gnus. That's why I was reluctant
to even include it in the comparison.
10:54 a.m.
Quoting Hrvoje Niksic <hniksic(a)xemacs.org>:
The problem is that people who tried it claim that XEmacs 21.5 is
much
*slower* than 21.4, especially for Gnus. That's why I was reluctant
to even include it in the comparison.
I've been following this list since almost one year now and I recall
that Ben Wing used to post performance improvement patches for 21.5
at the beginning of 2005 (split out of a bigger patch on request of
other XEmacs developers). Have they ever been applied?
Regards,
--
Jérôme Marant
12:11 p.m.
>>>> "Hrvoje" == Hrvoje Niksic
<hniksic(a)xemacs.org> writes:
Hrvoje> The problem is that people who tried it claim that XEmacs
Hrvoje> 21.5 is much *slower* than 21.4, especially for Gnus.
Hrvoje> That's why I was reluctant to even include it in the
Hrvoje> comparison.
Well, I'm running *with* full error-checking in Mule in 21.5, and on a
Pentium II 450MHz with a slow disk it takes about 30 seconds to enter
emacs-devel (about 2000 unread at the moment).
--
School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
Ask not how you can "do" free software business;
ask what your business can "do for" free software.
2:17 p.m.
"Stephen J. Turnbull" <stephen(a)xemacs.org> writes:
>>>>> "Hrvoje" == Hrvoje Niksic
<hniksic(a)xemacs.org> writes:
Hrvoje> The problem is that people who tried it claim that XEmacs
Hrvoje> 21.5 is much *slower* than 21.4, especially for Gnus.
Hrvoje> That's why I was reluctant to even include it in the
Hrvoje> comparison.
Well, I'm running *with* full error-checking in Mule in 21.5, and on
a Pentium II 450MHz with a slow disk it takes about 30 seconds to
enter emacs-devel (about 2000 unread at the moment).
OK. Now try a group with 4000 unread. Does it take 60 seconds or 120
seconds? (Also note that disk speed shouldn't be relevant for nnml,
which reads data from a single .overview file.)
It's not about the raw speed; it's the quadratic characteristic that
is highly undesirable for large-volume groups.
12:30 p.m.
>>>> "Stephen" == Stephen J Turnbull
<stephen(a)xemacs.org> writes:
>>>> "Hrvoje" == Hrvoje Niksic
<hniksic(a)xemacs.org> writes:
Hrvoje> The problem is that people who
tried it claim that XEmacs
Hrvoje> 21.5 is much *slower* than 21.4, especially for Gnus.
Hrvoje> That's why I was reluctant to even include it in the
Hrvoje> comparison.
Stephen> Well, I'm running *with* full error-checking in Mule in
Stephen> 21.5, and on a Pentium II 450MHz with a slow disk it takes
Stephen> about 30 seconds to enter emacs-devel (about 2000 unread
Stephen> at the moment).
I just want to point that may be you have not considered all possible
reasons for the speed loss of Xemacs against GNU.
I yesterday did some testing, latest official GNU emacs with no-gnu0.4
vrs xemacs 21.4.17 no mule.
While GNU emacs was on WARP speed all the time even for copying and
moving a lot of msg between imap folder, xemacs was well slow.
Uwe Brauer
5:58 p.m.
>>>> "Uwe" == Uwe Brauer <oub(a)mat.ucm.es>
writes:
Uwe> I just want to point that may be you have not considered all
Uwe> possible reasons for the speed loss of Xemacs against GNU.
Sure. I can name a number of them. Contributions on any of them
would be welcome.
Startup:
(1) Packages vs. monolithic distribution. We do a _lot_ more stats.
I suspect this has regressed somewhat, too.
(2) Portable dumper. I doubt this is _big_, but certainly it has to
be slower than unexec.
Runtime:
(1) The extent issues we've been discussing.
(2) I suspect redisplay is slowed dramatically by code kluged in to
ensure correct display of the native widgets. I know for sure that in
Xft most everything gets displayed twice on every redisplay.
(3) Regexp issues. It's not hard to write regexps that behave badly
in loops, and XEmacs's regexp implementation has had problems with
corner cases.
(4) Algorithms. I suspect that for most packages GNU has a decided
advantage in terms of the amount of developer time that goes into
optimizing for them, and there's a strong bias toward designing things
that are efficient for GNU, and then hacking in XEmacs support later.
Even for stuff at the C level, this is often a problem. Naive ports
of stuff like the syntax table property have caused huge performance
problems.
--
School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
Ask not how you can "do" free software business;
ask what your business can "do for" free software.
7:04 p.m.
>>>> "Stephen" == Stephen J Turnbull
<stephen(a)xemacs.org> writes:
>>>> "Uwe" == Uwe Brauer <oub(a)mat.ucm.es>
writes:
Stephen> Sure. I can name a number of them. Contributions on any
Stephen> of them would be welcome.
Stephen> (2) I suspect redisplay is slowed dramatically by code
Stephen> kluged in to ensure correct display of the native widgets.
Stephen> I know for sure that in Xft most everything gets displayed
Stephen> twice on every redisplay.
Stephen> (3) Regexp issues. It's not hard to write regexps that
Stephen> behave badly in loops, and XEmacs's regexp implementation
Stephen> has had problems with corner cases.
Stephen> (4) Algorithms. I suspect that for most packages GNU has
Stephen> a decided advantage in terms of the amount of developer
Stephen> time that goes into optimizing for them, and there's a
Stephen> strong bias toward designing things that are efficient for
Stephen> GNU, and then hacking in XEmacs support later.
Ok I see, how could I test that in the case I mentioned (copying and
moving a lot of msg from imap folders to nnml folder, or even between
imap folders) is slow?
Can I run some kind of profiler to find out, in of the points
mentioned above could be the culprit?
Uwe
9:37 p.m.
Uwe Brauer <oub(a)mat.ucm.es> writes:
>>>>> "Stephen" == Stephen J Turnbull
<stephen(a)xemacs.org> writes:
>>>>> "Uwe" == Uwe Brauer <oub(a)mat.ucm.es> writes:
Stephen> Sure. I can name a number of them. Contributions on any
Stephen> of them would be welcome.
Stephen> (2) I suspect redisplay is slowed dramatically by code
Stephen> kluged in to ensure correct display of the native widgets.
Stephen> I know for sure that in Xft most everything gets displayed
Stephen> twice on every redisplay.
Stephen> (3) Regexp issues. It's not hard to write regexps that
Stephen> behave badly in loops, and XEmacs's regexp implementation
Stephen> has had problems with corner cases.
Stephen> (4) Algorithms. I suspect that for most packages GNU has
Stephen> a decided advantage in terms of the amount of developer
Stephen> time that goes into optimizing for them, and there's a
Stephen> strong bias toward designing things that are efficient for
Stephen> GNU, and then hacking in XEmacs support later.
Ok I see, how could I test that in the case I mentioned (copying and
moving a lot of msg from imap folders to nnml folder, or even between
imap folders) is slow?
Can I run some kind of profiler to find out, in of the points
mentioned above could be the culprit?
Hi Uwe, you could try some of
a* profile-command - Run COMMAND, profiling the execution and displaying
the results.
a* profile-expression - Eval EXPR, profiling the execution and displaying the
results.
a* profile-key-sequence - Dispatch the key sequence KEYS, profile the execution
and show the results.
a* profile-results - Print profiling info INFO to STREAM in a pretty
format.
l* profile-sexp - Profile some lisp expression.
--
Adrian Aichner
mailto:adrian＠xemacs.org
http://www.xemacs.org/
10:24 a.m.
Adrian Aichner wrote:
Uwe Brauer <oub(a)mat.ucm.es> writes:
>>>>>>"Stephen" == Stephen J Turnbull
<stephen(a)xemacs.org> writes:
>>>>>>
>>>>>>
>>>>>>"Uwe" == Uwe Brauer <oub(a)mat.ucm.es> writes:
>>>>>>
>>>>>>
> Stephen> Sure. I can name a number of them. Contributions on any
> Stephen> of them would be welcome.
>
>
> Stephen> (2) I suspect redisplay is slowed dramatically by code
> Stephen> kluged in to ensure correct display of the native widgets.
> Stephen> I know for sure that in Xft most everything gets displayed
> Stephen> twice on every redisplay.
>
> Stephen> (3) Regexp issues. It's not hard to write regexps that
> Stephen> behave badly in loops, and XEmacs's regexp implementation
> Stephen> has had problems with corner cases.
>
> Stephen> (4) Algorithms. I suspect that for most packages GNU has
> Stephen> a decided advantage in terms of the amount of developer
> Stephen> time that goes into optimizing for them, and there's a
> Stephen> strong bias toward designing things that are efficient for
> Stephen> GNU, and then hacking in XEmacs support later.
>
>Ok I see, how could I test that in the case I mentioned (copying and
>moving a lot of msg from imap folders to nnml folder, or even between
>imap folders) is slow?
>
>Can I run some kind of profiler to find out, in of the points
>mentioned above could be the culprit?
>
>
Hi Uwe, you could try some of
a* profile-command - Run COMMAND, profiling the execution and displaying
the results.
a* profile-expression - Eval EXPR, profiling the execution and displaying the
results.
a* profile-key-sequence - Dispatch the key sequence KEYS, profile the execution
and show the results.
a* profile-results - Print profiling info INFO to STREAM in a pretty
format.
l* profile-sexp - Profile some lisp expression.
you can also simply do `M-x toggle-profiling', do something or other,
and then do `M-x toggle-profiling' again. this will print out profiling
results for the code run while profiling was enabled. before doing this
though, it would be much better to recompile xemacs
`--without-error-checking', which can majorly distort the results.
ben
5:58 a.m.
>>>> "Ben" == Ben Wing <ben(a)xemacs.org>
writes:
Ben> you can also simply do `M-x toggle-profiling', do something
Ben> or other, and then do `M-x toggle-profiling' again. this
Ben> will print out profiling results for the code run while
Ben> profiling was enabled. before doing this though, it would be
Ben> much better to recompile xemacs `--without-error-checking',
Ben> which can majorly distort the results.
I realize that timing will be way off, but does error checking also
affect the counts?
--
School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
Ask not how you can "do" free software business;
ask what your business can "do for" free software.
7:10 a.m.
Stephen J. Turnbull wrote:
>>>>>"Ben" == Ben Wing <ben(a)xemacs.org>
writes:
>>>>>
>>>>>
Ben> you can also simply do `M-x toggle-profiling', do something
Ben> or other, and then do `M-x toggle-profiling' again. this
Ben> will print out profiling results for the code run while
Ben> profiling was enabled. before doing this though, it would be
Ben> much better to recompile xemacs `--without-error-checking',
Ben> which can majorly distort the results.
I realize that timing will be way off, but does error checking also
affect the counts?
it shouldn't affect the gc or function-call counts.
5:52 p.m.
>>>> "APA" == Adrian Aichner
<adrian(a)xemacs.org> writes:
APA> Uwe Brauer <oub(a)mat.ucm.es> writes:
[snip]
APA> a* profile-command - Run COMMAND, profiling the
APA> execution and displaying the results.
Hello Adrian,
Thanks this command worked fine, I did various tests, either having all
my packages loaded or not, I attach the one, with all packages used.
I looked for profile commands in gnus emacs, there is no per default,
google did not really help,
Could somebody of the gnu folks[1] please point me out how to profile a
command in either gnu emacs 21 or 22.
Thanks
Uwe Brauer
[1] thats why I post also to the gnus group
7:24 p.m.
Uwe Brauer <oub(a)mat.ucm.es> writes:
>>>>> "APA" == Adrian Aichner
<adrian(a)xemacs.org> writes:
APA> Uwe Brauer <oub(a)mat.ucm.es> writes:
[snip]
APA> a* profile-command - Run COMMAND, profiling the
APA> execution and displaying the results.
Hello Adrian,
Thanks this command worked fine, I did various tests, either having all
my packages loaded or not, I attach the one, with all packages used.
I looked for profile commands in gnus emacs, there is no per default,
google did not really help,
Could somebody of the gnu folks[1] please point me out how to profile a
command in either gnu emacs 21 or 22.
M-x elp-instrument-package RET gnus RET
Do something
M-x elp-results RET
--
David Kastrup, Kriemhildstr. 15, 44793 Bochum
10:50 a.m.
>>>> "David" == David Kastrup
<dak(a)gnu.org> writes:
David> M-x elp-instrument-package RET gnus RET
David> Do something
David> M-x elp-results RET
Thanks, this function also exists in xemacs (and I did not know about
it), in any case
the information is very aehm spare:
for emacs-21 I obtain:
--8<------------------------schnipp------------------------->8---
Function Name Call Count Elapsed Time Average Time
========================= ========== ============ ============
gnus-summary-copy-article 1 14.03726 14.03726
--8<------------------------schnapp------------------------->8---
for xemacs 21.4.17 I get
--8<------------------------schnipp------------------------->8---
Function Name Call Count Elapsed Time Average Time
========================= ========== ============ ============
gnus-summary-copy-article 1 15.869385 15.869385
--8<------------------------schnapp------------------------->8---
However the profile-command of xemacs give me detailed information of
the sort:
--8<------------------------schnipp------------------------->8---
Function Name Ticks/Total %Usage Calls GC-Usage/ Total
===========================================/===== ====== ===== ========/=======
(in garbage collection) 237/ 0 39.109
next-single-property-change 70/ 0 11.551 24664
get-text-property 33/ 0 5.446 29299
--8<------------------------schnapp------------------------->8---
Etc etc
I tried to compile xemacs, profile package under emacs-21 without
success. Does there exist any add-on package for more profiling
details?
Thanks
Uwe Brauer
11:48 a.m.
Uwe Brauer <oub(a)mat.ucm.es> writes:
>>>>> "David" == David Kastrup
<dak(a)gnu.org> writes:
David> M-x elp-instrument-package RET gnus RET
David> Do something
David> M-x elp-results RET
Thanks, this function also exists in xemacs (and I did not know about
it), in any case
the information is very aehm spare:
for emacs-21 I obtain:
--8<------------------------schnipp------------------------->8---
Function Name Call Count Elapsed Time Average Time
========================= ========== ============ ============
gnus-summary-copy-article 1 14.03726 14.03726
--8<------------------------schnapp------------------------->8---
for xemacs 21.4.17 I get
--8<------------------------schnipp------------------------->8---
Function Name Call Count Elapsed Time Average Time
========================= ========== ============ ============
gnus-summary-copy-article 1 15.869385 15.869385
--8<------------------------schnapp------------------------->8---
Well, you should instrument the package _after_ loading all relevant
functions, or just the autoloads get instrumented.
--
David Kastrup, Kriemhildstr. 15, 44793 Bochum
6:50 p.m.
Uwe Brauer <oub(a)mat.ucm.es> writes:
>>>>> "David" == David Kastrup
<dak(a)gnu.org> writes:
David> M-x elp-results RET
Thanks, this function also exists in xemacs (and I did not know
about it), in any case the information is very aehm spare:
You should probably also instrument the `nnml', `imap', `mm' and `mml'
packages (and maybe some more depending on which operation you're
benchmarking -- I don't recall).
11:07 p.m.
Jesper Harder <jesper.harder(a)gmail.com> writes:
Uwe Brauer <oub(a)mat.ucm.es> writes:
>>>>>> "David" == David Kastrup <dak(a)gnu.org> writes:
>
> David> M-x elp-results RET
>
> Thanks, this function also exists in xemacs (and I did not know
> about it), in any case the information is very aehm spare:
You should probably also instrument the `nnml', `imap', `mm' and `mml'
packages (and maybe some more depending on which operation you're
benchmarking -- I don't recall).
And it helps to run the test first before instrumenting it: that way
not only the autoloads get instrumented.
--
David Kastrup, Kriemhildstr. 15, 44793 Bochum
10:32 a.m.
>>>> "David" == David Kastrup
<dak(a)gnu.org> writes:
>
>>>>>>> "David" == David Kastrup <dak(a)gnu.org>
writes:
>>
David> M-x elp-results RET
>>
>> Thanks, this function also exists in xemacs (and I did not know
>> about it), in any case the information is very aehm spare:
>
> You should probably also instrument the `nnml', `imap', `mm' and
`mml'
> packages (and maybe some more depending on which operation you're
> benchmarking -- I don't recall).
David> And it helps to run the test first before instrumenting it: that way
David> not only the autoloads get instrumented.
Ok, thanks, I think now more or less I know how to proceed. As I said
before the situation is sort of strange: for a certain amount of msg,
say n, both emacsen are almost equal, gnu emacs a little faster, once
the number of msg > n, then xemacs got stacked.
I should add, that the information of the elp packages is detailed the
profile-command of xemacs gives far more information. I don't know how
important profiling is for emacs, but may that pkg could be ported?
2:02 a.m.
Uwe Brauer wrote:
>>>>>"David" == David Kastrup
<dak(a)gnu.org> writes:
>>>>>
>>>>>
>>
>>>>>>>> "David" == David Kastrup <dak(a)gnu.org>
writes:
>>>
David> M-x elp-results RET
>>>
>>> Thanks, this function also exists in xemacs (and I did not know
>>> about it), in any case the information is very aehm spare:
>>
>> You should probably also instrument the `nnml', `imap', `mm' and
`mml'
>> packages (and maybe some more depending on which operation you're
>> benchmarking -- I don't recall).
David> And it helps to run the test first before instrumenting it: that way
David> not only the autoloads get instrumented.
Ok, thanks, I think now more or less I know how to proceed. As I said
before the situation is sort of strange: for a certain amount of msg,
say n, both emacsen are almost equal, gnu emacs a little faster, once
the number of msg > n, then xemacs got stacked.
it would be very nice to know where the problem is. sometimes typing
Control-Shift-G in XEmacs will break out and give you a backtrace. (it
should always do that but maybe sometimes doesn't ...) If not, try
setting Options->Troubleshooting->Debug on Quit, then manually do `kill
-INT ###', where ### is the process id of XEmacs. (maybe it's `kill
-SIGINT ###', i forget)
I should add, that the information of the elp packages is detailed
the
profile-command of xemacs gives far more information. I don't know how
important profiling is for emacs, but may that pkg could be ported?
the problem is that the XEmacs profile stuff depends on C support, which
is how it's able to do all the nifty things it does.
ben
10:24 a.m.
>>>> "Ben" == Ben Wing <ben(a)xemacs.org>
writes:
Ben> Uwe Brauer wrote:
>
>
Ben> it would be very nice to know where the problem is. sometimes
Ben> typing
it seems that it has to do with the size of the msg. Mails with large
attachment are the problem in xemacs, but still I have to look that
up in detail.
Ben> Control-Shift-G in XEmacs will break out and give you a
Ben> backtrace. (it should always do that but maybe sometimes
Ben> doesn't ...) If not, try setting
Ben> Options->Troubleshooting->Debug on Quit, then manually do
Ben> `kill -INT ###', where ### is the process id of XEmacs. (maybe
Ben> it's `kill -SIGINT ###', i forget)
problem is that under recent X-free versions the shift key does not
work properly, that is control shift g, is the same as control g for
xemacs, but I do as you suggest set is debug on quit.
I report back,
Uwe
10:52 a.m.
>>>> "Ben" == Ben Wing <ben(a)xemacs.org>
writes:
Ben> Uwe Brauer wrote:
>
>
Ben> it would be very nice to know where the problem is.
Ben> sometimes typing Control-Shift-G in XEmacs will break out
Ben> and give you a backtrace. (it should always do that but
Ben> maybe sometimes doesn't ...) If not, try setting
Ben> Options->Troubleshooting->Debug on Quit, then manually do
Ben> `kill -INT ###', where ### is the process id of
Ben> XEmacs. (maybe it's `kill -SIGINT ###', i forget)
Ok, I continued with the testing:
- the problem does not occur when copying a large amount of
msg between nnml servers! Xemacs is as fast as gnu emacs
- the problem seem to be connected with `large' msg: msg with
an attachment >0.5 Mega. I tried this with xemacs -vanilla
either nognus-0.3 or the actual xemacs pkg release. Now
xemacs does not really get stack, after some Control g key
strokes I can continue typing (and even with
debug-on-signal t no useful backtrace is produced). However the
nnimap command got *stacked*. I can not leave neither the group
not the server
The command
lisp-process
lists me the imap command
imap open *nnimap* ucimap.ucm.es (none) network stream
connection 143(a)ucimap.ucm.es
could please somebody confirm this?
But I don't know how to kill that process, ps ux does not list that
process.
Again that problem does not occur in gnu emacs and now I begin to
understand why me complains about the slowness of the nnimap command
were ignored on the gnus list: it seems to be more of a xemacs
problem.
If someone could suggest me how to enhance the testing in order to
localise the problem...
Regards
Uwe Brauer
12:09 p.m.
Uwe Brauer <oub(a)mat.ucm.es> writes:
The command
lisp-process
lists me the imap command
imap open *nnimap* ucimap.ucm.es (none) network stream
connection 143(a)ucimap.ucm.es
But I don't know how to kill that process, ps ux does not list that
process.
Kill the buffer " *nnimap *ucimap.ucm.es" (note leading SPC).
If someone could suggest me how to enhance the testing in order to
localise the problem...
Instrument imap* and nnimap* for ELP. Note that nnimap is likely to
be slowed than nnml, but I'm not sure what magnitude of "slower" you
are talking about here.
12:55 p.m.
>>>> "Simon" == Simon Josefsson
<jas(a)extundo.com> writes:
Simon> Uwe Brauer <oub(a)mat.ucm.es> writes:
Simon> Kill the buffer " *nnimap *ucimap.ucm.es" (note leading SPC).
> If someone could suggest me how to enhance the testing in order
to
> localise the problem...
Simon> Instrument imap* and nnimap* for ELP. Note that nnimap is
I did elp-instrument-package gnus (which I thought include everyting)
Or or profile-command (gnus-summary-copy-article) for xemacs.
But right I do as you say
Simon> likely to be slowed than nnml, but I'm not sure what
Simon> magnitude of "slower" you are talking about here.
Get stacked, I mean not doing anything (xemacs) compared to perform a
operation (gnu emacs)
1:27 p.m.
Uwe Brauer <oub(a)mat.ucm.es> writes:
>>>>> "Simon" == Simon Josefsson
<jas(a)extundo.com> writes:
Simon> Uwe Brauer <oub(a)mat.ucm.es> writes:
Simon> Kill the buffer " *nnimap *ucimap.ucm.es" (note leading SPC).
>> If someone could suggest me how to enhance the testing in order to
>> localise the problem...
Simon> Instrument imap* and nnimap* for ELP. Note that nnimap is
I did elp-instrument-package gnus (which I thought include everyting)
No, the (nn)imap functions in Gnus doesn't use the gnus function name
prefix, you need to instrument them manually. Same for message, mm,
pgg, sieve etc.
Simon> likely to be slowed than nnml, but I'm not sure
what
Simon> magnitude of "slower" you are talking about here.
Get stacked, I mean not doing anything (xemacs) compared to perform a
operation (gnu emacs)
Aha. Ok. That's bad.
1:06 p.m.
>>>> "Simon" == Simon Josefsson
<jas(a)extundo.com> writes:
>
> But I don't know how to kill that process, ps ux does not list that
> process.
Simon> Kill the buffer " *nnimap *ucimap.ucm.es" (note leading SPC).
That buffer does not exist in my case. I can only see the process but
I don't have that buffer (in xemacs-21.4.17)
11:04 a.m.
>>>> "Ben" == Ben Wing <ben(a)xemacs.org>
writes:
>
>
Ben> it would be very nice to know where the problem is. sometimes
Ben> typing Control-Shift-G in XEmacs will break out and give you a
Ben> backtrace. (it should always do that but maybe sometimes
Ben> doesn't ...) If not, try setting
Ben> Options->Troubleshooting->Debug on Quit, then manually do
Ben> `kill -INT ###', where ### is the process id of XEmacs. (maybe
Ben> it's `kill -SIGINT ###', i forget)
I forgot to add: in my nnimap folder/groups I have activated Gcc (to
that group) so that, via the nnimap command, my reply to a mail is
copied to that nnimap group. That command as well gets *stacked*, but
not in gnu emacs!
So I have the feeling that nnimap is not optimized for xemacs to put
it mildly.
Uwe
10 p.m.
On Mon, Oct 24 2005, Uwe Brauer wrote:
Could somebody of the gnu folks[1] please point me out how to profile
a
command in either gnu emacs 21 or 22.
,----
| elp-instrument-function M-x ... RET
| Command: Instrument FUNSYM for profiling.
| elp-instrument-list M-x ... RET
| Command: Instrument for profiling, all functions in `elp-function-list'.
| elp-instrument-package M-x ... RET
| Command: Instrument for profiling, all functions which start with PREFIX.
| elp-results M-x ... RET
| Command: Display current profiling results.
`----
Bye, Reiner.
--
,,,
(o o)
---ooO-(_)-Ooo--- | PGP key available | http://rsteib.home.pages.de/
5:53 p.m.
>>>> "APA" == Adrian Aichner
<adrian(a)xemacs.org> writes:
APA> a* profile-command - Run COMMAND, profiling the
APA> execution and displaying the results.
Here comes the profile protocol, I am not sure how to interpret it
Uwe
Function Name Ticks/Total %Usage Calls GC-Usage/ Total
===========================================/===== ====== ===== ========/=======
(in garbage collection) 237/ 0 39.109
next-single-property-change 70/ 0 11.551 24664
get-text-property 33/ 0 5.446 29299
write-region-internal 19/ 0 3.135 177
add-text-properties 16/ 0 2.640 1067
insert 13/ 0 2.145 1746
replace-match 12/ 0 1.980 13268
search-forward 12/ 0 1.980 15157
kill-buffer 11/ 0 1.815 2020
redisplay-echo-area 10/ 0 1.650 436
gnus-topic-update-topic-line 8/ 0 1.320 485
byte-code 7/ 0 1.155 7105
accept-process-output 6/ 0 0.990 752
get-buffer-create 6/ 0 0.990 2535
all-after-change-function 5/ 0 0.825 29264
expand-file-name 5/ 0 0.825 3093
re-search-forward 5/ 0 0.825 6453
all-before-change-function 4/ 0 0.660 29264
file-name-as-directory 4/ 0 0.660 3399
gnus-xmas-format 4/ 0 0.660 857
read 4/ 0 0.660 5974
gnus-topic-find-groups 4/ 0 0.660 485
imap-parse-flag-list 3/ 0 0.495 514
gnuserv-buffer-p 3/ 0 0.495 2020
gnus-group-update-group 3/ 0 0.495 97
auto-capitalize 3/ 0 0.495 29264
string-match 3/ 0 0.495 3544
gnus-agent-lib-file 3/ 0 0.495 451
file-attributes 3/ 0 0.495 615
process-send-string 3/ 0 0.495 729
file-exists-p 3/ 0 0.495 613
select-buffer-killed-text 2/ 0 0.330 4040
gnus-topic-update-topics-containing-group
2/ 0 0.330 97
< 2/ 0 0.330 18886
when 2/ 0 0.330 7729
process-send-region 2/ 0 0.330 80
intern 2/ 0 0.330 7250
intern-soft 2/ 0 0.330 11368
gnus-find-method-for-group 2/ 0 0.330 1581
generate-new-buffer-name 2/ 0 0.330 2019
si:require 2/ 0 0.330 177
file-directory-p 2/ 0 0.330 580
nnimap-request-set-mark 2/ 0 0.330 160
put 2/ 0 0.330 1699
mm-auto-mode-alist 2/ 0 0.330 384
insert-file-contents-internal 2/ 0 0.330 464
member* 2/ 0 0.330 2020
match-string 2/ 0 0.330 9942
nnimap-possibly-change-group 2/ 0 0.330 211
gnus-topic-parent-topic 2/ 0 0.330 6924
imap-mailbox-unselect 1/ 0 0.165 80
(in redisplay) 1/ 0 0.165
gnus-list-of-unread-articles 1/ 0 0.165 95
gnuserv-buffer-clients 1/ 0 0.165 2020
gnus-backlog-request-article 1/ 0 0.165 80
utf7-encode 1/ 0 0.165 424
text-property-any 1/ 0 0.165 1218
#<compiled-function (uid Uid) "...(22)" [maxuid minuid uid] 2>
1/ 0 0.165 34
directory-files 1/ 0 0.165 80
gnus-get-function 1/ 0 0.165 274
get-buffer 1/ 0 0.165 721
buffer-name 1/ 0 0.165 3765
looking-at 1/ 0 0.165 2853
point-at-eol 1/ 0 0.165 2382
gnus-group-name-charset 1/ 0 0.165 855
gnus-server-to-method 1/ 0 0.165 114
put-text-property 1/ 0 0.165 2425
gnus-agent-group-pathname 1/ 0 0.165 580
bm-buffer-save 1/ 0 0.165 2020
mm-insert-file-contents 1/ 0 0.165 384
raw-append-message 1/ 0 0.165 516
clear-message 1/ 0 0.165 519
nnimap-mark-to-flag-1 1/ 0 0.165 1099
subst-char-in-region 1/ 0 0.165 160
imap-arrival-filter 1/ 0 0.165 728
log-message 1/ 0 0.165 515
imap-mailbox-put 1/ 0 0.165 1345
delete-backward-char 1/ 0 0.165 2183
boundp 1/ 0 0.165 7474
mapcar 1/ 0 0.165 6876
follow-generic-filter 1/ 0 0.165 728
format 1/ 0 0.165 3509
erase-buffer 1/ 0 0.165 699
apply 1/ 0 0.165 2529
#<compiled-function nil "...(7)" [tex-wcount-timer tex-wcount-mode
cancel-timer] 2>
1/ 0 0.165 2020
truncate 1/ 0 0.165 2260
gnus-request-article-this-buffer 1/ 0 0.165 80
preview-kill-buffer-cleanup 1/ 0 0.165 2020
sort 1/ 0 0.165 298
gnus-prin1-to-string 1/ 0 0.165 177
gnus-dribble-enter 1/ 0 0.165 177
gnus-summary-move-article 1/ 0 0.165 1
gnus-agent-save-group-info 1/ 0 0.165 17
nnimap-close-group 1/ 0 0.165 17
#<compiled-function (name-method) "...(12)" [method name-method throw
server-name] 3>
1/ 0 0.165 12512
#<compiled-function (info) "...(27)" [match server method info info-server
info-method 4 server-name (byte-code
®
ÆÇ "ÆÈD
«É
BC B )"
A＠ʶ«ËÌ
＠"ªËÍ
＠
A＠#
B ¶¬ B * [name-method name gnus-select-method alists gnus-server-method-cache method mapc
#<compiled-function (name-method) "...(12)" [method name-method throw
server-name] 3> #<compiled-function (server-alist) "...(5)" [server-alist
mapc #<compiled-function (name-method) "...(22)" [method
gnus-server-method-cache name-method gnus-methods-equal-p throw server-name] 3>] 3>
native (nil ) format %s %s:%s gnus-server-alist gnus-predefined-server-alist] 5)] 3>
1/ 0 0.165 7378
noninteractive 0/ 0 0.000 13
make-extent 0/ 0 0.000 1
set-window-buffer 0/ 0 0.000 1
gnus-summary-copy-article 0/ 0 0.000 1
remq 0/ 0 0.000 4040
gnus-async-prefetched-article-entry 0/ 0 0.000 80
split-string 0/ 0 0.000 2
gnus-group-auto-expirable-p 0/ 0 0.000 80
handle-pre-motion-command-current-command-is-motion
0/ 0 0.000 1
gnus-agent-possibly-alter-active 0/ 0 0.000 194
buffer-list 0/ 0 0.000 3
equal 0/ 0 0.000 53
spam-backend-statistical-p 0/ 0 0.000 23
imap-message-flags-del 0/ 0 0.000 80
char-width 0/ 0 0.000 97
valid-specifier-locale-p 0/ 0 0.000 2
gutter-specifier-p 0/ 0 0.000 4
match-data 0/ 0 0.000 528
file-chase-links 0/ 0 0.000 1
gnus-group-topic-level 0/ 0 0.000 488
gnus-group-timestamp-delta 0/ 0 0.000 97
recursion-depth 0/ 0 0.000 1
window-dedicated-p 0/ 0 0.000 2
gnus-group-set-parameter 0/ 0 0.000 95
gnus-topic-goto-topic 0/ 0 0.000 585
gnus-remassoc 0/ 0 0.000 289
stop-profiling 0/ 0 0.000 1
select-window 0/ 0 0.000 1
valid-instantiator-p 0/ 0 0.000 8
eval 0/ 0 0.000 1351
getenv 0/ 0 0.000 1
string-to-number 0/ 0 0.000 81
gnus-agent-check-overview-buffer 0/ 0 0.000 80
session-kill-buffer-hook 0/ 0 0.000 2020
set-buffer-dedicated-frame 0/ 0 0.000 1
vectorp 0/ 0 0.000 27
count-lines 0/ 0 0.000 160
#<compiled-function (x) "...(23)" [modenm2 modenm1 mode2 mode1 x
string-match] 3>
0/ 0 0.000 52
bufferp 0/ 0 0.000 3032
gnus-backlog-setup 0/ 0 0.000 80
gnus-methods-using 0/ 0 0.000 194
position 0/ 0 0.000 26
gnus-group-find-parameter 0/ 0 0.000 80
imap-find-next-line 0/ 0 0.000 2911
imap-capability 0/ 0 0.000 160
gnus-set-difference 0/ 0 0.000 17
set-extent-property 0/ 0 0.000 109
gnus-summary-buffer-name 0/ 0 0.000 97
buffers-menu-omit-invisible-buffers 0/ 0 0.000 102
itimer-disable-timeout 0/ 0 0.000 27
point 0/ 0 0.000 2134
verify-visited-file-modtime 0/ 0 0.000 25
set-window-configuration 0/ 0 0.000 1
gnus-group-highlight-line 0/ 0 0.000 97
mm-subst-char-in-string 0/ 0 0.000 757
read-minibuffer-internal 0/ 0 0.000 1
select-buffer-killed-default 0/ 0 0.000 4040
append 0/ 0 0.000 2142
nnimap-request-accept-article 0/ 0 0.000 80
mm-enable-multibyte 0/ 0 0.000 194
update-tab-in-gutter 0/ 0 0.000 2
nnimap-mark-to-flag 0/ 0 0.000 1099
nnimap-find-minmax-uid 0/ 0 0.000 17
gnus-ephemeral-group-p 0/ 0 0.000 97
imap-opened 0/ 0 0.000 35
gnus-current-topics 0/ 0 0.000 80
gnus-make-directory 0/ 0 0.000 177
gnus-close-group 0/ 0 0.000 17
gnus-summary-find-next 0/ 0 0.000 80
minibuffer-window 0/ 0 0.000 1
window-frame 0/ 0 0.000 1
symbol-value-in-buffer 0/ 0 0.000 128
nnmail-group-pathname 0/ 0 0.000 580
imap-mailbox-get 0/ 0 0.000 769
gnus-put-text-property 0/ 0 0.000 582
gnus-request-set-mark 0/ 0 0.000 160
#<compiled-function (buf) "...(10)" [buffers-tab-filter-functions buf tmp-buf
mapc #<compiled-function (fun) "...(11)" [tmp-buf first-buf buf fun nil]
3>] 3>
0/ 0 0.000 32
specifier-instance 0/ 0 0.000 8
appt-check 0/ 0 0.000 1
gnus-group-mark-group 0/ 0 0.000 17
imap-mailbox-expunge 0/ 0 0.000 17
nuke-x-symbol 0/ 0 0.000 1
gnus-group-topic-parameters 0/ 0 0.000 80
gnus-agent-request-article 0/ 0 0.000 80
display-time-no-mail-sign 0/ 0 0.000 1
nnheader-replace-duplicate-chars-in-string
0/ 0 0.000 580
mapconcat 0/ 0 0.000 321
run-hook-with-args 0/ 0 0.000 595
gnus-topic-insert-topic-line 0/ 0 0.000 485
gnus-cache-member-of-class 0/ 0 0.000 80
regexp-quote 0/ 0 0.000 47
display-message 0/ 0 0.000 516
process-buffer 0/ 0 0.000 2184
gnus-correct-length 0/ 0 0.000 97
gnus-cache-possibly-enter-article 0/ 0 0.000 80
glyphp 0/ 0 0.000 2
imap-parse-resp-text 0/ 0 0.000 1260
write-region 0/ 0 0.000 177
console-type 0/ 0 0.000 951
prin1-to-string 0/ 0 0.000 177
gnus-make-hashtable 0/ 0 0.000 160
gnus-agent-read-and-cache-local 0/ 0 0.000 160
frame-type 0/ 0 0.000 951
gnus-extent-start-open 0/ 0 0.000 97
keysyms-equal 0/ 0 0.000 16
gnus-topic-group-indentation 0/ 0 0.000 3
help-keymap-with-help-key 0/ 0 0.000 1
delete-char 0/ 0 0.000 291
utf7-decode 0/ 0 0.000 194
message 0/ 0 0.000 517
gnus-group-next-unread-group 0/ 0 0.000 18
nnimap-possibly-change-server 0/ 0 0.000 291
user-login-name 0/ 0 0.000 1
gnus-check-backend-function 0/ 0 0.000 17
mapc-internal 0/ 0 0.000 1281
itimer-function-arguments 0/ 0 0.000 2
current-window-configuration 0/ 0 0.000 1
overlays-in 0/ 0 0.000 2020
utf7-encode-internal 0/ 0 0.000 424
gnus-add-text-properties 0/ 0 0.000 1067
gnus-agent-save-alist 0/ 0 0.000 80
set-itimer-recorded-run-time 0/ 0 0.000 4
file-readable-p 0/ 0 0.000 25
visited-file-modtime 0/ 0 0.000 2020
kill-all-local-variables 0/ 0 0.000 1
utf7-decode-internal 0/ 0 0.000 194
register-swap-out 0/ 0 0.000 2020
gnus-copy-sequence 0/ 0 0.000 80
gnus-summary-show-thread 0/ 0 0.000 80
gnus-group-process-prefix 0/ 0 0.000 1
fmakunbound 0/ 0 0.000 1
decode-buffer 0/ 0 0.000 7
gnus-cache-file-contents 0/ 0 0.000 534
mark 0/ 0 0.000 6
gnus-agent-regenerate-group 0/ 0 0.000 80
itimer-time-difference 0/ 0 0.000 297
enqueue-eval-event 0/ 0 0.000 12
nnimap-request-update-info-internal 0/ 0 0.000 17
current-time-zone 0/ 0 0.000 1
spam-setup-widening 0/ 0 0.000 1
gnus-group-position-point 0/ 0 0.000 18
gnus-cache-request-article 0/ 0 0.000 80
process-status 0/ 0 0.000 2131
gnus-add-to-range 0/ 0 0.000 80
gnus-group-remove-excess-properties 0/ 0 0.000 97
gnus-activate-group 0/ 0 0.000 17
gnus-async-set-buffer 0/ 0 0.000 80
re-search-backward 0/ 0 0.000 80
gnus-group-set-info 0/ 0 0.000 95
and 0/ 0 0.000 486
gnus-message 0/ 0 0.000 515
display-time-file-nonempty-p 0/ 0 0.000 1
princ 0/ 0 0.000 565
gnus-buffer-live-p 0/ 0 0.000 177
imap-parse-resp-text-code 0/ 0 0.000 1260
set-specifier-dirty-flag 0/ 0 0.000 2
store-match-data 0/ 0 0.000 528
gnus-range-length 0/ 0 0.000 97
display-time-evaluate-list 0/ 0 0.000 1
bury-buffer 0/ 0 0.000 177
gnus-group-remove-parameter 0/ 0 0.000 95
eq 0/ 0 0.000 483
gnus-virtual-group-p 0/ 0 0.000 97
overlay-recenter 0/ 0 0.000 2020
gnus-current-topic 0/ 0 0.000 83
delete* 0/ 0 0.000 5
while 0/ 0 0.000 757
specifier-type 0/ 0 0.000 4
insert-string 0/ 0 0.000 436
itimer-is-idle 0/ 0 0.000 1215
point-marker 0/ 0 0.000 97
int-to-string 0/ 0 0.000 841
reftex-kill-buffer-hook 0/ 0 0.000 2020
gnus-agent-article-name 0/ 0 0.000 580
nnoo-server-opened 0/ 0 0.000 18
null 0/ 0 0.000 32
set-specifier 0/ 0 0.000 2
imap-mailbox-select-1 0/ 0 0.000 97
gnus-agent-method-p 0/ 0 0.000 291
buffer-substring 0/ 0 0.000 1476
gnus-cache-file-name 0/ 0 0.000 80
display-time-convert-load 0/ 0 0.000 1
marker-buffer 0/ 0 0.000 4040
fboundp 0/ 0 0.000 472
mod 0/ 0 0.000 1
remove-message 0/ 0 0.000 519
gnus-visual-p 0/ 0 0.000 80
butlast 0/ 0 0.000 12
#<compiled-function (r1 r2) "...(20)" [r2 r1] 2>
0/ 0 0.000 80
spam-backend-list 0/ 0 0.000 1
imap-message-map 0/ 0 0.000 17
device-type 0/ 0 0.000 951
gnus-summary-highlight-line-0 0/ 0 0.000 80
gnus-parse-active 0/ 0 0.000 17
atomic-extent-pre-hook 0/ 0 0.000 1
if 0/ 0 0.000 8680
length 0/ 0 0.000 757
nuke-selective-display 0/ 0 0.000 1
assoc 0/ 0 0.000 11520
gnus-mode-line-buffer-identification 0/ 0 0.000 275
gnus-summary-update-secondary-mark 0/ 0 0.000 80
vm-delete-buffer-frame 0/ 0 0.000 2020
gnus-group-insert-group-line-info 0/ 0 0.000 97
#<compiled-function (mark) "...(13)" [marks mark
imap-message-flag-permanent-p nnimap-mark-to-flag] 3>
0/ 0 0.000 160
set-gutter-dirty-p 0/ 0 0.000 2
gnus-group-topic-unread 0/ 0 0.000 1358
nnimap-mark-permanent-p 0/ 0 0.000 238
nnimap-before-find-minmax-bugworkaround
0/ 0 0.000 17
canonicalize-spec 0/ 0 0.000 2
nnheader-insert-file-contents 0/ 0 0.000 384
gnus-kill-all-overlays 0/ 0 0.000 160
gnus-agent-synchronize-group-flags 0/ 0 0.000 80
gnus-group-remove-mark 0/ 0 0.000 17
preview-clearout 0/ 0 0.000 2020
cdr 0/ 0 0.000 153
aset 0/ 0 0.000 582
nnheader-set-temp-buffer 0/ 0 0.000 81
gnus-topic-find-topology 0/ 0 0.000 2676
gnus-group-name-to-method 0/ 0 0.000 160
make-vector 0/ 0 0.000 257
imap-fetch 0/ 0 0.000 17
gnus-topic-parameters 0/ 0 0.000 400
gnuserv-kill-buffer-query-function 0/ 0 0.000 2020
tempbuf-check-buffers 0/ 0 0.000 1
#<compiled-function (pred) "...(42)" [info pred recent
nnimap-mark-permanent-p nnimap-mark-to-flag imap-mailbox-get flags gnus-info-set-entry
gnus-update-alist-soft gnus-compress-sequence imap-search nnimap-mark-to-predicate 3]
8>
0/ 0 0.000 238
#<compiled-function (s) "...(13)" [propname func result s UID] 4>
0/ 0 0.000 34
gnus-online 0/ 0 0.000 292
gnus-group-prefixed-name 0/ 0 0.000 228
nnheader-insert 0/ 0 0.000 17
gnus-cache-fully-p 0/ 0 0.000 80
decode-time 0/ 0 0.000 1
gnus-get-split-value 0/ 0 0.000 1
append-message 0/ 0 0.000 516
gnus-group-method 0/ 0 0.000 2
file-symlink-p 0/ 0 0.000 1
selected-frame 0/ 0 0.000 1035
symbol-name 0/ 0 0.000 3441
device-console 0/ 0 0.000 951
gnus-agent-save-local 0/ 0 0.000 80
mm-encode-coding-string 0/ 0 0.000 177
current-time-string 0/ 0 0.000 1
gnus-agent-load-local 0/ 0 0.000 274
add-timeout 0/ 0 0.000 27
#<compiled-function (symbol) "...(43)" [range symbol boundp symbol-name
(+dirty +method) prin1 princ
] 2>
0/ 0 0.000 241
gnus-parameters-get-parameter 0/ 0 0.000 80
console-on-window-system-p 0/ 0 0.000 3
nnimap-server-opened 0/ 0 0.000 18
progn 0/ 0 0.000 1649
copy-keymap 0/ 0 0.000 1
imap-parse-fetch 0/ 0 0.000 194
gnus-group-group-name 0/ 0 0.000 131
subtract-time 0/ 0 0.000 97
reverse 0/ 0 0.000 2
#<compiled-function (buffer) "...(18)" [buffers-tab-switch-to-buffer-function
buffers-tab-format-buffer-line-function buffer selected vector buffer-name :selected nil]
5>
0/ 0 0.000 6
valid-specifier-tag-p 0/ 0 0.000 2
current-message-label 0/ 0 0.000 2
rassoc 0/ 0 0.000 2
gnus-group-parameter-value 0/ 0 0.000 388
quote 0/ 0 0.000 582
gnus-topic-visible-p 0/ 0 0.000 485
remove* 0/ 0 0.000 1
insert-file-contents 0/ 0 0.000 464
buffer-disable-undo 0/ 0 0.000 82
vm-update-composition-buffer-name 0/ 0 0.000 2
nnheader-narrow-to-headers 0/ 0 0.000 80
add-spec-list-to-specifier 0/ 0 0.000 2
make-local-variable 0/ 0 0.000 2025
copy-sequence 0/ 0 0.000 1122
gnus-group-goto-group 0/ 0 0.000 97
minibuffer-depth 0/ 0 0.000 5
window-buffer 0/ 0 0.000 731
bm-repository-remove 0/ 0 0.000 2020
display-time-function 0/ 0 0.000 1
gnus-summary-update-mark 0/ 0 0.000 80
marker-position 0/ 0 0.000 6
gnus-server-equal 0/ 0 0.000 194
nnheader-translate-file-chars 0/ 0 0.000 1788
imap-message-flags-add 0/ 0 0.000 80
mark-marker 0/ 0 0.000 6
frame-selected-window 0/ 0 0.000 2
character-to-event 0/ 0 0.000 1
gnus-completing-read-with-default 0/ 0 0.000 1
display-time-can-do-graphical-display 0/ 0 0.000 3
nnimap-request-group 0/ 0 0.000 17
gnus-summary-set-article-display-arrow 0/ 0 0.000 240
select-buffers-tab-buffers-by-mode 0/ 0 0.000 32
prin1 0/ 0 0.000 81
extentp 0/ 0 0.000 4040
gnus-summary-position-point 0/ 0 0.000 339
reftex-view-crossref-when-idle 0/ 0 0.000 2
gnus-uncompress-range 0/ 0 0.000 17
#<compiled-function (pair) "...(31)" [day-of-download pair comp-list
compressed article-id nil] 4>
0/ 0 0.000 80
itimerp 0/ 0 0.000 3232
file-name-directory 0/ 0 0.000 97
gnus-create-hash-size 0/ 0 0.000 160
featurep 0/ 0 0.000 184
gnus-group-set-mode-line 0/ 0 0.000 274
0/ 0 0.000 97
gnus-group-get-parameter 0/ 0 0.000 468
default-gutter-position 0/ 0 0.000 2
event-live-p 0/ 0 0.000 1
gnus-agent-set-local 0/ 0 0.000 174
gnus-group-next-group 0/ 0 0.000 17
gnus-server-add-address 0/ 0 0.000 17
gnus-server-opened 0/ 0 0.000 18
use-local-map 0/ 0 0.000 1
mapc 0/ 0 0.000 1281
bm-lists 0/ 0 0.000 2020
itimer-uses-arguments 0/ 0 0.000 11
= 0/ 0 0.000 8020
#<compiled-function (article-id) "...(8)" [state article-id uncomp] 2>
0/ 0 0.000 969
search-backward 0/ 0 0.000 17
display-time-generate-load-glyphs 0/ 0 0.000 1
buffers-tab-items 0/ 0 0.000 2
log-message-filter 0/ 0 0.000 515
valid-image-instantiator-format-p 0/ 0 0.000 2
itimer-recorded-run-time 0/ 0 0.000 243
#<compiled-function (flag) "...(12)" [flag mapconcat identity ] 4>
0/ 0 0.000 160
number-to-string 0/ 0 0.000 1084
call-interactively 0/ 0 0.000 1
last 0/ 0 0.000 338
imap-current-mailbox-p 0/ 0 0.000 211
previous-single-property-change 0/ 0 0.000 83
itimer-run-expired-timers 0/ 0 0.000 27
imap-mailbox-select 0/ 0 0.000 97
gnus-put-text-property-excluding-characters-with-faces
0/ 0 0.000 97
event-modifiers 0/ 0 0.000 1
natnump 0/ 0 0.000 2952
balloon-help-go-away 0/ 0 0.000 1
nnheader-report 0/ 0 0.000 17
key-press-event-p 0/ 0 0.000 1
#<compiled-function nil "...(102)" [minibuffer-completion-predicate
minibuffer-confirm-incomplete completion minibuffer-completion-table event buffer-string
throw exit nil exact-minibuffer-completion-p t try-completion temp-minibuffer-message
[incomplete; confirm] [no completions; confirm] next-command-event quit-char
event-to-character dispatch-event inhibit-quit quit-flag last-command-event] 5 855346
nil>
0/ 0 0.000 1
completing-read 0/ 0 0.000 1
gnus-agent-update-view-total-fetched-for
0/ 0 0.000 80
gnus-group-real-prefix 0/ 0 0.000 3
spam-widening-needed-p 0/ 0 0.000 1
nnheader-concat 0/ 0 0.000 1031
imap-send-command-wait 0/ 0 0.000 632
characterp 0/ 0 0.000 33
disable-timeout 0/ 0 0.000 27
#<compiled-function (extent arg) "...(6)" [extent set-extent-property
start-open t] 4>
0/ 0 0.000 97
markerp 0/ 0 0.000 8080
gnus-remove-from-range 0/ 0 0.000 97
imap-message-put 0/ 0 0.000 354
delete-if 0/ 0 0.000 4
input-pending-p 0/ 0 0.000 25
gnus-run-hooks 0/ 0 0.000 275
insert-char 0/ 0 0.000 2394
gnus-article-mark-to-type 0/ 0 0.000 1040
#<compiled-function nil "...(225)" [uncached default-low default-high
gnus-spam-mark score mark spam-face gnus-summary-cancelled-face
gnus-summary-high-undownloaded-face gnus-summary-low-undownloaded-face
gnus-summary-normal-undownloaded-face gnus-summary-high-ticked-face
gnus-summary-low-ticked-face gnus-summary-normal-ticked-face
gnus-summary-high-ancient-face gnus-summary-low-ancient-face
gnus-summary-normal-ancient-face gnus-summary-high-unread-face
gnus-summary-low-unread-face gnus-summary-normal-unread-face gnus-summary-high-read-face
gnus-summary-low-read-face gnus-summary-normal-read-face gnus-dormant-mark
gnus-ticked-mark gnus-ancient-mark gnus-unread-mark gnus-canceled-mark] 2>
0/ 0 0.000 80
gnus-group-search-forward 0/ 0 0.000 18
gnus-remove-denial 0/ 0 0.000 17
remove 0/ 0 0.000 1
gnus-group-group-indentation 0/ 0 0.000 194
gnus-summary-goto-subject 0/ 0 0.000 240
map-extents 0/ 0 0.000 6317
imap-wait-for-tag 0/ 0 0.000 712
delete 0/ 0 0.000 98
make-string 0/ 0 0.000 585
gnus-check-server 0/ 0 0.000 1
gnus-group-name-decode 0/ 0 0.000 952
frame-property 0/ 0 0.000 2
gnus-region-active-p 0/ 0 0.000 2
let 0/ 0 0.000 854
canonicalize-inst-list 0/ 0 0.000 4
gnus-update-alist-soft 0/ 0 0.000 84
gnus-summary-work-articles 0/ 0 0.000 1
encode-time-value 0/ 0 0.000 97
string-to-int 0/ 0 0.000 2475
gnus-async-request-fetched-article 0/ 0 0.000 80
#<subr when> 0/ 0 0.000 7729
read-from-minibuffer 0/ 0 0.000 1
run-hooks 0/ 0 0.000 470
set-glyph-property 0/ 0 0.000 2
cl-position 0/ 0 0.000 26
format-buffers-tab-line 0/ 0 0.000 6
gnus-server-status 0/ 0 0.000 18
imap-message-appenduid-1 0/ 0 0.000 80
#<compiled-function (char) "...(8)" [char length char-width] 3>
0/ 0 0.000 97
vector 0/ 0 0.000 9
buffer-modified-p 0/ 0 0.000 597
- 0/ 0 0.000 97
imap-search 0/ 0 0.000 101
gnus-set-mode-line 0/ 0 0.000 1
imap-send-command 0/ 0 0.000 649
aref 0/ 0 0.000 7729
char-to-string 0/ 0 0.000 781
imap-parse-response 0/ 0 0.000 2183
gnus-agent-read-agentview 0/ 0 0.000 127
format-decode 0/ 0 0.000 218
gnus-group-decoded-name 0/ 0 0.000 758
gnus-compress-sequence 0/ 0 0.000 244
load-average 0/ 0 0.000 2
frame-device 0/ 0 0.000 951
gnus-request-accept-article 0/ 0 0.000 80
mapatoms 0/ 0 0.000 97
font-lock-pre-idle-hook 0/ 0 0.000 4
gnus-group-insert-group-line 0/ 0 0.000 97
frame-live-p 0/ 0 0.000 952
gnus-get-unread-articles-in-group 0/ 0 0.000 97
gnus-use-long-file-name 0/ 0 0.000 80
get-buffer-window 0/ 0 0.000 728
gnus-summary-highlight-line 0/ 0 0.000 80
overlay-lists 0/ 0 0.000 4040
concat 0/ 0 0.000 1403
1+ 0/ 0 0.000 7729
mm-disable-multibyte 0/ 0 0.000 17
setq 0/ 0 0.000 7729
valid-specifier-tag-set-p 0/ 0 0.000 2
mapcar-extents 0/ 0 0.000 2020
imap-send-command-1 0/ 0 0.000 649
auto-revert-buffers 0/ 0 0.000 5
throw 0/ 0 0.000 1188
nnimap-mark-to-predicate 0/ 0 0.000 67
ldiff 0/ 0 0.000 1
canonicalize-spec-list 0/ 0 0.000 2
balloon-help-displayed 0/ 0 0.000 1
set-window-hscroll 0/ 0 0.000 1
#<compiled-function (fun) "...(11)" [tmp-buf first-buf buf fun nil] 3>
0/ 0 0.000 32
itimer-restart 0/ 0 0.000 22
imap-current-mailbox 0/ 0 0.000 97
gnus-short-group-name 0/ 0 0.000 1
selected-window 0/ 0 0.000 729
set-buffer-modified-p 0/ 0 0.000 574
reset-buffer 0/ 0 0.000 1
handle-pre-motion-command 0/ 0 0.000 1
set-glyph-image 0/ 0 0.000 2
imap-message-append 0/ 0 0.000 80
gnus-topic-hierarchical-parameters 0/ 0 0.000 80
delq 0/ 0 0.000 2453
buffer-string 0/ 0 0.000 1476
gnus-group-get-new-news-this-group 0/ 0 0.000 1
find 0/ 0 0.000 26
#<compiled-function (keysym) "...(31)" [mods key keysym butlast keysyms-equal
last throw handle-pre-motion-command-current-command-is-motion t] 4>
0/ 0 0.000 24
gnus-topic-update-unreads 0/ 0 0.000 485
iso-acc-minibuf-setup 0/ 0 0.000 1
identity 0/ 0 0.000 6
sit-for 0/ 0 0.000 1
itimer-value 0/ 0 0.000 1551
device-live-p 0/ 0 0.000 951
require 0/ 0 0.000 177
buffer-file-name 0/ 0 0.000 2070
gnus-agent-read-local 0/ 0 0.000 160
gnus-read-move-group-name 0/ 0 0.000 1
abs 0/ 0 0.000 18
imap-parse-status 0/ 0 0.000 17
#<compiled-function (comp-list) "...(8)" [comp-list gnus-compress-sequence]
3>
0/ 0 0.000 80
force-mode-line-update 0/ 0 0.000 1
#<compiled-function (buffer) "...(34)" [tempbuf-timeout tempbuf-last-time
tempbuf-mode buffer ct get-buffer-window t tempbuf-post-command tempbuf-grace
tempbuf-time-diff tempbuf-expire] 3>
0/ 0 0.000 51
hash-table-count 0/ 0 0.000 4
buffer-enable-undo 0/ 0 0.000 1
event-type 0/ 0 0.000 1
gnus-agent-get-local 0/ 0 0.000 194
imap-message-flag-permanent-p 0/ 0 0.000 398
gnus-info-set-entry 0/ 0 0.000 84
#<compiled-function (item) "...(17)" [item format %d:%d %d] 4>
0/ 0 0.000 160
nnimap-verify-uidvalidity 0/ 0 0.000 97
truncate-command-history-for-gc 0/ 0 0.000 12
canonicalize-inst-pair 0/ 0 0.000 6
itimer-function 0/ 0 0.000 11
#<compiled-function (x) "...(13)" [gnus-valid-select-methods
gnus-select-method method x assoc symbol-name] 4>
0/ 0 0.000 194
car-less-than-car 0/ 0 0.000 3789
display-time-string-to-char-list 0/ 0 0.000 1
gnus-agent-update-files-total-fetched-for
0/ 0 0.000 80
nnoo-current-server 0/ 0 0.000 18
set-itimer-value-internal 0/ 0 0.000 173
gnus-agent-load-alist 0/ 0 0.000 260
gnus-delete-file 0/ 0 0.000 2
display-time-generate-time-glyphs 0/ 0 0.000 1
#<compiled-function (comp-list) "...(108)" [sequence comp-list last result
first ranges nil mapcar #<compiled-function (article-id) "...(8)" [state
article-id uncomp] 2> state] 5>
0/ 0 0.000 204
imap-current-mailbox-p-1 0/ 0 0.000 308
gnuserv-kill-buffer-function 0/ 0 0.000 2020
event-key 0/ 0 0.000 1
gnus-summary-remove-process-mark 0/ 0 0.000 1
buffer-size 0/ 0 0.000 674
balloon-help-pre-command-hook 0/ 0 0.000 1
find-if 0/ 0 0.000 26
current-time 0/ 0 0.000 2179
imap-range-to-message-set 0/ 0 0.000 160
x-symbol-setup-minibuffer 0/ 0 0.000 1
nnimap-update-unseen 0/ 0 0.000 17
display-time-convert-num 0/ 0 0.000 1
buffer-live-p 0/ 0 0.000 35
imap-mailbox-status 0/ 0 0.000 17
itimer-timer-driver 0/ 0 0.000 27
point-at-bol 0/ 0 0.000 3415
-------------------------------------------------------------------------------
Total 606 100.000 626828 0
Ticks/Total = Ticks this function/this function and descendants
Calls = Number of calls to this function
GC-Usage/Total = Lisp allocation this function/this function and descendants
One tick = 1 ms
6:46 p.m.
Uwe Brauer <oub(a)mat.ucm.es> writes:
Here comes the profile protocol, I am not sure how to interpret it
If "in garbage collection" comes first, it means you need to bump
`gc-cons-threshold' to something reasonable. 2000000 is probably a
good start.
Then, don't forget that the profile* family of functions *accumulate*
profile results, meaning that the displayed summary doesn't contain
the latest profiling run, but a sum of all profiling runs. This can
be surprising if you spend some time testing the profiler and then
want to produce "real" data. In that case you need to use `M-x
clear-profiling-info'.
4:28 a.m.
Hrvoje Niksic wrote:
Uwe Brauer <oub(a)mat.ucm.es> writes:
>Here comes the profile protocol, I am not sure how to interpret it
>
>
If "in garbage collection" comes first, it means you need to bump
`gc-cons-threshold' to something reasonable. 2000000 is probably a
good start.
Then, don't forget that the profile* family of functions *accumulate*
profile results, meaning that the displayed summary doesn't contain
the latest profiling run, but a sum of all profiling runs. This can
be surprising if you spend some time testing the profiler and then
want to produce "real" data. In that case you need to use `M-x
clear-profiling-info'.
actually, it hasn't done this for awhile; i changed it a couple of years
ago to not accumulate by default, since it seemed to me that it is very
confusing and rarely what you want. currently, if you want to continue
a profile session and not reset it, you have to use a C-u prefix to
profile-*.
ben
11:07 a.m.
Ben Wing <ben(a)xemacs.org> writes:
actually, it hasn't done this for awhile; i changed it a couple
of
years ago to not accumulate by default,
I stand corrected (and I agree with this change).
What I said about garbage collection time still stands, though.
Increasing gc-cons-threshold will both reduce the time spent in gc and
give you a more realistic benchmark.
10:55 a.m.
>>>> "Hrvoje" == Hrvoje Niksic
<hniksic(a)xemacs.org> writes:
Hrvoje> Uwe Brauer <oub(a)mat.ucm.es> writes:
> Here comes the profile protocol, I am not sure how to interpret
it
Hrvoje> If "in garbage collection" comes first, it means you need to bump
Hrvoje> `gc-cons-threshold' to something reasonable. 2000000 is probably a
Hrvoje> good start.
Hrvoje> Then, don't forget that the profile* family of functions
Hrvoje> *accumulate* profile results, meaning that the displayed
Hrvoje> summary doesn't contain the latest profiling run, but a sum
Hrvoje> of all profiling runs. This can be surprising if you spend
Hrvoje> some time testing the profiler and then want to produce
Hrvoje> "real" data. In that case you need to use `M-x
Hrvoje> clear-profiling-info'.
Thanks, I followed both advised now the profiling results start like
--8<------------------------schnipp------------------------->8---
Function Name Ticks/Total %Usage Calls GC-Usage/ Total
===========================================/===== ====== ===== ========/=======
(in garbage collection) 49/ 0 13.462
get-text-property 30/ 0 8.242 16616
next-single-property-change 30/ 0 8.242 13926
--8<------------------------schnapp------------------------->8---
And given your comment above I doubt that the change of
gc-cons-threshold had any effect.
I also start to wonder where the problem lies? so far copying 80 msg
from imap folder to imap folder did not show a significant difference
in speed between gnus emacs and xemacs, *however* when I try to copy
160 msg then gnus emacs needs nearly twice the time compared to the 80
while xemacs simply get stuck and I have to abort. I may try to
interpolate between the 80 and 160 msg in order to see where the two
start to diverge.
Uwe
11:56 a.m.
Uwe Brauer <oub(a)mat.ucm.es> writes:
Thanks, I followed both advised now the profiling results start like
--8<------------------------schnipp------------------------->8---
Function Name Ticks/Total %Usage Calls GC-Usage/ Total
===========================================/===== ====== ===== ========/=======
(in garbage collection) 49/ 0 13.462
get-text-property 30/ 0 8.242 16616
next-single-property-change 30/ 0 8.242 13926
--8<------------------------schnapp------------------------->8---
And given your comment above I doubt that the change of
gc-cons-threshold had any effect.
The effect is that GC time dropped from 237 ticks to 49 ticks (a
five-fold speedup!) and from 39% to 13% of time. That is quite
decent.
4:43 p.m.
>>>> "Uwe" == Uwe Brauer <oub(a)mat.ucm.es>
writes:
Uwe> I also start to wonder where the problem lies? so far copying
Uwe> 80 msg from imap folder to imap folder did not show a
Uwe> significant difference in speed between gnus emacs and
Uwe> xemacs, *however* when I try to copy 160 msg then gnus emacs
Uwe> needs nearly twice the time compared to the 80 while xemacs
Uwe> simply get stuck and I have to abort. I may try to
Uwe> interpolate between the 80 and 160 msg in order to see where
Uwe> the two start to diverge.
We would greatly appreciate it if you would do that. If the copying
goes via XEmacs/Gnus (as opposed to simply being an IMAP command),
then it is possible that there is a single message that is confusing
XEmacs for some reason. If we could identify that, then we can
probably fix a bug.
Similarly, it is possible that at some byte count (close to a multile
of the size of a buffer, for example) XEmacs gets wedged.
Steve
--
School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
Ask not how you can "do" free software business;
ask what your business can "do for" free software.
1:12 p.m.
Hrvoje Niksic wrote:
The problem is that people who tried it claim that XEmacs 21.5 is
much
*slower* than 21.4, especially for Gnus. That's why I was reluctant
to even include it in the comparison.
maybe they're comparing apples and oranges, i.e. error-checking and
no-error-checking. i'd be rather surprised if xemacs 21.5 actually got
slower, and i'd certainly like to know why.
ben
2:33 p.m.
>>>> "Ben" == Ben Wing <ben(a)xemacs.org>
writes:
Ben> Hrvoje Niksic wrote:
> The problem is that people who tried it claim that XEmacs 21.5 is
much
> *slower* than 21.4, especially for Gnus. That's why I was reluctant
> to even include it in the comparison.
>
>
>
Ben> maybe they're comparing apples and oranges, i.e.
error-checking and
Ben> no-error-checking. i'd be rather surprised if xemacs 21.5 actually
Ben> got slower, and i'd certainly like to know why.
I can tell you that 21.5-b18 is way, way faster than later versions.
In a GNUS buffer with about 1000 messages, 21.5-b18 was acceptably
fast on my Sun Blade 1500 at work. When I upgraded to 21.5-b22, it
takes 2-3 times longer. (I've sent a message about this to the list
before, with some profiling results.) Both were compiled with Mule
enabled. b18 had error-checking on, but b22 has it off.
In my day-to-day usage, this is the only thing I really notice as
being slower.
Ray
4:45 p.m.
Raymond Toy <raymond.toy(a)ericsson.com> wrote:
I can tell you that 21.5-b18 is way, way faster than later versions.
In a GNUS buffer with about 1000 messages, 21.5-b18 was acceptably
fast on my Sun Blade 1500 at work. When I upgraded to 21.5-b22, it
takes 2-3 times longer. (I've sent a message about this to the list
before, with some profiling results.) Both were compiled with Mule
enabled. b18 had error-checking on, but b22 has it off.
Which garbage collector are you using? I've had to go back to the
old (non-default) garbage collector, as the new one is unacceptably slow
for me.
--
Darryl Okahata
darrylo(a)soco.agilent.com
DISCLAIMER: this message is the author's personal opinion and does not
constitute the support, opinion, or policy of Agilent Technologies, or
of the little green men that have been following him all day.
11:01 p.m.
>>>> "Darryl" == Darryl Okahata
<darrylo(a)soco.agilent.com> writes:
Darryl> Raymond Toy <raymond.toy(a)ericsson.com> wrote:
> I can tell you that 21.5-b18 is way, way faster than later
versions.
> In a GNUS buffer with about 1000 messages, 21.5-b18 was acceptably
> fast on my Sun Blade 1500 at work. When I upgraded to 21.5-b22, it
> takes 2-3 times longer. (I've sent a message about this to the list
> before, with some profiling results.) Both were compiled with Mule
> enabled. b18 had error-checking on, but b22 has it off.
Darryl> Which garbage collector are you using? I've had to go back to
the
Darryl> old (non-default) garbage collector, as the new one is unacceptably slow
Darryl> for me.
I used the default GC. I should try the old GC, then.
Thanks,
Ray
10:28 p.m.
Raymond Toy wrote:
>>>>>"Ben" == Ben Wing <ben(a)xemacs.org>
writes:
>>>>>
>>>>>
Ben> Hrvoje Niksic wrote:
>> The problem is that people who tried it claim that XEmacs 21.5 is much
>> *slower* than 21.4, especially for Gnus. That's why I was reluctant
>> to even include it in the comparison.
>>
>>
>>
Ben> maybe they're comparing apples and oranges, i.e. error-checking and
Ben> no-error-checking. i'd be rather surprised if xemacs 21.5 actually
Ben> got slower, and i'd certainly like to know why.
I can tell you that 21.5-b18 is way, way faster than later versions.
In a GNUS buffer with about 1000 messages, 21.5-b18 was acceptably
fast on my Sun Blade 1500 at work. When I upgraded to 21.5-b22, it
takes 2-3 times longer. (I've sent a message about this to the list
before, with some profiling results.) Both were compiled with Mule
enabled. b18 had error-checking on, but b22 has it off.
In my day-to-day usage, this is the only thing I really notice as
being slower.
Ray
hmm. well, the byte-char algorithm was changed between b18 and b19.
but various other things changed between 18 and 22 as well. could you
try compiling b19 and see whether it gets slower or faster? ideally,
both b18 and b19 should be optimized, without error-checking.
ben
11:04 p.m.
>>>> "Ben" == Ben Wing <ben(a)xemacs.org>
writes:
Ben> Raymond Toy wrote:
[snip]
Ben> hmm. well, the byte-char algorithm was changed between b18 and b19.
Ben> but various other things changed between 18 and 22 as well. could you
Ben> try compiling b19 and see whether it gets slower or faster? ideally,
Ben> both b18 and b19 should be optimized, without error-checking.
From my old mails, I profiled b18 and b20; I never tried (or even
downloaded) b19 for some reason.
I'll give that a try soon.
And I see from my old mails, I tried b20 with the --disable-kkcc and
--disable-mc-alloc. Still slow, but somewhat better. Things got
better when I disabled error-checking, but still quite a bit slower.
Ray
2:22 a.m.
Raymond Toy wrote:
>>>>>"Ben" == Ben Wing <ben(a)xemacs.org>
writes:
>>>>>
>>>>>
Ben> Raymond Toy wrote:
[snip]
Ben> hmm. well, the byte-char algorithm was changed between b18 and b19.
Ben> but various other things changed between 18 and 22 as well. could you
Ben> try compiling b19 and see whether it gets slower or faster? ideally,
Ben> both b18 and b19 should be optimized, without error-checking.
>From my old mails, I profiled b18 and b20; I never tried (or even
downloaded) b19 for some reason.
I'll give that a try soon.
And I see from my old mails, I tried b20 with the --disable-kkcc and
--disable-mc-alloc. Still slow, but somewhat better. Things got
better when I disabled error-checking, but still quite a bit slower.
b20 isn't very different from b19. before you go to the trouble of
downloading it, could you repost the profiling info from b20? if it's
spending a lot of time in byte-char conversion, that should be
immediately obvious.
2:21 p.m.
>>>> "Ben" == Ben Wing <ben(a)xemacs.org>
writes:
Ben> Raymond Toy wrote:
> And I see from my old mails, I tried b20 with the --disable-kkcc
and
> --disable-mc-alloc. Still slow, but somewhat better. Things got
> better when I disabled error-checking, but still quite a bit slower.
>
>
Ben> b20 isn't very different from b19. before you go to the
trouble of
Ben> downloading it, could you repost the profiling info from b20? if it's
Ben> spending a lot of time in byte-char conversion, that should be
Ben> immediately obvious.
The relevant thread can be found here:
http://search.gmane.org/?query=Profile+results+for+21.5.21+and+21.5.18&am...
(The profile results are rather large.)
But in light of Stephen's message, perhaps I should try to recompile
b18 and b20 with error-checking off, and get profile results for
those.
But as I mentioned, b22 with error-checking off is still rather slow
and much slower than b18 with error-checking on.
Ray
10:39 a.m.
>>>> "Raymond" == Raymond Toy
<raymond.toy(a)ericsson.com> writes:
Raymond> Things got better when I disabled error-checking, but
Raymond> still quite a bit slower.
You *must* disable error-checking for the results to be at all
meaningful, unless (as I did) you simply want to post a "worst-case I
can think of" result. (In my case I had Mule, full error checking,
and a slow---by today's standards---heavily-loaded machine, and I
still wasn't seeing Hrvoje's "minutes".)
Speed losses due to error-checking are not considered at all when we
add more error checking, so they can vary wildly across versions, and
generally will become worse and worse over time.
--
School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
Ask not how you can "do" free software business;
ask what your business can "do for" free software.
12:14 p.m.
>>>> "Ben" == Ben Wing <ben(a)xemacs.org>
writes:
Ben> it's also still possible it's the text property
Ben> implementation. i know that the extent code works very badly
Ben> in the presence of a single extent that covers the entire
Ben> buffer
Urk. A text property is the obvious way to implement
language-dependent behavior, of course. It's really ugly to have a
buffer property that's actually a text property just because
performance sucks when an extent covers the whole buffer.
--
School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
Ask not how you can "do" free software business;
ask what your business can "do for" free software.
10:24 a.m.
Stephen J. Turnbull wrote:
>>>>>"Ben" == Ben Wing <ben(a)xemacs.org>
writes:
>>>>>
>>>>>
Ben> it's also still possible it's the text property
Ben> implementation. i know that the extent code works very badly
Ben> in the presence of a single extent that covers the entire
Ben> buffer
Urk. A text property is the obvious way to implement
language-dependent behavior, of course. It's really ugly to have a
buffer property that's actually a text property just because
performance sucks when an extent covers the whole buffer.
what do you mean? as i said before, i have mostly finished code that
will make text properties efficient by implementing them entirely in
non-overlapping intervals, as fsf does.
the problem with an extent covering the entire buffer is that it causes
the "stack of extents" [soe] caching mechanism that determines which
extents cover a particular position to entirely break down. i think the
problem is that the current algorithm at least needs to examine all
extents within the boundaries of the union of all extents covering the
current position in order to find the correct set of extents overlapping
an adjacent position. when there is an extent covering the whole
buffer, all soe operations have to scan all extents in the buffer. i'm
sure there are more sophisticated data structures to help with this
problem, but in general this is a very tough problem, and using simpler
solutions when possible (e.g. non-overlapping text properties). it
might be possible to simulate extents in general using non-overlapping
intervals, where multiple intervals could represent a single extent and
each interval contains parts of all extents crossing over them. this
interval would be guaranteed efficient and not too hard to implement, i
think -- in fact it's the same algorithm i'd be using for
text-properties, and could be extended to extents in general.
but first we need some more profiling data.
redisplay and certain other operations need to know about all extents
covering a particular position, which is cached using a structure.
map-extents, for example, when told to map over all extents overlapping
a region, uses the soe structure to determine the bounds within which it
needs to look. but when one extent covers the entire buffer, the
current code that maintains the soe and moves it to adjoining buffer
positions breaks down at least it is impossible to maintain the the
caching info from the soe will be useless, and it will always have to
uses ; but an extent over the entire buffer essentially invalidates
this cache, since updating it requires looking over
2:39 a.m.
>>>> "Ben" == Ben Wing <ben(a)xemacs.org>
writes:
Ben> Stephen J. Turnbull wrote:
> It's really ugly to have a buffer property that's
actually a
> text property just because performance sucks when an extent
> covers the whole buffer.
I assume the above is what you're talking about?
Ben> what do you mean? as i said before, i have mostly finished
Ben> code that will make text properties efficient by implementing
Ben> them entirely in non-overlapping intervals, as fsf does.
What I mean is that there are a number of things that we do as buffer
attributes or buffer-local variables (language environment, coded
character set, major mode for starters) that conceptually are text
properties. Think about it: wouldn't it be cool if instead of needing
a hack like mmm, we simply attached LISP-major-mode to an extent
covering the buffer, string-major-mode to strings, and
comment-major-mode to comments? Wouldn't it be cool if we could
attach xml-mode to individual strings in xml.el or use html-mode for
strings by default in blog-mode.el?
Ben> in general this is a very tough problem, and using simpler
Ben> solutions when possible (e.g. non-overlapping text
Ben> properties).
Whatever it is it has to be part of the implementation, not a
constraint on user code. (I'm pretty sure you agree, but it's not
clear to me from that sentence.)
Ben> it might be possible to simulate extents in general using
Ben> non-overlapping intervals, where multiple intervals could
Ben> represent a single extent and each interval contains parts of
Ben> all extents crossing over them. this interval would be
Ben> guaranteed efficient and not too hard to implement, i think
Ben> -- in fact it's the same algorithm i'd be using for
Ben> text-properties, and could be extended to extents in general.
Something like this? A linked list P of (buffer position, extent
list) pairs, with an intervals defined as [P[n].pos,P[n+1].pos) for
each n. An extent is a pair of pointers into P defining which
fragments are covered by the extent plus a property list. P[n].ext is
the list of extents covering that interval. Then for random access,
finding n is O(lg n), and finding the property is O(|P[n].ext|), which
is the best you can get, I guess. (For many operations, in particular
redisplay, which walk P in order, finding n is O(1), right?) P itself
only needs to be updated when text is inserted/deleted in the buffer,
and when an extent is created/deleted. |P| <= |# boundary points of
all extents|, so that is no worse than what we currently have, and we
never have to update the extents themselves.
Ben> but first we need some more profiling data.
Why? Basically in your explanation of how the soe works, you are
saying that large extents (not just extents covering the whole buffer,
but any extent that overlaps many other extents) are very expensive
because they make _all_ the overlapped extents inefficient. So that
means that code like
(let ((header (make-extent (message-beginning) (header-end)))
(body (make-extent (header-end) (message-end))))
(font-lock-fontify-buffer))
necessarily sucks. If I'm not totally missing the point, I have to
ask: Do you consider that restriction acceptable?
--
School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
Ask not how you can "do" free software business;
ask what your business can "do for" free software.
6:42 a.m.
>>>> "sjt" == Stephen J Turnbull
<stephen(a)xemacs.org> writes:
sjt> Why? Basically in your explanation of how the soe works, you
sjt> are saying that large extents (not just extents covering the
sjt> whole buffer, but any extent that overlaps many other
sjt> extents) are very expensive because they make _all_ the
sjt> overlapped extents inefficient. So that means that code like
(let ((header (make-extent (message-beginning) (header-end)))
(body (make-extent (header-end) (message-end))))
(font-lock-fontify-buffer))
sjt> necessarily sucks. If I'm not totally missing the point, I
sjt> have to ask: Do you consider that restriction acceptable?
For example, here's the output of (extents-at (point)) in a Gnus
summary buffer:
(#<extent (1, 10542) D paste-function text-prop 0xa062cec> ;; Uh-oh....
#<extent (10425, 10465) D paste-function gnus-number text-prop 0x9f51b64>
#<extent (10425, 10464) D paste-function text-prop 0xa068620>
#<extent (10432, 10463) HD paste-function text-prop 0x9f51d24>
#<extent [10432, 10463) 0x9f7b32c>)
Apparently there are 4 such extents per line, and one line per
article. And
One extent to rule them all, and in the slowness bind them.
--
School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
Ask not how you can "do" free software business;
ask what your business can "do for" free software.
1:56 a.m.
New subject: Speed difference in Gnus: a step in the right direction
>>>> "sjt" == Stephen J Turnbull
<stephen(a)xemacs.org> writes:
sjt> For example, here's the output of (extents-at (point)) in a Gnus
sjt> summary buffer:
sjt> (#<extent (1, 10542) D paste-function text-prop 0xa062cec> ;;
Uh-oh....
The presence of this monster extent is due to Gnus's abuse of text
properties as a "portable" way to unify overlays and text properties
across Emacsen. As usual, Gnus abuses turn out to be a great way to
find XEmacs bugs, though!
What happens is that because text properties coalesce, some property
that is applied to every line in the summary buffer ends up swallowing
the whole buffer. The text property in question is 'start-open, and
the value is t. (This is badly broken, since 'start-open is not a
property of the text, but of the extent itself; it cannot usefully be
a text property since extents carry only one text property. Ie, what
we have here is an extent whose only purpose is to announce that it
itself is start-open.)
BTW: I wonder if this is why custom, especially the face editor, seems
slow to start up; IIRC custom buffers have a buffer-wide read-only
property.
Quick fix: This patch eliminates this text property (completely safe
since this is a no-op as far as Gnus functionality is concerned), and
gives me a 33% speedup based on wall clock time in summary generation
on a 2500 message group in an XEmacs with Mule and full error checking
ON. NOT TESTED in an OPTIMIZED XEmacs yet. (Reading of headers is
still unconscionably slow for me, but I believe that's documented in
the Gnus manual for large nnml groups because of all the disk I/O
needed.)
--- gnus-xmas.el~ 2004-09-28 11:21:10.000000000 +0900
+++ gnus-xmas.el 2005-10-20 09:16:57.000000000 +0900
＠＠ -200,12 +200,10 ＠＠
(add-spec-to-specifier current-display-table table (current-buffer) nil)))
(defun gnus-xmas-add-text-properties (start end props &optional object)
- (add-text-properties start end props object)
- (put-text-property start end 'start-closed nil object))
+ (add-text-properties start end props object))
(defun gnus-xmas-put-text-property (start end prop value &optional object)
- (put-text-property start end prop value object)
- (put-text-property start end 'start-closed nil object))
+ (put-text-property start end prop value object))
(defun gnus-xmas-extent-start-open (point)
(map-extents (lambda (extent arg)
GNUS HACKERS TAKE NOTE: The function `gnus-xmas-extent-start-open'
maps over the extents near some arbitrary buffer position; I'm not
sure from Ben's descriptions whether this is an operation that is
affected by large extents. It's only used in the *Group* buffer IIRC,
so even if so, probably the performance hit is negligible for most
people.
--
School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
Ask not how you can "do" free software business;
ask what your business can "do for" free software.
7:03 a.m.
New subject: Speed difference in Gnus: a step in the right direction
Stephen J. Turnbull wrote:
>>>>>"sjt" == Stephen J Turnbull
<stephen(a)xemacs.org> writes:
>>>>>
>>>>>
sjt> For example, here's the output of (extents-at (point)) in a Gnus
sjt> summary buffer:
sjt> (#<extent (1, 10542) D paste-function text-prop 0xa062cec> ;;
Uh-oh....
The presence of this monster extent is due to Gnus's abuse of text
properties as a "portable" way to unify overlays and text properties
across Emacsen. As usual, Gnus abuses turn out to be a great way to
find XEmacs bugs, though!
What happens is that because text properties coalesce, some property
that is applied to every line in the summary buffer ends up swallowing
the whole buffer. The text property in question is 'start-open, and
the value is t. (This is badly broken, since 'start-open is not a
property of the text, but of the extent itself; it cannot usefully be
a text property since extents carry only one text property. Ie, what
we have here is an extent whose only purpose is to announce that it
itself is start-open.)
i think gnus is trying to do the equivalent of setting the
`front-sticky' and `rear-sticky' properties. one of the things i tried
to do in my new text-property implementation is implement these
properties directly.
GNUS HACKERS TAKE NOTE: The function
`gnus-xmas-extent-start-open'
maps over the extents near some arbitrary buffer position; I'm not
sure from Ben's descriptions whether this is an operation that is
affected by large extents.
yes, i think so. if i remember aright, the problem is in updating the
soe cache, which lists all extents overlapping some specific position.
map_extents() definitely uses the soe, and moves it appropriately as it
maps. (unless you give it some flag like `start-in-region' or
`end-in-region', which makes life easier; or unless you define the
constant DONT_USE_SOE [for debugging purposes only!], in which case it
just brute-forces things by starting from the beginning of the buffer.)
ben
10:15 a.m.
New subject: Speed difference in Gnus: a step in the right direction
"Stephen J. Turnbull" <stephen(a)xemacs.org> writes:
(Reading of headers is still unconscionably slow for me, but I
believe that's documented in the Gnus manual for large nnml groups
because of all the disk I/O needed.)
What disk I/O? nnml reads the headers from a single .overview file.
The way I understand it, nnml is supposed to be the "fast" one.
Reading the headers is fairly slow, but its speed has never been
O(n^2).
11 a.m.
New subject: Speed difference in Gnus: a step in the right direction
>>>> "Hrvoje" == Hrvoje Niksic
<hniksic(a)xemacs.org> writes:
Hrvoje> "Stephen J. Turnbull" <stephen(a)xemacs.org> writes:
> (Reading of headers is still unconscionably slow for me, but I
> believe that's documented in the Gnus manual for large nnml groups
> because of all the disk I/O needed.)
Hrvoje> What disk I/O? nnml reads the headers from a single .overview file.
Thanks for the correction. Something is dog-slow on my box, though.
Maybe my config is broken somehow. I'll profile it.
Hrvoje> The way I understand it, nnml is supposed to be the "fast" one.
Hrvoje> Reading the headers is fairly slow, but its speed has never been
Hrvoje> O(n^2).
My reading of the manual is that it's a tradeoff between entering the
group (nnml slow, nnfolder fast) and working with individual messages
(nnml fast, nnfolder slow).
--
School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
Ask not how you can "do" free software business;
ask what your business can "do for" free software.
11:14 a.m.
New subject: Speed difference in Gnus: a step in the right direction
"Stephen J. Turnbull" <stephen(a)xemacs.org> writes:
Hrvoje> The way I understand it, nnml is supposed to be the
Hrvoje> "fast" one. Reading the headers is fairly slow, but its
Hrvoje> speed has never been O(n^2).
My reading of the manual is that it's a tradeoff between entering
the group (nnml slow, nnfolder fast) and working with individual
messages (nnml fast, nnfolder slow).
I seem to understand the opposite. nnml is fast on entry because of
the overview index, and slow when working with individual messages
because it works with a bunch of small files (nnfolder uses an
in-memory buffer, presumably saved in the end).
For example, from the "Comparing Back Ends" node:
[nnml] maintains a Usenet-style active file [...] and also creates
"overview" files *for efficient group entry* [emphasis mine], as
has been defined for NNTP servers for some years now. It is
slower in mail-splitting, due to the creation of lots of files,
updates to the `nnml' active file, and additions to overview files
on a per-message basis, but it is extremely fast on access because
of what amounts to the indexing support provided by the active
file and overviews.
12:31 p.m.
New subject: Speed difference in Gnus: a step in the right direction
On Thu, Oct 20 2005, Stephen J. Turnbull wrote:
>>>>> "Hrvoje" == Hrvoje Niksic
<hniksic(a)xemacs.org> writes:
Hrvoje> "Stephen J. Turnbull" <stephen(a)xemacs.org> writes:
>> (Reading of headers is still unconscionably slow for me, but I
>> believe that's documented in the Gnus manual for large nnml groups
>> because of all the disk I/O needed.)
Hrvoje> What disk I/O? nnml reads the headers from a single
Hrvoje> .overview file.
Thanks for the correction. Something is dog-slow on my box, though.
Maybe my config is broken somehow. I'll profile it.
Scoring on non-overview headers or on body might be responsible for
this. But I guess you tested with scoring turned off. Or you have
set `nnml-nov-is-evil' to t?
Bye, Reiner.
--
,,,
(o o)
---ooO-(_)-Ooo--- | PGP key available | http://rsteib.home.pages.de/
11:39 a.m.
New subject: Speed difference in Gnus: a step in the right direction
Hrvoje Niksic wrote:
"Stephen J. Turnbull" <stephen(a)xemacs.org> writes:
>(Reading of headers is still unconscionably slow for me, but I
>believe that's documented in the Gnus manual for large nnml groups
>because of all the disk I/O needed.)
>
>
What disk I/O? nnml reads the headers from a single .overview file.
The way I understand it, nnml is supposed to be the "fast" one.
Reading the headers is fairly slow, but its speed has never been
O(n^2).
Hrvoje, if you're willing to try some profiling in a current 21.5, it
would give me some good data and we could figure out for sure where the
O(n^2) behavior is coming from. i know 21.5 has gotten slower due to
changes to the char-byte algorithm, but still, the profiling output yet
get would be immensely helpful (if you're feeling wary about the latest
21.5, you could use beta 18, although i wouldn't be modifying that.) If
you can present me with some profiles of the latest 21.5 (or of b18),
optimized w/o error-checking, of doing the operations that document the
N^2 behavior, I'll make a new 21.5 that adds appropriate profile
sections to help narrow down further what's going wrong, and also
probably make some changes based on the info you give me so that things
speed up -- e.g. see about fixing the byte-char slowdown. if it's text
props, that will take longer, but at least we'll have confirmed the
source of the problem rather than it being some yet other slowness that
i'm not aware of.
thanks,
ben
3:17 p.m.
New subject: Speed difference in Gnus: a step in the right direction
Ben Wing <ben(a)xemacs.org> writes:
Hrvoje, if you're willing to try some profiling in a current
21.5,
I don't have time for it now, and I'm reluctant to work on the 21.5
code base anyway, because of the reasons outlined earlier. I started
this thread to make other people aware of the problem.
Creating a large nnml group shouldn't be hard. For example, enter
Gnus, press "n" when asked to go offline, then press "y" when asked
whether to continue. You can create an empty nnml group using `G m
testgroup RET nnml RET'. Then enter an existing mbox using `G f
<file-name> RET' (don't mind if this is slow, it's supposed to be
slow). Press `C-x h' to select the whole buffer, followed by `B c
nnml:tesgroup RET'. You should have a working nnml group with a bunch
of messages.
4:36 a.m.
New subject: Speed difference in Gnus: a step in the right direction
Hrvoje Niksic wrote:
Ben Wing <ben(a)xemacs.org> writes:
>Hrvoje, if you're willing to try some profiling in a current 21.5,
>
>
I don't have time for it now, and I'm reluctant to work on the 21.5
code base anyway, because of the reasons outlined earlier. I started
this thread to make other people aware of the problem.
Creating a large nnml group shouldn't be hard. For example, enter
Gnus, press "n" when asked to go offline, then press "y" when asked
whether to continue. You can create an empty nnml group using `G m
testgroup RET nnml RET'. Then enter an existing mbox using `G f
<file-name> RET' (don't mind if this is slow, it's supposed to be
slow). Press `C-x h' to select the whole buffer, followed by `B c
nnml:tesgroup RET'. You should have a working nnml group with a bunch
of messages.
well, we all have limited time. if you ever change your mind, i'd be
glad for your help. in practice, profiling 21.5 is more useful than
21.4 since very little more work will be done on 21.4.
as for your reluctance to work on 21.5, it would help if you could
articulate your objections as to the direction of current work. the
main thing i'm working on now is making the internal encoding be
unicode, which seems very important. my proposal for a classifier-based
encoding detection scheme was only a floated proposal, something that
will get done, if at all, as part of a final project for a class of mine
-- which otherwise would be something not related to xemacs at all.
ben
2:56 p.m.
New subject: Speed difference in Gnus: a step in the right direction
>>>> "Ben" == Ben Wing <ben(a)xemacs.org>
writes:
Ben> Hrvoje Niksic wrote:
> Ben Wing <ben(a)xemacs.org> writes:
>> Hrvoje, if you're willing to try some profiling in a current 21.5,
> I don't have time for it now, and I'm reluctant to work on the 21.5
> code base anyway, because of the reasons outlined earlier. I
> started this thread to make other people aware of the problem.
>
> Creating a large nnml group shouldn't be hard. For example, enter
> Gnus, press "n" when asked to go offline, then press "y" when
asked
> whether to continue. You can create an empty nnml group using `G m
> testgroup RET nnml RET'. Then enter an existing mbox using `G f
> <file-name> RET' (don't mind if this is slow, it's supposed to be
> slow). Press `C-x h' to select the whole buffer, followed by `B c
> nnml:tesgroup RET'. You should have a working nnml group with a
> bunch of messages.
Ben> well, we all have limited time. if you ever change
your mind, i'd
Ben> be glad for your help. in practice, profiling 21.5 is more useful
Ben> than
Ben> 21.4 since very little more work will be done on 21.4.
Is this what you were after? This is from a profile-key-sequence in
21.5b20 on Linux, no error checking, entering a large (2200 articles)
nnml group.
Function Name Ticks/Total %Usage Calls GC-Usage/ Total
===========================================/===== ====== ===== ========/=======
(in garbage collection) 4458/ 4458 51.247 156
(profile overhead) 1036/ 1036 11.909
(in map-extents-internal) 750/ 755 8.622 218120 1152928/1152928
looking-at 332/ 337 3.817 88212 3140404/3140404
bq-process-2 290/ 1978 3.334 887495 15788648/21176540
featurep 190/ 190 2.184 100986
re-search-forward 146/ 149 1.678 89140 1182416/1182416
kill-all-local-variables 145/ 145 1.667 5198
gnus-xmas-format 84/ 625 0.966 5200 187488/6891068
string-match 75/ 75 0.862 35722 594960/ 594960
kill-buffer 66/ 131 0.759 10726 257448/ 772296
gnus-user-format-function-b 61/ 5068 0.701 2598 328568/53798736
byte-code 47/ 957 0.540 21691 1676504/9915304
gnus-summary-prepare-threads 44/ 7513 0.506 1 441084/67178696
decode-coding-region 42/ 43 0.483 5217 626040/ 626040
gnus-simplify-subject-fuzzy 38/ 281 0.437 4892 96/ 915424
(in char-byte conversion) 38/ 38 0.437 2060174
get-buffer-create 38/ 41 0.437 21122 11500600/11500600
bbdb-records 36/ 684 0.414 7413 480/8779088
bq-process-1 36/ 49 0.414 317845 4991652/4991652
replace-match 34/ 35 0.391 21075 3152/ 3152
verify-visited-file-modtime 29/ 31 0.333 4817 238832/ 238832
mm-subst-char-in-string 26/ 25 0.299 10551 1080/ 1248
gnus-set-work-buffer 25/ 32 0.287 4893 0/ 28
#<compiled-function nil "...(157)" [gnus-tmp-replied gnus-tmp-unread
gnus-tmp-header gnus-tmp-lines need val gnus-user-format-function-b char-to-string
gnus-put-text-property gnus-xmas-format %s%s %s%s 4 gnus-correct-length 0 make-string ?\
nil eval gnus-summary-from-or-to-or-newsgroups 20 gnus-correct-substring %s: %s
int-to-string gnus-summary-number-of-articles-in-thread boundp thread 2
gnus-tmp-score-char gnus-tmp-indentation gnus-tmp-opening-bracket gnus-tmp-from
gnus-tmp-closing-bracket gnus-mouse-face-prop gnus-mouse-face gnus-tmp-level
gnus-tmp-subject-or-nil] 12>
24/ 7068 0.276 2598 227004/65137268
cond 23/ 693 0.264 38666 0/6209820
let 20/ 4313 0.230 25140 0/44675488
save-current-buffer 19/ 4274 0.218 10388 0/44145164
search-forward 18/ 30 0.207 28533
buffer-local-variables 18/ 32 0.207 7415 8544912/8544912
put-text-property 17/ 88 0.195 16803 1149592/1149592
forward-word 16/ 16 0.184 11488 1178892/1178892
setq 16/ 63 0.184 170408 0/ 450608
char-syntax 16/ 16 0.184 310 1528956/1528956
match-string 15/ 16 0.172 52795 589708/ 589708
#<compiled-function (&rest args) "...(28)" [result args reverse cons] 4
663101>
15/ 92 0.172 28659 3553764/4700124
insert-file-contents-internal 14/ 14 0.161 1 128/ 128
sort 14/ 31 0.161 636
char-width 13/ 246 0.149 95419
set-buffer 13/ 13 0.149 10388
subst-char-in-region 11/ 178 0.126 2 48/ 48
when 11/ 2196 0.126 67203 432/27534744
gnus-summary-number-of-articles-in-thread
11/ 11 0.126 3835 0/ 14844
erase-buffer 11/ 11 0.126 10103 0/ 84
#<compiled-function (char) "...(8)" [char length char-width] 3>
10/ 257 0.115 89938
generate-new-buffer-name 10/ 10 0.115 10724 17192/ 17192
while 10/ 4018 0.115 37675 0/43926860
if 9/ 2004 0.103 138500 0/22217668
bq-list* 9/ 483 0.103 28659 216/7168788
charset-property 8/ 8 0.092 95419
skip-chars-forward 8/ 8 0.092 44460
mapcar 7/ 279 0.080 23315 1200396/1249544
bq-comma 6/ 8 0.069 47765 573180/ 573180
#<compiled-function (buffer &rest body) "...(8)" [body buffer
save-current-buffer set-buffer] 3 696316>
6/ 11 0.069 10388 1433568/1433568
read 6/ 21 0.069 9452
vectorp 6/ 5 0.069 892167
insert-buffer-substring 6/ 7 0.069 3768 190284/ 190284
buffer-string 6/ 34 0.069 20434 0/ 963120
backquote 6/ 2539 0.069 12149 0/29294804
save-excursion 6/ 858 0.069 4372 0/5875944
append 5/ 5 0.057 11650 74592/ 74592
or 5/ 154 0.057 91182 0/ 938124
insert 5/ 5 0.057 5519 0/ 168
mail-decode-encoded-word-string 5/ 462 0.057 5206 11544/7426796
gnus-extract-address-components 5/ 43 0.057 2598 425820/ 879336
mm-decode-coding-string 5/ 359 0.057 5217 216/6885340
gnus-summary-highlight-line 5/ 61 0.057 2598 0/ 197564
reverse 5/ 6 0.057 29590 1182936/1182936
gnus-score-string< 5/ 5 0.057 40193
member* 4/ 7 0.046 10726 514848/ 514848
scan-sexps 4/ 4 0.046 3496 350064/ 350064
gnus-summary-limit-children 4/ 4 0.046 2602 31152/ 35016
buffer-substring 4/ 7 0.046 20434 963120/ 963120
condition-case 4/ 40 0.046 3497 0/ 350100
bbdb-buffer 3/ 3 0.034 7413
gnus-score-followup 3/ 42 0.034 1 31272/ 159072
prin1-to-string 3/ 3 0.034 1 59390/ 59390
list 3/ 6 0.034 69467 2469660/2469660
mm-decode-coding-region 3/ 4 0.034 340 40800/ 40800
#<subr when> 3/ 18 0.034 67203 6467640/6467640
progn 3/ 1886 0.034 49249 0/21574624
charset-width 3/ 20 0.034 95419
narrow-to-region 3/ 5 0.034 12224 96/ 96
select-buffer-killed-text 2/ 8 0.023 10726
intern 2/ 2 0.023 11039 120456/ 120456
and 2/ 189 0.023 172066 0/1344516
unless 2/ 25 0.023 5192 0/ 458364
eval 2/ 7073 0.023 10406 0/65145236
gnus-correct-substring 2/ 15 0.023 261 23920/ 23920
regexp-quote 2/ 2 0.023 3198 126352/ 126352
point-at-eol 2/ 3 0.023 5207
mail-extract-address-components 2/ 4315 0.023 2598 0/44675156
get-text-property 2/ 16 0.023 16267
gnus-score-string 2/ 17 0.023 1 31224/ 158524
assq 2/ 2 0.023 21566
eq 2/ 0 0.023 216156
gnus-put-text-property-excluding-characters-with-faces
2/ 38 0.023 2599 116/ 197640
next-single-property-change 2/ 9 0.023 3226
gnuserv-buffer-clients 2/ 2 0.023 10726
gnus-score-headers 2/ 61 0.023 1 124908/ 461004
char-charset 2/ 2 0.023 95419
set-text-properties 2/ 3 0.023 2598
get-buffer 2/ 2 0.023 4925
(in redisplay) 2/ 2 0.023 7 7188/ 8124
gnus-thread-sort-by-number 2/ 2 0.023 17889
downcase-region 2/ 2 0.023 2596
bq-process 2/ 2011 0.023 12149 24/21353504
gnus-correct-length 2/ 278 0.023 10395 0/1079256
with-current-buffer 2/ 4309 0.023 10388 24/44675156
fundamental-mode 2/ 147 0.023 5196 72/ 72
(in internal-external conversion) 2/ 2 0.023 4851
funcall 2/ 7071 0.023 2603 0/65143140
delete-other-windows 2/ 2 0.023 1 6912/ 6912
goto-char 2/ 3 0.023 52737
select-buffer-killed-default 2/ 5 0.023 10726
natnump 2/ 3 0.023 40868
syntax-string-to-code 1/ 1 0.011 594 0/ 42768
mail-extr-safe-move-sexp 1/ 43 0.011 3496 0/ 350064
eobp 1/ 1 0.011 35836
gnus-inverse-list-range-intersection 1/ 1 0.011 1
copy-sequence 1/ 1 0.011 1735 66576/ 66576
put-char-table 1/ 1 0.011 594
gnus-list-of-unread-articles 1/ 1 0.011 2 53400/ 53400
gnus-thread-loop-p 1/ 1 0.011 322 9036/ 9036
delq 1/ 1 0.011 9573
quote 1/ 1 0.011 208182
#<compiled-function nil "...(166)" [gnus-dormant-mark uncached default-low
default-high score mark gnus-summary-cancelled gnus-summary-high-undownloaded
gnus-summary-low-undownloaded gnus-summary-normal-undownloaded gnus-summary-high-ticked
gnus-summary-low-ticked gnus-summary-normal-ticked gnus-summary-high-ancient
gnus-summary-low-ancient gnus-summary-normal-ancient gnus-summary-high-unread
gnus-summary-low-unread gnus-summary-normal-unread gnus-summary-high-read
gnus-summary-low-read gnus-summary-normal-read gnus-ticked-mark gnus-ancient-mark
gnus-unread-mark gnus-canceled-mark] 2>
1/ 1 0.011 2598
gnus-summary-from-or-to-or-newsgroups 1/ 48 0.011 2598 76/ 965208
gnus-sorted-nunion 1/ 1 0.011 1 26712/ 26712
not 1/ 1 0.011 57611
browse-url-delete-temp-file 1/ 1 0.011 10726
bbdb-record-name 1/ 1 0.011 1570
check-menu-syntax 1/ 4 0.011 634 0/ 256
mail-extr-skip-whitespace-forward 1/ 13 0.011 38978
make-string 1/ 2 0.011 7171 179848/ 179848
gnus-remove-from-range 1/ 9 0.011 2 0/ 83016
nnheader-nov-delete-outside-range 1/ 4 0.011 1
cons 1/ 2 0.011 76702 920424/ 920424
bbdb-record-getprop 1/ 1 0.011 1784
make-vector 1/ 1 0.011 3 21552/ 21552
gnus-update-read-articles 1/ 10 0.011 1 41856/ 125048
char-to-string 1/ 2 0.011 13586 326064/ 326064
run-hooks 1/ 67 0.011 2609 0/ 231500
rfc2047-decode-encoded-words 1/ 65 0.011 171 12964/ 504656
delete-region 1/ 1 0.011 632
coding-system-property 1/ 1 0.011 576 59696/ 59696
quoted-printable-decode-region 1/ 1 0.011 137 4560/ 4600
point-max 1/ 1 0.011 18830
= 1/ 1 0.011 27504
1/ 1 0.011 22181
fume-narrow-to-region 1/ 1 0.011 12224
widen 1/ 2 0.011 7796 48/ 48
gnus-run-hooks 1/ 64 0.011 2607 31296/ 256676
[... rest snipped ...]
Let me know if you want the full profile, or one from a recent CVS
build.
--J.
11:43 a.m.
New subject: Speed difference in Gnus: garbage generation/collection is the real culprit, at least here
Jan Rychter wrote:
>>>>>"Ben" == Ben Wing <ben(a)xemacs.org>
writes:
>>>>>
>>>>>
Ben> Hrvoje Niksic wrote:
>> Ben Wing <ben(a)xemacs.org> writes:
>>> Hrvoje, if you're willing to try some profiling in a current 21.5,
>> I don't have time for it now, and I'm reluctant to work on the 21.5
>> code base anyway, because of the reasons outlined earlier. I
>> started this thread to make other people aware of the problem.
>>
>> Creating a large nnml group shouldn't be hard. For example, enter
>> Gnus, press "n" when asked to go offline, then press "y"
when asked
>> whether to continue. You can create an empty nnml group using `G m
>> testgroup RET nnml RET'. Then enter an existing mbox using `G f
>> <file-name> RET' (don't mind if this is slow, it's supposed to
be
>> slow). Press `C-x h' to select the whole buffer, followed by `B c
>> nnml:tesgroup RET'. You should have a working nnml group with a
>> bunch of messages.
Ben> well, we all have limited time. if you ever change your mind, i'd
Ben> be glad for your help. in practice, profiling 21.5 is more useful
Ben> than
Ben> 21.4 since very little more work will be done on 21.4.
Is this what you were after? This is from a profile-key-sequence in
21.5b20 on Linux, no error checking, entering a large (2200 articles)
nnml group.
Function Name Ticks/Total %Usage Calls GC-Usage/ Total
===========================================/===== ====== ===== ========/=======
(in garbage collection) 4458/ 4458 51.247 156
(profile overhead) 1036/ 1036 11.909
(in map-extents-internal) 750/ 755 8.622 218120 1152928/1152928
looking-at 332/ 337 3.817 88212 3140404/3140404
bq-process-2 290/ 1978 3.334 887495 15788648/21176540
featurep 190/ 190 2.184 100986
re-search-forward 146/ 149 1.678 89140 1182416/1182416
kill-all-local-variables 145/ 145 1.667 5198
gnus-xmas-format 84/ 625 0.966 5200 187488/6891068
string-match 75/ 75 0.862 35722 594960/ 594960
kill-buffer 66/ 131 0.759 10726 257448/ 772296
gnus-user-format-function-b 61/ 5068 0.701 2598 328568/53798736
byte-code 47/ 957 0.540 21691 1676504/9915304
gnus-summary-prepare-threads 44/ 7513 0.506 1 441084/67178696
decode-coding-region 42/ 43 0.483 5217 626040/ 626040
gnus-simplify-subject-fuzzy 38/ 281 0.437 4892 96/ 915424
(in char-byte conversion) 38/ 38 0.437 2060174
get-buffer-create 38/ 41 0.437 21122 11500600/11500600
bbdb-records 36/ 684 0.414 7413 480/8779088
bq-process-1 36/ 49 0.414 317845 4991652/4991652
replace-match 34/ 35 0.391 21075 3152/ 3152
verify-visited-file-modtime 29/ 31 0.333 4817 238832/ 238832
mm-subst-char-in-string 26/ 25 0.299 10551 1080/ 1248
gnus-set-work-buffer 25/ 32 0.287 4893 0/ 28
#<compiled-function nil "...(157)" [gnus-tmp-replied gnus-tmp-unread
gnus-tmp-header gnus-tmp-lines need val gnus-user-format-function-b char-to-string
gnus-put-text-property gnus-xmas-format %s%s %s%s 4 gnus-correct-length 0 make-string ?\
nil eval gnus-summary-from-or-to-or-newsgroups 20 gnus-correct-substring %s: %s
int-to-string gnus-summary-number-of-articles-in-thread boundp thread 2
gnus-tmp-score-char gnus-tmp-indentation gnus-tmp-opening-bracket gnus-tmp-from
gnus-tmp-closing-bracket gnus-mouse-face-prop gnus-mouse-face gnus-tmp-level
gnus-tmp-subject-or-nil] 12>
24/ 7068 0.276 2598 227004/65137268
cond 23/ 693 0.264 38666 0/6209820
let 20/ 4313 0.230 25140 0/44675488
save-current-buffer 19/ 4274 0.218 10388 0/44145164
search-forward 18/ 30 0.207 28533
buffer-local-variables 18/ 32 0.207 7415 8544912/8544912
put-text-property 17/ 88 0.195 16803 1149592/1149592
forward-word 16/ 16 0.184 11488 1178892/1178892
setq 16/ 63 0.184 170408 0/ 450608
char-syntax 16/ 16 0.184 310 1528956/1528956
match-string 15/ 16 0.172 52795 589708/ 589708
#<compiled-function (&rest args) "...(28)" [result args reverse cons] 4
663101>
15/ 92 0.172 28659 3553764/4700124
insert-file-contents-internal 14/ 14 0.161 1 128/ 128
sort 14/ 31 0.161 636
char-width 13/ 246 0.149 95419
set-buffer 13/ 13 0.149 10388
subst-char-in-region 11/ 178 0.126 2 48/ 48
when 11/ 2196 0.126 67203 432/27534744
gnus-summary-number-of-articles-in-thread
11/ 11 0.126 3835 0/ 14844
erase-buffer 11/ 11 0.126 10103 0/ 84
#<compiled-function (char) "...(8)" [char length char-width] 3>
10/ 257 0.115 89938
generate-new-buffer-name 10/ 10 0.115 10724 17192/ 17192
while 10/ 4018 0.115 37675 0/43926860
if 9/ 2004 0.103 138500 0/22217668
bq-list* 9/ 483 0.103 28659 216/7168788
charset-property 8/ 8 0.092 95419
skip-chars-forward 8/ 8 0.092 44460
mapcar 7/ 279 0.080 23315 1200396/1249544
bq-comma 6/ 8 0.069 47765 573180/ 573180
#<compiled-function (buffer &rest body) "...(8)" [body buffer
save-current-buffer set-buffer] 3 696316>
6/ 11 0.069 10388 1433568/1433568
read 6/ 21 0.069 9452
vectorp 6/ 5 0.069 892167
insert-buffer-substring 6/ 7 0.069 3768 190284/ 190284
buffer-string 6/ 34 0.069 20434 0/ 963120
backquote 6/ 2539 0.069 12149 0/29294804
save-excursion 6/ 858 0.069 4372 0/5875944
append 5/ 5 0.057 11650 74592/ 74592
or 5/ 154 0.057 91182 0/ 938124
insert 5/ 5 0.057 5519 0/ 168
mail-decode-encoded-word-string 5/ 462 0.057 5206 11544/7426796
gnus-extract-address-components 5/ 43 0.057 2598 425820/ 879336
mm-decode-coding-string 5/ 359 0.057 5217 216/6885340
gnus-summary-highlight-line 5/ 61 0.057 2598 0/ 197564
reverse 5/ 6 0.057 29590 1182936/1182936
gnus-score-string< 5/ 5 0.057 40193
member* 4/ 7 0.046 10726 514848/ 514848
scan-sexps 4/ 4 0.046 3496 350064/ 350064
gnus-summary-limit-children 4/ 4 0.046 2602 31152/ 35016
buffer-substring 4/ 7 0.046 20434 963120/ 963120
condition-case 4/ 40 0.046 3497 0/ 350100
bbdb-buffer 3/ 3 0.034 7413
gnus-score-followup 3/ 42 0.034 1 31272/ 159072
prin1-to-string 3/ 3 0.034 1 59390/ 59390
list 3/ 6 0.034 69467 2469660/2469660
mm-decode-coding-region 3/ 4 0.034 340 40800/ 40800
#<subr when> 3/ 18 0.034 67203 6467640/6467640
progn 3/ 1886 0.034 49249 0/21574624
charset-width 3/ 20 0.034 95419
narrow-to-region 3/ 5 0.034 12224 96/ 96
select-buffer-killed-text 2/ 8 0.023 10726
intern 2/ 2 0.023 11039 120456/ 120456
and 2/ 189 0.023 172066 0/1344516
unless 2/ 25 0.023 5192 0/ 458364
eval 2/ 7073 0.023 10406 0/65145236
gnus-correct-substring 2/ 15 0.023 261 23920/ 23920
regexp-quote 2/ 2 0.023 3198 126352/ 126352
point-at-eol 2/ 3 0.023 5207
mail-extract-address-components 2/ 4315 0.023 2598 0/44675156
get-text-property 2/ 16 0.023 16267
gnus-score-string 2/ 17 0.023 1 31224/ 158524
assq 2/ 2 0.023 21566
eq 2/ 0 0.023 216156
gnus-put-text-property-excluding-characters-with-faces
2/ 38 0.023 2599 116/ 197640
next-single-property-change 2/ 9 0.023 3226
gnuserv-buffer-clients 2/ 2 0.023 10726
gnus-score-headers 2/ 61 0.023 1 124908/ 461004
char-charset 2/ 2 0.023 95419
set-text-properties 2/ 3 0.023 2598
get-buffer 2/ 2 0.023 4925
(in redisplay) 2/ 2 0.023 7 7188/ 8124
gnus-thread-sort-by-number 2/ 2 0.023 17889
downcase-region 2/ 2 0.023 2596
bq-process 2/ 2011 0.023 12149 24/21353504
gnus-correct-length 2/ 278 0.023 10395 0/1079256
with-current-buffer 2/ 4309 0.023 10388 24/44675156
fundamental-mode 2/ 147 0.023 5196 72/ 72
(in internal-external conversion) 2/ 2 0.023 4851
funcall 2/ 7071 0.023 2603 0/65143140
delete-other-windows 2/ 2 0.023 1 6912/ 6912
goto-char 2/ 3 0.023 52737
select-buffer-killed-default 2/ 5 0.023 10726
natnump 2/ 3 0.023 40868
syntax-string-to-code 1/ 1 0.011 594 0/ 42768
mail-extr-safe-move-sexp 1/ 43 0.011 3496 0/ 350064
eobp 1/ 1 0.011 35836
gnus-inverse-list-range-intersection 1/ 1 0.011 1
copy-sequence 1/ 1 0.011 1735 66576/ 66576
put-char-table 1/ 1 0.011 594
gnus-list-of-unread-articles 1/ 1 0.011 2 53400/ 53400
gnus-thread-loop-p 1/ 1 0.011 322 9036/ 9036
delq 1/ 1 0.011 9573
quote 1/ 1 0.011 208182
#<compiled-function nil "...(166)" [gnus-dormant-mark uncached default-low
default-high score mark gnus-summary-cancelled gnus-summary-high-undownloaded
gnus-summary-low-undownloaded gnus-summary-normal-undownloaded gnus-summary-high-ticked
gnus-summary-low-ticked gnus-summary-normal-ticked gnus-summary-high-ancient
gnus-summary-low-ancient gnus-summary-normal-ancient gnus-summary-high-unread
gnus-summary-low-unread gnus-summary-normal-unread gnus-summary-high-read
gnus-summary-low-read gnus-summary-normal-read gnus-ticked-mark gnus-ancient-mark
gnus-unread-mark gnus-canceled-mark] 2>
1/ 1 0.011 2598
gnus-summary-from-or-to-or-newsgroups 1/ 48 0.011 2598 76/ 965208
gnus-sorted-nunion 1/ 1 0.011 1 26712/ 26712
not 1/ 1 0.011 57611
browse-url-delete-temp-file 1/ 1 0.011 10726
bbdb-record-name 1/ 1 0.011 1570
check-menu-syntax 1/ 4 0.011 634 0/ 256
mail-extr-skip-whitespace-forward 1/ 13 0.011 38978
make-string 1/ 2 0.011 7171 179848/ 179848
gnus-remove-from-range 1/ 9 0.011 2 0/ 83016
nnheader-nov-delete-outside-range 1/ 4 0.011 1
cons 1/ 2 0.011 76702 920424/ 920424
bbdb-record-getprop 1/ 1 0.011 1784
make-vector 1/ 1 0.011 3 21552/ 21552
gnus-update-read-articles 1/ 10 0.011 1 41856/ 125048
char-to-string 1/ 2 0.011 13586 326064/ 326064
run-hooks 1/ 67 0.011 2609 0/ 231500
rfc2047-decode-encoded-words 1/ 65 0.011 171 12964/ 504656
delete-region 1/ 1 0.011 632
coding-system-property 1/ 1 0.011 576 59696/ 59696
quoted-printable-decode-region 1/ 1 0.011 137 4560/ 4600
point-max 1/ 1 0.011 18830
>= 1/ 1 0.011 27504
> 1/ 1 0.011 22181
>
>
fume-narrow-to-region 1/ 1 0.011 12224
widen 1/ 2 0.011 7796 48/ 48
gnus-run-hooks 1/ 64 0.011 2607 31296/ 256676
[... rest snipped ...]
Let me know if you want the full profile, or one from a recent CVS
build.
yuck. there is a *fantastic* amount of garbage being generated. 65 MB
for 2200 messages! (how big is the file itself?)
things to try:
[1] silly question -- did you build with optimization (configure
--with-optimization)? you should.
[2] make sure you do all your tests with `xemacs --vanilla' (see
comments below).
[1] use latest cvs if it's not too hard.
[2] first do a build with and without `with-kkcc=no'/`with-mc-alloc=no'
to see if the new garbage collector and allocator are making things
worse, and if so by how much.
[3] then, increase gc-cons-threshold to at least 2000000, if not higher,
and redo the profiling. (it was recently raised to 2000000)
anyway, clearly mule is *not* at fault here; look at the small amount of
time going to `(in char-byte conversion)' and `(in internal-external
conversion)'.
-------------------
[btw: following is a version of jan's profile, resorted to show maximum
gc uage in each function. (the left figure shows allocation in this
profile unit only, while the right shows allocation in this and all
children profile units. a "profile unit" is more or less a lisp
function, but can be smaller in some cases and includes many of the
native subsystems that potentially take lots of time.)]
but just look at the amount of cumulative time spent in backquote
processing (Chief Offender), and the 29 megs of crap generated! this
should *really* happen at compile time, so something is majorly wrong
here. Jan, are parts of GNUS somehow uncompiled or out-of-date (in
which case the interpreted versions will be used instead)? Do you have
some personal code you stick into certain hooks that is uncompiled and
uses backquotes?
If not either of these, I can only assume that GNUS itself is doing
something horrendous with backquotes that results in run-time expansion;
and it probably isn't doing the same under GNU Emacs.
Offender #2 is get-buffer-create, called 21122 times for 2200 articles,
and mostly (it seems) actually creating a buffer.
this seems an awful lot of buffer creation. if this still appears after
Jan runs --vanilla, could someone help track down what's happening?
Same for buffer-local-variables'. #<subr when> is once again a case of
a macro not getting compiled correctly. bq-process-1 is part of the
backquote mechanism, like bq-process-2; after that, i'm not really
sure. i need to put back the compiled-fun annotation "hack" that steve
baur (?) didn't like but which is great for these cases.
after this, `looking-at' should probably not be generating lisp objects.
`list' does it as part of its job, and so does the byte-compiler in
certain cases.
map_extents() is also highly suspicious, as it shouldn't really be
allocating.
Function Name Ticks/Total %Usage Calls GC-Usage/ Total
===========================================/===== ====== ===== ========/=======
bq-process-2 290/ 1978 3.334 887495 15788648/21176540
get-buffer-create 38/ 41 0.437 21122 11500600/11500600
buffer-local-variables 18/ 32 0.207 7415 8544912/8544912
#<subr when> 3/ 18 0.034 67203 6467640/6467640
bq-process-1 36/ 49 0.414 317845 4991652/4991652
#<compiled-function (&rest args) "...(28)" [result args reverse cons] 4
663101> 15/ 92 0.172 28659
3553764/4700124
looking-at 332/ 337 3.817 88212 3140404/3140404
list 3/ 6 0.034 69467 2469660/2469660
byte-code 47/ 957 0.540 21691 1676504/9915304
char-syntax 16/ 16 0.184 310 1528956/1528956
#<compiled-function (buffer &rest body) "...(8)" [body buffer
save-current-buffer set-buffer] 3 696316> 6/
11 0.069 10388 1433568/1433568
mapcar 7/ 279 0.080 23315 1200396/1249544
reverse 5/ 6 0.057 29590 1182936/1182936
re-search-forward 146/ 149 1.678 89140 1182416/1182416
forward-word 16/ 16 0.184 11488 1178892/1178892
(in map-extents-internal) 750/ 755 8.622 218120 1152928/1152928
put-text-property 17/ 88 0.195 16803 1149592/1149592
buffer-substring 4/ 7 0.046 20434 963120/ 963120
cons 1/ 2 0.011 76702 920424/ 920424
decode-coding-region 42/ 43 0.483 5217 626040/ 626040
string-match 75/ 75 0.862 35722 594960/ 594960
match-string 15/ 16 0.172 52795 589708/ 589708
bq-comma 6/ 8 0.069 47765 573180/ 573180
member* 4/ 7 0.046 10726 514848/ 514848
gnus-summary-prepare-threads 44/ 7513 0.506 1 441084/67178696
gnus-extract-address-components 5/ 43 0.057 2598 425820/ 879336
scan-sexps 4/ 4 0.046 3496 350064/ 350064
gnus-user-format-function-b 61/ 5068 0.701 2598 328568/53798736
char-to-string 1/ 2 0.011 13586 326064/ 326064
kill-buffer 66/ 131 0.759 10726 257448/ 772296
verify-visited-file-modtime 29/ 31 0.333 4817 238832/ 238832
12:19 p.m.
New subject: Speed difference in Gnus: a step in the right direction
Hrvoje Niksic wrote:
What disk I/O? nnml reads the headers from a single .overview file.
Unless told *not* to use NOV databases, by gnus-nov-is-evil. Perhaps
Stephen should check that.
--
Didier Verna, didier(a)lrde.epita.fr, http://www.lrde.epita.fr/~didier
EPITA / LRDE, 14-16 rue Voltaire Tel.+33 (1) 44 08 01 85
94276 Le Kremlin-Bicêtre, France Fax.+33 (1) 53 14 59 22 didier(a)xemacs.org
12:36 p.m.
New subject: Speed difference in Gnus: a step in the right direction
On Thu, Oct 20 2005, Stephen J. Turnbull wrote:
GNUS HACKERS TAKE NOTE: The function
`gnus-xmas-extent-start-open'
maps over the extents near some arbitrary buffer position; I'm not
sure from Ben's descriptions whether this is an operation that is
affected by large extents. It's only used in the *Group* buffer IIRC,
so even if so, probably the performance hit is negligible for most
people.
If you want to reach the Gnus developers, it would make sense to Cc
ding(a)gnus.org. [ But since Hrvoje instructed me that this topic is
not interesting for Gnus developers, I'll leave this up to you. ]
Bye, Reiner.
--
,,,
(o o)
---ooO-(_)-Ooo--- | PGP key available | http://rsteib.home.pages.de/
9:29 a.m.
Stephen J. Turnbull wrote:
>>>>>"Ben" == Ben Wing <ben(a)xemacs.org>
writes:
>>>>>
>>>>>
Ben> Stephen J. Turnbull wrote:
>> It's really ugly to have a buffer property that's actually a
>> text property just because performance sucks when an extent
>> covers the whole buffer.
I assume the above is what you're talking about?
yes.
Ben> what do you mean? as i said before, i have mostly
finished
Ben> code that will make text properties efficient by implementing
Ben> them entirely in non-overlapping intervals, as fsf does.
What I mean is that there are a number of things that we do as buffer
attributes or buffer-local variables (language environment, coded
character set, major mode for starters) that conceptually are text
properties. Think about it: wouldn't it be cool if instead of needing
a hack like mmm, we simply attached LISP-major-mode to an extent
covering the buffer, string-major-mode to strings, and
comment-major-mode to comments? Wouldn't it be cool if we could
attach xml-mode to individual strings in xml.el or use html-mode for
strings by default in blog-mode.el?
Ben> in general this is a very tough problem, and using simpler
Ben> solutions when possible (e.g. non-overlapping text
Ben> properties).
Whatever it is it has to be part of the implementation, not a
constraint on user code. (I'm pretty sure you agree, but it's not
clear to me from that sentence.)
yes, i do. once i get the new text property implementation down, then
any program that only uses text properties and not directly uses extents
or overlays, or which only uses extents/overlays covering small sections
of the buffer, will be fast. however, in the long run that's not
enough; we'd like for extents to be fast, period, and not make the user
go through the text property interface. (the reason why this is harder
is that the text property interface is higher-level and doesn't
explicitly give you a concept of an entity tied to a particular range.)
if people normally used text properties, and tended to only use a small
number of extents, then we might consider as a stop-gap measure
separating the two internally into a sort of parallel structure -- this
is much like what FSF does, with its overlays, i think. however, it
might turn out to be not much easier to implement that than to just move
entirely to the "correct" implementation of extents in terms of
non-overlapping intervals.
Ben> it might be possible to simulate extents in general using
Ben> non-overlapping intervals, where multiple intervals could
Ben> represent a single extent and each interval contains parts of
Ben> all extents crossing over them. this interval would be
Ben> guaranteed efficient and not too hard to implement, i think
Ben> -- in fact it's the same algorithm i'd be using for
Ben> text-properties, and could be extended to extents in general.
Something like this? A linked list P of (buffer position, extent
list) pairs, with an intervals defined as [P[n].pos,P[n+1].pos) for
each n. An extent is a pair of pointers into P defining which
fragments are covered by the extent plus a property list. P[n].ext is
the list of extents covering that interval. Then for random access,
finding n is O(lg n), and finding the property is O(|P[n].ext|), which
is the best you can get, I guess. (For many operations, in particular
redisplay, which walk P in order, finding n is O(1), right?) P itself
only needs to be updated when text is inserted/deleted in the buffer,
and when an extent is created/deleted. |P| <= |# boundary points of
all extents|, so that is no worse than what we currently have, and we
never have to update the extents themselves.
this sounds about right, although i'm a little confused about what
"finding n" means, and finding anything in a linked list is O(n), not
O(lg n). we maintain an ordered gap array (or whatever) of
non-overlapping intervals as defined by the extents, where each interval
is the maximum stretch not crossed by an extent endpoint. Each extent
contains a pointer to the first and last interval covered by the
extent. most likely, these will be indices into the gap array, and
hence need to be adjusted as we do gap movement in the array; this is
just like what happens currently with the endpoints of an extent in the
buffer. we can avoid this update if the gap array, rather than
containing the interval structures themselves, contains pointers to
heap-allocated interval objects, and each such object contains a forward
and probably backward pointers to the next/previous object, so that
essentially the intervals are in a combination linked-list and array.
the array, btw, is necessary so that we can locate the interval
surrounding a particular position in O(lg n) time. it's not possible to
do this in a pure linked list; you'd get O(n) instead. another data
structure would involve having the intervals be objects with pointers
back to the array position they came from. doing gap movement in the
array requires updating the intervals rather than the extents; there
could potentially be significant savings if there are a whole lot of
overlapping extents but in practice this seems unlikely. in reality i'm
somewhat loath to do any of these alternative schemes because it would
add a lot of space overhead, would fragment the memory, etc. in
practice, i think that extents are modified much less often than they
are accessed by redisplay, and you only get O(n) behavior when inserting
or deleting extents when the gap is far away from where you previously
were. but i'm certainly willing to be proven wrong by appropriate
profiling data.
the crucial thing, then, is that each interval contains a some sort of
list of all the extents overlapping that interval. this list will
probably be small; so either a linked list or a Stynarr could be used.
(A Stynarr is a new struct that i invented in my unicode-internal ws.
it is based on existing face-cachel code. the idea is that it uses a
fixed array of small size (e.g. 4 or 6) that's built into the object and
pointer to a dynarr, which is used to handle overflow. it's useful when
you need to maintain a list of objects and handle an arbitrary # of
objects on the list, but where in practice it very rarely exceeds a
small # and would like to avoid lots and lots of heap allocation.)
we might still have to maintain the display-order and e-order for extent
endpoints, like we currently do; i'll have to look into the code, but i
bet there will still be places that need to look stuff up by endpoints,
and it may not be too easy to derive this from the interval data.
what do you mean "we never have to update the extents themselves"?
Ben> but first we need some more profiling data.
Why? Basically in your explanation of how the soe works, you are
saying that large extents (not just extents covering the whole buffer,
but any extent that overlaps many other extents) are very expensive
because they make _all_ the overlapped extents inefficient.
why, because i have limited time and i need to schedule things according
to how badly they slow things down. i see that my attempts to speed up
byte<->char conversion actually slowed things down significantly for
Raymond, although they indeed sped things up in my profiling for
whatever i was running into problems with. i think it's because i
deleted the "known range" cache that was the first line of attack. i
did this because fsf didn't have any such thing and seemed to be doing
fine; but they may have a lot more code internally that operates in byte
positions rather than char positions [or even in pairs of the two; that
seems common]. so i'm going to look into putting them back.
meanwhile, i'm mostly done with the rewrite of char tables for
unicode-internal. there isn't much coding left before i will start
compiling; but there's definitely a good deal of work to be done. for
one, the font handling is still by charset, meaning that unicode chars
for which there's no charset won't be displayable! yuck. that requires
some significant work. for another char-classes haven't been
implemented yet. and i don't know what the hell to do with ccl;
hopefully it really won't be necessary because i've expanded the notion
of charset to include any rectangular subset of 256x256 (so we can
properly have a windows-1252 charset, e.g.) and i'm planning on creating
a general mbcs coding system to replace the ad-hoc shift-jis and big5
coding systems; it could also be used for windows-125* coding systems
plus lots of simple coding systems (e.g. iso-8859-1) that are currently
handled using iso2022.
questions for you, stephen:
[1] how are unicode fonts handled under X windows?
[2] now that i look more at ccl, i see that it's utterly tied to the old mule
encoding, as it expects to look directly at the internal encoding of the text. it would
have to be radically restructured for it to work on a character-by-character level, not
byte-by-byte. do you think it's reasonable to just make ccl be an optional feature
that is compiled in only with old mule?
So that
means that code like
(let ((header (make-extent (message-beginning) (header-end)))
(body (make-extent (header-end) (message-end))))
(font-lock-fontify-buffer))
necessarily sucks. If I'm not totally missing the point, I have to
ask: Do you consider that restriction acceptable?
no, not in the long run.
11:19 a.m.
I'll be looking more carefully at this later, but a couple of quick
corrections and clarifications.
I see your point about priorities, it's especially clearer when you
put it in terms of redisplay access vs. extent modifications as the
typical cases.
>>>> "Ben" == Ben Wing <ben(a)xemacs.org>
writes:
Ben> yes, i do. once i get the new text property implementation down, then
Ben> any program that only uses text properties and not directly uses
Ben> extents or overlays, or which only uses extents/overlays covering
Ben> small sections of the buffer, will be fast.
That's useful to know; I wasn't sure.
> Something like this? A linked list P of (buffer position, extent
This was originally an array, which is where the "finding n is O(lg
n)" came from.
> list) pairs, with an intervals defined as [P[n].pos,P[n+1].pos)
for
> each n. An extent is a pair of pointers into P defining which
> fragments are covered by the extent plus a property list. P[n].ext is
> the list of extents covering that interval. Then for random access,
> finding n is O(lg n), and finding the property is O(|P[n].ext|), which
"Finding n" == "finding the n such that P[n].pos = some_position".
> is the best you can get, I guess. (For many operations, in
particular
> redisplay, which walk P in order, finding n is O(1), right?) P itself
> only needs to be updated when text is inserted/deleted in the buffer,
> and when an extent is created/deleted. |P| <= |# boundary points of
> all extents|, so that is no worse than what we currently have, and we
> never have to update the extents themselves.
Ben> what do you mean "we never have to update the extents
Ben> themselves"?
Since the extents are pointers to *links*, and
links will be inserted/deleted only when fragments are split/coalesced,
which occurs only when an extent is "created with an endpoint not in
the list"/deleted,
the pointers in the extents never need to be updated.
You make the same point. What I missed is:
Ben> the array, btw, is necessary so that we can locate the
Ben> interval surrounding a particular position in O(lg n) time. it's not
Ben> possible to do this in a pure linked list; you'd get O(n) instead.
Ben> [1] how are unicode fonts handled under X windows?
Not, as far as I can tell.
Ben> [2] now that i look more at ccl, i see that it's utterly tied
Ben> to the old mule encoding, as it expects to look directly at
Ben> the internal encoding of the text. it would have to be
Ben> radically restructured for it to work on a
Ben> character-by-character level, not byte-by-byte. do you think
Ben> it's reasonable to just make ccl be an optional feature that
Ben> is compiled in only with old mule?
Yes. I think that CCL has four interesting applications that I know
of: shift-jis, big5, mule-ucs, and fast table-based character sets
(eg, KOI8). The first two have been in C forever, and the latter two
are in C in the unicode support now (at least in principle).
There are also some minor hacks from Aidan and others.
It might be worth using the CCL interpreter as a general fast
arithmetic engine for small numbers, but rip out the Mule stuff.
--
School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
Ask not how you can "do" free software business;
ask what your business can "do for" free software.
12:26 p.m.
Stephen J. Turnbull wrote:
I'll be looking more carefully at this later, but a couple of
quick
corrections and clarifications.
I see your point about priorities, it's especially clearer when you
put it in terms of redisplay access vs. extent modifications as the
typical cases.
me likewise. i'll just add a few [rather long!] comments below.
>>>>>"Ben" == Ben Wing <ben(a)xemacs.org> writes:
>>>>>
>>>>>
Ben> yes, i do. once i get the new text property implementation down, then
Ben> any program that only uses text properties and not directly uses
Ben> extents or overlays, or which only uses extents/overlays covering
Ben> small sections of the buffer, will be fast.
That's useful to know; I wasn't sure.
>> Something like this? A linked list P of (buffer position, extent
This was originally an array, which is where the "finding n is O(lg
n)" came from.
>> list) pairs, with an intervals defined as [P[n].pos,P[n+1].pos) for
>> each n. An extent is a pair of pointers into P defining which
>> fragments are covered by the extent plus a property list. P[n].ext is
>> the list of extents covering that interval. Then for random access,
>> finding n is O(lg n), and finding the property is O(|P[n].ext|), which
"Finding n" == "finding the n such that P[n].pos = some_position".
>> is the best you can get, I guess. (For many operations, in particular
>> redisplay, which walk P in order, finding n is O(1), right?) P itself
>> only needs to be updated when text is inserted/deleted in the buffer,
>> and when an extent is created/deleted. |P| <= |# boundary points of
>> all extents|, so that is no worse than what we currently have, and we
>> never have to update the extents themselves.
Ben> what do you mean "we never have to update the extents
Ben> themselves"?
Since the extents are pointers to *links*, and
links will be inserted/deleted only when fragments are split/coalesced,
which occurs only when an extent is "created with an endpoint not in
the list"/deleted,
the pointers in the extents never need to be updated.
You make the same point. What I missed is:
Ben> the array, btw, is necessary so that we can locate the
Ben> interval surrounding a particular position in O(lg n) time. it's not
Ben> possible to do this in a pure linked list; you'd get O(n) instead.
Ben> [1] how are unicode fonts handled under X windows?
Not, as far as I can tell.
well, there are some fonts that have ...-devanagari-10646-... in them,
which are presumably unicode. maybe we create a special unicode-bcs[?]
charset, 256x256, for such fonts -- and make sure we check for the
presence of characters in a
particular font before using it. the question is, what do other modern
Unicode-based applications do under X windows for displaying Unicode
chars? (think Firefox, KDE's web browser, etc) What about xft?
i'm asking this because the code to handle char->font mapping needs to
be totally redone (e.g. it's currently phrased in terms of char*set* to
font!) and i need some info on how X does it before thinking about
doing it. in windows it's easy because all fonts are indexed by unicode
and have some bit fields corresponding to specfic Unicode subranges
indicating, more or less, which subranges they support. (OTOH i haven't
yet figured out exactly what this means -- supports at least one char in
the range? supports "more than a few"? "many"? "all"?)
almost certainly
i'll need a cache of characters->fonts.
since you have a general handle on these issues better than i do, maybe
you could present the general model you have in mind for font mapping?
assume something like this:
1. the goal is: redisplay gets a stretch of characters and the combined
faces for each (if you want, just assume you've been given a stretch
entirely covered by the same combined face) and has to figure out the
appropriate font for each one and the width of each character.
(currently, it optimizes by using the charset and assuming one font per
charset; that allows it to make one width check per stretch of
characters in the same charset. but this has to go, and i don't think
anything equivalent e.g. over unicode subranges would be sound, as some
fonts support only parts of certain subranges so we'd have to use
another font in some cases.)
2. you also have available the buffer that the text came from (from
which you can derive a `language' object; not yet implemented, but will
be; maybe better renamed to a `culture' object, as per .NET, since it
refers to something more than just pure language and might well want to
differentiate "simplified chinese" from "traditional chinese" as
different 'languages/cultures/whatever'; OTOH, 'culture' in .NET is
really something different, something like 'en-US' or `arb-EGY', which
is not what we're looking for). you also may have available a
'language' (or whatever) taken from an extent. in fact, you probably
just have a single character-specific `language' object and you don't
know or care where it comes from.
3. maybe you have a language->font mapping inside of a face. feel free
to choose the appropriate mapping and interface.
4. while you're at it, feel free to design the appropriate interface for
mapping languages (or whatever) to a precedence list of charsets, for
unicode conversion (including both the mapping itself and the lisp
commands used to control this mapping).
5. assume that our goal is to find the best font given the language and
associated preferences, but that in all cases we need to find *some*
font, if it's available. displaying a "box" or whatever is a last
resort, and ideally should happen only when there is *no* font on the
system (or at least, no font from all those available to look through,
i.e. specified as global defaults). assume also that we want this to be
reasonably fast, so we will probably need to cache char->font mappings.
since presumably it's much more expensive to get at a font's data than
to look through it, maybe we put in some special hackery whenever we go
looking in a font to see what other characters are available in that
font (maybe in some reasonable-sized chunk surrounding the char we're
looking for, or even throughout the whole damn font). We can store that
info in a range table or something. maybe we also have special hacks to
make sure that ascii goes fast.
6. try to do it in a system-independent way as possible, indicating, as
necessary, how certain system-dependent things would be handled [see
comments above about windows vs. x-windows in font handling].
maybe this sounds overwhelming, so don't take it as a command :-) i'm
just interested in the model of char/language->font mapping in your
head, since you've obviously thought a lot about this and you work first
hand with it in your sjt-xft workspace, and i'm confused in a lot of
these areas.
Ben> [2] now that i look more at ccl, i see that it's
utterly tied
Ben> to the old mule encoding, as it expects to look directly at
Ben> the internal encoding of the text. it would have to be
Ben> radically restructured for it to work on a
Ben> character-by-character level, not byte-by-byte. do you think
Ben> it's reasonable to just make ccl be an optional feature that
Ben> is compiled in only with old mule?
Yes. I think that CCL has four interesting applications that I know
of: shift-jis, big5, mule-ucs, and fast table-based character sets
(eg, KOI8). The first two have been in C forever, and the latter two
are in C in the unicode support now (at least in principle).
the problem of shift-jis, big5 and koi8 just goes away with the new
extended charsets. i don't know about mule-ucs, but it's just a stopgap
way of providing unicode support before the real support is implemented,
right?
ccl could be fixed by making it work on characters not bytes, e.g.
having commands to separate a character into a charset and its octets
(although this depends on the current language, etc.). but this would
be a real undertaking and i'd need to see real evidence of significant
things that could be done in ccl but no other reasonable way before
considering this. just ripping out certain things and making it
non-mule-specific is another possibility; for the moment i'll just have
it be old-mule only.
There are also some minor hacks from Aidan and others.
It might be worth using the CCL interpreter as a general fast
arithmetic engine for small numbers, but rip out the Mule stuff.
ben
4:48 a.m.
>>>> "Ben" == Ben Wing <ben(a)xemacs.org>
writes:
Ben> well, there are some fonts that have ...-devanagari-10646-... in them,
Ben> which are presumably unicode.
Yeah, I know, but I can't get them to work (haven't tried in a while,
though, because I'm using Xft almost all the time now). AFAICT the
standard X11R6 functions didn't handle them at all, and the XFree86
extensions wanted UTF-8 and did something magic internally.
Basically I think the way to go is fontconfig/Xft.
Ben> i need some info on how X does it before thinking about
Ben> doing it.
Xft simply does Unicode, period, AFAICS. (There may be an exception
for 8 bit encodings, but I hope that the 8 bit functions are
restricted to ISO-8859-1---aka Plane 0, Row 00 of Unicode---as they
should be.) I'm not sure how they go about using fonts indexed by
other registries, although I know that Xft can render traditional
bitmap fonts (BDF, PCF at least). Xft and X Unicode stuff is terribly
documented as far as I can tell, and where it is documented (eg,
Markus Kuhn's standard for Unicode X keysyms) the implementations
don't conform.
Ben> 1. the goal is: redisplay gets a stretch of characters and
Ben> the combined faces for each (if you want, just assume you've
Ben> been given a stretch entirely covered by the same combined
Ben> face) and has to figure out the appropriate font for each one
Ben> and the width of each character. (currently, it optimizes by
Ben> using the charset and assuming one font per charset; that
Ben> allows it to make one width check per stretch of characters
Ben> in the same charset. but this has to go, and i don't think
Ben> anything equivalent e.g. over unicode subranges would be
Ben> sound, as some fonts support only parts of certain subranges
Ben> so we'd have to use another font in some cases.)
I think that in general language segments are going to be large (99%
of the time covering the whole buffer). So you break up the region to
be displayed into segments by language, first. Then pass (segment,
language) pairs to the face-merging machinery, which returns a list of
(subsegment, language, face) triples. Then you pass those to the
rendering engine, which chooses font, and possibly other
characteristics, according to language. For example, in Japanese
italic is rarely used; underlines and boldface are preferred. So it
should be possible for an "emphasis" face would map to italic fonts
and no underlining for English, while in Japanese it maps to mincho
(same as the default face) and underlining, while "strong emphasis"
maps to bold and no underline for both languages.
If the first choice font doesn't contain the glyph for some character,
you fallback, first going down later choices for that language, then
inheriting from parents of the face (up to default), then finally
querying the system for any font with a glyph for that character (you
can do this in fontconfig).
Ben> 2. you also have available the buffer that the text came from
It's not clear to me that that's really useful, except at the very top
level. As I say, I believe that the vast majority of the time you'll
have only a few language segments. I don't think what I propose would
be inefficient for a document with many intralinear glosses, even.
Ben> maybe better renamed to a `culture' object
That's not really right, as you point out. We may eventually want
"culture" objects, but they should affect far more than redisplay.
How about "dialect" as something finer than language and a component
of culture?
Ben> 3. maybe you have a language->font mapping inside of a face. feel
Ben> free to choose the appropriate mapping and interface.
This is going to be hard. The reason is that fontconfig and Windows,
at least, have their own mechanisms for grouping languages. We
probably want to be careful that our mechanism can be configured to be
compatible with that.
4. while you're at it, feel free to design the appropriate interface
for mapping languages (or whatever) to a precedence list of charsets,
for unicode conversion (including both the mapping itself and the lisp
commands used to control this mapping).
Ben> 5. assume that our goal is to find the best font given the
Ben> language and associated preferences, but that in all cases we
Ben> need to find *some* font, if it's available. displaying a
Ben> "box" or whatever is a last resort, and ideally should happen
Ben> only when there is *no* font on the system (or at least, no
Ben> font from all those available to look through, i.e. specified
Ben> as global defaults).
Take a look at the fontconfig manual (in the general-docs package of
XEmacs as texinfo). I recommend this not because it's perfect; there
are warts. However, it seems to satisfy the vast majority of users
out there and it is the underlying mechanism for Xft and (I believe)
GTK v2. fontconfig/Xft are standard extensions in X11R6.8 (at worst,
it may be in X11R6.6 too).
Ben> assume also that we want this to be reasonably fast, so we
Ben> will probably need to cache char-> font mappings.
I don't know if fontconfig can give us that, so we may have to do some
redundant work.
Ben> maybe this sounds overwhelming, so don't take it as a command :-)
Actually, this is something I've been thinking about a lot. I don't
think it's really that hard, once we get rid of Mule charsets and
start thinking in terms of font (etc) repertoires defined as Unicode
subsets. Implementation is scary, so I'm very glad you're going to be
helping a lot with that.
Ben> the problem of shift-jis, big5 and koi8 just goes away with
Ben> the new extended charsets. i don't know about mule-ucs, but
Ben> it's just a stopgap way of providing unicode support before
Ben> the real support is implemented, right?
Sorry for the imprecision. That's exactly what I meant.
Ben> i'd need to see real evidence of significant things that
Ben> could be done in ccl but no other reasonable way before
Ben> considering this.
No, let's junk it (but check me on the last sentence of the next
paragraph! we don't want to lose KOI8 and so on, and we do need the
ability to define such at runtime).
CCL arithmetic is fast (because it doesn't do any of the magical LISP
stuff) and it can safely be called from redisplay (for the same
reason). That's all you can say for it, except that it can be done at
runtime. But `load-unicode-mapping-table' means that you can just
cons up a coded character set in a buffer, and install it directly.
Now, if we could do font-lock in CCL ... but we can't.
--
School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
Ask not how you can "do" free software business;
ask what your business can "do for" free software.
5:58 a.m.
Stephen J. Turnbull wrote:
>>>>>"Ben" == Ben Wing <ben(a)xemacs.org>
writes:
>>>>>
>>>>>
Ben> well, there are some fonts that have ...-devanagari-10646-... in them,
Ben> which are presumably unicode.
Yeah, I know, but I can't get them to work (haven't tried in a while,
though, because I'm using Xft almost all the time now). AFAICT the
standard X11R6 functions didn't handle them at all, and the XFree86
extensions wanted UTF-8 and did something magic internally.
theoretically it should just use straight UCS-2, no?
Basically I think the way to go is fontconfig/Xft.
Ben> i need some info on how X does it before thinking about
Ben> doing it.
Xft simply does Unicode, period, AFAICS. (There may be an exception
for 8 bit encodings, but I hope that the 8 bit functions are
restricted to ISO-8859-1---aka Plane 0, Row 00 of Unicode---as they
should be.) I'm not sure how they go about using fonts indexed by
other registries, although I know that Xft can render traditional
bitmap fonts (BDF, PCF at least). Xft and X Unicode stuff is terribly
documented as far as I can tell, and where it is documented (eg,
Markus Kuhn's standard for Unicode X keysyms) the implementations
don't conform.
if they use other registries then they have to have some conversion
between unicode and national charsets (not esp. hard).
Ben> 1. the goal is: redisplay gets a stretch of characters
and
Ben> the combined faces for each (if you want, just assume you've
Ben> been given a stretch entirely covered by the same combined
Ben> face) and has to figure out the appropriate font for each one
Ben> and the width of each character. (currently, it optimizes by
Ben> using the charset and assuming one font per charset; that
Ben> allows it to make one width check per stretch of characters
Ben> in the same charset. but this has to go, and i don't think
Ben> anything equivalent e.g. over unicode subranges would be
Ben> sound, as some fonts support only parts of certain subranges
Ben> so we'd have to use another font in some cases.)
I think that in general language segments are going to be large (99%
of the time covering the whole buffer). So you break up the region to
be displayed into segments by language, first. Then pass (segment,
language) pairs to the face-merging machinery, which returns a list of
(subsegment, language, face) triples. Then you pass those to the
rendering engine, which chooses font, and possibly other
characteristics, according to language. For example, in Japanese
italic is rarely used; underlines and boldface are preferred. So it
should be possible for an "emphasis" face would map to italic fonts
and no underlining for English, while in Japanese it maps to mincho
(same as the default face) and underlining, while "strong emphasis"
maps to bold and no underline for both languages.
If the first choice font doesn't contain the glyph for some character,
you fallback, first going down later choices for that language, then
inheriting from parents of the face (up to default), then finally
querying the system for any font with a glyph for that character (you
can do this in fontconfig).
ok. redisplay currently segments first by merged face and then by
charset. as a first pass i don't think we need to change that; the only
time this loses is when someone wants to do something weird like [a] use
a ligating script like arabic or devanagari; [b] switch faces in the
middle of a word. for the moment i'm not even worrying about ligating
at all, although dealing with this is not impossibly hard (basically,
redisplay has to calculate widths based on a segment, not
glyph-by-glyph; it already passes these segments on to the
display-specific code, which draws them a segment at a time, which more
or less does correct ligating behavior, at least under windows).
mostly what i'm confused about is how exactly we map characters to
fonts, see below ...
Ben> 3. maybe you have a language->font mapping inside of a
face. feel
Ben> free to choose the appropriate mapping and interface.
This is going to be hard. The reason is that fontconfig and Windows,
at least, have their own mechanisms for grouping languages. We
probably want to be careful that our mechanism can be configured to be
compatible with that.
well, we don't have to worry too much about this, i think; presumably,
`language' is an XEmacs concept that's used somehow or other to derive a
list of fonts to look through. what i don't understand is what the
proper interface onto the `face' object should be, to specify
language-specific fonts; i assume you have a better idea since you've
had to do this work under xft.
4. while you're at it, feel free to design the appropriate
interface
for mapping languages (or whatever) to a precedence list of charsets,
for unicode conversion (including both the mapping itself and the lisp
commands used to control this mapping).
Ben> 5. assume that our goal is to find the best font given the
Ben> language and associated preferences, but that in all cases we
Ben> need to find *some* font, if it's available. displaying a
Ben> "box" or whatever is a last resort, and ideally should happen
Ben> only when there is *no* font on the system (or at least, no
Ben> font from all those available to look through, i.e. specified
Ben> as global defaults).
Take a look at the fontconfig manual (in the general-docs package of
XEmacs as texinfo). I recommend this not because it's perfect; there
are warts. However, it seems to satisfy the vast majority of users
out there and it is the underlying mechanism for Xft and (I believe)
GTK v2. fontconfig/Xft are standard extensions in X11R6.8 (at worst,
it may be in X11R6.6 too).
ok, i'll look into this.
Ben> assume also that we want this to be reasonably fast, so
we
Ben> will probably need to cache char-> font mappings.
I don't know if fontconfig can give us that, so we may have to do some
redundant work.
we can do this ourselves; a cache won't be too hard.
Ben> maybe this sounds overwhelming, so don't take it as a
command :-)
Actually, this is something I've been thinking about a lot. I don't
think it's really that hard, once we get rid of Mule charsets and
start thinking in terms of font (etc) repertoires defined as Unicode
subsets. Implementation is scary, so I'm very glad you're going to be
helping a lot with that.
well, it'll take time. i did a bunch of work on char tables recently,
but it was time consuming ... and there's still more to go before i can
even start compiling. first pass will not display non-charset fonts
(but at least it won't eat the chars), and it will be littered with
functions named ichar_charset_obsolete_me_baby_please(). (well, there's
only one of those; but there are other functions that deserve the same
treatment, e.g. ensure_face_cachel_contains_charset().)
Now, if we could do font-lock in CCL ... but we can't.
well, most font-lock slowness is in the regexp or extent code, not in
the lisp.
ben
8:49 a.m.
Stephen J. Turnbull wrote:
>>>>>"Ben" == Ben Wing <ben(a)xemacs.org>
writes:
>>>>>
>>>>>
Ben> well, there are some fonts that have ...-devanagari-10646-... in them,
Ben> which are presumably unicode.
Yeah, I know, but I can't get them to work (haven't tried in a while,
though, because I'm using Xft almost all the time now). AFAICT the
standard X11R6 functions didn't handle them at all, and the XFree86
extensions wanted UTF-8 and did something magic internally.
Basically I think the way to go is fontconfig/Xft.
actually, it doesn't look as bad as i thought. there is such a thing as
XwcDrawImageString, which is a standard part of x windows and which
accepts a Unicode string as well as an XFontSet, which collects a bunch
of fonts together [although what it does if more than one font in the
fontset contains a glyph for a particular character is anyone's guess
...]. the definition of XCreateFontSet has a bunch of bogosity about
locales and charsets and fonts that possibly must exist for a particular
charset depending on the "locale definition in Xlib", and naturally such
definitions are nowhere to be found, but there's probably a reasonable
way of handling this that will work fairly generally.
10:59 a.m.
>>>> "Ben" == Ben Wing <ben(a)xemacs.org>
writes:
Ben> actually, it doesn't look as bad as i thought. there is such
Ben> a thing as XwcDrawImageString, which is a standard part of x
Ben> windows and which accepts a Unicode string as well as an
Ben> XFontSet
Well, actually it accepts wide characters. I have no idea what it
does with them, though, and I believe the standard is "implementation
dependent". I'm pretty sure on SunOS with JLE 4.1.3 it was documented
to take constant-width EUC-JP.
--
School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
Ask not how you can "do" free software business;
ask what your business can "do for" free software.
1:33 p.m.
Stephen J. Turnbull wrote:
>>>>> "Ben" == Ben Wing
<ben(a)xemacs.org> writes:
Ben> actually, it doesn't look as bad as i thought. there is such
Ben> a thing as XwcDrawImageString, which is a standard part of x
Ben> windows and which accepts a Unicode string as well as an
Ben> XFontSet
Well, actually it accepts wide characters. I have no idea what it
does with them, though, and I believe the standard is "implementation
dependent". I'm pretty sure on SunOS with JLE 4.1.3 it was documented
to take constant-width EUC-JP.
The Xlib documentation says "in the form supported by the host C
language environment", which I take to mean "whatever the OS normally
uses for wchar_t".
If __STDC_ISO_10646__ is defined, then wchar_t is ISO-10646. Other
than that, I'm not aware of any standard mechanism to determine the
system's encoding of wide characters.
--
Glynn Clements <glynn(a)gclements.plus.com>
6:05 a.m.
Stephen J. Turnbull wrote:
Yes. I think that CCL has four interesting applications that I know
of: shift-jis, big5, mule-ucs, and fast table-based character sets
(eg, KOI8). The first two have been in C forever, and the latter two
are in C in the unicode support now (at least in principle).
There are also some minor hacks from Aidan and others.
well, i can think of at least one example still, which is the chinese hz
encoding. although now that i look, i see that china-util.el does
indeed implement this, and doesn't use ccl.
ben
11:27 a.m.
>>>> "Hrvoje" == Hrvoje Niksic
<hniksic(a)xemacs.org> writes:
[snip]
Hrvoje> For the longest time I believed this to be a result of
Hrvoje> XEmacs's implementation of text properties on top of
Hrvoje> extents. I planned to convert Gnus to use extents
Hrvoje> natively, in the hope that it would speed up summary buffer
Hrvoje> generation. But according to profiling, most of the time
Hrvoje> is spent in `insert', at least once you discount excessive
Hrvoje> GC by increasing gc-cons-theshold. I'm not sure what
Hrvoje> XEmacs is doing wrong here. I suspect Mule is to blame,
Hrvoje> but I can't prove it.
No, I don't think so, I use no-mule xemacs-21.4.17 and it is slow as
well.
Uwe Brauer
6967
days inactive
6982
days old
75 comments
15 participants
participants (15)
-
Adrian Aichner -
Ben Wing -
Darryl Okahata -
David Kastrup -
Didier Verna -
Glynn Clements -
Hrvoje Niksic -
Jan Rychter -
Jesper Harder -
Jérôme Marant -
Raymond Toy -
Reiner Steib -
Simon Josefsson -
Stephen J. Turnbull -
Uwe Brauer