4:31 p.m.
Hi,
I've added some issues with the test suite. issue875, issue876, and
issue878 related to availability of packages -- several tests won't
run without packages (or even crash the test-harness so it doesn't
report the previous results), and others give different results
depending on availability of packages. Aidan, if you have a minute to
spare I'd appreciate comments on refactoring the lisp/mule/*-util.el
files, see issue876 for a slightly more detailed description.
issue877 is a RFE for SkipIf to maintain a count of skipped tests and
report that in the summaries.
If anybody has run into oddities or issues with the test suite, this
would be a good time to report them! Even if you don't have time to
write a "good" report (mine are hardly models of good reporting!)
Steve
_______________________________________________
XEmacs-Beta mailing list
XEmacs-Beta(a)xemacs.org
http://lists.xemacs.org/mailman/listinfo/xemacs-beta
7:52 a.m.
>>>> Stephen J Turnbull <stephen(a)xemacs.org> writes:
[...] several tests won't run without packages (or even crash
the
test-harness so it doesn't report the previous results), and others
give different results depending on availability of packages.
I guess this is why the test fails in the buildbot? (Reason why I
configured it to ignore failures there...)
Example from buildbot:
http://www.lidell.nu/xemacs-buildbot/builders/x86%20gentoo%20cfg0/builds/...
base64-tests.el: 1234 of 1234 tests successful (100%).
byte-compiler-tests.el: 66 of 66 tests successful (100%).
c-tests.el: 126 of 126 tests successful (100%).
case-tests.el: 9436 of 9436 tests successful (100%).
ccl-tests.el: 4570 of 4570 tests successful (100%).
completion-tests.el: 126 of 126 tests successful (100%).
database-tests.el: 20 of 20 tests successful (100%).
dired-tests.el: 32 of 32 tests successful (100%).
extent-tests.el: 200 of 200 tests successful (100%).
face-tests.el: No tests run.
file-tests.el: 10 of 10 tests successful (100%).
files-tests.el: 20 of 20 tests successful (100%).
hash-table-tests.el: 11892 of 11892 tests successful (100%).
iso-ir-196-test.el: 2 of 2 tests successful (100%).
keymap-tests.el: 28 tests completed (aborted).
lisp-reader-tests.el: 90 of 90 tests successful (100%).
lisp-tests.el: 14298 of 14308 tests successful ( 99%).
md5-tests.el: 56 of 56 tests successful (100%).
mule-tests.el: 703 tests completed (aborted).
os-tests.el: 12 of 12 tests successful (100%).
process-tests.el: 36 of 36 tests successful (100%).
query-coding-tests.el: 2548 of 2548 tests successful (100%).
regexp-tests.el: 6594 of 6594 tests successful (100%).
region-tests.el: 28 of 28 tests successful (100%).
register-tests.el: 12 of 12 tests successful (100%).
search-tests.el: 120 of 120 tests successful (100%).
symbol-tests.el: 252 of 252 tests successful (100%).
syntax-ppss-tests.el: 8 of 8 tests successful (100%).
syntax-tests.el: 148 of 154 tests successful ( 96%).
tag-tests.el: 6 of 6 tests successful (100%).
weak-tests.el: 140 of 140 tests successful (100%).
Yours
--
%% Mats
_______________________________________________
XEmacs-Beta mailing list
XEmacs-Beta(a)xemacs.org
http://lists.xemacs.org/mailman/listinfo/xemacs-beta
9:43 a.m.
Mats Lidell writes:
I guess this is why the test fails in the buildbot? (Reason why I
configured it to ignore failures there...)
Example from buildbot:
http://www.lidell.nu/xemacs-buildbot/builders/x86%20gentoo%20cfg0/builds/...
[...]
byte-compiler-tests.el: 66 of 66 tests successful (100%).
IIRC, these tests are incomplete due to lack of defadvice (protected
by SkipIf).
face-tests.el: No tests run.
Possibly due to missing packages.
keymap-tests.el: 28 tests completed (aborted).
Due to missing packages.
lisp-tests.el: 14298 of 14308 tests successful ( 99%).
These are real failures.
mule-tests.el: 703 tests completed (aborted).
This error is probably due to missing packages (Jerry fixed ethiopic
in packages but not in core).
syntax-tests.el: 148 of 154 tests successful ( 96%).
These are real failures. I think they're wrapped with ExpectFailure
though.
I believe that only the crashes in keymap-tests and mule-tests would
be recorded by the buildbot as build failures.
Steve
_______________________________________________
XEmacs-Beta mailing list
XEmacs-Beta(a)xemacs.org
http://lists.xemacs.org/mailman/listinfo/xemacs-beta
6:06 a.m.
>>>> Stephen J Turnbull <stephen(a)xemacs.org> writes:
I believe that only the crashes in keymap-tests and mule-tests would
be recorded by the buildbot as build failures.
It would be more practical for the buildbot if "make check" only would
run the test cases that are contained in the core. Then it would be
easy to let a test failure mark the whole build as a failure.
Why not move test cases that depends on packages to those packages
instead or, maybe a pragmatic compromise, that we have a
check-packages target in core for those!?
Yours
--
%% Mats
_______________________________________________
XEmacs-Beta mailing list
XEmacs-Beta(a)xemacs.org
http://lists.xemacs.org/mailman/listinfo/xemacs-beta
9:02 a.m.
Mats Lidell writes:
>>>>> Stephen J Turnbull <stephen(a)xemacs.org>
writes:
> I believe that only the crashes in keymap-tests and mule-tests would
> be recorded by the buildbot as build failures.
It would be more practical for the buildbot if "make check" only would
run the test cases that are contained in the core. Then it would be
easy to let a test failure mark the whole build as a failure.
Why not move test cases that depends on packages to those packages
If the feature is in core, it should be tested when we build core.
instead or, maybe a pragmatic compromise, that we have a
check-packages target in core for those!?
We already can test for the package (or the function/variable, if
autoloaded) being required in each function using the SkipIf macro. I
think that works as well as what you're suggesting for making buildbot
failures be meaningful (ie, only failing on actual regressions).
There are four issues (that I know of) at present in the test-harness
framework that are buildbot-relevant.
(1) KnownBug is appropriate, but not used, so bugs we already know
about trigger 'bot failures. (I don't have examples offhand so I
could be wrong about this one. 2, 3, and 4 are confirmed, though.)
(2) SkipIf is needed, but not used, and the test suite actually
crashes, triggering a 'bot failure.
(3) SkipIf could be used, but then the feature would not be tested at
all, so we should find a better way to test that works with just
core XEmacs.
(4) SkipIf doesn't report skipped tests in the summaries, so it looks
like all tests are passing but in fact some are skipped so we
don't know.
_______________________________________________
XEmacs-Beta mailing list
XEmacs-Beta(a)xemacs.org
http://lists.xemacs.org/mailman/listinfo/xemacs-beta
5:28 p.m.
>>>> Stephen J Turnbull <stephen(a)xemacs.org> writes:
(Have been out travelling so trying to catch up so I might miss some
details. Nice to see some mail activity... keep it up!)
If the feature is in core, it should be tested when we build core.
Of course but then de package dependencies should be removed so that
it only depends on core functionality. (As I get it you have fixed
this for some cases right?)
[...]
There are four issues (that I know of) at present in the
test-harness
framework that are buildbot-relevant.
[1, 2, 3, 4 ...]
OK. So your saying that by applying KnownBug, SkipIf and rewriting
tests consistently we could come to a situation where we could start
letting buildbot signal a failure on error code from "make check"?
Sounds like a small project to do that.
Isn't there some way to tell XEmacs to ignore the package paths
completely so developers would see the same environment as buildbot
does when running "make check"? So that it will be obvious that you
have used a non core function in the unit test?
Yours
--
%% Mats
_______________________________________________
XEmacs-Beta mailing list
XEmacs-Beta(a)xemacs.org
http://lists.xemacs.org/mailman/listinfo/xemacs-beta
6:54 p.m.
Mats Lidell writes:
>>>>> Stephen J Turnbull <stephen(a)xemacs.org>
writes:
Isn't there some way to tell XEmacs to ignore the package paths
completely so developers would see the same environment as buildbot
does when running "make check"? So that it will be obvious that you
have used a non core function in the unit test?
Yes. xemacs --no-packages will do that, as will building with
--with-prefix=no.
But I'm leery of making --no-packages the default for tests because
it's hard enough to get people to write tests as it is.
_______________________________________________
XEmacs-Beta mailing list
XEmacs-Beta(a)xemacs.org
http://lists.xemacs.org/mailman/listinfo/xemacs-beta
2:35 p.m.
Ar an triú lá déag de mí Deireadh Fómhair, scríobh Stephen J. Turnbull:
> face-tests.el: No tests run.
Possibly due to missing packages.
No, font-mgr not available.
> keymap-tests.el: 28 tests completed
(aborted).
Due to missing packages.
This should work without packages, I’d be interested in the backtrace, but
am not going to follow up now.
> lisp-tests.el: 14298 of 14308 tests successful (
99%).
These are real failures.
> mule-tests.el: 703 tests completed (aborted).
This error is probably due to missing packages (Jerry fixed ethiopic
in packages but not in core).
> syntax-tests.el: 148 of 154 tests successful ( 96%).
These are real failures. I think they're wrapped with ExpectFailure
though.
I believe that only the crashes in keymap-tests and mule-tests would
be recorded by the buildbot as build failures.
That’s the way it should work, definitely.
--
‘Tramadol is further fed to cattle […] when working them […] (as draft
animals) so that the animals do not get tired quickly. …’
— Angewandte Chemie, Sept 2014, describing the social context of
(synthetic) tramadol having been found in Cameroon tree roots.
_______________________________________________
XEmacs-Beta mailing list
XEmacs-Beta(a)xemacs.org
http://lists.xemacs.org/mailman/listinfo/xemacs-beta
3468
days inactive
3477
days old
7 comments
3 participants
participants (3)
-
Aidan Kehoe -
Stephen J. Turnbull -
The XEmacs Package Smoketest