>>>> "SJT" == Stephen J Turnbull
<stephen(a)xemacs.org> writes:
SJT> In the above framework, tests would be available in the same
SJT> distribution as the sources being tested. However, with packagization
SJT> they would become a true regression test, and would also allow us (and
SJT> users) to backtrack pre-existing bugs. (For example, every XEmacs
SJT> released in the 21.5 series fails the same tests in regexp-tests.el
SJT> that 21.4 does, as do 21.1.14 and 21.1.9. I think we can presume that
SJT> 21.2.xx all do, too.)
I hope you're not suggesting that there be only one set of tests, to
be applied to all different versions of the software. The point of a
test suite is to reflect what works in any particular version. One
job of the release engineer is to ensure that the test suite always
passes 100%, so no regressions enter the code base. The key to this
working is that the test suite evolves along with the code it's
testing.
You can't run 21.4's test suite against 21.1. Not only would you get
failures, but the test suite would just hang because the 21.1 code
infloops.
Test suites are the ultimate non-portable software, and rightly so.
Martin