Re: Separating the tests from the code being tested considered harmful

mb> What happens in practice is that people write tests for code mb> that they've just changed. In particular, everyone who has mb> just changed the behavior of the code will test it "by hand". mb> It is a natural further step to include it in the test suite, I see no evidence in ChangeLogs that this is "natural". I see a few core people (you, Ben, Hrvoje, Yoshiki, myself, Mike Sperber) writing all the tests, and with the exception of updating the Mule tests for 21.5's unicode (Ben) and weak-stuff tests (Mike)[1], I'm the only person who's contributed new tests since the release of 21.4, although Ben and Mike have both contributed bug fixes. Furthermore, the bug fixes have not been backported[2], although I think they could be, at first glance. I see essentially no tests for package code, period, although some of the packages may have their own test suites. mb> Those tests will not work on prior versions, preciesely mb> because they were _designed_ to work differently. I don't plan to move the tests out of the core tree until I've had a chance to check this assertion against the tests and the versions. I suspect that there are not all that many of these, but we'll see. mb> Version-specific tests are evil. In particular, they don't mb> interact well with branches. Yeah, well, we don't have any problems with interacting with branches at the moment. :-( mb> More logical would be the too-horrible-to-contemplate mb> (when (featurep 'bugfix-40346823) ...) I've contemplated it. Big deal. Not feasible in this project. mb> The only kinds of tests that should be decoupled from the mb> implementation are specification-based tests. Experience mb> shows that producing such tests is no fun, and the free mb> software community has not produced such tests. If a test worked in 21.1, and it works in 21.4, and it works in 21.5, I think there's an implicit specification there. At that point, the test itself becomes the specification. Not a terribly desirable form of specification, but better than nothing. And if the test tests behavior documented in the User Manual or the Lispref or the docstring, I think that's specification enough. I agree with you that we're not likely to see true specification-based tests. The question is whether we can't do better than we're currently doing. SJT> OTOH, packaging test-harness and tests is likely to get more SJT> tests run _and_ written IMO. mb> I strongly disagree. Both the dropping of the "no test mb> failures" policy If you were going to back it up with code, I'd say wizard, let's put it to the Board. I'd like to have that policy, but I am not able to back it up with code, and I don't see anyone else doing it, either. Either they're ignoring the test results, or they're not running the tests. Neither one says much for current trust in the test suite. Something needs to be done; saying "if Martin were active, there'd be a no-failures-in-releases policy" doesn't cut it. I think risking public embarrassment by at least having the testers see the failures might light a few fires under a few seats. What else can Da Mgmt do? Vin can implement it by fiat in 21.4, of course, and maybe he should. I'm not going to lie by disabling failing tests for which we have a specification (regexps) or which have an implicit specification because no intentional behavior change has been implemented (a couple of other failures we are observing in 21.5). And I'll scream if somebody else does. _That_ would surely decrease the level of trust in the tests. mb> and the proposed de-coupling of the test suite decrease the mb> level of trust in the test results and discourage addition of mb> new tests. _If_ running the recent tests against older XEmacsen showed many false positives because the new tests test for new specs (explicitly or implicitly), yes, that would somewhat decrease confidence. And I will do that before releasing a tests package, let alone removing the tests currently in core. Well, I think that just doing and reporting the exercise of trying to make a test suite package and running it on the various versions available would be useful in terms of raising consciousness about the test suite. I may do it on that basis even if the idea of really moving tests to a package turns out to be nonsense. Footnotes: [1] The nature of those additions tends to support your argument. [2] Which supports my contention that writing tests doesn't seem to be a natural activity. -- Institute of Policy and Planning Sciences http://turnbull.sk.tsukuba.ac.jp University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Ask not how you can "do" free software business; ask what your business can "do for" free software.