Separating the tests from the code being tested considered harmful

mb> If you backport a fix, you can backport the regression test mb> that goes with it. Porting the test is usually much easier. SJT> This is incorrect thinking, for a practical and a theoretical reason. SJT> In practice, it is not done, until somebody scratches their SJT> perfectionist itch. Often years later, as many of our developers SJT> aren't even aware that they should be writing tests, and it is a skill SJT> that very few of us have, as yet. It's nearly completely undocumented SJT> (I wrote what docs exist AFAICT; comments in test-harness.el don't SJT> count---if you can find those, you've already passed the halfway point SJT> in the obstacle course). What happens in practice is that people write tests for code that they've just changed. In particular, everyone who has just changed the behavior of the code will test it "by hand". It is a natural further step to include it in the test suite, since the test is effectively already written - only additional packaging is required. Those tests will not work on prior versions, preciesely because they were _designed_ to work differently. SJT> In theory, this ignores the fact that the test should be run against SJT> previous versions still in use (_not_ in current distribution) to SJT> determine whether they also suffer from the bug. ... SJT> As I made clear, I see the necessity for version-specific tests. SJT> I Version-specific tests are evil. In particular, they don't interact well with branches. More logical would be the too-horrible-to-contemplate (when (featurep 'bugfix-40346823) ...) SJT> think it is very bad practice to duplicate the vast majority of tests SJT> which should be passed by all recent Emacsen or which can require a SJT> feature. Your answer to that is "if we were disciplined, we'd do it SJT> my way." Too bad; there are a lot of things that would work better in SJT> this project if we raised the level of discipline. This aspect of SJT> discipline is a very low priority. The only kinds of tests that should be decoupled from the implementation are specification-based tests. Experience shows that producing such tests is no fun, and the free software community has not produced such tests. It is left to the Plum Halls of the world to create such test suites and charge big bucks for them. XEmacs is unlikely to have specification-based tests available, because it is unlikely to conform to any formal specification (for example, we would be unlikely to pass a Common Lisp test suite if one were available). SJT> OTOH, packaging test-harness and tests is likely to get more tests run SJT> _and_ written IMO. I strongly disagree. Both the dropping of the "no test failures" policy and the proposed de-coupling of the test suite decrease the level of trust in the test results and discourage addition of new tests. Martin