[AC21.5] Track "unexpected errors" in test-harness

APPROVE COMMIT 21.5 This patch tracks the unexpected errors which do not get reported as "failures" by the per file code, and reports the count, as well as the file(s) where they occurred, in the summary report. Not fully tested, but it seems to work. After a bit of consideration, I decided that most of the ones I've seen are things like missing packages or executables, or the Mac's funky handling of non-ASCII file names. These really are not failures of XEmacs proper, and aren't likely to occur in actual practice. So I think this is appropriate. I'd be happy to entertain arguments that they should be converted to per-file failures. Also, the file names that are printed are absolute paths. Personally, I think that's a good idea; I keep bunches of workspaces around, and I'm not always in the right one when reading beta.err. Index: tests/ChangeLog =================================================================== RCS file: /pack/xemacscvs/XEmacs/xemacs/tests/ChangeLog,v retrieving revision 1.80 diff -u -r1.80 ChangeLog --- tests/ChangeLog 3 Jun 2006 17:51:03 -0000 1.80 +++ tests/ChangeLog 24 Jun 2006 11:37:48 -0000 ＠＠ -0,0 +1,4 ＠＠ +2006-06-24 Stephen J. Turnbull <stephen(a)xemacs.org&gt; + + * automated/test-harness.el: Improve handling of unexpected errors. + Index: tests/automated/test-harness.el =================================================================== RCS file: /pack/xemacscvs/XEmacs/xemacs/tests/automated/test-harness.el,v retrieving revision 1.20 diff -u -r1.20 test-harness.el --- tests/automated/test-harness.el 3 May 2004 15:08:51 -0000 1.20 +++ tests/automated/test-harness.el 24 Jun 2006 11:37:49 -0000 ＠＠ -45,6 +45,23 ＠＠ (require 'bytecomp) +(defvar unexpected-test-suite-failures 0 + "Cumulative number of unexpected failures since test-harness was loaded. + +\"Unexpected failures\" are those caught by a generic handler established +outside of the test context. As such they involve an abort of the test +suite for the file being tested. + +They often occur during preparation of a test or recording of the results. +For example, an executable used to generate test data might not be present +on the system, or a system error might occur while reading a data file.") + +(defvar unexpected-test-suite-failure-files nil + "List of test files causing unexpected failures.") + +;; Declared for dynamic scope; _do not_ initialize here. +(defvar unexpected-test-file-failures) + (defvar test-harness-test-compiled nil "Non-nil means the test code was compiled before execution.") ＠＠ -134,6 +151,7 ＠＠ (setq body (cons (read buffer) body))) (end-of-file nil) (error + (incf unexpected-test-file-failures) (princ (format "Unexpected error %S reading forms from buffer\n" error-info)))) `(lambda () ＠＠ -144,7 +162,6 ＠＠ (defvar missing-message-failures) (defvar other-failures) - (defvar unexpected-test-suite-failure) (defvar trick-optimizer) ,＠(nreverse body)))) ＠＠ -158,6 +175,7 ＠＠ (wrong-error-failures 0) (missing-message-failures 0) (other-failures 0) + (unexpected-test-file-failures 0) ;; #### perhaps this should be a defvar, and output at the very end ;; OTOH, this way AC types can use a null EMACSPACKAGEPATH to find ＠＠ -165,7 +183,6 ＠＠ (skipped-test-reasons (make-hash-table :test 'equal)) (trick-optimizer nil) - (unexpected-test-suite-failure nil) (debug-on-error t) (pass-stream nil)) (with-output-to-temp-buffer "*Test-Log*" ＠＠ -320,7 +337,7 ＠＠ (condition-case error-info (funcall (test-harness-read-from-buffer inbuffer)) (error - (setq unexpected-test-suite-failure t) + (incf unexpected-test-file-failures) (princ (format "Unexpected error %S while executing interpreted code\n" error-info)) (message "Unexpected error %S while executing interpreted code." error-info) ＠＠ -341,6 +358,7 ＠＠ (condition-case error-info (if code (funcall code)) (error + (incf unexpected-test-file-failures) (princ (format "Unexpected error %S while executing byte-compiled code\n" error-info)) (message "Unexpected error %S while executing byte-compiled code." error-info) ＠＠ -383,7 +401,11 ＠＠ (cons (list filename passes total) test-harness-file-results-alist)) (message "%s" summary-msg)) - (when unexpected-test-suite-failure + (when (> unexpected-test-file-failures 0) + (setq unexpected-test-suite-failure-files + (cons filename unexpected-test-suite-failure-files)) + (setq unexpected-test-suite-failures + (+ unexpected-test-suite-failures unexpected-test-file-failures)) (message "Test suite execution failed unexpectedly.")) (fmakunbound 'Assert) (fmakunbound 'Check-Error) ＠＠ -510,7 +532,23 ＠＠ (/ (* 100 nsucc) ntest)) (message test-harness-null-summary-template (concat basename ":"))) - (setq results (cdr results)))))) + (setq results (cdr results))))) + (when (> unexpected-test-suite-failures 0) + (message "\n***** There %s %d unexpected test suite %s in %s:" + (if (= unexpected-test-suite-failures 1) "was" "were") + unexpected-test-suite-failures + (if (= unexpected-test-suite-failures 1) "failure" "failures") + (if (= (length unexpected-test-suite-failure-files) 1) + "file" + "files")) + (while unexpected-test-suite-failure-files + (let ((line (pop unexpected-test-suite-failure-files))) + (while (and (< (length line) 61) + unexpected-test-suite-failure-files) + (setq line + (concat line " " + (pop unexpected-test-suite-failure-files)))) + (message line))))) (message "\nDone") (kill-emacs (if error 1 0)))) -- Graduate School of Systems and Information Engineering University of Tsukuba http://turnbull.sk.tsukuba.ac.jp/ Tennodai 1-1-1 Tsukuba 305-8573 JAPAN Economics of Information Communication and Computation Systems Experimental Economics, Microeconomic Theory, Game Theory