11:01 a.m.
src/ChangeLog addition:
2006-11-21 Aidan Kehoe <kehoea(a)parhasard.net>
* mule-coding.c (iso2022_decode):
Only take the lower seven bits of any eight-bit character that
would be illegal in UTF-8, when handling ISO/IR 196 escapes.
tests/ChangeLog addition:
2006-11-21 Aidan Kehoe <kehoea(a)parhasard.net>
* automated/mule-tests.el (featurep):
Add a test that ISO/IR 196 escape handling in ISO-2022-based
charsets don't choke on invalid bytes in UTF-8 text.
XEmacs Trunk source patch:
Diff command: cvs -q diff -u
Files affected: tests/automated/mule-tests.el
===================================================================
RCS src/mule-coding.c
===================================================================
RCS
Index: src/mule-coding.c
===================================================================
RCS file: /pack/xemacscvs/XEmacs/xemacs/src/mule-coding.c,v
retrieving revision 1.38
diff -u -r1.38 mule-coding.c
--- src/mule-coding.c 2006/06/03 17:50:55 1.38
+++ src/mule-coding.c 2006/11/21 10:54:28
＠＠ -1949,8 +1949,11 ＠＠
counter = 1;
}
else
- /* ASCII, or the lower control characters. */
- Dynarr_add (dst, c);
+ /* ASCII, or the lower control characters.
+
+ Perhaps we should signal an error if the character is in
+ the range 0x80-0xc0; this is illegal UTF-8. */
+ Dynarr_add (dst, (c & 0x7f));
break;
case 1:
Index: tests/automated/mule-tests.el
===================================================================
RCS file: /pack/xemacscvs/XEmacs/xemacs/tests/automated/mule-tests.el,v
retrieving revision 1.11
diff -u -r1.11 mule-tests.el
--- tests/automated/mule-tests.el 2006/11/20 19:21:56 1.11
+++ tests/automated/mule-tests.el 2006/11/21 10:54:28
＠＠ -441,6 +441,12 ＠＠
(eq (aref ccl-vector 4)
(encode-char (make-char 'control-1 31) 'ucs)))))
+ ;; This used to crash, at least in debug builds:
+
+ (Assert (decode-coding-string
+ (string ?\33 ?\45 ?\107 ?\306 ?\222 ?\215 ?\306)
+ 'iso-2022-jp))
+
;;---------------------------------------------------------------
;; Test charset-in-* functions
;;---------------------------------------------------------------
--
Santa Maradona, priez pour moi!
_______________________________________________
XEmacs-Patches mailing list
XEmacs-Patches(a)xemacs.org
http://calypso.tux.org/cgi-bin/mailman/listinfo/xemacs-patches
1:42 a.m.
QUERY
Aidan Kehoe writes:
Index: tests/automated/mule-tests.el
===================================================================
RCS file: /pack/xemacscvs/XEmacs/xemacs/tests/automated/mule-tests.el,v
retrieving revision 1.11
diff -u -r1.11 mule-tests.el
--- tests/automated/mule-tests.el 2006/11/20 19:21:56 1.11
+++ tests/automated/mule-tests.el 2006/11/21 10:54:28
＠＠ -441,6 +441,12 ＠＠
(eq (aref ccl-vector 4)
(encode-char (make-char 'control-1 31) 'ucs)))))
+ ;; This used to crash, at least in debug builds:
Please put (former) crashes in a separate file (eg, crash-tests.el),
since a crash would prevent running the rest of the tests.
Suggestions for how to handle this automatically are welcome; I think
that probably we should run a separate XEmacs for it, and maybe this
should be optional.
_______________________________________________
XEmacs-Patches mailing list
XEmacs-Patches(a)xemacs.org
http://calypso.tux.org/cgi-bin/mailman/listinfo/xemacs-patches
1:48 p.m.
Ar an dara lá is fiche de mí na Samhain, scríobh Stephen J. Turnbull:
> + ;; This used to crash, at least in debug builds:
Please put (former) crashes in a separate file (eg, crash-tests.el),
since a crash would prevent running the rest of the tests.
Suggestions for how to handle this automatically are welcome; I think
that probably we should run a separate XEmacs for it, and maybe this
should be optional.
That doesn’t reflect existing practise, and is not really fleshed out. I
agree that it would be better to move potentially-crashing tests into
separate files, and extend the test architecture to deal with that sensibly,
but that’s a separate issue from this change.
--
Santa Maradona, priez pour moi!
_______________________________________________
XEmacs-Patches mailing list
XEmacs-Patches(a)xemacs.org
http://calypso.tux.org/cgi-bin/mailman/listinfo/xemacs-patches
6:49 p.m.
Aidan Kehoe writes:
I agree that it would be better to move potentially-crashing tests
into separate files, and extend the test architecture to deal with
that sensibly, but that’s a separate issue from this change.
I didn't ask you to extend the test architecture, just to move the
known crash into a separate file. Please just do that. Suggestions
on architecture are welcome, but can wait.
_______________________________________________
XEmacs-Patches mailing list
XEmacs-Patches(a)xemacs.org
http://calypso.tux.org/cgi-bin/mailman/listinfo/xemacs-patches
9:25 p.m.
Ar an ceathrú lá is fiche de mí na Samhain, scríobh Stephen J. Turnbull:
> I agree that it would be better to move potentially-crashing
tests
> into separate files, and extend the test architecture to deal with
> that sensibly, but that’s a separate issue from this change.
I didn't ask you to extend the test architecture, just to move the
known crash into a separate file. Please just do that.
Okay, will get to it Saturday.
Suggestions on architecture are welcome, but can wait.
--
Santa Maradona, priez pour moi!
_______________________________________________
XEmacs-Patches mailing list
XEmacs-Patches(a)xemacs.org
http://calypso.tux.org/cgi-bin/mailman/listinfo/xemacs-patches
10:17 p.m.
New subject: [COMMIT] Move the ISO-IR 196 test, which may crash, to a separate file.
APPROVE COMMIT
NOTE: This patch has been committed.
tests/ChangeLog addition:
2006-11-25 Aidan Kehoe <kehoea(a)parhasard.net>
* automated/iso-ir-196-test.el:
* automated/mule-tests.el (featurep):
Move the tested added in the previous commit to a separate file.
XEmacs Trunk source patch:
Diff command: cvs -q diff -Nu
Files affected: tests/automated/mule-tests.el
===================================================================
RCS tests/automated/iso-ir-196-test.el
===================================================================
RCS
Index: tests/automated/iso-ir-196-test.el
===================================================================
RCS file: iso-ir-196-test.el
diff -N iso-ir-196-test.el
--- /dev/null Sat Nov 25 23:00:05 2006
+++ iso-ir-196-test.el Sat Nov 25 22:59:40 2006
＠＠ -0,0 +1,36 ＠＠
+;; Copyright (C) 2006 Free Software Foundation, Inc.
+
+;; Author: Aidan Kehoe <kehoea(a)parhasard.net>
+;; Maintainers: Aidan Kehoe <kehoea(a)parhasard.net>
+;; Created: 2006
+;; Keywords: tests
+
+;; This file is part of XEmacs.
+
+;; XEmacs is free software; you can redistribute it and/or modify it
+;; under the terms of the GNU General Public License as published by
+;; the Free Software Foundation; either version 2, or (at your option)
+;; any later version.
+
+;; XEmacs is distributed in the hope that it will be useful, but
+;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+;; General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with XEmacs; see the file COPYING. If not, write to the Free
+;; Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA
+;; 02111-1307, USA.
+
+;;; Synched up with: Not in FSF.
+
+;;; Commentary:
+
+;; Previously, we reacted badly to invalid sequences of non-UTF-8 octets
+;; when handling UTF-8 encode in ISO 2022.
+
+(when (featurep 'mule)
+ ;; This used to crash, at least in debug builds:
+ (Assert (decode-coding-string
+ (string ?\33 ?\45 ?\107 ?\306 ?\222 ?\215 ?\306)
+ 'iso-2022-jp)))
Index: tests/automated/mule-tests.el
===================================================================
RCS file: /pack/xemacscvs/XEmacs/xemacs/tests/automated/mule-tests.el,v
retrieving revision 1.12
diff -u -u -r1.12 mule-tests.el
--- tests/automated/mule-tests.el 2006/11/23 13:43:29 1.12
+++ tests/automated/mule-tests.el 2006/11/25 21:59:40
＠＠ -441,12 +441,6 ＠＠
(eq (aref ccl-vector 4)
(encode-char (make-char 'control-1 31) 'ucs)))))
- ;; This used to crash, at least in debug builds:
-
- (Assert (decode-coding-string
- (string ?\33 ?\45 ?\107 ?\306 ?\222 ?\215 ?\306)
- 'iso-2022-jp))
-
;;---------------------------------------------------------------
;; Test charset-in-* functions
;;---------------------------------------------------------------
--
Santa Maradona, priez pour moi!
_______________________________________________
XEmacs-Patches mailing list
XEmacs-Patches(a)xemacs.org
http://calypso.tux.org/cgi-bin/mailman/listinfo/xemacs-patches
10:17 p.m.
New subject: [COMMIT] Move the ISO-IR 196 test, which may crash, to a separate file.
APPROVE COMMIT
NOTE: This patch has been committed.
tests/ChangeLog addition:
2006-11-25 Aidan Kehoe <kehoea(a)parhasard.net>
* automated/iso-ir-196-test.el:
* automated/mule-tests.el (featurep):
Move the tested added in the previous commit to a separate file.
XEmacs Trunk source patch:
Diff command: cvs -q diff -Nu
Files affected: tests/automated/mule-tests.el
===================================================================
RCS tests/automated/iso-ir-196-test.el
===================================================================
RCS
Index: tests/automated/iso-ir-196-test.el
===================================================================
RCS file: iso-ir-196-test.el
diff -N iso-ir-196-test.el
--- /dev/null Sat Nov 25 23:00:05 2006
+++ iso-ir-196-test.el Sat Nov 25 22:59:40 2006
＠＠ -0,0 +1,36 ＠＠
+;; Copyright (C) 2006 Free Software Foundation, Inc.
+
+;; Author: Aidan Kehoe <kehoea(a)parhasard.net>
+;; Maintainers: Aidan Kehoe <kehoea(a)parhasard.net>
+;; Created: 2006
+;; Keywords: tests
+
+;; This file is part of XEmacs.
+
+;; XEmacs is free software; you can redistribute it and/or modify it
+;; under the terms of the GNU General Public License as published by
+;; the Free Software Foundation; either version 2, or (at your option)
+;; any later version.
+
+;; XEmacs is distributed in the hope that it will be useful, but
+;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+;; General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with XEmacs; see the file COPYING. If not, write to the Free
+;; Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA
+;; 02111-1307, USA.
+
+;;; Synched up with: Not in FSF.
+
+;;; Commentary:
+
+;; Previously, we reacted badly to invalid sequences of non-UTF-8 octets
+;; when handling UTF-8 encode in ISO 2022.
+
+(when (featurep 'mule)
+ ;; This used to crash, at least in debug builds:
+ (Assert (decode-coding-string
+ (string ?\33 ?\45 ?\107 ?\306 ?\222 ?\215 ?\306)
+ 'iso-2022-jp)))
Index: tests/automated/mule-tests.el
===================================================================
RCS file: /pack/xemacscvs/XEmacs/xemacs/tests/automated/mule-tests.el,v
retrieving revision 1.12
diff -u -u -r1.12 mule-tests.el
--- tests/automated/mule-tests.el 2006/11/23 13:43:29 1.12
+++ tests/automated/mule-tests.el 2006/11/25 21:59:40
＠＠ -441,12 +441,6 ＠＠
(eq (aref ccl-vector 4)
(encode-char (make-char 'control-1 31) 'ucs)))))
- ;; This used to crash, at least in debug builds:
-
- (Assert (decode-coding-string
- (string ?\33 ?\45 ?\107 ?\306 ?\222 ?\215 ?\306)
- 'iso-2022-jp))
-
;;---------------------------------------------------------------
;; Test charset-in-* functions
;;---------------------------------------------------------------
--
Santa Maradona, priez pour moi!
_______________________________________________
XEmacs-Patches mailing list
XEmacs-Patches(a)xemacs.org
http://calypso.tux.org/cgi-bin/mailman/listinfo/xemacs-patches
2:14 a.m.
New subject: [Q] Handle bytes in the range 0x80-0xC0 better when dealing with ISO-IR 196.
QUERY
Aidan Kehoe writes:
+ /* ASCII, or the lower control characters.
+
+ Perhaps we should signal an error if the character is in
+ the range 0x80-0xc0; this is illegal UTF-8. */
+ Dynarr_add (dst, (c & 0x7f));
Please do. This is corrupting the data.
I don't have a clue how to recover from it, but the user should at
least be told.
_______________________________________________
XEmacs-Patches mailing list
XEmacs-Patches(a)xemacs.org
http://calypso.tux.org/cgi-bin/mailman/listinfo/xemacs-patches
2:13 p.m.
New subject: [Q] Handle bytes in the range 0x80-0xC0 better when dealing with ISO-IR 196.
Ar an dara lá is fiche de mí na Samhain, scríobh Stephen J. Turnbull:
> + /* ASCII, or the lower control characters.
> +
> + Perhaps we should signal an error if the character is in
> + the range 0x80-0xc0; this is illegal UTF-8. */
> + Dynarr_add (dst, (c & 0x7f));
Please do. This is corrupting the data.
I don't have a clue how to recover from it, but the user should at
least be told.
I have a tentative plan to add a charset to XEmacs, 256 characters of which
reflect corrupt Unicode data. These 256 characters will be generated by
Unicode-oriented coding systems when they encounter invalid data:
(decode-coding-string "\x80\x80" 'utf-8)
=> "\200\200" ;; With funky redisplay properties once display tables and
;; char tables are integrated. Which, whee, is more work.
And will be ignored by them when writing:
(encode-coding-string (decode-coding-string "\x80\x80" 'utf-8) 'utf-8)
=> ""
This will allow applications like David Kastrup’s reconstruct-utf-8
sequences-from-fragmentary-TeX-error-messages to be possible, while not
contradicting the relevant Unicode standards. With Unicode as the internal
encoding, there’s no need to have a separate Mule character set; we can
stick their codes somewhere above the astral planes. But we should maintain
the same syntax code for them. Note also that, as far as I can work out,
these 256 codes will be sufficient for representing error data for all the
other Unicode-oriented representations well as UTF-8.
--
Santa Maradona, priez pour moi!
_______________________________________________
XEmacs-Patches mailing list
XEmacs-Patches(a)xemacs.org
http://calypso.tux.org/cgi-bin/mailman/listinfo/xemacs-patches
2:18 p.m.
APPROVE COMMIT
NOTE: this patch has been committed.
Ar an chéad lá is fiche de mí na Samhain, scríobh Aidan Kehoe:
[...]
--
Santa Maradona, priez pour moi!
_______________________________________________
XEmacs-Patches mailing list
XEmacs-Patches(a)xemacs.org
http://calypso.tux.org/cgi-bin/mailman/listinfo/xemacs-patches
6601
days inactive
6605
days old
9 comments
3 participants
participants (3)
-
Aidan Kehoe -
Stephen J. Turnbull -
Stephen J. Turnbull