#'query-coding-region and invalid Unicode sequences.
- First Post
- Replies
- Stats
-
Go to
- ----- 2024 -----
- May
- April
- March
- February
- January
- ----- 2023 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2022 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2021 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2020 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2019 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2018 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2017 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2016 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2015 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2014 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2013 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2012 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2011 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2010 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2009 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2008 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2007 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2006 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2005 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2004 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2003 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2002 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2001 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 2000 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 1999 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
- February
- January
- ----- 1998 -----
- December
- November
- October
- September
- August
- July
- June
- May
- April
- March
At the moment, #'query-coding-region ignores invalid Unicode sequences, it
treats them as always encodable--which they are, it is clear what they
should correspond to when written to disk. But Unicode says they are not
encodable. In the interactive code which will eventually use
#'query-coding-region I want to make this clear, and offer the user the
option of treating them like other unencodable characters (that is, removing
them from the file, or choosing other characters, for the most part), or of
ignoring this and going ahead and writing them to disk. So,
#'query-coding-region would need two modes, one where such sequences are
treated as encodable, one where they give an error (or give a range table
describing where to find the problematic characters.)
What would be a good way to provide this? Should I re-order the existing
parameters (now would be the time to do it) and add a new one after BUFFER,
or add it at the end? What should the argument be called?
--
¿Dónde estará ahora mi sobrino Yoghurtu Nghe, que tuvo que huir
precipitadamente de la aldea por culpa de la escasez de rinocerontes?
_______________________________________________
XEmacs-Beta mailing list
XEmacs-Beta(a)xemacs.org
http://calypso.tux.org/cgi-bin/mailman/listinfo/xemacs-beta
On 2009-01-14, Aidan Kehoe <kehoea(a)parhasard.net> wrote:
At the moment, #'query-coding-region ignores invalid Unicode sequences, it
treats them as always encodable--which they are, it is clear what they
should correspond to when written to disk. But Unicode says they are not
encodable.
What do you mean by "invalid Unicode sequence"? How do they get into
the buffer in the first place?
_______________________________________________
XEmacs-Beta mailing list
XEmacs-Beta(a)xemacs.org
http://calypso.tux.org/cgi-bin/mailman/listinfo/xemacs-beta
Ar an ceathrú lá déag de mí Eanair, scríobh Julian Bradfield:
On 2009-01-14, Aidan Kehoe <kehoea(a)parhasard.net> wrote:
>
> At the moment, #'query-coding-region ignores invalid Unicode sequences,
> it treats them as always encodable--which they are, it is clear what
> they should correspond to when written to disk. But Unicode says they
> are not encodable.
What do you mean by "invalid Unicode sequence"?
XEmacs characters that reflect that Unicode coding systems encountered
invalid octet sequences on disk. E.g. the output of
(decode-coding-string "\xd8\x00\x00\x01" 'utf-16-be) ;; Invalid surrogates
or
(decode-coding-string "\xe4" 'utf-8) ;; Attempt to decode Latin 1 as utf-8
We produce them so that loading a, for example, koi8-r file as utf-8, making
a single modification, and saving it does not necessarily trash the
non-ASCII content.
How do they get into the buffer in the first place?
unicode.c:1743 and the code that uses that macro.
Pre v23 GNU Emacs have the eight-bit-graphic character set, which they used
for this situation with UTF-8. 23 doesn’t seem to deal with this situation
well, as far as I can tell, I don’t get the option to write the invalid
sequence to disk.
The attached file is UTF-16 with byte order mark, and has an invalid
sequence after the first ". ". Current GNU Emacs deals with it badly. XEmacs
deals with it a little better, but not in a particularly stellar way right
now either.
--
¿Dónde estará ahora mi sobrino Yoghurtu Nghe, que tuvo que huir
precipitadamente de la aldea por culpa de la escasez de rinocerontes?
_______________________________________________
XEmacs-Beta mailing list
XEmacs-Beta(a)xemacs.org
http://calypso.tux.org/cgi-bin/mailman/listinfo/xemacs-beta
Ar an ceathrú lá déag de mí Eanair, scríobh Aidan Kehoe:
[...] So, #'query-coding-region would need two modes, one where
such
[invalid Unicode] sequences are treated as encodable, one where they give
an error (or give a range table describing where to find the problematic
characters.)
What would be a good way to provide this? Should I re-order the existing
parameters (now would be the time to do it) and add a new one after BUFFER,
or add it at the end? What should the argument be called?
So, yeah, I’m still interested in answers to these questions.
--
¿Dónde estará ahora mi sobrino Yoghurtu Nghe, que tuvo que huir
precipitadamente de la aldea por culpa de la escasez de rinocerontes?
_______________________________________________
XEmacs-Beta mailing list
XEmacs-Beta(a)xemacs.org
http://calypso.tux.org/cgi-bin/mailman/listinfo/xemacs-beta
5573
days inactive
5592
days old
3 comments
2 participants
tags (0)
participants (2)
-
Aidan Kehoe -
Julian Bradfield