11:14 p.m.
================================================================
Dear Bug Team!
When opening a text file containing the words "méi wéi Europäësch
Compétitivitéit" (Luxembourgish for "more than European
Competitivity"), the text appears garbled in some strange Chinese
character set (Zh/Big5). When opening the same text in vi, it appears
normal. As I'm typing this in emacs, I'm not sure how this will arrive
at the other end...
I admit that the Chinese might overtake Europe one day, but IMHO emacs
is way ahead of its time here ;-)
Is there a possibility to de-activate these political forecasts? ;-)
Thanks,
Alain
================================================================
System Info to help track down your bug:
---------------------------------------
uname -a: Linux mozart 2.6.9 #1 SMP Wed Dec 1 14:42:08 UTC 2004 i686 i686 i386 GNU/Linux
./configure 'i386-suse-linux' '--with-gcc' '--dynamic'
'--debug=no' '--error-checking=none' '--cflags=-O2 -march=i586
-mcpu=i686 -fmessage-length=0 -Wall -g -Wall -Wno-switch -pipe -O2 -march=i586 -mcpu=i686
-fmessage-length=0 -Wall -g -fno-strict-aliasing' '--ldflags='
'--prefix=/usr' '--exec-prefix=/usr' '--bindir=/usr/X11R6/bin'
'--datadir=/usr/share'
'--archlibdir=/usr/lib/xemacs/21.5.18/i386-suse-linux'
'--docdir=/usr/lib/xemacs/21.5.18/i386-suse-linux'
'--lispdir=/usr/share/xemacs/21.5.18/lisp'
'--etcdir=/usr/share/xemacs/21.5.18/etc' '--statedir=/var/lib/xemacs'
'--moduledir=/usr/lib/xemacs/21.5.18/i386-suse-linux/modules'
'--infodir=/usr/share/xemacs/info' '--mandir=/usr/share/man/man1'
'--with-database=gnudbm,berkdb' '--with-ncurses' '--with-tty=yes'
'--with-site-lisp' '--with-sound=native' '--with-gpm'
'--pdump' '--with-xpm' '--with-gif' '--with-tiff'
'--with-jpeg' '--with-png' '--with-x' '--with-athena=3d'
'--with-menubars=lucid' '--with-scrollbars=lucid' '--with!
-dialogs=athena' '--with-widgets=athena'
'--x-includes=/usr/X11R6/include' '--x-libraries=/usr/X11R6/lib'
'--with-mule' '--with-xim=xlib' '--with-xfs'
'--with-clash-detection' '--with-ldap' '--with-pop'
XEmacs 21.5-b18 "chestnut" (+CVS-20041021) configured for
`i386-suse-linux'.
Compilation Environment and Installation Defaults:
Source code location: /usr/src/packages/BUILD/xemacs-21.5.18
Installation prefix: /usr
Operating system description file: `s/linux.h'
Machine description file: `m/intel386.h'
Compiler version: gcc (GCC) 3.3.5 20050117 (prerelease) (SUSE Linux)
- GCC specs file: /usr/lib/gcc-lib/i586-suse-linux/3.3.5/specs
- Compiler command: gcc -Wall -Wno-switch -Winline -Wmissing-prototypes
-Wsign-compare -Wundef -Wstrict-prototypes -Wpacked -Wunused-parameter -O2 -march=i586
-mcpu=i686 -fmessage-length=0 -Wall -g -Wall -Wno-switch -pipe -O2 -march=i586 -mcpu=i686
-fmessage-length=0 -Wall -g -fno-strict-aliasing
libc version:
Relocating allocator for buffers: no
GNU version of malloc: yes
- Using Doug Lea's new malloc from the GNU C Library.
Window System:
Compiling in support for the X window system:
- X Windows headers location: /usr/X11R6/include
- X Windows libraries location: /usr/X11R6/lib
- Handling WM_COMMAND properly.
Compiling in support for the Athena widget set:
- Athena headers location: X11/Xaw3d
- Athena library to link: Xaw3d
Using Lucid menubars.
Using Lucid scrollbars.
Using Athena dialog boxes.
Using Athena native widgets.
TTY:
Compiling in support for ncurses.
Compiling in support for GPM (General Purpose Mouse).
Images:
Compiling in support for GIF images (builtin).
Compiling in support for XPM images.
Compiling in support for PNG images.
Compiling in support for JPEG images.
Compiling in support for TIFF images.
Compiling in support for X-Face message headers.
Sound:
Compiling in support for sound (native).
Databases:
Compiling in support for Berkeley database.
Compiling in support for GNU DBM.
Compiling in support for LDAP.
Internationalization:
Compiling in support for Mule (multi-lingual Emacs).
Compiling in support for XIM (X11R5+ I18N input method).
- Using raw Xlib to provide XIM support.
- Using XFontSet to provide bilingual menubar.
Compiling in support for Canna on Mule.
Compiling in support for the WNN input method on Mule.
Mail:
Compiling in support for POP mail retrieval.
Compiling in support for "dot-locking" mail spool file locking method.
Other Features:
Inhibiting IPv6 canonicalization at startup.
Compiling in support for dynamic shared object modules.
Using the new portable dumper.
Load-Path Lisp Shadows:
----------------------
(/home/aknaff/startup/emacs/mode-compile
/usr/share/xemacs/xemacs-packages/lisp/prog-modes/mode-compile
/usr/share/xemacs/mule-packages/lisp/mule-base/ethio-util
/usr/share/xemacs/21.5.18/lisp/mule/ethio-util
/usr/share/xemacs/mule-packages/lisp/mule-base/china-util
/usr/share/xemacs/21.5.18/lisp/mule/china-util
/usr/share/xemacs/mule-packages/lisp/mule-base/cyril-util
/usr/share/xemacs/21.5.18/lisp/mule/cyril-util
/usr/share/xemacs/mule-packages/lisp/mule-base/korea-util
/usr/share/xemacs/21.5.18/lisp/mule/korea-util
/usr/share/xemacs/mule-packages/lisp/mule-base/viet-util
/usr/share/xemacs/21.5.18/lisp/mule/viet-util
/usr/share/xemacs/mule-packages/lisp/mule-base/japan-util
/usr/share/xemacs/21.5.18/lisp/mule/japan-util
/usr/share/xemacs/mule-packages/lisp/mule-ucs/unicode
/usr/share/xemacs/21.5.18/lisp/unicode
/usr/share/xemacs/xemacs-packages/lisp/build/build-report
/usr/share/xemacs/21.5.18/lisp/build-report
/usr/share/xemacs/site-packages/lisp/term/gnome
/usr/share/xemacs/21.5.18/lisp/gnome)
Installed XEmacs Packages:
-------------------------
(zenirc ver: 1.15 upstream: 2.112)
(xwem ver: 1.19 upstream: lg(a)xwem.org--2004/xwem--main--2.0--version-0)
(xslt-process ver: 1.11 upstream: 1.2.1)
(xslide ver: 1.09 upstream: 0.2.2)
(xlib ver: 1.13 upstream: lg(a)xwem.org--2004/xlib--main--2.0--version-0)
(xemacs-devel ver: 1.7 upstream: No-Upstream-Ver)
(xemacs-base ver: 1.94 upstream: No-Upstream-Ver)
(x-symbol ver: 1.1 upstream: 4.5.1)
(w3 ver: 1.3 upstream: 4.0pre47)
(vm ver: 7.18 upstream: 7.17)
(viper ver: 1.37 upstream: 3.09)
(view-process ver: 1.13 upstream: 2.4)
(vhdl ver: 1.19 upstream: 3.32.20)
(vc-cc ver: 1.22 upstream: No-Upstream-Ver)
(vc ver: 1.4 upstream: No-Upstream-Ver)
(tramp ver: 1.22 upstream: 2.0.47)
(tpu ver: 1.14 upstream: 4.2X)
(tooltalk ver: 1.15 upstream: No-Upstream-Ver)
(tm ver: 1.37 upstream: No-Upstream-Ver)
(time ver: 1.14 upstream: 1.17)
(textools ver: 1.15 upstream: No-Upstream-Ver)
(text-modes ver: 1.82 upstream: No-Upstream-Ver)
(texinfo ver: 1.28 upstream: No-Upstream-Ver)
(supercite ver: 1.2 upstream: 3.55x3)
(strokes ver: 1.1 upstream: No-Upstream-Ver)
(speedbar ver: 1.27 upstream: 0.14beta4)
(sounds-wav ver: 1.12 upstream: No-Upstream-Ver)
(sounds-au ver: 1.12 upstream: No-Upstream-Ver)
(sml-mode ver: 0.11 upstream: 3.9.5)
(slider ver: 1.15 upstream: 0.3x1)
(sieve ver: 1.17 upstream: No-Upstream-Ver)
(sh-script ver: 1.18 upstream: 2.0e)
(sgml ver: 1.11 upstream: No-Upstream-Ver)
(semantic ver: 1.19 upstream: 1.4.2)
(scheme ver: 1.14 upstream: No-Upstream-Ver)
(sasl ver: 1.15 upstream: 1.14.4)
(ruby-modes ver: 1.02 upstream: 1.6.8)
(rmail ver: 1.14 upstream: No-Upstream-Ver)
(riece ver: 1.17 upstream: 1.0.6)
(reftex ver: 1.34 upstream: 4.21)
(python-modes ver: 1.06 upstream: No-Upstream-Ver)
(psgml-dtds ver: 1.03 upstream: No-Upstream-Ver)
(psgml ver: 1.43 upstream: 1.3.1)
(ps-print ver: 1.11 upstream: 6.5.6)
(prog-modes ver: 2.01 upstream: No-Upstream-Ver)
(pgg ver: 1.05 upstream: 0.1)
(perl-modes ver: 1.06 upstream: No-Upstream-Ver)
(pcomplete ver: 1.03 upstream: 1.1.6)
(pcl-cvs ver: 1.66 upstream: R-2_9_9)
(pc ver: 1.26 upstream: No-Upstream-Ver)
(os-utils ver: 1.35 upstream: No-Upstream-Ver)
(ocaml ver: 0.05 upstream: 3.06)
(net-utils ver: 1.44 upstream: N/A)
(mmm-mode ver: 1.02 upstream: 0.4.7)
(misc-games ver: 1.18 upstream: No-Upstream-Ver)
(mine ver: 1.16 upstream: 1.9)
(mh-e ver: 1.28 upstream: 7.4.2)
(mew ver: 1.18 upstream: 1.94.2)
(mailcrypt ver: 2.14 upstream: 3.5.8)
(mail-lib ver: 1.7 upstream: No-Upstream-Ver)
(liece ver: 1.13 upstream: 1.4.9)
(jde ver: 1.46 upstream: 2.3.2)
(ispell ver: 1.29 upstream: 3.6)
(ilisp ver: 1.33 upstream: 5.12.0)
(igrep ver: 1.13 upstream: 2.111)
(idlwave ver: 1.31 upstream: 5.1)
(ibuffer ver: 1.09 upstream: No-Upstream-Ver)
(hm--html-menus ver: 1.23 upstream: 5.9)
(haskell-mode ver: 1.07 upstream: 1.45)
(gnus ver: 1.82 upstream: 5.10.6)
(gnats ver: 1.16 upstream: 3.101)
(general-docs ver: 1.03 upstream: No-Upstream-Ver)
(games ver: 1.15 upstream: 1.04)
(fsf-compat ver: 1.15 upstream: No-Upstream-Ver)
(frame-icon ver: 1.11 upstream: No-Upstream-Ver)
(fortran-modes ver: 1.04 upstream: No-Upstream-Ver)
(forms ver: 1.15 upstream: 2.37)
(footnote ver: 1.16 upstream: 0.18x)
(eudc ver: 1.39 upstream: 1.32)
(eterm ver: 1.15 upstream: No-Upstream-Ver)
(ess ver: 1.12 upstream: 5.2.3)
(eshell ver: 1.08 upstream: 2.4.1)
(escreen ver: 1.01 upstream: 1.16)
(erc ver: 0.12 upstream: Version 4.0 Revision: 1.675)
(emerge ver: 1.11 upstream: No-Upstream-Ver)
(elib ver: 1.11 upstream: 1.0)
(eieio ver: 1.05 upstream: 0.17)
(efs ver: 1.33 upstream: 1.23)
(edt ver: 1.13 upstream: No-Upstream-Ver)
(edit-utils ver: 2.19 upstream: No-Upstream-Ver)
(ediff ver: 1.5 upstream: 2.75)
(edebug ver: 1.21 upstream: No-Upstream-Ver)
(ecrypto ver: 0.19 upstream: 2.0)
(ecb ver: 1.22 upstream: 2.31)
(docbookide ver: 0.07000000000000001 upstream: 0.1)
(dired ver: 1.16 upstream: 7.12)
(dictionary ver: 1.15 upstream: 1.8)
(debug ver: 1.17 upstream: No-Upstream-Ver)
(crisp ver: 1.14 upstream: 1.34)
(cookie ver: 1.15 upstream: No-Upstream-Ver)
(clearcase ver: 1.09 upstream: /main/laptop/156)
(cc-mode ver: 1.43 upstream: 5.30.9)
(calendar ver: 1.23 upstream: No-Upstream-Ver)
(calc ver: 1.26 upstream: 2.02fX3)
(c-support ver: 1.2 upstream: No-Upstream-Ver)
(build ver: 1.14 upstream: 2.02)
(bbdb ver: 1.25 upstream: 2.34)
(auctex ver: 1.42 upstream: 11.53)
(apel ver: 1.27 upstream: 10.2)
(ada ver: 1.14 upstream: 2.27)
(Sun ver: 1.16 upstream: No-Upstream-Ver)
(mule-ucs ver: 1.06 upstream: 0.84)
(mule-base ver: 1.46 upstream: No-Upstream-Ver)
(locale ver: 1.22 upstream: No-Upstream-Ver)
(leim ver: 1.22 upstream: No-Upstream-Ver)
(latin-unity ver: 1.1 upstream: 1.10)
(egg-its ver: 1.27 upstream: No-Upstream-Ver)
(edict ver: 1.16 upstream: 0.9.9)
Installed Modules:
-----------------
Features:
--------
(mail-abbrevs xemacsbug shadow sendmail rfc822 efs-cu dired-faces
dired-xemacs dired vc-hooks gnuserv cc-mode cc-menus cc-cmds cc-styles
cc-align cc-engine cc-vars cc-defs regexp-opt cus-face un-define
latin-euro-input quail overlay latin-unity-tables latin-unity-latin10
latin-unity-latin9 latin-unity-latin8 latin-unity-latin7
latin-unity-vars latin-unity ispell x-compose zenirc-autoloads
xwem-autoloads edmacro xslt-process-autoloads xslide-autoloads
xlib-autoloads xemacs-devel-autoloads xemacs-base-autoloads
x-symbol-autoloads w3-autoloads vm-autoloads viper-autoloads
view-process-autoloads vhdl-autoloads vc-cc-autoloads vc-autoloads
tramp-autoloads tpu-autoloads tooltalk-autoloads tm-autoloads
time-autoloads textools-autoloads text-modes-autoloads
texinfo-autoloads supercite-autoloads strokes-autoloads
speedbar-autoloads sounds-wav-autoloads sounds-au-autoloads
sml-mode-autoloads slider-autoloads sieve-autoloads
sh-script-autoloads sgml-autoloads semantic-autoloads scheme-autoloads
sasl-autoloads ruby-modes-autoloads rmail-autoloads riece-autoloads
reftex-autoloads python-modes-autoloads psgml-dtds-autoloads
psgml-autoloads ps-print-autoloads prog-modes-autoloads pgg-autoloads
perl-modes-autoloads pcomplete-autoloads pcl-cvs-autoloads
pc-autoloads os-utils-autoloads ocaml-autoloads net-utils-autoloads
mmm-mode-autoloads misc-games-autoloads mine-autoloads mh-e-autoloads
mew-autoloads mailcrypt-autoloads mail-lib-autoloads liece-autoloads
jde-autoloads ispell-autoloads ilisp-autoloads igrep-autoloads
idlwave-autoloads ibuffer-autoloads hm--html-menus-autoloads
haskell-mode-autoloads gnus-autoloads gnats-autoloads
general-docs-autoloads games-autoloads fsf-compat-autoloads
frame-icon-autoloads fortran-modes-autoloads forms-autoloads
footnote-autoloads eudc-autoloads eterm-autoloads ess-autoloads
eshell-autoloads escreen-autoloads erc-autoloads emerge-autoloads
elib-autoloads eieio-autoloads efs-autoloads edt-autoloads
edit-utils-autoloads ediff-autoloads ediff-hook edebug-autoloads
ecrypto-autoloads ecb-autoloads docbookide-autoloads dired-autoloads
dictionary-autoloads debug-autoloads crisp-autoloads cookie-autoloads
clearcase-autoloads cc-mode-autoloads calendar-autoloads
calc-autoloads c-support-autoloads build-autoloads bbdb-autoloads
auctex-autoloads apel-autoloads ada-autoloads Sun-autoloads
mule-ucs-autoloads mule-base-autoloads locale-autoloads leim-autoloads
latin-unity-autoloads egg-its-autoloads edict-autoloads
modules-autoloads mule-autoloads auto-autoloads disp-table loadhist
rsz-minibuf auto-show fontl-hooks x-iso8859-1 canna-leim tibetan lao
devanagari indian slovenian czech romanian ccl code-cmds gutter-items
menubar-items x-menubar mode-motion mouse behavior itimer auto-save
lisp-mode easymenu iso8859-1 page buff-menu lib-complete cus-file
derived newcomment env text-props frame obsolete cus-start custom
widget cl-extra cl cl-19 packages backquote CANNA wnn unicode
lucid-scrollbars cut-buffer lucid-menubars athena-dialogs x
c-balloon-help tty-frames tty toolbar native-sound scrollbar
unix-processes multicast network-streams subprocesses modules
menu-accelerator-support menubar berkeley-db dbm md5 xemacs xim mule
gutter tiff png gif jpeg xface xpm xbm lisp-float-type file-coding
linux dialog devices window-system base64)
Recent keystrokes:
-----------------
f4 t e s TAB RET f4 t BS C-g f12 RET f4 t e s TAB RET
f12 RET button1 button1up f4 t e s TAB RET f12 RET
button1 button1up f4 t e s TAB RET f12 RET f4 t e TAB
RET f12 RET f4 t e s TAB RET f12 RET f4 t e s TAB RET
f12 RET button1 button1up f4 t e s TAB RET f12 RET
button1 button1up f4 t e s TAB RET f12 RET f4 t e s
TAB RET f12 RET f4 t e s TAB RET f12 RET misc-user
Recent messages (most recent first):
-----------------------------------
Parsing /home/aknaff/.mailrc... done
Parsing /home/aknaff/.mailrc...
Loading xemacsbug...done
Loading xemacsbug...
Quit
Notice: Unable to instantiate font for charset chinese-big5-2, face default
Loading efs-cu...done
Loading efs-cu...
Reading directory /home/aknaff/Social/Betisiers/...done
Reading directory /home/aknaff/Social/Betisiers/...
Loading dired...done
Loading dired-mule...done
Loading dired-mule...
Loading dired...
Loading vc-hooks...done
Loading vc-hooks...
Loading gnuserv...done
Loading gnuserv...
Loading cc-mode...done
Loading cc-mode...
6:40 a.m.
>>>> "Alain" == Alain Knaff
<aknaff(a)localhost.localdomain> writes:
Alain> When opening a text file containing the words "m嶯 w嶯
Alain> Europ儇sch Comp彋itivit嶯t" (Luxembourgish for "more than
Alain> European Competitivity"), the text appears garbled in some
Alain> strange Chinese character set (Zh/Big5).
Either set the LANG variable or use `set-language-environment'. If
your environment is LANG=C or LANG=POSIX (the default), XEmacs uses a
rather random set of preferences for guessing encodings.
If your LANG variable _is_ set to something sane (such as
de_DE.ISO8859-15----just an example, sorry, but I don't know how to
spell "Luxembourg Luxembourgish" in ISO 639/3166), please try
killing the buffer with the garbled text, M-x set-language-environment
RET TAB to get a list of known environments, pick one as close to what
you want as possible, and try finding the file again.
Either way, please tell us your LANG (or LC_ALL or LC_CTYPE if you're
a locale guru) values and the language environment so we can improve
this for future versions.
If your LANG variable is not set or is set to some other language/
encoding than the one normal for Luxembourgish, and setting it
properly and restarting XEmacs doesn't fix things, please let us know
what you set it to, and what the inappropriate result is.
--
School of Systems and Information Engineering http://turnbull.sk.tsukuba.ac.jp
University of Tsukuba Tennodai 1-1-1 Tsukuba 305-8573 JAPAN
Ask not how you can "do" free software business;
ask what your business can "do for" free software.
9:14 p.m.
Hi Alain,
I had the same problem and believe I found the correct cause and an
acceptable solution -- at least what I describe below works for me
since a long time.
I assume your file is in some encoding like ISO-8859-1? (If it is
UTF-8, I can't explain this behaviour.)
You may already have noticed that the problem goes away if the file
does not contain consecutive occurrences of characters outside ASCII,
that is, no words like "Europäësch"?
First the solution: In the file "src/mule-coding.c", near the end
of the function "iso2022_detect", replace the clause
else if (data->odd_high_byte_groups > 0 &&
data->even_high_byte_groups > 0)
SET_DET_RESULTS (st, iso2022, DET_SOMEWHAT_UNLIKELY);
with
else if (data->odd_high_byte_groups > 0 &&
data->even_high_byte_groups > 0)
{
SET_DET_RESULTS (st, iso2022, DET_SOMEWHAT_UNLIKELY);
if (data->odd_high_byte_groups >= 2 * data->even_high_byte_groups)
DET_RESULT (st, iso_8_1) = DET_SOMEWHAT_LIKELY;
}
and recompile XEmacs.
Second, the explanation: This part of XEmacs is responsible for a kind
of statistical recognition of file encodings. To do so among other things
it counts the number of occurrences of runs of even and odd lengths of
bytes >= 0xA0 in the first few KB of the file.
If the file is encoded in something like ISO-8859-1 most such runs will
be one byte long (e.g. "Compétitivitéit" contains two such runs), but
sometimes there will be some of length two, too, e.g. in "Europäësch".
If both occur, the clause above will trigger and erroneously reduce the
likelihood of encodings like ISO-8859-1 below the default level, leaving
Zh/Big5 as the most likely encoding.
The changed version above simply checks that the relative frequency
of occurrence of runs of even and odd lengths is compatible with the
assumption that the language/encoding in question uses bytes >= 0xA0
mostly singly, but sometimes in pairs, and increases the likelihood of
encodings like ISO-8859-1 in this case.
Hope this helps
Yours
Lutz Euler
6892
days inactive
6893
days old
2 comments
3 participants
participants (3)
-
Alain Knaff -
Lutz Euler -
Stephen J. Turnbull