[Bug: 21.4.22] Strange character conversion problem (UTF8)

================================================================ Dear Bug Team, the following strange bug has occurred on my system: 1) Observation: It appears that large text files are divided into blocks of 2^16=65536 bytes, when they are saved on a disk. 2) Fact: Storing some UTF8 characters (e.g. umlauts) requires two bytes. 3) Bug: If the first byte of such a character is the last byte of of a block (in other words, if a two byte character is stored in two different blocks of size 65536), then this character is replaced by EF BF BD = replacement character (3 bytes), so that the text file contains a corrupted character. Any help would be greatly appreciated. Markus ================================================================ System Info to help track down your bug: --------------------------------------- uname -a: Linux lgw01-11 3.19.0-42-generic #48~14.04.1-Ubuntu SMP Fri Dec 18 10:24:49 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux ./configure '--with-sound=none,native' '--with-x11' '--extra-verbose' '--with-site-lisp' '--with-database=berkdb' '--statedir=/var/lib' '--infodir=/usr/share/info/xemacs21' '--infopath=/usr/share/info/xemacs21' '--prefix=/usr' '--dynamic' '--error-checking=none' '--debug=no' '--with-pop' '--with-file-coding' '--pdump' '--with-system-malloc' '--with_menubars=lucid' '--with_scrollbars=lucid' '--with_dialogs=athena' '--docdir=/usr/lib/xemacs-21.4.22/x86_64-linux-gnu/mule/' '--with-mule' '--with-canna=no' '--with-wnn=no' '--with-xim=xlib' '--with-xfs' '--package-path=~/.xemacs:~/.xemacs/packages:~/.xemacs/xemacs-packages::/usr/share/xemacs21/site-packages:/usr/share/xemacs21/mule-packages:/usr/share/xemacs21/xemacs-packages' 'x86_64-linux-gnu' XEmacs 21.4.22 "Instant Classic" configured for `x86_64-pc-linux'. Compilation / Installation: Source code location: /build/xemacs21-W0b37d/xemacs21-21.4.22 Installation prefix: /usr Operating system description file: `s/linux.h' Not using any machine description file Compiler: gcc -g -O3 -Wall -Wno-switch -Winline -Wmissing-prototypes -Wsign-compare -fgnu89-inline -fno-strict-aliasing -Wshadow Compiler version: gcc (Ubuntu 5.3.1-4ubuntu1) 5.3.1 20151219 Compiler specs file: specs. Relocating allocator for buffers: no GNU version of malloc: no - User chose not to use GNU allocators. Need to guess glibc1/2/etc here Window System: Compiling in support for the X window system: - X Windows headers location: - X Windows libraries location: - Handling WM_COMMAND properly. Compiling in support for the Athena widget set: - Athena headers location: X11/Xaw - Athena library to link: Xaw Using Lucid menubars. Using Lucid scrollbars. Using Athena dialog boxes. Using Athena native widgets. TTY: Compiling in support for ncurses. Compiling in support for GPM (General Purpose Mouse). Images: Compiling in support for GIF images (builtin). Compiling in support for XPM images. Compiling in support for PNG images. Compiling in support for JPEG images. Compiling in support for TIFF images. Compiling in support for X-Face message headers. Sound: Compiling in support for sound (native). Databases: Compiling in support for Berkeley database. Compiling in support for LDAP. Internationalization: Compiling in support for Mule (multi-lingual Emacs). Compiling in support for file coding. Compiling in support for XIM (X11R5+ I18N input method). - Using raw Xlib to provide XIM support. - Using XFontSet to provide bilingual menubar. Mail: Compiling in support for POP mail retrieval. Compiling in support for "dot-locking" mail spool file locking method. Other Features: Inhibiting IPv6 canonicalization at startup. Compiling in support for dynamic shared object modules. Using the new portable dumper. Load-Path Lisp Shadows: ---------------------- (/usr/share/xemacs21/site-lisp/a2ps/a2ps /usr/share/xemacs21/site-lisp/a2ps/a2ps /usr/share/xemacs21/site-lisp/a2ps/a2ps-print /usr/share/xemacs21/site-lisp/a2ps/a2ps-print /usr/share/xemacs21/site-lisp/cmake-data/cmake-mode /usr/share/xemacs21/site-lisp/cmake-data/cmake-mode /usr/share/xemacs21/site-lisp/dictionaries-common/flyspell /usr/share/xemacs21/site-lisp/dictionaries-common/flyspell /usr/share/xemacs21/site-lisp/dictionaries-common/debian-ispell /usr/share/xemacs21/site-lisp/dictionaries-common/debian-ispell /usr/share/xemacs21/site-lisp/dictionaries-common/ispell /usr/share/xemacs21/site-lisp/dictionaries-common/ispell /usr/share/xemacs21/site-lisp/git/git /usr/share/xemacs21/site-lisp/git/git /usr/share/xemacs21/site-lisp/git/git-blame /usr/share/xemacs21/site-lisp/git/git-blame /usr/share/xemacs21/site-lisp/latex-cjk-common/cjk-enc /usr/share/xemacs21/site-lisp/latex-cjk-common/cjk-enc /usr/share/xemacs21/site-lisp/latex-cjk-common/cjkspace /usr/share/xemacs21/site-lisp/latex-cjk-common/cjkspace /usr/share/xemacs21/site-lisp/latex-cjk-common/cjktilde /usr/share/xemacs21/site-lisp/latex-cjk-common/cjktilde /usr/share/xemacs21/site-lisp/latex-cjk-thai/thai-word /usr/share/xemacs21/site-lisp/latex-cjk-thai/thai-word /etc/emacs/site-start /usr/share/xemacs21/site-packages/lisp/site-start /usr/share/xemacs21/site-lisp/dictionaries-common/ispell /usr/share/xemacs21/xemacs-packages/lisp/ispell/ispell /usr/share/xemacs21/xemacs-packages/lisp/cedet-common/working /usr/share/xemacs21/xemacs-packages/lisp/semantic/working /usr/share/xemacs21/xemacs-packages/lisp/cedet-common/sformat /usr/share/xemacs21/xemacs-packages/lisp/semantic/sformat /usr/share/xemacs21/site-lisp/dictionaries-common/flyspell /usr/share/xemacs21/xemacs-packages/lisp/text-modes/flyspell /usr/share/xemacs21/xemacs-packages/lisp/xemacs-base/easy-mmode /usr/lib/xemacs-21.4.22/lisp/easy-mmode ~/.emacsdir/custom /usr/lib/xemacs-21.4.22/lisp/custom /usr/share/xemacs21/xemacs-packages/lisp/build/build-report /usr/lib/xemacs-21.4.22/lisp/build-report /usr/share/xemacs21/site-lisp/cmake-data/cmake-mode /usr/share/emacs/site-lisp/cmake-mode /usr/share/xemacs21/site-lisp/debian-startup /usr/share/emacs/site-lisp/debian-startup) Internationalization Settings: ------------------------- Environment: Value of LC_ALL : nil Value of LC_COLLATE : nil Value of LC_CTYPE : nil Value of LC_MESSAGES : nil Value of LC_MONETARY : de_DE.UTF-8 Value of LC_NUMERIC : de_DE.UTF-8 Value of LC_TIME : de_DE.UTF-8 Value of LANG : en_US.UTF-8 Lisp locale settings: current-language-environment => "English" default-buffer-file-coding-system => iso-2022-8 default-process-coding-system => [not available] (current-locale) => [not available] keyboard-coding-system => nil terminal-coding-system => nil (coding-priority-list) => (utf-8 iso-7 no-conversion iso-8-1 iso-8-2 iso-8-designate iso-lock-shift shift-jis big5 ucs-4) Coding system aliases: 'native is not a coding system alias 'file-name is aliased to binary 'mswindows-multibyte-system-default is not a coding system alias Installed XEmacs Packages: ------------------------- (zenirc ver: 1.16 upstream: 2.112) (xwem ver: 1.22 upstream: lg(a)xwem.org--2005/xwem--main--2.1--versionfix-1) (xslt-process ver: 1.12 upstream: 1.2.1) (xslide ver: 1.09 upstream: 0.2.2) (xlib ver: 1.14 upstream: lg(a)xwem.org--2005/xlib--main--2.1--version-0) (xetla ver: 1.02 upstream: steve(a)eicq.org--2005/xetla--main--1.1--version-0) (xemacs-devel ver: 1.79 upstream: No-Upstream-Ver) (xemacs-base ver: 2.19 upstream: No-Upstream-Ver) (x-symbol ver: 1.11 upstream: 4.5.1) (w3 ver: 1.35 upstream: 4.0pre47) (vm ver: 8.07 upstream: 8.0.12-devo) (viper ver: 1.65 upstream: 3.09) (view-process ver: 1.13 upstream: 2.4) (vhdl ver: 1.22 upstream: 3.33.12) (vc-cc ver: 1.22 upstream: No-Upstream-Ver) (vc ver: 1.45 upstream: No-Upstream-Ver) (tramp ver: 1.4 upstream: 2.0.56) (tpu ver: 1.14 upstream: 4.2X) (tooltalk ver: 1.15 upstream: No-Upstream-Ver) (tm ver: 1.39 upstream: No-Upstream-Ver) (time ver: 1.14 upstream: 1.17) (textools ver: 1.15 upstream: No-Upstream-Ver) (text-modes ver: 1.96 upstream: No-Upstream-Ver) (texinfo ver: 1.3 upstream: No-Upstream-Ver) (supercite ver: 1.21 upstream: 3.55x3) (strokes ver: 1.1 upstream: No-Upstream-Ver) (speedbar ver: 1.29 upstream: 1.0pre4) (sounds-wav ver: 1.12 upstream: No-Upstream-Ver) (sounds-au ver: 1.12 upstream: No-Upstream-Ver) (sml-mode ver: 0.12 upstream: 3.9.5) (slider ver: 1.16 upstream: 0.3x1) (sieve ver: 1.18 upstream: No-Upstream-Ver) (sh-script ver: 1.24 upstream: 2.0f) (sgml ver: 1.11 upstream: No-Upstream-Ver) (semantic ver: 1.21 upstream: 2.0pre4) (scheme ver: 1.17 upstream: No-Upstream-Ver) (sasl ver: 1.16 upstream: 1.14.4) (ruby-modes ver: 1.02 upstream: 1.6.8) (rmail ver: 1.14 upstream: No-Upstream-Ver) (riece ver: 1.23 upstream: 3.1.2) (reftex ver: 1.34 upstream: 4.21) (re-builder ver: 1.05 upstream: 1.20) (python-modes ver: 1.13 upstream: 5.0.0) (psgml-dtds ver: 1.03 upstream: No-Upstream-Ver) (psgml ver: 1.45 upstream: 1.3.2) (ps-print ver: 1.12 upstream: 6.5.6) (prog-modes ver: 2.19 upstream: No-Upstream-Ver) (pgg ver: 1.07 upstream: 0.1) (perl-modes ver: 1.14 upstream: No-Upstream-Ver) (pcomplete ver: 1.05 upstream: 1.1.6) (pcl-cvs ver: 1.68 upstream: R-2_9_9) (pc ver: 1.28 upstream: No-Upstream-Ver) (os-utils ver: 1.41 upstream: No-Upstream-Ver) (oo-browser ver: 1.05 upstream: 4.08) (ocaml ver: 0.06 upstream: 3.06) (net-utils ver: 1.56 upstream: N/A) (mmm-mode ver: 1.03 upstream: 0.4.8) (misc-games ver: 1.22 upstream: No-Upstream-Ver) (mine ver: 1.16 upstream: 1.9) (mh-e ver: 1.31 upstream: 7.4.2) (mew ver: 1.19 upstream: 1.94.2) (mailcrypt ver: 2.14 upstream: 3.5.8) (mail-lib ver: 1.8 upstream: No-Upstream-Ver) (jde ver: 1.52 upstream: 2.3.5.1) (ispell ver: 1.32 upstream: 3.6) (ilisp ver: 1.34 upstream: 5.12.0) (igrep ver: 1.16 upstream: 2.111) (idlwave ver: 1.32 upstream: 5.1) (ibuffer ver: 1.09 upstream: No-Upstream-Ver) (hyperbole ver: 1.17 upstream: 5.0) (hm--html-menus ver: 1.24 upstream: 5.9) (haskell-mode ver: 1.11 upstream: 2.1) (guided-tour ver: 0.52 upstream: No-Upstream-Ver) (gnus ver: 1.93 upstream: 5.10.8) (gnats ver: 1.17 upstream: 3.101) (general-docs ver: 1.05 upstream: No-Upstream-Ver) (games ver: 1.18 upstream: 1.04) (fsf-compat ver: 1.17 upstream: No-Upstream-Ver) (frame-icon ver: 1.11 upstream: No-Upstream-Ver) (fortran-modes ver: 1.05 upstream: No-Upstream-Ver) (forms ver: 1.15 upstream: 2.37) (footnote ver: 1.16 upstream: 0.18x) (eudc ver: 1.4 upstream: 1.32) (eterm ver: 1.17 upstream: No-Upstream-Ver) (eshell ver: 1.17 upstream: 0fc80a3f6bb3bb59f42e9ff83cc8b89bf90fe658) (escreen ver: 1.01 upstream: 1.16) (erc ver: 0.22 upstream: Version 5.1.2 Revision: 1.796.2.6) (emerge ver: 1.11 upstream: No-Upstream-Ver) (elib ver: 1.13 upstream: 1.0) (eieio ver: 1.06 upstream: 1.0pre4) (efs ver: 1.34 upstream: 1.24) (edt ver: 1.14 upstream: No-Upstream-Ver) (edit-utils ver: 2.4 upstream: No-Upstream-Ver) (ediff ver: 1.77 upstream: 2.75) (edebug ver: 1.22 upstream: No-Upstream-Ver) (ede ver: 1.01 upstream: 1.0pre4) (ecrypto ver: 0.21 upstream: 2.0) (ecb ver: 1.22 upstream: 2.31) (easypg ver: 1.02 upstream: 0.0.16) (docbookide ver: 0.08 upstream: 0.1) (dired ver: 1.19 upstream: 7.16) (dictionary ver: 1.16 upstream: 1.8) (debug ver: 1.18 upstream: No-Upstream-Ver) (crisp ver: 1.15 upstream: 1.34) (cookie ver: 1.15 upstream: No-Upstream-Ver) (cogre ver: 1.02 upstream: 0.5) (clearcase ver: 1.1 upstream: /main/laptop/165) (cedet-common ver: 1.01 upstream: 1.0pre4) (cc-mode ver: 1.45 upstream: 5.30.10) (calendar ver: 1.38 upstream: No-Upstream-Ver) (calc ver: 1.26 upstream: 2.02fX3) (c-support ver: 1.22 upstream: No-Upstream-Ver) (build ver: 1.14 upstream: 2.02) (bbdb ver: 1.32 upstream: 2.35) (auctex ver: 1.51 upstream: 11.84) (apel ver: 1.33 upstream: 10.6) (ada ver: 1.14 upstream: 2.27) (Sun ver: 1.16 upstream: No-Upstream-Ver) (skk ver: 1.24 upstream: 10.62a) (mule-ucs ver: 1.17 upstream: 0.84) (mule-base ver: 1.53 upstream: No-Upstream-Ver) (lookup ver: 1.15 upstream: 1.0) (locale ver: 1.28 upstream: No-Upstream-Ver) (leim ver: 1.28 upstream: No-Upstream-Ver) (latin-unity ver: 1.2 upstream: 1.20) (latin-euro-standards ver: 1.08 upstream: 1.08) (egg-its ver: 1.27 upstream: No-Upstream-Ver) (edict ver: 1.16 upstream: 0.9.9) Installed Modules: ----------------- Features: -------- (mail-abbrevs xemacsbug shadow sendmail rfc822 preview env prv-xemacs byte-optimize overlay tex-bar tex-buf toolbar-x outline font-latex bytecomp byte-compile latex easy-mmode edmacro tex-style tex advice advice-preload regexp-opt efs-cu guided-tour un-define latin-euro-standards latin-euro-latin10 latin-euro-latin9 latin-euro-latin8 latin-euro-latin7 mule-uni font disp-table font-lock cus-face paren blink-paren pending-del executable mwheel zenirc-autoloads xwem-autoloads xslt-process-autoloads xslide-autoloads xlib-autoloads xetla-autoloads xemacs-devel-autoloads xemacs-base-autoloads x-symbol-autoloads w3-autoloads vm-autoloads viper-autoloads view-process-autoloads vhdl-autoloads vc-cc-autoloads vc-autoloads tramp-autoloads tpu-autoloads tooltalk-autoloads tm-autoloads time-autoloads textools-autoloads text-modes-autoloads texinfo-autoloads supercite-autoloads strokes-autoloads speedbar-autoloads sounds-wav-autoloads sounds-au-autoloads sml-mode-autoloads slider-autoloads sieve-autoloads sh-script-autoloads sgml-autoloads semantic-autoloads scheme-autoloads sasl-autoloads ruby-modes-autoloads rmail-autoloads riece-autoloads reftex-autoloads re-builder-autoloads python-modes-autoloads psgml-dtds-autoloads psgml-autoloads ps-print-autoloads prog-modes-autoloads pgg-autoloads perl-modes-autoloads pcomplete-autoloads pcl-cvs-autoloads pc-autoloads os-utils-autoloads oo-browser-autoloads ocaml-autoloads net-utils-autoloads mmm-mode-autoloads misc-games-autoloads mine-autoloads mh-e-autoloads mew-autoloads mailcrypt-autoloads mail-lib-autoloads jde-autoloads ispell-autoloads ilisp-autoloads igrep-autoloads idlwave-autoloads ibuffer-autoloads hyperbole-autoloads hm--html-menus-autoloads haskell-mode-autoloads guided-tour-autoloads gnus-autoloads gnats-autoloads general-docs-autoloads games-autoloads fsf-compat-autoloads frame-icon-autoloads fortran-modes-autoloads forms-autoloads footnote-autoloads eudc-autoloads eterm-autoloads eshell-autoloads escreen-autoloads erc-autoloads emerge-autoloads elib-autoloads eieio-autoloads efs-autoloads edt-autoloads edit-utils-autoloads ediff-autoloads edebug-autoloads ede-autoloads ecrypto-autoloads ecb-autoloads easypg-autoloads docbookide-autoloads dired-autoloads dictionary-autoloads debug-autoloads crisp-autoloads cookie-autoloads cogre-autoloads clearcase-autoloads cedet-common-autoloads cc-mode-autoloads calendar-autoloads calc-autoloads c-support-autoloads build-autoloads bbdb-autoloads tex-site auctex-autoloads apel-autoloads ada-autoloads Sun-autoloads skk-autoloads mule-ucs-autoloads mule-base-autoloads lookup-autoloads locale-autoloads leim-autoloads latin-unity-autoloads latin-euro-standards-autoloads egg-its-autoloads edict-autoloads lisp-autoloads loadhist auto-show fontl-hooks x-iso8859-1 slovenian czech romanian ccl mule-help code-cmds gutter-items menubar-items x-menubar mode-motion mouse itimer auto-save lisp-mode easymenu iso8859-1 page buff-menu lib-complete cus-file derived frame text-props obsolete cus-start custom widget cl-extra mini-cl cl cl-19 packages backquote very-early-lisp file-coding mule lucid-scrollbars cut-buffer lucid-menubars athena-dialogs x c-balloon-help tty-frames tty toolbar native-sound scrollbar unix-processes multicast network-streams subprocesses modules menu-accelerator-support menubar berkeley-db md5 xemacs xim gutter tiff png gif jpeg xface xpm xbm lisp-float-type linux dialog devices window-system base64) Recent keystrokes: ----------------- misc-user Recent messages (most recent first): ----------------------------------- Loading xemacsbug...done Loading xemacsbug... Applying style hooks... done Applying style hooks... Loading preview...done Loading env...done Loading env... Loading byte-optimize...done Loading byte-optimize... Loading preview... Loading tex-bar...done Loading tex-bar... Loading font-latex...done Loading bytecomp...done Loading bytecomp... Loading font-latex... Loading latex...done Loading easy-mmode...done Loading easy-mmode... Loading edmacro...done