[Bug: 21.4.22] weird file corruption with utf8 charset

Hi XEmacs Team! I may have found a weird bug. If you have a sufficiently large utf8 text file with accented characters, some of them will be corrupted silently upon saving. Run "xemacs -q". Eval the following lisp code: (require 'un-define) C-x f testfile.txt M-x set-buffer-file-coding-system RET utf-8 RET M-x set-input-method RET latin-1-postfix RET Type some characters like a'e'u'i' (the input method will convert these to accented chars). Now you have somme accented characters on one line. Now multiply this line till your testfile is larger than 1 MiB. Save it (C-x s !). Open the file with midnight commander viewer (F3). Search (F7) for the hexa value EF (tick the hexadecimal checkbox in the search dialog). If you find one, that's a bug, it should not be there, inspect the file at that point, you probably see some garbage. Or you can use the command hd testfile | grep 'ef ' Here is an example file I created that contains corrupted sequences, 1.4 MiB: https://www.dropbox.com/s/2kprme5e4m3b2cc/report-bigfile.txt?dl=0 System Info to help track down your bug: --------------------------------------- uname -a: Linux biber 3.2.0-4-amd64 #1 SMP Debian 3.2.54-2 i686 GNU/Linux ./configure '--with-sound=none,native' '--with-x11' '--extra-verbose' '--with-site-lisp' '--with-database=berkdb' '--statedir=/var/lib' '--infodir=/usr/share/info/xemacs21' '--infopath=/usr/share/info/xemacs21' '--prefix=/usr' '--dynamic' '--error-checking=none' '--debug=no' '--with-pop' '--with-file-coding' '--pdump' '--with-system-malloc' '--with_menubars=lucid' '--with_scrollbars=lucid' '--with_dialogs=athena' '--docdir=/usr/lib/xemacs-21.4.22/i486-linux-gnu/mule/' '--with-mule' '--with-canna=no' '--with-wnn=no' '--with-xim=xlib' '--with-xfs' '--package-path=~/.xemacs:~/.xemacs/packages:~/.xemacs/xemacs-packages::/usr/share/xemacs21/site-packages:/usr/share/xemacs21/mule-packages:/usr/share/xemacs21/xemacs-packages' 'i486-linux-gnu' XEmacs 21.4.22 "Instant Classic" configured for `i486-pc-linux'. Compilation / Installation: Source code location: /build/xemacs21-eaml6F/xemacs21-21.4.22 Installation prefix: /usr Operating system description file: `s/linux.h' Machine description file: `m/intel386.h' Compiler: gcc -g -O3 -Wall -Wno-switch -Winline -Wmissing-prototypes -Wsign-compare -fno-strict-aliasing -Wshadow Compiler version: gcc (Debian 4.8.2-16) 4.8.2 Compiler specs file: specs. Relocating allocator for buffers: no GNU version of malloc: no - User chose not to use GNU allocators. Need to guess glibc1/2/etc here Window System: Compiling in support for the X window system: - X Windows headers location: - X Windows libraries location: - Handling WM_COMMAND properly. Compiling in support for the Athena widget set: - Athena headers location: X11/Xaw - Athena library to link: Xaw Using Lucid menubars. Using Lucid scrollbars. Using Athena dialog boxes. Using Athena native widgets. TTY: Compiling in support for ncurses. Compiling in support for GPM (General Purpose Mouse). Images: Compiling in support for GIF images (builtin). Compiling in support for XPM images. Compiling in support for PNG images. Compiling in support for JPEG images. Compiling in support for TIFF images. Compiling in support for X-Face message headers. Sound: Compiling in support for sound (native). Databases: Compiling in support for Berkeley database. Compiling in support for LDAP. Internationalization: Compiling in support for Mule (multi-lingual Emacs). Compiling in support for file coding. Compiling in support for XIM (X11R5+ I18N input method). - Using raw Xlib to provide XIM support. - Using XFontSet to provide bilingual menubar. Mail: Compiling in support for POP mail retrieval. Compiling in support for "dot-locking" mail spool file locking method. Other Features: Inhibiting IPv6 canonicalization at startup. Compiling in support for dynamic shared object modules. Using the new portable dumper. Internationalization Settings: ------------------------- Environment: Value of LC_ALL : nil Value of LC_COLLATE : nil Value of LC_CTYPE : en_US.UTF-8 Value of LC_MESSAGES : nil Value of LC_MONETARY : nil Value of LC_NUMERIC : nil Value of LC_TIME : nil Value of LANG : C Lisp locale settings: current-language-environment => "latin-2" default-buffer-file-coding-system => #<coding_system iso-8859-2> default-process-coding-system => [not available] (current-locale) => [not available] keyboard-coding-system => #<coding_system iso-8859-2> terminal-coding-system => #<coding_system iso-8859-2> (coding-priority-list) => (utf-8 iso-8-2 iso-8-1 no-conversion iso-7 iso-8-designate iso-lock-shift shift-jis big5 ucs-4) Coding system aliases: 'native is not a coding system alias 'file-name is aliased to iso-8859-1 'mswindows-multibyte-system-default is not a coding system alias Installed XEmacs Packages: ------------------------- (zenirc ver: 1.16 upstream: 2.112) (xwem ver: 1.22 upstream: lg(a)xwem.org--2005/xwem--main--2.1--versionfix-1) (xslt-process ver: 1.12 upstream: 1.2.1) (xslide ver: 1.09 upstream: 0.2.2) (xlib ver: 1.14 upstream: lg(a)xwem.org--2005/xlib--main--2.1--version-0) (xetla ver: 1.02 upstream: steve(a)eicq.org--2005/xetla--main--1.1--version-0) (xemacs-devel ver: 1.79 upstream: No-Upstream-Ver) (xemacs-base ver: 2.19 upstream: No-Upstream-Ver) (x-symbol ver: 1.11 upstream: 4.5.1) (w3 ver: 1.35 upstream: 4.0pre47) (vm ver: 8.07 upstream: 8.0.12-devo) (viper ver: 1.65 upstream: 3.09) (view-process ver: 1.13 upstream: 2.4) (vhdl ver: 1.22 upstream: 3.33.12) (vc-cc ver: 1.22 upstream: No-Upstream-Ver) (vc ver: 1.45 upstream: No-Upstream-Ver) (tramp ver: 1.4 upstream: 2.0.56) (tpu ver: 1.14 upstream: 4.2X) (tooltalk ver: 1.15 upstream: No-Upstream-Ver) (tm ver: 1.39 upstream: No-Upstream-Ver) (time ver: 1.14 upstream: 1.17) (textools ver: 1.15 upstream: No-Upstream-Ver) (text-modes ver: 1.96 upstream: No-Upstream-Ver) (texinfo ver: 1.3 upstream: No-Upstream-Ver) (supercite ver: 1.21 upstream: 3.55x3) (strokes ver: 1.1 upstream: No-Upstream-Ver) (speedbar ver: 1.29 upstream: 1.0pre4) (sounds-wav ver: 1.12 upstream: No-Upstream-Ver) (sounds-au ver: 1.12 upstream: No-Upstream-Ver) (sml-mode ver: 0.12 upstream: 3.9.5) (slider ver: 1.16 upstream: 0.3x1) (sieve ver: 1.18 upstream: No-Upstream-Ver) (sh-script ver: 1.24 upstream: 2.0f) (sgml ver: 1.11 upstream: No-Upstream-Ver) (semantic ver: 1.21 upstream: 2.0pre4) (scheme ver: 1.17 upstream: No-Upstream-Ver) (sasl ver: 1.16 upstream: 1.14.4) (ruby-modes ver: 1.02 upstream: 1.6.8) (rmail ver: 1.14 upstream: No-Upstream-Ver) (riece ver: 1.23 upstream: 3.1.2) (reftex ver: 1.34 upstream: 4.21) (re-builder ver: 1.05 upstream: 1.20) (python-modes ver: 1.13 upstream: 5.0.0) (psgml-dtds ver: 1.03 upstream: No-Upstream-Ver) (psgml ver: 1.45 upstream: 1.3.2) (ps-print ver: 1.12 upstream: 6.5.6) (prog-modes ver: 2.19 upstream: No-Upstream-Ver) (pgg ver: 1.07 upstream: 0.1) (perl-modes ver: 1.14 upstream: No-Upstream-Ver) (pcomplete ver: 1.05 upstream: 1.1.6) (pcl-cvs ver: 1.68 upstream: R-2_9_9) (pc ver: 1.28 upstream: No-Upstream-Ver) (os-utils ver: 1.41 upstream: No-Upstream-Ver) (oo-browser ver: 1.05 upstream: 4.08) (ocaml ver: 0.06 upstream: 3.06) (net-utils ver: 1.56 upstream: N/A) (mmm-mode ver: 1.03 upstream: 0.4.8) (misc-games ver: 1.22 upstream: No-Upstream-Ver) (mine ver: 1.16 upstream: 1.9) (mh-e ver: 1.31 upstream: 7.4.2) (mew ver: 1.19 upstream: 1.94.2) (mailcrypt ver: 2.14 upstream: 3.5.8) (mail-lib ver: 1.8 upstream: No-Upstream-Ver) (jde ver: 1.52 upstream: 2.3.5.1) (ispell ver: 1.32 upstream: 3.6) (ilisp ver: 1.34 upstream: 5.12.0) (igrep ver: 1.16 upstream: 2.111) (idlwave ver: 1.32 upstream: 5.1) (ibuffer ver: 1.09 upstream: No-Upstream-Ver) (hyperbole ver: 1.17 upstream: 5.0) (hm--html-menus ver: 1.24 upstream: 5.9) (haskell-mode ver: 1.11 upstream: 2.1) (guided-tour ver: 0.52 upstream: No-Upstream-Ver) (gnus ver: 1.93 upstream: 5.10.8) (gnats ver: 1.17 upstream: 3.101) (general-docs ver: 1.05 upstream: No-Upstream-Ver) (games ver: 1.18 upstream: 1.04) (fsf-compat ver: 1.17 upstream: No-Upstream-Ver) (frame-icon ver: 1.11 upstream: No-Upstream-Ver) (fortran-modes ver: 1.05 upstream: No-Upstream-Ver) (forms ver: 1.15 upstream: 2.37) (footnote ver: 1.16 upstream: 0.18x) (eudc ver: 1.4 upstream: 1.32) (eterm ver: 1.17 upstream: No-Upstream-Ver) (eshell ver: 1.17 upstream: 0fc80a3f6bb3bb59f42e9ff83cc8b89bf90fe658) (escreen ver: 1.01 upstream: 1.16) (erc ver: 0.22 upstream: Version 5.1.2 Revision: 1.796.2.6) (emerge ver: 1.11 upstream: No-Upstream-Ver) (elib ver: 1.13 upstream: 1.0) (eieio ver: 1.06 upstream: 1.0pre4) (efs ver: 1.34 upstream: 1.24) (edt ver: 1.14 upstream: No-Upstream-Ver) (edit-utils ver: 2.4 upstream: No-Upstream-Ver) (ediff ver: 1.77 upstream: 2.75) (edebug ver: 1.22 upstream: No-Upstream-Ver) (ede ver: 1.01 upstream: 1.0pre4) (ecrypto ver: 0.21 upstream: 2.0) (ecb ver: 1.22 upstream: 2.31) (easypg ver: 1.02 upstream: 0.0.16) (docbookide ver: 0.08 upstream: 0.1) (dired ver: 1.19 upstream: 7.16) (dictionary ver: 1.16 upstream: 1.8) (debug ver: 1.18 upstream: No-Upstream-Ver) (crisp ver: 1.15 upstream: 1.34) (cookie ver: 1.15 upstream: No-Upstream-Ver) (cogre ver: 1.02 upstream: 0.5) (clearcase ver: 1.1 upstream: /main/laptop/165) (cedet-common ver: 1.01 upstream: 1.0pre4) (cc-mode ver: 1.45 upstream: 5.30.10) (calendar ver: 1.38 upstream: No-Upstream-Ver) (calc ver: 1.26 upstream: 2.02fX3) (c-support ver: 1.22 upstream: No-Upstream-Ver) (build ver: 1.14 upstream: 2.02) (bbdb ver: 1.32 upstream: 2.35) (auctex ver: 1.51 upstream: 11.84) (apel ver: 1.33 upstream: 10.6) (ada ver: 1.14 upstream: 2.27) (Sun ver: 1.16 upstream: No-Upstream-Ver) (skk ver: 1.24 upstream: 10.62a) (mule-ucs ver: 1.17 upstream: 0.84) (mule-base ver: 1.53 upstream: No-Upstream-Ver) (lookup ver: 1.15 upstream: 1.0) (locale ver: 1.28 upstream: No-Upstream-Ver) (leim ver: 1.28 upstream: No-Upstream-Ver) (latin-unity ver: 1.2 upstream: 1.20) (latin-euro-standards ver: 1.08 upstream: 1.08) (egg-its ver: 1.27 upstream: No-Upstream-Ver) (edict ver: 1.16 upstream: 0.9.9) Installed Modules: ----------------- Features: -------- (xemacsbug shadow tabify org-clock cal-iso org-table nnml nnfolder utf7 gnus-encrypt starttls gnus-demon guided-tour blink-cursor icomplete minibuf-electric permanent-buffers rsz-minibuf org-toc per-window-point tramp-vc vc lorem-ipsum gitsum json minibuffer-complete-cycle tramp-smb tramp trampver darcsum add-log ediff-merg ediff-diff ediff-wind ediff-tbar ediff-help ediff-util ediff-mult ediff-init ediff vc-git semantic-edit semantic-html semantic-format semantic-util-modes semantic-util semantic inversion semantic-lex semantic-tag semantic-fw mode-local find-func working fame sb-html sb-texinfo speedbar sb-image ezimage dframe assoc org-wl org-w3m org-vm org-rmail org-mhe org-mew org-irc org-jsinfo org-infojs org-html org-info org-gnus org-docview org-bibtex org-bbdb xrdb-mode apt-sources sh-script vc-rcs gnuserv scroll-in-place quail perlplus cperl-mode redshank slime-indentation slime-asdf slime-fancy slime-fontifying-fu slime-package-fu slime-mdot-fu slime-references slime-scratch slime-presentations slime-highlight-edits slime-fuzzy slime-fancy-inspector slime-c-p-c slime-editing-commands slime-autodoc slime-enclosing-context slime-parse slime-repl slime hyperspec newcomment paredit xref xrefactory hideshow guess-offset align ispell imdb cyclebuffer w3m-form ibuf-macs w3m-ucs supercite regi passwd rk-nym erc-track erc-nickserv erc-button erc-pcomplete erc-complete hippie-exp erc-match erc-fill erc-stamp erc erc-compat randomsig gnus-topic nnir nnimap imap vc-hooks vc-xemacs novice gnus-delay gnus-draft gnus-agent gnus-srvr gnus-score score-mode nnvirtual nntp gnus-cache nndraft nnmh gnus-fun sa-learn-invoke bbdb-pgp mailcrypt bbdb-gnus bbdb-snarf mail-extr bbdb-com bbdb em-alias em-banner em-basic em-cmpl em-dirs em-glob em-pred em-hist em-ls em-prompt em-script term ehelp electric em-term esh-opt em-unix esh-toggle env esh-var esh-io esh-ext esh-proc esh-arg esh-cmd esh-module esh-mode esh-util eshell records-util records records-autoloads records-dindex records-index records-vars calc-macs calc-ext emoney filesets2 textstats excerpt remarks file-properties tellib bm tinybookmark tinylib tinyeat tinylibm tinylibb tinyliba checkdoc gse-number-rect texinfo view-less view filladapt dictionary link connection mmm-mode mmm-univ mmm-class mmm-region mmm-utils css-mode apropos javascript-mode html-mode jsp-html-helper-mode asp-html-helper-mode php-html-helper-mode html-helper-mode sgml-mode psgml psgml-html cc-mode cc-fonts cc-awk cc-menus cc-cmds cc-styles cc-align cc-engine cc-langs cc-vars cc-defs cc-bytecomp igrep matlab shell gud tags etags tempo home-end reftex reftex-vars latex tex-style tex morse protbuf iso-acc backup-dir iswitchb highlight-current-line egocentric goto-addr hyper-apropos browse-kill-ring joc-toggle-buffer highline paren mic-paren quickurl highlight-completion boxquote rect tf diminish http-post http-get hexl nethack nethack-cmd nethack-api nethack-keys gamegrid nethack-compat malyon malyon-mode ll-debug skeleton stripes color-alternation-mode apt-upgrade apt-utils jka-compr rk-todo longlines planner-w3m w3m unicode mule-ucs-unicode un-data mucs mucs-ccl trans-util mucs-type mucs-error browse-url image-mode timezone w3m-hist w3m-fb w3m-xmas path-util w3m-ccl pccl pccl-20 w3m-favicon w3m-image w3m-proc w3m-util poem poem-xm pces pces-xfc pces-20 pces-xm poe-xemacs broken poe pym static apel-ver product planner-gnus gnus-msg gnus-art mm-uu mml2015 pgg password pgg-parse pgg-def mm-view planner-rank planner-multi crm planner-deadline planner-id compile planner sort autorevert muse-wiki muse-html muse-xml-common muse-colors muse-mode pcomplete muse-project cus-edit cus-load muse-publish muse muse-protocols muse-regexps tumme cal-china lunar solar cal-dst cal-persia cal-islam cal-julian cal-hebrew holidays diary-lib appt time lifebalance supermemo timer-funcs pdb calc mpg123 telnet completer time-stamp auto-capitalize whitespace ffap bookmark eldoc footnote org-learn org-publish org-exp org-exp-blocks org-agenda org byte-optimize bytecomp byte-compile org-footnote org-src org-list org-faces org-compat org-macs calendar cal-xemacs cal-compat gnus-sum nnoo gnus-group gnus-undo nnmail mail-source format-spec gnus-start gnus-spec gnus-int gnus-range gnus-win gnus gnus-ems gnus-xmas wid-edit noutline outline remember bufcomp light ibuffer ace-jump-mode mu ansi-color silly-mail shop timer query-repace-by-example vvb-mode efs-cu balloon-help sawfish inf-lisp comint ring pp regexp-opt font-lock thingatpt info wdired dired-faces dired-xemacs dired picture edmacro message messagexmas mml mml-sec mml-smime smime dig mm-decode mm-bodies mm-encode mailcap lpr mail-parse rfc2045 rfc2231 rfc2047 qp ietf-drums mail-abbrevs sendmail rfc822 nnheader nnheaderxm run-at-time gnus-util netrc time-date parse-time mm-util mail-prsvr mail-utils mailheader canlock sha1 hex-util un-define latin-euro-standards latin-euro-latin10 latin-euro-latin9 latin-euro-latin8 latin-euro-latin7 mule-uni cl-compat debian-el debian-el-loaddefs w3m-load mmm-auto mmm-vars cus-face mmm-compat overlay emacs-goodies-el emacs-goodies-custom emacs-goodies-loaddefs emacs-goodies-el-autoloads executable cedet-autogen advice advice-preload autoload autoload-operators easy-mmode mwheel zenirc-autoloads xwem-autoloads xslt-process-autoloads xslide-autoloads xlib-autoloads xetla-autoloads xemacs-devel-autoloads xemacs-base-autoloads x-symbol-autoloads w3-autoloads vm-autoloads viper-autoloads view-process-autoloads vhdl-autoloads vc-cc-autoloads vc-autoloads tramp-autoloads tpu-autoloads tooltalk-autoloads tm-autoloads time-autoloads textools-autoloads text-modes-autoloads texinfo-autoloads supercite-autoloads strokes-autoloads speedbar-autoloads sounds-wav-autoloads sounds-au-autoloads sml-mode-autoloads slider-autoloads sieve-autoloads sh-script-autoloads sgml-autoloads semantic-autoloads scheme-autoloads sasl-autoloads ruby-modes-autoloads rmail-autoloads riece-autoloads reftex-autoloads re-builder-autoloads python-modes-autoloads psgml-dtds-autoloads psgml-autoloads ps-print-autoloads prog-modes-autoloads pgg-autoloads perl-modes-autoloads pcomplete-autoloads pcl-cvs-autoloads pc-autoloads os-utils-autoloads oo-browser-autoloads ocaml-autoloads net-utils-autoloads mmm-mode-autoloads misc-games-autoloads mine-autoloads mh-e-autoloads mew-autoloads mailcrypt-autoloads mail-lib-autoloads jde-autoloads ispell-autoloads ilisp-autoloads igrep-autoloads idlwave-autoloads ibuffer-autoloads hyperbole-autoloads hm--html-menus-autoloads haskell-mode-autoloads guided-tour-autoloads gnus-autoloads gnats-autoloads general-docs-autoloads games-autoloads fsf-compat-autoloads frame-icon-autoloads fortran-modes-autoloads forms-autoloads footnote-autoloads eudc-autoloads eterm-autoloads eshell-autoloads escreen-autoloads erc-autoloads emerge-autoloads elib-autoloads eieio-autoloads efs-autoloads edt-autoloads edit-utils-autoloads ediff-autoloads edebug-autoloads ede-autoloads ecrypto-autoloads ecb-autoloads easypg-autoloads docbookide-autoloads dired-autoloads dictionary-autoloads debug-autoloads crisp-autoloads cookie-autoloads cogre-autoloads clearcase-autoloads cedet-common-autoloads cc-mode-autoloads calendar-autoloads calc-autoloads c-support-autoloads build-autoloads bbdb-autoloads tex-site auctex-autoloads apel-autoloads ada-autoloads Sun-autoloads skk-autoloads mule-ucs-autoloads mule-base-autoloads lookup-autoloads locale-autoloads leim-autoloads latin-unity-autoloads latin-euro-standards-autoloads egg-its-autoloads edict-autoloads lisp-autoloads loadhist auto-show fontl-hooks x-iso8859-1 slovenian czech romanian ccl mule-help code-cmds gutter-items menubar-items x-menubar mode-motion mouse itimer auto-save lisp-mode easymenu iso8859-1 page buff-menu lib-complete cus-file derived frame text-props obsolete cus-start custom widget cl-extra mini-cl cl cl-19 packages backquote very-early-lisp file-coding mule lucid-scrollbars cut-buffer lucid-menubars athena-dialogs x c-balloon-help tty-frames tty toolbar native-sound scrollbar unix-processes multicast network-streams subprocesses modules menu-accelerator-support menubar berkeley-db md5 xemacs xim gutter tiff png gif jpeg xface xpm xbm lisp-float-type linux dialog devices window-system base64) -- Udv, Richard _______________________________________________ XEmacs-Beta mailing list XEmacs-Beta(a)xemacs.org http://lists.xemacs.org/mailman/listinfo/xemacs-beta