Re: [V] patch to help efs ensure synchonization with ftp process (long)

Sorry for the cross-posting, but each responder to the original message that I sent to xemacs-patches suggested other places to send this patch. I've attached the original email, patch and changelog. Michael> The problem seems to be that your process buffer (or your Michael> ftp client) is somehow echoing all commands which it Michael> shouldn't. Could you post more data on your configuration Michael> (via M-x describe-installation RET) to xemacs-beta, maybe Michael> with some information about if this duplication also Michael> happens, say, in shell mode or something. Andy> You need to send this to the efs mlist andy As a point of information, the efs list looks like there hasn't been any activity on it since 1998. I am referring to Andy Norman's site, http://www-uk.hpl.hp.com/people/ange/efs/ If there is another list, I'd appreciate the address. I currently cannot reach www.xemacs.org to check for an efs mailing list. I am not subscribed to xemacs-beta, so please CC: me on responses. I don't know why xemacs-beta is the place for this, but here it is. I'm doing ten things at once, so excuse me if I sound incoherent anywhere. NB: the string of equal signs are just to separate sections of this email for readability. regards, davep ========================================================================= ftp in xemacs buffer, M-x ftp Ftp to Host: some-host Connected to some-host.crl.dec.com. 220 ProFTPD 1.2.2rc1 Server (ProFTPD Default Installation) [some-host] 331 Password required for davep. 230 User davep logged in. Remote system type is UNIX. Using binary mode to transfer files. ftp> progress off progress off Progress bar off. ftp> quote cwd ~/diary/ quote cwd ~/diary/ 550 ~/diary/: Not a directory ftp> ========================================================================= This does indeed show extra echoing. running ftp in an xterm and typing commands by hand doesn't show extra echoing. running ftp in an xterm with input from a file doesn't show extra echoing. (the outputs are available if you need them) I'm running the standard ftp client, on a pretty standard FreeBSD system. I've done a lot of xemacs customization in elisp, but (famous last words) None of it should affect an ftp buffer. That said, any idea of what I did to affect the ftp buffer? ========================================================================= output from M-x describe-current-coding-system in efs' ftp buffer: Coding system for saving this buffer: Not set locally, use the default. Default coding system (for new files): ISO8 -- iso-2022-8 Coding system for keyboard input: nil Coding system for terminal output: nil Coding systems for process I/O: encoding input to the process: Binary -- binary decoding output from the process: Auto:t -- undecided-mac Priority order for recognizing coding systems when reading files: 1. iso-2022-7 2. raw-text 3. ctext 4. iso-2022-lock Other coding systems cannot be distinguished automatically from these, and therefore cannot be recognized automatically with the present coding system priorities. Particular coding systems specified for certain file names: OPERATION TARGET PATTERN CODING SYSTEM(s) --------- -------------- ---------------- File I/O "\\.\\(?:bz2\\|elc\\|g\\(if\\|z\\)\\|jp\\(eg\\|g\\)\\|png\\|t\\(ar\\|gz\\|iff\\)\\|[Zo]\\)\\'" binary "TUTORIAL\\.\\(?:hr\\|pl\\|ro\\)\\'" iso-8859-2 Process I/O nothing specified Network I/O nothing specified ========================================================================= output from M-x describe-installation: uname -a: FreeBSD 4.6.2-RELEASE FreeBSD 4.6.2-RELEASE #1: Tue Aug 20 13:34:40 EDT 2002 i386 ./configure '--with-x11' '--with-mule' '--x-includes=/usr/X11R6/include' '--x-libraries=/usr/X11R6/lib' '--site-libraries=/usr/local/lib' '--site-includes=/usr/local/include' '--with-xface' '--with-sound=native' '--with-site-lisp' '--with-pop' '--with-xfs' '--with-jpeg' '--with-png' '--with-tiff' '--with-ldap=yes' '--infopath=/usr/local/lib/xemacs/info:/usr/local/info:/usr/X11R6/info:/usr/info:/usr/local/lib/texmf/doc/info:/usr/lib/texmf/doc:/usr/share/info' '--with-clash-detection' '--with-database=berkdb' '--with-xpm' '--with-xim=xlib' '--with-athena=xaw' '--with-menubars=lucid' '--with-scrollbars=motif' '--with-dialogs=motif' '--with-widgets=motif' '--with-canna=no' '--with-wnn=no' '--with-wnn6=no' '--x-libraries=/usr/X11R6/lib' '--x-includes=/usr/X11R6/include' '--prefix=/usr/local' XEmacs 21.4.9 "Informed Management" configured for `i386-unknown-freebsd4.6.2'. Compilation / Installation: Source code location: /sundry/build/xemacs/xemacs-21.4.9 Installation prefix: /usr/local Additional header files: /usr/local/include Additional libraries: /usr/local/lib Runtime library search path: /usr/local/lib:/usr/X11R6/lib Operating system description file: `s/freebsd.h' Machine description file: `m/intel386.h' Compiler: gcc -g -O3 -Wall -Wno-switch -Winline -Wmissing-prototypes -Wsign-compare -Wshadow -Wpointer-arith Relocating allocator for buffers: yes GNU version of malloc: yes Window System: Compiling in support for the X window system: - X Windows headers location: /usr/X11R6/include - X Windows libraries location: /usr/X11R6/lib - Handling WM_COMMAND properly. Using Lucid menubars. Using Motif scrollbars. Using Motif dialog boxes. Using Motif native widgets. TTY: Compiling in support for ncurses. Images: Compiling in support for GIF images (builtin). Compiling in support for XPM images. Compiling in support for PNG images. Compiling in support for JPEG images. Compiling in support for TIFF images. Compiling in support for X-Face message headers. Sound: Compiling in support for sound (native). Databases: Compiling in support for Berkeley database. Compiling in support for LDAP. Compiling in support for PostgreSQL. - Using PostgreSQL header file: libpq-fe.h - Using PostgreSQL V7 bindings. Internationalization: Compiling in support for Mule (multi-lingual Emacs). Compiling in support for XIM (X11R5+ I18N input method). - Using raw Xlib to provide XIM support. - Using XFontSet to provide bilingual menubar. Mail: Compiling in support for POP mail retrieval. Compiling in support for "flock" mail spool file locking method. Other Features: Inhibiting IPv6 canonicalization at startup. Compiling in support for dynamic shared object modules. Hi all, This will most likely be rejected, but if it starts some activity on efs then that will be enough. Ever since I switched to xemacs, I've had problems with efs that I never saw with ange-ftp under fsf emacs. The main problem is getting: ``Unable to identify remote host type'' errors. The happens perhaps 1 in 5 times when I first use efs. Less often is a writable file being fetched as read-only. Even less often is a directory being fetched instead of the desired file. I'm pretty sure that the first error is caused by efs getting out of sync with the ftp process. It is not unreasonable to believe that this could be the cause of the other errors, too. The contents of an ftp buffer in which the first kind of error occurs follows: Connected to some-node.crl.dec.com. 220 ProFTPD 1.2.2rc1 Server (ProFTPD Default Installation) [some-node] quote user "some-user" ftp> quote user "some-user" 331 Password required for some-user. quote pass Turtle Power! ftp> quote pass actual-password-characters! 230 User some-user logged in. progress off ftp> progress off Progress bar off. hash ftp> hash quote cwd ~/diary/ Hash mark printing on (1024 bytes/hash mark). ls "-al" /tmp/some-user/efsbAKpin ftp> quote cwd ~/diary/ There is definitely a timing problem. In this trace you can see that the hash command is sent before the ftp program has issued its prompt after the progress off command. (the progress off command is also sent out of sync but that is non-fatal) Here is what then happens: efs-process-filter matches "^ftp> " and chews it off. Then efs-process-handle-line matches ``Hash mark printing on (1024 bytes/hash mark).'' when it uses "^[Hh]ash$" to detect the results of the hash command. Efs now thinks the hash command is done. It isn't and we're out of sync. The hash mark size is never determined, and subsequent commands also get incorrect responses. Efs gets confused and errors out. Simply putting messages inside the "^ftp>" chewer slows things enough to "fix" the bug, so the timing is pretty close. Chewing ftp prompts, as mentioned in some comments, does not fix the out of sync condition. What I've tried to do is make things really synchronous by waiting for the ftp prompt after non-async commands. However, efs is a tad complex, so I'm not sure what I've done is correct. This patch does let me do hundreds of efs edits without any errors. If anyone who knows more about efs could look at this I would appreciate it. Since I've become a telecommuter, efs has become essential in getting my work done. I'm also very curious to know if anyone else has seen errors like these. thanks, davep --- efs.el.ORIG Thu Sep 19 18:27:07 2002 +++ efs.el Thu Sep 19 20:54:03 2002 ＠＠ -1421,6 +1421,9 ＠＠ (defvar efs-process-busy nil) (make-variable-buffer-local 'efs-process-busy) +(defvar efs-process-prompt-seen nil) +(make-variable-buffer-local 'efs-process-prompt-seen) + (defvar efs-process-result-line "") (make-variable-buffer-local 'efs-process-result-line) ＠＠ -2594,6 +2597,8 ＠＠ (setq str (efs-process-handle-hash str) efs-process-string (concat efs-process-string str))) (efs-process-log-string proc str) + (if (string-match efs-process-prompt-regexp str) + (setq efs-process-prompt-seen t)) (while (and efs-process-busy (string-match "\n" efs-process-string)) (let ((line (substring efs-process-string ＠＠ -2606,7 +2611,8 ＠＠ ;; never get prompts in the wrong place. Just to be safe, ;; chew them off. (while (string-match efs-process-prompt-regexp line) - (setq line (substring line (match-end 0)))) + (setq efs-process-prompt-seen t + line (substring line (match-end 0)))) (efs-process-handle-line line proc))) ;; has the ftp client finished? if so then do some clean-up ＠＠ -3422,6 +3428,7 ＠＠ efs-process-result-line "" efs-process-result-cont-lines "" efs-process-busy t + efs-process-prompt-seen nil efs-process-msg (and efs-verbose msg) efs-process-continue cont efs-process-server-confused nil ＠＠ -3466,7 +3473,8 ＠＠ ;; Some clients die after the command is sent, if the server ;; times out. Don't wait on dead processes. (efs-kbd-quit-protect proc - (while (and efs-process-busy + (while (and (or efs-process-busy + (not efs-process-prompt-seen)) ;; Need to recheck nowait, since it may get reset ;; in a cont. (null efs-process-nowait) 2002-09-19 David A. Panariti <davep.xemacs(a)meduseld.net&gt; * efs.el (efs-process-filter): check for efs-ftp-prompt-regexp and set efs-process-prompt-seen. * (efs-raw-send-cmd): check for efs-process-prompt-seen when nowait is nil. This ensures synchronization with the ftp process.