4:02 p.m.
The discussion about XEmacs blowing with really large mail messages
reminded me of a concern that I had a long time ago about *macs and
subprocesses.
When a UNIX process wants to create a child, it fork()s and exec()s.
The fork() duplicates the parent's address space. This was seen as
silly when 9 times out of 10 the child simply calls exec and thus was
born vfork(). vfork() has this problem, though that the child briefly
runs in the parents address space. The implications of this statement
are serious and have resulted in two things. First, standards authors
have had no choice but to say that you can't guarantee anything aside
from exec() or _exit() will work. Second, application developers have
been bitten by all kinds of nasty bugs and have shied away from vfork.
The net result is that vfork() could probably be removed from the UNIX
ABI and nobody would shed a tear.
However, some processes get ridiculously large. Fork/exec on a >100MB
process can be quite silly... and knowing that there have been some
important bug fixes within Solaris' vfork I decided it was time to
have a second look.
I hacked process-unix.c to use vfork() instead of fork(), built, and
ran. I then loaded up a whole bunch of large files I have kicking
around until I ran out of memory (for those really curious, there's
pmap output at the bottom of this mail). I then started using things
like M-x shell to create subprocesses. I kept going until I exhausted
the per process file descriptor limit. Were my XEmacs using fork(),
those shells would never have gotten started because the calls to
fork() would fail.
So, for those folks playing with 36MB mail attachments for which
mmencode needs to be called as a subprocess, seems like vfork might
not be so bad after all...
Rather than shun vfork, perhaps careful examination and
rationalization of all the code that gets run in the child after the
[v]fork() would be a plan...
Of course, the right answer is for UNIX at large to adopt the Plan 9
rfork()...
Comments?
P.S. I'm going to run with vfork() instead of fork() for a little
while to see if I run into any nasties that weren't uncovered by my
experiment. I'll keep you posted.
-----------
pmap output:
3174: src/xemacs
Address Kbytes Resident Shared Private Permissions Mapped File
00010000 5280 4208 - 4208 read/exec xemacs
00546000 2408 2160 - 2160 read/write/exec xemacs
007A0000 30640 9264 - 9264 read/write/exec [ heap ]
D5C00000 5584 5448 - 5448 read/write [ anon ]
D6400000 10152 9688 - 9688 read/write [ anon ]
D7400000 4936 4888 - 4888 read/write [ anon ]
D7C00000 16344 12696 - 12696 read/write [ anon ]
D9400000 10152 7904 - 7904 read/write [ anon ]
DA400000 7096 6504 - 6504 read/write [ anon ]
DAC00000 11448 8480 - 8480 read/write [ anon ]
DBC00000 7992 7016 - 7016 read/write [ anon ]
DC400000 11448 7952 - 7952 read/write [ anon ]
DD400000 6304 4912 - 4912 read/write [ anon ]
DDC00000 5584 3752 - 3752 read/write [ anon ]
DE400000 4936 3952 - 3952 read/write [ anon ]
DEC00000 4936 3160 - 3160 read/write [ anon ]
DF400000 11448 6560 - 6560 read/write [ anon ]
E0800000 6304 4104 - 4104 read/write [ anon ]
E1000000 4936 2912 - 2912 read/write [ anon ]
E1800000 18400 10472 - 10472 read/write [ anon ]
E2C00000 7096 3408 - 3408 read/write [ anon ]
E3C00000 16344 7024 - 7024 read/write [ anon ]
E4C00000 11448 4840 - 4840 read/write [ anon ]
E5800000 14512 5504 - 5504 read/write [ anon ]
E6800000 16344 5832 - 5832 read/write [ anon ]
E7C00000 10152 4912 - 4912 read/write [ anon ]
E8800000 12888 5320 - 5320 read/write [ anon ]
E9800000 26208 9048 - 9048 read/write [ anon ]
EB400000 26208 8528 - 8528 read/write [ anon ]
ED000000 7096 2200 - 2200 read/write [ anon ]
ED800000 4936 2536 - 2536 read/write [ anon ]
EE000000 33192 7864 - 7864 read/write [ anon ]
F0400000 7096 1776 - 1776 read/write [ anon ]
F0C00000 20704 5552 - 5552 read/write [ anon ]
F2400000 37368 6144 - 6144 read/write [ anon ]
F4C00000 26208 3944 - 3944 read/write [ anon ]
F6C00000 6304 1688 - 1688 read/write [ anon ]
F7400000 5584 3360 - 3360 read/write [ anon ]
F7C00000 26208 6592 - 6592 read/write [ anon ]
F9800000 10152 1144 - 1144 read/write [ anon ]
FA400000 9000 2304 - 2304 read/write [ anon ]
FB000000 5584 1496 - 1496 read/write [ anon ]
FB800000 9000 1400 - 1400 read/write [ anon ]
FC400000 7096 1720 - 1720 read/write [ anon ]
FCC00000 16344 2272 - 2272 read/write [ anon ]
FDC80000 3848 3080 - 3080 read/write [ anon ]
FE080000 3856 352 - 352 read/write [ anon ]
FE700000 720 664 - 664 read/write [ anon ]
FE880000 1032 992 - 992 read/write [ anon ]
FE9B0000 344 336 - 336 read/write [ anon ]
FEA60000 8 8 - 8 read/write [ anon ]
FEA70000 8 8 - 8 read/write [ anon ]
FEA80000 360 344 - 344 read/write [ anon ]
FEAE0000 344 336 - 336 read/write [ anon ]
FEB60000 8 - - - read/write [ anon ]
FEB80000 8 - - - read/write [ anon ]
FEB90000 8 - - - read/write [ anon ]
FEBA0000 8 8 - 8 read/write [ anon ]
FEBB0000 16 16 8 8 read/exec libmp.so.2
FEBC2000 8 8 8 - read/write/exec libmp.so.2
FEBD0000 80 24 16 8 read/exec libICE.so.6
FEBF2000 8 - - - read/write/exec libICE.so.6
FEBF4000 8 - - - read/write/exec [ anon ]
FEC00000 656 608 592 16 read/exec libc.so.1
FECB2000 32 32 - 32 read/write/exec libc.so.1
FECC0000 8 8 - 8 read/write [ anon ]
FECD0000 16 16 8 8 read/exec libc_psr.so.1
FECE0000 40 16 8 8 read/exec libSM.so.6
FECF8000 8 - - - read/write/exec libSM.so.6
FED00000 512 384 368 16 read/exec libnsl.so.1
FED8E000 40 40 - 40 read/write/exec libnsl.so.1
FED98000 32 8 - 8 read/write/exec [ anon ]
FEDB0000 8 8 - 8 read/write/exec [ anon ]
FEDC0000 24 24 16 8 read/exec libgen.so.1
FEDD4000 16 16 - 16 read/write/exec libgen.so.1
FEDE0000 32 32 24 8 read/exec libsocket.so.1
FEDF6000 16 16 - 16 read/write/exec libsocket.so.1
FEE00000 88 72 64 8 read/exec libm.so.1
FEE24000 8 8 8 - read/write/exec libm.so.1
FEE30000 168 72 - 72 read/exec libcurses.so.1
FEE68000 40 8 - 8 read/write/exec libcurses.so.1
FEE72000 8 - - - read/write/exec [ anon ]
FEE80000 8 - - - read/write [ anon ]
FEE90000 448 440 432 8 read/exec libX11.so.4
FEF00000 24 24 - 24 read/write/exec libX11.so.4
FEF10000 336 336 320 16 read/exec libXt.so.4
FEF74000 24 24 - 24 read/write/exec libXt.so.4
FEF7A000 8 8 - 8 read/write/exec [ anon ]
FEF80000 488 112 104 8 read/exec libtt.so.2
FF00A000 48 - - - read/write/exec libtt.so.2
FF020000 80 72 64 8 read/exec libXext.so.0
FF042000 8 8 8 - read/write/exec libXext.so.0
FF050000 8 8 - 8 read/write/exec [ anon ]
FF060000 80 56 48 8 read/exec libXmu.so.4
FF082000 16 16 - 16 read/write/exec libXmu.so.4
FF090000 384 192 160 32 read/exec libDtSvc.so.1
FF0F0000 24 24 - 24 read/write/exec libDtSvc.so.1
FF100000 2032 1160 1048 112 read/exec libXm.so.4
FF30C000 96 88 - 88 read/write/exec libXm.so.4
FF330000 56 8 - 8 read/exec libXpm.so.4.6
FF34C000 8 - - - read/write/exec libXpm.so.4.6
FF360000 160 40 - 40 read/exec libpng.so.2.1.0
FF396000 16 - - - read/write/exec libpng.so.2.1.0
FF3A0000 8 8 - 8 read/exec libdl.so.1
FF3B0000 120 120 112 8 read/exec ld.so.1
FF3DC000 8 8 - 8 read/write/exec ld.so.1
FF3DE000 8 8 - 8 read/write/exec [ anon ]
FFBCC000 144 88 - 88 read/write/exec [ stack ]
-------- ------ ------ ------ ------
total Kb 576464 242832 3416 239416
5:28 p.m.
>>>> "PA" == Per Abrahamsen
<abraham(a)dina.kvl.dk> writes:
PA> Isn't fork copy-on-write these days?
Imprecise.
Fork duplicates the address space.
Ignoring the details of a particular VM (virtual memory)
implementation, the confusion folks seem to have is that the address
space may ultimately point at copy on write _pages_.
This concept may be easier to understand when you start thinking about
what the failure mode(s) should be. In UNIX we don't have the notion
that a process may arbitrarily fail when a copy-on-write operation
fails (we do have signals, but none describe COW failure) whereas
fork() _is_ defined as possibly failing with ENOMEM.
6:17 p.m.
Georg Nikodym writes:
This concept may be easier to understand when you start thinking
about
what the failure mode(s) should be. In UNIX we don't have the notion
that a process may arbitrarily fail when a copy-on-write operation
fails (we do have signals, but none describe COW failure) whereas
fork() _is_ defined as possibly failing with ENOMEM.
I think 4.3 BSD just swung the meataxe and SIGKILL'd the process
if it overcommitted swap space. So that's what I'd expect to
happen. Yeah, it's mondo unclean. Waiting for another process
to die and give up space would lead to deadlock situations I'm
sure.
6:08 p.m.
Georg Nikodym writes:
| When a UNIX process wants to create a child, it fork()s and exec()s.
| The fork() duplicates the parent's address space. This was seen as
| silly when 9 times out of 10 the child simply calls exec and thus was
| born vfork(). vfork() has this problem, though that the child briefly
| runs in the parents address space. The implications of this statement
| are serious and have resulted in two things. First, standards authors
| have had no choice but to say that you can't guarantee anything aside
| from exec() or _exit() will work. Second, application developers have
| been bitten by all kinds of nasty bugs and have shied away from vfork.
| The net result is that vfork() could probably be removed from the UNIX
| ABI and nobody would shed a tear.
Yes, I know. I got several emails on this topic and if point people to
vfork() it seems they don't see vfork() as a solution. In this case we
had a proccess doing a fork() when it had grown to 300MB. This was a
server and it would do fork() multiple times during its life.
| So, for those folks playing with 36MB mail attachments for which
| mmencode needs to be called as a subprocess, seems like vfork might
| not be so bad after all...
If you could your hacking a patch I could build a _Management Edition_
of XEmacs. ;-) In fact this was just to point out that there might be
other issues to solve first before move to 64 bit computing. WRT 64
bit OS 36MB are ridiculess (and I guess even 36GB are).
| Rather than shun vfork, perhaps careful examination and
| rationalization of all the code that gets run in the child after the
| [v]fork() would be a plan...
I agree here and I guess this will become a issue anyway. Most of the
people have just started using MIME together with XEmacs. The more
they use XEmacs with larger data the more they feel the need for
performance improvement. (I remember there was a discussion about W3
rendering machine a while ago.)
| Of course, the right answer is for UNIX at large to adopt the Plan 9
| rfork()...
I never got my hands on Plan 9 but a review in a magazine was rarely
talking about these internal things.
--Marcus
4:33 p.m.
>>>> "mt" == Marcus Thiessel
<marcus(a)xemacs.org> writes:
mt> If you could your hacking a patch I could build a _Management
mt> Edition_ of XEmacs. ;-) In fact this was just to point out that
mt> there might be other issues to solve first before move to 64 bit
mt> computing. WRT 64 bit OS 36MB are ridiculess (and I guess even
mt> 36GB are).
Sure, I'll try to develop a patch.
mt> I never got my hands on Plan 9 but a review in a magazine was
mt> rarely talking about these internal things.
Plan 9 has an rfork() call that took an argument which is a bitmap of
resources to share between parent and child.
I believe Linux has something like this. I looked at under the covers
and saw the do_fork() in the kernel takes a clone_flags argument which
can control fairly precisely what gets duplicated in the child.
7:57 a.m.
Georg Nikodym writes:
| >>>>> "mt" == Marcus Thiessel <marcus(a)xemacs.org>
writes:
| Plan 9 has an rfork() call that took an argument which is a bitmap of
| resources to share between parent and child.
Do you have any additional pointers to web resources? I would be
interested in the things that have changed.
| I believe Linux has something like this. I looked at under the covers
| and saw the do_fork() in the kernel takes a clone_flags argument which
| can control fairly precisely what gets duplicated in the child.
I thought the clone was implemented for the use with kernel
threads.
2:40 p.m.
>>>> "mt" == Marcus Thiessel
<marcus(a)xemacs.org> writes:
mt> Do you have any additional pointers to web resources? I would be
mt> interested in the things that have changed.
http://cm.bell-labs.com/magic/man2html/2/fork
7:56 p.m.
Georg Nikodym wrote:
When a UNIX process wants to create a child, it fork()s and exec()s.
The fork() duplicates the parent's address space. This was seen as
silly when 9 times out of 10 the child simply calls exec and thus was
born vfork(). vfork() has this problem, though that the child briefly
runs in the parents address space. The implications of this statement
are serious and have resulted in two things. First, standards authors
have had no choice but to say that you can't guarantee anything aside
from exec() or _exit() will work.
Which makes vfork() rather useless for spawning subprocesses. If you
can guarantee that dup2() (or similar) will work, then vfork() may be
of some use.
--
Glynn Clements <glynn(a)sensei.co.uk>
4:42 p.m.
>>>> "GC" == Glynn Clements
<glynn(a)sensei.co.uk> writes:
GC> Which makes vfork() rather useless for spawning subprocesses. If
GC> you can guarantee that dup2() (or similar) will work, then
GC> vfork() may be of some use.
In Solaris, dup2 has always worked because the file table hangs of the
user structure of the process which is duplicated in the child.
(Some confusion seems to come from the fact that children that call
exit() instead of _exit() flush and close stdio FILE buffers...)
The bug that Solaris had until 2.6 was that the signal disposition
wasn't handled correctly because a lot of the signal junk is actually
contained within libc data area (dynamic) and thus part of the address
space. So, when a child diddled signal stuff, it actually wacked the
parents stuff. Again, this has been fixed since 2.6.
6:15 a.m.
On Wed, 4 Nov 1998, Georg Nikodym wrote:
When a UNIX process wants to create a child, it fork()s and exec()s.
The fork() duplicates the parent's address space. This was seen as
silly when 9 times out of 10 the child simply calls exec and thus was
born vfork(). vfork() has this problem, though that the child briefly
runs in the parents address space. The implications of this statement
are serious and have resulted in two things. First, standards authors
have had no choice but to say that you can't guarantee anything aside
from exec() or _exit() will work. Second, application developers have
been bitten by all kinds of nasty bugs and have shied away from vfork.
The net result is that vfork() could probably be removed from the UNIX
ABI and nobody would shed a tear.
I don't believe this is the case. While fork() does duplicate the address
space, it does not have to physically duplicate the memory used by the
process. First, mapped files (the executable, shared libs) are read-only
and thus can be shared. Also, most modern OSes will implement
copy-on-write in this situation. The original memory image will be shared
on a page-by-page basis until one process writes to the shared page. At
that point, the page is duplicated.
So, { fork(); exec(); } is a relatively efficient operation.
The overhead that I believe that vfork is attempting to eliminate is that
of the actual management of these internal data structures when
duplicating a process. vfork allows the child to run until _exit or exec
using the kernel structures of the parent. When the _exit/exec call
"completes", the parent resumes. The exec call will create a new process.
This saves the overhead of (1) marking the shared pages as being shared at
the time of the fork only to (2) release the shared pages back to the
parent upon the subsequent exec.
Comments?
This is very system dependent, but this is how it's "supposed to" work.
-Justin
vallon(a)mindspring.com
1:42 p.m.
On Thu, 5 Nov 1998, Justin Vallon wrote:
On Wed, 4 Nov 1998, Georg Nikodym wrote:
> When a UNIX process wants to create a child, it fork()s and exec()s.
> The fork() duplicates the parent's address space. This was seen as
> silly when 9 times out of 10 the child simply calls exec and thus was
> born vfork(). vfork() has this problem, though that the child briefly
> runs in the parents address space. The implications of this statement
> are serious and have resulted in two things. First, standards authors
> have had no choice but to say that you can't guarantee anything aside
> from exec() or _exit() will work. Second, application developers have
> been bitten by all kinds of nasty bugs and have shied away from vfork.
> The net result is that vfork() could probably be removed from the UNIX
> ABI and nobody would shed a tear.
M-: (douse fire)
So, if your OS (Solaris?) does not overcommit vm pages, then at fork()
time, free swap pages will be earmarked as potentially required if the
process(es) write to all of their shared pages? Since the parent has a
100Mb swap footprint, you'll basically need 100Mb free swap to be able to
do {fork();exec();} due to the high-water-mark, even though no (few)
shared pages will ever be copied-due-to-write.
-Justin
vallon(a)mindspring.com
7:32 p.m.
Justin Vallon wrote:
> > When a UNIX process wants to create a child, it fork()s and
exec()s.
> > The fork() duplicates the parent's address space. This was seen as
> > silly when 9 times out of 10 the child simply calls exec and thus was
> > born vfork(). vfork() has this problem, though that the child briefly
> > runs in the parents address space. The implications of this statement
> > are serious and have resulted in two things. First, standards authors
> > have had no choice but to say that you can't guarantee anything aside
> > from exec() or _exit() will work. Second, application developers have
> > been bitten by all kinds of nasty bugs and have shied away from vfork.
> > The net result is that vfork() could probably be removed from the UNIX
> > ABI and nobody would shed a tear.
M-: (douse fire)
So, if your OS (Solaris?) does not overcommit vm pages, then at fork()
time, free swap pages will be earmarked as potentially required if the
process(es) write to all of their shared pages?
Yep.
Since the parent has a 100Mb swap footprint, you'll basically
need
100Mb free swap to be able to do {fork();exec();} due to the
high-water-mark,
Yep.
even though no (few) shared pages will ever be copied-due-to-write.
But you can't know that this will be the case at the time that fork()
was called; the child might not call exec().
If you get around the problem by overcommitting, then you are left
with a child process which can die suddenly just because it wrote to
one page too many. This isn't a good idea if you want to write robust
programs.
The solution is to provide a usable vfork(), i.e. one where you can
guarantee being able to perform a minimal set of useful operations
between the vfork() and the exec(). dup2() is an obvious candidate;
signal handling would also be pretty high on the list.
--
Glynn Clements <glynn(a)sensei.co.uk>
4:29 p.m.
Sorry, real life interfered with my ability to participate in a timely
manner...
>>>> "JV" == Justin Vallon
<vallon(a)mindspring.com> writes:
JV> On Wed, 4 Nov 1998, Georg Nikodym wrote:
> When a UNIX process wants to create a child, it fork()s and
> exec()s. The fork() duplicates the parent's address space. This
> was seen as silly when 9 times out of 10 the child simply calls
> exec and thus was born vfork(). vfork() has this problem, though
> that the child briefly runs in the parents address space. The
> implications of this statement are serious and have resulted in
> two things. First, standards authors have had no choice but to
> say that you can't guarantee anything aside from exec() or _exit()
> will work. Second, application developers have been bitten by all
> kinds of nasty bugs and have shied away from vfork. The net
> result is that vfork() could probably be removed from the UNIX ABI
> and nobody would shed a tear.
JV> I don't believe this is the case. While fork() does duplicate
JV> the address space, it does not have to physically duplicate the
JV> memory used by the process. First, mapped files (the executable,
JV> shared libs) are read-only and thus can be shared. Also, most
JV> modern OSes will implement copy-on-write in this situation. The
JV> original memory image will be shared on a page-by-page basis
JV> until one process writes to the shared page. At that point, the
JV> page is duplicated.
Absolutely. But whether or not a physical page is copied into has no
bearing on the allocation of a virtual page.
JV> So, { fork(); exec(); } is a relatively efficient operation.
Duplicating an address space is not trivial. An SVR4 address space
(as) consists of a list of segments (seg) which each have a list of
page structures (page). So, in my 500MB+ XEmacs, there's still 62500
pages to be duplicated.
vfork does _no_ address space duplication. So to say that fork() is
not much more expensive is one of the convenient lies of
simplification for under qualified application programmers that later
become net.myth. (sorry wrong rant)
A number of you have raise the notion of over-commitment. I agree
that a case can be made here. However, the argument then distills
down to one of philosophy:
On the one hand you check to see that everything you could ever want
to do will be possible (ie, some minimal guarantee) and if not fail
early in such a way that the application can decide how to solve the
problem.
Or on the other hand, don't guarantee anything and create a situation
where an application can be shot randomly by the kernel if it
misbehaves.
The latter approach is seductive but has some serious problems.
The main one being the application programmers cannot protect or
harden themselves against this class of failure. To many commercial
software providers (ignoring any statements about the quality of their
stuff) this approach is untenable.
With AIX, IBM introduced SIGDANGER (a signal that will be fatally
delivered to random processes when the system is running out of
memory). The hue and cry in the industry drove IBM to make the
delivery of SIGDANGER optional. Nobody (here at least) wants to
repeat that bit of history.
Once again the real answer is to invent a new interface that actually
does what everybody wants and provides the safety guarantees the
everybody needs. Possible but hard, both technically and politically.
In the meantime, for the purposes of XEmacs, vfork on Solaris 2.6 and
up is safe.
5:13 p.m.
>>>> "GN" == Georg Nikodym <georgn＠Canada>
writes:
GN> Duplicating an address space is not trivial. An SVR4 address
GN> space (as) consists of a list of segments (seg) which each have a
GN> list of page structures (page). So, in my 500MB+ XEmacs, there's
GN> still 62500 pages to be duplicated.
As has been pointed out by a Sun lurker, I'm full of crap here.
The as and the segments are duplicated, but the page structures are
not.
Clearly more research on my part is required.
My statement about the non-triviality of address space duplication
stands, though[1].
Footnotes:
[1] Vahalia, Uresh. Section 14.7.3 Process Creation, _UNIX
Internals, The New Frontiers_. Prentice Hall, Inc. 1996.
9513
days inactive
9515
days old
14 comments
6 participants
participants (6)
-
Georg Nikodym -
Glynn Clements -
Justin Vallon -
Kyle Jones -
Marcus Thiessel -
Per Abrahamsen