O_CLOFORK use case

[Joshua Hudson]

I ran headlong into the use case for O_CLOFORK and got into a locking
debate over it.

The actual use case involves squashing a thread race between two
threads. If a file is opened for write in one thread with O_CLOEXEC
while another thread calls fork(), a race condition can happen where
the thread that closes the handle misses the out of disk error because
the child process closed the handle last inside execve().

The decades old idiom for replacing config files isn't safe in
multi-threaded code. Yipe.

    int h = open(".configfile~", O_WRONY | O_EXCL | O_CLOEXEC, 0666);
    if (h < 0) { perror(".configfile"); return 1; }
    ssize_t delta = 0;
    while ((delta = write(h, newconfigdata, newconfiglen)) > 0) {
        newconfigdata += delta;
        newconfiglen -= delta;
    }
    if (delta < 0) { perror(".configfile"); return 1; }
    if (close(h)) { perror(".configfile"); return 1; }
    rename(".configfile~", ".configfile");

To fix it, we have to put locks around close() and fork()/vfork(). Ugh.