The problem is that while it's well known how to do safe locks with NFS,
maintaining cache coherency of the file you're locking isn't. This is
the problem that gets you and many other programs. Mail programs that
rely on dot-locking can lose or corrupt mail because of this, yet there
are a lot of them about.
On Fri, Dec 11, 1998 at 10:06:07AM -0500, Jim Nance wrote:
> I am trying to get an application that works under Solaris to work under
> Linux. Its a distributed processing application and it shares data through
> a common file system and synchronizes itself using locks in the file system.
The behaviour depends on what kind of lock you use: hard link to the
data file, separate lock file, fcntl(), lockf(), flock().
The only ones which synchronise the cached data are fcntl(), and lockf()
which simply calls fcntl(). And they didn't do so until kernel 2.1.131.
Many programs on other clients don't use fcntl(), so if you're
interacting with one of those, you to use a lock file. But that doesn't
synchronise the data. So you have to use both :-)
If you're in control of the program doing the locking, you can just use
fcntl().
> messages. If you run it on 2 linux machines you will eventually see one
> process get stuck waiting for the lock. It remains stuck even if you kill
> the other process and remove the lock file by hand. It appears that it
> has cached the results of the stat() call and is not actually sending them
> to the remote machine.
Nearly. The remote machine has cached stat() info, and isn't requesting
new info.
> After a minute or so it realizes that the lock
> is gone and it starts working again.
That's about right.
> I am using 2.1.131+ac7 as my kernel
> and I am using HJ's knfs nfs server. I have also tried running the program
> from directories served off Solaris and Digital Unix NFS servers and the
> problem is still there.
Indeed, it's purely a client problem.
You might have better luck calling stat() on the link you've just
created, instead of the file you're linking from. That should have an
up to date n_link value.
That won't get the right data though! (Because the data file is also
cached, and fiddling with locks won't flush that cache. It will read
the right data the first time). One solution is to use a lock as above,
and to write new data to new files. The other is to use fcntl() for
locking, and kernel 2.1.131 or later. fcntl() in 2.1.131 explicitly
revalidates the read data cache, and flushes the write data cache.
But beware that for fcntl() locking to do its thing, don't use the
`nolock' mount option to mount the NFS directory, and you have to have a
working statd on the client and lockd on the server (the kernel provides
lockd on the client side, and on the kernel side if you use knfsd).
statd is available as part of H.J.Lu's knfsd package.
You can also turn off a certain amount of NFS client cacheing: try these
mount options `acregmin=0,acregmax=0' to disable attribute cacheing of
files. To do the same for directories as well (but I don't think you
need this): `acdirmin=0,acdirmax=0,acregmin=0,acregmax=0'. But beware
this can still mean you use old cached data in the data file if your
transactions may take less then 1 second.
Summary:
1. Your program should work fine if you use fcntl() for locking
instead of linking to a lockfile, and your NFS is setup so that
works.
2. stat() the new link to get up to date n_link info, but beware that
the datafile can still contain old, locally cached data.
3. You can also use mount options to disable attribute cacheing.
I don't know of any way to disable data cacheing, which is a
problem if transactions may take less then 1 second, as new data
isn't detected for 60 seconds if it is written less then 1 second
after the last data written.
Enjoy,
-- Jamie
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/