Re: Process Migration on Linux - Impossible?

Russell Coker - mailing lists account (bofh@snoopy.virtual.net.au)
Sun, 28 Sep 97 13:15:29 +1100


>Say we want to migrate process X. The daemon talks to the daemon at the
>target machine (I'm not concerned with a wonderfull selection of a target,
>as that's load _balancing_, not just load _sharing_), and the daemon at
>the target convinces the kernel to create a new process, which will have
>special needs, as all it's files _and_ code will be on another machine.

The files/code etc don't necessarily have to be on another machine. If
you have a group of machines with NFS mounts for all significant file
systems then the process can just be loaded from the file with the same
name on the destination machine. For shared libraries IFF all machines
have the same versions installed then loading from the local libraries on
the hard drive instead of over the network would be a huge win. With the
way sym-links are used in linux such as the following: lrwxrwxrwx 1 root
root 13 Jun 15 01:09 libc.so.6 -> libc-2.0.4.so
An application could just load to /lib/libc.so.6 and it could then move
to any machine with a file named /lib/libc-2.0.4.so. If that file isn't
available then you could refuse to allow it to migrate to that machine.

As for user files if all machines had the same /home and /var/spool/mail
directories then the same scheme could work. A potentially significant
problem is /tmp. For example when you run a compiler which has cpp writing
to /tmp/something which the main compiler will expect to read then it will
be difficult to migrate things (NFS mounted /tmp is a bad thing).
The same thing would work if you had a cluster of machines with the
omirrd running (when omirrd gets fixed and is runnable again).

>I intend to alter all the f_ops, mm_ops, whetever_ops structures of this
>process, so that whenever it faults on a page, the routines on my kernel
>module get called, and so it talks to the daemon, which talks to the
>daemon on the source machine, which talks to the kernel on the source,
>which will find da damn page and send it back all the way through.

I think it might be easier to just move the data across as one operation
and have the executable loaded from file on the target machine. Moving a
page at a time will make VM management very tricky.

>File ops will have a similar treatment, but then there is some other
>concerns, like IPC. Rerouting signals is simple, and so are messages, but
>shared memory, for instance, would be a major pain in the ass, if not
>impossible at all. Simple tweaks like having all processes involved to
>fault on the shared page when it's accessed and then distribute the page
>(if someone faulted on it) to the other machine on the next schedule()
>would maybe work, but looks horrible.

I suggest that you just not support shared memory. If the process has
shared memory then it's not distributable. In version 2.0 after you've
graduated you can support this. ;-)

>Another great job would have to be put to redirecting all syscalls which
>are not as simple as a getpid(). Maybe they can be redirected some way
>similar to the page faults. Also obvious is the fact that this migrated
>process will be in a state where it won't get on the run queue, or things
>will get funny. I don't know if it's better to create a new state, or use
>a similar one such as UNINTERRUPTIBLE.

What do you really have to redirect apart from signals and access to
/dev/*?

>Now for the real thing: Does anyone consider this project so improbable as
>to not even a very simple "Hello World!" be able to migrate?

That depends, is "hello world" statically linked? ;-)

-- 
-----------------------------------------------------------
In return for "mailbag contention" errors from buggy Exchange
servers I'll set my mail server to refuse mail from your domain.
The same response applies when a message to a postmaster
account bounces.
"Russell Coker - mailing lists account" <bofh@snoopy.virtual.net.au>
-----------------------------------------------------------