Re: Checkpoint/restart (was Re: [PATCH 0/4] - v2 - Object creationwith a specified id)
From: Serge E. Hallyn
Date: Fri Jul 18 2008 - 12:18:56 EST
Quoting Oren Laadan (orenl@xxxxxxxxxxxxxxx):
> Serge E. Hallyn wrote:
>> Quoting Dave Hansen (dave@xxxxxxxxxxxxxxxxxx):
>>> On Wed, 2008-07-09 at 18:58 -0700, Eric W. Biederman wrote:
>>>> In the worst case today we can restore a checkpoint by replaying all of
>>>> the user space actions that took us to get there. That is a tedious
>>>> and slow approach.
>>> Yes, tedious and slow, *and* minimally invasive in the kernel. Once we
>>> have a tedious and slow process, we'll have some really good points when
>>> we try to push the next set of patches to make it less slow and tedious.
>>> We'll be able to describe an _actual_ set of problems to our fellow
>>> kernel hackers.
>>> So, the checkpoint-as-a-corefile idea sounds good to me, but it
>>> definitely leaves a lot of questions about exactly how we'll need to do
>>> the restore.
>> Talking with Dave over irc, I kind of liked the idea of creating a new
>> fs/binfmt_cr.c that executes a checkpoint-as-a-coredump file.
>> One thing I do not like about the checkpoint-as-coredump is that it begs
>> us to dump all memory out into the file. Our plan/hope was to save
>> ourselves from writing out most memory by:
>> 1. associating a separate swapfile with each container
>> 2. doing a swapfile snapshot at each checkpoint
>> 3. dumping the pte entries (/proc/self/)
>> If we do checkpoint-as-a-coredump, then we need userspace to coordinate
>> a kernel-generated coredump with a user-generated (?) swapfile snapshot.
>> But I guess we figure that out later.
> I'm not sure how this approach integrates with (a) live migration (and
> the iterative process of sending over memory modified since previous
> iteration), and (b) incremental checkpoint (where except for the first
> snapshot, additional snapshots only save what changed since the previous
Oh, well I was seeing them as pretty orthogonal actually. The reason is
that my checkpoint-as-a-coredump file would NOT include memory contents.
I'm still hoping that we can lock a container into its own nfs-mounted
swapfile, and take a snapshot of that at checkpoint. Let the snapshot
solution take care of the incremental snapshot.
Attached are one kernel and one cryo patch, purely for a toy
implementation. It doesn't even *quite* work yet. Here are the notes
I wrote when I put it down wednesday afternoon:
The kernel patch impelments (only for x86_32) sys_checkpoint. You
can create a checkpoint with the following test program:
ret = syscall(327, "output.kckpt");
printf("ret was %d\n", ret);
if (ret == -1)
Make output.kckpt executable and run it. Works fine.
With the cryo patch, you can do:
./cr -p <sleeppid> -f o1.bin
and you'll see <sleeppid.kckpt> created. Make that
executable and run it, and you'll see it runs as though
it were the original sleep program.
On the other hand, two problems with using cryo:
1. the checkpointed app segfaults
2. if you restart with "cr -r -f o1.bin" it
Matt has already pointed out some coding errors so *really* it's
just to show one way we could integrate a binary format handler
for partial restart with more userspace control. I'd be too
embarassed to send it out, but figure I should send it out before