... (yes, transparent means that
it does not require LD_PRELOAD or collaboration of the application!
nor does it require userspace virtualizations of so many things
already provided by the kernel today), more generic, more flexible,
provides more guarantees, cover more types or states of resources,
and can perform significantly better.
I still haven't understood why you object to the DMTCP use of LD_PRELOAD.
How will the user app ever know that we used LD_PRELOAD, since we remove
LD_PRELOAD from the environment before the user app libraries and main
can begin? And, if you really object to LD_PRELOAD, then there are
other ways to capture control. Similarly, I'll have to understand better
I don't object to it per se - it's actually pretty useful oftentimes.
But in our context, it has limitations. For example, it does not
cover static applications, nor apps that call syscalls directly
using int 0x80.
For static apps, we would use other interposition techniques. And yes,
we haven't implemented support of static apps so far, because our
user base hasn't asked for it. We do handle apps that use the
syscall system call to make system calls. We don't handle apps
that directly use "int 0x80". Again, there are ways to do this, but
our user base hasn't asked for it.
In general, please keep in mind the principles that you rightly had
to remind me of in a previous post. :-) Our two pieces of work are coming
from two different directions with two different visions. Linux C/R wants
to be so transparent that no user app can ever detect it. DMTCP wants to be
transparent enough that any reasonable use case is covered.
In particular, DMTCP considers distributed computations to be equally
valid use cases for the core DMTCP C/R. I also agree that Linux C/R can be
extended to cover distributed apps -- either through userland extensions,
or maybe with techniques like in your excellent CLUSTER-2005 paper.
Also, it conflicts with LD_PRELOAD possibly needed
for other software (like valgrind) - for which again you would need
yet another per-app wrapper, at the very least.
DMTCP does not conflict with the fact that valgrind uses LD_PRELOAD.
We add dmtcphijack.so to the beginning of LD_PRELOAD before the user app
starts. We then remove it before the app really starts. The LD_PRELOAD
requests of valgrind continue to be honored. It all works.
what you mean by the _collaboration of the application_. DMTCP operates
on unmodified application binaries.
I mean that the applications needs to be scheduled and to run to
participate in its own checkpoint. You use syscall interposition
and signals games to do exactly that - gain control over the app
and run your library's code. This has at least three negatives:
first, some apps don't want to or can't run - e.g. ptraced, or
swapped (think incremental checkpoint: why swap everything in ?!);
Second, the coordination can take significant time, especially if
many tasks/threads and resources are involved; Third, it modifies
the state of the app - if something goes wrong while you use c/r
to migrate an app, you impact the app.
If it helps, then think of a wrapper as just another function,
that calls an inner function. Object-oriented programming uses this
principle all the time. Similarly, the glibc wrapper around a kernel
API is just one more of these functions. Another way to view this is
through the idea of layers. Each layer of the software receives a call
from the layer above and may call to the next layer below. As you're
already aware, this is a basic principle of O/S design, and so
the O/S is full of wrappers. We're just inserting one more layer ---
this time between the user app and the glibc layer.
I still don't fully understand what you mean by "collaboration", but
it sounds like your definition reduces to the the use of system call
wrappers. In that case, I agree that if DMTCP were not allowed to use
system call wrappers, then DMTCP would fall apart. Aside from that
almost tautology, I don't understand why system call wrappers are inherently
bad. Glibc puts system call wrappers around almost every kernel system call.
Glibc even reserves two signals solely for its own use.
Basically, if _transparent_ means
that one is not allowed to use anything at all from userland, then I
agree with you that no userland checkpointing can ever be transparent.
But, I think that's a biased definition of _transparent_. :-)
"Transparent" c/r means "invisible" to the user/apps, i.e. that
you don't restrict the user or the app in what they do and how
they do it.
Did you ever try to 'ltrace skype' ? there exists useful and
popular software that doesn't like being spied after...
We have not tried to 'ltrace skype'. But ltrace is using PTRACE.
Note that DMTCP does not use PTRACE. I imagine the more interesting question
is if we ever tried 'dmtcp_checkpoint skype'. No, we have not, but
it sounds like an interesting experiment. We'd love to do it, and
discuss with you whatever we learn. In the offline discussion, perhaps
we can take a shortcut and have you describe the skype tricks to us,
so that we can give you a quick first guess.
Anyway, there's one other obvious issue with skype for both Linux C/R
and DMTCP. Skype is talking to a remote app that is probably not under
checkpoint control.
And even if both ends are under checkpoint control,
Skype is probably not a good use case for C/R, but if it were, it might
indeed be a difficult problem. (I'd have to think about it.)
As before, remember that we are talking about two different approaches:
- in-kernel C/R and capturing every possible application;
- userland C/R and covering the actual use cases that one finds in practice