[go: up one dir, main page]

|
|
Log in / Subscribe / Register

Checkpoint/restart in user space

By Jonathan Corbet
October 29, 2013

2013 Kernel Summit
There has long been a desire for the ability to checkpoint a set of processes (save their state to disk) and restore those processes at some future time, possibly on a different system. For almost as long, Linux has lacked that feature, but those days are coming to an end. Pavel Emelyanov led a session during the 2013 Kernel Summit's plenary day to update the audience on the status of this functionality.

Pavel started with the history of this feature. Early attempts to add checkpoint/restart went with an entirely in-kernel approach. The resulting patch set was large and invasive; it looked like a maintenance burden and never got much acceptance from the broader development community. Eventually, some developers realized that the APIs provided by the kernel were nearly sufficient to allow the creation of a checkpoint/restore mechanism that ran almost [Pavel Emelyanov] entirely in user space. All that was needed was a few additions here and there; as of the 3.11 kernel, all of those additions have been merged and user-space checkpoint/restart works. Live migration is supported as well.

Pavel had some requests for developers designing kernel interfaces in the future. Whenever new resources are added to a process, he asked, please provide a call to query the current state. A classic example is timers; developers added interfaces to create and arm timers, but nothing to query them, so the checkpoint/restart developers had to fill that in. He also requested that any user-visible identifiers exposed by the kernel not be global; instead, they should be per-process identifiers like file descriptors. If identifiers must be global — he gave process IDs as an example — it will be necessary to create a namespace around them so that the same identifiers can be restored with a checkpointed process.

Now that the basic functionality works, some interesting new features are being worked on. One of these checkpoints all processes in the system, but keeps the contents of their memory in place. It then boots into a new kernel with kexec and restores the processes quickly, using the saved memory whenever possible. This, Pavel said, is the path toward a seamless kernel upgrade.

Andrew Morton expressed his amazement that all of this functionality works, especially given that the checkpoint/restore developers added very little in the way of new kernel code. Is there, he asked, anything that doesn't work? Pavel responded that they have tried a lot of stuff, including web servers, command-line utilities, huge high-performance computing applications, and more. Almost everything will checkpoint and restore just fine.

Andrew then refined his question: could you write an application that is not checkpointable? The answer is "yes"; the usual problem is the use of external resources that cannot be checkpointed. For example, Unix-domain sockets where one end is held by a process that is not being checkpointed will block things; syslog can apparently be a problem in this regard. Work is being done to solve this problem for a set of known services; the systemd folks want it, Pavel added. Unknown devices are another problematic resource; there is a library hook that can be used to add support for specific devices if their state can be obtained and restored.

Beyond that, though, this long-sought functionality seems to work at last.

[Next: A kernel.org update].

Index entries for this article
KernelCheckpointing
ConferenceKernel Summit/2013


to post comments

Checkpoint/restart in user space

Posted Oct 30, 2013 18:41 UTC (Wed) by nix (subscriber, #2304) [Link] (9 responses)

Yeah. I imagine it would be problematic if you checkpointed something using a network connection and the far end timed it out while it was checkpointed. (e.g., what happens if you checkpoint an X client? The X server will presume it dead after a fairly short time...)

The 'checkpoint everything, kexec, and uncheckpoint' mechanism looks awesome. The sooner it works fully, the happier I'll be. It's a big hammer, unlike, say, ksplice, but it should work well enough. (Kernel modesetting probably helps here, since it is at least possible to checkpoint things like an X server, where before KMS that was a hopeless dream.)

Checkpoint/restart in user space

Posted Oct 30, 2013 19:44 UTC (Wed) by raven667 (subscriber, #5198) [Link] (7 responses)

I was thinking about that and it seems a simple solution to just let the TCP connection be immediately reset by the remote side, the application should have some way of handling that in any case, checkpointing or not.

Checkpoint/restart in user space

Posted Oct 30, 2013 19:48 UTC (Wed) by dlang (guest, #313) [Link] (6 responses)

the thing is that if the process isn't asleep for very long, and the IP moves with the process, the TCP connection may be able to recover.

Checkpoint/restart in user space

Posted Oct 30, 2013 21:33 UTC (Wed) by raven667 (subscriber, #5198) [Link] (5 responses)

I should have been more clear what I meant by the "remote side", I meant the remote end of the TCP connection. If TCP can't recover it's no real loss because the network stack and application can handle dropped TCP connections without trouble. You see the same thing when devices suspend and resume on another network segment.

Checkpoint/restart in user space

Posted Nov 1, 2013 22:24 UTC (Fri) by nix (subscriber, #2304) [Link] (4 responses)

the network stack and application can handle dropped TCP connections without trouble
My X server disagrees -- or rather its connected clients do. X connection drops, oops you're dead.

For some sorts of TCP-connected application what you say is true -- but not for all, by any means.

Checkpoint/restart in user space

Posted Nov 2, 2013 17:59 UTC (Sat) by Oddscurity (guest, #46851) [Link] (1 responses)

Is there something like tmux or screen for X, where you can pick up a session after a disconnection?

Checkpoint/restart in user space

Posted Nov 2, 2013 21:04 UTC (Sat) by mmorrow (guest, #83845) [Link]

Checkpoint/restart in user space

Posted Nov 2, 2013 19:23 UTC (Sat) by raven667 (subscriber, #5198) [Link] (1 responses)

I understand. What I meant to say is that a dropped tcp connection is a common error with expected behavior, as opposed to having some sort of obscure error in the middle of you application in some operation that normally doesn't fail and no one handles errors for.

Checkpoint/restart in user space

Posted Nov 3, 2013 19:59 UTC (Sun) by nix (subscriber, #2304) [Link]

Oh, agreed, it's better than a sudden bus error or whatever.

Checkpoint/restart in user space

Posted Nov 26, 2013 21:23 UTC (Tue) by scientes (guest, #83068) [Link]

But unlike ksplice you can upgrade your kernel.

Booting a kernel into a checkpointed user-space

Posted Oct 31, 2013 10:21 UTC (Thu) by ncm (guest, #165) [Link]

Is this is a better way to handle hot-plugging? I understand that this wouldn't meet needs for continuous-availability and power-management use cases, but maybe it would allow solutions for those cases to be simpler if they didn't need to satisfy everyone else.

Checkpoint/restart in user space

Posted Nov 7, 2013 2:46 UTC (Thu) by heijo (guest, #88363) [Link]

Can it checkpoint applications using OpenGL direct rendering?

Can it checkpoint KVM?

Checkpoint/restart in user space

Posted Nov 25, 2013 17:06 UTC (Mon) by dowdle (subscriber, #659) [Link]

Here is a recording of Pavel's presentation. The audio is a little low.

https://www.youtube.com/watch?v=R2Net9eItBc


Copyright © 2013, Eklektix, Inc.
This article may be redistributed under the terms of the Creative Commons CC BY-SA 4.0 license
Comments and public postings are copyrighted by their creators.
Linux is a registered trademark of Linus Torvalds