Implement Snapshot/Restore API for workload pods in Kata - #467
Open
michaelxu2288 wants to merge 8 commits into
Open
Implement Snapshot/Restore API for workload pods in Kata#467michaelxu2288 wants to merge 8 commits into
michaelxu2288 wants to merge 8 commits into
Conversation
…eparation In its current form, the only use of Snapshot/Restore api for a given hypervisor is for VM templating, configured via the BootToBeTemplate and BootFromTemplate configs. Replace these hypervisor-level configs with a new FileBackedMemory abstraction so that other usecases for SaveVM/RestoreVM can use it.
…andbox) Signed-off-by: Michael Xu <t-michaelxu@microsoft.com>
`kata-runtime restore --from <snapshot>` (plus the shim's restore mode) rebuilds a Sandbox from a snapshot dir, boots the VM from its saved memory (copy-on-write), adopts the live agent + hypervisor, and runs it as a real kata-managed sandbox (own shim, lifecycle, management API). Proves the guest RAM survives a snapshot; the clone is not yet a networked pod.
…ainerd and CNI Drive restore from a pod annotation (`restore_from`) instead of the hand-run CLI: RunPodSandbox dispatches to RestoreSandbox, and the restored clone comes up as a real containerd-managed, CNI-networked, isolated pod. Adopt the pod's CNI netns, hotplug the CNI NIC, re-IP the guest NIC by MAC, install routes, and neutralize the frozen snapshot NIC so the clone shares no network identity with its source.
Restore a Kata snapshot as a real, networked, isolated Kubernetes pod through the `io.katacontainers.restore-from` pod annotation on the normal CRI path. The restored VM keeps its single guest NIC and rebinds it to the fresh CNI tap via CLH `vm.restore` net_fds (no second NIC, no ghost neutralization, no CLH change). The VM restores paused and resumes at workload StartContainer; the sole workload is adopted host-only (no re-create in the guest); the guest NIC identity is replaced to the exact target CNI MAC/IP/routes and verified before a TAP-up/no-redirect fence is activated. Fail-only: any restore anomaly aborts and reaps the VM. Squash of the net_fds refactor; drops the standalone restore CLI launcher and the perf/timing instrumentation.
A restored sandbox reconnects to an already-live snapshotted agent, so the first hybrid-vsock trivial handshake reliably hits the known kernel vsock race where the agent's first reply is dropped. The dialer then waits the full hardcoded 10s handshakeTimeout before commonDialer redials (which succeeds instantly), making restore ~10s slower than the ~130ms of real work. To work around this, improve hybridVSockDialer so that the timeout for connection attempts starts small and grows to the 10s maximum. This means early CONN drops are recovered from quickly.
The backoff loop in hybridVSockDialer derives a per-attempt context via context.WithTimeout(ctx, timeout). Some callers reach the dialer with a nil context - notably the VM factory path, whose kataAgent connects with a nil k.ctx. The previous dialer ignored the context, so a nil parent was harmless; the backoff version dereferences it and panics with cannot create context
There was a problem hiding this comment.
Pull request overview
This PR introduces a snapshot/restore workflow for Kata “workload pods”, spanning: (1) hypervisor-level VM snapshot/restore primitives, (2) runtime sandbox/container bookkeeping to adopt a restored guest, (3) network “fencing” to safely re-bind host networking to a restored guest NIC, and (4) shim + kata-runtime CLI plumbing to trigger snapshotting and to dispatch restore based on an OCI annotation.
Changes:
- Add VM snapshot/restore APIs across hypervisors (notably Cloud Hypervisor + QEMU) and plumb them through virtcontainers VM/Sandbox layers.
- Implement sandbox restore flow (
RestoreSandbox,RestoreContainer,FinalizeRestoreNetwork) including guest identity remapping and TC-based network fencing. - Add shim management
/snapshotendpoint and akata-runtime snapshotCLI, plus an annotation (io.katacontainers.restore-from) to trigger restore.
Reviewed changes
Copilot reviewed 41 out of 41 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| src/runtime/virtcontainers/vm.go | Adds NewVMFromSnapshot, RestoreNetEndpoints, and snapshots via Save(snapshotDir) / restore via hypervisor. |
| src/runtime/virtcontainers/vm_test.go | Updates VM tests for new Save(dir) signature and basic NewVMFromSnapshot coverage. |
| src/runtime/virtcontainers/virtframework.go | Updates virtframework hypervisor stubs for new Save/Restore signatures. |
| src/runtime/virtcontainers/veth_endpoint.go | Adds restore-network “fence” attach path to prepare TAP FDs without bridging/hotplug. |
| src/runtime/virtcontainers/stratovirt.go | Adds Save/Restore methods (currently stubbed). |
| src/runtime/virtcontainers/sandbox.go | Tracks restoreNetFence; adds Sandbox PauseVM/SaveVM/ResumeVM helpers; restores fence cleanup in removeNetwork. |
| src/runtime/virtcontainers/restore.go | New restore implementation: rekeys persist state, restores VM from snapshot, adopts containers, and finalizes network identity. |
| src/runtime/virtcontainers/remote.go | Updates remote hypervisor interface implementation for Save/Restore. |
| src/runtime/virtcontainers/qemu.go | Reworks template-related logic into FileBackedMemory; implements QEMU Save/Restore via migration device state file in snapshot dir. |
| src/runtime/virtcontainers/qemu_test.go | Updates QEMU tests for FileBackedMemory and SaveVM(dir). |
| src/runtime/virtcontainers/qemu_amd64.go | Removes vmFactory/template flag plumbing. |
| src/runtime/virtcontainers/qemu_amd64_test.go | Removes vmFactory-related CPU model test cases. |
| src/runtime/virtcontainers/pkg/vcmock/types.go | Extends vcmock sandbox struct with restore hooks. |
| src/runtime/virtcontainers/pkg/vcmock/sandbox.go | Implements vcmock methods for restore and VM pause/save/resume. |
| src/runtime/virtcontainers/pkg/annotations/annotations.go | Adds RestoreFrom annotation key. |
| src/runtime/virtcontainers/pkg/agent/protocols/client/client.go | Refactors HybridVsock dialer to use ctx-aware retry windows and bounded handshake reads. |
| src/runtime/virtcontainers/persist/api/config.go | Removes template-specific persisted fields (MemoryPath/DevicesStatePath/BootToBeTemplate/BootFromTemplate). |
| src/runtime/virtcontainers/persist.go | Drops template-specific fields from persisted config load/dump paths. |
| src/runtime/virtcontainers/network_linux.go | Adds restore fencing helpers: prepare/activate/cleanup TC redirects; validates TC filter model on restore. |
| src/runtime/virtcontainers/mock_hypervisor.go | Updates mock hypervisor for new Save/Restore interface. |
| src/runtime/virtcontainers/mock_hypervisor_test.go | Updates mock hypervisor tests for SaveVM(dir). |
| src/runtime/virtcontainers/kata_agent.go | Switches agent RPC container/exec IDs to restored guest IDs via c.agentID() / c.guestExecID(). |
| src/runtime/virtcontainers/interfaces.go | Extends VCSandbox interface with restore + VM pause/save/resume methods. |
| src/runtime/virtcontainers/hypervisor.go | Introduces FileBackedMemoryConfig; changes hypervisor interface to SaveVM(dir) and adds RestoreVM. |
| src/runtime/virtcontainers/hypervisor_config_linux.go | Removes template-config validation hook from config validation. |
| src/runtime/virtcontainers/hypervisor_config_linux_test.go | Removes template-validation unit tests. |
| src/runtime/virtcontainers/fc.go | Adds Save/Restore methods (currently stubbed). |
| src/runtime/virtcontainers/fc_test.go | Updates firecracker tests for SaveVM(dir). |
| src/runtime/virtcontainers/factory/template/template_linux.go | Switches factory template to FileBackedMemory, snapshots to directory, uses NewVMFromSnapshot, and patches CLH snapshot sharing mode. |
| src/runtime/virtcontainers/factory/factory_linux.go | Resets FileBackedMemory in factory config reset. |
| src/runtime/virtcontainers/container.go | Adds agentID() / guestExecID() helpers for restored guest ID mapping. |
| src/runtime/virtcontainers/clh.go | Adds CLH snapshot timeout sizing; implements RestoreVM; adds PatchCLHSnapshotMemoryPrivate; implements net_fds restore staging and SCM_RIGHTS restore request. |
| src/runtime/virtcontainers/clh_test.go | Updates CLH tests for SaveVM(dir) and changed snapshot destination URL behavior. |
| src/runtime/pkg/containerd-shim-v2/start.go | Adds restored-sandbox start behavior: defer pause task IO arming; finalize restore network and resume at workload start. |
| src/runtime/pkg/containerd-shim-v2/shim_management.go | Adds /snapshot handler; implements pause/save/snapshot/resume flow and snapshot manifest/config patching. |
| src/runtime/pkg/containerd-shim-v2/service.go | Tracks restoredSandbox state. |
| src/runtime/pkg/containerd-shim-v2/create.go | Dispatches restore via annotations.RestoreFrom; adopts workload container via RestoreContainer. |
| src/runtime/pkg/containerd-shim-v2/container.go | Adds restorePauseIOArmPending flag to container state. |
| src/runtime/cmd/kata-runtime/snapshot.go | Adds kata-runtime snapshot create/delete commands. |
| src/runtime/cmd/kata-runtime/main.go | Registers the snapshot CLI command. |
| src/agent/src/netlink.rs | Adds restore-replace interface operation keyed off a raw_flags bit; improves route EEXIST handling. |
Comments suppressed due to low confidence (1)
src/runtime/virtcontainers/fc.go:958
- RestoreVM returns nil even though Firecracker restore is not implemented. Returning success here will leave the VM un-restored and likely cause confusing follow-on failures. Return an explicit not-supported error.
func (fc *firecracker) RestoreVM(ctx context.Context, snapshotDir string) error {
// Firecracker does not support snapshot/restore in this implementation.
return nil
}
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+639
to
643
| memPath := clh.config.FileBackedMemory.Path | ||
| // Double-check that the memory file exists before using it in the VM config, to avoid hitting a less clear error from cloud hypervisor when it tries to access the non-existing memory file. | ||
| if _, err := os.Stat(memPath); os.IsNotExist(err) { | ||
| return fmt.Errorf("memory file %s does not exist", memPath) | ||
| } |
Comment on lines
+1806
to
1809
|
|
||
| if err := os.WriteFile(configPath, modifiedConfig, 0644); err != nil { | ||
| return fmt.Errorf("failed to write modified snapshot config: %w", err) | ||
| } |
There was a problem hiding this comment.
Yes, the standard should be 0600 permission bits for these artifacts.
Comment on lines
+399
to
+404
| if err := s.doSnapshot(context.Background(), destDir); err != nil { | ||
| logger.WithError(err).Error("snapshot failed") | ||
| w.WriteHeader(http.StatusInternalServerError) | ||
| w.Write([]byte(err.Error())) | ||
| return | ||
| } |
Comment on lines
+394
to
+397
| destDir := strings.TrimSpace(string(body)) | ||
| if destDir == "" { | ||
| destDir = filepath.Join(SnapshotBaseDir, s.id) | ||
| } |
There was a problem hiding this comment.
At the very least we should be doing as the bot suggests here and normalizing the path. There are all sorts of ways such a mechanism can be used maliciously, such as with /normal/looking/path/../../../root/super-secret/not-your-snapshot/
Comment on lines
+207
to
+212
| // snapshot storage is caller-owned, so Kata trusts the annotation path. | ||
| snapDir := filepath.Clean(restoreFrom) | ||
| if !filepath.IsAbs(snapDir) { | ||
| snapDir = filepath.Join(SnapshotBaseDir, snapDir) | ||
| } | ||
| shimLog.WithFields(logrus.Fields{"restore-from": restoreFrom, "snapshot-dir": snapDir, "sandbox": r.ID}).Info("dispatching restore-from-snapshot") |
There was a problem hiding this comment.
Yea, in an attempt to make the API permissible/flexible for consumers of this API we have introduced a big security problem. Maybe we'll have to go back to named snapshots under a well-known /run/vc/vm/snapshots/ dir. This centralization would have the added benefit of allowing us to set up a tmpfs mount on that directory, meaning snapshots are cached in RAM.
Comment on lines
+950
to
+953
| func (fc *firecracker) SaveVM(snapshotDir string) error { | ||
| // Firecracker does not support snapshot/restore in this implementation. | ||
| return nil | ||
| } |
Comment on lines
+1114
to
1122
| func (s *stratovirt) SaveVM(snapshotDir string) error { | ||
| // StratoVirt does not support snapshot/restore in this implementation. | ||
| return nil | ||
| } | ||
|
|
||
| func (s *stratovirt) RestoreVM(ctx context.Context, snapshotDir string) error { | ||
| // StratoVirt does not support snapshot/restore in this implementation. | ||
| return nil | ||
| } |
Comment on lines
29
to
36
| } else if conf.ImagePath == "" && conf.InitrdPath == "" { | ||
| return fmt.Errorf("Missing image and initrd path") | ||
| } else if conf.ImagePath != "" && conf.InitrdPath != "" { | ||
| return fmt.Errorf("Image and initrd path cannot be both set") | ||
| } | ||
|
|
||
| if err := conf.CheckTemplateConfig(); err != nil { | ||
| return err | ||
| } | ||
|
|
||
| if conf.NumVCPUsF == 0 { | ||
| conf.NumVCPUsF = defaultVCPUs |
|
@michaelxu2288 please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.
Contributor License AgreementContribution License AgreementThis Contribution License Agreement (“Agreement”) is agreed to by the party signing below (“You”),
|
Camelron
reviewed
Jul 27, 2026
| // names the source memory path (e.g. /run/vc/vm/template/memory) outside destDir. | ||
| // On restore CLH opens that path to back the memory zone, so a relocated snapshot | ||
| // (or a GC'd source) would fail. Pointing it at the in-dir memory-ranges makes the | ||
| // snapshot dir's memory self-contained. (Disks are not rewritten here; that is |
There was a problem hiding this comment.
This comment reads as overly AI-driven. It compulsively talks about future concerns such as Disks when we just want to concisely describe the actual codepath as-written.
Camelron
reviewed
Jul 27, 2026
| }, | ||
| } | ||
|
|
||
| var deleteSnapshotCommand = cli.Command{ |
There was a problem hiding this comment.
This is probably the most important cleanup item. This is essentially a built-in foot-nuke for kata that allows you to rm -rf any directory on the system.
An example of foot-nuking:
kata-runtime snapshot delete --path /
would recursively delete the entire OS with no survivors.
A happy middle-ground for the PoC can be to only look in --path for the specific named artifacts [memory-ranges, persist.json, state.json, etc]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds snapshot and restore API for Kata workload pods.
A snapshot captures a running pod's VM to disk. Running
kata-runtime snapshotagainst a live Cloud Hypervisor sandbox writes the guest memory into a chosen snapshot directory and resumes it, so the source pod only stops for the duration of the save. The snapshot artifact holds the entire guest state which lets the same process be brought back later on another pod using COW.Restore is driven entirely from a pod annotation. Adding
io.katacontainers.restore-fromto a normal pod spec routes RunPodSandbox through the custom restore path in this PR instead of a cold boot, so the restored clone comes up as an ordinary containerd-managed, CNI-networked pod. The VM boots from the snapshot's COW, which leaves the snapshot file untouched and lets several clones share one snapshot, and the frozen process resumes exactly where it left off. the VM comes back paused and only resumes once the workload reaches StartContainer.
Because the clone must not answer on the source pod's IP, restore gives it a fresh network identity. the VM keeps the single guest NIC from the snapshot and rebinds it to the new pod's CNI tap through Cloud Hypervisor's
vm.restorenet_fds path, which passes the host-side tap file descriptors and leaves the guest device and the CLH binary unchanged. After resume, the guest NIC is reprogrammed to the target CNI's MAC, IP, and then read back to confirm the change landed before any traffic is allowed.The entire restore path is fail-only which means any inconsistency during restore aborts and reaps the VM immediately and throws a failure message.