Skip to content

runtime-rs: Add support for OpenVMM - #466

Open
sprt wants to merge 11 commits into
msft-mainfrom
sprt/msft-main-ovmm-oop
Open

runtime-rs: Add support for OpenVMM#466
sprt wants to merge 11 commits into
msft-mainfrom
sprt/msft-main-ovmm-oop

Conversation

@sprt

@sprt sprt commented Jul 24, 2026 •
edited
Loading

Copy link
Copy Markdown

This is a backport of my upstream PR into msft-main.

Commits in order:

  • The upstream PR is contained in the first 3 commits (with some tweaks to adapt to msft-main).
  • 2 commits to support VFIO passthrough - please review carefully.
  • 1 commit to port the MSFT resource management behavior to OpenVMM - please review carefully.
  • + commits to onboard the OpenVMM build to node-builder and fix the fork CI.
    • You'll notice these are fixup! commits so that they'll be trivial to squash during the next msft-main rebase using git rebase --autosquash

Testing:

  • I can create a busybox pod after building/deploying with make -C tools/osbuilder/node-builder/azure-linux USE_OPENVMM=yes USE_RUNTIME_RS=yes clean all deploy
  • Testing VFIO passthrough is out of scope - Jocelyn will do this in an ulterior PR

Note: When passing BUILD_TYPE=debug to the above make command (setting all 3 enable_debug=true configs), pod creation fails - debugging now.

sprt added 3 commits July 24, 2026 11:08
@sprt
runtime-rs: add support for OpenVMM
1516fb0 
* Adds a new backend to runtime-rs where Kata talks to the OpenVMM process
  via protobufs over ttrpc.
* The protobufs are vendored from the OpenVMM repo via a new make target
  and the Rust bindings are autogenerated when `cargo build` runs, all matching
  prior art.
* Introduced a new config file: configuration-openvmm-azure-runtime-rs.toml

Generated-By: GitHub Copilot
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
Original-Commit: 4caf4ea
Original-Commit-Date: 2026-07-22
@sprt
ci: add testing for OpenVMM
50b7231 
* Introduced 2 new CIs:
  * kata-containers-ci-on-push / run-k8s-tests-on-aks / run-k8s-tests (cbl-mariner, openvmm-azure-runtime-rs, normal)
  * kata-containers-ci-on-push / run-k8s-tests-on-aks / run-k8s-tests (cbl-mariner, openvmm-azure-runtime-rs, small)
* OpenVMM is built from source following prior art.

Generated-By: GitHub Copilot
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
Original-Commit: 92014f7
Original-Commit-Date: 2026-07-22
@sprt
packaging: strip OpenVMM debug information
b6fffc4 
Remove DWARF debug information from the OpenVMM binary before packaging it.
This reduces the compressed OpenVMM asset by about 87 MiB and keeps the
combined release tarball below GitHub’s 2 GiB asset limit.

This should be reverted when the CI allows.

Generated-By: GitHub Copilot
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
Original-Commit: 161969b
Original-Commit-Date: 2026-07-22
@sprt
runtime-rs: support PCI functions in guest paths
0cadd99 
Represent each guest PCI path element as a device/function pair so
multifunction root ports can be reported to kata-agent without losing the
function number.

Keep function-zero formatting and constructors compatible. Reject nonzero
functions at the legacy add-swap RPC boundary instead of truncating them.

Generated-By: GitHub Copilot
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
@sprt
sprt force-pushed the sprt/msft-main-ovmm-oop branch 3 times, most recently from 086f30f to b1aadd3 Compare July 27, 2026 19:00
sprt added 4 commits July 27, 2026 17:27
@sprt
runtime-rs/openvmm: support cold-plug VFIO over TTRPC
90e8a62 
Cold-plug modern VFIO PCI devices through the external OpenVMM VM service.
Discover devices through CDI or the raw OCI specification, send full host
BDFs over TTRPC, and report deterministic function-aware guest PCI paths
back to the agent.

Reserve 16 multifunction root ports and use the GPU-sized MMIO apertures
from the in-process implementation. Legacy VFIO and post-launch VFIO
changes remain unsupported.

Reference-Commit: f12d92d

Reference-Commit: 4c19e86

Reference-Commit: 016bd39

Generated-By: GitHub Copilot
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
@sprt
runtime-rs/openvmm: apply resource management patches
f0ea3ba 
This ports the Microsoft resource management patches from f7f6cf4 and e5bf9cd
to OpenVMM.

Generated-By: GitHub Copilot
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
@sprt
fixup! node-builder: add initial node-builder recipes
bb1cb4a 
node-builder: force regenerating config files

It seems the upstream makefile doesn't regenerate the config if the generated
config exists, so remove it before rebuilding.

Generated-By: GitHub Copilot
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
@sprt
fixup! node-builder: add initial node-builder recipes
137782b 
* Onboard the new upstream configuration-clh-azure-runtime-rs.toml.
* Set the new upstream IMAGENAME_AZURE to kata-containers.img (instead
  of kata-containers-mariner.img) to preserve fork behavior.

Generated-By: GitHub Copilot
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
@sprt
sprt force-pushed the sprt/msft-main-ovmm-oop branch from b1aadd3 to 25f9bff Compare July 27, 2026 22:32
sprt added 3 commits July 27, 2026 17:38
@sprt
fixup! node-builder: add initial node-builder recipes
c393dad 
Extend node-builder to support OpenVMM.

Generated-By: GitHub Copilot
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
@sprt
fixup! ci: switch default branch to msft-preview
8fd921c 
Replace the stale msft-preview branch references with msft-main. We forgot this
follow-up when msft-main was reset to msft-preview.

Generated-By: GitHub Copilot
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
@sprt
fixup! ci: switch default branch to msft-preview
ca332cb 
This allows fixup! commits to make future rebases easier:
`git rebase --autosquash` will automatically squash such commits into the
corresponding earlier commits to keep the history clean.

Generated-By: GitHub Copilot
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
@sprt
sprt force-pushed the sprt/msft-main-ovmm-oop branch from 25f9bff to ca332cb Compare July 27, 2026 22:38
@sprt
sprt marked this pull request as ready for review July 27, 2026 22:39
Copilot AI review requested due to automatic review settings July 27, 2026 22:39
Copilot AI reviewed Jul 27, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR backports upstream work to add an OpenVMM hypervisor backend to runtime-rs (implemented as an external, out-of-process VMM controlled over TTRPC), plus follow-on commits to support VFIO passthrough, align Microsoft static resource-management behavior for OpenVMM, and wire the new hypervisor into packaging, node-builder, and CI for the msft-main branch.

Changes:

  • Add an OpenVMM hypervisor implementation for runtime-rs (TTRPC client/server bindings, process lifecycle wrapper, OpenVMM-specific PCIe topology/device management, config template, and feature gating).
  • Extend VFIO handling (CDI/Pod Resources + raw OCI fallback), and update PCI path representation to support multifunction devices (function != 0).
  • Update packaging, node-builder, and GitHub workflows to build/ship OpenVMM assets and to target msft-main instead of msft-preview.

Reviewed changes

Copilot reviewed 70 out of 71 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
versions.yaml Pins an OpenVMM repo + commit SHA under hypervisor assets.
tools/testing/gatekeeper/skips.py Changes default gatekeeper target branch to msft-main.
tools/packaging/static-build/openvmm/Dockerfile Adds a builder image for compiling OpenVMM from source.
tools/packaging/static-build/openvmm/build.sh Adds a wrapper to build OpenVMM via the new container builder.
tools/packaging/static-build/openvmm/build-static-openvmm.sh Implements the actual OpenVMM-from-source build, MSRV pinning, and artifact staging.
tools/packaging/scripts/lib.sh Adds get_openvmm_image_name() helper for builder image tagging.
tools/packaging/kata-deploy/shim-components.json Adds an openvmm-azure-runtime-rs shim component mapping.
tools/packaging/kata-deploy/local-build/Makefile Adds openvmm-tarball target plumbing.
tools/packaging/kata-deploy/local-build/kata-deploy-binaries.sh Adds install_openvmm() to build/install a static OpenVMM binary into kata-deploy outputs.
tools/packaging/kata-deploy/helm-chart/kata-deploy/values.yaml Adds Helm values for enabling openvmm-azure-runtime-rs.
tools/packaging/kata-deploy/helm-chart/kata-deploy/templates/runtimeclasses.yaml Adds default resource requests for the OpenVMM runtimeclass.
tools/packaging/kata-deploy/binary/src/utils/system.rs Adds OpenVMM shim to the rust shim list.
tools/packaging/kata-deploy/binary/src/artifacts/install.rs Adds OpenVMM shim recognition and maps it to the openvmm hypervisor binary; adds tests.
tools/osbuilder/node-builder/azure-linux/README.md Documents node-builder usage for OpenVMM + runtime-rs.
tools/osbuilder/node-builder/azure-linux/package_install.sh Installs CLH/OpenVMM runtime-rs configs side-by-side (optionally).
tools/osbuilder/node-builder/azure-linux/package_build.sh Adds OpenVMM toggles/flags and cleans up generated runtime-rs config outputs.
tools/osbuilder/node-builder/azure-linux/Makefile Exports USE_OPENVMM default.
tools/osbuilder/node-builder/azure-linux/common.sh Introduces OpenVMM config filenames and /usr/bin/openvmm location.
tools/osbuilder/node-builder/azure-linux/clean.sh Cleans runtime-go + runtime-rs debug config outputs more precisely.
tests/static-checks.sh Excludes vendored OpenVMM proto sources from license-header checks.
tests/spellcheck/kata-dictionary.txt Adds OpenVMM and vmservice to the spellcheck dictionary.
tests/hypervisor_helpers.sh Adds openvmm-azure-runtime-rs to the test hypervisor list.
src/runtime-rs/Makefile Adds OpenVMM hypervisor option, config generation, azure image naming, kubelet root dir var, and feature joining logic.
src/runtime-rs/crates/runtimes/virt_container/src/sandbox.rs Adds raw VFIO device discovery fallback + deduping + OpenVMM restore support.
src/runtime-rs/crates/runtimes/virt_container/src/lib.rs Registers OpenVMM config plugin and instantiates OpenVMM hypervisor when selected.
src/runtime-rs/crates/runtimes/virt_container/Cargo.toml Adds an openvmm feature wired to the hypervisor crate feature.
src/runtime-rs/crates/runtimes/Cargo.toml Exposes openvmm feature at the runtimes crate level.
src/runtime-rs/crates/resource/src/cpu_mem/swap.rs Validates PCI function==0 for swap PCI paths; adds unit tests.
src/runtime-rs/crates/resource/src/cpu_mem/initial_size.rs Adds tests covering static sizing defaults vs explicit resources.
src/runtime-rs/crates/hypervisor/src/utils.rs Adds a shared vCPU TID scanner used by CH/OpenVMM.
src/runtime-rs/crates/hypervisor/src/openvmm/vmm_instance.rs Implements OpenVMM external process wrapper + TTRPC client lifecycle + tests.
src/runtime-rs/crates/hypervisor/src/openvmm/protos/vmservice.proto Vendors OpenVMM vmservice proto definition.
src/runtime-rs/crates/hypervisor/src/openvmm/protos/README.md Documents proto vendoring/update process for OpenVMM vmservice.
src/runtime-rs/crates/hypervisor/src/openvmm/protos/google/protobuf/struct.proto Vendors protobuf dependency used by the OpenVMM proto.
src/runtime-rs/crates/hypervisor/src/openvmm/protos/google/protobuf/empty.proto Vendors protobuf dependency used by the OpenVMM proto.
src/runtime-rs/crates/hypervisor/src/openvmm/mod.rs Adds the OpenVMM hypervisor implementation and Persist integration.
src/runtime-rs/crates/hypervisor/src/openvmm/inner.rs Adds OpenVMM internal state and hotplug port bookkeeping.
src/runtime-rs/crates/hypervisor/src/openvmm/inner_hypervisor.rs Implements OpenVMM VM creation/topology, cold-plug VFIO assignment, and lifecycle RPCs.
src/runtime-rs/crates/hypervisor/src/openvmm/inner_device.rs Implements OpenVMM block hotplug and VFIO lifecycle constraints; adds tests.
src/runtime-rs/crates/hypervisor/src/lib.rs Wires OpenVMM module + exports OpenVMM hypervisor name behind feature flag.
src/runtime-rs/crates/hypervisor/src/device/topology.rs Updates PCIe topology helper to use new PciSlot::new() API.
src/runtime-rs/crates/hypervisor/src/device/pci_path.rs Refactors PciSlot/PciPath to support PCI function numbers; updates tests.
src/runtime-rs/crates/hypervisor/src/device/driver/vfio_device/mod.rs Exposes DeviceAddress needed for OpenVMM VFIO handling.
src/runtime-rs/crates/hypervisor/src/ch/inner_hypervisor.rs Reuses shared vCPU TID scanner for CH instead of local implementation.
src/runtime-rs/crates/hypervisor/Makefile Adds update-proto target to refetch pinned OpenVMM vmservice.proto.
src/runtime-rs/crates/hypervisor/Cargo.toml Adds optional protobuf dependency and openvmm feature gating.
src/runtime-rs/crates/hypervisor/build.rs Generates OpenVMM TTRPC bindings when openvmm feature is enabled and patches service name.
src/runtime-rs/crates/hypervisor/.gitignore Ignores generated OpenVMM binding outputs.
src/runtime-rs/config/configuration-openvmm-azure-runtime-rs.toml.in Adds an OpenVMM runtime-rs Azure config template including static resource mgmt keys.
src/runtime-rs/Cargo.toml Exposes openvmm feature at the runtime-rs top level.
src/libs/kata-types/src/config/mod.rs Re-exports OpenVMM config and name from kata-types config module.
src/libs/kata-types/src/config/hypervisor/openvmm.rs Adds OpenVMM config plugin (validation + defaults).
src/libs/kata-types/src/config/hypervisor/mod.rs Wires OpenVMM module into kata-types hypervisor config.
Cargo.lock Pulls protobuf into the lockfile for the new optional dependency.
.github/workflows/static-checks.yaml Updates workflow env to msft-main.
.github/workflows/scorecard.yaml Updates scorecard docs link and branch triggers to msft-main.
.github/workflows/run-k8s-tests-on-aks.yaml Adds OpenVMM matrix entries to AKS tests.
.github/workflows/release-s390x.yaml Updates branch/tag handling to msft-main.
.github/workflows/release-ppc64le.yaml Updates branch/tag handling to msft-main.
.github/workflows/release-arm64.yaml Updates branch/tag handling to msft-main.
.github/workflows/release-amd64.yaml Updates branch/tag handling to msft-main.
.github/workflows/push-oras-tarball-cache.yaml Updates branch trigger to msft-main.
.github/workflows/payload-after-push.yaml Updates branch trigger to msft-main.
.github/workflows/osv-scanner-scheduled.yaml Updates branch trigger to msft-main.
.github/workflows/osv-scanner-pr.yaml Updates PR branch filter to msft-main.
.github/workflows/docs.yaml Updates branch trigger to msft-main.
.github/workflows/commit-message-check.yaml Allows fixup! commit subjects and updates contributing link to msft-main.
.github/workflows/codeql.yml Updates branch filters to msft-main.
.github/workflows/ci-on-push.yaml Updates PR branch filter to msft-main.
.github/workflows/build-kubectl-image.yaml Updates branch trigger to msft-main.
.github/workflows/build-kata-static-tarball-amd64.yaml Adds openvmm component to the static tarball build matrix.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/runtime-rs/Makefile
Comment on lines +97 to +104
ARCH_SUPPORT_OPENVMM := x86_64 aarch64
ifneq ($(filter $(ARCH),$(ARCH_SUPPORT_OPENVMM)),)
USE_OPENVMM := true
else
USE_OPENVMM := false
$(info OpenVMM does not support ARCH $(ARCH), disabled. \
Specify "USE_OPENVMM=true" to force enable.)
endif
Comment thread tools/osbuilder/node-builder/azure-linux/package_build.sh
Comment on lines +55 to +65
if [[ "${USE_OPENVMM}" == "yes" ]]; then
runtime_rs_make_flags+=(
"HYPERVISOR=openvmm-runtime-rs"
"USE_OPENVMM=true"
)
else
runtime_rs_make_flags+=(
"HYPERVISOR=cloud-hypervisor"
"USE_OPENVMM=false"
)
fi
Comment thread src/runtime-rs/crates/hypervisor/src/utils.rs
Comment on lines +490 to +494
let comm_path = tid_path.join(&tid_str).join("comm");

if !comm_path.exists() {
return Err(anyhow!("comm path was not found."));
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

At least 2 approving reviews are required to merge this pull request.

Assignees

No one assigned

Labels

ok-to-test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@sprt