PoC code for SWIFT v2 addition to Kata - #424
Conversation
Bug: https://microsoft.visualstudio.com/OS/_workitems/edit/43668151 Rationale: This is a temporary solution for optimizing memory usage for the current mechanism of requesting resources through pod Limit annotations: - if no Limits are specified and hence WorkloadMemMB is 0, set a default value 'StaticWorkloadDefaultMem' to allocate a default amount of memory for use for containers in the sandbox in addition to the base memory - if Limits are specified, the base memory and the sum of Limits are allocated. The end user needs to be aware of the minimum memory requirements for their pods, otherwise the pod will be stuck in the ContainerCreating state Testing: Manual testing, creating pods with Limits and without limits, and with two containers where each container has a limit, tested with integration in a SPEC file where the config variables were set via environment variables via the make command Adapted by @mfrw from 3.1.0 to apply to 3.2.0 Signed-off-by: Muhammad Falak R Wani <mwani@microsoft.com> Signed-off-by: Manuel Huber <mahuber@microsoft.com> runtime: Remove unused VMM options for mem alloc - We only ever tested these fork changes with CLH+MSHV - Remove these options as we don't use QEMU/FC Signed-off-by: Manuel Huber <mahuber@microsoft.com>
Add Microsoft mandatory file SECURITY.md Signed-off-by: Saul Paredes <saulparedes@microsoft.com>
Add pull_request_template.md Signed-off-by: Saul Paredes <saulparedes@microsoft.com>
This adds our team as reviewers for PRs automatically again. Signed-off-by: Chris Co <chrco@microsoft.com> Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
This fixes the below error when attempting to access the debug console when all debug_console_enabled=true and all 3 enable_debug options are true: level=error msg="error create pseudo tty" error="open /dev/ptmx: operation not permitted" Signed-off-by: Aurelien Bombo <abombo@microsoft.com>
This branch starts introducing additional scripting to build, deploy and evaluate the components used in AKS' Pod Sandboxing and Confidential Containers preview features. This includes the capability to build the IGVM file and its reference measurement file for remote attestation. Signed-off-by: Manuel Huber <mahuber@microsoft.com> tools: Improve igvm-builder and node-builder/azure-linux scripting - Support for Mariner 3 builds using OS_VERSION variable - Improvements to IGVM build process and flow as described in README - Adoption of using only cloud-hypervisor-cvm on CBL-Mariner Signed-off-by: Manuel Huber <mahuber@microsoft.com> tools: Add package-tools-install functionality - Add script to install kata-containers(-cc)-tools bits - Minor improvements in README.md - Minor fix in package_install - Remove echo outputs in package_build Signed-off-by: Manuel Huber <mahuber@microsoft.com> tools: Enable setting IGVM SVN - Allow setting SVN parameter for IGVM build scripting Signed-off-by: Manuel Huber <mahuber@microsoft.com> node-builder: introduce BUILD_TYPE variable This lets developers build and deploy Kata in debug mode without having to make manual edits to the build scripts. With BUILD_TYPE=debug (default is release): * The agent is built in debug mode. * The agent is built with a permissive policy (using allow-all.rego). * The shim debug config file is used, ie. we create the symlink configuration-clh-snp-debug.toml <- configuration-clh-snp.toml. For example, building and deploying Kata-CC in debug mode is now as simple as: make BUILD_TYPE=debug all-confpods deploy-confpods Also do note that make still lets you override the other variables even after setting BUILD_TYPE. For example, you can use the production shim config with BUILD_TYPE=debug: make BUILD_TYPE=debug SHIM_USE_DEBUG_CONFIG=no all-confpods deploy-confpods Signed-off-by: Aurélien Bombo <abombo@microsoft.com> node-builder: introduce SHIM_REDEPLOY_CONFIG See README: when SHIM_REDEPLOY_CONFIG=no, the shim configuration is NOT redeployed, so that potential config changes made directly on the host during development aren't lost. Signed-off-by: Aurélien Bombo <abombo@microsoft.com> node-builder: Use img for Pod Sandboxing Switch from UVM initrd to image format Signed-off-by: Manuel Huber <mahuber@microsoft.com> node-builder: Adapt README instructions - Sanitize containerd config snippet - Set podOverhead for Kata runtime class Signed-off-by: Manuel Huber <mahuber@microsoft.com> tools: Adapt AGENT_POLICY_FILE path - Adapt path in uvm_build.sh script to comply with the usptream changes we pulled in Signed-off-by: Manuel Huber <mahuber@microsoft.com> node-builder: Use Azure Linux 3 as default path - update recipe and node-builder scripting - change default value on rootfs-builder Signed-off-by: Manuel Huber <mahuber@microsoft.com> node-builder: Deploy-only for AzL3 VMs - split deployment sections in node-builder README.md - install jq, curl dependencies within IGVM script - add path parameter to UVM install script Signed-off-by: Manuel Huber <mahuber@microsoft.com> node-builder: Minor updates to README.md - no longer install make package, is part of meta package - remove superfluous popd - add note on permissive policy for ConfPods UVM builds Signed-off-by: Manuel Huber <mahuber@microsoft.com> node-builder: Updates to README.md - with the latest 3.2.0.azl4 package on PMC, can remove OS_VERSION parameter and use the make deploy calls instead of copying files by hand for variant I (now aligned with Variant II) - with the latest changes on msft-main, set the podOverhead to 600Mi Signed-off-by: Manuel Huber <mahuber@microsoft.com> node-builder: Fix SHIM_USE_DEBUG_CONFIG behavior Using a symlink would create a cycle after calling this script again when copying the final configuration at line 74 so we just use cp instead. Also, I moved this block to the end of the file to properly override the final config file. Signed-off-by: Aurélien Bombo <abombo@microsoft.com> node-builder: Build and install debug configuration for pod sandboxing For ease of debugging, install a configuration-clh-debug.toml for pod sandboxing as we do in Conf pods. Signed-off-by: Cameron Baird <cameronbaird@microsoft.com> runtime: remove clh-snp config file usage in makefile Not needed to build vanilla kata Signed-off-by: Saul Paredes <saulparedes@microsoft.com> package_tools_install.sh: include nsdax.gpl.c Include nsdax.gpl.c Signed-off-by: Saul Paredes <saulparedes@microsoft.com>
After these changes:
1. The value of the K8s runtime class memory overhead:
- Covers the memory usage from all the Host-side components (mainly
the Kata Shim and the VMM).
- Doesn't include the memory usage from any Guest-side components.
2. The value of a pod memory limit specified by the user:
- Is equal to the memory size of the Pod VM.
- Includes the memory usage from all the Guest-side components
(mainly user's workload, the Guest kernel, and the Kata Agent)
- Doesn't include the memory usage from any Host-side components.
Signed-off-by: Dan Mihai <dmihai@microsoft.com>
- similar to the static_sandbox_default_workload_mem option, assign a default number of vcpus to the VM when no limits are given, 1 vcpu in this case - similar to commit c7b8ee9, do not allocate additional vcpus when limits are provided Signed-off-by: Manuel Huber <mahuber@microsoft.com>
Build the mariner guest image using IMAGE_SIZE_ALIGNMENT_MB=2. Signed-off-by: Dan Mihai <dmihai@microsoft.com>
- Change Makefile to point to fork - Change versions.yaml to point to proper version on fork - Do not regenerate the binding - the current definitions are invalid - Definitions will be fixed with upcoming versions such as v41.0.120 Signed-off-by: Manuel Huber <mahuber@microsoft.com>
This commit introduces changes merged in upstream PR 9153 of relaxing the timeout for calling CLH's CreateVM+BootVM APIs. Further, the commit increases the timeout to 100s to handle guest boot with large memory requests. Signed-off-by: Sumedh Alok Sharma <sumsharma@microsoft.com>
The file is currently being ignored as the CodeQL analysis task is configured as 'default'. In order to configure this as an 'advanced' task, one needs to push a CodeQL file. However, we cannot push as this file already exists. As we don't want to change the file's path, I am temporarily removing this file. Signed-off-by: Manuel Huber <mahuber@microsoft.com>
Enabling advanced CodeQL logic, re-adding upstream's codeql.yml with the only modifications being branch specifications. This should align fork and upstream CodeQL task logic. Signed-off-by: Manuel Huber <mahuber@microsoft.com>
This reverts commit 3cfdd53.
This also fixes a shellcheck error and lets us require the
shellcheck-required job:
In ./tools/osbuilder/node-builder/azure-linux/uvm_build.sh line 34:
if [ -z "${UVM_KERNEL_HEADER_DIR}}" ]; then
^-- SC2157 (error): Argument to -z is always false due to literal strings.
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
This fixes the below static check error to follow up on the infra fix from kata-containers#11646: 2025-07-31T19:32:45.0031829Z time="2025-07-31T19:32:44.990004665Z" level=fatal msg="found 2 parse errors:\nfile=\"tools/osbuilder/node-builder/azure-linux/README.md\": duplicate heading: \"Set up environment\" (heading: {Name:Set up environment MDName:Set up environment LinkName:set-up-environment Level:2})\nfile=\"tools/osbuilder/node-builder/azure-linux/README.md\": duplicate heading: \"Install build dependencies\" (heading: {Name:Install build dependencies MDName:Install build dependencies LinkName:install-build-dependencies Level:2})" commit=1d17f56b1aa7a880468b8e25d14467c92dca8eeb name=kata-check-markdown pid=9075 source=check-markdown version=0.0.1 Note: that is likely flagged because having two headings with the same name, even under different sections, makes it impossible to create a canonical heading link in Markdown. This should eventually be squashed into the node-builder commit. Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
This addresses the following errors from `make test` to allow us to require that upstream CI: https://github.com/microsoft/kata-containers/actions/runs/16656407213/job/47142422035?pr=392#step:13:53 Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
As we adopted containerd2, we remove references to our prior forked containerd version. Signed-off-by: Manuel Huber <mahuber@microsoft.com>
This auto-detects the repo by default (instead of having to specify KATA_DEV_MODE=true) so that forked repos can leverage the static-checks.yaml CI check without modification. An alternative would have been to pass the repo in static-checks.yaml. However, because of the matrix, this would've changed the check name, which is a pain to handle in either the gatekeeper/GH UI. Example fork failure: https://github.com/microsoft/kata-containers/actions/runs/16656407213/job/47142421739#step:8:75 I've tested this change to work in a fork. Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
This adds SECURITY.md to the list of GH-native files that should be excluded by the reference checker. Today this is useful for downstreams who already have a SECURITY.md file for compliance reasons. When Kata onboards that file, this commit will also be required. Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
This is needed to avoid the following static-checks error: 2025-08-05T21:27:20.0028337Z [static-checks.sh:808] ERROR: Document tools/osbuilder/node-builder/azure-linux/README.md is not referenced This commit is to be squashed into the node-builder commit. Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
This should be squashed into d1eb0ac. This is to avoid the following static-checks error: 2025-08-05T21:39:49.8540588Z .github/pull_request_template.md 2025-08-05T21:39:49.8570049Z ERROR: Invalid URL 'https://nvd.nist.gov/vuln/detail/CVE-YYYY-XXXX' found in the following files: Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
For our Kata UVM, we know we need at least 128MB of memory to prevent instability in the guest. Enforce this constraint with a descriptive error to prevent users from destabilizing the UVM with faulty k8s configurations. Signed-off-by: Cameron Baird <cameronbaird@microsoft.com>
Re-add DEFDISABLEIMAGENVDIMM=true to package_build.sh to fix a regression causing us to use pmem. Signed-off-by: Cameron Baird <cameronbaird@microsoft.com>
This would cause weird issues for downstreams which default branch is not "main". Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
This change mirrors host networking into the guest as before, but now also includes the default gateway neighbor entry for each interface. Pods using overlay/synthetic gateways (e.g., 169.254.1.1) can hit a first-connect race while the guest performs the initial ARP. Preseeding the gateway neighbor removes that latency and makes early connections (e.g., to the API Service) deterministic. Signed-off-by: Saul Paredes <saulparedes@microsoft.com>
This is the strictest possible setting for Zizmor. Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
This fixes all such issues, ie.: https://github.com/kata-containers/kata-containers/security/code-scanning/459 https://github.com/kata-containers/kata-containers/security/code-scanning/508 https://github.com/kata-containers/kata-containers/security/code-scanning/510 Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
Background: * `pull_request` runs on the PR branch code and has access to secrets ONLY if the PR is from microsoft/kata-containers (i.e. NOT from an external contributor who forked the repo). * `pull_request_target` runs on the trusted main branch code by default and has access to secrets for any PR. Reference: https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows#pull_request Upstream uses `pull_request_target` (and manually checks out the PR code) to have access to secrets for PRs from external contributors, however we don't expect external PRs, hence we can use `pull_request`. Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
This runs Zizmor on pushes to any branch, not just main.
This is useful for:
1. Testing changes in feature branches with the manually-triggered CI.
2. Forked repos that may use a different name than "main" for their
default branch.
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
Address issues just introduced and set actionlint as a required by removing the path filter. Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
`${kernel_name,,}` is bash 4.0 and not posix compliant, so doesn't
work on macos, so switch to `tr` which is more widely
supported
Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Avoid the error `ERROR: unsupported architecture: arm64` in install_go.sh on darwin Signed-off-by: stevenhorsman <steven@uk.ibm.com>
Update the two workflows that used setup-go to instead call `install_go.sh` script, which handles installing the correct version of golang Signed-off-by: stevenhorsman <steven@uk.ibm.com>
golang 1.25 has been released, so 1.23 is EoL, so we should update to ensure we don't end up with security issues Signed-off-by: stevenhorsman <steven@uk.ibm.com>
As part of the go 1.24.6 bump there are errors about the incorrect use of a errorf, so switch to the non-formatting version, or add the format string as appropriate Signed-off-by: stevenhorsman <steven@uk.ibm.com>
The Go installation step was broken because the checkout action was checking out the code in a subdirectory: https://github.com/kata-containers/kata-containers/actions/runs/18265538456/job/51999316919 Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
We can't test this PR because the workflow needs this trigger, so adding this will allow testing future PRs. Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
Downloading Go from storage.googleapis.com fails intermittently with a 403 (see error below) so we switch to go.dev as referenced at https://go.dev/dl/. /tmp/install-go-tmp.Rw5Q4thEWr ~/work/kata-containers/kata-containers /usr/bin/go [install_go.sh:85] INFO: removing go version go1.24.9 linux/amd64 [install_go.sh:94] INFO: Download go version 1.24.6 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 298 100 298 0 0 2610 0 --:--:-- --:--:-- --:--:-- 2614 [install_go.sh:97] INFO: Install go gzip: stdin: not in gzip format tar: Child returned status 1 tar: Error is not recoverable: exiting now [install_go.sh:99] ERROR: sudo tar -C /usr/local/ -xzf go1.24.6.linux-amd64.tar.gz https://github.com/kata-containers/kata-containers/actions/runs/18602801597/job/53045072109?pr=11947#step:5:17 Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
If memory limit is set and less than minimum, set it to minimum. This is to to account for kata-containers@0ec3403 Signed-off-by: Saul Paredes <saulparedes@microsoft.com>
As of https://microsoft.visualstudio.com/OS/_workitems/edit/48222512?src=WorkItemMention&src-action=artifact_link , we are able to run privileged containers on kata, so allow them through the webhook. Signed-off-by: Saul Paredes <saulparedes@microsoft.com>
The repro below would show this error in the logs (in debug mode only):
fatal runtime error: IO Safety violation: owned file descriptor already closed
The issue was that the `pseudo.slave` file descriptor was being owned by
multiple variables simultaneously. When any of those variables would go out
of scope, they would close the same file descriptor, which is undefined
behavior.
To fix this, we clone: we create a new file descriptOR that refers to the same
file descriptION as the original. When the cloned descriptor is closed, this
affect neither the original descriptor nor the description. Only when the last
descriptor is closed does the kernel cleans up the description.
Note that we purposely consume (not clone) the original descriptor with
`child_stdin` as `pseudo` is NOT dropped automatically.
Repro
-----
Prerequisites:
- Use Rust 1.80+.
- Build the agent in debug mode.
$ cat busybox.yaml
apiVersion: v1
kind: Pod
metadata:
name: busybox
spec:
containers:
- image: busybox:latest
name: busybox
runtimeClassName: kata
$ kubectl apply -f busyboox.yaml
pod/busybox created
$ kubectl exec -it busybox -- sh
error: Internal error occurred: Internal error occurred: error executing
command in container: failed to exec in container: failed to start exec
"e6c602352849647201860c1e1888d99ea3166512f1cc548b9d7f2533129508a9":
cannot enter container 76a499cbf747b9806689e51f6ba35e46d735064a3f176f9be034777e93a242d5,
with err ttrpc: closed
Fixes: kata-containers#11054
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
Allow users to build the Kata Agent using INIT_DATA=no to disable the detect_initdata_device() code loop and associated debug log output. Future additional improvements related to Init Data are tracked by kata-containers#11532. Signed-off-by: Dan Mihai <dmihai@microsoft.com>
Disable detect_initdata_device for the AKS-Kata Agent. This feature was unused but caused the execution of a non-trivial amount of code and many log entries during Agent init. Signed-off-by: Dan Mihai <dmihai@microsoft.com>
b325f56 to
640e3bf
Compare
| } | ||
|
|
||
| if isPhysical { | ||
| if s.config.HypervisorConfig.ColdPlugVFIO == config.NoPort { |
There was a problem hiding this comment.
It is still needed, but this check was moved into "createPhysicalEndpoint" because the VFIO Disabled config flag only applies to VFIO interfaces, and we can't check the VFIO Disabled config flag before we know the interface is an actual VFIO interface. In the previous code, the flow assumed that "isPhysical" automatically means "is VFIO", which is what I changed here, so this check cannot be done as early in the code anymore.
| } else { | ||
| // The network namespace would have been deleted at this point | ||
| // if it has not been created by virtcontainers. | ||
| if !netNsCreated { |
There was a problem hiding this comment.
I am on the fence for this one. I took the same approach as the existing veth code:
Adding logging here can significantly increase the number of log lines but the usefulness of the log line may be arguable (the netNsCreated flag is an internal flag to prevent detaching a network interface multiple times, as "Detach" can be called multiple times, so, if we add a log line here, we will see a call with "netNsCreated" for both the successful deletion, and the harmless subsequent calls to Detach. This may make the logs confusing).
640e3bf to
a406af4
Compare
Signed-off-by: Aurélien Bombo <abombo@microsoft.com>
| if err != nil { | ||
| return nil, err | ||
| } | ||
| lastIdx = int(n) |
Check failure
Code scanning / CodeQL
Incorrect conversion between integer types High
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 5 months ago
In general, the issue arises because strconv.ParseInt is invoked with a 64-bit size, but the result is then converted to int with no check. To fix this pattern, either (a) parse with a bit size matching the target type (e.g., 32 for 32-bit int), or (b) keep parsing as 64-bit and add explicit bounds checks to ensure the value fits into the target type before converting.
Here, lastIdx is an int used as an index-like counter derived from the numeric suffix of an endpoint name. The safest and simplest change that preserves behavior is to (1) parse the numeric suffix as a 32-bit integer (since int is at least 32 bits) using strconv.ParseInt with bit size 32, (2) check that the parsed value is non-negative and within the int range, and then (3) convert to int. Because we must avoid extra imports and math is not currently imported in this file, we cannot rely on math.MaxInt32 directly; however, if we parse with bit size 32 and then convert to int, the value is guaranteed to fit in int on 32-bit and 64-bit Go architectures (where int is 32 or 64 bits respectively). We should also guard against negative values defensively, even though the regex extracts only digits.
Concretely, in addSingleEndpoint in src/runtime/virtcontainers/network_linux.go, modify the block that extracts and parses the numeric suffix: change the ParseInt call to use bit size 32, and replace lastIdx = int(n) with a small validity check, e.g., ensure n >= 0 before assigning and otherwise return an error. All other behavior remains the same.
| @@ -134,10 +134,13 @@ | ||
| lastEndpoint := n.eps[len(n.eps)-1] | ||
| re := regexp.MustCompile("[0-9]+") | ||
| matchStr := re.FindString(lastEndpoint.Name()) | ||
| n, err := strconv.ParseInt(matchStr, 10, 64) | ||
| n, err := strconv.ParseInt(matchStr, 10, 32) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
| if n < 0 { | ||
| return nil, fmt.Errorf("invalid endpoint index parsed from name %q: %d", lastEndpoint.Name(), n) | ||
| } | ||
| lastIdx = int(n) | ||
| } | ||
| if idx <= lastIdx { |
Merge Checklist
Summary
This PR captures the PoC code for SWIFT v2 support in Kata. The code does two things:
The code mostly modifies the physical network path to provide the above support. It also uses the same approach as VETH for all the non-VF / non SR-IOV network adapters that are being added to a Kata UVM.
Associated issues
Links to CVEs
Test Methodology
Manual validation of the PoC. No regression testing (this is pure PoC code)