Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions confidential-containers/attestation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,14 @@ Attestation is required for any feature that depends on secrets, including:
* Using sealed secrets
* Requesting secrets directly from workloads

A complete attestation process should also cover the Kata Agent API surface: which operations the
Comment thread
manuelh-dev marked this conversation as resolved.
untrusted host is allowed to request inside the guest.
An agent security policy constrains that surface.
Generate and attach that policy as part of the workload manifest so attestation of the guest
includes that constrained Agent API surface.
Refer to :ref:`Attach a Kata Agent Security Policy <coco-agent-security-policy>`.
The agent security policy is distinct from the Trustee policy that decides whether to release a KBS resource.

Key Concepts
============

Expand Down
73 changes: 64 additions & 9 deletions confidential-containers/configure-workloads.rst
Original file line number Diff line number Diff line change
Expand Up @@ -28,20 +28,22 @@ As a :ref:`Container User <coco-persona-container-user>`, use this page to confi
A Confidential Container workload is a standard Kubernetes pod that runs inside a TEE-protected
virtual machine and requests one or more GPUs through the NVIDIA Kata sandbox device plugin.
Compared with a traditional GPU pod, a Confidential Container workload pod manifest differs in
three ways:
various ways. In particular, the pod manifest:

* It selects a TEE-aware Kata runtime class instead of the default ``runc``-based runtime.
* It requests GPU and NVSwitch resources using the resource types advertised by the NVIDIA
* Selects a TEE-aware Kata runtime class instead of the default ``runc``-based runtime.
* Requests GPU and NVSwitch resources using the resource types advertised by the NVIDIA
Kata sandbox device plugin, which can be either default names or model-specific names.
* For NVSwitch-based HGX systems, it requests every GPU and NVSwitch on the node together so
* For NVSwitch-based HGX systems, requests every GPU and NVSwitch on the node together so
that all devices reside inside the same Confidential Container virtual machine.
* For an attested production deployment, includes a Kata agent security policy that limits
which Agent API calls the untrusted host can make into the guest.

**Before this page:** Complete the :doc:`Detailed Install Guide <confidential-containers-deploy>` and verify the cluster with :doc:`Run a Sample Workload <run-sample-workload>` (``Test PASSED`` in pod logs).
For install steps, refer to :doc:`Prerequisites <prerequisites>` and :doc:`Detailed Install Guide <confidential-containers-deploy>`.

This page describes each of these decisions and provides single-GPU and multi-GPU passthrough
manifest examples that you can copy and adapt to your environment.
The install sample uses a minimal manifest.
The install sample uses a minimal manifest and does not attach an agent security policy.

********************************
Select a Container Runtime Class
Expand All @@ -63,7 +65,7 @@ Select the runtime class based on the CPU TEE on the target worker node:
- ``kata-qemu-nvidia-gpu-tdx``

The ``kata-deploy`` chart also installs a ``kata-qemu-nvidia-gpu`` runtime class.
That class is intended for non-confidential Kata workloads.
That class is intended for non-confidential Kata workloads.

.. _coco-resource-types:

Expand All @@ -72,7 +74,7 @@ Reference GPU and NVSwitch Resource Types
*****************************************

The NVIDIA Kata sandbox device plugin advertises GPUs and NVSwitches to Kubernetes as extended resources.
Your pod manifest requests those resources under ``resources.limits``.
Your pod manifest requests those resources under ``resources.limits``.
You can use either the default resource types or model-specific resource types.

By default, every passthrough GPU is advertised as ``nvidia.com/pgpu`` and every NVSwitch is advertised as ``nvidia.com/nvswitch``.
Expand Down Expand Up @@ -102,9 +104,9 @@ Use the model-specific resource name in workloads that must target a specific ac
limits:
nvidia.com/GH100_H200_141GB: "1"


Set the ``NODE_NAME`` environment variable to the node you want to check:

.. code-block:: console

$ export NODE_NAME="<node-name>"
Expand Down Expand Up @@ -315,9 +317,62 @@ Run a Multi-GPU Workload

$ kubectl delete -f multi-gpu-kata.yaml

.. _coco-agent-security-policy:

***********************************
Attach a Kata Agent Security Policy
***********************************

The Kata agent runs inside the guest virtual machine and manages the container lifecycle.
Because the Kata shim on the host is outside the TEE, the host can still issue Agent API calls
into the guest unless you restrict those Agent API calls.
Comment thread
manuelh-dev marked this conversation as resolved.
An *agent security policy* is a Rego policy that the agent enforces so that only the operations
your workload declared are allowed.

The :doc:`sample workload <run-sample-workload>` omits an agent policy so that you can confirm
GPU passthrough with the smallest possible manifest.
Do not treat that YAML as a production template.

The Kata Containers ``genpolicy`` tool reads your Kubernetes YAML, infers the intended Agent API
calls, encodes the policy in base64, and appends it as an annotation on the same file.

#. Download ``genpolicy`` from the latest Kata Containers release that is compatible with this
reference architecture.
For usage details, refer to the
`Agent Policy generation tool <https://github.com/kata-containers/kata-containers/blob/main/src/tools/genpolicy/README.md>`_
documentation.

#. Run ``genpolicy`` against the manifest to deploy:

.. code-block:: console

$ genpolicy -y cuda-vectoradd-kata.yaml

Review the generated policy before you apply the manifest.
The automatically-generated policy is a starting point.
The policy can allow operations you do not want, such as ``kubectl exec``, or omit operations your
workload needs.

.. important::

Review and, if needed, edit the policy so that it matches your threat model before you
use it.

#. Apply the annotated manifest:

.. code-block:: console

$ kubectl apply -f cuda-vectoradd-kata.yaml

**********
Next Steps
**********

* Refer to :doc:`Attestation <attestation>` for Trustee concepts and a local connectivity test.
A complete attestation process should cover the Kata Agent API surface as well as the TEE.
The agent security policy contained in the workload annotation is included in the measured
payload that is attested with the guest.
The agent security policy is distinct from the Trustee attestation policy that decides whether
to release secrets.
* Refer to :doc:`Managing the Confidential Computing Mode <configure-cc-mode>` to change the CC mode on GPUs at the cluster or node level.
* Refer to :doc:`Troubleshooting <troubleshooting>` if a workload does not schedule or the pod stays ``Pending``.
4 changes: 2 additions & 2 deletions confidential-containers/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,7 @@ Learn
Roles, responsibilities, and documentation navigation by persona.


.. grid-item-card:: :octicon:`server;1.5em;sd-mr-1` Supported Platforms
.. grid-item-card:: :octicon:`server;1.5em;sd-mr-1` Supported Platforms and Software Components
:link: supported-platforms
:link-type: doc

Expand Down Expand Up @@ -189,7 +189,7 @@ Configuration
:link: configure-workloads
:link-type: doc

Runtime classes, resource types, and multi-GPU passthrough.
Runtime classes, resource types, multi-GPU passthrough, and agent security policy.

.. grid-item-card:: :octicon:`gear;1.5em;sd-mr-1` Managing the Confidential Computing Mode
:link: configure-cc-mode
Expand Down
4 changes: 2 additions & 2 deletions confidential-containers/llms.txt
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ Scope notes for agents and readers:
- [Personas](https://docs.nvidia.com/datacenter/cloud-native/confidential-containers/latest/personas.html): Target roles — hardware IT admin, host OS admin, Kubernetes cluster admin, security engineer, container user (plus application-owner and enterprise/compliance stakeholders).

## Prerequisites & platforms
- [Supported Platforms](https://docs.nvidia.com/datacenter/cloud-native/confidential-containers/latest/supported-platforms.html): Validated versions — OS/kernel, containerd, Kubernetes, Kata, GPU Operator — and supported CPUs (AMD Genoa/Milan, Intel ER/GR) and GPUs (H100/H200/B200/RTX Pro 6000, single- and multi-GPU PPCIe).
- [Supported Platforms and Software Components](https://docs.nvidia.com/datacenter/cloud-native/confidential-containers/latest/supported-platforms.html): Validated versions — OS/kernel, containerd, Kubernetes, Kata, GPU Operator — and supported CPUs (AMD Genoa/Milan, Intel ER/GR) and GPUs (H100/H200/B200/RTX Pro 6000, single- and multi-GPU PPCIe).
- [Prerequisites](https://docs.nvidia.com/datacenter/cloud-native/confidential-containers/latest/prerequisites.html): Required starting state — Kubernetes cluster, containerd, BIOS (hardware virtualization, ACS, IOMMU), host OS/kernel, and feature gates (e.g. KubeletPodResourcesGet) — before installing Kata and the GPU Operator.

## Install
Expand All @@ -33,7 +33,7 @@ Scope notes for agents and readers:

## Run & configure
- [Run a Sample Workload](https://docs.nvidia.com/datacenter/cloud-native/confidential-containers/latest/run-sample-workload.html): Deploy cuda-vectoradd-kata.yaml on the CC RuntimeClass; success = `Test PASSED` in the pod logs.
- [Configuring Workloads](https://docs.nvidia.com/datacenter/cloud-native/confidential-containers/latest/configure-workloads.html): Workload spec options for confidential GPU pods.
- [Configuring Workloads](https://docs.nvidia.com/datacenter/cloud-native/confidential-containers/latest/configure-workloads.html): Workload spec options for confidential GPU pods, including how to attach a Kata agent security policy.
- [Managing Confidential Computing Mode](https://docs.nvidia.com/datacenter/cloud-native/confidential-containers/latest/configure-cc-mode.html): Turning GPU CC mode on/off and verifying cc.mode.state.

## Attestation
Expand Down
29 changes: 15 additions & 14 deletions confidential-containers/overview.rst
Original file line number Diff line number Diff line change
Expand Up @@ -31,14 +31,14 @@ Background
**********

NVIDIA GPUs power the training and deployment of Large Language Models (LLMs) that define the state of the art in AI reasoning and capability.
As organizations adopt these models in regulated industries such as financial services, healthcare, and the public sector, protecting model intellectual property and sensitive user data becomes essential.
The model deployment landscape is also evolving to include public clouds, enterprise on-premises, and edge.
As organizations adopt these models in regulated industries such as financial services, healthcare, and the public sector, protecting model intellectual property and sensitive user data becomes essential.
The model deployment landscape is also evolving to include public clouds, enterprise on-premises, and edge.
A zero-trust posture on cloud-native platforms such as Kubernetes is essential to secure assets (model IP and enterprise private data) from untrusted infrastructure with privileged user access.

Confidential Computing (CC) addresses this gap by using hardware-based Trusted Execution Environments (TEEs), such as AMD SEV-SNP and Intel TDX, with NVIDIA Confidential Computing capabilities to provide isolation, memory encryption, and integrity verification during processing. In addition to isolation, CC provides Remote Attestation, which allows workload owners to cryptographically verify the state of a TEE before providing secrets or sensitive data.

`Confidential Containers <https://github.com/confidential-containers>`__ (CoCo) is the cloud-native approach of CC on Kubernetes.
The Confidential Containers project leverages Kata Containers to provide the sandboxing capabilities.
The Confidential Containers project leverages Kata Containers to provide the sandboxing capabilities.
`Kata Containers <https://katacontainers.io/>`_ is an open-source project that provides lightweight Utility Virtual Machines (UVMs) that feel and perform like containers while providing strong workload isolation. Along with the Confidential Containers project, Kata enables the orchestration of secure, GPU-accelerated workloads in Kubernetes.

.. _coco-use-cases:
Expand All @@ -47,7 +47,7 @@ The Confidential Containers project leverages Kata Containers to provide the san
Use Cases
*********

The target for Confidential Containers is to enable model providers (closed and open source) and Enterprises to use the advancements of Gen AI, agnostic to the deployment model (Cloud, Enterprise, or Edge).
The target for Confidential Containers is to enable model providers (closed and open source) and Enterprises to use the advancements of Gen AI, agnostic to the deployment model (Cloud, Enterprise, or Edge).

* For Model Providers: It enables the expansion of reach by allowing expensive, proprietary model weights to be deployed on-site at customer data centers without exposing the intellectual property (IP) to the customer's infrastructure administrators.
* For Adopters: It provides low-latency access to state-of-the-art frontier models within their own sovereign environment, ensuring their private prompts and data never leave their controlled premises while maintaining the security of the model provider's IP.
Expand Down Expand Up @@ -127,14 +127,14 @@ These components include:
* NVIDIA Confidential Computing Manager (cc-manager) for Kubernetes: Sets the confidential computing (CC) mode on the NVIDIA GPUs.
By default, the Confidential Computing Manager will transition all NVIDIA GPUs to Confidential Computing mode, if they are not already in that mode.
* NVIDIA Kata Sandbox Device Plugin: Creates host-side Container Device Interface (CDI) specifications for GPU passthrough and discovers NVIDIA GPUs along with their capabilities, advertises these to Kubernetes, and allocates GPUs during pod deployment.
Allocatable GPU resources are advertised as type ``nvidia.com/pgpu`` by default.
Allocatable GPU resources are advertised as type ``nvidia.com/pgpu`` by default.
* NVIDIA VFIO Manager: Binds discovered NVIDIA GPUs and NVSwitches to the vfio-pci driver for VFIO passthrough.

Refer to the :doc:`NVIDIA GPU Operator <gpuop:overview>` documentation for more information on the NVIDIA GPU Operator or the :ref:`GPU Operator Cluster Topology Considerations <coco-gpu-operator-components>` section for more information on selecting nodes for Confidential Containers.

**Node Feature Discovery (NFD)**

Bootstraps the node by advertising the node features using labels to make sophisticated scheduling decisions, such as installing the Kata/CoCo stack only on the nodes that support the CC prerequisites for CPU and GPU.
Bootstraps the node by advertising the node features using labels to make sophisticated scheduling decisions, such as installing the Kata/CoCo stack only on the nodes that support the CC prerequisites for CPU and GPU.
This directs the Operator to install node feature rules that detect CPU security features and the NVIDIA GPU hardware.

Refer to the `Node Feature Discovery documentation <https://kubernetes-sigs.github.io/node-feature-discovery/>`_ for upstream usage and reference material.
Expand All @@ -143,19 +143,20 @@ This component is typically deployed and managed by default by the GPU Operator.

**Snapshotter (for example, Nydus)**

Handles the container image "guest pull" functionality.
Used as a remote snapshotter, it bypasses image pulls on the host.
Handles the container image "guest pull" functionality.
Used as a remote snapshotter, it bypasses image pulls on the host.
Instead, the snapshotter fetches and unpacks encrypted and signed container images directly inside the protected guest memory, keeping proprietary contents hidden and ensuring image integrity.


**Kata Agent and Agent Security Policy**

Runs inside the guest VM to manage the container lifecycle while enforcing a strict, immutable agent security policy based on Rego (regorus).
Runs inside the guest VM to manage the container lifecycle while enforcing a strict, immutable agent security policy based on Rego (regorus).
This blocks the untrusted host from executing unauthorized commands, such as a malicious ``kubectl exec``.
For how to generate and attach a policy to a workload, refer to :ref:`Attach a Kata Agent Security Policy <coco-agent-security-policy>`.

**Trustee and Attestation Service**

Attestation and key brokering framework (which includes the Key Broker Service and Attestation Service).
Attestation and key brokering framework (which includes the Key Broker Service and Attestation Service).
It acts as the cryptographic gatekeeper, verifying hardware/software evidence and only releasing secrets if the environment is proven secure.


Expand Down Expand Up @@ -219,7 +220,7 @@ The following features are supported with Confidential Containers:


* Composite :doc:`attestation <attestation>` using Trustee and the NVIDIA Remote Attestation Service (NRAS).
* Generating Kata Agent Security Policies using the `genpolicy tool <https://github.com/kata-containers/kata-containers/blob/main/src/tools/genpolicy/README.md>`_.
* Generating Kata Agent Security Policies.
* Use of `signed sealed secrets <https://confidentialcontainers.org/docs/features/sealed-secrets/>`_.
* Access to authenticated registries for container image guest-pull.
* Container image signature verification and encrypted container images.
Expand All @@ -235,6 +236,7 @@ Limitations and Restrictions
****************************

* NVIDIA supports the GPU Operator and confidential computing with the containerd runtime only.

* All GPUs on the host must be configured for Confidential Computing.
Configuring only a subset of GPUs on a node is not supported.
For multi-GPU passthrough, all GPUs must be assigned to a single confidential VM.
Expand Down Expand Up @@ -281,11 +283,11 @@ To deploy on your cluster, start with the **Install** section:
.. grid:: 2
:gutter: 3

.. grid-item-card:: :octicon:`server;1.5em;sd-mr-1` Supported Platforms
.. grid-item-card:: :octicon:`server;1.5em;sd-mr-1` Supported Platforms and Software Components
:link: supported-platforms
:link-type: doc

Hardware, OS, and component versions validated for general availability (GA).
Validated hardware, OS, and component versions.

.. grid-item-card:: :octicon:`checklist;1.5em;sd-mr-1` Prerequisites
:link: prerequisites
Expand All @@ -306,4 +308,3 @@ To deploy on your cluster, start with the **Install** section:
Verify the deployment; success is ``Test PASSED`` in pod logs.

After installation, refer to the **Configuration** section for workload configuration, CC mode management, and attestation.

Loading