diff --git a/confidential-containers/attestation.rst b/confidential-containers/attestation.rst index a8d43a75e..0286cc660 100644 --- a/confidential-containers/attestation.rst +++ b/confidential-containers/attestation.rst @@ -38,6 +38,14 @@ Attestation is required for any feature that depends on secrets, including: * Using sealed secrets * Requesting secrets directly from workloads +A complete attestation process should also cover the Kata Agent API surface: which operations the +untrusted host is allowed to request inside the guest. +An agent security policy constrains that surface. +Generate and attach that policy as part of the workload manifest so attestation of the guest +includes that constrained Agent API surface. +Refer to :ref:`Attach a Kata Agent Security Policy `. +The agent security policy is distinct from the Trustee policy that decides whether to release a KBS resource. + Key Concepts ============ diff --git a/confidential-containers/configure-workloads.rst b/confidential-containers/configure-workloads.rst index 049126dd3..709c185bb 100644 --- a/confidential-containers/configure-workloads.rst +++ b/confidential-containers/configure-workloads.rst @@ -28,20 +28,22 @@ As a :ref:`Container User `, use this page to confi A Confidential Container workload is a standard Kubernetes pod that runs inside a TEE-protected virtual machine and requests one or more GPUs through the NVIDIA Kata sandbox device plugin. Compared with a traditional GPU pod, a Confidential Container workload pod manifest differs in -three ways: +various ways. In particular, the pod manifest: -* It selects a TEE-aware Kata runtime class instead of the default ``runc``-based runtime. -* It requests GPU and NVSwitch resources using the resource types advertised by the NVIDIA +* Selects a TEE-aware Kata runtime class instead of the default ``runc``-based runtime. +* Requests GPU and NVSwitch resources using the resource types advertised by the NVIDIA Kata sandbox device plugin, which can be either default names or model-specific names. -* For NVSwitch-based HGX systems, it requests every GPU and NVSwitch on the node together so +* For NVSwitch-based HGX systems, requests every GPU and NVSwitch on the node together so that all devices reside inside the same Confidential Container virtual machine. +* For an attested production deployment, includes a Kata agent security policy that limits + which Agent API calls the untrusted host can make into the guest. **Before this page:** Complete the :doc:`Detailed Install Guide ` and verify the cluster with :doc:`Run a Sample Workload ` (``Test PASSED`` in pod logs). For install steps, refer to :doc:`Prerequisites ` and :doc:`Detailed Install Guide `. This page describes each of these decisions and provides single-GPU and multi-GPU passthrough manifest examples that you can copy and adapt to your environment. -The install sample uses a minimal manifest. +The install sample uses a minimal manifest and does not attach an agent security policy. ******************************** Select a Container Runtime Class @@ -63,7 +65,7 @@ Select the runtime class based on the CPU TEE on the target worker node: - ``kata-qemu-nvidia-gpu-tdx`` The ``kata-deploy`` chart also installs a ``kata-qemu-nvidia-gpu`` runtime class. -That class is intended for non-confidential Kata workloads. +That class is intended for non-confidential Kata workloads. .. _coco-resource-types: @@ -72,7 +74,7 @@ Reference GPU and NVSwitch Resource Types ***************************************** The NVIDIA Kata sandbox device plugin advertises GPUs and NVSwitches to Kubernetes as extended resources. -Your pod manifest requests those resources under ``resources.limits``. +Your pod manifest requests those resources under ``resources.limits``. You can use either the default resource types or model-specific resource types. By default, every passthrough GPU is advertised as ``nvidia.com/pgpu`` and every NVSwitch is advertised as ``nvidia.com/nvswitch``. @@ -102,9 +104,9 @@ Use the model-specific resource name in workloads that must target a specific ac limits: nvidia.com/GH100_H200_141GB: "1" - + Set the ``NODE_NAME`` environment variable to the node you want to check: - + .. code-block:: console $ export NODE_NAME="" @@ -315,9 +317,62 @@ Run a Multi-GPU Workload $ kubectl delete -f multi-gpu-kata.yaml +.. _coco-agent-security-policy: + +*********************************** +Attach a Kata Agent Security Policy +*********************************** + +The Kata agent runs inside the guest virtual machine and manages the container lifecycle. +Because the Kata shim on the host is outside the TEE, the host can still issue Agent API calls +into the guest unless you restrict those Agent API calls. +An *agent security policy* is a Rego policy that the agent enforces so that only the operations +your workload declared are allowed. + +The :doc:`sample workload ` omits an agent policy so that you can confirm +GPU passthrough with the smallest possible manifest. +Do not treat that YAML as a production template. + +The Kata Containers ``genpolicy`` tool reads your Kubernetes YAML, infers the intended Agent API +calls, encodes the policy in base64, and appends it as an annotation on the same file. + +#. Download ``genpolicy`` from the latest Kata Containers release that is compatible with this + reference architecture. + For usage details, refer to the + `Agent Policy generation tool `_ + documentation. + +#. Run ``genpolicy`` against the manifest to deploy: + + .. code-block:: console + + $ genpolicy -y cuda-vectoradd-kata.yaml + + Review the generated policy before you apply the manifest. + The automatically-generated policy is a starting point. + The policy can allow operations you do not want, such as ``kubectl exec``, or omit operations your + workload needs. + + .. important:: + + Review and, if needed, edit the policy so that it matches your threat model before you + use it. + +#. Apply the annotated manifest: + + .. code-block:: console + + $ kubectl apply -f cuda-vectoradd-kata.yaml + ********** Next Steps ********** +* Refer to :doc:`Attestation ` for Trustee concepts and a local connectivity test. + A complete attestation process should cover the Kata Agent API surface as well as the TEE. + The agent security policy contained in the workload annotation is included in the measured + payload that is attested with the guest. + The agent security policy is distinct from the Trustee attestation policy that decides whether + to release secrets. * Refer to :doc:`Managing the Confidential Computing Mode ` to change the CC mode on GPUs at the cluster or node level. * Refer to :doc:`Troubleshooting ` if a workload does not schedule or the pod stays ``Pending``. diff --git a/confidential-containers/index.rst b/confidential-containers/index.rst index 94e3f84f2..9427eea6e 100644 --- a/confidential-containers/index.rst +++ b/confidential-containers/index.rst @@ -143,7 +143,7 @@ Learn Roles, responsibilities, and documentation navigation by persona. - .. grid-item-card:: :octicon:`server;1.5em;sd-mr-1` Supported Platforms + .. grid-item-card:: :octicon:`server;1.5em;sd-mr-1` Supported Platforms and Software Components :link: supported-platforms :link-type: doc @@ -189,7 +189,7 @@ Configuration :link: configure-workloads :link-type: doc - Runtime classes, resource types, and multi-GPU passthrough. + Runtime classes, resource types, multi-GPU passthrough, and agent security policy. .. grid-item-card:: :octicon:`gear;1.5em;sd-mr-1` Managing the Confidential Computing Mode :link: configure-cc-mode diff --git a/confidential-containers/llms.txt b/confidential-containers/llms.txt index b0ed13dfc..a462d147f 100644 --- a/confidential-containers/llms.txt +++ b/confidential-containers/llms.txt @@ -24,7 +24,7 @@ Scope notes for agents and readers: - [Personas](https://docs.nvidia.com/datacenter/cloud-native/confidential-containers/latest/personas.html): Target roles — hardware IT admin, host OS admin, Kubernetes cluster admin, security engineer, container user (plus application-owner and enterprise/compliance stakeholders). ## Prerequisites & platforms -- [Supported Platforms](https://docs.nvidia.com/datacenter/cloud-native/confidential-containers/latest/supported-platforms.html): Validated versions — OS/kernel, containerd, Kubernetes, Kata, GPU Operator — and supported CPUs (AMD Genoa/Milan, Intel ER/GR) and GPUs (H100/H200/B200/RTX Pro 6000, single- and multi-GPU PPCIe). +- [Supported Platforms and Software Components](https://docs.nvidia.com/datacenter/cloud-native/confidential-containers/latest/supported-platforms.html): Validated versions — OS/kernel, containerd, Kubernetes, Kata, GPU Operator — and supported CPUs (AMD Genoa/Milan, Intel ER/GR) and GPUs (H100/H200/B200/RTX Pro 6000, single- and multi-GPU PPCIe). - [Prerequisites](https://docs.nvidia.com/datacenter/cloud-native/confidential-containers/latest/prerequisites.html): Required starting state — Kubernetes cluster, containerd, BIOS (hardware virtualization, ACS, IOMMU), host OS/kernel, and feature gates (e.g. KubeletPodResourcesGet) — before installing Kata and the GPU Operator. ## Install @@ -33,7 +33,7 @@ Scope notes for agents and readers: ## Run & configure - [Run a Sample Workload](https://docs.nvidia.com/datacenter/cloud-native/confidential-containers/latest/run-sample-workload.html): Deploy cuda-vectoradd-kata.yaml on the CC RuntimeClass; success = `Test PASSED` in the pod logs. -- [Configuring Workloads](https://docs.nvidia.com/datacenter/cloud-native/confidential-containers/latest/configure-workloads.html): Workload spec options for confidential GPU pods. +- [Configuring Workloads](https://docs.nvidia.com/datacenter/cloud-native/confidential-containers/latest/configure-workloads.html): Workload spec options for confidential GPU pods, including how to attach a Kata agent security policy. - [Managing Confidential Computing Mode](https://docs.nvidia.com/datacenter/cloud-native/confidential-containers/latest/configure-cc-mode.html): Turning GPU CC mode on/off and verifying cc.mode.state. ## Attestation diff --git a/confidential-containers/overview.rst b/confidential-containers/overview.rst index 4aa525f0e..c655fb623 100644 --- a/confidential-containers/overview.rst +++ b/confidential-containers/overview.rst @@ -31,14 +31,14 @@ Background ********** NVIDIA GPUs power the training and deployment of Large Language Models (LLMs) that define the state of the art in AI reasoning and capability. -As organizations adopt these models in regulated industries such as financial services, healthcare, and the public sector, protecting model intellectual property and sensitive user data becomes essential. -The model deployment landscape is also evolving to include public clouds, enterprise on-premises, and edge. +As organizations adopt these models in regulated industries such as financial services, healthcare, and the public sector, protecting model intellectual property and sensitive user data becomes essential. +The model deployment landscape is also evolving to include public clouds, enterprise on-premises, and edge. A zero-trust posture on cloud-native platforms such as Kubernetes is essential to secure assets (model IP and enterprise private data) from untrusted infrastructure with privileged user access. Confidential Computing (CC) addresses this gap by using hardware-based Trusted Execution Environments (TEEs), such as AMD SEV-SNP and Intel TDX, with NVIDIA Confidential Computing capabilities to provide isolation, memory encryption, and integrity verification during processing. In addition to isolation, CC provides Remote Attestation, which allows workload owners to cryptographically verify the state of a TEE before providing secrets or sensitive data. `Confidential Containers `__ (CoCo) is the cloud-native approach of CC on Kubernetes. -The Confidential Containers project leverages Kata Containers to provide the sandboxing capabilities. +The Confidential Containers project leverages Kata Containers to provide the sandboxing capabilities. `Kata Containers `_ is an open-source project that provides lightweight Utility Virtual Machines (UVMs) that feel and perform like containers while providing strong workload isolation. Along with the Confidential Containers project, Kata enables the orchestration of secure, GPU-accelerated workloads in Kubernetes. .. _coco-use-cases: @@ -47,7 +47,7 @@ The Confidential Containers project leverages Kata Containers to provide the san Use Cases ********* -The target for Confidential Containers is to enable model providers (closed and open source) and Enterprises to use the advancements of Gen AI, agnostic to the deployment model (Cloud, Enterprise, or Edge). +The target for Confidential Containers is to enable model providers (closed and open source) and Enterprises to use the advancements of Gen AI, agnostic to the deployment model (Cloud, Enterprise, or Edge). * For Model Providers: It enables the expansion of reach by allowing expensive, proprietary model weights to be deployed on-site at customer data centers without exposing the intellectual property (IP) to the customer's infrastructure administrators. * For Adopters: It provides low-latency access to state-of-the-art frontier models within their own sovereign environment, ensuring their private prompts and data never leave their controlled premises while maintaining the security of the model provider's IP. @@ -127,14 +127,14 @@ These components include: * NVIDIA Confidential Computing Manager (cc-manager) for Kubernetes: Sets the confidential computing (CC) mode on the NVIDIA GPUs. By default, the Confidential Computing Manager will transition all NVIDIA GPUs to Confidential Computing mode, if they are not already in that mode. * NVIDIA Kata Sandbox Device Plugin: Creates host-side Container Device Interface (CDI) specifications for GPU passthrough and discovers NVIDIA GPUs along with their capabilities, advertises these to Kubernetes, and allocates GPUs during pod deployment. - Allocatable GPU resources are advertised as type ``nvidia.com/pgpu`` by default. + Allocatable GPU resources are advertised as type ``nvidia.com/pgpu`` by default. * NVIDIA VFIO Manager: Binds discovered NVIDIA GPUs and NVSwitches to the vfio-pci driver for VFIO passthrough. Refer to the :doc:`NVIDIA GPU Operator ` documentation for more information on the NVIDIA GPU Operator or the :ref:`GPU Operator Cluster Topology Considerations ` section for more information on selecting nodes for Confidential Containers. **Node Feature Discovery (NFD)** -Bootstraps the node by advertising the node features using labels to make sophisticated scheduling decisions, such as installing the Kata/CoCo stack only on the nodes that support the CC prerequisites for CPU and GPU. +Bootstraps the node by advertising the node features using labels to make sophisticated scheduling decisions, such as installing the Kata/CoCo stack only on the nodes that support the CC prerequisites for CPU and GPU. This directs the Operator to install node feature rules that detect CPU security features and the NVIDIA GPU hardware. Refer to the `Node Feature Discovery documentation `_ for upstream usage and reference material. @@ -143,19 +143,20 @@ This component is typically deployed and managed by default by the GPU Operator. **Snapshotter (for example, Nydus)** -Handles the container image "guest pull" functionality. -Used as a remote snapshotter, it bypasses image pulls on the host. +Handles the container image "guest pull" functionality. +Used as a remote snapshotter, it bypasses image pulls on the host. Instead, the snapshotter fetches and unpacks encrypted and signed container images directly inside the protected guest memory, keeping proprietary contents hidden and ensuring image integrity. **Kata Agent and Agent Security Policy** -Runs inside the guest VM to manage the container lifecycle while enforcing a strict, immutable agent security policy based on Rego (regorus). +Runs inside the guest VM to manage the container lifecycle while enforcing a strict, immutable agent security policy based on Rego (regorus). This blocks the untrusted host from executing unauthorized commands, such as a malicious ``kubectl exec``. +For how to generate and attach a policy to a workload, refer to :ref:`Attach a Kata Agent Security Policy `. **Trustee and Attestation Service** -Attestation and key brokering framework (which includes the Key Broker Service and Attestation Service). +Attestation and key brokering framework (which includes the Key Broker Service and Attestation Service). It acts as the cryptographic gatekeeper, verifying hardware/software evidence and only releasing secrets if the environment is proven secure. @@ -219,7 +220,7 @@ The following features are supported with Confidential Containers: * Composite :doc:`attestation ` using Trustee and the NVIDIA Remote Attestation Service (NRAS). -* Generating Kata Agent Security Policies using the `genpolicy tool `_. +* Generating Kata Agent Security Policies. * Use of `signed sealed secrets `_. * Access to authenticated registries for container image guest-pull. * Container image signature verification and encrypted container images. @@ -235,6 +236,7 @@ Limitations and Restrictions **************************** * NVIDIA supports the GPU Operator and confidential computing with the containerd runtime only. + * All GPUs on the host must be configured for Confidential Computing. Configuring only a subset of GPUs on a node is not supported. For multi-GPU passthrough, all GPUs must be assigned to a single confidential VM. @@ -281,11 +283,11 @@ To deploy on your cluster, start with the **Install** section: .. grid:: 2 :gutter: 3 - .. grid-item-card:: :octicon:`server;1.5em;sd-mr-1` Supported Platforms + .. grid-item-card:: :octicon:`server;1.5em;sd-mr-1` Supported Platforms and Software Components :link: supported-platforms :link-type: doc - Hardware, OS, and component versions validated for general availability (GA). + Validated hardware, OS, and component versions. .. grid-item-card:: :octicon:`checklist;1.5em;sd-mr-1` Prerequisites :link: prerequisites @@ -306,4 +308,3 @@ To deploy on your cluster, start with the **Install** section: Verify the deployment; success is ``Test PASSED`` in pod logs. After installation, refer to the **Configuration** section for workload configuration, CC mode management, and attestation. - diff --git a/confidential-containers/personas.rst b/confidential-containers/personas.rst index 8c33c7925..87f8e1c81 100644 --- a/confidential-containers/personas.rst +++ b/confidential-containers/personas.rst @@ -67,10 +67,10 @@ Depending on your role, you may complete several sections or only a subset. - Installs and manages the Kubernetes cluster and the Confidential Containers software stack. - :doc:`Prerequisites ` * - :ref:`Security Engineer ` - - Validates Confidential Computing configuration, attestation policy, and secret release for workloads. + - Validates Confidential Computing configuration, attestation policy, agent security policy, and secret release for workloads. - :doc:`Attestation Quickstart ` * - :ref:`Container User ` - - Deploys confidential GPU workloads on a prepared cluster. + - Deploys confidential GPU workloads on a prepared cluster, including generating and attaching an agent security policy. - :doc:`Configuring Workloads ` .. _coco-persona-hardware-it-administrator: @@ -140,6 +140,7 @@ Relevant pages: * :doc:`Reference Architecture `: understand the use cases, trust model, and how workloads are isolated from the infrastructure. * :doc:`Attestation Quickstart `: stand up a local Trustee instance and verify connectivity. Attestation is required for workloads that use secrets, encrypted container images, or authenticated registries. + Review the :ref:`Kata agent security policy ` so the untrusted host cannot issue unauthorized Agent API calls into the guest. For production attestation workflows, secret management, and policy configuration, refer to the upstream `Confidential Containers attestation documentation `_. @@ -155,5 +156,5 @@ This persona works primarily with Kubernetes workload manifests and does not req Relevant pages: -* :doc:`Configuring Workloads `: runtime class selection, GPU and NVSwitch resource types, and single- or multi-GPU passthrough manifests. +* :doc:`Configuring Workloads `: runtime class selection, GPU and NVSwitch resource types, single- or multi-GPU passthrough manifests, and attaching a Kata agent security policy. * :doc:`Run a Sample Workload `: run the reference workload to confirm the cluster is ready before deploying your own application. diff --git a/confidential-containers/run-sample-workload.rst b/confidential-containers/run-sample-workload.rst index 544156c0f..ea4a3c5d4 100644 --- a/confidential-containers/run-sample-workload.rst +++ b/confidential-containers/run-sample-workload.rst @@ -117,7 +117,7 @@ For runtime class selection, resource type naming, multi-GPU passthrough, and ad cuda-vectoradd-kata 1/1 Running 0 10s The pod could also say ``Completed`` if the container already completed successfully. - + If the pod stays ``Pending`` for more than a few minutes, use the :doc:`Troubleshooting ` guide for more details on viewing logs and potential causes for the pod being stuck. #. View the logs from the pod after the container starts: @@ -168,7 +168,7 @@ For runtime class selection, resource type naming, multi-GPU passthrough, and ad Normal Created 71s kubelet Created container: cuda-vectoradd Normal Started 68s kubelet Started container cuda-vectoradd - The ``Runtime Class Name`` should match the runtime class used to create the pod and you will see that the pod ran successfully. + The ``Runtime Class Name`` should match the runtime class used to create the pod and you will see that the pod ran successfully. The example output shows ``Runtime Class Name: kata-qemu-nvidia-gpu-tdx``. @@ -195,7 +195,7 @@ Next Steps deployment, refer to the upstream `Confidential Containers NVIDIA attestation guide `__. -If you'd like to continue to learn about Confidential Container configuraion, review the following pages: +If you'd like to continue to learn about Confidential Container configuration, review the following pages: * Refer to :doc:`Configuring Workloads ` for runtime class selection, resource types, and multi-GPU passthrough manifests. * Refer to :doc:`Managing the Confidential Computing Mode ` to change the CC mode on GPUs at the cluster or node level. diff --git a/confidential-containers/supported-platforms.rst b/confidential-containers/supported-platforms.rst index e12c37cfc..012eddc7e 100644 --- a/confidential-containers/supported-platforms.rst +++ b/confidential-containers/supported-platforms.rst @@ -18,9 +18,9 @@ .. _coco-supported-platforms: -################### -Supported Platforms -################### +############################################## +Supported Platforms and Software Components +############################################## Following are the platforms supported by the NVIDIA Confidential Containers Reference Architecture. diff --git a/gpu-operator/confidential-containers-deploy.rst b/gpu-operator/confidential-containers-deploy.rst index 252fd5d37..cebc28418 100644 --- a/gpu-operator/confidential-containers-deploy.rst +++ b/gpu-operator/confidential-containers-deploy.rst @@ -23,7 +23,7 @@ Documentation for configuring the GPU Operator for Confidential Containers is av Deploy Confidential Containers with the NVIDIA GPU Operator on Kubernetes. - .. grid-item-card:: :octicon:`server;1.5em;sd-mr-1` Supported Platforms + .. grid-item-card:: :octicon:`server;1.5em;sd-mr-1` Supported Platforms and Software Components :link: https://docs.nvidia.com/datacenter/cloud-native/confidential-containers/latest/supported-platforms.html :link-type: url