Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 15 additions & 4 deletions docs/apps/compute.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,11 @@ To deploy an image from a private registry (e.g., Amazon ECR, Google Artifact Re

Enter your SSH public key to configure access to the instance, select a GPU instance type, and click Deploy.

Optional settings on the create form:

- **Enable logging** — collect container output and show it on the deployment **Logs** tab.
- **Annotations** — optional key/value metadata on the instance.

<Frame>
<img src="/images/comp_1.png" alt="Compute instance launch page" style={{ borderRadius: '0.5rem' }} />
</Frame>
Expand Down Expand Up @@ -74,16 +79,22 @@ The interactive shell depends on what your image provides:

The instance includes the libraries from your image. Additional packages can be installed with your preferred package manager.

### In-console Terminal

While the instance is **Active**, the deployment **Terminal** tab opens a shell in the browser without SSH from your machine. Use SSH (above) for `scp`/`SFTP` and port forwarding. See [Managing Deployments](/resources/managing_deployments) for Terminal availability, pause/resume, clone, and logs.

Compute deployments are not revision-aware in the console: edits update the instance in place and there is no rollback table.


## What's next

<CardGroup cols={3}>
<Card
title="Clients"
icon="terminal"
href="/clients/setup"
title="Managing Deployments"
icon="sliders"
href="/resources/managing_deployments"
>
Learn how to interact with the NVIDIA CCluster programmatically
In-console Terminal, logs, pause, and clone.
</Card>
<Card
title="Managing Vault Objects"
Expand Down
14 changes: 9 additions & 5 deletions docs/apps/dynamo.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,11 @@ Open the deployment to access:
- **Terminal** — an interactive shell for an active deployment.
- **Usage** — deployment resource usage.

Editing a Dynamo deployment creates a new revision and starts an operator-managed rolling update. You can inspect historical revision status and logs or roll back to an earlier revision. Pausing or resuming the whole deployment is supported; pausing only an in-progress Dynamo rollout is not.
<Note>
Changing worker **Min Replicas** or **Max Replicas** on a running Dynamo deployment scales in place and **does not create a new revision**. Editing the spec (model, protocol, extra arguments, and similar) creates a new revision. See [Managing Deployments](/resources/managing_deployments) for Terminal, pause/resume, clone, and the shared revision model.
</Note>

Editing a Dynamo deployment creates a new revision and starts an operator-managed rolling update. You can inspect historical revision status and logs, clone a deployment, or roll back to an earlier revision. Pausing or resuming the whole deployment is supported; pausing only an in-progress Dynamo rollout is not.

The console's **Edit** and **Clone** flows currently support aggregated deployments only. For a disaggregated deployment, use the Dynamo API to update supported fields or create a copy from the `worker_pools` configuration returned by the API.

Expand Down Expand Up @@ -190,11 +194,11 @@ The current integration has a single Frontend and does not expose Frontend high

<CardGroup cols={3}>
<Card
title="Securing Endpoints"
icon="lock"
href="/resources/private"
title="Managing Deployments"
icon="sliders"
href="/resources/managing_deployments"
>
Protect your endpoint with a Bearer token or mTLS client certificate.
Terminal, pause, clone, revisions, and replica scaling.
</Card>
<Card
title="NVIDIA Dynamo Frontend"
Expand Down
45 changes: 35 additions & 10 deletions docs/apps/inference.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,34 @@ At a minimum, you need to specify:
- **Health check path** — the endpoint used to verify readiness (e.g., `/health`, `/`). For gRPC deployments, CCluster uses TCP socket checks automatically, so this field is ignored.
- **Image Registry Username / Password** — required only if the image is hosted on a private registry (e.g., Docker Hub private repo). You can also use credentials stored in your [Vault](/resources/vault).

Under the **Optional Details** tab:
The create wizard splits optional settings across **Advanced** and **Scaling**.

**Advanced**

- **Enable logging** — collect container stdout/stderr and show it on the deployment **Logs** tab. If this is off, that revision has no logs in the console.
- **Enable Model Caching** — store model files in a node-local persistent cache. Restarts and replicas scheduled on the same node can reuse the files instead of downloading them again.
- **Session affinity** — best-effort sticky routing using the `X-Session-Id` request header. Requests with the same header value land on the same pod, which helps KV-cache reuse for agentic workloads. Requests without the header are routed at random. Affinity is **not** durable: scaling, rollouts, restarts, or readiness-probe transitions can remap sessions to different pods. Do not use it for irreplaceable in-pod state.
- **Health check path** — the endpoint used to verify readiness (also listed in the required fields above).
- **Add command** — entrypoint command and arguments. If left empty, the image's default entrypoint is used.
- **Autoscaling** — set the min and max scale for your deployment. CCluster scales replicas based on max concurrency (maximum in-flight requests per replica). Default is infinity.
- **Environment variables** — pass additional environment variables to the container (e.g., `HF_TOKEN`).
- **Metrics port** / **Metrics path** — optional Prometheus scrape endpoint. Set **both** together, or leave both empty to use the defaults (the container port and `/metrics`).
- **Environment variables** — pass additional variables to the container (e.g., `HF_TOKEN`).
- **Config file** — mount a file into the container. Provide **Filename**, **Mount Path (directory)**, and **File Content**. For worked examples, see [gRPC Inference Endpoints](/resources/grpc) and [Deploying TensorRT-LLM](/resources/trtllm).
- **Annotations** — optional key/value metadata on the deployment.

**Scaling**

CCluster scales replicas from in-flight request concurrency (**Max Concurrency**). Leave concurrency unset for no concurrency target (effectively unlimited).

| Setting | Description |
| --- | --- |
| **Min Replicas** | Lower bound for replica count. `0` allows scale-to-zero when demand drops. |
| **Max Replicas** | Upper bound for replica count. Set equal to **Min Replicas** for a fixed size. |
| **Max Concurrency** | Target in-flight requests per replica. Required when **Max Replicas** is greater than **Min Replicas**. |
| **Cooldown Period (s)** | Scale-down stabilization time, from `0` to `3600`. |

<Note>
Changing min or max replica counts on a running deployment scales in place and **does not create a new revision**. Spec changes such as image, command, or affinity do. See [Managing Deployments](/resources/managing_deployments).
</Note>

<Frame>
<img src="/images/inf_1.png" alt="General Inference deployment page" style={{ borderRadius: '0.5rem' }} />
Expand Down Expand Up @@ -109,6 +132,8 @@ Click on the deployment to view the details page, logs and monitoring informatio
<img src="/images/inf_4.png" alt="Deployment details panel after launching a general inference endpoint" style={{ borderRadius: '0.5rem' }} />
</Frame>

From the detail page you can stream **Logs**, open the in-browser **Terminal** while the deployment is Active, pause or resume, clone, edit (which creates a new revision), or roll back. See [Managing Deployments](/resources/managing_deployments).

Once the deployment status is ready, the container port is going to be exposed under the endpoint url shown in the details page.

### Accessing your endpoint
Expand Down Expand Up @@ -140,20 +165,20 @@ For gRPC deployments, see [gRPC Inference Endpoints](/resources/grpc) for connec
## What's next

<CardGroup cols={3}>
<Card
title="Managing Deployments"
icon="sliders"
href="/resources/managing_deployments"
>
Logs, Terminal, pause, clone, revisions, and replica scaling.
</Card>
<Card
title="Securing Endpoints"
icon="lock"
href="/resources/private"
>
Learn how to protect your endpoints with Bearer tokens and mTLS certificates
</Card>
<Card
title="Clients"
icon="terminal"
href="/clients/setup"
>
Learn how to interact with the NVIDIA CCluster programmatically
</Card>
<Card
title="LLM Serving"
icon="messages"
Expand Down
11 changes: 6 additions & 5 deletions docs/apps/jobs.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ Expand the **Optional Details** section to tune how the job executes. All of the
- **Enable logging** — toggle to collect and view logs from the job's pods.
- **Environment variables** — pass additional environment variables to the container (e.g., `HF_TOKEN`).
- **Annotations** — attach custom key/value metadata to the job.
- **Config file** — mount a file into the container. Provide **Filename**, **Mount Path (directory)**, and **File Content**. Use this for startup scripts or small configs your command can read.

<Note>
The environment variable names `JOB_COMPLETION_INDEX` and `JOB_COMPLETIONS` are reserved by the platform and cannot be used for your own variables. CCluster runs jobs in **indexed** completion mode and automatically injects both variables into every replica so each one can determine which slice of work it owns:
Expand All @@ -49,19 +50,19 @@ You can integrate your own private cluster into CCluster through bring-your-own-

## 4. Monitor your job

Once created, your job appears in the deployments listing view alongside its current status. Click into the job to view its details page, including per-pod status and, when **Enable logging** is turned on, the logs from each pod.
Once created, your job appears in the deployments listing view alongside its current status. Click into the job to view its details page, including per-pod status and, when **Enable logging** is turned on, the logs from each pod. While pods are running, the **Terminal** tab opens an in-browser shell. Jobs are not revision-aware and have no pause/resume or replica autoscaling. See [Managing Deployments](/resources/managing_deployments).

Because jobs are batch workloads, there is no endpoint to call — the job runs until your container exits. A job is considered complete once the requested number of **Completions** finish successfully, and failed pods are retried up to the **Backoff Limit** (or until the **Active Deadline Seconds** is reached).

## What's next

<CardGroup cols={3}>
<Card
title="General Inference"
icon="circle-nodes"
href="/apps/inference"
title="Managing Deployments"
icon="sliders"
href="/resources/managing_deployments"
>
Deploy long-running, scalable inference endpoints for your models.
Follow logs and use the in-console Terminal while the job is running.
</Card>
<Card
title="Deploying Custom Models"
Expand Down
31 changes: 27 additions & 4 deletions docs/apps/llm.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -36,12 +36,35 @@ Each configuration is accompanied by detailed analysis on:

These insights help you choose the configuration that best meets your needs.

### Optional details

Expand **Optional Details** for runtime, observability, and scaling controls. These are independent of the preset (or Custom) performance configuration.

- **Enable logging** — collect container output and show it on the deployment **Logs** tab.
- **Enable Model Caching** — store model files in a node-local persistent cache so restarts and replicas on the same node can skip re-downloading weights.
- **Enable session affinity** — best-effort sticky routing via the `X-Session-Id` request header. Requests with the same header value land on the same pod; requests without it are routed at random. Affinity is not durable across scaling, rollouts, or restarts.
- **Environment variables** — extra variables passed to the serving container.
- **Annotations** — optional key/value metadata.

| Setting | Description |
| --- | --- |
| **Min Replicas** | Lower bound for replica count. |
| **Max Replicas** | Upper bound for replica count. Set equal to **Min Replicas** for a fixed size. |
| **Max Concurrency** | Target in-flight requests per replica. Required when max is greater than min. |
| **Cooldown Period (s)** | Scale-down stabilization time, from `0` to `3600`. |

<Note>
Scaling min or max replicas on a running deployment does **not** create a new revision. Changing the serving spec does. See [Managing Deployments](/resources/managing_deployments).
</Note>

### (Optional) Performance customization
For advanced users, NVIDIA CCluster also offers an option to customize their model performance configuration. Simply click the "Custom" configuration to gain full control over several tunable parameters.

## 3. Deploy and integrate
Finally, click "Deploy". Once the deployment is ready in a few minutes, copy the endpoint url and go to `https://<endpoint_url>/docs` to find the list of API endpoints to start using your LLM deployment. We offer API compatibility with CServe, OpenAI, and Cortex, making integration with other applications seamless.

From the deployment detail page you can stream logs, open the in-browser **Terminal**, pause or resume, clone, edit (new revision), or roll back. See [Managing Deployments](/resources/managing_deployments).

Endpoints are never public, so every request must authenticate with a Bearer token or a client certificate. The example below uses a Bearer token; see [Securing Endpoints](/resources/private) for all options.

```bash
Expand Down Expand Up @@ -77,11 +100,11 @@ For more details on how to use the LLM deployment, please refer to the [examples

<CardGroup cols={3}>
<Card
title="Clients"
icon="terminal"
href="/clients/setup"
title="Managing Deployments"
icon="sliders"
href="/resources/managing_deployments"
>
Learn how to interact with the NVIDIA CCluster programmatically
Logs, Terminal, pause, clone, revisions, and replica scaling.
</Card>
<Card
title="Securing Endpoints"
Expand Down
Loading
Loading