Cyberwave Cloud Node

Turn any computer into a Cloud Node instance you can use to run inference or training with Cyberwave. This subfolder contains the documentation of the package that helps developers create Cloud Node (GPU) instances that are cyberwave-ready in a few lines of code.

How it Works

The Cloud Node connects to the Cyberwave MQTT broker to receive commands. This means:

• No public URL required - The node connects outbound to MQTT, no need to expose ports
• Firewall friendly - Works behind NAT and firewalls
• Real-time communication - Commands are received instantly via MQTT subscription

Quickstart

Let's say you have a codebase that runs training and inference. You want that to be part of a Cyberwave pipeline.

First, you install the package

pip install cyberwave-cloud-node

# or if you want the compiled version

sudo apt-get install cyberwave-cloud-node

The same apt registry also carries cyberwave-cloud-node-dev and cyberwave-cloud-node-staging for explicit channel installs. Default cyberwave-cloud-node is tagged production releases; the channel packages conflict because they ship the same cyberwave-cloud-node binary.

On non-apt platforms, prerelease Python wheels are published to the Buildkite Python registry and consumed automatically by cyberwave compute install --channel dev|staging. Stable pip installs continue to use the public PyPI release.

Then to the root of your repository you add a yaml file like this

# cyberwave.yml
cyberwave-cloud-node:
  install_script: ./install.sh          # install what you need in the cloud GPU
  inference: python ./inference.py --params {body}  # {body} renders to JSON params from MQTT
  simulate: python ./simulate.py --params {body}
  training: python ./training.py --params {body}
  profile_slug: gpu-a100                # optional: node profile (default: "default")
  heartbeat_interval: 30                # optional: heartbeat interval in seconds
  mqtt_host: mqtt.cyberwave.com         # optional: custom MQTT broker
  mqtt_port: 8883                       # optional: custom MQTT port (default: 8883 with TLS)

Behind the scenes, Cyberwave Cloud Node will take care of:

• Running the install script on startup and notifying you if it fails
• Registering with the Cyberwave backend to get a UUID and slug
• Storing identity locally for re-registration support
• Connecting to MQTT to receive commands
• Sending periodic heartbeats
• Processing inference and training requests as independent OS processes
• Monitoring workload processes and collecting results when they complete
• Graceful shutdown without interrupting running workloads:
  • Rejects new commands during shutdown
  • Workloads continue running as independent processes
  • Flushes logs and notifies backend before terminating

Authentication

The Cloud Node needs an API token to communicate with Cyberwave. You can provide it in several ways (in order of priority):

1. Environment variable: export CYBERWAVE_API_KEY=your-token
2. .env file in current directory:

   # .env
   CYBERWAVE_API_KEY=your-token
   CYBERWAVE_WORKSPACE_SLUG=my-workspace

3. .env file in ~/.cyberwave/.env (shared config)
4. Stored credentials from cyberwave-cli login (~/.cyberwave/credentials.json)

If you've already logged in with cyberwave-cli, the Cloud Node will automatically use those credentials.

On start, the node validates the token against the backend before connecting to MQTT. If the token is invalid or revoked, it exits immediately with a clear error (HTTP 401/403) instead of silently looping on rejected heartbeats. A transient backend outage is logged as a warning and does not block startup.

Environment Variables

Required

• CYBERWAVE_API_KEY: Your Cyberwave API token

Optional - API & Workspace

• CYBERWAVE_WORKSPACE_SLUG: Your workspace slug
• CYBERWAVE_INSTANCE_SLUG: Instance slug hint (useful for automated deployments)
• CYBERWAVE_BASE_URL: API URL (default: https://api.cyberwave.com)

Optional - MQTT

• CYBERWAVE_MQTT_HOST: MQTT broker host (default: mqtt.cyberwave.com)
• CYBERWAVE_MQTT_PORT: MQTT broker port (default: 8883)
• CYBERWAVE_MQTT_USERNAME: MQTT username if required
• CYBERWAVE_MQTT_PASSWORD: MQTT password if required; falls back to CYBERWAVE_API_KEY
• CYBERWAVE_ENVIRONMENT: Environment prefix for MQTT topics (empty for production)

Optional - Commands (alternative to cyberwave.yml)

• CYBERWAVE_INSTALL_SCRIPT: Install script command
• CYBERWAVE_INFERENCE_CMD: Inference command template
• CYBERWAVE_SIMULATE_CMD: Simulation command template
• CYBERWAVE_TRAINING_CMD: Training command template
• CYBERWAVE_PROFILE_SLUG: Node profile slug (default: "default")
• CYBERWAVE_HEARTBEAT_INTERVAL: Heartbeat interval in seconds (default: 30)
• CYBERWAVE_NODE_ENVIRONMENT_UUID: Reserve this node for a single Cyberwave environment. Only that environment's workloads are scheduled on the node; workloads from other environments (or with no environment) skip it. Can also be set via environment_uuid in cyberwave.yml or the --environment CLI flag (CLI flag > config file > env var).

CLI Usage

# Start the cloud node (backend assigns UUID and slug)
export CYBERWAVE_API_KEY=your-token-here
cyberwave-cloud-node start

# With a slug hint (backend may use this or assign a different one)
cyberwave-cloud-node start --slug my-gpu-node

# With custom config file
cyberwave-cloud-node start --config ./path/to/cyberwave.yml

# With profile override
cyberwave-cloud-node start --profile gpu-a100

# Reserved for a single environment (only that environment's workloads run here)
cyberwave-cloud-node start --environment 11111111-2222-3333-4444-555555555555

# With custom MQTT broker (local dev, no TLS)
cyberwave-cloud-node start --mqtt-host localhost --mqtt-port 1883

# Verbose logging
cyberwave-cloud-node start -v

Note: The --slug parameter is a hint. The backend is the owner of UUIDs and slugs - it may use your hint or assign a different one.

Programmatic Usage

from cyberwave_cloud_node import CloudNode, CloudNodeConfig

# From config file (recommended)
node = CloudNode.from_config_file()
node.run()

# With a slug hint
node = CloudNode.from_config_file(slug="my-gpu-node")
node.run()

# From environment variables
node = CloudNode.from_env()
node.run()

# Or fully programmatic
config = CloudNodeConfig(
    install_script="./install.sh",
    inference="python inference.py --params {body}",
    training="python train.py --params {body}",
    profile_slug="gpu-a100",
    heartbeat_interval=30,
    mqtt_host="mqtt.cyberwave.com",
    mqtt_port=8883,
)
node = CloudNode(config=config, slug="my-gpu-node")
node.run()

Instance Identity

After successful registration, the backend assigns a UUID and slug to your node. This identity is stored locally in ~/.cyberwave/instance_identity.json for:

• Re-registration: If the node restarts, it will re-register with the same identity
• Debugging: You can inspect the file to see your node's assigned UUID and slug

MQTT Topics

The cloud node subscribes to command topics and publishes responses:

Command Topics (subscribes to):

• cyberwave/cloud-node/{instance_uuid}/command
• cyberwave/cloud-node/{slug}/command

Response Topic (publishes to):

• cyberwave/cloud-node/{instance_uuid}/response

Command Message Format

{
  "command": "inference",
  "request_id": "unique-request-id",
  "params": {
    "model": "gpt-4",
    "input": "Hello world"
  }
}

Supported commands:

• inference - Run the inference command
• simulate - Run the simulation command
• training - Run the training command
• status - Get node status (includes active workloads)
• cancel - Cancel a running workload by PID or request_id

Response Message Format

{
  "status": "ok",
  "request_id": "unique-request-id",
  "slug": "my-gpu-node",
  "instance_uuid": "abc-123",
  "output": "Command output here"
}

On error:

{
  "status": "error",
  "request_id": "unique-request-id",
  "slug": "my-gpu-node",
  "instance_uuid": "abc-123",
  "error": "Error message here"
}

Cancelling Workloads

To cancel a running workload, send a cancel command:

Cancel by PID:

{
  "command": "cancel",
  "request_id": "unique-request-id",
  "params": {
    "pid": 12345
  }
}

Cancel by workload request_id:

{
  "command": "cancel",
  "request_id": "unique-request-id",
  "params": {
    "workload_request_id": "original-training-request-id"
  }
}

Cancel with specific signal:

{
  "command": "cancel",
  "request_id": "unique-request-id",
  "params": {
    "pid": 12345,
    "signal": "SIGKILL"
  }
}

Supported signals:

• SIGTERM (default) - Graceful termination, allows cleanup
• SIGINT - Interrupt signal (like Ctrl+C)
• SIGKILL - Force kill, immediate termination

Cancel Response:

{
  "status": "ok",
  "request_id": "unique-request-id",
  "slug": "my-gpu-node",
  "instance_uuid": "abc-123",
  "output": {
    "message": "Workload cancelled with SIGTERM",
    "pid": 12345,
    "workload_type": "training",
    "workload_request_id": "original-training-request-id",
    "signal": "SIGTERM"
  }
}

Process-Based Workload Management

Training and inference jobs run as independent OS processes that survive Cloud Node restarts. This design ensures:

How It Works

1. Command Received: When a training/inference command arrives via MQTT

   • Cloud Node spawns a detached subprocess (using start_new_session=True)
   • Process runs independently with its own PID
   • Output streams to log files in ~/.cyberwave/workload_logs/

2. Background Monitoring: The workload monitor loop:

   • Checks every 5 seconds if workload processes are still alive
   • Collects results when processes complete
   • Publishes completion status and output back via MQTT

3. Node Capacity & Status:

   • Cloud Node tracks active workloads by PID
   • Rejects new workloads when busy (configurable for concurrent workloads)
   • Status command returns:

     {
       "slug": "my-gpu-node",
       "instance_uuid": "abc-123",
       "is_busy": true,
       "active_workloads": [
         {
           "pid": 12345,
           "type": "training",
           "request_id": "abc-123",
           "running_for_seconds": 3600
         }
       ]
     }

4. Stale-Workload Self-Healing:

   • The node re-checks each tracked workload with Cyberwave and terminates the ones that are no longer active, so a leftover process cannot keep the node busy forever
   • Runs at startup, before every heartbeat, and before rejecting an incoming workload for being busy — so the capacity the node reports reflects what is actually still running
   • An inconclusive check (Cyberwave temporarily unreachable, for example) always keeps the process: a transient network problem never kills a healthy workload
   • A process that cannot be terminated stays tracked, so the node keeps reporting itself busy rather than accepting work it cannot run
   • Terminating a stale workload never emits a duplicate completion for it
   • The check is time-boxed so it can never delay the node's heartbeat; an unfinished pass is simply retried on the next one

5. Result Uploads Keep the Node Busy:

   • A workload is not finished when its process exits — the node still collects its output and uploads its result files, which for large artifacts takes minutes
   • The node reports itself busy for that whole window and declines new workloads, so an upload is never starved by a job started on top of it
   • A declined workload is not lost: Cyberwave puts it back in the queue and retries it, with a growing delay, once a host is genuinely free
   • The node also serialises its own start handling, so two start requests that arrive at the same moment can never both land on it

6. Workload Cancellation:

   • Backend can cancel workloads by PID or request_id
   • Supports graceful (SIGTERM) or force (SIGKILL) termination
   • Sends cancellation notification to original workload request

7. Graceful Shutdown: When Cloud Node shuts down (Ctrl+C, SIGTERM, SIGINT):

   • ✅ Stops accepting new commands
   • ✅ Cancels background monitoring tasks
   • ✅ Logs running workload PIDs and output file locations
   • ✅ Workloads continue running in background
   • ✅ Results will be available in log files

Benefits

• Resilience: Cloud Node can restart/upgrade without killing training jobs
• Fast Shutdown: No waiting - shutdown completes immediately
• Process Independence: Training jobs aren't tied to Cloud Node lifecycle
• Crash Recovery: Workloads survive Cloud Node crashes
• Simple Monitoring: Just check PID to see if workload is alive

Output Files

Workload output is streamed to:

~/.cyberwave/workload_logs/
├── training_<request_id>.stdout.log
├── training_<request_id>.stderr.log
├── inference_<request_id>.stdout.log
└── inference_<request_id>.stderr.log

These files are also streamed to the backend during the run (not only at exit) for live monitoring: a background loop tails each active workload's stdout/stderr from the last-sent byte offset and forwards new content over the same MQTT log path. The completion/cancel handler sends only the remaining tail, so nothing is double-sent. This is what powers the "Policy decision logs" panel for online RL inference.

The same tailed content is also mirrored to the node's own stdout/stderr, so it shows up in docker logs for the container. Without this, docker logs only shows the supervisor's logs: the workload runs as a detached subprocess whose fds point at the log files above, never at the container stdout. Each mirrored line is tagged [workload <id> <stdout|stderr>] so it stays distinguishable from the supervisor's own logging-formatted lines, and stdout/stderr remain separated (the two files, and their MQTT streams, are untouched). Enable per-step inference detail with CYBERWAVE_RL_DEBUG=1 (see cyberwave-rl-task).

• CYBERWAVE_WORKLOAD_LOG_STREAM_INTERVAL (default 3, seconds): cadence of the live tail — this also sets how quickly mirrored output reaches docker logs. Lower = more responsive panel, more MQTT traffic.
• The periodic log-flush loop is now crash-resistant: a single unexpected error no longer permanently stops log persistence (previously any exception other than MQTTError/TimeoutError killed the flush loop and silenced all further logs).

Manifest Schema

Starting with cyberwave>=0.3.46, cyberwave.yml is validated against a Pydantic v2 schema (cyberwave.manifest.ManifestSchema). Both key formats work:

# New format (preferred)
cyberwave:
  inference: inference.py

# Legacy format (still supported)
cyberwave-cloud-node:
  install_script: ./install.sh
  inference: python server.py {body}

Migration notes

Old convention	New convention	Notes
cyberwave-cloud-node: wrapper key	cyberwave: wrapper key	Both accepted; validator warns on legacy key
install_script: field	install: field	Both accepted; effective_install normalises
All fields in extra: dict silently	extra = "forbid" — unknown fields error	Use --lenient to demote to warnings during migration
Shell-only dispatch	Module dispatch for .py values	inference: inference.py calls infer() directly

Validating locally

cyberwave manifest validate cyberwave.yml
cyberwave manifest validate cyberwave.yml --lenient

New fields available

• workers: — list of .py files using @cw.on_frame hooks (loaded at startup)
• requirements: — pip package specs (parsed; installation not yet performed)
• models: — model IDs to pre-download (parsed; download not yet performed)
• input: — input declaration (string normalised to list)
• gpu: — hardware routing flag
• resources: — memory / cpus constraints
• version: — schema version (default "1")

TODO

• Try it on a real repo - I cloned https://github.com/cyberwave-os/cyberwave-compute-smolvla into this workspace
• Add a Nuitka github action to build it as a binary

MuJoCo Docker Example

Inside this monorepo there is a runnable local example for the simulate command at cyberwave-cloud-nodes/tests/cyberwave-sim/. It starts:

• a cyberwave-cloud-node worker container
• a simulation service container (cyberwave-sim)

Use that example when you want your local machine to behave like a real simulation cloud node with minimal manual setup.