WebSocket Protocol

WebSocket Protocol

The computing-provider communicates with Swan Inference over a persistent WebSocket connection at <WebSocketURL>/ws. All messages are JSON with a type field.

---

Connection flow

Provider                          Swan Inference
   │──── connect ws://.../ws ────▶│
   │──── register ───────────────▶│  announce models + auth token
   │◀─── ack (registered) ────────│
   │──── model_health_update ────▶│  send initial health status
   │
   │  [ steady state ]
   │
   │◀─── inference ───────────────│  incoming request
   │──── stream_chunk (×N) ──────▶│  streaming response
   │──── stream_end ─────────────▶│
   │
   │◀─── warmup ──────────────────│  pre-load a model
   │──── ack (warmup result) ────▶│
   │
   │──── heartbeat (every 54s) ──▶│
   │◀─── ack ─────────────────────│

---

Message types

register → server

Sent immediately after connecting. Authenticates the provider and declares available models.

{
  "type": "register",
  "request_id": "uuid",
  "payload": {
    "node_id": "04c8257b...",
    "node_name": "my-provider",
    "token": "sk-prov-xxxxxxxxxxxxxxxxxxxx",
    "models": ["meta-llama/Llama-3.2-3B-Instruct"],
    "capabilities": [],
    "hardware": {
      "gpu_type": "rtx3080",
      "gpu_model": "NVIDIA RTX 3080",
      "vram_gb": 10,
      "gpu_count": 1,
      "serving_engine": "sglang"
    }
  }
}

inference ← server

Incoming inference request. The request field is a raw OpenAI-compatible JSON body.

{
  "type": "inference",
  "request_id": "uuid",
  "payload": {
    "endpoint_id": "uuid",
    "model_id": "meta-llama/Llama-3.2-3B-Instruct",
    "stream": true,
    "request": { "model": "...", "messages": [...], "max_tokens": 512 }
  }
}

stream_chunk → server

One SSE chunk from the model server, forwarded to Swan Inference.

{
  "type": "stream_chunk",
  "payload": {
    "request_id": "uuid",
    "chunk": { "id": "...", "choices": [...] },
    "done": false
  }
}

stream_end → server

Signals end of stream with usage stats.

{
  "type": "stream_end",
  "payload": {
    "request_id": "uuid",
    "latency_ms": 1234,
    "tokens_input": 42,
    "tokens_output": 128
  }
}

heartbeat → server

Sent every ~54 seconds to maintain the connection and report metrics.

{
  "type": "heartbeat",
  "payload": {
    "node_id": "04c8257b...",
    "timestamp": 1718000000,
    "models": ["meta-llama/Llama-3.2-3B-Instruct"],
    "model_health": {
      "meta-llama/Llama-3.2-3B-Instruct": "healthy"
    },
    "metrics": {
      "gpu_utilization": 0.45,
      "vram_used_gb": 7.2
    }
  }
}

model_health_update → server

Sent when a model's health status changes (healthy / degraded / unhealthy).

{
  "type": "model_health_update",
  "payload": {
    "node_id": "04c8257b...",
    "model_health": {
      "meta-llama/Llama-3.2-3B-Instruct": "healthy"
    },
    "timestamp": 1718000000
  }
}

warmup ← server

Swan Inference asks the provider to pre-load a model to reduce cold-start latency.

{
  "type": "warmup",
  "payload": {
    "model_id": "meta-llama/Llama-3.2-3B-Instruct",
    "warmup_type": "inference"
  }
}

The provider responds with an ack carrying WarmupResponse fields (load_time_ms, memory_mb, success).

ack (both directions)

Generic acknowledgment.

{
  "type": "ack",
  "request_id": "uuid",
  "payload": {
    "request_id": "uuid",
    "success": true,
    "message": "registered successfully"
  }
}

error (both directions)

{
  "type": "error",
  "payload": {
    "request_id": "uuid",
    "code": 401,
    "message": "invalid provider API key"
  }
}

---

Health check

The computing-provider polls each model endpoint every 30 seconds with GET <endpoint>/v1/models. A 200 response marks the model healthy; three consecutive failures mark it unhealthy and Swan Inference stops routing requests to it.

---

Notes

• WebSocketURL in config.toml must not include /ws — the client appends it automatically. ws://localhost:8081 → connects to ws://localhost:8081/ws.
• Max message size: 1 MB.
• Ping/pong keep-alive: 54 s ping period, 60 s read deadline.
• On reconnect the provider re-sends register and model_health_update automatically.