[Feature] Presets#4058
Open
peterschmidt85 wants to merge 15 commits into
Open
Conversation
Sequential experimental trials for endpoint preset creation: the agent searches serving configurations across hardware, frameworks, and variants in `dstack` tasks, records reproducible trial records (trials.jsonl), promotes the best to a verified service, and saves the preset. Verified across nine live e2e sessions (Qwen2.5-0.5B, Qwen3-32B on H100, Qwen3-32B under $1/hr — best result 533 tok/s on a single RTX 5090 via NVFP4 + EAGLE3). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ycle list - Preset ID minted at session start and stable through run names, the session directory, and the final preset; sessions survive Ctrl+C, hard kills, and host restarts and continue with --resume <preset id> - Agent deaths without a submitted report retry in-process; the retry budget refills only when the failed attempt made progress - Durable workspace under the session directory with a /tmp symlink alias (Unix socket path limit); byte-offset persistence prevents duplicated output on resume; stale running manifests downgrade to interrupted - Flat ~/.dstack/presets/<id>/ store holding preset.yaml next to session internals; lazy migration from the models--* layout; delete archives to .archive/ instead of destroying - dstack endpoint preset list shows in-flight and interrupted sessions with colored statuses, trial progress after the status, best-trial benchmark and GPU, -w watch mode, and -v details - Top-level base:/repo: configuration shorthand; nested model: deprecated unless model.name is set; --base/--repo filters on list and delete Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- A watchdog process holds a pipe from the CLI and kills the agent's process tree the moment the pipe snaps — covering deaths the CLI cannot react to (SIGKILL included), which previously left an orphaned agent able to keep submitting GPU runs. Graceful stops disarm the watchdog. - POSIX kills the agent's process group; Windows kills the process tree - Presets and sessions sort newest-first within each base-model group, matching dstack ps Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…sion A final report echoing a known secret (e.g. the bearer token in benchmark.command) was rejected wholesale, discarding an otherwise verified session at its very last step. Scrub known secret values from the report structure before validation; the bearer check still rejects unknown leaked tokens. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ish list
- Configuration `type: preset`; flat `dstack preset {list,get,create,apply,delete}`
command tree; user-facing strings follow (internal names are a follow-up)
- `apply` selects among repeatable `--id` candidates (given order,
capacity-aware) replacing the machine-local `preset:` configuration field
- The bearer-token report guard only rejects credential-shaped values;
prose such as "auth via bearer header from env" failed two live sessions
- Preset rows show `success (n/m)` from the completed creation session;
trial count is the record count again (one long-lived task commonly hosts
several trials, so distinct task names undercount)
- List polish: dimmed base group rows and trial progress, green `success`
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- BENCHMARK renders identically with and without -v; the verbose-only detail dump is gone (full data remains in `preset get --json`) - -v adds exactly two things: a `ctx=` benchmark prefix (replacing the CONTEXT column) and a dimmed indented `repo=` on the preset's own row - `apply` shows the selected preset with the same `ctx= con= tok/s TTFT` string Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- `prompt:` in the configuration, inline or as a file `path:` relative to the configuration file; validated and capped - The agent contract gains the text via template directives in system_prompt.md (`<!--?prompt:...-->`); without a prompt the rendered contract stays byte-identical. The default no-patching limits gain an escape clause the prompt can invoke explicitly - The resolved prompt is pinned per session and kept across resumes, with a warning when the configuration changes meanwhile - The list shows `verifying` once a running session's trial budget is spent and the final service is being deployed and verified Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ation - The configuration `name` names the preset, claimed at session start: creating with a used name detaches it from the current holder (preset or session of any status) after confirmation, docker-tag style; the old preset survives nameless. `get`, `delete`, and `apply --id` accept names wherever they accept ids; the list gains a NAME column - Name-less creation is allowed; run names derive from a model slug - `create` shows an apply-style plan before starting the agent — Project, User, the effective fleets, and their offers (agent-free, best-effort) — and always asks for confirmation; `-y` skips - `print_offers` extracted from `print_run_plan` for reuse Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The agent's streams go to workspace files instead of a pipe into the CLI, so the agent survives CLI death and a later attach continues parsing from the persisted offsets. The session manifest tracks the agent's own pid, and list liveness keys on it, so a CLI-less agent shows as running (clauding/verifying) rather than interrupted. - Ctrl+C during create/attach asks to stop or detach; detaching (or any hard CLI death) leaves the agent working and visible - `dstack preset attach <id|name>` follows a detached session to completion and finalizes it; `stop <id|name>` terminates the agent and offers to stop its active runs - Stopping a session offers to stop its non-terminal runs; the cost warning only shows when the user keeps them for a later resume - Preset name conflicts prompt to reassign the name, leaving the old preset name-less - The watchdog is removed: CLI death now means detach, which is safe because detached sessions stay visible and attachable Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The stream readers and the record mirrors each open a separate _OffsetStore on the same file with disjoint keys; writing the whole in-memory view dropped the other store's offsets, causing duplicated mirror records on the next attach or resume. Update only the written key, re-reading the file first. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Statuses use the same palette as `dstack ps` (verified=grey, clauding= sea_green3, verifying/interrupted/failed matching), and NAME becomes a trailing dimmed column so the data columns stay contiguous. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`model` is what makes the run a served, readiness-checked model endpoint — i.e. a preset. Tell the agent a service that never passes the readiness probe is a real failure, not something to work around by removing `model`. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
# Conflicts: # src/dstack/_internal/cli/commands/endpoint.py # src/dstack/_internal/cli/utils/run.py
… read Replace `dstack preset attach` with `dstack preset logs [-f]`: dump a preset creation's log at any status; `-f` re-owns and finalizes a detached or running creation, and extra followers tail read-only instead of being rejected. Reconcile-on-read: `list`, `get`, and `apply` finalize a completed-but-orphaned creation from its on-disk report, so the preset is saved regardless of how the CLI went away (detached, killed, or crashed). Use advisory file locks (flock/msvcrt) for creation ownership instead of pid claims — no steal race, no stale locks, kernel auto-release on exit. Ctrl+C during create reports Detached/interrupted via the interrupt handler; stop auto-stops runs with a spinner and never re-saves a finished creation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rename the preset feature's internal identifiers, modules, and directories from endpoint/Endpoint to preset/Preset. CLI-only — no server, core, or API code is involved (that design was already removed). Notable moves: cli/services/endpoints -> cli/services/presets, cli/commands/endpoint.py -> preset.py, and the models split into configurations.py (PresetConfiguration) and presets.py (Preset). Also drops the redundant `endpoint` session-manifest key, which held the same value as `name`. Require max_trials for preset creation instead of silently defaulting to 3: create now errors unless it is set in the configuration or passed via --max-trials. The model field stays optional so apply and partial configs are unaffected. Trim the saved-preset line to `Preset <id> saved`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
peterschmidt85
marked this pull request as ready for review
July 22, 2026 18:33
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
dstack preset— agent-driven search for the best model serving configuration. A headless agent (Claude Code) runs sequential trials on real fleets — hardware, framework, quantization, engine parameters — benchmarks each, deploys the best as a verifieddstackservice, and saves the result as a reusable preset. Purely client-side: works against any existing dstack server.Key changes
type: presetconfiguration:base:orrepo:,max_trials(required),concurrency,context_length,fleets,env, andprompt:— custom agent instructions (inline or a filepath:) that can reshape objectives and explicitly extend the default limitsdstack presetCLI:create(--resume ID),list— a live dashboard of presets and in-progress creations,logs [-f]— dump or follow a creation's log,apply(repeatable--id),get,delete,stopname(optional): the name is a mutable pointer, docker-tag style — re-creating with a used name reassigns it after confirmation, leaving the old preset name-less;get/delete/--idaccept namescreateshows an apply-style plan first — the effective fleets and their offers, agent-free — and asks for confirmation before the agent startsclauding/verifying);logs -fre-follows it to completion (extra viewers tail read-only),stopends it and its runs. However the CLI goes away, a completed creation is finalized and saved on the next read (list/get/apply) — reconcile-on-readHow to test
Install from this branch:
Prerequisites: any dstack server with a GPU-capable backend (the feature is client-side, so an existing deployment works as is), at least one fleet, and an authenticated
claudeCLI.Interrupt a
createwith Ctrl+C and continue it withdstack preset create -f preset.dstack.yml --resume <ID>.🤖 Generated with Claude Code