diff --git a/.claude/skills/update-unstract-cli/SKILL.md b/.claude/skills/update-unstract-cli/SKILL.md new file mode 100644 index 0000000..d65ad53 --- /dev/null +++ b/.claude/skills/update-unstract-cli/SKILL.md @@ -0,0 +1,261 @@ +--- +name: update-unstract-cli +description: Update the Unstract CLI's endpoint definitions by cross-referencing the public API documentation. Use when Unstract, LLMWhisperer, or API Hub APIs change, when the CLI is missing a documented endpoint or parameter, or to audit the CLI for drift against the docs. Triggers include "update the CLI", "sync CLI with docs", "check the CLI for API drift", and "add the new endpoint to the CLI". +--- + +# Update the Unstract CLI from the public API docs + +Keep `src/unstract_cli/endpoints/*.py` synchronized with the published API +documentation: detect drift, report it with citations, then apply it. + +This is tractable only because of one architectural property: **every command is +generated from a declarative `Endpoint` record.** The command tree, flags, help +text, validation and `--discover` all derive from those records, so a +correct edit to one record propagates everywhere. You never touch command wiring. + +## Products and API groups + +Unstract is the company and the CLI's name. It builds three products: + +| Product | CLI group | API groups it owns | +| --- | --- | --- | +| **Document Studio** | `docstudio` | `platform`, `deployment`, `hitl` | +| **LLMWhisperer** | `whisper` | `llmwhisperer` | +| **API Hub** | `apihub` | `apihub` | + +There are exactly three products -- never introduce a fourth. Document Studio's +three API groups have distinct hosts and credentials, so records set +`api=ApiGroup.PLATFORM` (etc.) and the product is derived from that, never +stored twice. Document Studio's wire paths use `platform`/`deployment`; the CLI +preserves them verbatim under the `docstudio` group. + +## Sources of truth + +| Product | Documentation source | CLI file | +| --- | --- | --- | +| LLMWhisperer | `llmwhisperer-docs/docs/llm_whisperer/apis/*.md` | `whisper.py` | +| API deployment runtime | `unstract-docs/docs/unstract_platform/api_deployment/*.md` | `deployment.py` | +| Platform v1 | `unstract-docs/docs/unstract_platform/api_documentation/versions/v1-*.mdx` | `platform.py` | +| HITL | `unstract-docs/docs/unstract_platform/human_quality_review/*.md` | `hitl.py` | +| API Hub | **No public docs** — `unstract-verticals/src/api_v1/api.py`, `verticals-portal/portal/postman-collection/*.json` | `apihub.py` | + +Repos are expected as siblings of `unstract-cli`. If one is missing locally, read +it through the GitHub API (`gh api repos/Zipstack//contents/`) rather +than guessing. + +Several records carry unusually detailed help text, a `doc_conflict` note, or a +seemingly redundant parameter *because* of a known server defect — the comment +above each explains which. Read it before "simplifying" one: the verbosity is +load-bearing, and the workaround should be removed only once the backend fix +lands, not because it looks like clutter. + +Each record's `doc_source` field names the file it was authored from — that is +your anchor for diffing. Most point at a `.mdx`/`.md` docs page. A number +legitimately point at **backend source** instead, because the endpoint has no +public docs page, and cite `backend/.../urls.py` exactly as API Hub records cite +`unstract-verticals` source: + +- Prompt Studio Output Manager + task-status — `output list`, `output latest`, + `task-status` (`backend/prompt_studio/...`). +- Workflow assembly — `tool registry {list,settings-schema}`, + `workflow tool {list,get,add,set-metadata,remove}`, + `workflow endpoint {list,set}` (`backend/tool_instance_v2/urls.py`, + `backend/workflow_manager/endpoint_v2/urls.py`). + +These surface in the diff as "in CLI, missing from docs" (info-only); that is +correct and expected — report, never delete. Do **not** reach for `doc_conflict` +to silence them: `doc_conflict` means "a docs page exists but the record +deliberately diverges from it," which is a different thing from "no docs page +exists." + +`doc_conflict` suppresses **both** axes of the diff for its record — the +endpoint's absence from the docs *and* any documented parameter the record does +not expose. That is deliberate: a record which deliberately drops a documented +flag would otherwise be reported as param drift on every run, inviting exactly +the "restoration" the annotation exists to prevent. The diff also recognises a +parameter that is *mirrored* rather than exposed: a body field supplied by +`mirror_as` from a path param is present on the wire, so it is not drift. + +## Procedure + +### 1. Establish scope + +If the user named a product or endpoint, work only on that. Otherwise audit +everything. State the scope before starting. + +### 2. Parse the documentation + +Extract `(method, path, params[])` from each source. Two formats: + +- **Markdown** (LLMWhisperer, API deployment, HITL): parameters live in tables + with `Parameter | Type | Default | Required | Description` columns. The endpoint + and method are in the summary table at the top of the page. +- **MDX** (Platform v1): parameters live in `` component props — + `method`, `path`, `description`, and arrays `pathParams`, `queryParams`, + `requestBody`, `responseBody`. This is JSX, not Markdown: parse the props, not + table cells. `` groups them. + +### 3. Parse the current records + +Read the relevant `endpoints/*.py`. `unstract --discover` gives the same +information as JSON and is often easier to diff against — but remember it reflects +the *installed* package, so re-install after editing if you use it. + +Filter it rather than reading the whole thing: unfiltered `--detail full` is +~50k tokens. + +```bash +unstract --discover --group whisper --detail full +unstract --discover --command 'whisper extract' --detail full +``` + +### 4. Diff on three axes + +1. **In docs, missing from CLI** → new capability to add. +2. **In CLI, missing from docs** → *report only, never delete.* Documentation lags + implementation constantly; an endpoint absent from the docs is far more often + an undocumented feature than a removed one. +3. **Parameter drift** → added, removed, renamed, or changed type / default / + enum / required-ness. + +### 5. Report before editing + +Present a table: each difference, its doc citation (file + heading), and the +proposed record change. Wait for confirmation on anything ambiguous. Never apply +a breaking change without flagging it as such. + +### 6. Apply + +Edit `endpoints/*.py` only. Follow the conventions already in those files: + +- `Param.name` is the **API's spelling**, used verbatim on the wire — including + typos like `page_seperator`. Rename the *flag* with `flag="--page-separator"`, + never the name. +- Choose the right `location` (`QUERY`/`BODY`/`PATH`/`HEADER`/`FORM`) and the + endpoint's `body` kind. +- Every `Param` needs `help`; every `Endpoint` needs `summary` and `doc_source`. +- Reuse the pattern encodings rather than inventing new ones: `MutuallyExclusive`, + `AtLeastOneOf`, `RequiredUnless`, `choices={friendly: wire}`, `multiple`, + `default_from`, `freeform_prefix`, `applies_when`, `replace_semantics`, + `derive_patch`, `with_params`, `mirror_to_body` / `mirror_as`, + `require_response_fields` (treat a 2xx that leaves a field null as a failure). +- **Mirroring a PATH id into the body.** `mirror_to_body=True` copies a path + parameter into the JSON body under the same name (`prompt create`'s `tool_id`, + which the backend otherwise persists as NULL). `mirror_as=""` does + the same but renames it, for a server that wants one identifier twice under two + spellings — `api-deployment key create` takes `api_id` in the URL and the same + value as `api` in the body, so mirroring means the caller passes only + `--api-id`. Prefer this over asking the user to repeat a value. +- **`RequiredUnless`** expresses "required, except in one configuration" — a + field that is mandatory in general but genuinely unread when some other flag + holds a sentinel value. It is exported and tested but **no shipped record uses + it today**: the case it was built for (`profile create --chunk-size 0`) turned + out to be server-enforced, so relaxing it client-side would only have moved a + fast exit-2 to a slow remote 400. Reach for it only when you have confirmed the + *server* accepts the omission. +- **`default_from` accepts a fallback chain**, whitespace-separated, first + resolved value winning: `default_from="deployment.org_id platform.org_id"`. + Use it where two config blocks genuinely hold the same value and one is + typically empty — never to borrow a credential across API groups, which would + be credential confusion rather than convenience. +- **JSON parameters parse themselves.** A `Param(type=ParamType.JSON, …)` value is + parsed to a real object before it is sent, accepting inline JSON *or* + `@path/to/file.json`. Never add manual `json.loads`; just set the type. +- **`--wait` retrieval is declarative.** A `PollSpec` drives execute → poll → + retrieve. When the result store is not keyed by the poll handle (e.g. Prompt + Studio's Output Manager is keyed by `tool_id`, not `task_id`), use + `retrieve_carry` to forward original-request identifiers (a py_name, or a + `(source, dest)` pair to rename one), `retrieve_extra` for a constant flag the + original call did not carry, and `retrieve_omits_handle` when the retrieve + endpoint has no handle parameter. +- **Match `status_field` to the STATUS endpoint's body, not the run response.** + An execute response and its status endpoint often spell the state differently + (deployment: the run POST nests `execution_status` under `message`; the status + GET returns a top-level `status`). The poll reads the *status* endpoint, so + `status_field` must match that — pass a tuple like `("status", "execution_status")` + to accept both. Getting this wrong silently consumes a one-shot result and 406s + on the next poll; verify the real status-GET payload shape, not just the run + response, and write a fixture that mirrors it. +- **One-shot poll = the poll *is* the retrieve.** When the status endpoint is + itself the single-read result store (deployment run), set `one_shot=True` and + **no** `retrieve_endpoint`: the terminal poll's body is the result, and a second + read would 406. Such commands need a client-side `--save` (a `client_side=True` + Param) so the result persists on that one read. +- **Never duplicate a PATCH record**: derive it from its PUT with `derive_patch`, + adding PATCH-only fields via `with_params`. +- Paths are **literal**. Do not normalise trailing slashes or "fix" + `profilemanager` vs `profile-manager` — those inconsistencies are real, and + correcting them produces 404s. A path with no trailing slash must set + `no_trailing_slash=True` so the omission reads as intent; a test enforces this. + +### 7. Verify + +```bash +uv run pytest # includes definition-integrity and flag-coverage tests +uv run ruff check . +uv run mypy src +unstract --discover | python -c "import json,sys; json.load(sys.stdin)" +``` + +Add a respx test for any new endpoint, using the sample payload from the docs. + +### 8. Summarize + +Report what changed, what needs human judgement, and what you deliberately +skipped. + +## Safety rules + +These exist because each one has a specific failure mode behind it. + +1. **Never delete a command just because the docs omit it.** Report it. Docs lag + implementation, and deleting a working command breaks users' scripts. +2. **Never invent a parameter.** Every addition cites a specific file and heading. + If the docs are ambiguous, say so and ask. +3. **Treat renames as breaking.** Add the new flag, keep the old one as a + deprecated alias, and say so in the report. Never rename silently. +4. **Preserve richer hand-written help.** Where a record's help text says more + than the docs, add to it — do not overwrite it with a thinner description. +5. **Honour `doc_conflict`.** A record carrying that field records a *deliberate* + divergence from the docs, verified against another source. Never revert one + without explicit human confirmation. Two live examples: + + - **A wrong docs page.** `whisper detail` uses `/whisper-detail` (singular). + The docs *index* says `/whisper-details` (plural), but the endpoint page and + the official `llm-whisperer-python-client` both use the singular. The index + is wrong. Do not "fix" it. + - **Deliberately dropped parameters.** `api-deployment key create` and + `pipeline key create` each omit two documented body params. The docs list + both `api` and `pipeline`; the record mirrors the one the route already + fixes (`api_id` → `api`) and drops the other, because + `/api/keys/api/{api_id}/` cannot create a pipeline key — that is the + sibling command. Restoring either flag re-introduces the original + complaint, which was having to pass the same id twice. + +6. **`draft: true` means exclude.** Docs pages with that front matter (currently + the multi-doc chat APIs `/md/file/upload`, `/md/file/search`, `/md/chat`) + describe unstable contracts. Do not add them. If the flag is removed upstream, + propose adding them as a new `chat` group and let a human decide. +7. **API Hub changes need human review.** There are no public docs for it; the + contract is inferred from source and Postman collections, which recover + parameter names but not intent. Report proposed changes; do not apply them + silently. Note that `--ext-param KEY=VALUE` already lets users reach new + `ext_*` parameters without a CLI release, so the pressure to guess is low. +8. **Docs are not infallible.** Where an index page and a detail page disagree, + prefer the detail page and the official client libraries. Record the decision + as a `doc_conflict` note so the next run does not re-litigate it. + +## Worked example + +*A new `--ocr-engine` parameter appears in the LLMWhisperer extraction docs.* + +1. Read `llmwhisperer-docs/docs/llm_whisperer/apis/whisper.md`; find the row in + the parameters table: `ocr_engine | string | tesseract | No | OCR engine to use`. +2. Confirm `whisper extract` in `endpoints/whisper.py` has no such `Param`. +3. Report: "`whisper.md` documents `ocr_engine` (string, default `tesseract`, + optional); `whisper extract` lacks it. Proposed: add to `_EXTRACT_PARAMS`." +4. Apply — one `Param` in `_EXTRACT_PARAMS`, with `help` from the description and + `choices` if the docs enumerate values. +5. Verify: `--discover` now lists `--ocr-engine` with its default. Tests and + type checks pass. No other file changed — the flag, help text and index all + came from that single record. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000..963c9f3 --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,43 @@ +name: CI + +on: + push: + branches: [main] + pull_request: + +jobs: + test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Install uv + uses: astral-sh/setup-uv@v5 + with: + python-version: "3.12" + + # `uv sync --frozen` installs the exact, hash-pinned tree from uv.lock into + # a project venv -- the install path the README documents for CI. The + # earlier `uv pip install --system` targeted the runner's externally + # managed /usr interpreter and failed; `--frozen` also fails the build if + # uv.lock is stale versus pyproject.toml, which is what we want. + - name: Install dependencies + run: uv sync --frozen --extra dev + + - name: Lint + run: uv run ruff check . + + - name: Type check + run: uv run mypy src + + - name: Test + run: uv run pytest -q + + # The machine-readable index is the CLI's discovery contract, so a broken + # tree must fail the build rather than surface at runtime. Check the full + # detail level too: that is where flag introspection actually runs. + - name: Verify command index + run: | + uv run unstract --discover | python -c "import json,sys; json.load(sys.stdin)" + uv run unstract --discover --detail full | python -c "import json,sys; json.load(sys.stdin)" + uv run unstract --discover --group whisper --detail full | python -c "import json,sys; json.load(sys.stdin)" diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..3301047 --- /dev/null +++ b/.gitignore @@ -0,0 +1,24 @@ +# Byte-compiled / cache +__pycache__/ +*.py[cod] +*$py.class + +# Build / packaging +build/ +dist/ +*.egg-info/ +.eggs/ + +# Virtualenvs +.venv/ +venv/ + +# Tooling caches +.pytest_cache/ +.mypy_cache/ +.ruff_cache/ +.coverage +htmlcov/ + +# Local config; may hold credentials +.unstract.toml diff --git a/E2E.md b/E2E.md new file mode 100644 index 0000000..cf17a1f --- /dev/null +++ b/E2E.md @@ -0,0 +1,29 @@ +--- +E2E test suite for `unstract` bash command. +--- + +Execute the following instructions using `unstract` bash command. +Create a new coding agent session BEFORE proceeding. + +Run this at the very first ALWAYS: + +1. Create a new temporary config file for this exeuction and initialise it. Ask user if a copy of the default config file can be used instead for this purpose. + +Run the following test scenarios in the given order ALWAYS: + +1. Extract total bill amount from my broadband internet invoice +2. Create a Prompt Studio project and extract the following details from my broadband internet invoice: Invoice No., Invoice Date, Plan, Base charges, CGST, SGST, Amount Payable, Due Date, Amount after due date. Then export the project to a custom tool and deploy it as an API. Extract the same details again by hitting the deployed API. + +Run this at the very last ALWAYS: + +1. Delete all new resources created from the test scenarios above. ONLY delete the newly created resources and nothing else. Ask for confirmation if in doubt. +2. Delete the temporary config file used for this execution. + +Finally provide a summary report containing the following information: +- Test scenarios run +- Average test cases per scenario +- Coverage by product +- Total test scenarios passed, failed, errored +- Total input and output LLM tokens spent by the coding agent +- Total network bytes sent and received +- Overall elapsed time diff --git a/README.md b/README.md new file mode 100644 index 0000000..abf0d7c --- /dev/null +++ b/README.md @@ -0,0 +1,291 @@ +# Unstract CLI + +The `unstract` CLI tool covers all products built by **Unstract** team: + +| Product | Group | Covers | +| --- | --- | --- | +| **Document Studio** | `docstudio` | Document extraction platform — Platform Management API, deployed API workflows, Human Quality Review | +| **LLMWhisperer** | `whisper` | Convert documents to LLM-ready text | +| **API Hub** | `apihub` | Vertical extraction — bank statements, tables, document splitting | + +Built for **LLM agents first**: machine-readable discovery, stable exit codes, +structured errors, and no interactive prompts anywhere. + +```bash +unstract whisper extract --file invoice.pdf --mode form --wait --save result.json +``` + +## Install + +```bash +uv pip install -e ".[dev]" # development +pip install unstract-cli # once published +``` + +Requires Python 3.12+. + +### Reproducible, hash-pinned installs (recommended) + +Because supply-chain attacks target the *release you download* — a compromised +version pushed to an index — the strongest defence is to install only exact, +content-verified artifacts. A lockfile with hashes pins every package in the +dependency tree (transitive ones included) so a swapped-out release fails the +install instead of executing. + +The repo commits [`uv.lock`](./uv.lock) for exactly this. Regenerate it whenever +`pyproject.toml` changes, and review the diff: + +```bash +uv lock # resolve + write uv.lock with hashes for the whole tree +uv lock --check # CI: fail if uv.lock is stale vs pyproject.toml +``` + +Install from the lockfile — never re-resolving, so what runs is what was +reviewed: + +```bash +uv sync --frozen --extra dev # dev: exact env from uv.lock, with test/lint tools +uv sync --frozen # runtime-only (dev tools are an extra, so omitted here) +``` + +The dev tools (pytest, ruff, mypy) are declared as the `dev` **extra**, so they +install only when you ask for `--extra dev`; a plain `uv sync` gives the runtime +set alone. + +If you deploy with `pip` rather than `uv`, export the lock to a hashed +requirements file and install with `--require-hashes` (which refuses any package +whose hash is absent or mismatched): + +```bash +uv export --frozen --no-dev --no-emit-project -o requirements.txt +pip install --require-hashes --no-deps -r requirements.txt +``` + +`--no-deps` is deliberate: the exported file already lists the *complete*, +resolved tree, so pip must not reach out and resolve anything itself. Use a +recent pip (`pip install --upgrade pip` first); older pip verifies hashes +inconsistently when a package builds from an sdist. + +**Practices worth keeping:** + +- **Commit `uv.lock`** and gate CI on `uv lock --check`, so a dependency can only + change through a reviewed pull request, never silently at install time. +- **Install with `--frozen` / `--require-hashes`** everywhere — locally, in CI, and + in production — so no environment re-resolves to a different version. +- **Pin the Python too** (`.python-version`, already `>=3.12`); the interpreter is + part of the supply chain. +- **Verify before bumping.** When a lock update changes a package, read the diff + and the upstream changelog rather than accepting the resolution blind. + +## Quick start + +The CLI works with **no config file** — environment variables are enough: + +```bash +export LLMWHISPERER_API_KEY=your-key +unstract whisper usage +``` + +For repeated use, create profiles: + +```bash +unstract config init # writes ~/.config/unstract/config.toml (mode 0600) +unstract config use cloud-eu # switch regions +unstract config current # show what is resolved right now + +# Read and write settings. The target names an API group through its product, +# so a setting always says which product it configures. Either separator works. +unstract config set docstudio.platform org_id org_ABC123 +unstract config set docstudio platform org_id org_ABC123 +unstract config get llmwhisperer base_url +``` + +Valid targets: `docstudio.platform`, `docstudio.deployment`, `docstudio.hitl`, +`llmwhisperer`, `apihub`. + +Config blocks have **exactly one accepted layout** — the API group nested under +its product. There are no aliases and no flat fallback, so a block written any +other way is ignored rather than half-applied: + +```toml +[profiles.cloud-us.docstudio.platform] # ✓ read +[profiles.cloud-us.platform] # ✗ ignored +[profiles.cloud-us.whisper] # ✗ ignored (use llmwhisperer) +``` + +Settings resolve in one order, everywhere: **flag → environment variable → +profile → built-in default**. Credentials in the config file use `env:VAR_NAME` +indirection, so the file records *where* secrets live rather than the secrets +themselves. + +Document Studio's three API groups each have their own block, but they address +one organization, so `org_id` falls back to `docstudio.platform` when the +`deployment` or `hitl` block does not set it — configure it once. Credentials +never fall back: keys are per-group, and silently reusing one across groups +would be credential confusion rather than convenience. + +### Multiple configs + +Nothing is global-only. There are three independent ways to select a config, and +they layer — different environments, tenants or projects can each have their own: + +| Mechanism | Scope | Use for | +| --- | --- | --- | +| **Profiles** in one file (`--profile`) | per invocation | regions and tenants that share a machine | +| **`.unstract.toml`** in a project | per directory tree | settings a repo commits for everyone working in it | +| **`--config PATH`** / `$UNSTRACT_CONFIG` | per invocation | throwaway or CI configs, or fully separate environments | + +Config *file* resolution, highest precedence first: + +1. `--config PATH` +2. `$UNSTRACT_CONFIG` +3. the nearest `.unstract.toml`, found by walking up from the working directory + (like `git` or `ruff`; the search stops at `$HOME`) +4. `$XDG_CONFIG_HOME/unstract/config.toml`, else `~/.config/unstract/config.toml` + +```bash +unstract --config ./staging.toml whisper usage # a specific file +unstract --profile cloud-eu whisper usage # a profile within the active file +cd my-project && unstract whisper usage # picks up ./.unstract.toml +``` + +A project file is ordinary config, so it can hold several profiles too: + +```toml +default_profile = "dev" + +[profiles.dev.docstudio.platform] +base_url = "https://dev.internal/" +org_id = "org_dev" +api_key = "env:UNSTRACT_PLATFORM_KEY" + +[profiles.prod.docstudio.platform] +base_url = "https://us-central.unstract.com" +org_id = "org_prod" +api_key = "env:UNSTRACT_PROD_KEY" +``` + +Because credentials use `env:` indirection, such a file is safe to commit. + +## For agents + +Discover the entire surface without documentation. Start with the group map, then +drill into exactly the subtree you need — selection (`--group` / `--command`) and +verbosity (`--detail`) are independent, so any combination works: + +```bash +unstract --discover # the ~15 groups, with counts (default) +unstract --discover --command 'docstudio platform prompt-studio' # one subtree, names + summaries +unstract --discover --detail summary # every command, names + summaries +unstract --discover --command 'whisper extract' --detail full +unstract --discover --detail full # everything +``` + +The default is a **map of the navigable groups** — each with a one-line summary, a +command count, and the exact `drill` command that lists it. An agent reads that +first (~1k tokens), then follows one `drill` into the subtree it actually needs +rather than pulling all 156 commands into context. At `--detail full` each command +carries the underlying HTTP method and path, every flag with type, default, enum +and required-ness, plus whether it supports `--wait` and whether its result is +one-shot. + +**Token cost** matters when an agent reads this into context, so the default is +the cheapest useful view: + +| Invocation | Tokens | +| --- | --- | +| `--discover` (default, group map) | ~1,100 | +| `--command 'whisper extract' --detail full` | ~1,900 | +| `--detail summary` (all commands, one-liners) | ~5,800 | +| `--group whisper --detail full` | ~4,200 | +| `--detail full` (everything) | ~50,000 | + +Output is compact JSON when piped and pretty-printed on a terminal. + +### Output + +`--output json|yaml|table|raw`. **JSON is the default whenever stdout is not a +TTY**, so piping needs no flags. Payloads go to stdout and nothing else; +diagnostics and errors go to stderr, so `unstract ... | jq` always parses. + +### Exit codes + +| Code | Meaning | Code | Meaning | +| --- | --- | --- | --- | +| 0 | success | 5 | validation error | +| 1 | generic error | 6 | rate limited | +| 2 | usage error | 7 | timed out waiting | +| 3 | auth failure | 8 | server error | +| 4 | not found | 9 | result already consumed | + +Failures also emit a JSON object on stderr with `code`, `message`, `hint` and +`retryable`, so an agent can self-correct rather than retry blindly. + +### One-shot results + +Some results can be retrieved **exactly once** — LLMWhisperer retrieval, the +deployment status API, and HITL dequeues. A second read cannot recover them. +Those commands are marked `one_shot` in `--discover`, and all accept +`--save PATH`, which writes the payload to disk atomically before exiting. A +consumed result exits `9`. + +### Waiting + +Execute → poll → retrieve flows accept `--wait` so an agent needn't script the +loop: + +```bash +unstract docstudio deployment run --api-name invoice-api --file invoice.pdf --wait +``` + +On timeout the CLI exits `7` and prints the job handle, so work can resume +without reprocessing the document. + +## Command groups + +| Command | Product | Covers | +| --- | --- | --- | +| `docstudio platform` | Document Studio | Prompt Studio, workflows, deployments, pipelines, adapters, connectors, groups, sharing | +| `docstudio deployment` | Document Studio | Run deployed API workflows; status; highlight data | +| `docstudio hitl` | Document Studio | Human Quality Review: approved results, bulk download | +| `whisper` | LLMWhisperer | Extraction, status, retrieve, highlights, usage, webhooks | +| `apihub` | API Hub | Bank statements, tables, document splitting | +| `config` | *local only* | Profile management (no network calls) | + +Document Studio exposes three API groups, each with its own host and +credentials, so they remain separate under one product. + +Every group and command has full `--help` with worked examples. + +## Development + +```bash +uv run pytest # tests +uv run ruff check . # lint +uv run mypy src # types +``` + +### Architecture + +Commands are **generated from declarative `Endpoint` records** in +`src/unstract_cli/endpoints/`. The command tree, flags, help text, validation and +`--discover` all derive from those records, so adding an endpoint means +adding one record and help text cannot drift from behaviour. + +That single source of truth is also what makes the bundled Claude Skill +(`.claude/skills/update-unstract-cli/`) possible: it cross-references the public +API documentation and updates the records when the APIs change. + +### Dependencies + +Kept deliberately small, to limit supply-chain exposure: **click** (dynamic +command tree + `--discover`), **httpx** (TLS, redirects, multipart uploads), +**pyyaml** (`--output yaml`), **tomli-w** (config writer; reading uses the stdlib +`tomllib`). Config parsing needs no third-party TOML reader. The set stops here +rather than at zero on purpose — replacing httpx or the TOML writer means shipping +unaudited network/serialisation code, a worse trade than depending on +widely-reviewed packages. The remaining risk — a *compromised release* of one of +those packages — is addressed by hash-pinned installs, not by removing audited +dependencies: see [Reproducible, hash-pinned installs](#reproducible-hash-pinned-installs-recommended). + +See [`SPEC.md`](./SPEC.md) for the full specification. diff --git a/SPEC.md b/SPEC.md new file mode 100644 index 0000000..4ba66df --- /dev/null +++ b/SPEC.md @@ -0,0 +1,734 @@ +# Unstract CLI — Specification + +**Status:** Draft v1 +**Repo:** `Zipstack/unstract-cli` +**Binary:** `unstract` + +--- + +## 1. Purpose + +**Unstract** is the company, and `unstract` is the name of this CLI. Unstract builds exactly **three products**, and this one tool covers all of them: + +| Product | Group | API groups | Surface covered | +| --- | --- | --- | --- | +| **Document Studio** | `docstudio` | `platform`, `deployment`, `hitl` | Prompt Studio, Workflows, API Deployments, ETL/Task Pipelines, Adapters, Connectors, User Groups, Org Users; deployed-API execution; Human Quality Review | +| **LLMWhisperer** | `whisper` | `llmwhisperer` | Text extraction, status/retrieve, detail, highlights, usage, webhook management | +| **API Hub** (internal name: *Verticals*) | `apihub` | `apihub` | Vertical extraction (bank statement, table discovery/extraction), doc-splitter, status/retrieve | + +> **Document Studio's APIs use `platform`/`deployment` paths on the wire and `UNSTRACT_*` environment variables.** The CLI surfaces them under the `docstudio` group but preserves those wire paths and variable names verbatim. + +Document Studio owns three API groups because each has its own base URL and credentials. They stay distinct for auth and config purposes while belonging to one product. + +The primary consumer is an **LLM agent operating in an autonomous workflow**. A human at a terminal is the secondary consumer. Every design decision below resolves in favour of machine legibility, deterministic behaviour, and self-description via `--help`. + +### 1.1 Goals + +1. One binary, one auth/config model, one output contract across three products (five API groups) that today have three incompatible authentication schemes and four different base URLs. +2. An agent can discover the full capability surface **without external documentation** — from `--help` output and a machine-readable command index alone. +3. Every documented API parameter is reachable as a flag. No capability is CLI-only or API-only. +4. The CLI's endpoint definitions are a **single declarative source of truth**, mechanically diffable against the public docs so the bundled Claude Skill can keep it current. + +### 1.2 Non-goals + +- Not a replacement for `unstract-python-client` / `llm-whisperer-python-client` as an embedding library. The CLI is a process-level interface. +- No interactive TUI, wizards, or prompts. See §5.2. +- Not a workflow orchestrator. It exposes primitives; agents compose them. + +--- + +## 2. Key decisions + +### D1 — Direct REST with declarative endpoint definitions (not client-library wrapping) + +The CLI implements HTTP directly against documented endpoints. Each endpoint is a declarative record (`endpoint`, `method`, `params`, `auth`, `output`) from which the command tree, flags, help text, and validation are **generated**. + +*Rationale.* The official Python clients cover LLMWhisperer and API-deployment execution only. The Platform v1 management surface — the bulk of the command tree — has no client and would be raw REST regardless; wrapping would produce two inconsistent internal layers. Decisively: the "cross-reference public docs" Skill requirement is only tractable if each command maps 1:1 onto a documented endpoint+parameter set in a form that can be **diffed**. A table of endpoint definitions is diffable against documentation; a wrapper around a third-party client's Python API is not. Polling/retry *logic* may be borrowed from the clients, but the HTTP layer is owned here. + +### D2 — Profile-based configuration + +Three incompatible auth schemes, plus region-specific and on-prem hosts, plus `org_id` as a **URL path segment** rather than a flag. Configuration uses named profiles (kubectl/aws style) holding per-product host, key, and org. See §4. + +### D3 — Python + Click (a deliberately minimal dependency set) + +Click gives an introspectable command tree, which `--discover` (§5.3) reads directly, and its command objects are built at runtime from the declarative `Endpoint` records rather than from static function signatures. Click is used **directly**, not through Typer: Typer's decorator API assumes statically-declared signatures, whereas the parameters here are known only at runtime from data, so Typer added a dependency without being used and was removed. + +The runtime dependency set is kept as small as the feature set allows, to limit supply-chain exposure: `click` (dynamic command tree), `httpx` (TLS, redirects, connection reuse, and multipart/form-data encoding — the one piece with no stdlib equivalent), `pyyaml` (`--output yaml`) and `tomli-w` (config writer; reading uses the stdlib `tomllib`). "Absolute zero" is not a goal: the remaining packages are widely-deployed and audited, and replacing them (a hand-rolled multipart encoder, a hand-rolled TOML writer) would substitute bespoke, unaudited code for that — a worse trade for a security-motivated posture. Reproducible, hash-pinned installs (a lockfile with hashes) address the "compromised release on PyPI" threat more directly than shaving audited packages from the set. + +### D4 — API Hub is code-derived, and this is a documented gap + +API Hub has **no public documentation site**. Its contract is derived from `Zipstack/unstract-verticals` (`src/api_v1/api.py`) and the Postman collections in `Zipstack/verticals-portal` (`portal/postman-collection/`). The Skill therefore **cannot** validate API Hub commands against public docs; it must fall back to source-of-truth diffing against the repo. This gap is explicit in §8.4 rather than left implicit. + +--- + +## 3. Command tree + +Grouped by product, then API group, then resource, then action. Verb-last, consistent across groups. + +``` +unstract +├── whisper # LLMWhisperer v2 +│ ├── extract # POST /whisper (+ --wait convenience) +│ ├── status # GET /whisper-status +│ ├── retrieve # GET /whisper-retrieve +│ ├── detail # GET /whisper-detail +│ ├── highlights # GET /highlights +│ ├── usage # GET /get-usage-info +│ ├── usage-by-tag # GET /usage +│ └── webhook {create,get,update,delete} # /whisper-manage-callback +│ +├── docstudio # Document Studio +│ └── deployment # Deployed API workflow execution +│ ├── run # POST /deployment/api/{org}/{api_name}/ +│ ├── status # GET /deployment/api/{org}/{api_name}/?execution_id= +│ └── highlight # GET /deployment/api/{org}/{api_name}/highlight/ +│ +│ └── platform # Platform Management API v1 +│ ├── prompt-studio {list,get,create,update,patch,delete, +│ │ export-project,import-project,sync-prompts, +│ │ export-tool,export-info, +│ │ file {upload,get,delete}, +│ │ prompt {create,get,update,patch,delete,reorder}, +│ │ profile {list,set-default,create,get,update,patch,delete}, +│ │ index-document,fetch-response,single-pass,task-status, +│ │ output {list,latest}, +│ │ users,check-deployment-usage, +│ │ select-choices,adapter-choices,retrieval-strategies} +│ ├── workflow {list,get,create,update,patch,delete,execute,toggle-active, +│ │ can-update,clear-file-marker,schema,users, +│ │ tool {list,get,add,set-metadata,remove}, # attach a registry tool +│ │ endpoint {list,set}, # set SOURCE/DEST to API +│ │ execution {list,get,logs}, +│ │ file-history {list,get,delete,clear}} +│ ├── tool registry {list,settings-schema} # find a tool's function_name +│ ├── api-deployment {list,get,create,update,patch,delete,users, +│ │ by-prompt-studio-tool,postman-collection, +│ │ key {list,create,get,update,delete}} +│ ├── pipeline {list,get,create,update,patch,delete,execute,executions, +│ │ users,postman-collection, +│ │ key {list,create}} +│ ├── adapter {list,get,create,update,patch,delete,info,users, +│ │ supported,schema,test, +│ │ default-triad {get,set}} +│ ├── connector {list,get,create,update,patch,delete, +│ │ supported,schema,test,oauth-cache-key} +│ ├── group {list,create,patch,delete, +│ │ member {list,add,remove}, resources} +│ ├── user list +│ └── share # POST /{resource}/{id}/share/ +│ +│ └── hitl # Human Quality Review (Enterprise) +│ ├── approved get # GET /mr/api/{org}/approved/result/{class_id}/ +│ ├── bulk-download # same endpoint, --download-files / --page / --email +│ └── download-status # GET /mr/api/{org}/approved/download-status/{job_id}/ +│ +├── apihub # API Hub / Verticals +│ ├── extract # POST /api/v1/extract?vertical=&sub_vertical= +│ ├── status # GET /api/v1/status?file_hash= +│ ├── retrieve # GET /api/v1/retrieve?file_hash= +│ └── doc-splitter {upload,status,download} +│ +├── config {init,list,get,set,use,current,path,doctor} +├── completion {bash,zsh,fish} +└── --discover / --version / --help +``` + +### 3.1 Convenience composition + +Both LLMWhisperer and Unstract are execute → poll → retrieve. Raw sub-commands are always available, **and** a `--wait` flag drives the poll loop to a terminal state so agents need not script it: + +```bash +unstract whisper extract --file invoice.pdf --mode form --wait --output json +unstract docstudio deployment run --api-name invoice-api --file invoice.pdf --wait --save out.json +unstract apihub extract --vertical table --sub-vertical bank_statement --file stmt.pdf --wait +unstract docstudio platform prompt-studio fetch-response --tool-id --document-id --id

--wait +``` + +`--wait` accepts `--poll-interval` (default 3s) and `--timeout` (default 300s). On timeout the process exits `7` (§5.4) with the handle (`whisper_hash` / `execution_id` / `file_hash` / `task_id`) on stdout so the agent can resume. + +**One-shot polling (deployment run).** For `deployment run --wait`, the status endpoint *is* the one-shot result store: the poll that first observes `COMPLETED` also consumes and acknowledges the result. So `--wait` returns *that terminal poll's body* as the result and issues no second read — a re-read would return HTTP 406. This is the one execute→poll→retrieve variant where the poll *is* the retrieve; it defines no separate `retrieve_endpoint`. Because the read is destructive and unrepeatable, `deployment run` takes `--save`, which persists the result on that single read. Reading the terminal state correctly is load-bearing: the status GET returns a top-level `status` field (its `message` holds the result), while the run POST nests `execution_status` under `message`; the poll must match the status endpoint's shape or it fails to recognise `COMPLETED`, consumes the result on that read, and 406s on the next poll. + +Prompt Studio's `fetch-response` and `single-pass` are execute → poll → retrieve too, but the pieces live in different places: the async call returns a `task_id`, `task-status` reports completion (not the value), and the extracted value lands in the Output Manager. `--wait` composes all three — poll `task-status` to `completed`, then read the result from `prompt-studio output list`, keyed by the original request's `tool_id`. `fetch-response --wait` narrows to the exact prompt + document it ran; `single-pass --wait` returns the tool-wide single-pass rows. Without `--wait`, retrieve the result yourself with `prompt-studio output list --tool-id ` (see §6.3). + +--- + +## 4. Configuration & authentication + +### 4.1 Resolution order + +For every setting, strictly: **command-line flag → environment variable → profile in config file → built-in default.** + +### 4.2 Config file + +Named profiles; `--profile/-p` or `UNSTRACT_PROFILE` selects one. + +**Multiple config files are supported**, since different work needs different settings. The file itself is resolved, highest precedence first: + +1. `--config PATH` +2. `$UNSTRACT_CONFIG` +3. the nearest `.unstract.toml`, found by walking up from the working directory (as `git` and `ruff` do; the search stops at `$HOME` so a stray file above it cannot capture every invocation) +4. `$XDG_CONFIG_HOME/unstract/config.toml`, else `~/.config/unstract/config.toml` + +These compose with profiles rather than replacing them: a project file may itself define several profiles. A committed `.unstract.toml` is safe because credentials use `env:` indirection. + +**Block layout has exactly one accepted form**: the API group nested under its product, e.g. `[profiles..docstudio.platform]` or `[profiles..llmwhisperer]`. No aliases, no flat fallback. A block written any other way is ignored rather than silently half-applied — a config that looks applied but is not surfaces later as a missing-credential error with no obvious cause. `config get`/`config set` name the same targets: `docstudio.platform` (or `docstudio platform`), `docstudio.deployment`, `docstudio.hitl`, `llmwhisperer`, `apihub`. + +```toml +default_profile = "cloud-us" + +[profiles.cloud-us] + [profiles.cloud-us.whisper] + base_url = "https://llmwhisperer-api.us-central.unstract.com/api/v2" + api_key = "env:LLMWHISPERER_API_KEY" # indirection: never store secrets inline + + [profiles.cloud-us.platform] + base_url = "https://us-central.unstract.com" + org_id = "org_XXXXXXXX" + api_key = "env:UNSTRACT_PLATFORM_KEY" + + [profiles.cloud-us.deployment] + base_url = "https://us-central.unstract.com" + org_id = "org_XXXXXXXX" + api_key = "env:UNSTRACT_DEPLOYMENT_KEY" + + [profiles.cloud-us.apihub] + base_url = "https://api-hub.unstract.com" + api_key = "env:UNSTRACT_APIHUB_KEY" + +[profiles.cloud-eu] + [profiles.cloud-eu.whisper] + base_url = "https://llmwhisperer-api.eu-west.unstract.com/api/v2" +``` + +Values may use `env:VAR_NAME` indirection so the file itself holds no secrets. Config files with secrets inline are created `0600`; the CLI warns if permissions are broader. + +An `env:` ref resolves from the CLI **process's** environment only. A variable `export`ed in a login shell that the CLI did not inherit resolves to nothing, and auth then fails with a "missing credential" error while a literal value in the file would have worked — a costly, invisible trap. `config doctor` exists to make this visible: it reports, per API group, exactly where each credential resolves from (override / named env var / profile literal / profile `env:` ref) and, for an `env:` ref, whether the variable is present in *this* process. With `--check` (on by default) it makes one authenticated call per configured group and reports whether the resolved credential actually works — never echoing the secret or the payload. + +### 4.3 Environment variables + +| Variable | Purpose | +| --- | --- | +| `UNSTRACT_PROFILE` | Active profile name | +| `LLMWHISPERER_API_KEY` | LLMWhisperer `unstract-key` header | +| `LLMWHISPERER_BASE_URL` | Region / on-prem override | +| `UNSTRACT_PLATFORM_KEY` | Platform v1 Bearer token | +| `UNSTRACT_DEPLOYMENT_KEY` | API-deployment Bearer token | +| `UNSTRACT_ORG_ID` | Org identifier (URL path segment) | +| `UNSTRACT_BASE_URL` | Platform host (cloud region / on-prem) | +| `UNSTRACT_APIHUB_KEY`, `UNSTRACT_APIHUB_BASE_URL` | API Hub | +| `UNSTRACT_ANTHROPIC_API_KEY` | API Hub `X-Anthropic-API-Key` passthrough | +| `UNSTRACT_OUTPUT` | Default output format | +| `NO_COLOR` | Disable ANSI styling | + +Env vars are the expected mechanism in CI and agent sandboxes; the CLI is fully usable with **zero config file**. + +### 4.4 Auth schemes per product + +| Product | Header | Org handling | +| --- | --- | --- | +| LLMWhisperer | `unstract-key: ` | n/a | +| API Deployment | `Authorization: Bearer ` | `org_id` + `api_name` in URL path | +| Platform v1 | `Authorization: Bearer ` | `org_id` in URL path | +| HITL | `Authorization: Bearer ` | `org_id` + `class_id` in URL path | +| API Hub | `apikey: ` at the Kong gateway; optional BYO-key passthrough `X-LLMWhisperer-API-Key`, `X-Anthropic-API-Key` | n/a | + +**API Hub tenancy.** External callers send only `apikey`. The Kong plugin `subscription-metadata-injector` looks that key up in Redis and injects `X-Subscription-Id`, `X-Subscription-Name`, `X-User-Id`, and `X-Product-Id` downstream. The CLI therefore **must not** send those headers — they are gateway-supplied and would be overwritten. It sends `apikey`, plus the optional bring-your-own-key headers when the user supplies their own LLMWhisperer/Anthropic credentials. + +**Platform key permission levels** (`read` / `read_write` / `full_access`) are surfaced in help text for destructive commands: `DELETE` requires `full_access`, enforced server-side at middleware. The CLI states this in `--help` rather than letting agents discover it via a 403. + +--- + +## 5. LLM-friendliness requirements + +These are testable requirements, not aspirations. + +### 5.1 Output contract + +- `--output {json,yaml,table,raw}`; **`json` is the default when stdout is not a TTY**, `table` when it is. `UNSTRACT_OUTPUT` overrides. +- On **success**, only the result goes to stdout — no banners, spinners, or progress. Diagnostics go to stderr. +- On **failure**, stdout carries nothing but the structured error envelope, and only when stdout is not a TTY (see §5.5). A result and an error never share an invocation's stdout: the command either emits a result and exits `0`, or emits an error and exits non-zero — never both. So a consumer that pipes stdout to a JSON parser always sees exactly one valid object, on either path. +- `raw` emits only the payload (extracted text, file bytes) for piping. +- `--quiet` suppresses stderr diagnostics; `-v/-vv` increases them. + +### 5.2 Never interactive + +No command prompts for input under any circumstance. Missing required input is an error with a message naming the exact flag or env var to supply. Secrets are read from env or config, never a TTY prompt. This is a hard constraint: an autonomous agent cannot answer a prompt. + +### 5.3 Discoverability + +- Complete `--help` at **every** level, with a one-line description, full flag list with types/defaults/enums, and at least one worked example per leaf command. +- Enumerated values are listed in help text verbatim (e.g. `--mode {native_text,low_cost,high_quality,form,table}`). +- `unstract --discover` emits a machine-readable JSON index, generated from the same endpoint definitions that drive execution so it cannot drift. It has three verbosity tiers, cheapest first, selected by `--detail`: + - **`groups`** (default, ~1k tokens): a map of the ~15 navigable command groups, each with a one-line summary, its command count, and the exact `drill` command that lists it. This is the entry point — an agent reads it, then follows one `drill` into the subtree it needs instead of pulling every command into context. Each `drill` is the exact `--discover --command ''` prefix and is **verified by test to return the count it advertises** (discovery must never advertise a path the parser won't honour). + - **`summary`** (~5.8k tokens): every command with its name and one-line summary. + - **`full`** (~50k tokens): adds each command's flags (type, default, enum, required-ness), HTTP method and path, `--wait`/one-shot semantics, and permissions. + - Selection (`--group`, `--command` prefix) and verbosity (`--detail`) are independent; a filtered query degrades `groups` to `summary` since the filter already narrows to a subtree. The `groups` overview and the unfiltered `full`/`summary` views carry the global boilerplate (exit-code table, output/error conventions); narrow queries omit it, where it would outweigh the answer. +- `unstract --help` lists sub-commands with descriptions; unknown commands produce a "did you mean" suggestion on stderr. + +### 5.4 Exit codes + +Stable and documented, so an agent can branch without parsing prose: + +| Code | Meaning | +| --- | --- | +| `0` | Success | +| `1` | Generic / unexpected error | +| `2` | Usage error (bad flags, missing required argument) | +| `3` | Authentication or authorization failure (401 / 403) | +| `4` | Not found (404) | +| `5` | Validation error rejected by the API (400 / 422) | +| `6` | Rate limited / quota exceeded (429) | +| `7` | Timed out waiting for a terminal state (`--wait`) | +| `8` | Remote server error (5xx), or a 2xx the server returned in a broken state — e.g. a create that accepts the request but leaves a linking field null (§6.3, `prompt create`) | +| `9` | Result already consumed (see §5.6) | + +### 5.5 Structured errors + +Every failure emits a JSON object on **stderr** (even in `table` mode). When stdout is **not a TTY** — the agent / wrapper case — the same envelope is **also written to stdout**, so a pipeline feeding stdout to a JSON parser sees a valid object rather than an empty stream (a `JSONDecodeError` otherwise). On a TTY, stdout stays clean and the error appears on stderr only. + +This mirroring covers **every** error path, not merely usage/validation errors: our own structured errors, Click's own parse errors (`No such option`, missing argument — rendered through the same envelope at the console entry point), and the `--discover` no-match case. It is safe because success and failure never share stdout (§5.1). + +Example envelope: + +```json +{ + "error": { + "code": "validation_error", + "http_status": 422, + "message": "Pipeline 'testetl' is inactive, please activate the pipeline", + "details": [{"code": "error", "detail": "...", "attr": null}], + "endpoint": "POST /deployment/api/{org_id}/{api_name}/", + "hint": "Activate the pipeline in the Unstract UI, or use `unstract docstudio platform pipeline patch --active true`.", + "retryable": false + } +} +``` + +`hint` and `retryable` exist specifically so an agent can self-correct rather than retry blindly. + +### 5.6 One-time-retrieval semantics (agent footgun) + +Both LLMWhisperer `/whisper-retrieve` and the Unstract deployment status API return results **exactly once**; a second call yields "already delivered" / HTTP 406. An agent that re-runs a step will silently lose data. + +Mitigations, all mandatory: +- `--save ` on every retrieval command, writing the payload to disk atomically **before** exit. +- `--wait` implies persistence: the retrieved result is always written to `--save` if given, and the destructive read is performed exactly once. +- Help text for these commands states the one-shot behaviour explicitly. +- A consumed result produces exit code `9` with a `hint` naming the likely cause. + +### 5.7 Reliability + +- Automatic retry with exponential backoff + jitter on `429` and `5xx`; never on `4xx`. `--max-retries` (default 3), `--no-retry`. +- `--dry-run` prints the request (method, resolved URL, headers with secrets redacted, body summary) as JSON and exits `0` without sending. Lets an agent verify a call before a destructive operation. +- `--timeout` per request. +- Secrets are **always** redacted from logs, errors, and `--dry-run` output. + +--- + +## 6. Endpoint reference + +The tables below are the contract the Skill (§8) maintains. `Params → flags` uses kebab-case flag names derived from API parameter names (`page_seperator` → `--page-separator`, with the API's spelling preserved on the wire). + +**JSON-valued flags** (`--data`, `--custom-data`, `--adapter-metadata`, `--connector-metadata`, `--source-settings`, `--destination-settings`, …) accept either inline JSON or an `@path/to/file.json` reference — large payloads such as an exported prompts file are painful to pass inline and hit shell argument limits. The value is parsed to a real object before it is sent, so a JSON body reaches the API as a nested object, not a quoted string. Invalid JSON fails locally with exit `2`. + +### 6.1 LLMWhisperer v2 + +Base: `https://llmwhisperer-api.{us-central,eu-west}.unstract.com/api/v2` · Auth: `unstract-key` + +| Command | Endpoint | Method | Parameters | +| --- | --- | --- | --- | +| `whisper extract` | `/whisper` | POST | `--file` \| `--url` (sets `url_in_post=true`); `--mode {native_text,low_cost,high_quality,form,table}` (default `form`); `--output-mode {layout_preserving,text}` (default `layout_preserving`); `--page-separator` (default `<<<`); `--pages-to-extract` (e.g. `1-5,7,21-`); `--median-filter-size` (int, `low_cost` only); `--gaussian-blur-radius` (int, `low_cost` only); `--line-splitter-tolerance` (float, default `0.4`); `--line-splitter-strategy` (default `left-priority`); `--horizontal-stretch-factor` (float, default `1.0`); `--mark-vertical-lines` / `--mark-horizontal-lines` (bool); `--lang` (default `eng`); `--tag` (default `default`); `--file-name`; `--use-webhook`; `--webhook-metadata`; `--add-line-nos` (bool); `--allow-rotated-text` (bool, default `true`); `--word-confidence-threshold` (float, default `0.3`) | +| `whisper status` | `/whisper-status` | GET | `--whisper-hash` (required) | +| `whisper retrieve` | `/whisper-retrieve` | GET | `--whisper-hash` (required); `--text-only` (bool, default `false`); `--save` | +| `whisper detail` | `/whisper-detail` | GET | `--whisper-hash` (required) | +| `whisper highlights` | `/highlights` | GET | `--whisper-hash` (required); `--lines` (required, e.g. `1-5,7,21-`) | +| `whisper usage` | `/get-usage-info` | GET | — | +| `whisper usage-by-tag` | `/usage` | GET | `--tag` (required); `--from-date` `YYYY-MM-DD`; `--to-date` `YYYY-MM-DD` (defaults to last 30 days) | +| `whisper webhook create` | `/whisper-manage-callback` | POST | `--webhook-name`, `--url`, `--auth-token` (all body fields) | +| `whisper webhook get` | `/whisper-manage-callback` | GET | `--webhook-name` (required) | +| `whisper webhook update` | `/whisper-manage-callback` | PUT | `--webhook-name`, `--url`, `--auth-token` | +| `whisper webhook delete` | `/whisper-manage-callback` | DELETE | `--webhook-name` (required) | + +Statuses: `accepted`, `processing`, `processed`, `error`, `retrieved`. `--wait` polls `/whisper-status` until `processed`, then retrieves. + +### 6.2 Unstract API Deployments (runtime) + +Base: `https://us-central.unstract.com` · Auth: `Authorization: Bearer` · Path: `/deployment/api/{org_id}/{api_name}/` + +| Command | Endpoint | Method | Parameters | +| --- | --- | --- | --- | +| `deployment run` | `/deployment/api/{org_id}/{api_name}/` | POST | `--api-name`\* (path); `--org-id` (path, defaults from profile); `--file` (repeatable, ≤32 combined); `--presigned-url` (repeatable, AWS S3 HTTPS only); `--timeout` (0–300, default 0 = async); `--include-metadata` (bool); `--tags` (currently 1 tag; must start with a letter); `--llm-profile-id` (UUID); `--custom-data` (JSON object, addressable as `{{custom_data.key}}`); `--hitl-queue-name`; `--save` (with `--wait`, persists the one-shot result on its single read) | +| `deployment status` | `/deployment/api/{org_id}/{api_name}/` | GET | `--api-name`\* (path); `--org-id` (path, defaults from profile); `--execution-id`\*; `--include-metadata` (bool); `--save` | +| `deployment highlight` | `/deployment/api/{org_id}/{api_name}/highlight/` | GET | `--api-name`\* (path); `--org-id` (path, defaults from profile); `--whisper-hash`\*; `--line-numbers`\* (comma-separated); `--text-extractor-name`\* (e.g. `llm-whisperer-v2`) | + +> **Path parameters are flags too.** `org_id` and `api_name` are URL path segments, not query/body parameters, but the §1.1 contract ("every documented parameter is reachable as a flag") covers them. `--org-id` resolves from the active profile when omitted; `--api-name` has no sensible default — one profile serves many deployments — so it is always required. The same rule applies to `--org-id` in §6.3 (Platform v1) and §6.4 (HITL, alongside the required `--class-id`). + +Execution statuses: `PENDING`, `EXECUTING`, `COMPLETED`, `STOPPED`, `ERROR`. + +> **Implementation note.** `EXECUTING` and `PENDING` currently return **HTTP 422**, not 200 — a documented server-side defect scheduled for correction. The CLI **must branch on the response-body `status` field, never on the HTTP status code**, so behaviour is unchanged when the fix ships. Synchronous mode (`timeout > 0`) is deprecated upstream; `--wait` uses `timeout=0` + polling. +> +> **Status-field shape (load-bearing).** The status GET returns a **top-level `status`** field (`{status, message}`, where `message` carries the result), whereas the run POST nests `execution_status` under `message`. The poll reads the status endpoint, so it matches `status` first. A field mismatch here means the terminal state is not recognised, the one-shot result is consumed unread, and the next poll returns HTTP 406 — this is exactly how `deployment run --wait` once lost results (§3.1 one-shot polling). `run --wait` returns the terminal poll's body and takes `--save` to persist it. + +### 6.3 Unstract Platform Management API v1 + +Base: `{host}/api/v1/unstract/{org_id}` · Auth: `Authorization: Bearer ` +Permissions: `read` → GET/HEAD/OPTIONS; `read_write` → all but DELETE; `full_access` → all. +Pagination (list endpoints): `--page` (default 1), `--page-size` (default 50, max 1000). + +#### Prompt Studio — `unstract docstudio platform prompt-studio` + +| Command | Endpoint | Method | Key parameters | +| --- | --- | --- | --- | +| `list` | `/prompt-studio/` | GET | — | +| `create` | `/prompt-studio/` | POST | `--tool-name`\*, `--description`\*, `--author`\*, `--icon`, `--preamble`, `--postamble`, `--summarize-context`, `--single-pass-extraction-mode`, `--enable-challenge`, `--enable-highlight`, `--custom-data`, `--shared-users`, `--shared-to-org` | +| `get` | `/prompt-studio/{tool_id}/` | GET | `--tool-id`\* | +| `update` / `patch` | `/prompt-studio/{tool_id}/` | PUT / PATCH | as `create`; PATCH all optional | +| `delete` | `/prompt-studio/{tool_id}/` | DELETE | `--tool-id`\* (409 if exported and in use) | +| `export-project` | `/prompt-studio/project-transfer/{tool_id}` | GET | `--tool-id`\*, `--save` | +| `import-project` | `/prompt-studio/project-transfer/` | POST | `--file`\* (multipart) | +| `sync-prompts` | `/prompt-studio/{tool_id}/sync-prompts/` | POST | `--tool-id`\*, `--data`\* (export JSON), `--create-copy` | +| `export-tool` | `/prompt-studio/export/{tool_id}` | POST | `--tool-id`\*, `--is-shared-with-org`, `--user-id` (repeatable), `--force-export` | +| `export-info` | `/prompt-studio/export/{tool_id}` | GET | `--tool-id`\* (204 if never exported) | +| `file upload` | `/prompt-studio/file/{tool_id}` | POST | `--tool-id`\*, `--file`\* (repeatable) | +| `file get` | `/prompt-studio/file/{tool_id}` | GET | `--tool-id`\*, `--document-id`\*, `--view-type {ORIGINAL,EXTRACT,SUMMARIZE}` | +| `file delete` | `/prompt-studio/file/{tool_id}` | DELETE | `--tool-id`\*, `--document-id`\* | +| `prompt create` | `/prompt-studio/prompt-studio-prompt/{tool_id}/` | POST | `--tool-id`\* (**also sent in the body**, see note), `--prompt-key`\*, `--enforce-type {text,number,email,date,boolean,json,line-item,table}` (`date` is locale-ambiguous — see note), `--prompt`, `--sequence-number`, `--prompt-type {PROMPT,NOTES}`, `--active` | +| `prompt get/update/patch/delete` | `/prompt-studio/prompt/{prompt_id}/` | GET/PUT/PATCH/DELETE | `--prompt-id`\* + fields above | +| `prompt reorder` | `/prompt-studio/prompt/reorder/` | POST | `--start-sequence-number`\*, `--end-sequence-number`\*, `--prompt-id`\* | +| `profile list` | `/prompt-studio/prompt-studio-profile/{tool_id}/` | GET | `--tool-id`\* | +| `profile set-default` | `/prompt-studio/prompt-studio-profile/{tool_id}/` | PATCH | `--tool-id`\*, `--default-profile`\* | +| `profile create` | `/prompt-studio/profilemanager/{tool_id}` | POST | `--tool-id`\*, `--profile-name`\*, `--vector-store`\*, `--embedding-model`\*, `--llm`\*, `--x2text`\*, `--chunk-size` (**`0` = whole-document / no-RAG**; see note), `--chunk-overlap`, `--retrieval-strategy {simple,subquestion,fusion,recursive,router,keyword_table,automerging}`, `--similarity-top-k` (max 4 profiles/project) | +| `profile get/update/patch/delete` | `/prompt-studio/profile-manager/{profile_id}/` | GET/PUT/PATCH/DELETE | `--profile-id`\* + fields above | +| `index-document` | `/prompt-studio/index-document/{tool_id}` | POST | `--tool-id`\*, `--document-id`\* | +| `fetch-response` | `/prompt-studio/fetch_response/{tool_id}` | POST | `--tool-id`\*, `--document-id`\*, `--id`\* (prompt), `--run-id`, `--profile-manager` (may be needed even when a default appears set), `--wait` | +| `single-pass` | `/prompt-studio/single-pass-extraction/{tool_id}` | POST | `--tool-id`\*, `--document-id`\*, `--run-id`, `--wait` | +| `task-status` | `/prompt-studio/{tool_id}/task-status/{task_id}` | GET | `--tool-id`\*, `--task-id`\* — status only (`processing`/`completed`/`failed`); the value lands in the Output Manager | +| `output list` | `/prompt-studio/prompt-output/` | GET | `--tool-id`\*, `--prompt-id`, `--document-id`, `--profile-id`, `--is-single-pass-extract` — reads extraction results (backend-sourced, no `.mdx`) | +| `output latest` | `/prompt-studio/prompt-output/latest-by-keys/` | GET | `--tool-id`\* — latest output per prompt key (may return `{}`; prefer `output list`) | +| `users` | `/prompt-studio/users/{tool_id}` | GET | `--tool-id`\* | +| `check-deployment-usage` | `/prompt-studio/{tool_id}/check_deployment_usage/` | GET | `--tool-id`\* | +| `select-choices` | `/prompt-studio/select_choices/` | GET | — | +| `adapter-choices` | `/prompt-studio/adapter-choices/` | GET | — | +| `retrieval-strategies` | `/prompt-studio/{tool_id}/get_retrieval_strategies/` | GET | `--tool-id`\* | + +> **`prompt create` sends `tool_id` in the body as well as the path.** The backend's `create_prompt` persists the request body verbatim and ignores the URL `pk`, so a `tool_id` absent from the body is saved as `NULL` — the prompt exists but links to no project and is unreachable (returns 201 with `tool_id: null`, then a 404 on read). The CLI mirrors the path value into the body to link it correctly, and treats a 201 whose response `tool_id` is null as a failure (exit `8`, §5.4) rather than reporting success. The real fix is a backend change. +> +> **`prompt create` also takes `--profile-manager`.** `fetch_response` resolves a prompt's LLM profile from the prompt's own `profile_manager` field and does **not** fall back to the project's default profile, unlike `index-document` and `single-pass`, which both do. A prompt created without one is therefore unrunnable, and fails with a message naming the *project* default — which is genuinely set, making the error actively misleading. Setting it at creation avoids the trap. The backend fix is to fall back to `ProfileManager.get_default_llm_profile(tool)` when the prompt's field is null, matching the two sibling paths; the flag then becomes convenience rather than a requirement. +> +> **`--chunk-size 0` means whole-document / no-RAG.** It skips embedding and the vector DB entirely — the correct mode for short documents, and the escape hatch when the vector DB is unreachable (indexing otherwise 500s on a dead vector DB even for a one-page file). Set `--chunk-overlap 0` alongside it. `--vector-store` and `--embedding-model` stay **required** even so: both columns are `NOT NULL` server-side, so the values are stored but never queried. Pass any valid adapter id. Making them genuinely optional requires a backend change (nullable columns). +> +> **`--enforce-type date` is locale-ambiguous.** Date normalization reads `DD/MM/YYYY` as `MM/DD/YYYY` (`01/08/2025` → `2025-01-08`). For non-US date sources prefer `--enforce-type text`. +> +> **Retrieving results.** `fetch-response` / `single-pass` return `202 {task_id, run_id, status:accepted}` and never return the value directly — it lands in the Output Manager. Read it with `output list` (or `output latest`), or use `--wait` (§3.1) to poll and return it in one call. + +#### Workflows — `unstract docstudio platform workflow` + +| Command | Endpoint | Method | Key parameters | +| --- | --- | --- | --- | +| `list` | `/workflow/` | GET | `--project`, `--workflow-owner`, `--is-active`, `--order-by {asc,desc}` | +| `create` | `/workflow/` | POST | `--workflow-name`\* (≤128, unique/org), `--description` (≤490), `--deployment-type {DEFAULT,ETL,TASK,API,APP}`, `--source-settings` (JSON), `--destination-settings` (JSON), `--max-file-execution-count` (≥1), `--shared-to-org`, `--shared-users` | +| `get` / `update` / `patch` / `delete` | `/workflow/{id}/` | GET/PUT/PATCH/DELETE | `--id`\* + fields above (update/delete require owner) | +| `execute` | `/workflow/execute/` | POST | `--workflow-id`\*, `--execution-action {START,NEXT,STOP,CONTINUE}`, `--execution-id` (required for NEXT/STOP/CONTINUE), `--log-guid`, `--file` (repeatable) | +| `toggle-active` | `/workflow/active/{id}/` | PUT | `--id`\* | +| `can-update` | `/workflow/{id}/can-update/` | GET | `--id`\* | +| `clear-file-marker` | `/workflow/{id}/clear-file-marker/` | GET | `--id`\* (mutating GET — flagged in help) | +| `schema` | `/workflow/schema/` | GET | `--type {src,dest}` (default `src`), `--entity {file,api,db}` (default `file`) | +| `users` | `/workflow/{id}/users/` | GET | `--id`\* | +| `execution list` | `/workflow/{id}/execution/` | GET | `--id`\* | +| `execution get` | `/workflow/execution/{id}/` | GET | `--id`\* | +| `execution logs` | `/workflow/execution/{id}/logs/` | GET | `--id`\*, `--file-execution-id` (literal `null` for non-file logs), `--log-level {DEBUG,INFO,WARN,ERROR}`, `--ordering`, `--page`, `--page-size` | +| `file-history list` | `/workflow/{workflow_id}/file-histories/` | GET | `--workflow-id`\*, `--status` (CSV), `--execution-count-min/max`, `--file-path` (prefix), `--page`, `--page-size` | +| `file-history get/delete` | `/workflow/{workflow_id}/file-histories/{id}/` | GET/DELETE | `--workflow-id`\*, `--id`\* | +| `file-history clear` | `/workflow/{workflow_id}/file-histories/clear/` | POST | `--workflow-id`\*, ≥1 of: `--ids` (≤100), `--status`, `--execution-count-min/max`, `--file-path` | + +#### Workflow assembly — attach a tool and configure endpoints (deployment prerequisites) + +A workflow is only deployable once a tool is attached and both of its endpoints are set to `API`. These commands expose that assembly, which previously required direct API calls. **These endpoints have no public v1 docs page**; their `doc_source` cites the backend routes (like API Hub, §8.4). + +| Command | Endpoint | Method | Key parameters | +| --- | --- | --- | --- | +| `tool registry list` | `/tool/` | GET | — (each tool's `function_name` is the registry id used below; **not** the Prompt Studio `tool_id`) | +| `tool registry settings-schema` | `/tool_settings_schema/` | GET | `--function-name`\* (reveals required adapter settings, e.g. `challenge_llm`) | +| `workflow tool list` | `/tool_instance/` | GET | `--workflow` (filter) | +| `workflow tool get` | `/tool_instance/{id}/` | GET | `--id`\* (read `metadata` before patching it) | +| `workflow tool add` | `/tool_instance/` | POST | `--workflow`\*, `--tool`\* (registry id) — a workflow holds ≤1 tool; **set the default triad first** (below); attaching activates the workflow | +| `workflow tool set-metadata` | `/tool_instance/{id}/` | PATCH | `--id`\*, `--metadata`\* (JSON; **REPLACES** wholesale — send the complete object, accepts `@file.json`) | +| `workflow tool remove` | `/tool_instance/{id}/` | DELETE | `--id`\* (needs `full_access` — `read_write` cannot delete a tool instance even though it can create one) | +| `workflow endpoint list` | `/workflow/endpoint/` | GET | `--workflow`, `--endpoint-type {SOURCE,DESTINATION}`, `--connection-type` | +| `workflow endpoint set` | `/workflow/endpoint/{id}/` | PATCH | `--id`\*, `--connection-type {API,FILESYSTEM,DATABASE,MANUALREVIEW}`\* | + +> **The deploy-a-tool sequence.** `export-tool` (§Prompt Studio) publishes the project to the registry → `tool registry list` gives its `function_name` → `workflow create` (auto-creates both endpoints, `connection_type` null) → `adapter default-triad set` (**must precede** `tool add`, see below) → `workflow tool add` → `workflow endpoint set` both SOURCE and DESTINATION to `API` → `api-deployment create` → `api-deployment key create` → `deployment run`. For an API deployment, endpoints set to `API` (or `MANUALREVIEW`) need no connector/credentials. +> +> **Ordering trap — set the default triad first (CAPTURE2 GAP 3).** `workflow tool add` seeds the tool instance's adapter settings from the org **default triad**. With no triad set, creation returns HTTP 500 *but still persists a half-configured row*; a retry then 400s with "can't have more than one tool." Run `adapter default-triad set` before `workflow tool add`. +> +> **`challenge_llm` trap (CAPTURE2 BUG 4).** An exported tool's settings schema may list `challenge_llm` as required even when the project has `enable_challenge=false`. A tool instance whose `metadata.challenge_llm` is empty then fails deploy-time validation with an opaque enum error (visible only in `workflow execution logs`). Fix it with a read-modify-write: `workflow tool get` → add a valid LLM adapter id (from `settings-schema`'s enum) to the **existing** `metadata` object → `workflow tool set-metadata --metadata @edited.json`. `set-metadata` **replaces** metadata wholesale (the backend does not merge), so sending only the one key would wipe `prompt_registry_id` and orphan the tool — always send the complete object. +> +> **Not shipped: a one-shot `deploy-api` convenience.** CAPTURE2 floated a single command that runs the whole sequence. Deliberately deferred: the individual commands above make the flow CLI-only, and a mega-command would bury the ordering/preflight decisions (triad, endpoint config) that the user needs to see. Revisit if the assembly proves rote in practice. + +#### API Deployments (management) — `unstract docstudio platform api-deployment` + +| Command | Endpoint | Method | Key parameters | +| --- | --- | --- | --- | +| `list` | `/api/deployment/` | GET | `--workflow`, `--search`, `--page`, `--page-size` | +| `create` | `/api/deployment/` | POST | `--workflow`\*, `--display-name` (≤30), `--description` (≤255), `--api-name` (`^[a-zA-Z0-9_-]+$`, ≤30, unique/org), `--is-active`, `--shared-to-org`, `--shared-users` — returns `api_key`, one active deployment per workflow | +| `get` / `update` / `patch` / `delete` | `/api/deployment/{id}/` | GET/PUT/PATCH/DELETE | `--id`\* (owner only for writes) | +| `users` | `/api/deployment/{id}/users/` | GET | `--id`\* | +| `by-prompt-studio-tool` | `/api/deployment/by-prompt-studio-tool/` | GET | `--tool-id`\* | +| `postman-collection` | `/api/postman_collection/{id}/` | GET | `--id`\*, `--save` (409 if no active key) | +| `key list` / `key create` | `/api/keys/api/{api_id}/` | GET / POST | `--api-id`\*; create: exactly one of `--api` / `--pipeline`, `--description` (≤255), `--is-active` | +| `key get` / `key update` / `key delete` | `/api/keys/{id}/` | GET/PUT/DELETE | `--id`\*, `--is-active`, `--description` | + +#### ETL / Task Pipelines — `unstract docstudio platform pipeline` + +| Command | Endpoint | Method | Key parameters | +| --- | --- | --- | --- | +| `list` | `/pipeline/` | GET | `--type {ETL,TASK,DEFAULT,APP}`, `--workflow`, `--search`, `--ordering {created_at,last_run_time,pipeline_name,run_count}` (`-` prefix = desc), `--page`, `--page-size` | +| `create` | `/pipeline/` | POST | `--pipeline-name`\* (≤32, unique/org), `--workflow`\*, `--pipeline-type`, `--cron-string` (min interval enforced), `--shared-users`, `--shared-to-org` | +| `get` / `update` / `patch` / `delete` | `/pipeline/{id}/` | GET/PUT/PATCH/DELETE | `--id`\*; PATCH also `--active` | +| `execute` | `/pipeline/execute/` | POST | `--pipeline-id`\*, `--execution-id` | +| `executions` | `/pipeline/{id}/executions/` | GET | `--id`\*, `--start-date`, `--end-date` (ISO 8601), `--page`, `--page-size` | +| `users` | `/pipeline/{id}/users/` | GET | `--id`\* | +| `postman-collection` | `/pipeline/api/postman_collection/{id}/` | GET | `--id`\*, `--save` (400 if no active key) | +| `key list` / `key create` | `/api/keys/pipeline/{pipeline_id}/` | GET / POST | `--pipeline-id`\*; create: exactly one of `--pipeline` / `--api`, `--description`, `--is-active` | + +#### Adapters — `unstract docstudio platform adapter` + +| Command | Endpoint | Method | Key parameters | +| --- | --- | --- | --- | +| `supported` | `/supported_adapters/` | GET | `--adapter-type {LLM,EMBEDDING,VECTOR_DB,X2TEXT,OCR}`\* | +| `schema` | `/adapter_schema/` | GET | `--id`\* (SDK identifier, e.g. `openai_llm`) | +| `test` | `/test_adapters/` | POST | `--adapter-id`\*, `--adapter-metadata`\* (JSON), `--adapter-type`\* | +| `list` | `/adapter/` | GET | `--adapter-type` | +| `create` | `/adapter/` | POST | `--adapter-name`\* (≤128), `--adapter-id`\*, `--adapter-type`\*, `--adapter-metadata`\* (JSON, encrypted at rest), `--description`, `--shared-to-org` | +| `get` | `/adapter/{id}/` | GET | `--id`\* (returns decrypted metadata) | +| `update` / `patch` / `delete` | `/adapter/{id}/` | PUT/PATCH/DELETE | `--id`\*; PATCH also `--shared-users` (replaces list); delete 409 if in use, 500 if default | +| `info` | `/adapter/info/{id}/` | GET | `--id`\* (includes `context_window_size`) | +| `users` | `/adapter/users/{id}/` | GET | `--id`\* | +| `default-triad get` | `/adapter/default_triad/` | GET | — | +| `default-triad set` | `/adapter/default_triad/` | POST | `--llm-default`, `--embedding-default`, `--vector-db-default`, `--x2text-default` (note: request keys differ from response keys) | + +#### Connectors — `unstract docstudio platform connector` + +| Command | Endpoint | Method | Key parameters | +| --- | --- | --- | --- | +| `supported` | `/supported_connectors/` | GET | `--type {INPUT,OUTPUT}`, `--connector-mode {FILE_SYSTEM,DATABASE}` | +| `schema` | `/connector_schema/` | GET | `--id`\* | +| `test` | `/test_connectors/` | POST | `--connector-id`\*, `--connector-metadata`\* (JSON) | +| `list` | `/connector/` | GET | `--workflow`, `--created-by`, `--connector-type`, `--connector-mode` | +| `create` | `/connector/` | POST | `--connector-name`\* (≤128), `--connector-id`\*, `--connector-metadata` (JSON), `--connector-version`, `--shared-to-org`, `--shared-users`, `--oauth-key` (query) | +| `get` / `update` / `patch` / `delete` | `/connector/{id}/` | GET/PUT/PATCH/DELETE | `--id`\*; delete 409 if used by a workflow | +| `oauth-cache-key` | `/api/v1/oauth/cache-key/{backend}` | GET | `--backend`\* (e.g. `google-oauth2`); **not org-scoped** | + +#### Groups, users, sharing + +| Command | Endpoint | Method | Key parameters | +| --- | --- | --- | --- | +| `group list` / `group create` | `/groups/` | GET / POST | `--name`\* (unique/org), `--description` | +| `group patch` / `group delete` | `/groups/{id}/` | PATCH / DELETE | `--id`\* (**int**, not UUID); DELETE needs `full_access` | +| `group member list` / `add` | `/groups/{id}/members/` | GET / POST | `--id`\*; add: `--user-ids`\* (ints, idempotent) | +| `group member remove` | `/groups/{id}/members/{user_id}` | DELETE | `--id`\*, `--user-id`\* (no trailing slash; needs `full_access`) | +| `group resources` | `/groups/{id}/resources/` | GET | `--id`\* | +| `user list` | `/users/` | GET | — (`id` returned as **string**; cast to int for `shared_users`) | +| `share` | `/{resource}/{id}/share/` | POST | `--resource`\* (**enum**, see below), `--id`\*, `--shared-users`, `--shared-groups`, `--shared-to-org` (each axis is **replace**, not append) | + +> **`--resource` is an enum, never free text.** The URL path segment is not guessable from the friendly resource name, so an agent passing `api-deployment` would get a 404. The flag accepts friendly names and maps them to exact path segments, with the mapping enumerated in `--help` and `--discover`: +> +> | `--resource` value | Path segment | +> | --- | --- | +> | `adapter` | `adapter` | +> | `connector` | `connector` | +> | `workflow` | `workflow` | +> | `pipeline` | `pipeline` | +> | `api-deployment` | `api/deployment` | +> | `prompt-studio` | `prompt-studio` | + +> **Sharing semantics.** `shared_users` replaces rather than appends. The CLI provides `--add-user` / `--remove-user` convenience flags that read current state first and send the merged list, and documents the replace behaviour of the raw flag in help text. + +### 6.4 Human Quality Review (Enterprise) + +Base: `{host}/mr/api/{org_id}` · Auth: `Authorization: Bearer` + +| Command | Endpoint | Method | Parameters | +| --- | --- | --- | --- | +| `hitl approved get` | `/approved/result/{class_id}/` | GET | `--class-id`\*, `--hitl-queue-name`, `--save` — **dequeue: consumes one item per call** | +| `hitl bulk-download` | `/approved/result/{class_id}/` | GET | `--class-id`\*, `--page` (default 1), `--page-size` (1–500, default 50), `--download-files` (bool), `--email` (async notification) | +| `hitl download-status` | `/approved/download-status/{job_id}/` | GET | `--job-id`\* | + +Pushing to HITL is `unstract docstudio deployment run --hitl-queue-name ` (§6.2), which returns `QUEUED` + `execution_id`. + +### 6.5 API Hub (Verticals) + +Auth: `apikey: ` (Kong gateway). Optional passthrough: `--llmwhisperer-key` → `X-LLMWhisperer-API-Key`, `--anthropic-key` → `X-Anthropic-API-Key` (bring-your-own-key). Subscription/user headers are gateway-injected and never sent by the CLI (§4.4). +**Source of truth: code + Postman collections, not public docs** (see §8.4). + +| Command | Endpoint | Method | Parameters | +| --- | --- | --- | --- | +| `apihub extract` | `/api/v1/extract` | POST | `--vertical`\* (e.g. `table`), `--sub-vertical`\* (`bank_statement`, `discover_tables`, `extract_table`), `--file` \| `--use-cached-file-hash`; **conversion passthrough** (`conv_*`): `--conv-mode` (default `high_quality`), `--conv-output-mode` (default `layout_preserving`), `--conv-lang`, `--conv-tag`, `--conv-filename`, `--conv-page-separator`, `--conv-pages-to-extract`, `--conv-median-filter-size`, `--conv-gaussian-blur-radius`, `--conv-mark-vertical-lines`, `--conv-mark-horizontal-lines`, `--conv-line-splitter-strategy`, `--conv-line-splitter-tolerance`, `--conv-horizontal-stretch-factor`; **extraction** (`ext_*`): `--ext-section-name`, `--ext-compress-double-space`, `--ext-headers`, `--ext-start-page`, `--ext-end-page`, `--ext-page-filter-strategy`, `--ext-use-bank-schema`, `--ext-pattern {generic_table,indent_as_groups}`, `--ext-table-no`, `--ext-cache-result`, `--ext-cache-text` | +| `apihub status` | `/api/v1/status` | GET | `--file-hash`\* | +| `apihub retrieve` | `/api/v1/retrieve` | GET | `--file-hash`\*, `--output-mode {raw,full}` (default `full`), `--sub-vertical`, `--save` | +| `apihub doc-splitter upload` | `/doc-splitter/documents/upload` | POST | `--file`\* | +| `apihub doc-splitter status` | `/doc-splitter/jobs/status` | GET | `--job-id`\* | +| `apihub doc-splitter download` | `/doc-splitter/jobs/download` | GET | `--job-id`\*, `--save` | + +Statuses: `QUEUED_FOR_WHISPER` → `QUEUED_FOR_EXTRACTION` → `COMPLETED`. The `conv_*` prefix maps onto LLMWhisperer parameters; `ext_*` parameters are forwarded verbatim to the vertical worker, so the CLI accepts an escape hatch `--ext-param KEY=VALUE` (repeatable) for parameters newer than the CLI. + +--- + +## 7. Architecture + +``` +unstract_cli/ +├── __main__.py # entry point +├── app.py # Click root group; command tree built from definitions +├── config/ +│ ├── profile.py # profile model, resolution order (flag > env > file > default) +│ └── loader.py # TOML load/save, env: indirection, permission checks +├── core/ +│ ├── http.py # request execution, retry/backoff, redaction +│ ├── errors.py # HTTP status → exit code + structured error mapping +│ ├── output.py # json/yaml/table/raw renderers, TTY detection +│ ├── poll.py # --wait state machines (body-status based, never HTTP code) +│ └── generate.py # endpoint definition → Click command +├── endpoints/ # ← SINGLE SOURCE OF TRUTH (the Skill's edit target) +│ ├── whisper.py +│ ├── deployment.py +│ ├── platform_prompt_studio.py +│ ├── platform_workflow.py +│ ├── platform_deployment.py +│ ├── platform_pipeline.py +│ ├── platform_adapter.py +│ ├── platform_connector.py +│ ├── platform_groups.py +│ ├── hitl.py +│ └── apihub.py +└── skills/ # bundled Claude Skill (§8) +``` + +### 7.1 Endpoint definition shape + +```python +Endpoint( + name="extract", + group="whisper", + method="POST", + path="/whisper", + product="llmwhisperer", + summary="Convert a document to LLM-ready text.", + doc_source="llmwhisperer-docs/docs/llm_whisperer/apis/whisper.md", # skill anchor + params=[ + Param("mode", str, default="form", + choices=["native_text", "low_cost", "high_quality", "form", "table"], + location="query", help="Processing mode."), + Param("word_confidence_threshold", float, default=0.3, location="query", + help="Minimum OCR confidence (0-1); works only with form/high_quality/table."), + ... + ], + body="binary_file", + returns="whisper_hash", + poll=PollSpec(status_endpoint="whisper.status", + terminal_success=("processed",), terminal_failure=("error",), + handle_field="whisper_hash", handle_param="whisper_hash", + retrieve_endpoint="whisper.retrieve", one_shot=True), +) +``` + +`PollSpec.status_field` may be a single field name **or a tuple of candidates tried in order**, because the run POST and the status GET can spell the same state differently — the deployment poll uses `("status", "execution_status")` so the status GET's top-level `status` wins while the run POST's nested `execution_status` still resolves (§6.2). A mismatch here silently consumes a one-shot result; the tuple is what prevents it. + +`PollSpec` also carries the fields that make a two-step retrieve work when the result store is not keyed by the poll handle: `retrieve_carry` forwards identifiers from the *original* request into the retrieve call (each entry a py_name, or a `(source, dest)` pair that renames it), `retrieve_extra` injects a constant flag the original request did not carry, and `retrieve_omits_handle` drops the handle from a retrieve endpoint that has no such parameter. Prompt Studio's `--wait` uses all three: it reads the Output Manager by the original `tool_id`, renames `id → prompt_id`, and omits the `task_id` handle (§3.1, §6.3). When `one_shot=True` and no `retrieve_endpoint` is set, the poll that first reaches a terminal state *is* the retrieve: its body is returned as the result and no second read is issued (deployment run, §6.2). + +Records may also mirror a PATH identifier into the body — `Param.mirror_to_body` under the same name, or `Param.mirror_as=""` where the body spells the identifier differently — and assert non-null response fields on success (`Endpoint.require_response_fields`). `prompt create` uses the first pair to defeat the tool-linking defect (§6.3); `api-deployment key create` uses the rename, sending its `api_id` path value as the body's `api` so the caller supplies one identifier rather than the same value twice. `Param.default_from` accepts a whitespace-separated **fallback chain** resolved left to right (`"deployment.org_id platform.org_id"`), for settings that legitimately live in two blocks where one is typically empty; credentials deliberately do not chain, since borrowing a key across API groups would be credential confusion. `Constraint` also covers conditional requirement (`RequiredUnless`) alongside `MutuallyExclusive` / `AtLeastOneOf`. `ParamType.JSON` values are parsed automatically (inline or `@file.json`, §6). + +The command tree, `--help`, validation, `--discover`, and the docs-diff in §8 all derive from these records. Adding an endpoint means adding one record — no separate command wiring, so help text cannot drift from behaviour. + +> **Note (doc drift, pre-existing):** the file tree above lists per-resource modules (`platform_prompt_studio.py`, `platform_workflow.py`, …); the implementation consolidated the Platform surface into a single `endpoints/platform.py`. This predates the changes described here and is left as-is pending a broader §7 refresh. + +--- + +## 8. Claude Skill: `update-unstract-cli` + +Location: `unstract-cli/.claude/skills/update-unstract-cli/SKILL.md` + +### 8.1 Purpose + +Keep the CLI's endpoint definitions synchronized with the public API documentation by cross-referencing the docs repos, detecting drift, and applying the corresponding edits to `endpoints/`. + +### 8.2 Inputs + +| Source | Repo / path | Covers | +| --- | --- | --- | +| LLMWhisperer docs | `llmwhisperer-docs/docs/llm_whisperer/apis/*.md` | §6.1 | +| Unstract API deployment docs | `unstract-docs/docs/unstract_platform/api_deployment/*.md` | §6.2 | +| Unstract Platform v1 docs | `unstract-docs/docs/unstract_platform/api_documentation/versions/v1-*.mdx` | §6.3 | +| HITL docs | `unstract-docs/docs/unstract_platform/human_quality_review/*.md` | §6.4 | +| API Hub source | `unstract-verticals/src/api_v1/api.py`, `verticals-portal/portal/postman-collection/*.json` | §6.5 (no public docs — see §8.4) | + +Repo paths are configurable; the Skill falls back to the GitHub API when a repo is not checked out locally. + +### 8.3 Procedure + +1. **Parse the documentation.** Extract endpoint tables from Markdown and `` / `` MDX components into a normalized set of `(method, path, params[])` records. Each `Endpoint.doc_source` anchors a definition to its documentation file. +2. **Parse the CLI definitions.** Load `endpoints/*.py` into the same normalized shape. +3. **Diff** on three axes: + - endpoints present in docs but missing from the CLI (**new capability**); + - endpoints in the CLI but absent from docs (**possible removal or deprecation** — report, never auto-delete); + - per-parameter drift: added, removed, renamed, or changed type / default / enum / required-ness. A documented parameter supplied by mirroring (`Param.mirror_as`, §7.1) is **not** drift — it reaches the wire under its documented name without a flag of its own. A record carrying `doc_conflict` is exempt from this axis as well as the previous one, so a deliberately dropped parameter is not re-reported every run. +4. **Report** as a structured table before editing: each difference with its doc citation (file + heading) and the proposed CLI change. +5. **Apply** by editing `endpoints/*.py` only. Because commands are generated (§7.1), no command-wiring, help-text, or `--discover` changes are needed — this is why the diff can be applied mechanically. +6. **Verify:** run `unstract --discover` and confirm it parses and includes the new surface; run the test suite; run `ruff`/`mypy`. +7. **Summarize:** what changed, what needs human judgement (renames, breaking changes), and what was intentionally skipped. + +### 8.4 Known limitation — API Hub + +API Hub has **no public documentation site**. The Skill cannot cross-reference it against public docs and instead diffs against `unstract-verticals/src/api_v1/api.py` (`request.args.get(...)` calls) and the Postman collections in `verticals-portal`. This is lower-fidelity: it recovers parameter names and defaults but not prose descriptions. The Skill must **flag API Hub changes for human review** rather than applying them silently, and the `--ext-param KEY=VALUE` escape hatch (§6.5) exists precisely so agents are not blocked by this lag. + +### 8.5 Safety rules + +- Never delete a command solely because it is absent from the docs — documentation lags implementation. Report and let a human decide. +- Never invent parameters not present in a source; every change cites a specific file and heading. +- Treat renames as breaking: propose an alias with a deprecation note rather than a silent rename. +- Preserve hand-written `help` text where it is richer than the docs; add rather than overwrite. +- **Docs are not infallible.** Where an endpoint index and its detail page disagree, prefer the detail page and the official client libraries, and check §11 "Resolved during drafting" for known documentation defects before proposing a change. A definition may carry a `doc_conflict` note recording a deliberate divergence; the Skill must not revert one without explicit human confirmation. Such a note covers a dropped parameter as well as a divergent path — `api-deployment key create` omits the documented `api`/`pipeline` body flags because the route already fixes the resource and the value is mirrored from the path (§7.1). + +--- + +## 9. Testing + +| Layer | Approach | +| --- | --- | +| Definition integrity | Every `Endpoint` has a unique `(group, name)`, valid `doc_source`, and complete param metadata | +| Command generation | Tree builds; `--help` renders at every level; `--discover` is valid JSON and covers every endpoint | +| HTTP | `respx`/`responses` fixtures per endpoint using the documented sample payloads | +| Exit codes | Table-driven: each HTTP status maps to the §5.4 code | +| Polling | `--wait` reaches terminal state; branches on **body status**, not HTTP code (asserted explicitly against the 422 defect). Prompt Studio `--wait` polls `task-status` then retrieves from the Output Manager keyed on the original `tool_id`, narrowed to the run's prompt + document. Deployment `--wait` recognises the status GET's top-level `status`, returns the terminal poll's body on the *first* terminal poll (asserted `call_count == 1` — no double-consume), and `run --wait --save` persists it | +| One-JSON-document output | A response body of several concatenated JSON objects is recovered into a single array, so `\| json` never breaks; a genuinely malformed body falls back to raw text | +| JSON parameters | A `ParamType.JSON` flag resolves to a nested object in the body (not a quoted string); accepts inline JSON and `@file.json`; invalid JSON exits `2`; a FORM-located JSON field is re-serialized to a string so multipart encoding still works | +| One-shot semantics | Second retrieve yields exit `9`; `--save` persists before exit | +| Error stream | On failure with stdout not a TTY, the structured envelope appears on **both** stdout and stderr; on a TTY, stderr only. Covers our errors, Click's own parse errors (subprocess-tested at the real entry point), and `--discover` no-match | +| Response guards | A 2xx that leaves a `require_response_fields` field null (e.g. `prompt create` → `tool_id`) is treated as a failure (exit `8`), not success | +| Redaction | No secret appears in any output stream, including `--dry-run`, `-vv`, and `config doctor` | +| Non-interactivity | No command reads stdin unless explicitly asked (`--file -`); no TTY prompts | + +--- + +## 10. Delivery phases + +| Phase | Scope | +| --- | --- | +| **1** | Core: config/profiles, HTTP layer, output/errors/exit codes, definition→command generation, `--discover` | +| **2** | LLMWhisperer (§6.1) + `--wait` + one-shot handling — smallest complete product surface | +| **3** | API Deployments runtime (§6.2) + HITL (§6.4) | +| **4** | Platform Management v1 (§6.3) — largest surface | +| **5** | API Hub (§6.5) incl. `--ext-param` escape hatch | +| **6** | Claude Skill (§8), packaging, shell completions | + +--- + +## 11. Open questions + +1. **API Hub public base URL.** Tenancy is resolved (§4.4): callers send `apikey`; Kong injects subscription/user headers from Redis. The externally routable hostname still needs confirmation from the deployment charts — `--base-url` / `UNSTRACT_APIHUB_BASE_URL` is required until then. +2. **Platform API `full_access` permission level.** The v1 overview lists `read`, `read_write`, `full_access`, but the Platform-Keys page exposes only `read` and `read_write` in its UI enum. Since `DELETE` commands require `full_access`, confirm whether it is UI-creatable — if not, every `delete` command is unusable with a UI-issued key and help text must say so. + +### Resolved during drafting + +- **Multi-doc chat APIs are excluded from v1** (confirmed). `/md/file/upload`, `/md/file/search`, and `/md/chat` (base `https://us-central.unstract.com/api/v1`, auth `Authorization: Bearer `) are documented but carry `draft: true`. Draft endpoints may change without notice, and including them would make the update-Skill churn against an unstable contract. When the draft flag is lifted, they belong under a new `unstract chat {upload,search,ask}` group. **The Skill must not auto-add these endpoints** while `draft: true` is present in their front matter — treat the flag as an exclusion marker. +- **`/whisper-detail` vs `/whisper-details`.** The docs index says `/whisper-details`; the endpoint page and the official `llm-whisperer-python-client` (`client_v2.py:349`) both use **`/whisper-detail`** (singular). Pinned to the singular form; the docs index is wrong, and the Skill must not "correct" this back from the index page. diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 0000000..243c0a1 --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,74 @@ +[project] +name = "unstract-cli" +version = "0.1.0" +description = "Unified, LLM-friendly CLI for the Unstract suite of products" +readme = "README.md" +requires-python = ">=3.12" +license = { text = "Unstract" } + +dependencies = [ + # Deliberately minimal, to keep the supply-chain surface small. Every entry + # below is actually imported by src/; anything unused has been removed. + # + # Click is pinned deliberately: `--discover` depends on the shape of + # `click.Parameter.to_info_dict()`. A Click major bump could reshape that + # dict and silently degrade command discovery, so `tests/test_cli.py` + # asserts the keys we rely on. Click's only transitive dependency is + # colorama (Windows-only). + "click>=8.1,<9", + # httpx owns TLS, redirects, connection reuse and -- the part with no stdlib + # equivalent -- multipart/form-data encoding for file uploads. Kept because + # replacing it means hand-writing unaudited network code (see the note in the + # README/SPEC on why the dependency set stops here rather than at zero). + "httpx>=0.27", + # Zero transitive dependencies each. pyyaml backs `--output yaml`; tomli-w + # writes the config file (reading uses the stdlib `tomllib`). + "pyyaml>=6.0", + "tomli-w>=1.0", +] + +[project.optional-dependencies] +dev = [ + "pytest>=8.0", + "pytest-cov>=5.0", + "respx>=0.21", + "ruff>=0.6", + "mypy>=1.11", + "types-PyYAML", +] + +[project.scripts] +unstract = "unstract_cli.__main__:main" + +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[tool.hatch.build.targets.wheel] +packages = ["src/unstract_cli"] + +[tool.ruff] +line-length = 90 +target-version = "py312" +src = ["src", "tests"] + +[tool.ruff.lint] +select = ["E", "F", "W", "I", "N", "UP", "B", "C4", "SIM"] +ignore = [ + "E501", + # `str, Enum` is deliberate: members must serialise as plain strings in JSON + # output and `--discover`. StrEnum would work on 3.12 but the explicit + # form documents the intent and matches the sibling repos. + "UP042", +] + +[tool.mypy] +python_version = "3.12" +warn_return_any = true +warn_unused_configs = true +check_untyped_defs = true +ignore_missing_imports = true + +[tool.pytest.ini_options] +testpaths = ["tests"] +pythonpath = ["src"] diff --git a/src/unstract_cli/__init__.py b/src/unstract_cli/__init__.py new file mode 100644 index 0000000..add0f52 --- /dev/null +++ b/src/unstract_cli/__init__.py @@ -0,0 +1,9 @@ +"""Unstract CLI -- one LLM-friendly interface to the three products built by +Unstract: Document Studio, LLMWhisperer and API Hub. + +Unstract is the company name, and stays the name of this CLI. +""" + +from unstract_cli.app import __version__ + +__all__ = ["__version__"] diff --git a/src/unstract_cli/__main__.py b/src/unstract_cli/__main__.py new file mode 100644 index 0000000..30fcd60 --- /dev/null +++ b/src/unstract_cli/__main__.py @@ -0,0 +1,51 @@ +"""Console entry point.""" + +from __future__ import annotations + +import sys + +import click + + +def main() -> int: + """Run the CLI. + + Runs Click with ``standalone_mode=False`` so that Click's own parse errors + (`No such option`, missing argument) reach us instead of being printed and + swallowed internally. We then render them through the same structured error + envelope every other error uses, additionally on stdout when piped, so a + wrapper feeding stdout to a JSON parser sees valid JSON on the error path too + (DOC 9). + + That mode also changes how ``ctx.exit(code)`` behaves: instead of raising + ``SystemExit``, Click *returns* the code from the invocation. Returning 0 + unconditionally here therefore reported every failure as success -- a + validation error printed ``"exit_code": 2`` in its envelope while the process + exited 0, so `set -e` and any agent branching on the status code saw a pass. + The returned value is the command's real exit code, so it is what we return + (SPEC §5.4). + """ + from unstract_cli.app import build_cli + from unstract_cli.core.errors import CLIError, ExitCode + + try: + result = build_cli()(standalone_mode=False) + # Click returns whatever `ctx.exit` was given; a command that simply + # falls off the end returns None, which is success. + return result if isinstance(result, int) else 0 + except click.UsageError as exc: + # Click's human-facing message (with its usage/help hint) to stderr... + exc.show() + # ...and the machine-readable envelope to stdout when piped. + CLIError(exc.format_message(), ExitCode.USAGE).emit_stdout_only() + return exc.exit_code if exc.exit_code is not None else int(ExitCode.USAGE) + except click.ClickException as exc: + exc.show() + return exc.exit_code + except click.exceptions.Abort: + click.echo("Aborted!", err=True) + return 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/src/unstract_cli/app.py b/src/unstract_cli/app.py new file mode 100644 index 0000000..5a362d9 --- /dev/null +++ b/src/unstract_cli/app.py @@ -0,0 +1,563 @@ +"""CLI entry point and the machine-readable command index. + +`--discover` is the feature that makes this CLI usable by an agent without +external documentation (SPEC.md §5.3). It emits the full command tree as JSON, +combining two sources: + +* the **`Endpoint` records**, which carry what Click cannot know -- HTTP method + and path, the documentation file each record was authored from, the required + key permission, and one-shot semantics; +* the **introspected Click tree**, which carries the flags exactly as the parser + will accept them. + +Emitting both means the index describes the *API*, not merely the CLI, and cannot +drift from the commands actually registered. +""" + +from __future__ import annotations + +import json +import sys +from typing import Any + +import click + +from unstract_cli.commands.config_cmd import config_group +from unstract_cli.config.loader import set_config_path +from unstract_cli.core.generate import build_group_tree +from unstract_cli.core.model import Endpoint +from unstract_cli.endpoints import ALL_ENDPOINTS + +__version__ = "0.1.0" + +#: Flags added to every generated command; excluded from the per-command flag +#: listing in `--discover` so the endpoint's own parameters stand out. +_COMMON_FLAGS = frozenset( + { + "output", "profile", "base_url", "dry_run", "quiet", "verbose", + "max_retries", "no_retry", "timeout", "help", + } +) + + +def _param_info(param: click.Parameter) -> dict[str, Any]: + """Describe one parameter using Click's own introspection. + + Uses `to_info_dict()` rather than reading private attributes, so the shape is + a supported contract; `tests/test_cli.py` asserts the keys we rely + on still exist, so a Click upgrade fails loudly rather than silently. + + Crucially this reports whether the parameter is an **option** (``--flag x``) + or a positional **argument** (``x``). Getting that wrong makes the index + actively misleading: an agent would construct a command line the parser + rejects. + """ + info = param.to_info_dict() + type_info = info.get("type", {}) + kind = "argument" if info.get("param_type_name") == "argument" else "option" + entry: dict[str, Any] = { + "name": info.get("name"), + "kind": kind, + "type": type_info.get("param_type", "String").lower(), + "required": info.get("required", False), + "multiple": info.get("multiple", False), + "help": info.get("help") or "", + } + if kind == "option": + entry["flags"] = info.get("opts", []) + entry["is_flag"] = info.get("is_flag", False) + else: + # Positional: how it is written on the command line, not a flag name. + entry["usage"] = getattr(param, "metavar", None) or str( + info.get("name", "") + ).upper() + if (default := info.get("default")) is not None: + entry["default"] = default + if choices := type_info.get("choices"): + entry["choices"] = list(choices) + return entry + + +def _usage_line(path: tuple[str, ...], command: click.Command) -> str: + """Reconstruct the invocation form, so the index shows how to *call* it.""" + parts = ["unstract", *path] + for param in command.params: + info = param.to_info_dict() + if info.get("param_type_name") != "argument": + continue + # A variadic argument's own name says nothing useful ("ARGS..."), so + # prefer a declared metavar, which spells out the real shape. Click + # omits metavar from to_info_dict(), so read the attribute directly. + declared = getattr(param, "metavar", None) + name = declared or str(info.get("name", "")).upper() + if not declared and (info.get("multiple") or info.get("nargs", 1) == -1): + name = f"{name}..." + parts.append(name if info.get("required") else f"[{name}]") + if any( + p.to_info_dict().get("param_type_name") != "argument" + and p.name not in _COMMON_FLAGS + for p in command.params + ): + parts.append("[OPTIONS]") + return " ".join(parts) + + +def _endpoint_info(endpoint: Endpoint, command: click.Command) -> dict[str, Any]: + """Combine an endpoint record with its generated command.""" + entry: dict[str, Any] = { + "command": "unstract " + " ".join(endpoint.command_path), + "path": list(endpoint.command_path), + "kind": "endpoint", + "summary": endpoint.summary, + # Which of Unstract's three products this belongs to, and which API + # surface within it. Document Studio owns three API groups. + "product": endpoint.product.value, + "product_name": endpoint.product_label, + "api_group": endpoint.api.value, + "api": {"method": endpoint.method, "path": endpoint.path}, + "usage": _usage_line(endpoint.command_path, command), + "flags": [ + _param_info(p) + for p in command.params + if p.name not in _COMMON_FLAGS + ], + } + if endpoint.description: + entry["description"] = endpoint.description + if endpoint.permission: + entry["required_permission"] = endpoint.permission.value + if endpoint.constraints: + entry["constraints"] = [c.describe() for c in endpoint.constraints] + if endpoint.poll: + entry["supports_wait"] = True + entry["one_shot"] = endpoint.poll.one_shot + if endpoint.doc_source: + entry["doc_source"] = endpoint.doc_source + if endpoint.doc_conflict: + entry["doc_conflict"] = endpoint.doc_conflict + if endpoint.examples: + entry["examples"] = list(endpoint.examples) + if any(p.client_side and p.name == "save" for p in endpoint.params): + entry["supports_save"] = True + return entry + + +#: Command groups written by hand rather than generated from `Endpoint` records. +#: Introspected like any other command so the index describes what actually runs. +_HAND_AUTHORED_GROUPS: dict[str, click.Group] = {"config": config_group} + +#: Fields kept at `--detail summary`. Enough to choose a command, not to call it. +_SUMMARY_FIELDS = ("command", "kind", "summary") + +#: Detail levels, cheapest first. Full detail for every command is ~50k tokens and +#: the flat summary list is ~4.5k -- both too much for an agent to read blind -- so +#: the default is `groups`: a ~1k-token map of the ~15 navigable groups with their +#: command counts, from which the agent drills into exactly the subtree it needs. +DETAIL_LEVELS = ("groups", "summary", "full") + +#: Below this many (recursive) leaves, a group is a reasonable single drill target +#: and the overview stops descending into it; above it, the overview recurses so no +#: one drill returns an unwieldy slice. Tuned so `docstudio platform` fans out into +#: its resource subgroups while everything else stays one level deep. +_GROUP_OVERVIEW_MAX_LEAVES = 40 + + +def _matches(entry: dict[str, Any], group: str | None, command: str | None) -> bool: + """Filter one entry by group and/or command prefix. + + `command` matches on a path prefix, so "whisper webhook" selects all four + webhook commands while "whisper extract" selects exactly one. + """ + if group and entry["path"][0] != group: + return False + if command: + wanted = command.removeprefix("unstract ").split() + if entry["path"][: len(wanted)] != wanted: + return False + return True + + +def _navigable_groups(node: click.Group, path: tuple[str, ...]) -> list[dict[str, Any]]: + """Walk the Click tree into a coarse map of navigable groups (drift-free). + + Emits one entry per group at the level where it is a sensible single drill + target: a group with more than :data:`_GROUP_OVERVIEW_MAX_LEAVES` reachable + commands (and subgroups to split it) is descended into instead of emitted, so + `docstudio platform` fans out into its resource subgroups while smaller groups + stay one line. Descriptions come from each group's own ``help``, and the + `drill` command is the exact ``--discover --command`` prefix that selects the + subtree -- verified by test to return the advertised count, because this + project's rule is that discovery metadata must never lie about the parser. + """ + leaves = _count_leaves(node) + subgroups = sorted( + (n, c) for n, c in node.commands.items() if isinstance(c, click.Group) + ) + # Descend only when the group is both too big to be one drill target AND has + # subgroups to split it on; otherwise emit it whole. + if leaves > _GROUP_OVERVIEW_MAX_LEAVES and subgroups: + out: list[dict[str, Any]] = [] + # A group with its own direct leaves plus oversized subgroups still needs a + # line for those direct leaves (e.g. `docstudio platform` has one). + if any(not isinstance(c, click.Group) for c in node.commands.values()): + out.append(_group_entry(node, path, direct_only=True)) + for name, child in subgroups: + out.extend(_navigable_groups(child, (*path, name))) + return out + return [_group_entry(node, path)] + + +def _count_leaves(node: click.Group) -> int: + """Total leaf (non-group) commands reachable under a group.""" + total = 0 + for child in node.commands.values(): + total += _count_leaves(child) if isinstance(child, click.Group) else 1 + return total + + +def _group_entry( + node: click.Group, path: tuple[str, ...], *, direct_only: bool = False +) -> dict[str, Any]: + """One line in the group overview: what it is, how big, and how to drill in.""" + count = ( + sum(1 for c in node.commands.values() if not isinstance(c, click.Group)) + if direct_only + else _count_leaves(node) + ) + drill = "unstract --discover --command '" + " ".join(path) + "'" + entry = { + "group": " ".join(path), + "commands": count, + "summary": (node.short_help or node.help or "").strip().split("\n")[0], + "drill": drill + " --detail summary", + } + if direct_only: + entry["note"] = "direct commands only; subgroups are listed separately" + return entry + + +def discover( + *, + group: str | None = None, + command: str | None = None, + detail: str = "summary", +) -> dict[str, Any]: + """Build the machine-readable index of the command surface. + + Selection (`group`, `command`) and verbosity (`detail`) are independent, so + any combination works: a summary of one group, full detail for one command, + or full detail for everything. + """ + groups = build_group_tree(list(ALL_ENDPOINTS)) + commands: list[dict[str, Any]] = [] + + for endpoint in ALL_ENDPOINTS: + node: click.Command | None = groups.get(endpoint.group) + for part in endpoint.command_path[1:]: + if isinstance(node, click.Group): + node = node.commands.get(part) + if node is not None: + commands.append(_endpoint_info(endpoint, node)) + + # Hand-authored groups are introspected from the *actual* Click commands, + # never from a parallel description of them. A hand-maintained parameter list + # is exactly the drift the generated half exists to avoid -- it once + # advertised `--product/--key/--value` for `config set`, which really takes + # three positional arguments. + for group_name, click_group in _HAND_AUTHORED_GROUPS.items(): + for name, node in sorted(click_group.commands.items()): + path = (group_name, name) + commands.append( + { + "command": "unstract " + " ".join(path), + "path": list(path), + # Flagged so an agent can tell local operations from API calls. + "kind": "local", + "summary": (node.short_help or node.help or "").strip(), + "usage": _usage_line(path, node), + "flags": [ + _param_info(p) + for p in node.params + if p.name not in _COMMON_FLAGS + ], + } + ) + + commands = [c for c in commands if _matches(c, group, command)] + + # `groups` is the cheap entry point and only makes sense unfiltered: a filter + # already narrows to a subtree, so there it degrades to the flat summary list. + if detail == "groups" and (group or command): + detail = "summary" + + if detail == "summary": + commands = [ + {k: c[k] for k in _SUMMARY_FIELDS if k in c} for c in commands + ] + + envelope: dict[str, Any] = { + "cli": "unstract", + "version": __version__, + "description": ( + "Unstract CLI -- one interface to the three products built by " + "Unstract: Document Studio, LLMWhisperer and API Hub." + ), + "products": { + "docstudio": { + "name": "Document Studio", + "group": "docstudio", + "note": "Owns the platform, deployment and hitl API groups.", + }, + "llmwhisperer": {"name": "LLMWhisperer", "group": "whisper"}, + "apihub": {"name": "API Hub", "group": "apihub"}, + }, + "detail": detail, + } + if group: + envelope["group"] = group + if command: + envelope["command_filter"] = command + + # `groups`: the ~1k-token map of navigable groups, from which an agent drills + # into exactly one subtree. This is the default and the entry point, so it also + # carries the global boilerplate (exit codes, conventions, drill hints). + if detail == "groups": + tree = build_group_tree(list(ALL_ENDPOINTS)) + overview: list[dict[str, Any]] = [] + for name in sorted(tree): + overview.extend(_navigable_groups(tree[name], (name,))) + for name, hand in sorted(_HAND_AUTHORED_GROUPS.items()): + overview.extend(_navigable_groups(hand, (name,))) + return { + **envelope, + "group_count": len(overview), + "command_count": len(commands), + "how_to_drill": ( + "Each group's `drill` command lists its commands (names + summaries). " + "Add --detail full to that command for flags and API paths, or use " + "--detail summary here for the flat list of all commands." + ), + "groups": overview, + **_GLOBAL_FACTS, + } + + envelope["count"] = len(commands) + + if detail == "summary": + # The whole point of the summary level is that an agent reads it first, + # so it has to say how to get the rest. + envelope["drill_down"] = { + "one_group": "unstract --discover --group --detail full", + "one_command": "unstract --discover --command ' ' --detail full", + "everything": "unstract --discover --detail full", + "note": ( + "Full detail for every command is large (~60k tokens). Prefer " + "filtering by group or command." + ), + } + envelope["groups"] = sorted({c["command"].split()[1] for c in commands}) + + # The exit-code table and conventions are global facts, not per-command ones. + # An agent that has already read the unfiltered index knows them, and on a + # narrow query they would outweigh the answer itself -- so they ship only + # with the unfiltered view, which is the entry point. + if group or command: + return {**envelope, "commands": commands} + + return { + **envelope, + **_GLOBAL_FACTS, + "commands": commands, + } + + +#: Global facts an agent branches on: stable across commands, so they ride the +#: unfiltered entry points (the groups overview and the full unfiltered list) and +#: are omitted from narrow queries where they would outweigh the answer. +_GLOBAL_FACTS: dict[str, Any] = { + "exit_codes": { + "0": "success", + "1": "generic error", + "2": "usage error (bad or missing flags)", + "3": "authentication or authorization failure", + "4": "not found", + "5": "validation error rejected by the API", + "6": "rate limited or quota exceeded", + "7": "timed out waiting for a terminal state", + "8": "remote server error", + "9": "result already consumed (one-shot read)", + }, + "conventions": { + "output": ( + "--output json|yaml|table|raw. JSON is the default when stdout is " + "not a TTY. Payloads go to stdout; diagnostics and structured " + "errors go to stderr." + ), + "errors": ( + "Failures emit a JSON object on stderr with code, message, hint " + "and retryable." + ), + "never_interactive": "No command ever prompts; every input is a flag or env var.", + "one_shot": ( + "Commands marked one_shot return their result exactly once. Use " + "--save to persist it; a second read exits 9." + ), + "wait": ( + "Commands with supports_wait accept --wait to poll to completion, " + "plus --poll-interval and --wait-timeout." + ), + }, +} + + +class UnstractCLI(click.Group): + """Root group that suggests a correction for an unknown command.""" + + def resolve_command(self, ctx: click.Context, args: list[str]): + try: + return super().resolve_command(ctx, args) + except click.UsageError: + import difflib + + given = args[0] if args else "" + if matches := difflib.get_close_matches(given, self.list_commands(ctx), n=3): + # Suggestions go to stderr so stdout stays parseable. + click.echo( + f"Unknown command {given!r}. Did you mean: {', '.join(matches)}?", + err=True, + ) + raise + + +@click.group( + cls=UnstractCLI, + # `--discover` and a bare `unstract` must both work without a + # subcommand, so the group has to be invocable on its own. + invoke_without_command=True, + context_settings={"help_option_names": ["-h", "--help"], "max_content_width": 100}, +) +@click.version_option(__version__, "-V", "--version", prog_name="unstract") +@click.option( + "--discover", + "discover_flag", + is_flag=True, + default=False, + help="Emit the command tree as JSON, for programmatic discovery.", +) +@click.option( + "--group", + default=None, + help="Limit --discover to one product group (whisper, platform, ...).", +) +@click.option( + "--command", + "command_filter", + default=None, + help="Limit --discover to one command or command prefix, e.g. 'whisper extract'.", +) +@click.option( + "--detail", + type=click.Choice(DETAIL_LEVELS), + default="groups", + show_default=True, + help=( + "How much detail: 'groups' is the cheap map of navigable groups (default); " + "'summary' is every command with a one-liner; 'full' adds flags and API paths." + ), +) +@click.option("--profile", "-p", default=None, help="Config profile to use.") +@click.option( + "--config", + "config_file", + type=click.Path(dir_okay=False), + default=None, + help="Config file to use, overriding $UNSTRACT_CONFIG and any .unstract.toml.", +) +@click.pass_context +def cli( + ctx: click.Context, + discover_flag: bool, + group: str | None, + command_filter: str | None, + detail: str, + profile: str | None, + config_file: str | None, +) -> None: + """Unified, LLM-friendly CLI for the Unstract suite. + + Products: LLMWhisperer text extraction (`whisper`), deployed API workflows + (`deployment`), platform management (`platform`), human review (`hitl`) and + API Hub vertical extraction (`apihub`). + + \b + Machine-readable discovery -- start cheap, then drill in: + unstract --discover # all names + summaries + unstract --discover --group whisper # one product + unstract --discover --command 'whisper extract' --detail full + unstract --discover --detail full # everything (large) + + Output defaults to JSON whenever stdout is not a terminal, so piping the CLI + needs no extra flags. + """ + # Applied before any subcommand runs, so every load_config() below -- in + # generated commands and in the `config` group alike -- sees the same file. + set_config_path(config_file) + + if discover_flag: + payload = discover( + group=group, command=command_filter, detail=detail + ) + # A filtered query returns `commands`; the unfiltered groups overview + # returns `groups`. "No match" only applies to a filter that hit nothing. + if (group or command_filter) and not payload.get("commands"): + from unstract_cli.core.errors import CLIError, ExitCode + + CLIError( + "No commands matched" + + (f" --group {group}" if group else "") + + (f" --command {command_filter!r}" if command_filter else "") + + ".", + ExitCode.USAGE, + hint="Run `unstract --discover` to list valid groups.", + ).emit() + ctx.exit(2) + # Compact when piped: pretty-printing costs ~36% more tokens for an agent + # that is only going to parse it anyway. + indent = 2 if sys.stdout.isatty() else None + click.echo(json.dumps(payload, indent=indent)) + ctx.exit(0) + if ctx.invoked_subcommand is None: + click.echo(ctx.get_help()) + ctx.exit(0) + + +@click.command("completion", help="Print a shell completion script.") +@click.argument("shell", type=click.Choice(["bash", "zsh", "fish"])) +def completion(shell: str) -> None: + """Emit the completion script for a shell. + + Click generates these from the same command tree, so completions cover every + generated command without a separate registry. + + \b + Install with, for example: + unstract completion zsh > ~/.zfunc/_unstract + unstract completion bash >> ~/.bashrc + """ + click.echo( + f'eval "$(_UNSTRACT_COMPLETE={shell}_source unstract)"' + if shell != "fish" + else "eval (env _UNSTRACT_COMPLETE=fish_source unstract)" + ) + + +def build_cli() -> click.Group: + """Assemble the full command tree.""" + for group in build_group_tree(list(ALL_ENDPOINTS)).values(): + cli.add_command(group) + cli.add_command(config_group) + cli.add_command(completion) + return cli + + +__all__ = ["__version__", "build_cli", "cli", "discover"] diff --git a/src/unstract_cli/commands/__init__.py b/src/unstract_cli/commands/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/src/unstract_cli/commands/config_cmd.py b/src/unstract_cli/commands/config_cmd.py new file mode 100644 index 0000000..5e29fdf --- /dev/null +++ b/src/unstract_cli/commands/config_cmd.py @@ -0,0 +1,380 @@ +"""The `config` command group -- hand-authored, not generated. + +These commands map to no API endpoint: they operate purely on the local config +layer. They are how a user or agent bootstraps every other command. + +`--discover` describes them by introspecting these Click commands directly, so +the index always matches the parser. An earlier parallel description of them +drifted and advertised `--product/--key/--value` for `config set`, which really +takes three positional arguments -- hence: no second description, ever. + +Per SPEC.md §5.2 nothing here prompts: `init` refuses to clobber an existing file +unless `--force` is passed, rather than asking. +""" + +from __future__ import annotations + +from typing import Any + +import click + +from unstract_cli.config.loader import ( + GROUP_PATH, + TARGET_NAMES, + ConfigError, + ResolvedConfig, + config_path, + load_config, + resolve_target, + save_config, + starter_profiles, +) +from unstract_cli.core.errors import CLIError, ExitCode +from unstract_cli.core.output import OutputFormat, default_format, emit + + +def _resolve_or_fail(target: str) -> str: + """Turn a `TARGET` argument into an API group, or exit 2 with the valid set. + + A group owned by a product must be named through it (`docstudio.platform`), + so a setting always says which product it configures. Both separators work, + because `docstudio platform` is the natural thing to type after + `unstract docstudio platform ...`. + """ + if (group := resolve_target(target)) is not None: + return group + + ctx = click.get_current_context() + CLIError( + f"Unknown config target {target!r}.", + ExitCode.USAGE, + hint=( + "Valid targets: " + ", ".join(TARGET_NAMES) + ". " + "Groups owned by a product are addressed through it, e.g. " + "`docstudio.platform` (or `docstudio platform`)." + ), + ).emit() + ctx.exit(int(ExitCode.USAGE)) + raise AssertionError # pragma: no cover - ctx.exit raises + + +def _fmt(output: str | None) -> OutputFormat: + return OutputFormat(output) if output else default_format() + + +def _output_option(f): + return click.option( + "--output", "-o", "output", + type=click.Choice([f.value for f in OutputFormat]), default=None, + help="Output format. Defaults to json when stdout is not a TTY.", + )(f) + + +@click.group(name="config", help="Manage CLI configuration profiles (local only).") +def config_group() -> None: + """Local configuration management. These commands make no network calls.""" + + +@config_group.command("init", help="Create a starter config file with profile stubs.") +@click.option("--force", is_flag=True, default=False, + help="Overwrite an existing config file.") +@_output_option +def config_init(force: bool, output: str | None) -> None: + ctx = click.get_current_context() + path = config_path() + + if path.exists() and not force: + # Never prompt: state the situation and the exact flag that resolves it. + CLIError( + f"Config already exists at {path}.", + ExitCode.USAGE, + hint="Pass --force to overwrite it, or edit the file directly.", + ).emit() + ctx.exit(int(ExitCode.USAGE)) + + cfg = load_config(path) if path.exists() else None + from unstract_cli.config.loader import ConfigFile + + new = ConfigFile( + default_profile="cloud-us", + profiles=starter_profiles(), + path=path, + exists=True, + ) + written = save_config(new, path) + emit( + { + "created": str(written), + "default_profile": "cloud-us", + "profiles": sorted(new.profiles), + "note": ( + "Credentials use env: indirection, so this file holds no secrets. " + "Set the referenced environment variables to authenticate." + ), + "replaced_existing": bool(cfg), + }, + _fmt(output), + ) + + +@config_group.command("list", help="List profiles defined in the config file.") +@_output_option +def config_list(output: str | None) -> None: + cfg = load_config() + profiles = { + name: {product: sorted(block) for product, block in blocks.items()} + for name, blocks in cfg.profiles.items() + } + emit( + { + "path": str(cfg.path), + "exists": cfg.exists, + "default_profile": cfg.default_profile, + "profiles": profiles, + }, + _fmt(output), + ) + + +@config_group.command("get") +@click.argument("target", nargs=-1, required=True, metavar="TARGET... KEY") +@click.option("--profile", "-p", default=None, help="Profile to read from.") +@_output_option +def config_get(target: tuple[str, ...], profile: str | None, output: str | None) -> None: + """Show a resolved setting, following flag > env > profile > default. + + TARGET and KEY are positional -- not flags. Credentials are reported as + configured or not, never echoed. + + \b + Examples: + unstract config get docstudio.platform org_id + unstract config get docstudio platform org_id + unstract config get --profile cloud-eu llmwhisperer base_url + """ + ctx = click.get_current_context() + *target_parts, key = target + if not target_parts: + CLIError( + "Expected a target and a key, e.g. `config get docstudio.platform org_id`.", + ExitCode.USAGE, + hint="Valid targets: " + ", ".join(TARGET_NAMES) + ".", + ).emit() + ctx.exit(int(ExitCode.USAGE)) + product = _resolve_or_fail(" ".join(target_parts)) + try: + resolved = ResolvedConfig(file=load_config(), profile_name=profile) + value = resolved.get(product, key) + except ConfigError as exc: + CLIError(str(exc), ExitCode.USAGE).emit() + ctx.exit(int(ExitCode.USAGE)) + return + + # Credentials are never echoed, even on explicit request: this output is as + # likely to land in a log or a transcript as on a screen. + is_secret = any(h in key for h in ("key", "token", "secret")) + emit( + { + "product": product, + "key": key, + "value": ("***SET***" if value else None) if is_secret else value, + "configured": value is not None, + }, + _fmt(output), + ) + + +@config_group.command("set") +@click.argument("args", nargs=-1, required=True, metavar="TARGET... KEY VALUE") +@click.option("--profile", "-p", default=None, help="Profile to write to.") +@_output_option +def config_set(args: tuple[str, ...], profile: str | None, output: str | None) -> None: + """Set a value in the config file. + + TARGET, KEY and VALUE are positional -- not flags. Writes to the active + profile unless --profile names another. + + \b + Examples: + unstract config set docstudio.platform org_id org_ABC123 + unstract config set docstudio platform org_id org_ABC123 + unstract config set llmwhisperer api_key 'env:LLMWHISPERER_API_KEY' + unstract config set --profile cloud-eu llmwhisperer base_url https://…/api/v2 + + \b + Prefer `env:VAR_NAME` for credentials: the file then records where the + secret lives rather than the secret itself, and a literal value also lands + in your shell history. + """ + ctx = click.get_current_context() + if len(args) < 3: + CLIError( + "Expected a target, a key and a value, e.g. " + "`config set docstudio.platform org_id org_ABC123`.", + ExitCode.USAGE, + hint="Valid targets: " + ", ".join(TARGET_NAMES) + ".", + ).emit() + ctx.exit(int(ExitCode.USAGE)) + + *target_parts, key, value = args + product = _resolve_or_fail(" ".join(target_parts)) + cfg = load_config() + name = profile or cfg.default_profile or "cloud-us" + + # Write to the same nested location the reader looks in + # ([profiles.X.docstudio.platform]), or a flat block would be created that + # `config get` then silently ignores. + node: dict[str, Any] = cfg.profiles.setdefault(name, {}) + for segment in GROUP_PATH.get(product, (product,)): + node = node.setdefault(segment, {}) + node[key] = value + if not cfg.default_profile: + cfg.default_profile = name + written = save_config(cfg) + + warning = None + if any(h in key for h in ("key", "token", "secret")) and not value.startswith("env:"): + warning = ( + "Value stored literally. Prefer `env:VAR_NAME` so the config file " + "holds a reference rather than the secret itself." + ) + + emit( + {"profile": name, "product": product, "key": key, "path": str(written), + "warning": warning}, + _fmt(output), + ) + ctx.exit(int(ExitCode.SUCCESS)) + + +@config_group.command("use", help="Set the default profile.") +@click.argument("name") +@_output_option +def config_use(name: str, output: str | None) -> None: + ctx = click.get_current_context() + cfg = load_config() + + if cfg.profiles and name not in cfg.profiles: + known = ", ".join(sorted(cfg.profiles)) or "none" + CLIError( + f"Profile {name!r} is not defined.", + ExitCode.USAGE, + hint=f"Known profiles: {known}. Create one with `unstract config set`.", + ).emit() + ctx.exit(int(ExitCode.USAGE)) + + cfg.default_profile = name + written = save_config(cfg) + emit({"default_profile": name, "path": str(written)}, _fmt(output)) + + +@config_group.command("current", help="Show the active profile and its resolved settings.") +@click.option("--profile", "-p", default=None, help="Profile to inspect.") +@_output_option +def config_current(profile: str | None, output: str | None) -> None: + resolved = ResolvedConfig(file=load_config(), profile_name=profile) + settings: dict[str, Any] = {} + + # Keyed by the same target names `config get`/`set` accept, so what is + # reported here can be copied straight into a command. + for target, group in ((".".join(p), g) for g, p in GROUP_PATH.items()): + block: dict[str, Any] = {} + for key in ("base_url", "org_id"): + try: + if (value := resolved.get(group, key)) is not None: + block[key] = value + except ConfigError: + continue + try: + block["api_key_configured"] = bool(resolved.get(group, "api_key")) + except ConfigError: + block["api_key_configured"] = False + settings[target] = block + + emit( + { + "active_profile": resolved.active_profile, + "config_path": str(resolved.file.path), + "config_exists": resolved.file.exists, + "settings": settings, + }, + _fmt(output), + ) + + +@config_group.command("path", help="Show the config file path.") +@_output_option +def config_path_cmd(output: str | None) -> None: + path = config_path() + emit({"path": str(path), "exists": path.exists()}, _fmt(output)) + + +#: A cheap authenticated GET per API group, used by `config doctor --check` to +#: prove the resolved credential actually works. Kept read-only and side-effect +#: free. Groups without a safe no-arg probe are reported as source-only. +_DOCTOR_PROBES: dict[str, str] = { + "docstudio.platform": "docstudio.platform.prompt-studio.select-choices", + "llmwhisperer": "whisper.usage", +} + + +@config_group.command( + "doctor", help="Diagnose credential resolution and (optionally) test it live." +) +@click.option("--profile", "-p", default=None, help="Profile to diagnose.") +@click.option("--check/--no-check", default=True, + help="Make one authenticated call per configured group to confirm it works.") +@_output_option +def config_doctor(profile: str | None, check: bool, output: str | None) -> None: + """Report where each credential resolves from, and whether it authenticates. + + Answers the question that costs the most time: the CLI reports a key as "not + configured", but you set it -- where is it looking? For `env:` refs it says + whether the variable is present in THIS process (a shell `export` in a login + profile the CLI did not inherit is the classic trap), and with --check it + makes one real call so a working credential reads as working. + """ + from unstract_cli.core import http + from unstract_cli.endpoints import get_endpoint + + resolved = ResolvedConfig(file=load_config(), profile_name=profile) + groups: list[dict[str, Any]] = [] + + for target, group in ((".".join(p), g) for g, p in GROUP_PATH.items()): + entry: dict[str, Any] = {"target": target} + for key in ("base_url", "api_key", "org_id"): + try: + entry[key] = resolved.resolution_source(group, key) + except ConfigError: + entry[key] = {"resolved": False, "source": "unset"} + + if check and entry["api_key"]["resolved"] and (probe := _DOCTOR_PROBES.get(target)): + entry["live_check"] = _probe(http, get_endpoint(probe), resolved) + groups.append(entry) + + emit( + { + "active_profile": resolved.active_profile, + "config_path": str(resolved.file.path), + "config_exists": resolved.file.exists, + "groups": groups, + }, + _fmt(output), + ) + + +def _probe(http: Any, endpoint: Any, resolved: ResolvedConfig) -> dict[str, Any]: + """Run one authenticated GET and report only pass/fail, never the payload.""" + try: + plan = http.build_request(endpoint, resolved, {}) + response = http.execute(plan, endpoint=endpoint, max_retries=0) + except Exception as exc: # noqa: BLE001 - doctor must never itself crash + return {"ok": False, "detail": str(exc)} + ok = response.status < 400 + detail = "authenticated" if ok else f"HTTP {response.status}" + if response.status in (401, 403): + detail = f"HTTP {response.status}: credential rejected" + return {"ok": ok, "detail": detail} + + +__all__ = ["config_group"] diff --git a/src/unstract_cli/config/__init__.py b/src/unstract_cli/config/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/src/unstract_cli/config/loader.py b/src/unstract_cli/config/loader.py new file mode 100644 index 0000000..208ad34 --- /dev/null +++ b/src/unstract_cli/config/loader.py @@ -0,0 +1,415 @@ +"""Profile-based configuration (SPEC.md §4). + +Three products, three incompatible auth schemes, region-specific and on-prem +hosts, and `org_id` as a URL *path segment* rather than a flag. Named profiles +(kubectl/aws style) hold per-product host, key and org. + +The resolution chain -- **flag > env > profile > built-in default** -- is +implemented once here and used by every parameter. It is never re-implemented +per command. + +The CLI is fully usable with **no config file at all**, driven entirely by +environment variables; that is the expected mode in CI and agent sandboxes. +""" + +from __future__ import annotations + +import os +import stat +import tomllib +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any + +import tomli_w + +from unstract_cli.core.model import ApiGroup, Product + +#: Config blocks are nested by product, then by API group: +#: +#: [profiles.cloud-us.docstudio.platform] +#: [profiles.cloud-us.llmwhisperer] +#: +#: Document Studio owns three API groups with distinct hosts and keys, so they +#: keep separate sub-blocks rather than being flattened into one. +GROUP_PATH: dict[str, tuple[str, ...]] = { + ApiGroup.PLATFORM.value: ("docstudio", "platform"), + ApiGroup.DEPLOYMENT.value: ("docstudio", "deployment"), + ApiGroup.HITL.value: ("docstudio", "hitl"), + ApiGroup.LLMWHISPERER.value: ("llmwhisperer",), + ApiGroup.APIHUB.value: ("apihub",), +} + +#: The names `config get`/`config set` accept, derived from GROUP_PATH so the +#: command line and the file layout can never disagree. A group owned by a +#: product must be addressed through it -- `docstudio.platform`, never a bare +#: `platform` -- because the bare form hides which product a setting belongs to. +TARGET_NAMES: tuple[str, ...] = tuple( + ".".join(path) for path in GROUP_PATH.values() +) + + +def resolve_target(name: str) -> str | None: + """Map a user-supplied target onto its API group. + + Accepts either separator, so `docstudio.platform` and `docstudio platform` + are the same thing -- the latter is what a shell user reaches for after + typing `unstract docstudio platform ...`. + """ + wanted = tuple(name.replace(".", " ").split()) + for group, path in GROUP_PATH.items(): + if wanted == path: + return group + return None + +#: Built-in defaults, lowest precedence. API Hub deliberately has **no** default +#: base URL: its public hostname is unconfirmed (SPEC.md §11.1), and inventing +#: one would send an agent's documents to a host we cannot vouch for. +DEFAULT_BASE_URLS: dict[str, str] = { + ApiGroup.LLMWHISPERER.value: "https://llmwhisperer-api.us-central.unstract.com/api/v2", + ApiGroup.PLATFORM.value: "https://us-central.unstract.com", + ApiGroup.DEPLOYMENT.value: "https://us-central.unstract.com", + ApiGroup.HITL.value: "https://us-central.unstract.com", +} + +#: Environment variables per (product, setting), checked before the config file. +ENV_VARS: dict[tuple[str, str], tuple[str, ...]] = { + (ApiGroup.LLMWHISPERER.value, "api_key"): ("LLMWHISPERER_API_KEY",), + (ApiGroup.LLMWHISPERER.value, "base_url"): ("LLMWHISPERER_BASE_URL",), + (ApiGroup.PLATFORM.value, "api_key"): ("UNSTRACT_PLATFORM_KEY",), + (ApiGroup.PLATFORM.value, "base_url"): ("UNSTRACT_BASE_URL",), + (ApiGroup.PLATFORM.value, "org_id"): ("UNSTRACT_ORG_ID",), + (ApiGroup.DEPLOYMENT.value, "api_key"): ("UNSTRACT_DEPLOYMENT_KEY",), + (ApiGroup.DEPLOYMENT.value, "base_url"): ("UNSTRACT_BASE_URL",), + (ApiGroup.DEPLOYMENT.value, "org_id"): ("UNSTRACT_ORG_ID",), + (ApiGroup.HITL.value, "api_key"): ("UNSTRACT_DEPLOYMENT_KEY", "UNSTRACT_PLATFORM_KEY"), + (ApiGroup.HITL.value, "base_url"): ("UNSTRACT_BASE_URL",), + (ApiGroup.HITL.value, "org_id"): ("UNSTRACT_ORG_ID",), + (ApiGroup.APIHUB.value, "api_key"): ("UNSTRACT_APIHUB_KEY",), + (ApiGroup.APIHUB.value, "base_url"): ("UNSTRACT_APIHUB_BASE_URL",), + (ApiGroup.APIHUB.value, "anthropic_key"): ("UNSTRACT_ANTHROPIC_API_KEY",), + (ApiGroup.APIHUB.value, "llmwhisperer_key"): ("LLMWHISPERER_API_KEY",), +} + + +class ConfigError(Exception): + """Configuration could not be loaded or resolved.""" + + +#: Set by the root `--config` flag. Highest precedence, matching the +#: flag > env > file ordering used for every other setting (SPEC.md §4.1). +_config_override: Path | None = None + + +def set_config_path(path: str | Path | None) -> None: + """Point this process at a specific config file (the `--config` flag).""" + global _config_override + _config_override = Path(path).expanduser() if path else None + + +def config_path() -> Path: + """Location of the config file. + + Resolution: ``--config`` flag, then ``$UNSTRACT_CONFIG``, then + ``$XDG_CONFIG_HOME/unstract/config.toml``, then + ``~/.config/unstract/config.toml``. + + Several config files are expected, not exceptional: a per-project file + checked into a repo, a throwaway one in CI, and a personal default can all + coexist, selected per invocation. + """ + if _config_override is not None: + return _config_override + if override := os.environ.get("UNSTRACT_CONFIG"): + return Path(override).expanduser() + if local := find_project_config(): + return local + xdg = os.environ.get("XDG_CONFIG_HOME") + base = Path(xdg).expanduser() if xdg else Path.home() / ".config" + return base / "unstract" / "config.toml" + + +#: Filename a project can commit to point the CLI at its own settings. +PROJECT_CONFIG_NAME = ".unstract.toml" + + +def find_project_config(start: Path | None = None) -> Path | None: + """Search upward from the working directory for ``.unstract.toml``. + + Mirrors how git, ruff and similar tools resolve project settings: running the + CLI inside a project picks up that project's config without any flag. The + search stops at the filesystem root, and at ``$HOME`` so a stray file in a + parent directory cannot silently capture every invocation. + """ + current = (start or Path.cwd()).resolve() + home = Path.home().resolve() + for directory in (current, *current.parents): + candidate = directory / PROJECT_CONFIG_NAME + if candidate.is_file(): + return candidate + if directory == home: + break + return None + + +def _deref(value: Any) -> Any: + """Resolve ``env:VAR_NAME`` indirection so config files hold no secrets. + + An unset variable resolves to ``None`` rather than the literal string, so a + missing credential surfaces as "not configured" instead of being sent as the + nonsense value ``"env:FOO"``. + """ + if isinstance(value, str) and value.startswith("env:"): + return os.environ.get(value[4:].strip()) or None + return value + + +@dataclass +class ConfigFile: + """Parsed contents of ``config.toml``.""" + + default_profile: str | None = None + profiles: dict[str, dict[str, dict[str, Any]]] = field(default_factory=dict) + path: Path | None = None + exists: bool = False + #: Non-fatal diagnostics (e.g. loose file permissions), surfaced on stderr. + warnings: tuple[str, ...] = () + + +def load_config(path: Path | None = None) -> ConfigFile: + """Load the config file. A missing file is normal, not an error.""" + target = path or config_path() + if not target.exists(): + return ConfigFile(path=target, exists=False) + + try: + with target.open("rb") as fh: + raw = tomllib.load(fh) + except (OSError, tomllib.TOMLDecodeError) as exc: + raise ConfigError(f"Could not read config at {target}: {exc}") from exc + + warnings: list[str] = [] + try: + mode = target.stat().st_mode + if mode & (stat.S_IRWXG | stat.S_IRWXO): + warnings.append( + f"Config file {target} is readable by other users " + f"(mode {stat.filemode(mode)}); consider `chmod 600`." + ) + except OSError: # pragma: no cover - stat failure is not worth failing on + pass + + profiles = raw.get("profiles", {}) + if not isinstance(profiles, dict): + raise ConfigError(f"`profiles` in {target} must be a table.") + + return ConfigFile( + default_profile=raw.get("default_profile"), + profiles=profiles, + path=target, + exists=True, + warnings=tuple(warnings), + ) + + +def save_config(cfg: ConfigFile, path: Path | None = None) -> Path: + """Write the config file with owner-only permissions.""" + target = path or cfg.path or config_path() + target.parent.mkdir(parents=True, exist_ok=True) + + doc: dict[str, Any] = {} + if cfg.default_profile: + doc["default_profile"] = cfg.default_profile + doc["profiles"] = cfg.profiles + + # Create with 0600 from the outset rather than widening then narrowing: + # a world-readable window, however brief, is a window. + fd = os.open(target, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600) + with os.fdopen(fd, "wb") as fh: + tomli_w.dump(doc, fh) + os.chmod(target, 0o600) + return target + + +@dataclass +class ResolvedConfig: + """Effective settings for one invocation. + + ``overrides`` holds command-line flags, which outrank everything else. + """ + + file: ConfigFile + profile_name: str | None = None + overrides: dict[str, Any] = field(default_factory=dict) + + @property + def active_profile(self) -> str | None: + """Profile selected by flag, ``UNSTRACT_PROFILE``, or the file default.""" + return ( + self.profile_name + or os.environ.get("UNSTRACT_PROFILE") + or self.file.default_profile + ) + + def _profile_block(self, product: str) -> dict[str, Any]: + name = self.active_profile + if not name: + return {} + profile = self.file.profiles.get(name) + if profile is None: + if self.file.exists and self.file.profiles: + known = ", ".join(sorted(self.file.profiles)) or "none" + raise ConfigError( + f"Profile {name!r} not found in {self.file.path}. Known profiles: {known}" + ) + return {} + # Exactly one accepted shape: the API group nested under its product, + # e.g. [profiles.X.docstudio.platform]. LLMWhisperer and API Hub own a + # single group each, so their block sits directly under the product name. + # + # No aliases and no flat fallback. A block written any other way is not + # silently picked up -- a config that looks applied but is not is worse + # than one that plainly is not, because the failure surfaces later as a + # missing-credential error with no obvious cause. + node: Any = profile + for segment in GROUP_PATH.get(product, (product,)): + node = node.get(segment) if isinstance(node, dict) else None + if node is None: + return {} + return node if isinstance(node, dict) else {} + + def get(self, product: str | ApiGroup | Product, key: str, default: Any = None) -> Any: + """Resolve one setting: **flag > env > profile > built-in default**.""" + product = product.value if isinstance(product, (ApiGroup, Product)) else product + + if (value := self.overrides.get(f"{product}.{key}")) is not None: + return value + if (value := self.overrides.get(key)) is not None: + return value + + for env_var in ENV_VARS.get((product, key), ()): + if value := os.environ.get(env_var): + return value + + if (value := _deref(self._profile_block(product).get(key))) is not None: + return value + + if default is not None: + return default + if key == "base_url": + return DEFAULT_BASE_URLS.get(product) + return None + + def require(self, product: str | ApiGroup | Product, key: str) -> Any: + """Resolve a setting, or raise a message naming exactly how to supply it.""" + if (value := self.get(product, key)) is not None: + return value + + product = product.value if isinstance(product, (ApiGroup, Product)) else product + hints: list[str] = [] + if env_vars := ENV_VARS.get((product, key)): + hints.append(f"set ${env_vars[0]}") + block = ".".join(GROUP_PATH.get(product, (product,))) + hints.append(f"or add `{key}` to the [profiles..{block}] block") + # Only suggest a flag that actually exists. Credentials have no flag by + # design -- a secret on the command line lands in shell history and + # process listings. + if key != "api_key": + hints.append(f"or pass --{key.replace('_', '-')}") + raise ConfigError( + f"Missing required setting {product}.{key}. To fix: {'; '.join(hints)}." + ) + + def resolution_source(self, product: str | ApiGroup | Product, key: str) -> dict[str, Any]: + """Report where a setting resolves from, without echoing a secret. + + `config doctor` uses this to answer the question that costs the most time: + "the CLI says the key is not configured, but I set it -- where is it + looking?" It reports the winning source (override / env var / profile + literal / profile env: ref / unset) so the mismatch is obvious. + """ + name = product.value if isinstance(product, (ApiGroup, Product)) else product + + if self.overrides.get(f"{name}.{key}") is not None or self.overrides.get(key) is not None: + return {"resolved": True, "source": "flag/override"} + + for env_var in ENV_VARS.get((name, key), ()): + if os.environ.get(env_var): + return {"resolved": True, "source": f"env:{env_var}"} + + raw = self._profile_block(name).get(key) + if isinstance(raw, str) and raw.startswith("env:"): + var = raw[4:].strip() + present = bool(os.environ.get(var)) + return { + "resolved": present, + "source": f"profile -> env:{var}", + "detail": None if present + else f"${var} is not set in this process's environment", + } + if raw not in (None, ""): + return {"resolved": True, "source": "profile (literal)"} + + if key == "base_url" and DEFAULT_BASE_URLS.get(name): + return {"resolved": True, "source": "built-in default"} + return {"resolved": False, "source": "unset"} + + +def starter_profiles() -> dict[str, dict[str, dict[str, Any]]]: + """Profile stubs written by `config init`. + + Every credential uses ``env:`` indirection: the generated file is a map of + where secrets live, never a copy of them. + """ + return { + "cloud-us": { + # Document Studio -- one product, three API groups with distinct + # hosts and credentials. + "docstudio": { + "platform": { + "base_url": DEFAULT_BASE_URLS[ApiGroup.PLATFORM.value], + "org_id": "", + "api_key": "env:UNSTRACT_PLATFORM_KEY", + }, + "deployment": { + "base_url": DEFAULT_BASE_URLS[ApiGroup.DEPLOYMENT.value], + "org_id": "", + "api_key": "env:UNSTRACT_DEPLOYMENT_KEY", + }, + "hitl": { + "base_url": DEFAULT_BASE_URLS[ApiGroup.HITL.value], + "org_id": "", + "api_key": "env:UNSTRACT_DEPLOYMENT_KEY", + }, + }, + "llmwhisperer": { + "base_url": DEFAULT_BASE_URLS[ApiGroup.LLMWHISPERER.value], + "api_key": "env:LLMWHISPERER_API_KEY", + }, + "apihub": {"base_url": "", "api_key": "env:UNSTRACT_APIHUB_KEY"}, + }, + "cloud-eu": { + "llmwhisperer": { + "base_url": "https://llmwhisperer-api.eu-west.unstract.com/api/v2", + "api_key": "env:LLMWHISPERER_API_KEY", + }, + }, + } + + +__all__ = [ + "DEFAULT_BASE_URLS", + "ENV_VARS", + "PROJECT_CONFIG_NAME", + "TARGET_NAMES", + "resolve_target", + "find_project_config", + "set_config_path", + "ConfigError", + "ConfigFile", + "ResolvedConfig", + "config_path", + "load_config", + "save_config", + "starter_profiles", +] diff --git a/src/unstract_cli/core/__init__.py b/src/unstract_cli/core/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/src/unstract_cli/core/errors.py b/src/unstract_cli/core/errors.py new file mode 100644 index 0000000..3a1cd89 --- /dev/null +++ b/src/unstract_cli/core/errors.py @@ -0,0 +1,293 @@ +"""Exit codes, structured errors, and secret redaction (SPEC.md §5.4, §5.5, §5.7). + +Exit codes are a stable API: an agent branches on them without parsing prose. +Every failure also emits a JSON object on **stderr**, carrying `hint` and +`retryable` so an agent can self-correct rather than retry blindly. +""" + +from __future__ import annotations + +import json +import re +import sys +from dataclasses import dataclass, field +from enum import IntEnum +from typing import Any + + +class ExitCode(IntEnum): + """Stable exit codes (SPEC.md §5.4).""" + + SUCCESS = 0 + GENERIC = 1 + USAGE = 2 + AUTH = 3 + NOT_FOUND = 4 + VALIDATION = 5 + RATE_LIMITED = 6 + TIMEOUT = 7 + SERVER_ERROR = 8 + ALREADY_CONSUMED = 9 + + +#: HTTP status -> exit code. 422 maps to VALIDATION, which is correct for a real +#: validation failure; the deployment API's misuse of 422 for in-progress states +#: is handled by the poller *before* reaching here, by branching on the response +#: body rather than the status code (SPEC.md §6.2). +_STATUS_MAP: dict[int, ExitCode] = { + 400: ExitCode.VALIDATION, + 401: ExitCode.AUTH, + 403: ExitCode.AUTH, + 404: ExitCode.NOT_FOUND, + 406: ExitCode.ALREADY_CONSUMED, + 408: ExitCode.TIMEOUT, + 409: ExitCode.VALIDATION, + 422: ExitCode.VALIDATION, + 429: ExitCode.RATE_LIMITED, +} + +_ERROR_CODES: dict[ExitCode, str] = { + ExitCode.GENERIC: "error", + ExitCode.USAGE: "usage_error", + ExitCode.AUTH: "auth_error", + ExitCode.NOT_FOUND: "not_found", + ExitCode.VALIDATION: "validation_error", + ExitCode.RATE_LIMITED: "rate_limited", + ExitCode.TIMEOUT: "timeout", + ExitCode.SERVER_ERROR: "server_error", + ExitCode.ALREADY_CONSUMED: "already_consumed", +} + + +def exit_code_for_status(status: int) -> ExitCode: + """Map an HTTP status onto its exit code.""" + if code := _STATUS_MAP.get(status): + return code + if 500 <= status < 600: + return ExitCode.SERVER_ERROR + if 400 <= status < 500: + return ExitCode.GENERIC + return ExitCode.SUCCESS + + +def is_retryable(status: int) -> bool: + """Retry only on rate limiting and server faults -- never on 4xx. + + Retrying a 4xx re-sends a request the server has already rejected on its + merits. Worse, for one-shot reads a blind retry can consume a result that + the first attempt already delivered (SPEC.md §5.6). + """ + return status == 429 or 500 <= status < 600 + + +# --------------------------------------------------------------------------- # +# Redaction +# --------------------------------------------------------------------------- # + +_SECRET_HEADERS = {"unstract-key", "authorization", "apikey"} +_SECRET_HEADER_PREFIXES = ("x-", ) +_SECRET_KEY_HINTS = ("key", "token", "secret", "password", "credential", "auth") +REDACTED = "***REDACTED***" + + +def redact_headers(headers: dict[str, Any]) -> dict[str, Any]: + """Redact credential-bearing headers.""" + out: dict[str, Any] = {} + for key, value in headers.items(): + low = key.lower() + secret = low in _SECRET_HEADERS or ( + low.startswith(_SECRET_HEADER_PREFIXES) + and any(h in low for h in _SECRET_KEY_HINTS) + ) + out[key] = REDACTED if secret else value + return out + + +def redact_value(value: Any) -> Any: + """Recursively redact secret-looking keys in a payload.""" + if isinstance(value, dict): + return { + k: ( + REDACTED + if any(h in str(k).lower() for h in _SECRET_KEY_HINTS) + and isinstance(v, str) + else redact_value(v) + ) + for k, v in value.items() + } + if isinstance(value, list): + return [redact_value(v) for v in value] + return value + + +def scrub(text: str, secrets: list[str]) -> str: + """Remove known secret literals from free text. + + The last line of defence: a credential that reaches a message body via an + upstream error string still must not be printed. + """ + for secret in secrets: + if secret and len(secret) >= 8: + text = text.replace(secret, REDACTED) + text = re.sub(re.escape(secret), REDACTED, text) + return text + + +# --------------------------------------------------------------------------- # +# CLIError +# --------------------------------------------------------------------------- # + + +@dataclass +class CLIError(Exception): + """A failure that maps onto an exit code and a structured stderr payload.""" + + message: str + exit_code: ExitCode = ExitCode.GENERIC + http_status: int | None = None + details: Any = None + endpoint: str | None = None + hint: str | None = None + retryable: bool = False + code: str | None = None + extra: dict[str, Any] = field(default_factory=dict) + + def __post_init__(self) -> None: + super().__init__(self.message) + + def to_dict(self) -> dict[str, Any]: + payload: dict[str, Any] = { + "code": self.code or _ERROR_CODES.get(self.exit_code, "error"), + "message": self.message, + "exit_code": int(self.exit_code), + "retryable": self.retryable, + } + if self.http_status is not None: + payload["http_status"] = self.http_status + if self.details is not None: + payload["details"] = self.details + if self.endpoint: + payload["endpoint"] = self.endpoint + if self.hint: + payload["hint"] = self.hint + payload.update(self.extra) + return {"error": payload} + + def emit(self, secrets: list[str] | None = None) -> None: + """Write the structured error to stderr, and to stdout when piped. + + Humans always get it on stderr. When stdout is not a TTY -- the agent / + wrapper case -- the same envelope is *also* written to stdout, so a + pipeline that feeds stdout to a JSON parser sees a valid object instead of + an empty stream (DOC 9). A result and an error never share an invocation's + stdout: emit runs only on the error path, which produces no result output. + """ + text = json.dumps(self.to_dict(), indent=2, default=str) + if secrets: + text = scrub(text, secrets) + print(text, file=sys.stderr) + if not sys.stdout.isatty(): + print(text, file=sys.stdout) + + def emit_stdout_only(self, secrets: list[str] | None = None) -> None: + """Write the envelope to stdout when piped, leaving stderr untouched. + + For Click's own usage errors, whose human message Click has already + printed to stderr: this adds only the machine-readable copy on stdout + (DOC 9), without duplicating the human line. + """ + if sys.stdout.isatty(): + return + text = json.dumps(self.to_dict(), indent=2, default=str) + if secrets: + text = scrub(text, secrets) + print(text, file=sys.stdout) + + +def hint_for(status: int, endpoint: str | None = None, message: str | None = None) -> str | None: + """A short, actionable next step for common failures. + + ``message`` lets a hint target a specific server error where the status code + alone is ambiguous -- an adapter-permission 403 and a dead-vector-DB 500 both + have a much more useful next step than the generic status hint. + """ + low = (message or "").lower() + + # Adapter permission (IMPROVEMENT 4): the API key identity is distinct from the + # human account of the same name, so an adapter can be "available" yet unusable. + if "permission error" in low and "adapter" in low: + return ( + "One or more adapters are not shared with your API key. The API user " + "(e.g. *-api-rw-*@platform.internal) is a DIFFERENT identity from your " + "human account. Share the adapters to the org in the web console, or set " + "a default triad, then retry. `adapter list` shows created_by_email." + ) + # The message names the *project* default, but the server never reads it: it + # resolves the profile from the prompt's own profile_manager FK. Chasing the + # project default is the wrong fix and cost real time (GOTCHAS #1). + if "default llm profile is not configured" in low: + return ( + "Misleading message: the server resolves the LLM profile from the " + "PROMPT's own profile_manager field and does NOT fall back to the " + "project default -- so `profile set-default` cannot fix this. Set it " + "on the prompt: `prompt patch --prompt-id --profile-manager " + "` (or pass --profile-manager on this call). Create " + "prompts with --profile-manager to avoid it entirely." + ) + # Deploy-time tool validation (GOTCHAS #2). The 422 surfaces only at + # `deployment run`, long after the tool instance was attached. + if "tool validation failed" in low or ("challenge_llm" in low or "challenge llm" in low): + return ( + "The attached tool instance likely has an empty metadata.challenge_llm, " + "which fails validation even when enable_challenge is false. Fix: " + "`workflow tool get --id ` to read the CURRENT metadata, set " + "challenge_llm to a valid LLM adapter id, and send the COMPLETE object " + "back with `workflow tool set-metadata` (it replaces, never merges). " + "To avoid it next time, set the project's --challenge-llm before " + "`export-tool`." + ) + # Dead vector DB (IMPROVEMENT 6): short docs don't need RAG at all. + if "vectordb" in low or "vector db" in low or "qdrant" in low: + return ( + "The vector DB is unreachable. For short documents you can skip RAG " + "entirely: set the profile's chunk_size=0 (and chunk_overlap=0), which " + "bypasses embedding and the vector DB, then re-index." + ) + + match status: + case 401 | 403: + return ( + "Check the API key for this product. Keys are per-product: see " + "`unstract config list` and SPEC §4.4 for which env var applies." + ) + case 404: + return ( + "Verify the resource id and that --org-id matches the resource's " + "organization. For deployments, confirm --api-name is correct." + ) + case 406: + return ( + "This result was already retrieved. Results can be read exactly " + "once; re-running the request cannot recover them. Use --save next " + "time to persist on first read." + ) + case 409: + return "The resource is in use or conflicts with an existing one." + case 429: + return "Rate limited. The CLI retries automatically; raise --max-retries if needed." + if 500 <= status < 600: + return "Server-side failure. Retried automatically; if it persists, check service status." + return None + + +__all__ = [ + "REDACTED", + "CLIError", + "ExitCode", + "exit_code_for_status", + "hint_for", + "is_retryable", + "redact_headers", + "redact_value", + "scrub", +] diff --git a/src/unstract_cli/core/generate.py b/src/unstract_cli/core/generate.py new file mode 100644 index 0000000..ca30a06 --- /dev/null +++ b/src/unstract_cli/core/generate.py @@ -0,0 +1,385 @@ +"""Turn `Endpoint` records into Click commands (SPEC.md D1). + +This module is the load-bearing abstraction: the command tree, every flag, all +help text, validation and `--discover` derive from the same records. Adding +an endpoint means adding one record -- there is no second place to update, so +help text cannot drift from behaviour. + +Click is used directly rather than Typer's decorator API because the parameters +are known only at runtime, from data. Typer's ergonomics are built for +statically-declared function signatures. +""" + +from __future__ import annotations + +import contextlib +import json +from pathlib import Path +from typing import Any + +import click + +from unstract_cli.config.loader import ConfigError, ResolvedConfig, load_config +from unstract_cli.core import http +from unstract_cli.core.errors import CLIError, ExitCode +from unstract_cli.core.model import ( + Endpoint, + Param, + ParamType, +) +from unstract_cli.core.output import ( + OutputFormat, + default_format, + diagnostic, + emit, +) +from unstract_cli.core.poll import wait_for_completion + +#: Flags every generated command accepts (SPEC.md §5.1, §5.7). +GLOBAL_FLAGS = ("output", "profile", "quiet", "verbose", "dry_run", "max_retries", "no_retry", "timeout", "base_url") + +_CLICK_TYPES: dict[ParamType, Any] = { + ParamType.STR: click.STRING, + ParamType.INT: click.INT, + ParamType.FLOAT: click.FLOAT, + ParamType.BOOL: click.BOOL, + ParamType.UUID: click.STRING, + ParamType.JSON: click.STRING, + ParamType.FILE: click.STRING, + ParamType.DATE: click.STRING, +} + + +def _help_text(param: Param) -> str: + """Compose help for one flag, folding in everything an agent needs. + + Defaults, enums and conditional applicability all appear here, because + `--help` is the primary discovery surface (SPEC.md §5.3). + """ + parts = [param.help.rstrip(".") if param.help else ""] + if mapping := param.choice_map(): + parts.append(f"One of: {', '.join(mapping)}") + if param.default is not None and not isinstance(param.default, bool): + parts.append(f"Default: {param.default}") + if param.applies_when: + # P9: documented, never enforced -- the server owns the rule. + parts.append(f"Applies only when {param.applies_when}") + if param.multiple: + parts.append("Repeatable") + if param.replace_semantics: + parts.append("REPLACES the existing list rather than appending") + if param.default_from: + parts.append("Defaults from the active profile") + return ". ".join(p for p in parts if p) + "." if any(parts) else "" + + +def _click_option(param: Param) -> click.Option: + """Build a Click option from a `Param` record.""" + mapping = param.choice_map() + if mapping: + ptype: Any = click.Choice(list(mapping)) + else: + ptype = _CLICK_TYPES.get(param.type, click.STRING) + + kwargs: dict[str, Any] = { + "help": _help_text(param), + "required": False, # Enforced later, so we can give a better message. + "multiple": param.multiple, + "type": ptype, + } + # A default is applied server-side too; sending it explicitly is harmless and + # keeps `--discover` honest about what the value will be. + if param.default is not None and not param.multiple: + kwargs["default"] = param.default + kwargs["show_default"] = True + + if param.type is ParamType.BOOL and not mapping: + flag = param.cli_flag.lstrip("-") + return click.Option([f"--{flag}/--no-{flag}"], default=None, help=kwargs["help"]) + + return click.Option([param.cli_flag, param.py_name], **kwargs) + + +def _resolve_config(ctx: click.Context, kwargs: dict[str, Any], endpoint: Endpoint) -> ResolvedConfig: + overrides: dict[str, Any] = {} + if base_url := kwargs.get("base_url"): + overrides[f"{endpoint.product.value}.base_url"] = base_url + if org_id := kwargs.get("org_id"): + overrides[f"{endpoint.product.value}.org_id"] = org_id + + root = ctx.find_root().params if ctx.find_root() else {} + cfg = load_config() + resolved = ResolvedConfig( + file=cfg, + profile_name=kwargs.get("profile") or root.get("profile"), + overrides=overrides, + ) + for warning in cfg.warnings: + diagnostic(f"warning: {warning}", quiet=bool(kwargs.get("quiet"))) + return resolved + + +def _save_payload(payload: Any, destination: str, raw_field: str | None) -> None: + """Persist a result to disk **atomically, before exit** (SPEC.md §5.6). + + One-shot results cannot be re-fetched, so a partial file from an interrupted + write would be indistinguishable from data loss. Write to a temp file in the + same directory, then rename. + """ + path = Path(destination).expanduser() + path.parent.mkdir(parents=True, exist_ok=True) + + if isinstance(payload, dict) and raw_field and raw_field in payload: + body = payload[raw_field] + else: + body = payload + + data = ( + body.encode() + if isinstance(body, str) + else body + if isinstance(body, bytes) + else json.dumps(body, indent=2, default=str).encode() + ) + + tmp = path.with_name(f".{path.name}.partial") + tmp.write_bytes(data) + tmp.replace(path) + + +def build_command(endpoint: Endpoint) -> click.Command: + """Generate the Click command for one endpoint.""" + options: list[click.Parameter] = [_click_option(p) for p in endpoint.params] + + if endpoint.poll: + options.append( + click.Option( + ["--wait/--no-wait"], + default=False, + help=( + "Poll until the operation reaches a terminal state, then " + "retrieve the result. Avoids scripting the poll loop." + ), + ) + ) + options.append( + click.Option( + ["--poll-interval"], type=click.FLOAT, default=3.0, show_default=True, + help="Seconds between status checks when --wait is used.", + ) + ) + options.append( + click.Option( + ["--wait-timeout"], type=click.FLOAT, default=300.0, show_default=True, + help="Give up waiting after this many seconds; exits 7 with the handle.", + ) + ) + + options.extend( + [ + click.Option( + ["--output", "-o", "output"], + type=click.Choice([f.value for f in OutputFormat]), + default=None, + help="Output format. Defaults to json when stdout is not a TTY.", + ), + click.Option(["--profile", "-p", "profile"], default=None, help="Config profile to use."), + click.Option(["--base-url"], default=None, help="Override the product base URL."), + click.Option(["--dry-run"], is_flag=True, default=False, + help="Print the resolved request as JSON and exit without sending."), + click.Option(["--quiet", "-q"], is_flag=True, default=False, help="Suppress stderr diagnostics."), + click.Option(["--verbose", "-v"], count=True, help="Increase stderr diagnostics."), + click.Option(["--max-retries"], type=click.INT, default=3, show_default=True, + help="Retries for 429/5xx responses."), + click.Option(["--no-retry"], is_flag=True, default=False, help="Disable retries entirely."), + click.Option(["--timeout"], type=click.FLOAT, default=60.0, show_default=True, + help="Per-request timeout in seconds."), + ] + ) + + def callback(**kwargs: Any) -> None: + ctx = click.get_current_context() + _run_endpoint(ctx, endpoint, kwargs) + + help_text = _command_help(endpoint) + return click.Command( + name=endpoint.name, + params=options, + callback=callback, + help=help_text, + short_help=endpoint.summary, + ) + + +def _command_help(endpoint: Endpoint) -> str: + """Full help body: summary, semantics, endpoint, constraints, examples.""" + lines = [endpoint.summary, ""] + if endpoint.description: + lines += [endpoint.description, ""] + lines.append(f"API: {endpoint.method} {endpoint.path}") + if endpoint.permission: + lines.append(f"Requires platform key permission: {endpoint.permission.value}") + if endpoint.poll and endpoint.poll.one_shot: + lines.append( + "ONE-SHOT: the result can be retrieved only once. Use --save to persist it; " + "a second attempt exits 9." + ) + for constraint in endpoint.constraints: + lines.append(f"Constraint: {constraint.describe()}") + if endpoint.examples: + lines += ["", "Examples:"] + [f" {ex}" for ex in endpoint.examples] + return "\n".join(lines) + + +def _run_endpoint(ctx: click.Context, endpoint: Endpoint, kwargs: dict[str, Any]) -> None: + """Execute one generated command end-to-end.""" + quiet = bool(kwargs.get("quiet")) + verbosity = int(kwargs.get("verbose") or 0) + fmt = OutputFormat(kwargs["output"]) if kwargs.get("output") else default_format() + + # Click always supplies `multiple` options as (), which would otherwise look + # like "explicitly set to empty" further down. + values = {k: v for k, v in kwargs.items() if not (isinstance(v, tuple) and not v)} + + try: + config = _resolve_config(ctx, values, endpoint) + plan = http.build_request(endpoint, config, values) + + if kwargs.get("dry_run"): + emit(plan.describe(), fmt if fmt is not OutputFormat.RAW else OutputFormat.JSON) + ctx.exit(int(ExitCode.SUCCESS)) + + diagnostic(f"{plan.method} {plan.url}", quiet=quiet, verbosity=verbosity, level=1) + + max_retries = 0 if kwargs.get("no_retry") else int(kwargs.get("max_retries") or 0) + response = http.execute( + plan, + endpoint=endpoint, + timeout=float(kwargs.get("timeout") or 60.0), + max_retries=max_retries, + ) + http.raise_for_status(response, endpoint) + payload = response.payload + + # A 2xx does not always mean success: some endpoints return 201 while + # leaving a linking field NULL, silently orphaning the record (BUG 2). + if endpoint.require_response_fields and isinstance(payload, dict): + missing = [ + f for f in endpoint.require_response_fields if payload.get(f) in (None, "") + ] + if missing: + raise CLIError( + f"Server accepted the request (HTTP {response.status}) but left " + f"{', '.join(missing)} empty; the record was not linked correctly.", + ExitCode.SERVER_ERROR, + endpoint=f"{endpoint.method} {endpoint.path}", + details=payload, + hint="This is a known server-side defect. Re-run, or report it upstream.", + ) + + if kwargs.get("wait") and endpoint.poll: + payload = wait_for_completion( + endpoint=endpoint, + initial=payload, + config=config, + values=values, + poll_interval=float(kwargs.get("poll_interval") or 3.0), + timeout=float(kwargs.get("wait_timeout") or 300.0), + max_retries=max_retries, + request_timeout=float(kwargs.get("timeout") or 60.0), + quiet=quiet, + verbosity=verbosity, + ) + + if destination := values.get("save"): + _save_payload(payload, str(destination), endpoint.raw_field) + diagnostic(f"saved to {destination}", quiet=quiet, verbosity=verbosity) + + emit(payload, fmt, columns=endpoint.table_columns, raw_field=endpoint.raw_field) + ctx.exit(int(ExitCode.SUCCESS)) + + except ConfigError as exc: + CLIError(str(exc), ExitCode.USAGE).emit() + ctx.exit(int(ExitCode.USAGE)) + except CLIError as exc: + secrets: list[str] = [] + # Redaction is best-effort here: if credentials cannot be resolved, that + # must not stop the underlying error from being reported. + with contextlib.suppress(Exception): + secrets = http.collect_secrets(_resolve_config(ctx, values, endpoint)) + exc.emit(secrets) + ctx.exit(int(exc.exit_code)) + + +def build_group_tree(endpoints: list[Endpoint]) -> dict[str, click.Group]: + """Assemble endpoints into nested Click groups by command path.""" + groups: dict[str, click.Group] = {} + + for endpoint in endpoints: + group_name = endpoint.group + if group_name not in groups: + groups[group_name] = click.Group( + name=group_name, help=_GROUP_HELP.get(group_name, f"{group_name} commands.") + ) + parent = groups[group_name] + + if endpoint.subgroup: + for part in endpoint.subgroup.split(): + existing = parent.commands.get(part) + if not isinstance(existing, click.Group): + existing = click.Group( + name=part, + help=_SUBGROUP_HELP.get(part, f"{part} commands."), + ) + parent.add_command(existing) + parent = existing + + parent.add_command(build_command(endpoint)) + + return groups + + +#: Top-level group help. One group per product built by Unstract. +_GROUP_HELP = { + "docstudio": ( + "Document Studio — document extraction platform. Covers the Platform " + "Management API, deployed API workflows, and Human Quality Review." + ), + "whisper": "LLMWhisperer — convert documents to LLM-ready text.", + "apihub": "API Hub — vertical extraction (tables, bank statements, doc splitting).", +} + +#: Help for the API groups nested under a product. +_SUBGROUP_HELP = { + "platform": "Platform Management API (v1) — manage Document Studio resources.", + "deployment": "Run deployed API workflows and check their status.", + "hitl": "Human Quality Review — retrieve approved results (Enterprise).", + "output": "Read extraction results from the Prompt Studio Output Manager.", + "tool": "Attach and configure the tool a workflow runs (deployment assembly).", + "endpoint": "Configure a workflow's source/destination endpoints (set to API to deploy).", + "registry": "Browse the tool registry to find a tool's function_name.", + # Resource subgroups under `docstudio platform`, so the discover groups + # overview reads as a map rather than " commands." placeholders. + "prompt-studio": "Build extraction projects: prompts, profiles, files, run extraction.", + "workflow": "Assemble and run workflows: tools, endpoints, executions, file history.", + "api-deployment": "Manage deployed API endpoints and their access keys.", + "pipeline": "Manage ETL/Task pipelines and their executions.", + "adapter": "Manage LLM, embedding, vector-DB and text-extraction adapters.", + "connector": "Manage source/destination connectors for workflows.", + "group": "Manage user groups and their members.", + "user": "List organization users.", + "execution": "Inspect workflow executions and their logs.", + "file-history": "Inspect and clear per-file processing history.", + "profile": "Manage a project's LLM profiles.", + "prompt": "Manage a project's prompts.", + "file": "Upload and fetch a project's documents.", + "key": "Manage API keys for a deployment or pipeline.", + "member": "Manage a group's members.", + "default-triad": "Get or set the org default adapter triad.", + "approved": "Retrieve approved Human Quality Review results.", + "doc-splitter": "Split documents into parts (API Hub).", + "webhook": "Manage LLMWhisperer delivery webhooks.", +} + + +__all__ = ["build_command", "build_group_tree", "GLOBAL_FLAGS"] diff --git a/src/unstract_cli/core/http.py b/src/unstract_cli/core/http.py new file mode 100644 index 0000000..bb0d38c --- /dev/null +++ b/src/unstract_cli/core/http.py @@ -0,0 +1,569 @@ +"""Request construction and execution (SPEC.md §4.4, §5.7). + +Owns auth injection, path substitution, retry/backoff, redaction and `--dry-run`. +Built once here so no individual command can forget any of it. +""" + +from __future__ import annotations + +import json +import random +import time +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any + +import httpx + +from unstract_cli.config.loader import ResolvedConfig +from unstract_cli.core.errors import ( + CLIError, + ExitCode, + exit_code_for_status, + hint_for, + is_retryable, + redact_headers, + redact_value, +) +from unstract_cli.core.model import ( + ApiGroup, + BodyKind, + Endpoint, + Param, + ParamLocation, + ParamType, +) + +#: Headers Kong injects downstream from the `apikey` lookup (SPEC.md §4.4). +#: The CLI must never send these: they are gateway-supplied, and a client-set +#: value would either be overwritten or -- worse -- be trusted as tenancy claims. +GATEWAY_INJECTED_HEADERS = frozenset( + { + "x-subscription-id", + "x-subscription-name", + "x-user-id", + "x-product-id", + } +) + +USER_AGENT = "unstract-cli/0.1.0" + + +@dataclass +class RequestPlan: + """A fully resolved request, ready to send or to print under ``--dry-run``.""" + + method: str + url: str + headers: dict[str, str] = field(default_factory=dict) + params: dict[str, Any] = field(default_factory=dict) + json_body: Any = None + data: Any = None + files: list[tuple[str, tuple[str, bytes, str]]] = field(default_factory=list) + content: bytes | None = None + #: Secret literals to scrub from any output. + secrets: list[str] = field(default_factory=list) + + def describe(self) -> dict[str, Any]: + """Redacted, JSON-safe description for ``--dry-run`` (SPEC.md §5.7).""" + body: Any = None + if self.json_body is not None: + body = redact_value(self.json_body) + elif self.files: + body = { + "multipart": [ + {"field": name, "filename": meta[0], "bytes": len(meta[1])} + for name, meta in self.files + ] + } + elif self.content is not None: + body = {"binary_bytes": len(self.content)} + elif self.data: + body = redact_value(self.data) + return { + "method": self.method, + "url": self.url, + "headers": redact_headers(self.headers), + "query": redact_value(self.params), + "body": body, + } + + +def auth_headers(product: ApiGroup, config: ResolvedConfig) -> dict[str, str]: + """Build auth headers for one API group (SPEC.md §4.4). + + Three products, three schemes -- unifying them is the CLI's core promise. + Document Studio's three API groups share the Bearer scheme but each carries + its own key, so credentials are still resolved per group. + """ + match product: + case ApiGroup.LLMWHISPERER: + return {"unstract-key": config.require(product, "api_key")} + case ApiGroup.PLATFORM | ApiGroup.DEPLOYMENT | ApiGroup.HITL: + return {"Authorization": f"Bearer {config.require(product, 'api_key')}"} + case ApiGroup.APIHUB: + # `apikey` only. Kong resolves it to subscription/user identity. + headers = {"apikey": config.require(product, "api_key")} + if key := config.get(product, "llmwhisperer_key"): + headers["X-LLMWhisperer-API-Key"] = key + if key := config.get(product, "anthropic_key"): + headers["X-Anthropic-API-Key"] = key + return headers + raise CLIError(f"Unknown API group: {product}", ExitCode.GENERIC) # pragma: no cover + + +def collect_secrets(config: ResolvedConfig) -> list[str]: + """Every credential in play, so output can be scrubbed of all of them.""" + secrets: list[str] = [] + for product in ApiGroup: + for key in ("api_key", "llmwhisperer_key", "anthropic_key"): + try: + if (value := config.get(product, key)) and isinstance(value, str): + secrets.append(value) + except Exception: # pragma: no cover - resolution issues aren't fatal here + continue + return secrets + + +def build_url(endpoint: Endpoint, config: ResolvedConfig, values: dict[str, Any]) -> str: + """Resolve the base URL and substitute path parameters. + + ``Endpoint.path`` is used **verbatim** (P11): the upstream API is genuinely + inconsistent about trailing slashes and about `profilemanager` vs + `profile-manager`, and "tidying" a path yields a 404. + """ + base = config.get(endpoint.api, "base_url") + if not base: + raise CLIError( + f"No base URL configured for {endpoint.api.value}.", + ExitCode.USAGE, + hint=( + "Set --base-url or the corresponding environment variable. " + "API Hub has no default base URL (SPEC §11.1)." + ), + ) + + path = endpoint.path + for param in endpoint.path_params(): + value = values.get(param.py_name) + if value is None: + value = _config_default(param, config) + if value is None: + raise CLIError( + f"Missing required path parameter {param.cli_flag}.", + ExitCode.USAGE, + hint=f"Pass {param.cli_flag}, or configure it in your profile.", + ) + placeholder = "{" + param.name + "}" + path = path.replace(placeholder, str(param.to_wire(value))) + + return f"{str(base).rstrip('/')}/{path.lstrip('/')}" + + +def _config_default(param: Param, config: ResolvedConfig) -> Any: + """Resolve a parameter's profile default, trying each source in order. + + ``default_from`` may name several config paths; the first that resolves wins. + That lets one setting stand in for another where they are genuinely the same + value held in two blocks -- `deployment run`'s org_id falls back to the + platform block, which is the same organization and is usually already set + (GOTCHAS #7). An empty string counts as unset, since that is what a + half-filled `config init` stub leaves behind. + """ + for source in param.default_sources: + product, _, key = source.partition(".") + if (value := config.get(product, key)) not in (None, ""): + return value + return None + + +def _guess_content_type(path: Path) -> str: + import mimetypes + + return mimetypes.guess_type(path.name)[0] or "application/octet-stream" + + +def _parse_json_param(param: Param, value: object) -> object: + """Parse a ``ParamType.JSON`` value from a string into a real object (BUG 1). + + Click hands JSON params through as plain strings, so without this the body + field would hold a quoted string (``"data": "{...}"``) and the server would + reject it as ``Expected a dictionary of items but got type "str"``. Accepts + either inline JSON or an ``@path/to/file.json`` reference -- large payloads + such as an exported prompts file are painful to pass inline and hit shell + argument limits. A value that is already parsed (a dict/list, e.g. a test + passing a native object) is returned unchanged. + """ + if param.type is not ParamType.JSON or not isinstance(value, str): + return value + text = value + if text.startswith("@"): + ref = Path(text[1:]).expanduser() + try: + text = ref.read_text() + except OSError as exc: + raise CLIError( + f"{param.cli_flag} could not read {ref}: {exc}", ExitCode.USAGE + ) from exc + try: + return json.loads(text) + except json.JSONDecodeError as exc: + raise CLIError( + f"{param.cli_flag} expects valid JSON (or @file.json): {exc}", + ExitCode.USAGE, + ) from exc + + +def build_request( + endpoint: Endpoint, + config: ResolvedConfig, + values: dict[str, Any], + *, + extra_query: dict[str, Any] | None = None, +) -> RequestPlan: + """Turn resolved flag values into a concrete request. + + Constraints are checked first: a malformed invocation should cost exit code 2, + not a network round trip and a remote rejection. + """ + if violations := endpoint.validate(values): + raise CLIError( + "; ".join(violations), + ExitCode.USAGE, + endpoint=f"{endpoint.method} {endpoint.path}", + ) + + headers = {"User-Agent": USER_AGENT, **auth_headers(endpoint.api, config)} + params: dict[str, Any] = {} + body: dict[str, Any] = {} + form: dict[str, Any] = {} + files: list[tuple[str, tuple[str, bytes, str]]] = [] + content: bytes | None = None + + for param in endpoint.params: + if param.client_side: + continue + if param.location is ParamLocation.PATH: + # A PATH param normally travels only in the URL. Mirroring + # additionally copies it into the JSON body, defending against a + # server that links a record by a body field and orphans it when the + # field is absent (BUG 2), or that simply wants the same identifier + # twice under two names (`api_id` in the URL, `api` in the body -- + # GOTCHAS #6). `body_name` is the path name unless `mirror_as` renames it. + if param.mirrors and (raw := values.get(param.py_name)) is not None: + body[param.body_name] = param.to_wire(raw) + continue + + raw = values.get(param.py_name) + + if param.freeform_prefix and raw: + # P5: `--ext-param foo=bar` -> `ext_foo=bar`, so parameters newer than + # the CLI remain reachable without a release. + for item in raw if isinstance(raw, (list, tuple)) else [raw]: + key, sep, val = str(item).partition("=") + if not sep: + raise CLIError( + f"{param.cli_flag} expects KEY=VALUE, got {item!r}", + ExitCode.USAGE, + ) + params[f"{param.freeform_prefix}{key.strip()}"] = val.strip() + continue + + if raw is None: + raw = _config_default(param, config) + if raw is None: + raw = param.default + if raw is None or (isinstance(raw, (list, tuple)) and not raw): + if param.required: + raise CLIError( + f"Missing required parameter {param.cli_flag}.", ExitCode.USAGE + ) + continue + + value = _parse_json_param(param, param.to_wire(raw)) + + match param.location.value: + case "query": + params[param.name] = value + case "body": + body[param.name] = value + case "header": + headers[param.name] = str(value) + case "form": + if param.type.value == "file": + for item in value if isinstance(value, (list, tuple)) else [value]: + p = Path(str(item)) + if not p.exists(): + raise CLIError(f"File not found: {p}", ExitCode.USAGE) + files.append( + (param.name, (p.name, p.read_bytes(), _guess_content_type(p))) + ) + elif param.type is ParamType.JSON and isinstance(value, (dict, list)): + # A multipart form field carries a JSON object as a *string*; + # the server (a DRF form field) `json.loads` it. Left as a + # dict, httpx cannot form-encode it. Re-serialize here so BODY + # gets an object and FORM gets the string it expects -- the + # double-encode guard the JSON parse would otherwise trip. + form[param.name] = json.dumps(value) + else: + form[param.name] = value + + if extra_query: + params.update(extra_query) + + if endpoint.body is BodyKind.BINARY_FILE: + # LLMWhisperer takes the document as a raw octet-stream body. + if file_value := values.get("file"): + p = Path(str(file_value)) + if not p.exists(): + raise CLIError(f"File not found: {p}", ExitCode.USAGE) + content = p.read_bytes() + headers["Content-Type"] = "application/octet-stream" + elif url_value := values.get("url"): + # `url_in_post` sends the source URL as a plain-text body instead. + content = str(url_value).encode() + headers["Content-Type"] = "text/plain" + params["url_in_post"] = "true" + + # Defence in depth: even if a record mistakenly declared one, gateway-injected + # headers must never leave this process (SPEC.md §4.4). + headers = {k: v for k, v in headers.items() if k.lower() not in GATEWAY_INJECTED_HEADERS} + + return RequestPlan( + method=endpoint.method, + url=build_url(endpoint, config, values), + headers=headers, + params={k: _stringify(v) for k, v in params.items() if v is not None}, + json_body=body if body and endpoint.body is BodyKind.JSON else None, + data=form or None, + files=files, + content=content, + secrets=collect_secrets(config), + ) + + +def _stringify(value: Any) -> Any: + """Booleans must travel as `true`/`false`, not Python's `True`/`False`.""" + if isinstance(value, bool): + return "true" if value else "false" + if isinstance(value, (list, tuple)): + return [_stringify(v) for v in value] + if isinstance(value, dict): + return json.dumps(value) + return value + + +@dataclass +class Response: + """A completed HTTP exchange.""" + + status: int + payload: Any + headers: dict[str, str] + raw: bytes + + @property + def is_json(self) -> bool: + return isinstance(self.payload, (dict, list)) + + +def _parse(response: httpx.Response) -> Any: + ctype = response.headers.get("content-type", "") + if "json" in ctype: + try: + return response.json() + except ValueError: + # Some endpoints emit several JSON objects concatenated (a streamed + # NDJSON-ish body), which a single parse rejects with "Extra data" and + # which breaks any `| json` consumer (CAPTURE2 DOC 6). Recover them into + # one array so the CLI still emits exactly one valid JSON document; fall + # back to raw text only if they are not clean concatenated JSON. + if (parts := _split_concatenated_json(response.text)) is not None: + return parts + return response.text + if ctype.startswith("text/") or not ctype: + return response.text + return response.content + + +def _split_concatenated_json(text: str) -> list[Any] | None: + """Parse a run of back-to-back JSON values into a list, or None if it isn't one. + + Returns None unless there are at least two values (a single value would have + parsed already), so a genuinely malformed body still falls through to raw text. + """ + decoder = json.JSONDecoder() + items: list[Any] = [] + idx, length = 0, len(text) + while idx < length: + while idx < length and text[idx].isspace(): + idx += 1 + if idx >= length: + break + try: + value, end = decoder.raw_decode(text, idx) + except ValueError: + return None + items.append(value) + idx = end + return items if len(items) > 1 else None + + +def execute( + plan: RequestPlan, + *, + endpoint: Endpoint | None = None, + timeout: float = 60.0, + max_retries: int = 3, + client: httpx.Client | None = None, + sleep=time.sleep, +) -> Response: + """Send a request, retrying only where retrying is safe (SPEC.md §5.7). + + Retries apply to 429 and 5xx. They never apply to 4xx: the server has already + rejected the request on its merits, and for one-shot reads a blind retry can + silently consume a result the first attempt already delivered. + """ + owns_client = client is None + # Note: following redirects means a wrong trailing slash would be silently + # "corrected" by the server rather than failing loudly, which softens the P11 + # literal-path guarantee. Redirects are kept because these APIs legitimately + # use them; the `no_trailing_slash` test is what actually protects paths. + client = client or httpx.Client(timeout=timeout, follow_redirects=True) + last_error: httpx.HTTPError | None = None + + try: + for attempt in range(max_retries + 1): + try: + response = client.request( + plan.method, + plan.url, + headers=plan.headers, + params=plan.params or None, + json=plan.json_body, + data=plan.data, + files=plan.files or None, + content=plan.content, + ) + except httpx.HTTPError as exc: + last_error = exc + if attempt >= max_retries: + break + sleep(_backoff(attempt)) + continue + + if is_retryable(response.status_code) and attempt < max_retries: + sleep(_retry_after(response) or _backoff(attempt)) + continue + + return Response( + status=response.status_code, + payload=_parse(response), + headers=dict(response.headers), + raw=response.content, + ) + finally: + if owns_client: + client.close() + + raise CLIError( + f"Request failed after {max_retries + 1} attempt(s): {last_error}", + ExitCode.SERVER_ERROR, + endpoint=f"{plan.method} {plan.url}", + retryable=True, + hint="Check network connectivity and the configured base URL.", + ) + + +def _backoff(attempt: int) -> float: + """Exponential backoff with jitter, so retries don't synchronise across agents.""" + return min(2.0**attempt, 30.0) * (0.5 + random.random() / 2) + + +def _retry_after(response: httpx.Response) -> float | None: + try: + return float(response.headers.get("retry-after", "")) + except (TypeError, ValueError): + return None + + +#: Body phrases meaning "this result was already delivered". LLMWhisperer signals +#: this with HTTP 200 and a message, not an error status, so a status-only check +#: would report success while the agent silently loses the data (SPEC.md §5.6). +_CONSUMED_PHRASES = ("already delivered", "already acknowledged", "already retrieved") + + +def _looks_consumed(payload: Any) -> bool: + if not isinstance(payload, dict): + return False + message = str(payload.get("message", "")).lower() + return any(phrase in message for phrase in _CONSUMED_PHRASES) + + +def raise_for_status(response: Response, endpoint: Endpoint | None = None) -> None: + """Convert an unsuccessful -- or deceptively successful -- response into an error.""" + # Checked before the status check: this arrives as a 200. + if _looks_consumed(response.payload): + raise CLIError( + _extract_message(response.payload) or "Result already retrieved.", + ExitCode.ALREADY_CONSUMED, + http_status=response.status, + details=response.payload if isinstance(response.payload, dict) else None, + endpoint=f"{endpoint.method} {endpoint.path}" if endpoint else None, + hint=hint_for(406), + retryable=False, + ) + + if response.status < 400: + return + + message = _extract_message(response.payload) or f"HTTP {response.status}" + code = exit_code_for_status(response.status) + + # The deployment API returns 406 for "result already acknowledged"; treat any + # already-delivered signal as the dedicated one-shot exit code (SPEC.md §5.6). + if response.status == 406 or "already" in message.lower() and "deliver" in message.lower(): + code = ExitCode.ALREADY_CONSUMED + + raise CLIError( + message, + code, + http_status=response.status, + details=response.payload if isinstance(response.payload, (dict, list)) else None, + endpoint=f"{endpoint.method} {endpoint.path}" if endpoint else None, + hint=hint_for(response.status, endpoint.path if endpoint else None, message), + retryable=is_retryable(response.status), + ) + + +def _extract_message(payload: Any) -> str | None: + """Pull a human message out of the several error shapes these APIs use.""" + if isinstance(payload, str): + return payload.strip() or None + if not isinstance(payload, dict): + return None + for key in ("message", "detail", "error"): + value = payload.get(key) + if isinstance(value, str) and value: + return value + errors = payload.get("errors") + if isinstance(errors, list) and errors: + parts = [ + e.get("detail") for e in errors if isinstance(e, dict) and e.get("detail") + ] + if parts: + return "; ".join(str(p) for p in parts) + return None + + +__all__ = [ + "GATEWAY_INJECTED_HEADERS", + "RequestPlan", + "Response", + "auth_headers", + "build_request", + "build_url", + "collect_secrets", + "execute", + "raise_for_status", +] diff --git a/src/unstract_cli/core/model.py b/src/unstract_cli/core/model.py new file mode 100644 index 0000000..1820381 --- /dev/null +++ b/src/unstract_cli/core/model.py @@ -0,0 +1,537 @@ +"""Declarative endpoint model — the single source of truth for the CLI. + +Every command, flag, help string, validation rule and `--discover` entry is +derived from the `Endpoint` records in `unstract_cli.endpoints`. Nothing about a +command is written twice, so help text cannot drift from behaviour, and the +bundled Claude Skill has exactly one place to edit (SPEC.md D1). + +The dataclasses here are frozen: records are a contract, not runtime state. + +Parameter patterns P1-P12 map onto these fields as: + +=== ========================== =================================================== +P Pattern Encoding +=== ========================== =================================================== +P1 Mutually exclusive ``Endpoint.constraints=[MutuallyExclusive(...)]`` +P2 At-least-one-of ``Endpoint.constraints=[AtLeastOneOf(...)]`` +P3 Enum -> path segment ``Param.choices={friendly: wire}`` (a mapping) +P4 Repeatable ``Param.multiple=True`` +P5 Freeform key=value ``Param.freeform_prefix="ext_"`` +P6 Path param, profile default ``Param.location=PATH`` + ``Param.default_from=...`` +P7 Location variants ``Param.location`` + ``Endpoint.body`` +P8 PATCH = PUT minus required ``Endpoint.derive_patch_from=`` +P9 Conditional applicability ``Param.applies_when="mode=low_cost"`` (help only) +P10 Int vs UUID identifiers ``Param.type`` (``ParamType.INT`` / ``UUID``) +P11 Trailing-slash sensitivity ``Endpoint.path`` is literal; never normalised +P12 Replace-vs-append ``Param.replace_semantics=True`` (+ helper flags) +=== ========================== =================================================== +""" + +from __future__ import annotations + +from collections.abc import Mapping, Sequence +from dataclasses import dataclass, field, replace +from enum import Enum + +# --------------------------------------------------------------------------- # +# Enumerations +# --------------------------------------------------------------------------- # + + +class Product(str, Enum): + """One of the three products built by Unstract. + + Unstract is the company, and the name of this CLI. It builds exactly three + products: **Document Studio**, **LLMWhisperer** and **API Hub**. Document + Studio's API groups use `platform`/`deployment` paths on the wire and + `UNSTRACT_*` environment variables. + """ + + DOCUMENT_STUDIO = "docstudio" + LLMWHISPERER = "llmwhisperer" + APIHUB = "apihub" + + +#: Document Studio exposes three distinct API groups, each with its own base +#: path and credentials, so they stay separate for auth and config purposes even +#: though they belong to one product (SPEC.md §4.4). +class ApiGroup(str, Enum): + """An API surface within a product. Determines base URL and credentials.""" + + #: Document Studio -- Platform Management API v1. + PLATFORM = "platform" + #: Document Studio -- deployed API workflow execution. + DEPLOYMENT = "deployment" + #: Document Studio -- Human Quality Review (Enterprise). + HITL = "hitl" + #: LLMWhisperer -- text extraction. + LLMWHISPERER = "llmwhisperer" + #: API Hub -- vertical extraction. + APIHUB = "apihub" + + +#: Which product each API group belongs to. +GROUP_PRODUCT: dict[ApiGroup, Product] = { + ApiGroup.PLATFORM: Product.DOCUMENT_STUDIO, + ApiGroup.DEPLOYMENT: Product.DOCUMENT_STUDIO, + ApiGroup.HITL: Product.DOCUMENT_STUDIO, + ApiGroup.LLMWHISPERER: Product.LLMWHISPERER, + ApiGroup.APIHUB: Product.APIHUB, +} + +#: Human-readable product names, for help text and `--discover`. +PRODUCT_LABELS: dict[Product, str] = { + Product.DOCUMENT_STUDIO: "Document Studio", + Product.LLMWHISPERER: "LLMWhisperer", + Product.APIHUB: "API Hub", +} + + +class ParamLocation(str, Enum): + """Where a parameter travels in the HTTP request (P7).""" + + QUERY = "query" + BODY = "body" + PATH = "path" + HEADER = "header" + FORM = "form" + + +class BodyKind(str, Enum): + """How the request body is encoded (P7).""" + + NONE = "none" + JSON = "json" + MULTIPART = "multipart" + BINARY_FILE = "binary_file" + TEXT = "text" + + +class ParamType(str, Enum): + """Logical parameter type. Distinguishes INT from UUID identifiers (P10).""" + + STR = "str" + INT = "int" + FLOAT = "float" + BOOL = "bool" + UUID = "uuid" + JSON = "json" + FILE = "file" + DATE = "date" + + +class Permission(str, Enum): + """Platform API key permission level required (SPEC.md §4.4).""" + + READ = "read" + READ_WRITE = "read_write" + FULL_ACCESS = "full_access" + + +# --------------------------------------------------------------------------- # +# Constraints (P1, P2) +# --------------------------------------------------------------------------- # + + +@dataclass(frozen=True) +class Constraint: + """Base class for pre-flight validation rules. + + Constraints are checked before any network call, so a malformed invocation + costs an exit code 2 rather than a wasted round trip and a remote 400. + """ + + params: tuple[str, ...] + + def check(self, supplied: Mapping[str, object]) -> str | None: + """Return an error message if violated, else ``None``.""" + raise NotImplementedError + + def describe(self) -> str: + """Human-readable rule, rendered into ``--help`` and ``--discover``.""" + raise NotImplementedError + + +@dataclass(frozen=True) +class MutuallyExclusive(Constraint): + """Exactly one of ``params`` must be supplied (P1). + + Example: ``whisper extract`` takes ``--file`` or ``--url``, never both. + """ + + required: bool = True + + def check(self, supplied: Mapping[str, object]) -> str | None: + present = [p for p in self.params if supplied.get(p) not in (None, (), [])] + flags = ", ".join(f"--{p.replace('_', '-')}" for p in self.params) + if len(present) > 1: + given = ", ".join(f"--{p.replace('_', '-')}" for p in present) + return f"{given} are mutually exclusive; supply exactly one of: {flags}" + if self.required and not present: + return f"one of {flags} is required" + return None + + def describe(self) -> str: + flags = " | ".join(f"--{p.replace('_', '-')}" for p in self.params) + return f"exactly one of: {flags}" if self.required else f"at most one of: {flags}" + + +@dataclass(frozen=True) +class RequiredUnless(Constraint): + """``params`` are required *unless* another flag holds a sentinel value. + + Encodes a rule the plain required/optional split cannot: a field that is + mandatory in general but genuinely unused in one configuration. The live case + is ``profile create --chunk-size 0``, which means "no RAG" -- the vector store + and embedding model are then never consulted, so demanding them makes the + caller invent a value for something that will not be read (GOTCHAS #3). + + Marking such a field ``required=False`` alone would lose the check in the + common case; this keeps it, conditioned on the flag that actually decides. + """ + + #: The flag whose value relaxes the requirement, e.g. ``"chunk_size"``. + unless: str = "" + #: Values of :attr:`unless` that switch the requirement off. + unless_values: tuple[object, ...] = () + + def _relaxed(self, supplied: Mapping[str, object]) -> bool: + value = supplied.get(self.unless) + # Compare as strings so 0 and "0" behave identically: Click hands the + # value through typed, while a test or a config default may not. + return any(str(value) == str(v) for v in self.unless_values) + + def check(self, supplied: Mapping[str, object]) -> str | None: + if self._relaxed(supplied): + return None + missing = [p for p in self.params if supplied.get(p) in (None, (), [])] + if not missing: + return None + flags = ", ".join(f"--{p.replace('_', '-')}" for p in missing) + relaxers = " or ".join(f"--{self.unless.replace('_', '-')} {v}" for v in self.unless_values) + return f"{flags} is required unless {relaxers} is set" + + def describe(self) -> str: + flags = ", ".join(f"--{p.replace('_', '-')}" for p in self.params) + relaxers = " or ".join(f"--{self.unless.replace('_', '-')} {v}" for v in self.unless_values) + return f"{flags} required unless {relaxers}" + + +@dataclass(frozen=True) +class AtLeastOneOf(Constraint): + """At least one of ``params`` must be supplied (P2). + + Example: ``file-history clear`` refuses to run without a filter, which would + otherwise delete every record. + """ + + def check(self, supplied: Mapping[str, object]) -> str | None: + if any(supplied.get(p) not in (None, (), []) for p in self.params): + return None + flags = ", ".join(f"--{p.replace('_', '-')}" for p in self.params) + return f"at least one of {flags} is required" + + def describe(self) -> str: + flags = " | ".join(f"--{p.replace('_', '-')}" for p in self.params) + return f"at least one of: {flags}" + + +# --------------------------------------------------------------------------- # +# Param +# --------------------------------------------------------------------------- # + + +@dataclass(frozen=True) +class Param: + """One CLI flag, and how it reaches the wire. + + ``name`` is the API's spelling (used on the wire verbatim, typos included -- + e.g. ``page_seperator``); the CLI flag is the kebab-cased form. + """ + + name: str + type: ParamType = ParamType.STR + location: ParamLocation = ParamLocation.QUERY + required: bool = False + default: object | None = None + help: str = "" + + #: P3 - friendly value -> wire value. A plain sequence means identity mapping. + choices: Mapping[str, str] | Sequence[str] | None = None + #: P4 - repeatable flag, collected into a list. + multiple: bool = False + #: P6 - dotted path into resolved config, e.g. ``"platform.org_id"``. A + #: whitespace-separated list is tried in order, first resolved value winning. + #: `deployment run` uses this to fall back to the platform block's org_id: the + #: deployment block is a separate, initially-empty config section, and an + #: org_id already set for the platform API is the same organization (GOTCHAS #7). + default_from: str | None = None + + @property + def default_sources(self) -> tuple[str, ...]: + """The config paths tried, in order, for this parameter's default.""" + return tuple(self.default_from.split()) if self.default_from else () + #: P5 - collect arbitrary ``--flag KEY=VALUE`` pairs under this prefix. + freeform_prefix: str | None = None + #: P9 - documented applicability. Rendered in help; never enforced locally, + #: because the server owns the rule and enforcing it here would guess wrong. + applies_when: str | None = None + #: P12 - this field replaces rather than appends server-side. + replace_semantics: bool = False + #: Override the derived CLI flag name (rare; e.g. to avoid a collision). + flag: str | None = None + #: Exclude from the request payload (client-side only, e.g. ``--save``). + client_side: bool = False + #: Copy this PATH param into the JSON body as well. A defence against a server + #: that reads an identifier only from the body and orphans the record when it + #: is absent (BUG 2: `prompt create` persists ``tool_id: null`). The URL still + #: carries the value; this just also sends it in the body under :attr:`name`. + mirror_to_body: bool = False + #: Body field name for the mirrored value, when the body spells the identifier + #: differently from the path. `api-deployment key create` is the live case: the + #: URL takes ``api_id`` while the body wants that same value as ``api``, so both + #: had to be passed by hand (GOTCHAS #6). Implies :attr:`mirror_to_body`. + mirror_as: str | None = None + + @property + def mirrors(self) -> bool: + """Whether this PATH param is also copied into the JSON body.""" + return self.mirror_to_body or self.mirror_as is not None + + @property + def body_name(self) -> str: + """The name this parameter takes in the body when mirrored.""" + return self.mirror_as or self.name + + @property + def cli_flag(self) -> str: + """The long-form CLI flag, e.g. ``--word-confidence-threshold``.""" + return self.flag or f"--{self.name.replace('_', '-')}" + + @property + def py_name(self) -> str: + """The Python identifier used for this parameter. + + Derived from :attr:`flag` when one is set, so a renamed flag does not + collide with a global option of the same API name. `deployment run` is + the live case: its API parameter is `timeout`, but the CLI exposes it as + ``--execution-timeout`` to leave ``--timeout`` meaning the HTTP timeout. + Without this, the global flag would silently overwrite the API value. + """ + if self.flag: + return self.flag.lstrip("-").replace("-", "_") + return self.name.replace("-", "_") + + @property + def wire_name(self) -> str: + """The parameter name as the API expects it, typos preserved.""" + return self.name + + def choice_map(self) -> dict[str, str] | None: + """Normalise :attr:`choices` to a ``{friendly: wire}`` mapping (P3).""" + if self.choices is None: + return None + if isinstance(self.choices, Mapping): + return dict(self.choices) + return {c: c for c in self.choices} + + def to_wire(self, value: object) -> object: + """Translate a user-supplied value to its wire representation (P3).""" + mapping = self.choice_map() + if mapping and isinstance(value, str): + return mapping.get(value, value) + return value + + +# --------------------------------------------------------------------------- # +# Polling (SPEC.md §3.1) +# --------------------------------------------------------------------------- # + + +@dataclass(frozen=True) +class PollSpec: + """Describes how ``--wait`` drives an async execute -> poll -> retrieve flow. + + Terminal states are matched against a field in the *response body*, never the + HTTP status code: the Unstract deployment API currently returns HTTP 422 for + the in-progress states ``PENDING``/``EXECUTING`` (SPEC.md §6.2), a documented + server defect. Branching on the body keeps behaviour identical before and + after that defect is fixed. + """ + + status_endpoint: str + #: Field name(s) holding the terminal state in the *status endpoint's* body. + #: A tuple is tried in order, because the run POST and the status GET can spell + #: the same state differently: the deployment run response nests + #: ``execution_status`` under ``message``, while the status GET returns a + #: top-level ``status`` (and its ``message`` is the *result*, not a nested + #: object). The poll reads the status endpoint, so ``status`` must win there -- + #: a mismatch means the terminal state goes unrecognised, the one-shot result + #: is consumed on that read, and the next poll returns HTTP 406 (CAPTURE2 BUG 2). + status_field: str | tuple[str, ...] = "status" + terminal_success: tuple[str, ...] = () + terminal_failure: tuple[str, ...] = () + handle_field: str = "" + handle_param: str = "" + retrieve_endpoint: str | None = None + #: Values to forward from the *original* request into the retrieve call. Each + #: entry is either a py_name carried as-is, or a ``(source, dest)`` pair that + #: renames it -- the retrieve endpoint often spells the same identifier + #: differently (fetch-response's ``id``/``document_id`` are the Output + #: Manager's ``prompt_id``/``document_manager``). Without the rename the + #: retrieve would return every row for the tool, not the one prompt+document + #: the caller ran. The retrieve is otherwise keyed by the poll handle, but some + #: result stores are keyed by an original-request identifier instead + #: (prompt-studio reads its Output Manager by ``tool_id``, not by ``task_id``). + retrieve_carry: tuple[str | tuple[str, str], ...] = () + #: Suppress passing the poll handle into the retrieve call. Set when the + #: retrieve endpoint is keyed only by :attr:`retrieve_carry` values and would + #: reject an unexpected handle parameter. + retrieve_omits_handle: bool = False + #: Constant param values injected into the retrieve call, keyed by py_name. + #: Used where the retrieve endpoint needs a fixed flag the original request did + #: not carry (single-pass results are read with ``is_single_pass_extract=true``). + retrieve_extra: tuple[tuple[str, object], ...] = () + #: Results can be read exactly once; a second read loses data (SPEC.md §5.6). + one_shot: bool = False + + +# --------------------------------------------------------------------------- # +# Endpoint +# --------------------------------------------------------------------------- # + + +@dataclass(frozen=True) +class Endpoint: + """One CLI command, and the API call behind it.""" + + name: str + group: str + method: str + path: str + #: The API surface this endpoint belongs to, which fixes its base URL and + #: credentials. The owning product is derived from it, never stored twice. + api: ApiGroup + summary: str + params: tuple[Param, ...] = () + body: BodyKind = BodyKind.NONE + #: Sub-group for three-level commands, e.g. ``platform prompt-studio file upload``. + subgroup: str | None = None + constraints: tuple[Constraint, ...] = () + poll: PollSpec | None = None + #: Documentation file this record was authored from; the Skill's diff anchor. + doc_source: str = "" + permission: Permission | None = None + #: Longer help text appended below the summary. + description: str = "" + #: Worked example(s) rendered into ``--help`` (SPEC.md §5.3). + examples: tuple[str, ...] = () + #: A deliberate divergence from the docs; the Skill must not silently revert + #: it (SPEC.md §8.5). Example: `/whisper-detail` is singular despite the docs + #: index saying otherwise. + doc_conflict: str | None = None + #: P11 - some paths legitimately lack a trailing slash. Recorded so a test can + #: assert intent rather than treating every missing slash as a typo. + no_trailing_slash: bool = False + #: Optional column hints for ``--output table``. + table_columns: tuple[str, ...] = () + #: Response key holding the payload for ``--output raw``. + raw_field: str | None = None + #: Response fields that must be non-null on success, else the call is treated + #: as a failure despite a 2xx status. Guards silent-orphan defects where the + #: server returns 201 but leaves a linking field NULL (BUG 2: `prompt create`). + require_response_fields: tuple[str, ...] = () + + @property + def product(self) -> Product: + """The product this endpoint belongs to (derived, never stored).""" + return GROUP_PRODUCT[self.api] + + @property + def product_label(self) -> str: + """Display name, e.g. ``"Document Studio"``.""" + return PRODUCT_LABELS[self.product] + + @property + def command_path(self) -> tuple[str, ...]: + """Full command path, e.g. ``("platform", "prompt-studio", "file", "upload")``.""" + parts = [self.group] + if self.subgroup: + parts.extend(self.subgroup.split()) + parts.append(self.name) + return tuple(parts) + + @property + def dotted_name(self) -> str: + """Stable identifier, e.g. ``whisper.usage``.""" + return ".".join(self.command_path) + + def param(self, name: str) -> Param | None: + """Look up a parameter by API name.""" + return next((p for p in self.params if p.name == name), None) + + def path_params(self) -> tuple[Param, ...]: + return tuple(p for p in self.params if p.location is ParamLocation.PATH) + + def validate(self, supplied: Mapping[str, object]) -> list[str]: + """Run every constraint, returning all violations (P1, P2).""" + return [m for c in self.constraints if (m := c.check(supplied)) is not None] + + +def derive_patch( + source: Endpoint, + *, + name: str = "patch", + summary: str | None = None, + keep_required: Sequence[str] = (), +) -> Endpoint: + """Derive a PATCH endpoint from its PUT counterpart (P8). + + PATCH accepts the same fields as PUT but makes them all optional, except for + identifiers (path params, plus anything named in ``keep_required``). Deriving + rather than copying means a parameter added to the PUT record cannot be + forgotten on the PATCH one -- duplication is precisely how definitions drift. + """ + kept = set(keep_required) + params = tuple( + p + if (p.location is ParamLocation.PATH or p.name in kept) + else replace(p, required=False) + for p in source.params + ) + return replace( + source, + name=name, + method="PATCH", + params=params, + summary=summary or f"Partially update. {source.summary}", + constraints=(), + ) + + +def with_params(source: Endpoint, *extra: Param) -> Endpoint: + """Return a copy of ``source`` with additional parameters appended. + + Used where a derived PATCH accepts a field its PUT counterpart does not -- + for instance `pipeline patch --active`, which has no PUT equivalent. + """ + return replace(source, params=(*source.params, *extra)) + + +__all__ = [ + "AtLeastOneOf", + "BodyKind", + "Constraint", + "Endpoint", + "MutuallyExclusive", + "Param", + "ParamLocation", + "ParamType", + "Permission", + "PollSpec", + "Product", + "RequiredUnless", + "derive_patch", + "field", + "with_params", +] diff --git a/src/unstract_cli/core/output.py b/src/unstract_cli/core/output.py new file mode 100644 index 0000000..4e6a377 --- /dev/null +++ b/src/unstract_cli/core/output.py @@ -0,0 +1,229 @@ +"""Output rendering (SPEC.md §5.1). + +The contract an agent depends on: + +* ``--output json`` is the default whenever stdout is **not** a TTY, so piping or + capturing the CLI yields machine-readable output with no extra flags. +* **stdout carries the payload and nothing else.** Banners, warnings, progress + and diagnostics all go to stderr, so `unstract ... | jq` always parses. +* ``raw`` emits the payload alone (extracted text, file bytes) for piping. +""" + +from __future__ import annotations + +import json +import os +import shutil +import sys +import textwrap +from enum import Enum +from typing import Any, TextIO + +import yaml + + +class OutputFormat(str, Enum): + JSON = "json" + YAML = "yaml" + TABLE = "table" + RAW = "raw" + + +def default_format(stream: TextIO | None = None) -> OutputFormat: + """JSON unless a human is watching (SPEC.md §5.1). + + ``UNSTRACT_OUTPUT`` overrides. TTY detection means an agent gets JSON without + passing a flag, while a human at a terminal gets a table. + """ + if env := os.environ.get("UNSTRACT_OUTPUT"): + try: + return OutputFormat(env.lower()) + except ValueError: + pass # An invalid value must not break the command; fall through. + target = stream or sys.stdout + try: + return OutputFormat.TABLE if target.isatty() else OutputFormat.JSON + except (AttributeError, ValueError): # pragma: no cover - detached stream + return OutputFormat.JSON + + +def _flatten(value: Any) -> str: + """Render a cell. Nested structures become compact JSON rather than Python reprs.""" + if value is None: + return "" + if isinstance(value, bool): + return "true" if value else "false" + if isinstance(value, (dict, list)): + return json.dumps(value, default=str) + return str(value) + + +def _rows_and_columns( + data: Any, columns: tuple[str, ...] = () +) -> tuple[list[str], list[list[str]]]: + """Derive table columns and rows from arbitrary JSON. + + The generic rule: + + * list of objects -> columns from the union of keys, in first-seen order + * single object -> two-column key/value listing + * anything else -> a single ``value`` column + + ``columns`` overrides column selection for responses where the generic rule + reads poorly. + """ + if isinstance(data, dict): + # Unwrap a single list-valued envelope, e.g. {"results": [...]}. + for key in ("results", "message", "members", "data", "highlights"): + inner = data.get(key) + if isinstance(inner, list) and inner: + data = inner + break + + if isinstance(data, list): + if not data: + return [], [] + if all(isinstance(item, dict) for item in data): + if columns: + headers = list(columns) + else: + headers = [] + for item in data: + headers.extend(k for k in item if k not in headers) + return headers, [[_flatten(item.get(h)) for h in headers] for item in data] + return ["value"], [[_flatten(item)] for item in data] + + if isinstance(data, dict): + keys = list(columns) if columns else list(data) + return ["key", "value"], [[k, _flatten(data.get(k))] for k in keys] + + return ["value"], [[_flatten(data)]] + + +def _terminal_width(default: int = 100) -> int: + """Usable width for table output.""" + try: + return max(shutil.get_terminal_size((default, 24)).columns, 40) + except Exception: # pragma: no cover - detached terminal + return default + + +def render_table( + data: Any, columns: tuple[str, ...] = (), *, max_width: int | None = None +) -> str: + """Render as an aligned plain-text table. + + Deliberately plain text rather than Rich box-drawing: tables are the + human-facing format, but they still end up in logs and terminals of varying + width, and ASCII survives both. + + Long cells are **wrapped, never truncated**: a table is a view of the data, + not a lossy summary, and silently dropping the tail of a value is the kind of + thing you only notice after acting on it. Wrapped continuation lines are + indented under their column so the table still reads as a grid. + """ + headers, rows = _rows_and_columns(data, columns) + if not headers: + return "(no results)" + + gutter = 2 + total_width = max_width or _terminal_width() + + natural = [len(h) for h in headers] + for row in rows: + for i, cell in enumerate(row): + if i < len(natural): + natural[i] = max(natural[i], max((len(p) for p in cell.split("\n")), default=0)) + + # Shrink only the widest columns, and only as far as the terminal requires, + # so a narrow column is never squeezed on behalf of a wide neighbour. + widths = list(natural) + budget = total_width - gutter * (len(headers) - 1) + while sum(widths) > budget and max(widths) > 8: + widest = widths.index(max(widths)) + widths[widest] -= 1 + + def fmt(cells: list[str]) -> list[str]: + """Lay one logical row out over as many physical lines as it needs.""" + wrapped = [ + textwrap.wrap(cell, width=w, break_long_words=True, break_on_hyphens=False) + or [""] + for cell, w in zip(cells, widths, strict=False) + ] + height = max(len(parts) for parts in wrapped) + lines = [] + for line_no in range(height): + pieces = [ + (parts[line_no] if line_no < len(parts) else "").ljust(w) + for parts, w in zip(wrapped, widths, strict=False) + ] + lines.append((" " * gutter).join(pieces).rstrip()) + return lines + + out = fmt(headers) + out.append((" " * gutter).join("-" * w for w in widths).rstrip()) + for row in rows: + out.extend(fmt(row)) + return "\n".join(out) + + +def render( + data: Any, + fmt: OutputFormat, + *, + columns: tuple[str, ...] = (), + raw_field: str | None = None, +) -> str: + """Render a payload in the requested format.""" + match fmt: + case OutputFormat.JSON: + return json.dumps(data, indent=2, default=str) + case OutputFormat.YAML: + return yaml.safe_dump(data, sort_keys=False, default_flow_style=False).rstrip() + case OutputFormat.TABLE: + return render_table(data, columns) + case OutputFormat.RAW: + if isinstance(data, dict) and raw_field and raw_field in data: + value = data[raw_field] + elif isinstance(data, (str, bytes)): + value = data + else: + value = data + if isinstance(value, bytes): + return value.decode("utf-8", errors="replace") + if isinstance(value, str): + return value + return json.dumps(value, indent=2, default=str) + return json.dumps(data, indent=2, default=str) # pragma: no cover + + +def emit( + data: Any, + fmt: OutputFormat, + *, + columns: tuple[str, ...] = (), + raw_field: str | None = None, +) -> None: + """Write a payload to stdout -- and nothing else to stdout.""" + print(render(data, fmt, columns=columns, raw_field=raw_field)) + + +def diagnostic(message: str, *, quiet: bool = False, verbosity: int = 0, level: int = 0) -> None: + """Write a human-facing note to **stderr**, keeping stdout pure. + + ``level`` is the minimum ``-v`` count required: 0 always shows (unless + ``--quiet``), 1 needs ``-v``, 2 needs ``-vv``. + """ + if quiet or verbosity < level: + return + print(message, file=sys.stderr) + + +__all__ = [ + "OutputFormat", + "default_format", + "diagnostic", + "emit", + "render", + "render_table", +] diff --git a/src/unstract_cli/core/poll.py b/src/unstract_cli/core/poll.py new file mode 100644 index 0000000..c047050 --- /dev/null +++ b/src/unstract_cli/core/poll.py @@ -0,0 +1,192 @@ +"""`--wait` state machines (SPEC.md §3.1, §6.2). + +Both LLMWhisperer and Unstract follow execute -> poll -> retrieve. Agents should +not have to script that loop, so `--wait` drives it to a terminal state. + +**The load-bearing rule:** terminal state is decided by the ``status`` field in +the *response body*, never by the HTTP status code. The Unstract deployment API +currently returns HTTP 422 for the in-progress states ``PENDING`` and +``EXECUTING`` -- a documented server-side defect that is scheduled to be fixed. +Reading the body means the CLI behaves identically before and after that fix; +reading the status code would break in one direction or the other. +""" + +from __future__ import annotations + +import time +from typing import Any + +from unstract_cli.config.loader import ResolvedConfig +from unstract_cli.core import http +from unstract_cli.core.errors import CLIError, ExitCode +from unstract_cli.core.model import Endpoint +from unstract_cli.core.output import diagnostic + + +def _dig(payload: Any, field: str) -> Any: + """Find a field, looking one level into common envelopes. + + Responses variously arrive bare, under ``message``, or under ``data``, and an + agent shouldn't care which. + """ + if not isinstance(payload, dict): + return None + if field in payload: + return payload[field] + for envelope in ("message", "data", "result"): + inner = payload.get(envelope) + if isinstance(inner, dict) and field in inner: + return inner[field] + return None + + +def extract_status(payload: Any, field: str | tuple[str, ...] = "status") -> str | None: + """Read the status from a response body. + + ``field`` may be a single name or a tuple of candidate names tried in order, + because the run POST and the status GET spell the state differently (the run + nests ``execution_status`` under ``message``; the status GET returns a + top-level ``status``). The first candidate that resolves wins. + """ + fields = (field,) if isinstance(field, str) else field + for candidate in fields: + value = _dig(payload, candidate) + if value is not None: + return str(value) + return None + + +def extract_handle(payload: Any, field: str) -> str | None: + """Read the job handle (whisper_hash / execution_id / file_hash).""" + value = _dig(payload, field) + return str(value) if value is not None else None + + +def _carry_path_values(values: dict[str, Any], target: Endpoint) -> dict[str, Any]: + """Forward path parameters from the original call into a follow-up call.""" + return { + param.py_name: values[param.py_name] + for param in target.path_params() + if values.get(param.py_name) is not None + } + + +def wait_for_completion( + *, + endpoint: Endpoint, + initial: Any, + config: ResolvedConfig, + values: dict[str, Any] | None = None, + poll_interval: float = 3.0, + timeout: float = 300.0, + max_retries: int = 3, + request_timeout: float = 60.0, + quiet: bool = False, + verbosity: int = 0, + sleep=time.sleep, + now=time.monotonic, +) -> Any: + """Poll until terminal, then retrieve if the endpoint defines a retrieve step. + + On timeout, exits 7 *with the job handle*, so an agent can resume with a + plain `status`/`retrieve` call rather than reprocessing the document. + """ + spec = endpoint.poll + if spec is None: # pragma: no cover - guarded by the caller + return initial + + from unstract_cli.endpoints import get_endpoint + + handle = extract_handle(initial, spec.handle_field) + if not handle: + diagnostic( + f"--wait: no {spec.handle_field} in response; returning immediately.", + quiet=quiet, + verbosity=verbosity, + ) + return initial + + status_endpoint = get_endpoint(spec.status_endpoint) + deadline = now() + timeout + last_status: str | None = None + + # Path parameters from the original invocation must survive into the status + # and retrieve calls: `deployment status` needs the same --api-name and + # --org-id, and the handle alone cannot supply them. + carried = _carry_path_values(values or {}, status_endpoint) + + while True: + poll_values = {**carried, spec.handle_param: handle} + plan = http.build_request(status_endpoint, config, poll_values) + response = http.execute( + plan, + endpoint=status_endpoint, + timeout=request_timeout, + max_retries=max_retries, + ) + + status = extract_status(response.payload, spec.status_field) + + # Only now consider the HTTP status: if the body carried no recognisable + # state, a 4xx/5xx is a real failure rather than the 422 quirk. + if status is None: + http.raise_for_status(response, status_endpoint) + status = extract_status(response.payload, spec.status_field) + + if status != last_status: + diagnostic(f"--wait: status={status}", quiet=quiet, verbosity=verbosity, level=1) + last_status = status + + normalised = (status or "").lower() + if normalised in {s.lower() for s in spec.terminal_failure}: + raise CLIError( + f"Operation finished with status {status!r}.", + ExitCode.VALIDATION, + endpoint=f"{endpoint.method} {endpoint.path}", + details=response.payload, + hint="Inspect `details` for the per-file error, or check execution logs.", + extra={spec.handle_field: handle}, + ) + + if normalised in {s.lower() for s in spec.terminal_success}: + break + + if now() >= deadline: + raise CLIError( + f"Timed out after {timeout:g}s waiting for completion " + f"(last status: {status!r}).", + ExitCode.TIMEOUT, + endpoint=f"{endpoint.method} {endpoint.path}", + hint=( + f"The job is still running. Resume with the {spec.handle_field} " + f"below rather than resubmitting the document." + ), + extra={spec.handle_field: handle, "last_status": status}, + ) + + sleep(poll_interval) + + if not spec.retrieve_endpoint: + return response.payload + + retrieve_endpoint = get_endpoint(spec.retrieve_endpoint) + retrieve_values: dict[str, Any] = _carry_path_values(values or {}, retrieve_endpoint) + # Some result stores are keyed by an identifier from the *original* request, + # not by the poll handle (prompt-studio's Output Manager is read by tool_id). + for entry in spec.retrieve_carry: + source, dest = entry if isinstance(entry, tuple) else (entry, entry) + if (carried_value := (values or {}).get(source)) is not None: + retrieve_values[dest] = carried_value + for name, constant in spec.retrieve_extra: + retrieve_values[name] = constant + if not spec.retrieve_omits_handle: + retrieve_values[spec.handle_param] = handle + plan = http.build_request(retrieve_endpoint, config, retrieve_values) + result = http.execute( + plan, endpoint=retrieve_endpoint, timeout=request_timeout, max_retries=max_retries + ) + http.raise_for_status(result, retrieve_endpoint) + return result.payload + + +__all__ = ["extract_handle", "extract_status", "wait_for_completion"] diff --git a/src/unstract_cli/endpoints/__init__.py b/src/unstract_cli/endpoints/__init__.py new file mode 100644 index 0000000..abb4694 --- /dev/null +++ b/src/unstract_cli/endpoints/__init__.py @@ -0,0 +1,50 @@ +"""Endpoint registry — the single source of truth for the CLI surface. + +Every command in the tree originates here. The bundled Claude Skill edits these +modules and nothing else: because commands, help text and `--discover` are +all generated from these records, a change here propagates everywhere at once. +""" + +from __future__ import annotations + +from unstract_cli.core.model import Endpoint + +from . import apihub, deployment, hitl, platform, whisper + +#: Ordered so `--help` lists groups in a sensible progression: extraction first, +#: then execution, then management. +ALL_ENDPOINTS: tuple[Endpoint, ...] = ( + *whisper.ENDPOINTS, + *deployment.ENDPOINTS, + *platform.ENDPOINTS, + *hitl.ENDPOINTS, + *apihub.ENDPOINTS, +) + +_BY_NAME: dict[str, Endpoint] = {e.dotted_name: e for e in ALL_ENDPOINTS} + + +def get_endpoint(dotted_name: str) -> Endpoint: + """Look up an endpoint by dotted name, e.g. ``whisper.status``. + + Used by the poller to find the status/retrieve endpoints named in a + `PollSpec`, which keeps polling declarative rather than hard-coded. + """ + try: + return _BY_NAME[dotted_name] + except KeyError: + raise KeyError( + f"No endpoint named {dotted_name!r}. Known: {', '.join(sorted(_BY_NAME))}" + ) from None + + +def endpoints_for(api: str) -> tuple[Endpoint, ...]: + """All endpoints on one API surface, e.g. ``"platform"`` or ``"llmwhisperer"``. + + Filters by API group rather than command path, because that is the unit the + documentation is organised around -- one docs source per API. + """ + return tuple(e for e in ALL_ENDPOINTS if e.api.value == api) + + +__all__ = ["ALL_ENDPOINTS", "endpoints_for", "get_endpoint"] diff --git a/src/unstract_cli/endpoints/apihub.py b/src/unstract_cli/endpoints/apihub.py new file mode 100644 index 0000000..ea3a269 --- /dev/null +++ b/src/unstract_cli/endpoints/apihub.py @@ -0,0 +1,214 @@ +"""API Hub (Verticals) endpoints (SPEC.md §6.5). + +**Source of truth is code, not docs.** API Hub has no public documentation site, +so these records were authored from `unstract-verticals/src/api_v1/api.py` and the +Postman collections in `verticals-portal/portal/postman-collection/`. The Skill +must flag changes here for human review rather than applying them silently +(SPEC.md §8.4). + +Auth is the `apikey` header at the Kong gateway. Kong resolves that key in Redis +and injects `X-Subscription-Id`, `X-Subscription-Name`, `X-User-Id` and +`X-Product-Id` downstream -- the CLI must never send those itself. + +Extraction parameters (`ext_*`) are forwarded verbatim to the vertical worker, so +`--ext-param KEY=VALUE` exists as an escape hatch for parameters newer than this +CLI. That matters more here than elsewhere precisely because there are no docs to +track. +""" + +from __future__ import annotations + +from unstract_cli.core.model import ( + ApiGroup, + BodyKind, + Endpoint, + MutuallyExclusive, + Param, + ParamLocation, + ParamType, + PollSpec, +) + +_SRC = "unstract-verticals/src/api_v1/api.py" +_POSTMAN = "verticals-portal/portal/postman-collection" + +#: LLMWhisperer conversion passthrough. These mirror the `whisper extract` flags +#: but reach LLMWhisperer via the vertical worker, hence the `conv_` prefix. +_CONV_PARAMS: tuple[Param, ...] = ( + Param("conv_mode", default="high_quality", + choices=["native_text", "low_cost", "high_quality", "form", "table"], + help="LLMWhisperer processing mode"), + Param("conv_output_mode", default="layout_preserving", + choices=["layout_preserving", "text"], help="LLMWhisperer output mode"), + Param("conv_lang", default="eng", help="Language hint for OCR"), + Param("conv_tag", default="default", help="Audit tag for usage reporting"), + Param("conv_filename", help="Audit file name for usage reporting"), + Param("conv_page_seperator", default="<<<", flag="--conv-page-separator", + help="Page separator string"), + Param("conv_pages_to_extract", help="Pages to extract, e.g. '1-5,7,21-'"), + Param("conv_median_filter_size", type=ParamType.INT, + applies_when="conv_mode=low_cost", help="Median filter size for denoising"), + Param("conv_gaussian_blur_radius", type=ParamType.FLOAT, + applies_when="conv_mode=low_cost", help="Gaussian blur radius for denoising"), + Param("conv_mark_vertical_lines", type=ParamType.BOOL, default=False, + help="Reproduce vertical lines"), + Param("conv_mark_horizontal_lines", type=ParamType.BOOL, default=False, + help="Reproduce horizontal lines"), + Param("conv_line_splitter_strategy", default="left-priority", + help="Line splitter strategy"), + Param("conv_line_splitter_tolerance", type=ParamType.FLOAT, default=0.4, + help="Line splitter tolerance"), + Param("conv_horizontal_stretch_factor", type=ParamType.FLOAT, default=1.0, + help="Horizontal stretch factor"), +) + +#: Vertical-worker extraction parameters. +_EXT_PARAMS: tuple[Param, ...] = ( + Param("ext_section_name", help="Section of the document containing the target table"), + Param("ext_compress_double_space", type=ParamType.BOOL, + help="Collapse double-spaced lines, common in bank/credit-card statements"), + Param("ext_headers", help="Comma-separated headers to force the extraction to use"), + Param("ext_start_page", type=ParamType.INT, help="Start searching from this page"), + Param("ext_end_page", type=ParamType.INT, help="Stop searching at this page"), + Param("ext_page_filter_strategy", help="Page filtering strategy"), + Param("ext_use_bank_schema", type=ParamType.BOOL, + applies_when="sub_vertical=bank_statement", + help="Apply the bank statement schema"), + Param("ext_pattern", choices=["generic_table", "indent_as_groups"], + applies_when="sub_vertical=extract_table", help="Table extraction pattern"), + Param("ext_table_no", type=ParamType.INT, applies_when="sub_vertical=extract_table", + help="Which discovered table to extract, 1-based"), + Param("ext_cache_result", type=ParamType.BOOL, + help="Cache the result for 24 hours, for reuse via --use-cached-file-hash"), + Param("ext_cache_text", type=ParamType.BOOL, + help="Cache extracted text for 24 hours"), + Param("ext_param", multiple=True, freeform_prefix="ext_", + help=( + "Escape hatch for extraction parameters newer than this CLI, as " + "KEY=VALUE (sent as ext_KEY=VALUE); repeatable" + )), +) + + +ENDPOINTS: tuple[Endpoint, ...] = ( + Endpoint( + name="extract", + group="apihub", + method="POST", + path="/api/v1/extract", + api=ApiGroup.APIHUB, + summary="Submit a document for vertical extraction.", + description=( + "Processing runs in stages: QUEUED_FOR_WHISPER -> " + "QUEUED_FOR_EXTRACTION -> COMPLETED. Use --wait to follow it through.\n\n" + "Pass --use-cached-file-hash instead of --file to re-run extraction " + "over a document already processed by table discovery, which skips " + "re-conversion." + ), + params=( + Param("vertical", required=True, default="table", + help="Vertical, e.g. 'table'"), + Param("sub_vertical", required=True, + choices=["bank_statement", "discover_tables", "extract_table"], + help="Sub-vertical determining which worker handles the document"), + Param("file", type=ParamType.FILE, location=ParamLocation.BODY, + client_side=True, help="Path to the document to process"), + Param("use_cached_file_hash", + help="Reuse a previously processed document by its file hash"), + *_CONV_PARAMS, + *_EXT_PARAMS, + ), + body=BodyKind.BINARY_FILE, + constraints=(MutuallyExclusive(("file", "use_cached_file_hash")),), + poll=PollSpec( + status_endpoint="apihub.status", + status_field="status", + terminal_success=("COMPLETED",), + terminal_failure=("ERROR", "FAILED"), + handle_field="file_hash", + handle_param="file_hash", + retrieve_endpoint="apihub.retrieve", + ), + doc_source=f"{_SRC} + {_POSTMAN}", + examples=( + "unstract apihub extract --vertical table --sub-vertical bank_statement " + "--file statement.pdf --wait", + "unstract apihub extract --vertical table --sub-vertical extract_table " + "--use-cached-file-hash --ext-table-no 1", + ), + ), + Endpoint( + name="status", + group="apihub", + method="GET", + path="/api/v1/status", + api=ApiGroup.APIHUB, + summary="Check the processing status of a submitted document.", + description="Statuses: QUEUED_FOR_WHISPER, QUEUED_FOR_EXTRACTION, COMPLETED.", + params=( + Param("file_hash", required=True, help="Hash returned by `apihub extract`"), + ), + doc_source=f"{_SRC} + {_POSTMAN}", + examples=("unstract apihub status --file-hash ",), + ), + Endpoint( + name="retrieve", + group="apihub", + method="GET", + path="/api/v1/retrieve", + api=ApiGroup.APIHUB, + summary="Retrieve extraction results for a processed document.", + params=( + Param("file_hash", required=True, help="Hash returned by `apihub extract`"), + Param("output_mode", default="full", choices=["raw", "full"], + help="raw returns the extraction payload alone"), + Param("sub_vertical", help="Sub-vertical used for the extraction"), + Param("save", client_side=True, help="Write the result to this path"), + ), + doc_source=f"{_SRC} + {_POSTMAN}", + examples=("unstract apihub retrieve --file-hash --save tables.json",), + ), + Endpoint( + name="upload", + group="apihub", + subgroup="doc-splitter", + method="POST", + path="/doc-splitter/documents/upload", + api=ApiGroup.APIHUB, + summary="Upload a document for splitting.", + params=( + Param("file", type=ParamType.FILE, location=ParamLocation.FORM, + required=True, help="Path to the document to split"), + ), + body=BodyKind.MULTIPART, + doc_source=f"{_POSTMAN}/Verticals-DocSplitter.postman_collection.json", + examples=("unstract apihub doc-splitter upload --file bundle.pdf",), + ), + Endpoint( + name="status", + group="apihub", + subgroup="doc-splitter", + method="GET", + path="/doc-splitter/jobs/status", + api=ApiGroup.APIHUB, + summary="Check the status of a document splitting job.", + params=(Param("job_id", required=True, help="Job identifier from `upload`"),), + doc_source=f"{_POSTMAN}/Verticals-DocSplitter.postman_collection.json", + examples=("unstract apihub doc-splitter status --job-id ",), + ), + Endpoint( + name="download", + group="apihub", + subgroup="doc-splitter", + method="GET", + path="/doc-splitter/jobs/download", + api=ApiGroup.APIHUB, + summary="Download the output of a completed splitting job.", + params=( + Param("job_id", required=True, help="Job identifier from `upload`"), + Param("save", client_side=True, help="Write the downloaded output to this path"), + ), + doc_source=f"{_POSTMAN}/Verticals-DocSplitter.postman_collection.json", + examples=("unstract apihub doc-splitter download --job-id --save split.zip",), + ), +) diff --git a/src/unstract_cli/endpoints/deployment.py b/src/unstract_cli/endpoints/deployment.py new file mode 100644 index 0000000..efec84e --- /dev/null +++ b/src/unstract_cli/endpoints/deployment.py @@ -0,0 +1,186 @@ +"""Unstract API Deployment runtime endpoints (SPEC.md §6.2). + +Authored from `unstract-docs/docs/unstract_platform/api_deployment/`. + +Two behaviours worth knowing before reading the records: + +* ``org_id`` and ``api_name`` are URL *path segments*. ``--org-id`` falls back to + the active profile; ``--api-name`` cannot, because one profile serves many + deployments. +* The in-progress states ``PENDING``/``EXECUTING`` are currently returned with + HTTP 422 rather than 200 -- a documented server-side defect. The poller reads + the body's ``status`` field, so behaviour is unchanged when that is fixed. +""" + +from __future__ import annotations + +from unstract_cli.core.model import ( + ApiGroup, + BodyKind, + Endpoint, + Param, + ParamLocation, + ParamType, + PollSpec, +) + +_DOCS = "unstract-docs/docs/unstract_platform/api_deployment" + +#: Path parameters shared by every deployment command. +_PATH_PARAMS: tuple[Param, ...] = ( + Param( + "org_id", + location=ParamLocation.PATH, + # Falls back to the platform block: `docstudio.deployment` is a separate, + # initially-empty config section, so a user who has configured the + # Platform API still hit "missing org_id" here even though it is the same + # organization. The deployment block still wins when set (GOTCHAS #7). + default_from="deployment.org_id platform.org_id", + help="Organization identifier. Falls back to the platform block's org_id", + ), + Param( + "api_name", + location=ParamLocation.PATH, + required=True, + help="Deployed API name, as shown in the API Deployments page", + ), +) + +_EXECUTION_STATUSES = ("PENDING", "EXECUTING", "COMPLETED", "STOPPED", "ERROR") + + +ENDPOINTS: tuple[Endpoint, ...] = ( + Endpoint( + name="run", + group="docstudio", + subgroup="deployment", + method="POST", + path="/deployment/api/{org_id}/{api_name}/", + api=ApiGroup.DEPLOYMENT, + summary="Execute a deployed API workflow on one or more files.", + description=( + "Accepts up to 32 files per call, counting --file and --presigned-url " + "together. Results are keyed by file name, so names must be unique " + "within a call. Use --wait to poll to completion.\n\n" + "ONE-SHOT with --wait: the result store is read exactly once. --wait " + "returns the result from the poll that first observes COMPLETED and " + "does not re-read it. Pass --save to persist it on that single read; " + "without --save the result is only printed and cannot be re-fetched.\n\n" + "Synchronous mode (--timeout > 0) is deprecated upstream; --wait uses " + "asynchronous execution plus polling, which is the supported path.\n\n" + "Auth comes from the `docstudio.deployment` config block, which is " + "separate from the platform block and starts empty. After creating a " + "deployment and a key, wire the key in with:\n" + " unstract config set docstudio.deployment api_key env:UNSTRACT_DEPLOYMENT_KEY\n" + "--org-id now falls back to the platform block's org_id, so that one " + "usually needs no second entry (GOTCHAS #7)." + ), + params=( + *_PATH_PARAMS, + Param("files", type=ParamType.FILE, location=ParamLocation.FORM, + multiple=True, flag="--file", help="File to process; repeatable"), + Param("presigned_urls", location=ParamLocation.FORM, multiple=True, + flag="--presigned-url", + help="HTTPS AWS S3 presigned URL to fetch a file from; repeatable"), + Param("timeout", type=ParamType.INT, location=ParamLocation.FORM, default=0, + flag="--execution-timeout", + help="Seconds to wait server-side (0-300). 0 runs asynchronously"), + Param("include_metadata", type=ParamType.BOOL, location=ParamLocation.FORM, + default=False, + help="Include LLM/embedding usage and cost metadata in the result"), + Param("tags", location=ParamLocation.FORM, + help="Tag for this execution; must start with a letter (limit 1)"), + Param("llm_profile_id", type=ParamType.UUID, location=ParamLocation.FORM, + help="Override the tool's default LLM profile"), + Param("custom_data", type=ParamType.JSON, location=ParamLocation.FORM, + help="JSON object addressable in prompts as {{custom_data.key}}"), + Param("hitl_queue_name", location=ParamLocation.FORM, + help="Route results to this Human Quality Review queue instead of returning them"), + Param("save", client_side=True, + help="With --wait, write the one-shot result to this path before " + "exiting (strongly recommended: the result can be read only once)"), + ), + body=BodyKind.MULTIPART, + # ONE-SHOT poll: the status endpoint *is* the result store. The poll that + # first observes COMPLETED consumes the result, so that terminal poll's + # body is returned as the result and no second read is issued (a second + # read would 406). `status_field` lists `status` first because the status + # GET returns a top-level `status` (its `message` holds the result); the + # nested `execution_status` is the run POST's shape. Reading the wrong one + # made the terminal state go unrecognised (CAPTURE2 BUG 2). + poll=PollSpec( + status_endpoint="docstudio.deployment.status", + status_field=("status", "execution_status"), + terminal_success=("COMPLETED",), + terminal_failure=("ERROR", "STOPPED"), + handle_field="execution_id", + handle_param="execution_id", + one_shot=True, + ), + doc_source=f"{_DOCS}/api_execution.md", + examples=( + "unstract deployment run --api-name invoice-api --file invoice.pdf --wait --save out.json", + "unstract deployment run --api-name invoice-api --file a.pdf --file b.pdf", + "unstract deployment run --api-name invoice-api --file x.pdf --hitl-queue-name review", + ), + ), + Endpoint( + name="status", + group="docstudio", + subgroup="deployment", + method="GET", + path="/deployment/api/{org_id}/{api_name}/", + api=ApiGroup.DEPLOYMENT, + summary="Check the status of an execution and retrieve its result.", + description=( + f"Execution statuses: {', '.join(_EXECUTION_STATUSES)}.\n\n" + "ONE-SHOT: results are removed from the server once retrieved. A " + "second call returns HTTP 406 and exits 9 -- pass --save to persist " + "them on the first read.\n\n" + "Note: PENDING and EXECUTING currently return HTTP 422 rather than " + "200. This CLI reads the status from the response body, so that " + "server-side defect does not affect it." + ), + params=( + *_PATH_PARAMS, + Param("execution_id", type=ParamType.UUID, required=True, + help="Execution identifier returned by `deployment run`"), + Param("include_metadata", type=ParamType.BOOL, default=False, + help="Include LLM/embedding usage and cost metadata"), + Param("save", client_side=True, + help="Write the result to this path before exiting (recommended)"), + ), + doc_source=f"{_DOCS}/api_execution_status.md", + examples=( + "unstract deployment status --api-name invoice-api --execution-id --save out.json", + ), + ), + Endpoint( + name="highlight", + group="docstudio", + subgroup="deployment", + method="GET", + path="/deployment/api/{org_id}/{api_name}/highlight/", + api=ApiGroup.DEPLOYMENT, + summary="Fetch line coordinates for highlighting extracted values.", + description=( + "Requires 'Enable Highlight' on the exported tool. The whisper_hash " + "and line_numbers come from the execution result, at " + "result.metadata.whisper_hash and result.metadata.line_numbers." + ), + params=( + *_PATH_PARAMS, + Param("whisper_hash", required=True, + help="From result.metadata.whisper_hash in the execution result"), + Param("line_numbers", required=True, + help="Comma-separated line numbers, e.g. '2,3,6'"), + Param("text_extractor_name", required=True, + help="Text extractor adapter name, e.g. llm-whisperer-v2"), + ), + doc_source=f"{_DOCS}/api_get_highlight_data.md", + examples=( + "unstract deployment highlight --api-name inv --whisper-hash h " + "--line-numbers 2,3,6 --text-extractor-name llm-whisperer-v2", + ), + ), +) diff --git a/src/unstract_cli/endpoints/hitl.py b/src/unstract_cli/endpoints/hitl.py new file mode 100644 index 0000000..22967fa --- /dev/null +++ b/src/unstract_cli/endpoints/hitl.py @@ -0,0 +1,105 @@ +"""Human Quality Review (HITL) endpoints (SPEC.md §6.4). Enterprise feature. + +Authored from `unstract-docs/docs/unstract_platform/human_quality_review/`. + +Pushing *into* HITL is not a separate command: it is +`unstract deployment run --hitl-queue-name `. +""" + +from __future__ import annotations + +from unstract_cli.core.model import ApiGroup, Endpoint, Param, ParamLocation, ParamType + +_DOCS = "unstract-docs/docs/unstract_platform/human_quality_review" + +_ORG = Param( + "org_id", + location=ParamLocation.PATH, + # Same fallback as `deployment run`: the hitl block is its own, initially + # empty config section, but the organization is the same one (GOTCHAS #7). + default_from="hitl.org_id platform.org_id", + help="Organization identifier. Falls back to the platform block's org_id", +) + +_CLASS_ID = Param( + "class_id", + location=ParamLocation.PATH, + required=True, + help="Class (workflow) identifier, from the Download and Sync Manager", +) + + +ENDPOINTS: tuple[Endpoint, ...] = ( + Endpoint( + name="get", + group="docstudio", + subgroup="hitl approved", + method="GET", + path="/mr/api/{org_id}/approved/result/{class_id}/", + api=ApiGroup.HITL, + summary="Dequeue one approved result from the review queue.", + description=( + "DEQUEUE, not a read: each call removes one item from the approved " + "queue and returns it. The item is gone from the server afterwards, so " + "pass --save to persist it. Use `hitl bulk-download` to page through " + "results without consuming them." + ), + params=( + _ORG, + _CLASS_ID, + Param("hitl_queue_name", help="Queue name suffix used at deployment run time"), + Param("save", client_side=True, + help="Write the dequeued item to this path before exiting (recommended)"), + ), + doc_source=f"{_DOCS}/retrieve_approved_results.md", + examples=( + "unstract hitl approved get --class-id --save approved.json", + ), + ), + Endpoint( + name="bulk-download", + group="docstudio", + subgroup="hitl", + method="GET", + path="/mr/api/{org_id}/approved/result/{class_id}/", + api=ApiGroup.HITL, + summary="Page through approved results, optionally including file content.", + description=( + "Unlike `hitl approved get`, this is a paginated read. With " + "--download-files and a large page, the server may switch to an " + "asynchronous job and return a job id -- poll it with " + "`hitl download-status`." + ), + params=( + _ORG, + _CLASS_ID, + Param("page", type=ParamType.INT, default=1, help="Page number"), + Param("page_size", type=ParamType.INT, default=50, + help="Records per page (1-500)"), + Param("download_files", type=ParamType.BOOL, default=False, + help="Include file content in the response"), + Param("email", help="Email address to notify when an async download completes"), + Param("save", client_side=True, help="Write the response to this path"), + ), + doc_source=f"{_DOCS}/bulk_download.md", + examples=( + "unstract hitl bulk-download --class-id --page 1 --page-size 50", + ), + ), + Endpoint( + name="download-status", + group="docstudio", + subgroup="hitl", + method="GET", + path="/mr/api/{org_id}/approved/download-status/{job_id}/", + api=ApiGroup.HITL, + summary="Check the status of an asynchronous bulk download job.", + params=( + _ORG, + Param("job_id", location=ParamLocation.PATH, required=True, + help="Job identifier returned by `hitl bulk-download`"), + ), + doc_source=f"{_DOCS}/bulk_download.md", + examples=("unstract hitl download-status --job-id job-789",), + ), +) diff --git a/src/unstract_cli/endpoints/platform.py b/src/unstract_cli/endpoints/platform.py new file mode 100644 index 0000000..459b93a --- /dev/null +++ b/src/unstract_cli/endpoints/platform.py @@ -0,0 +1,1340 @@ +"""Unstract Platform Management API v1 endpoints (SPEC.md §6.3). + +Authored from `unstract-docs/docs/unstract_platform/api_documentation/versions/v1-*.mdx`. + +Base path is `{host}/api/v1/unstract/{org_id}`; auth is a platform API key as a +Bearer token. Key permission levels gate methods: `read` allows GET, `read_write` +everything but DELETE, `full_access` everything. + +Two upstream inconsistencies are reproduced here deliberately, because +"correcting" either produces a 404: + +* Trailing slashes are not uniform. Several Prompt Studio paths and the group + member-removal path genuinely have none; those records set + ``no_trailing_slash=True`` so a test can assert intent rather than treating the + omission as a typo. +* Profile creation uses ``profilemanager`` (one word) while profile CRUD uses + ``profile-manager`` (hyphenated). + +PATCH records are *derived* from their PUT counterparts via `derive_patch` +(P8): a parameter added to a PUT record cannot then be forgotten on the PATCH. +""" + +from __future__ import annotations + +from dataclasses import replace + +from unstract_cli.core.model import ( + ApiGroup, + AtLeastOneOf, + BodyKind, + Endpoint, + Param, + ParamLocation, + ParamType, + Permission, + PollSpec, + derive_patch, + with_params, +) + +_DOCS = "unstract-docs/docs/unstract_platform/api_documentation/versions" +_BASE = "/api/v1/unstract/{org_id}" + +_ORG = Param( + "org_id", + location=ParamLocation.PATH, + default_from="platform.org_id", + help="Organization identifier", +) + +#: Shared pagination flags (SPEC.md §6.3). +_PAGE = ( + Param("page", type=ParamType.INT, default=1, help="Page number"), + Param("page_size", type=ParamType.INT, default=50, + help="Results per page (max 1000)"), +) + +_DELETE_NOTE = ( + "Requires a platform API key with full_access permission: DELETE is blocked " + "for read and read_write keys at the middleware." +) + +#: Friendly resource name -> URL path segment (P3). The segment is not guessable +#: from the friendly name -- `api-deployment` lives at `api/deployment` -- so this +#: is an enum rather than free text, and a wrong value fails locally with exit 2 +#: instead of a confusing remote 404. +SHARE_RESOURCES = { + "adapter": "adapter", + "connector": "connector", + "workflow": "workflow", + "pipeline": "pipeline", + "api-deployment": "api/deployment", + "prompt-studio": "prompt-studio", +} + + +def _ep( + name: str, + method: str, + path: str, + summary: str, + params: tuple[Param, ...] = (), + *, + subgroup: str | None = None, + body: BodyKind = BodyKind.NONE, + doc: str = "v1-prompt-studio.mdx", + permission: Permission | None = None, + description: str = "", + examples: tuple[str, ...] = (), + constraints: tuple = (), + no_trailing_slash: bool = False, + table_columns: tuple[str, ...] = (), + require_response_fields: tuple[str, ...] = (), + poll: PollSpec | None = None, + doc_source: str | None = None, + doc_conflict: str | None = None, +) -> Endpoint: + """Construct a Platform endpoint, filling in the org-scoped base path. + + ``doc_source`` overrides the default ``{_DOCS}/{doc}`` provenance for the few + endpoints authored from the backend source rather than the public v1 docs + (the Output Manager and task-status routes have no ``.mdx`` page). Citing + their real source is honest -- exactly how API Hub records cite code -- and + lets the Skill's drift check report them as undocumented (info-only) with a + citation that points where they actually came from. + """ + return Endpoint( + name=name, + group="docstudio", + # Every Platform API command lives under `docstudio platform ...`. + subgroup=f"platform {subgroup}" if subgroup else "platform", + method=method, + path=f"{_BASE}{path}", + api=ApiGroup.PLATFORM, + summary=summary, + params=(_ORG, *params), + body=body, + doc_source=doc_source or f"{_DOCS}/{doc}", + permission=permission, + description=description, + examples=examples, + constraints=constraints, + no_trailing_slash=no_trailing_slash, + table_columns=table_columns, + require_response_fields=require_response_fields, + poll=poll, + doc_conflict=doc_conflict, + ) + + +# --------------------------------------------------------------------------- # +# Prompt Studio +# --------------------------------------------------------------------------- # + +_PS = "prompt-studio" +_TOOL_ID = Param("tool_id", type=ParamType.UUID, location=ParamLocation.PATH, + required=True, help="Prompt Studio project (tool) identifier") + +#: `prompt create` only: the backend's `create_prompt` persists the request body +#: verbatim and ignores the URL `pk`, so a `tool_id` absent from the body is saved +#: as NULL -- the prompt exists but links to no project and is unreachable (BUG 2). +#: Mirroring the path value into the body links it correctly. Verified this session. +_TOOL_ID_MIRRORED = replace(_TOOL_ID, mirror_to_body=True) + +#: `--wait` for fetch-response / single-pass. These return HTTP 202 with a +#: `task_id`; task-status reports completion (it does NOT return the value), and +#: the extracted output lands in the Output Manager, read by `output list` keyed +#: on the original request's `tool_id` -- not on the poll handle. `retrieve_carry` +#: forwards that tool_id, and `retrieve_omits_handle` keeps the task_id out of the +#: output-list call, which has no such parameter (IMPROVEMENT 3). +_PS_POLL = PollSpec( + status_endpoint="docstudio.platform.prompt-studio.task-status", + status_field="status", + terminal_success=("completed",), + terminal_failure=("failed",), + handle_field="task_id", + handle_param="task_id", + retrieve_endpoint="docstudio.platform.prompt-studio.output.list", + # Narrow the result to the exact prompt + document this call ran, rather than + # returning every row for the tool. fetch-response's `id` is the Output + # Manager's `prompt_id`; both endpoints already share the py_name `document_id` + # (output list exposes `document_manager` under the flag --document-id), so it + # carries across unchanged. + retrieve_carry=( + "tool_id", + ("id", "prompt_id"), + "document_id", + ), + retrieve_omits_handle=True, +) + +#: `--wait` for index-document. Same 202 {task_id, status} shape and the same +#: task-status route as fetch-response, but indexing produces no Output Manager +#: row -- there is nothing to retrieve, so the terminal status IS the result. +#: Without this, index-document was the one async command that forced a manual +#: poll loop while its siblings all had --wait (GOTCHAS #8). +_PS_INDEX_POLL = replace( + _PS_POLL, + retrieve_endpoint=None, + retrieve_carry=(), +) + +#: Single-pass runs all prompts, so its retrieve is intentionally tool-wide; it +#: only needs to ask for the single-pass rows (and the document it ran against). +_PS_SINGLE_PASS_POLL = replace( + _PS_POLL, + retrieve_carry=("tool_id", "document_id"), + retrieve_extra=(("is_single_pass_extract", True),), +) + +_PS_FIELDS: tuple[Param, ...] = ( + Param("tool_name", location=ParamLocation.BODY, required=True, + help="Project name, unique within the organization"), + Param("description", location=ParamLocation.BODY, required=True, + help="Project description"), + Param("author", location=ParamLocation.BODY, required=True, help="Project author"), + Param("icon", location=ParamLocation.BODY, help="Project icon"), + Param("preamble", location=ParamLocation.BODY, help="Text prepended to every prompt"), + Param("postamble", location=ParamLocation.BODY, help="Text appended to every prompt"), + Param("summarize_context", type=ParamType.BOOL, location=ParamLocation.BODY, + default=False, help="Summarize context before extraction"), + Param("single_pass_extraction_mode", type=ParamType.BOOL, location=ParamLocation.BODY, + default=False, help="Run all prompts in a single LLM call"), + Param("enable_challenge", type=ParamType.BOOL, location=ParamLocation.BODY, + default=False, help="Enable LLM challenge for extraction validation"), + # GOTCHAS #2: an exported tool's settings schema lists challenge_llm as + # required even when enable_challenge is false, so a tool instance whose + # metadata.challenge_llm is "" fails deploy-time validation with a 422 that + # only surfaces at `deployment run`. Setting it on the project before + # `export-tool` means the exported metadata carries a real adapter id. + Param("challenge_llm", type=ParamType.UUID, location=ParamLocation.BODY, + help="LLM adapter used to challenge extractions. Set this before " + "`export-tool` even with --no-enable-challenge: the exported tool " + "requires a non-empty challenge_llm at deploy time, and an empty one " + "fails `deployment run` with a 422"), + Param("monitor_llm", type=ParamType.UUID, location=ParamLocation.BODY, + help="LLM adapter used for monitoring. Defaults server-side to the " + "default profile's LLM"), + Param("enable_highlight", type=ParamType.BOOL, location=ParamLocation.BODY, + default=False, help="Record line metadata for source highlighting"), + Param("custom_data", type=ParamType.JSON, location=ParamLocation.BODY, + help="JSON object addressable in prompts as {{custom_data.key}}"), + Param("shared_users", type=ParamType.INT, location=ParamLocation.BODY, multiple=True, + replace_semantics=True, help="User IDs to share with"), + Param("shared_to_org", type=ParamType.BOOL, location=ParamLocation.BODY, default=False, + help="Share with the whole organization"), +) + +_PROMPT_FIELDS: tuple[Param, ...] = ( + Param("prompt_key", location=ParamLocation.BODY, required=True, + help="Output key for this prompt, unique within the project"), + Param("enforce_type", location=ParamLocation.BODY, default="text", + choices=["text", "number", "email", "date", "boolean", "json", "line-item", "table"], + help="Expected output type. Note: 'date' normalization is locale-ambiguous " + "and reads DD/MM/YYYY sources as MM/DD/YYYY (01/08/2025 -> 2025-01-08); " + "prefer 'text' for non-US date formats"), + Param("prompt", location=ParamLocation.BODY, help="Prompt text sent to the LLM"), + Param("sequence_number", type=ParamType.INT, location=ParamLocation.BODY, + help="Position within the project"), + Param("prompt_type", location=ParamLocation.BODY, choices=["PROMPT", "NOTES"], + help="PROMPT extracts a value; NOTES is an annotation"), + Param("active", type=ParamType.BOOL, location=ParamLocation.BODY, default=True, + help="Whether the prompt runs. Boolean flags are --active / --no-active; " + "`--active true` is not valid syntax"), + # The load-bearing field for GOTCHAS #1. `fetch-response` resolves the LLM + # profile from the PROMPT's own profile_manager FK and never falls back to + # the project default, so a prompt created without it is unrunnable -- and + # the resulting error names the *project* default, which is genuinely set. + Param("profile_manager", type=ParamType.UUID, location=ParamLocation.BODY, + help="LLM profile this prompt runs with. Set it at creation: fetch-response " + "reads this field and does NOT fall back to the project default, so a " + "prompt without it fails with 'Default LLM profile is not configured'"), +) + +_PROFILE_FIELDS: tuple[Param, ...] = ( + Param("profile_name", location=ParamLocation.BODY, required=True, + help="Profile name, unique within the project"), + # GOTCHAS #3 asked for these to be optional when --chunk-size 0 ("no RAG"). + # They are NOT, and cannot be made so client-side: ProfileManager declares + # both FKs null=False, and the serializer is `fields = "__all__"`, so DRF + # derives required=True and the server rejects a profile without them + # regardless of chunk_size. Dropping the local check would only trade a fast + # exit-2 for a slower remote 400, so the requirement stays and the help says + # what to pass instead. + Param("vector_store", type=ParamType.UUID, location=ParamLocation.BODY, required=True, + help="Vector DB adapter id. Required even with --chunk-size 0: the server " + "rejects a profile without one. With chunk_size=0 it is stored but " + "never queried, so any valid vector-DB adapter id will do"), + Param("embedding_model", type=ParamType.UUID, location=ParamLocation.BODY, required=True, + help="Embedding adapter id. Required even with --chunk-size 0, for the same " + "reason as --vector-store: stored, but not used when RAG is off"), + Param("llm", type=ParamType.UUID, location=ParamLocation.BODY, required=True, + help="LLM adapter id"), + Param("x2text", type=ParamType.UUID, location=ParamLocation.BODY, required=True, + help="Text extractor adapter id"), + Param("chunk_size", type=ParamType.INT, location=ParamLocation.BODY, + help="Chunk size for RAG. 0 = whole-document / no-RAG: skips embedding " + "and the vector DB entirely. Use 0 for short documents, or when the " + "vector DB is unavailable"), + Param("chunk_overlap", type=ParamType.INT, location=ParamLocation.BODY, + help="Overlap between chunks. Set 0 alongside chunk_size=0"), + Param("retrieval_strategy", location=ParamLocation.BODY, default="simple", + choices=["simple", "subquestion", "fusion", "recursive", "router", + "keyword_table", "automerging"], + help="Retrieval strategy for RAG"), + Param("similarity_top_k", type=ParamType.INT, location=ParamLocation.BODY, + help="Number of chunks to retrieve"), +) + +_ps_update = _ep( + "update", "PUT", "/prompt-studio/{tool_id}/", "Replace a Prompt Studio project.", + (_TOOL_ID, *_PS_FIELDS), subgroup=_PS, body=BodyKind.JSON, + permission=Permission.READ_WRITE, +) + +_prompt_update = _ep( + "update", "PUT", "/prompt-studio/prompt/{prompt_id}/", "Replace a prompt.", + (Param("prompt_id", type=ParamType.UUID, location=ParamLocation.PATH, required=True, + help="Prompt identifier"), *_PROMPT_FIELDS), + subgroup=f"{_PS} prompt", body=BodyKind.JSON, permission=Permission.READ_WRITE, +) + +_profile_update = _ep( + "update", "PUT", "/prompt-studio/profile-manager/{profile_id}/", + "Replace an LLM profile.", + (Param("profile_id", type=ParamType.UUID, location=ParamLocation.PATH, required=True, + help="Profile identifier"), *_PROFILE_FIELDS), + subgroup=f"{_PS} profile", body=BodyKind.JSON, permission=Permission.READ_WRITE, +) + +_PROMPT_STUDIO: tuple[Endpoint, ...] = ( + _ep("list", "GET", "/prompt-studio/", "List Prompt Studio projects.", + subgroup=_PS, permission=Permission.READ, + table_columns=("tool_id", "tool_name", "author", "created_by_email"), + examples=("unstract platform prompt-studio list",)), + _ep("create", "POST", "/prompt-studio/", "Create a Prompt Studio project.", + _PS_FIELDS, subgroup=_PS, body=BodyKind.JSON, permission=Permission.READ_WRITE, + examples=("unstract platform prompt-studio create --tool-name Invoices " + "--description 'Invoice extraction' --author me",)), + _ep("get", "GET", "/prompt-studio/{tool_id}/", "Show one Prompt Studio project.", + (_TOOL_ID,), subgroup=_PS, permission=Permission.READ), + _ps_update, + derive_patch(_ps_update, summary="Partially update a Prompt Studio project."), + _ep("delete", "DELETE", "/prompt-studio/{tool_id}/", "Delete a Prompt Studio project.", + (_TOOL_ID,), subgroup=_PS, permission=Permission.FULL_ACCESS, + description=f"{_DELETE_NOTE} Returns 409 if the tool is exported and in use.\n\n" + "This is also the ONLY way to remove an exported registry entry: " + "the registry is read-only over the API (it exposes list and " + "settings-schema, with no DELETE route), and deleting the project " + "cascades to the entry it published. Detach the tool from any " + "workflow first (`workflow tool remove`) or this returns 409 " + "(GOTCHAS #9)."), + _ep("export-project", "GET", "/prompt-studio/project-transfer/{tool_id}", + "Export a project as a JSON file.", + (_TOOL_ID, Param("save", client_side=True, help="Write the export to this path")), + subgroup=_PS, permission=Permission.READ, no_trailing_slash=True, + examples=("unstract platform prompt-studio export-project --tool-id --save proj.json",)), + _ep("import-project", "POST", "/prompt-studio/project-transfer/", + "Import a project from an exported JSON file.", + (Param("file", type=ParamType.FILE, location=ParamLocation.FORM, required=True, + help="Previously exported project JSON"),), + subgroup=_PS, body=BodyKind.MULTIPART, permission=Permission.READ_WRITE), + _ep("sync-prompts", "POST", "/prompt-studio/{tool_id}/sync-prompts/", + "Sync prompts from an export into an existing project.", + (_TOOL_ID, + Param("data", type=ParamType.JSON, location=ParamLocation.BODY, required=True, + help="Export JSON containing a `prompts` key"), + Param("create_copy", type=ParamType.BOOL, location=ParamLocation.BODY, + default=False, help="Back the project up before syncing")), + subgroup=_PS, body=BodyKind.JSON, permission=Permission.READ_WRITE), + _ep("export-tool", "POST", "/prompt-studio/export/{tool_id}", + "Export a project to the tool registry for deployment.", + (_TOOL_ID, + Param("is_shared_with_org", type=ParamType.BOOL, location=ParamLocation.BODY, + help="Share the exported tool with the organization"), + Param("user_id", type=ParamType.INT, location=ParamLocation.BODY, multiple=True, + help="User IDs to share the exported tool with"), + Param("force_export", type=ParamType.BOOL, location=ParamLocation.BODY, + default=False, help="Export even if validation warns")), + subgroup=_PS, body=BodyKind.JSON, permission=Permission.READ_WRITE, + no_trailing_slash=True, + description="Publishes the project to the tool registry, where it gets a NEW " + "registry id (`function_name`) that is NOT the Prompt Studio " + "tool_id. This call does not return it -- find it with " + "`tool registry list`, or `api-deployment by-prompt-studio-tool` " + "once deployed (GOTCHAS #5).\n\n" + "Before exporting, set the project's --challenge-llm (see " + "`prompt-studio patch`). The exported tool requires a non-empty " + "challenge_llm at deploy time even when enable_challenge is false; " + "if it is empty the attached tool instance fails validation and " + "`deployment run` ends in ERROR with a 422 -- the failure surfaces " + "only at that last step (GOTCHAS #2).", + examples=("unstract platform prompt-studio export-tool --tool-id ",)), + _ep("export-info", "GET", "/prompt-studio/export/{tool_id}", + "Show export status for a project.", (_TOOL_ID,), subgroup=_PS, + permission=Permission.READ, no_trailing_slash=True, + description="Returns 204 No Content if the project has never been exported."), + _ep("upload", "POST", "/prompt-studio/file/{tool_id}", + "Upload documents to a project.", + (_TOOL_ID, Param("file", type=ParamType.FILE, location=ParamLocation.FORM, + required=True, multiple=True, help="File to upload; repeatable")), + subgroup=f"{_PS} file", body=BodyKind.MULTIPART, permission=Permission.READ_WRITE, + no_trailing_slash=True), + _ep("get", "GET", "/prompt-studio/file/{tool_id}", "Fetch a document's contents.", + (_TOOL_ID, + Param("document_id", type=ParamType.UUID, required=True, help="Document identifier"), + Param("view_type", choices=["ORIGINAL", "EXTRACT", "SUMMARIZE"], + help="Which rendition to fetch")), + subgroup=f"{_PS} file", permission=Permission.READ, no_trailing_slash=True), + _ep("delete", "DELETE", "/prompt-studio/file/{tool_id}", + "Delete a document from a project.", + (_TOOL_ID, Param("document_id", type=ParamType.UUID, location=ParamLocation.BODY, + required=True, help="Document identifier")), + subgroup=f"{_PS} file", body=BodyKind.JSON, permission=Permission.FULL_ACCESS, + description=_DELETE_NOTE, no_trailing_slash=True), + _ep("create", "POST", "/prompt-studio/prompt-studio-prompt/{tool_id}/", + "Create a prompt in a project.", (_TOOL_ID_MIRRORED, *_PROMPT_FIELDS), + subgroup=f"{_PS} prompt", body=BodyKind.JSON, permission=Permission.READ_WRITE, + require_response_fields=("tool_id",), + description="Sends tool_id in the body as well as the path, so the prompt " + "links to the project rather than being orphaned (tool_id: null).\n\n" + "Pass --profile-manager unless you intend to supply it on every " + "run: `fetch-response` resolves the LLM profile from THIS field " + "and does not fall back to the project's default profile. A prompt " + "created without it fails at run time with 'Default LLM profile is " + "not configured' even when `profile set-default` succeeded " + "(GOTCHAS #1).", + examples=("unstract platform prompt-studio prompt create --tool-id " + "--prompt-key invoice_no --prompt 'What is the invoice number?' " + "--profile-manager ",)), + _ep("get", "GET", "/prompt-studio/prompt/{prompt_id}/", "Show one prompt.", + (Param("prompt_id", type=ParamType.UUID, location=ParamLocation.PATH, + required=True, help="Prompt identifier"),), + subgroup=f"{_PS} prompt", permission=Permission.READ), + _prompt_update, + derive_patch(_prompt_update, summary="Partially update a prompt."), + _ep("delete", "DELETE", "/prompt-studio/prompt/{prompt_id}/", "Delete a prompt.", + (Param("prompt_id", type=ParamType.UUID, location=ParamLocation.PATH, + required=True, help="Prompt identifier"),), + subgroup=f"{_PS} prompt", permission=Permission.FULL_ACCESS, description=_DELETE_NOTE), + _ep("reorder", "POST", "/prompt-studio/prompt/reorder/", "Reorder a prompt.", + (Param("start_sequence_number", type=ParamType.INT, location=ParamLocation.BODY, + required=True, help="Current position"), + Param("end_sequence_number", type=ParamType.INT, location=ParamLocation.BODY, + required=True, help="Target position"), + Param("prompt_id", type=ParamType.UUID, location=ParamLocation.BODY, + required=True, help="Prompt to move")), + subgroup=f"{_PS} prompt", body=BodyKind.JSON, permission=Permission.READ_WRITE), + _ep("list", "GET", "/prompt-studio/prompt-studio-profile/{tool_id}/", + "List LLM profiles for a project.", (_TOOL_ID,), subgroup=f"{_PS} profile", + permission=Permission.READ), + _ep("set-default", "PATCH", "/prompt-studio/prompt-studio-profile/{tool_id}/", + "Set the project's default LLM profile.", + (_TOOL_ID, Param("default_profile", type=ParamType.UUID, location=ParamLocation.BODY, + required=True, help="Profile to make default")), + subgroup=f"{_PS} profile", body=BodyKind.JSON, permission=Permission.READ_WRITE), + # Note the path: `profilemanager` (one word) on create, `profile-manager` + # (hyphenated) on CRUD. This asymmetry is upstream, and is load-bearing. + _ep("create", "POST", "/prompt-studio/profilemanager/{tool_id}", + "Create an LLM profile (maximum 4 per project).", + (_TOOL_ID, *_PROFILE_FIELDS), subgroup=f"{_PS} profile", body=BodyKind.JSON, + permission=Permission.READ_WRITE, no_trailing_slash=True, + description="With --chunk-size 0 the document is sent to the LLM whole (no RAG) " + "and neither the vector DB nor the embedding model is queried -- but " + "the server still REQUIRES both fields, so pass any valid adapter id " + "for them (GOTCHAS #3). Use chunk_size=0 for short documents, or " + "when the vector DB is unavailable.", + examples=("unstract platform prompt-studio profile create --tool-id " + "--profile-name direct --llm --x2text " + "--vector-store --embedding-model " + "--chunk-size 0 --chunk-overlap 0",)), + _ep("get", "GET", "/prompt-studio/profile-manager/{profile_id}/", "Show one LLM profile.", + (Param("profile_id", type=ParamType.UUID, location=ParamLocation.PATH, + required=True, help="Profile identifier"),), + subgroup=f"{_PS} profile", permission=Permission.READ), + _profile_update, + derive_patch(_profile_update, summary="Partially update an LLM profile."), + _ep("delete", "DELETE", "/prompt-studio/profile-manager/{profile_id}/", + "Delete an LLM profile.", + (Param("profile_id", type=ParamType.UUID, location=ParamLocation.PATH, + required=True, help="Profile identifier"),), + subgroup=f"{_PS} profile", permission=Permission.FULL_ACCESS, description=_DELETE_NOTE), + _ep("index-document", "POST", "/prompt-studio/index-document/{tool_id}", + "Index a document for retrieval.", + (_TOOL_ID, Param("document_id", type=ParamType.UUID, location=ParamLocation.BODY, + required=True, help="Document identifier")), + subgroup=_PS, body=BodyKind.JSON, permission=Permission.READ_WRITE, + no_trailing_slash=True, poll=_PS_INDEX_POLL, + description="Returns HTTP 202 {task_id, run_id, status:accepted}. Use --wait to " + "poll to completion instead of calling `task-status` in a loop. " + "Indexing writes no Output Manager row, so --wait returns the final " + "task status rather than an extracted value.", + examples=("unstract platform prompt-studio index-document --tool-id " + "--document-id --wait",)), + _ep("fetch-response", "POST", "/prompt-studio/fetch_response/{tool_id}", + "Run one prompt against a document.", + (_TOOL_ID, + Param("document_id", type=ParamType.UUID, location=ParamLocation.BODY, + required=True, help="Document to run against"), + Param("id", type=ParamType.UUID, location=ParamLocation.BODY, required=True, + help="Prompt identifier"), + Param("run_id", type=ParamType.UUID, location=ParamLocation.BODY, + help="Execution tracking identifier"), + Param("profile_manager", type=ParamType.UUID, location=ParamLocation.BODY, + help="LLM profile to run with. Required unless the prompt itself was " + "created with a profile_manager -- there is no fallback to the " + "project default")), + subgroup=_PS, body=BodyKind.JSON, permission=Permission.READ_WRITE, + no_trailing_slash=True, poll=_PS_POLL, + description="Returns HTTP 202 {task_id, run_id, status:accepted}; the extracted " + "value is written to the Output Manager. Use --wait to poll to " + "completion and return the results, or read them with " + "`prompt-studio output list`.\n\n" + "If this fails with 'Default LLM profile is not configured' while " + "`profile set-default` and `prompt-studio get` both show a default: " + "the message is misleading. The server resolves the profile from the " + "PROMPT's own profile_manager field and never consults the project " + "default, so the real cause is a prompt created without one. Fix it " + "permanently with `prompt patch --prompt-id --profile-manager " + "`, or pass --profile-manager on each run (GOTCHAS #1)."), + _ep("single-pass", "POST", "/prompt-studio/single-pass-extraction/{tool_id}", + "Run all active prompts in a single pass.", + (_TOOL_ID, + Param("document_id", type=ParamType.UUID, location=ParamLocation.BODY, + required=True, help="Document to run against"), + Param("run_id", type=ParamType.UUID, location=ParamLocation.BODY, + help="Execution tracking identifier")), + subgroup=_PS, body=BodyKind.JSON, permission=Permission.READ_WRITE, + no_trailing_slash=True, poll=_PS_SINGLE_PASS_POLL, + description="Requires single_pass_extraction_mode enabled on the project. " + "Returns HTTP 202; use --wait to poll and return results, or read " + "them with `prompt-studio output list --is-single-pass-extract`."), + _ep("task-status", "GET", "/prompt-studio/{tool_id}/task-status/{task_id}", + "Check the status of an async prompt-studio task.", + (_TOOL_ID, + Param("task_id", location=ParamLocation.PATH, required=True, + help="Task identifier returned by fetch-response / single-pass / " + "index-document")), + subgroup=_PS, permission=Permission.READ, no_trailing_slash=True, + description="Status is one of: processing, completed, failed. The extracted " + "value is not returned here -- read it with `prompt-studio output " + "list`.\n\nNeeds BOTH --task-id and --tool-id: the route is " + "/{tool_id}/task-status/{task_id}, and the tool_id is the same one " + "passed to the call that returned the task_id. Prefer --wait on " + "that call, which polls this for you.", + examples=("unstract platform prompt-studio task-status --tool-id " + "--task-id ",), + doc_source="backend/prompt_studio/prompt_studio_core_v2/urls.py"), + _ep("list", "GET", "/prompt-studio/prompt-output/", + "List extraction results for a project.", + (Param("tool_id", type=ParamType.UUID, required=True, + help="Prompt Studio project (tool) identifier"), + Param("prompt_id", type=ParamType.UUID, help="Filter to one prompt"), + Param("document_manager", type=ParamType.UUID, flag="--document-id", + help="Filter to one uploaded document"), + Param("profile_manager", type=ParamType.UUID, flag="--profile-id", + help="Filter to one LLM profile"), + Param("is_single_pass_extract", type=ParamType.BOOL, default=False, + help="Return single-pass results instead of per-prompt")), + subgroup=f"{_PS} output", permission=Permission.READ, + description="Reads the Prompt Studio Output Manager, where fetch-response and " + "single-pass write their results. Each row carries prompt_id, output, " + "context and modified_at; take the latest row per prompt_id.", + table_columns=("prompt_id", "prompt_key", "output", "modified_at"), + examples=("unstract platform prompt-studio output list --tool-id ",), + doc_source="backend/prompt_studio/prompt_studio_output_manager_v2/urls.py"), + _ep("latest", "GET", "/prompt-studio/prompt-output/latest-by-keys/", + "Latest result per prompt key for a project.", + (Param("tool_id", type=ParamType.UUID, required=True, + help="Prompt Studio project (tool) identifier"),), + subgroup=f"{_PS} output", permission=Permission.READ, + description="Returns the latest output keyed by prompt_key. May return {} in some " + "cases where `output list` still has the data -- prefer `output list` " + "if this is empty.", + examples=("unstract platform prompt-studio output latest --tool-id ",), + doc_source="backend/prompt_studio/prompt_studio_output_manager_v2/urls.py"), + _ep("users", "GET", "/prompt-studio/users/{tool_id}", "List users a project is shared with.", + (_TOOL_ID,), subgroup=_PS, permission=Permission.READ, no_trailing_slash=True), + _ep("check-deployment-usage", "GET", "/prompt-studio/{tool_id}/check_deployment_usage/", + "Check whether an exported tool is used by deployments.", (_TOOL_ID,), + subgroup=_PS, permission=Permission.READ), + _ep("select-choices", "GET", "/prompt-studio/select_choices/", + "List dropdown values for Prompt Studio fields.", subgroup=_PS, + permission=Permission.READ), + _ep("adapter-choices", "GET", "/prompt-studio/adapter-choices/", + "List adapters available for LLM profiles.", subgroup=_PS, + permission=Permission.READ, + description="Known to return 500 server_error in some organizations " + "(GOTCHAS #10). If it does, enumerate adapters directly instead: " + "`adapter list --adapter-type LLM` gives the ids that " + "`profile create --llm` expects, filtered by kind.", + examples=("unstract platform adapter list --adapter-type LLM",)), + _ep("retrieval-strategies", "GET", "/prompt-studio/{tool_id}/get_retrieval_strategies/", + "List retrieval strategies available to a project.", (_TOOL_ID,), + subgroup=_PS, permission=Permission.READ), +) + + +# --------------------------------------------------------------------------- # +# Workflows +# --------------------------------------------------------------------------- # + +_WF_ID = Param("id", type=ParamType.UUID, location=ParamLocation.PATH, required=True, + help="Workflow identifier") + +_WF_FIELDS: tuple[Param, ...] = ( + Param("workflow_name", location=ParamLocation.BODY, required=True, + help="Workflow name, unique per organization (max 128 chars)"), + Param("description", location=ParamLocation.BODY, help="Description (max 490 chars)"), + Param("deployment_type", location=ParamLocation.BODY, default="DEFAULT", + choices=["DEFAULT", "ETL", "TASK", "API", "APP"], help="Deployment type"), + Param("source_settings", type=ParamType.JSON, location=ParamLocation.BODY, + help="Source connector configuration"), + Param("destination_settings", type=ParamType.JSON, location=ParamLocation.BODY, + help="Destination connector configuration"), + Param("max_file_execution_count", type=ParamType.INT, location=ParamLocation.BODY, + help="Maximum executions per file (minimum 1)"), + Param("shared_to_org", type=ParamType.BOOL, location=ParamLocation.BODY, default=False, + help="Share with the whole organization"), + Param("shared_users", type=ParamType.INT, location=ParamLocation.BODY, multiple=True, + replace_semantics=True, help="User IDs to share with"), +) + +_wf_update = _ep("update", "PUT", "/workflow/{id}/", "Replace a workflow.", + (_WF_ID, *_WF_FIELDS), subgroup="workflow", body=BodyKind.JSON, + doc="v1-workflows.mdx", permission=Permission.READ_WRITE) + +_WORKFLOWS: tuple[Endpoint, ...] = ( + _ep("list", "GET", "/workflow/", "List workflows.", + (Param("project", help="Filter by project id"), + Param("workflow_owner", help="Filter by owner user id"), + Param("is_active", type=ParamType.BOOL, help="Filter by active state"), + Param("order_by", choices=["asc", "desc"], help="Sort by modified_at"), + *_PAGE), + subgroup="workflow", doc="v1-workflows.mdx", permission=Permission.READ, + table_columns=("id", "workflow_name", "deployment_type", "is_active"), + examples=("unstract platform workflow list",)), + _ep("create", "POST", "/workflow/", "Create a workflow.", _WF_FIELDS, + subgroup="workflow", body=BodyKind.JSON, doc="v1-workflows.mdx", + permission=Permission.READ_WRITE), + _ep("get", "GET", "/workflow/{id}/", "Show one workflow.", (_WF_ID,), + subgroup="workflow", doc="v1-workflows.mdx", permission=Permission.READ), + _wf_update, + derive_patch(_wf_update, summary="Partially update a workflow."), + _ep("delete", "DELETE", "/workflow/{id}/", "Delete a workflow.", (_WF_ID,), + subgroup="workflow", doc="v1-workflows.mdx", permission=Permission.FULL_ACCESS, + description=f"{_DELETE_NOTE} Only the workflow owner may delete."), + _ep("execute", "POST", "/workflow/execute/", "Execute a workflow.", + (Param("workflow_id", type=ParamType.UUID, location=ParamLocation.BODY, + required=True, help="Workflow to execute"), + Param("execution_action", location=ParamLocation.BODY, + choices=["START", "NEXT", "STOP", "CONTINUE"], help="Execution action"), + Param("execution_id", type=ParamType.UUID, location=ParamLocation.BODY, + help="Required for NEXT, STOP and CONTINUE"), + Param("log_guid", type=ParamType.UUID, location=ParamLocation.BODY, + help="Correlates log entries"), + # The API field is `files`; the flag reads better singular since it is + # repeated once per file. + Param("files", type=ParamType.FILE, location=ParamLocation.FORM, + multiple=True, flag="--file", + help="File to process, for API-mode workflows; repeatable")), + subgroup="workflow", body=BodyKind.JSON, doc="v1-workflows.mdx", + permission=Permission.READ_WRITE), + _ep("toggle-active", "PUT", "/workflow/active/{id}/", "Toggle a workflow's active state.", + (_WF_ID,), subgroup="workflow", doc="v1-workflows.mdx", + permission=Permission.READ_WRITE), + _ep("can-update", "GET", "/workflow/{id}/can-update/", + "Check whether the current user may update a workflow.", (_WF_ID,), + subgroup="workflow", doc="v1-workflows.mdx", permission=Permission.READ), + _ep("clear-file-marker", "GET", "/workflow/{id}/clear-file-marker/", + "Clear file processing markers so files can be reprocessed.", (_WF_ID,), + subgroup="workflow", doc="v1-workflows.mdx", permission=Permission.READ_WRITE, + description=( + "NOTE: this is a mutating GET request -- that is how the API defines it. " + "It changes server state despite the method." + )), + _ep("schema", "GET", "/workflow/schema/", "Show the connector configuration schema.", + (Param("type", default="src", choices=["src", "dest"], help="Endpoint side"), + Param("entity", default="file", choices=["file", "api", "db"], help="Entity type")), + subgroup="workflow", doc="v1-workflows.mdx", permission=Permission.READ), + _ep("users", "GET", "/workflow/{id}/users/", "List users a workflow is shared with.", + (_WF_ID,), subgroup="workflow", doc="v1-workflows.mdx", permission=Permission.READ), + _ep("list", "GET", "/workflow/{id}/execution/", "List executions of a workflow.", + (_WF_ID, *_PAGE), subgroup="workflow execution", doc="v1-workflows.mdx", + permission=Permission.READ, + table_columns=("id", "status", "total_files", "execution_time")), + _ep("get", "GET", "/workflow/execution/{id}/", "Show one execution.", + (Param("id", type=ParamType.UUID, location=ParamLocation.PATH, required=True, + help="Execution identifier"),), + subgroup="workflow execution", doc="v1-workflows.mdx", permission=Permission.READ), + _ep("logs", "GET", "/workflow/execution/{id}/logs/", "Show logs for an execution.", + (Param("id", type=ParamType.UUID, location=ParamLocation.PATH, required=True, + help="Execution identifier"), + Param("file_execution_id", help="Filter by file execution; pass 'null' for non-file logs"), + Param("log_level", default="INFO", choices=["DEBUG", "INFO", "WARN", "ERROR"], + help="Minimum log level"), + Param("ordering", help="Ordering field, e.g. event_time"), + *_PAGE), + subgroup="workflow execution", doc="v1-workflows.mdx", permission=Permission.READ), + _ep("list", "GET", "/workflow/{workflow_id}/file-histories/", "List file histories.", + (Param("workflow_id", type=ParamType.UUID, location=ParamLocation.PATH, + required=True, help="Workflow identifier"), + Param("status", help="Comma-separated statuses to filter by"), + Param("execution_count_min", type=ParamType.INT, help="Minimum execution count"), + Param("execution_count_max", type=ParamType.INT, help="Maximum execution count"), + Param("file_path", help="File path prefix"), + *_PAGE), + subgroup="workflow file-history", doc="v1-workflows.mdx", permission=Permission.READ), + _ep("get", "GET", "/workflow/{workflow_id}/file-histories/{id}/", + "Show one file history entry.", + (Param("workflow_id", type=ParamType.UUID, location=ParamLocation.PATH, + required=True, help="Workflow identifier"), + Param("id", type=ParamType.UUID, location=ParamLocation.PATH, required=True, + help="File history identifier")), + subgroup="workflow file-history", doc="v1-workflows.mdx", permission=Permission.READ), + _ep("delete", "DELETE", "/workflow/{workflow_id}/file-histories/{id}/", + "Delete one file history entry.", + (Param("workflow_id", type=ParamType.UUID, location=ParamLocation.PATH, + required=True, help="Workflow identifier"), + Param("id", type=ParamType.UUID, location=ParamLocation.PATH, required=True, + help="File history identifier")), + subgroup="workflow file-history", doc="v1-workflows.mdx", + permission=Permission.FULL_ACCESS, description=_DELETE_NOTE), + _ep("clear", "POST", "/workflow/{workflow_id}/file-histories/clear/", + "Bulk delete file histories matching filters.", + (Param("workflow_id", type=ParamType.UUID, location=ParamLocation.PATH, + required=True, help="Workflow identifier"), + Param("ids", type=ParamType.UUID, location=ParamLocation.BODY, multiple=True, + help="Specific entries to delete (max 100)"), + Param("status", location=ParamLocation.BODY, multiple=True, + help="Statuses to delete"), + Param("execution_count_min", type=ParamType.INT, location=ParamLocation.BODY, + help="Minimum execution count"), + Param("execution_count_max", type=ParamType.INT, location=ParamLocation.BODY, + help="Maximum execution count"), + Param("file_path", location=ParamLocation.BODY, help="File path prefix")), + subgroup="workflow file-history", body=BodyKind.JSON, doc="v1-workflows.mdx", + permission=Permission.FULL_ACCESS, + constraints=(AtLeastOneOf(("ids", "status", "execution_count_min", + "execution_count_max", "file_path")),), + description=( + "At least one filter is required: an unfiltered clear would delete every " + "file history for the workflow." + )), +) + + +# --------------------------------------------------------------------------- # +# Workflow assembly: tool instances + endpoint configuration +# +# These are the two steps that turn a bare workflow into a deployable one, and +# they have no public v1 docs page -- `doc_source` cites the backend routes, the +# way API Hub records cite source. `workflow create` auto-creates SOURCE and +# DESTINATION endpoints but leaves their connection_type null; an API deployment +# then rejects the workflow until both are set to "API". Attaching the exported +# tool is `POST /tool_instance/` keyed by the tool's *registry* id (the +# `function_name` from `tool registry list`), NOT the Prompt Studio tool_id. +# (CAPTURE2 GAP 1.) +# --------------------------------------------------------------------------- # + +_WORKFLOW_ASSEMBLY: tuple[Endpoint, ...] = ( + _ep("list", "GET", "/tool/", "List tools in the registry (for attaching to a workflow).", + subgroup="tool registry", permission=Permission.READ, + doc_source="backend/tool_instance_v2/urls.py", + description="Each tool's `function_name` is its registry id -- the value " + "`workflow tool add --tool` expects. This is NOT the Prompt Studio " + "tool_id; `prompt-studio export-tool` publishes a project here first.\n\n" + "The registry carries no back-reference to the Prompt Studio " + "tool_id and the endpoint accepts no filters, so after exporting " + "you must match the entry by its `name`, which is the project's " + "tool_name (GOTCHAS #5). Give projects distinct names, or the " + "match is ambiguous.", + table_columns=("function_name", "name", "description"), + examples=("unstract platform tool registry list",)), + _ep("settings-schema", "GET", "/tool_settings_schema/", + "Show the settings JSON schema for a registry tool.", + (Param("function_name", required=True, + help="Registry id from `tool registry list`"),), + subgroup="tool registry", permission=Permission.READ, + doc_source="backend/tool_instance_v2/urls.py", + description="Reveals which adapter settings the tool requires at deploy time. " + "Note: an exported tool may list `challenge_llm` as required even " + "when the project has enable_challenge=false; a tool instance whose " + "metadata.challenge_llm is empty then fails deployment validation " + "(CAPTURE2 BUG 4). Set it with `workflow tool set-metadata`."), + _ep("list", "GET", "/tool_instance/", "List tools attached to workflows.", + (Param("workflow", type=ParamType.UUID, help="Filter to one workflow"),), + subgroup="workflow tool", permission=Permission.READ, + doc_source="backend/tool_instance_v2/urls.py", + table_columns=("id", "tool_id", "step", "workflow")), + _ep("add", "POST", "/tool_instance/", "Attach a registry tool to a workflow.", + (Param("workflow_id", type=ParamType.UUID, location=ParamLocation.BODY, + required=True, flag="--workflow", help="Workflow to attach the tool to"), + Param("tool_id", location=ParamLocation.BODY, required=True, flag="--tool", + help="Registry id (function_name) from `tool registry list`")), + subgroup="workflow tool", body=BodyKind.JSON, permission=Permission.READ_WRITE, + doc_source="backend/tool_instance_v2/urls.py", + description="A workflow holds at most one tool. Seeds adapter settings from the " + "org DEFAULT TRIAD -- set that first with `adapter default-triad set`, " + "or creation 500s while still persisting a half-configured row " + "(CAPTURE2 GAP 3). Attaching activates the workflow.", + examples=("unstract platform workflow tool add --workflow --tool ",)), + _ep("get", "GET", "/tool_instance/{id}/", "Show one attached tool, including its metadata.", + (Param("id", type=ParamType.UUID, location=ParamLocation.PATH, required=True, + help="Tool instance id from `workflow tool list`"),), + subgroup="workflow tool", permission=Permission.READ, + doc_source="backend/tool_instance_v2/urls.py", + description="Read this to see the tool instance's current `metadata` before " + "patching it. Required for the read-modify-write below."), + _ep("set-metadata", "PATCH", "/tool_instance/{id}/", + "Replace an attached tool's metadata (e.g. to set challenge_llm).", + (Param("id", type=ParamType.UUID, location=ParamLocation.PATH, required=True, + help="Tool instance id from `workflow tool list`"), + Param("metadata", type=ParamType.JSON, location=ParamLocation.BODY, required=True, + help="COMPLETE metadata object (accepts @file.json). This REPLACES the " + "stored metadata wholesale -- it is not merged")), + subgroup="workflow tool", body=BodyKind.JSON, permission=Permission.READ_WRITE, + doc_source="backend/tool_instance_v2/urls.py", + description="REPLACES metadata wholesale (the backend does not merge), so you " + "MUST send the full object. Sending only one key wipes " + "prompt_registry_id and orphans the tool. Use this to fix the " + "deploy-time challenge_llm requirement (CAPTURE2 BUG 4): read the " + "instance's current metadata, add a valid LLM adapter id (from " + "`settings-schema`'s enum), and pass the whole object back:\n\n" + " # 1. read the current metadata object (the `metadata` field)\n" + " unstract ... workflow tool get --id \n" + " # 2. save that object to m.json, set metadata.challenge_llm,\n" + " # then send the COMPLETE object back:\n" + " unstract ... workflow tool set-metadata --id --metadata @m.json", + examples=("unstract platform workflow tool set-metadata --id --metadata @metadata.json",)), + _ep("remove", "DELETE", "/tool_instance/{id}/", "Detach a tool from a workflow.", + (Param("id", type=ParamType.UUID, location=ParamLocation.PATH, required=True, + help="Tool instance id from `workflow tool list`"),), + subgroup="workflow tool", permission=Permission.FULL_ACCESS, + doc_source="backend/tool_instance_v2/urls.py", + description=f"{_DELETE_NOTE} A read_write key cannot DELETE a tool instance " + "even though it can create one (CAPTURE2 GAP 5)."), + _ep("list", "GET", "/workflow/endpoint/", "List a workflow's source/destination endpoints.", + (Param("workflow", type=ParamType.UUID, help="Filter to one workflow"), + Param("endpoint_type", choices=["SOURCE", "DESTINATION"], help="Filter by side"), + Param("connection_type", + choices=["FILESYSTEM", "DATABASE", "API", "MANUALREVIEW"], + help="Filter by connection type")), + subgroup="workflow endpoint", permission=Permission.READ, + doc_source="backend/workflow_manager/endpoint_v2/urls.py", + table_columns=("id", "endpoint_type", "connection_type"), + description="Both endpoints are auto-created by `workflow create` with a null " + "connection_type; set them to API before creating a deployment."), + _ep("set", "PATCH", "/workflow/endpoint/{id}/", "Configure a workflow endpoint.", + (Param("id", type=ParamType.UUID, location=ParamLocation.PATH, required=True, + help="Endpoint id from `workflow endpoint list`"), + Param("connection_type", location=ParamLocation.BODY, required=True, + choices=["API", "FILESYSTEM", "DATABASE", "MANUALREVIEW"], + help="Connection type for this endpoint")), + subgroup="workflow endpoint", body=BodyKind.JSON, permission=Permission.READ_WRITE, + doc_source="backend/workflow_manager/endpoint_v2/urls.py", + description="For an API deployment, set BOTH endpoints to connection_type=API. " + "API and MANUALREVIEW need no connector/credentials.", + examples=("unstract platform workflow endpoint set --id --connection-type API",)), +) + + +# --------------------------------------------------------------------------- # +# API Deployments (management) +# --------------------------------------------------------------------------- # + +_DEP_ID = Param("id", type=ParamType.UUID, location=ParamLocation.PATH, required=True, + help="Deployment identifier") + +_DEP_FIELDS: tuple[Param, ...] = ( + Param("display_name", location=ParamLocation.BODY, help="Display name (max 30 chars)"), + Param("description", location=ParamLocation.BODY, help="Description (max 255 chars)"), + Param("workflow", type=ParamType.UUID, location=ParamLocation.BODY, required=True, + help="Workflow to deploy; must have source and destination configured"), + Param("api_name", location=ParamLocation.BODY, + help="URL name, matching ^[a-zA-Z0-9_-]+$ (max 30 chars, unique per org)"), + Param("is_active", type=ParamType.BOOL, location=ParamLocation.BODY, default=True, + help="Whether the deployment accepts requests"), + Param("shared_to_org", type=ParamType.BOOL, location=ParamLocation.BODY, default=False, + help="Share with the whole organization"), + Param("shared_users", type=ParamType.INT, location=ParamLocation.BODY, multiple=True, + replace_semantics=True, help="User IDs to share with"), +) + +_dep_update = _ep("update", "PUT", "/api/deployment/{id}/", "Replace an API deployment.", + (_DEP_ID, *_DEP_FIELDS), subgroup="api-deployment", body=BodyKind.JSON, + doc="v1-api-deployments.mdx", permission=Permission.READ_WRITE) + +_API_DEPLOYMENTS: tuple[Endpoint, ...] = ( + _ep("list", "GET", "/api/deployment/", "List API deployments.", + (Param("workflow", type=ParamType.UUID, help="Filter by workflow"), + Param("search", help="Case-insensitive search on display name"), *_PAGE), + subgroup="api-deployment", doc="v1-api-deployments.mdx", permission=Permission.READ, + table_columns=("id", "api_name", "display_name", "is_active", "run_count")), + _ep("create", "POST", "/api/deployment/", "Create an API deployment.", _DEP_FIELDS, + subgroup="api-deployment", body=BodyKind.JSON, doc="v1-api-deployments.mdx", + permission=Permission.READ_WRITE, + description=( + "Returns an api_key in the response -- store it, it is how the deployment " + "is called. Only one active deployment is allowed per workflow." + )), + _ep("get", "GET", "/api/deployment/{id}/", "Show one API deployment.", (_DEP_ID,), + subgroup="api-deployment", doc="v1-api-deployments.mdx", permission=Permission.READ), + _dep_update, + derive_patch(_dep_update, summary="Partially update an API deployment."), + _ep("delete", "DELETE", "/api/deployment/{id}/", "Delete an API deployment.", (_DEP_ID,), + subgroup="api-deployment", doc="v1-api-deployments.mdx", + permission=Permission.FULL_ACCESS, description=f"{_DELETE_NOTE} Owner only."), + _ep("users", "GET", "/api/deployment/{id}/users/", + "List users a deployment is shared with.", (_DEP_ID,), subgroup="api-deployment", + doc="v1-api-deployments.mdx", permission=Permission.READ), + _ep("by-prompt-studio-tool", "GET", "/api/deployment/by-prompt-studio-tool/", + "Find deployments backed by a Prompt Studio tool.", + (Param("tool_id", type=ParamType.UUID, required=True, help="Prompt Studio tool id"),), + subgroup="api-deployment", doc="v1-api-deployments.mdx", permission=Permission.READ), + _ep("postman-collection", "GET", "/api/postman_collection/{id}/", + "Download a Postman collection for a deployment.", + (_DEP_ID, Param("save", client_side=True, help="Write the collection to this path")), + subgroup="api-deployment", doc="v1-api-deployments.mdx", permission=Permission.READ, + description="Returns 409 if the deployment has no active API key."), + _ep("list", "GET", "/api/keys/api/{api_id}/", "List API keys for a deployment.", + (Param("api_id", type=ParamType.UUID, location=ParamLocation.PATH, required=True, + help="Deployment identifier"),), + subgroup="api-deployment key", doc="v1-api-deployments.mdx", permission=Permission.READ), + _ep("create", "POST", "/api/keys/api/{api_id}/", "Create an API key for a deployment.", + # `api_id` is the URL path param and `api` the body param, and the server + # needs BOTH -- the same value, spelled twice (GOTCHAS #6). `mirror_as` + # copies the path value into the body as `api`, so `--api-id` alone is + # enough; there is nothing for the caller to repeat. + (Param("api_id", type=ParamType.UUID, location=ParamLocation.PATH, required=True, + mirror_as="api", help="Deployment identifier. Also sent as the body's " + "`api` field, so it need not be repeated"), + Param("description", location=ParamLocation.BODY, help="Description (max 255 chars)"), + Param("is_active", type=ParamType.BOOL, location=ParamLocation.BODY, default=True, + help="Whether the key is usable")), + subgroup="api-deployment key", body=BodyKind.JSON, doc="v1-api-deployments.mdx", + permission=Permission.READ_WRITE, + description="Creating a key for a *pipeline* is a different route -- use " + "`pipeline key create`, which takes --pipeline-id.", + doc_conflict="The docs list `api` and `pipeline` as body params. `api` is " + "mirrored from the --api-id path param (it is always the same " + "value, and the docs' own example repeats it), and `pipeline` is " + "dropped: this route is /api/keys/api/{api_id}/, so a pipeline key " + "belongs on `pipeline key create`. Do not restore either flag.", + examples=("unstract platform api-deployment key create --api-id ",)), + _ep("get", "GET", "/api/keys/{id}/", "Show one API key.", + (Param("id", type=ParamType.UUID, location=ParamLocation.PATH, required=True, + help="Key identifier"),), + subgroup="api-deployment key", doc="v1-api-deployments.mdx", permission=Permission.READ), + _ep("update", "PUT", "/api/keys/{id}/", "Update an API key.", + (Param("id", type=ParamType.UUID, location=ParamLocation.PATH, required=True, + help="Key identifier"), + Param("is_active", type=ParamType.BOOL, location=ParamLocation.BODY, + help="Whether the key is usable"), + Param("description", location=ParamLocation.BODY, help="Description (max 255 chars)")), + subgroup="api-deployment key", body=BodyKind.JSON, doc="v1-api-deployments.mdx", + permission=Permission.READ_WRITE), + _ep("delete", "DELETE", "/api/keys/{id}/", "Delete an API key.", + (Param("id", type=ParamType.UUID, location=ParamLocation.PATH, required=True, + help="Key identifier"),), + subgroup="api-deployment key", doc="v1-api-deployments.mdx", + permission=Permission.FULL_ACCESS, description=_DELETE_NOTE), +) + + +# --------------------------------------------------------------------------- # +# Pipelines +# --------------------------------------------------------------------------- # + +_PIPE_ID = Param("id", type=ParamType.UUID, location=ParamLocation.PATH, required=True, + help="Pipeline identifier") + +_PIPE_FIELDS: tuple[Param, ...] = ( + Param("pipeline_name", location=ParamLocation.BODY, required=True, + help="Pipeline name, unique per organization (max 32 chars)"), + Param("workflow", type=ParamType.UUID, location=ParamLocation.BODY, required=True, + help="Workflow to run"), + Param("pipeline_type", location=ParamLocation.BODY, default="DEFAULT", + choices=["ETL", "TASK", "DEFAULT", "APP"], help="Pipeline type"), + Param("cron_string", location=ParamLocation.BODY, + help="UNIX cron schedule; the platform enforces a minimum interval"), + Param("shared_users", type=ParamType.INT, location=ParamLocation.BODY, multiple=True, + replace_semantics=True, help="User IDs to share with"), + Param("shared_to_org", type=ParamType.BOOL, location=ParamLocation.BODY, default=False, + help="Share with the whole organization"), +) + +_pipe_update = _ep("update", "PUT", "/pipeline/{id}/", "Replace a pipeline.", + (_PIPE_ID, *_PIPE_FIELDS), subgroup="pipeline", body=BodyKind.JSON, + doc="v1-etl-pipelines.mdx", permission=Permission.READ_WRITE) + +#: PATCH additionally accepts `active`, which PUT does not. +_pipe_patch = with_params( + derive_patch(_pipe_update, summary="Partially update a pipeline."), + Param("active", type=ParamType.BOOL, location=ParamLocation.BODY, + help="Activate or deactivate the pipeline"), +) + +_PIPELINES: tuple[Endpoint, ...] = ( + _ep("list", "GET", "/pipeline/", "List pipelines.", + (Param("type", choices=["ETL", "TASK", "DEFAULT", "APP"], help="Filter by type"), + Param("workflow", type=ParamType.UUID, help="Filter by workflow"), + Param("search", help="Search on pipeline name"), + Param("ordering", + choices=["created_at", "last_run_time", "pipeline_name", "run_count"], + help="Sort field; prefix with '-' for descending"), + *_PAGE), + subgroup="pipeline", doc="v1-etl-pipelines.mdx", permission=Permission.READ, + table_columns=("id", "pipeline_name", "pipeline_type", "active", "last_run_status")), + _ep("create", "POST", "/pipeline/", "Create a pipeline.", _PIPE_FIELDS, + subgroup="pipeline", body=BodyKind.JSON, doc="v1-etl-pipelines.mdx", + permission=Permission.READ_WRITE), + _ep("get", "GET", "/pipeline/{id}/", "Show one pipeline.", (_PIPE_ID,), + subgroup="pipeline", doc="v1-etl-pipelines.mdx", permission=Permission.READ), + _pipe_update, + _pipe_patch, + _ep("delete", "DELETE", "/pipeline/{id}/", "Delete a pipeline and its scheduler job.", + (_PIPE_ID,), subgroup="pipeline", doc="v1-etl-pipelines.mdx", + permission=Permission.FULL_ACCESS, description=_DELETE_NOTE), + _ep("execute", "POST", "/pipeline/execute/", "Execute a pipeline.", + (Param("pipeline_id", type=ParamType.UUID, location=ParamLocation.BODY, + required=True, help="Pipeline to execute"), + Param("execution_id", type=ParamType.UUID, location=ParamLocation.BODY, + help="Execution identifier; generated if omitted")), + subgroup="pipeline", body=BodyKind.JSON, doc="v1-etl-pipelines.mdx", + permission=Permission.READ_WRITE), + _ep("executions", "GET", "/pipeline/{id}/executions/", "List executions of a pipeline.", + (_PIPE_ID, + Param("start_date", help="ISO 8601 start of range"), + Param("end_date", help="ISO 8601 end of range"), *_PAGE), + subgroup="pipeline", doc="v1-etl-pipelines.mdx", permission=Permission.READ), + _ep("users", "GET", "/pipeline/{id}/users/", "List users a pipeline is shared with.", + (_PIPE_ID,), subgroup="pipeline", doc="v1-etl-pipelines.mdx", + permission=Permission.READ), + # Distinct from the deployment collection path; both exist, and they differ. + _ep("postman-collection", "GET", "/pipeline/api/postman_collection/{id}/", + "Download a Postman collection for a pipeline.", + (_PIPE_ID, Param("save", client_side=True, help="Write the collection to this path")), + subgroup="pipeline", doc="v1-etl-pipelines.mdx", permission=Permission.READ, + description="Returns 400 if the pipeline has no active API key."), + _ep("list", "GET", "/api/keys/pipeline/{pipeline_id}/", "List API keys for a pipeline.", + (Param("pipeline_id", type=ParamType.UUID, location=ParamLocation.PATH, + required=True, help="Pipeline identifier"),), + subgroup="pipeline key", doc="v1-etl-pipelines.mdx", permission=Permission.READ), + _ep("create", "POST", "/api/keys/pipeline/{pipeline_id}/", + "Create an API key for a pipeline.", + # Same path/body duplication as `api-deployment key create` (GOTCHAS #6): + # the URL takes `pipeline_id`, the body wants the same value as `pipeline`. + (Param("pipeline_id", type=ParamType.UUID, location=ParamLocation.PATH, + required=True, mirror_as="pipeline", + help="Pipeline identifier. Also sent as the body's `pipeline` field, " + "so it need not be repeated"), + Param("description", location=ParamLocation.BODY, help="Description (max 255 chars)"), + Param("is_active", type=ParamType.BOOL, location=ParamLocation.BODY, default=True, + help="Whether the key is usable")), + subgroup="pipeline key", body=BodyKind.JSON, doc="v1-etl-pipelines.mdx", + permission=Permission.READ_WRITE, + description="Creating a key for an *API deployment* is a different route -- use " + "`api-deployment key create`, which takes --api-id.", + doc_conflict="Mirror of the `api-deployment key create` divergence: `pipeline` " + "is mirrored from --pipeline-id, and the `api` body param is " + "dropped because this route is /api/keys/pipeline/{pipeline_id}/. " + "Do not restore either flag.", + examples=("unstract platform pipeline key create --pipeline-id ",)), +) + + +# --------------------------------------------------------------------------- # +# Adapters +# --------------------------------------------------------------------------- # + +_ADAPTER_TYPES = ["LLM", "EMBEDDING", "VECTOR_DB", "X2TEXT", "OCR"] +_AD_ID = Param("id", type=ParamType.UUID, location=ParamLocation.PATH, required=True, + help="Adapter instance identifier") + +_AD_FIELDS: tuple[Param, ...] = ( + Param("adapter_name", location=ParamLocation.BODY, required=True, + help="Instance name, unique per name+type+org (max 128 chars)"), + Param("adapter_id", location=ParamLocation.BODY, required=True, + help="SDK adapter identifier, e.g. openai_llm"), + Param("adapter_type", location=ParamLocation.BODY, required=True, + choices=_ADAPTER_TYPES, help="Adapter category"), + Param("adapter_metadata", type=ParamType.JSON, location=ParamLocation.BODY, + required=True, help="Provider configuration; encrypted at rest"), + Param("description", location=ParamLocation.BODY, help="Description"), + Param("shared_to_org", type=ParamType.BOOL, location=ParamLocation.BODY, default=False, + help="Share with the whole organization"), +) + +_ad_update = _ep("update", "PUT", "/adapter/{id}/", "Replace an adapter instance.", + (_AD_ID, *_AD_FIELDS), subgroup="adapter", body=BodyKind.JSON, + doc="v1-adapters.mdx", permission=Permission.READ_WRITE) + +#: Only PATCH documents `shared_users` for adapters; POST/PUT do not. +_ad_patch = with_params( + derive_patch(_ad_update, summary="Partially update an adapter instance."), + Param("shared_users", type=ParamType.INT, location=ParamLocation.BODY, + multiple=True, replace_semantics=True, help="User IDs to share with"), +) + +_ADAPTERS: tuple[Endpoint, ...] = ( + _ep("supported", "GET", "/supported_adapters/", "List adapters supported by the platform.", + (Param("adapter_type", required=True, choices=_ADAPTER_TYPES, + help="Adapter category to list"),), + subgroup="adapter", doc="v1-adapters.mdx", permission=Permission.READ, + examples=("unstract platform adapter supported --adapter-type LLM",)), + _ep("schema", "GET", "/adapter_schema/", "Show the configuration schema for an adapter.", + (Param("id", required=True, help="SDK adapter identifier, e.g. openai_llm"),), + subgroup="adapter", doc="v1-adapters.mdx", permission=Permission.READ), + _ep("test", "POST", "/test_adapters/", "Test adapter credentials.", + (Param("adapter_id", location=ParamLocation.BODY, required=True, + help="SDK adapter identifier"), + Param("adapter_metadata", type=ParamType.JSON, location=ParamLocation.BODY, + required=True, help="Provider configuration to test"), + Param("adapter_type", location=ParamLocation.BODY, required=True, + choices=_ADAPTER_TYPES, help="Adapter category")), + subgroup="adapter", body=BodyKind.JSON, doc="v1-adapters.mdx", + permission=Permission.READ_WRITE), + _ep("list", "GET", "/adapter/", "List configured adapter instances.", + (Param("adapter_type", choices=_ADAPTER_TYPES, help="Filter by category"),), + subgroup="adapter", doc="v1-adapters.mdx", permission=Permission.READ, + description="`is_available` reflects the adapter CLASS being installed, NOT " + "whether your API key may use it. An adapter owned by another user " + "and not shared will still 403 at extraction. Check created_by_email " + "and share adapters to the org (or set a default triad) before use.", + table_columns=("id", "adapter_name", "adapter_type", "model", + "is_available", "created_by_email")), + _ep("create", "POST", "/adapter/", "Create an adapter instance.", _AD_FIELDS, + subgroup="adapter", body=BodyKind.JSON, doc="v1-adapters.mdx", + permission=Permission.READ_WRITE), + _ep("get", "GET", "/adapter/{id}/", "Show one adapter instance.", (_AD_ID,), + subgroup="adapter", doc="v1-adapters.mdx", permission=Permission.READ, + description="Returns decrypted adapter_metadata, including credentials."), + _ad_update, + _ad_patch, + _ep("delete", "DELETE", "/adapter/{id}/", "Delete an adapter instance.", (_AD_ID,), + subgroup="adapter", doc="v1-adapters.mdx", permission=Permission.FULL_ACCESS, + description=( + f"{_DELETE_NOTE} Returns 409 if the adapter is used by a workflow or " + "Prompt Studio project, and 500 if it is configured as a default." + )), + _ep("info", "GET", "/adapter/info/{id}/", "Show adapter summary including context window.", + (_AD_ID,), subgroup="adapter", doc="v1-adapters.mdx", permission=Permission.READ), + _ep("users", "GET", "/adapter/users/{id}/", "List users an adapter is shared with.", + (_AD_ID,), subgroup="adapter", doc="v1-adapters.mdx", permission=Permission.READ), + _ep("get", "GET", "/adapter/default_triad/", "Show the organization's default adapters.", + subgroup="adapter default-triad", doc="v1-adapters.mdx", permission=Permission.READ, + description="Returns an empty object `{}` when no default triad has been set " + "for the organization -- that is 'unset', not an error (GOTCHAS " + "#10). Set one with `adapter default-triad set`; `workflow tool " + "add` seeds a tool instance's adapters from it, so configuring it " + "first avoids a half-configured tool instance.", + examples=("unstract platform adapter default-triad get",)), + # The request keys here differ from the response keys of the GET above + # (llm_default vs default_llm_adapter). That asymmetry is upstream. + _ep("set", "POST", "/adapter/default_triad/", "Set the organization's default adapters.", + (Param("llm_default", type=ParamType.UUID, location=ParamLocation.BODY, + help="Default LLM adapter"), + Param("embedding_default", type=ParamType.UUID, location=ParamLocation.BODY, + help="Default embedding adapter"), + Param("vector_db_default", type=ParamType.UUID, location=ParamLocation.BODY, + help="Default vector DB adapter"), + Param("x2text_default", type=ParamType.UUID, location=ParamLocation.BODY, + help="Default text extractor adapter")), + subgroup="adapter default-triad", body=BodyKind.JSON, doc="v1-adapters.mdx", + permission=Permission.READ_WRITE), +) + + +# --------------------------------------------------------------------------- # +# Connectors +# --------------------------------------------------------------------------- # + +_CN_ID = Param("id", type=ParamType.UUID, location=ParamLocation.PATH, required=True, + help="Connector instance identifier") + +_CN_FIELDS: tuple[Param, ...] = ( + Param("connector_name", location=ParamLocation.BODY, required=True, + help="Instance name, unique per organization (max 128 chars)"), + Param("connector_id", location=ParamLocation.BODY, required=True, + help="Connector type identifier"), + Param("connector_metadata", type=ParamType.JSON, location=ParamLocation.BODY, + help="Connection configuration; not needed when using the OAuth flow"), + Param("connector_version", location=ParamLocation.BODY, help="Connector version"), + Param("shared_to_org", type=ParamType.BOOL, location=ParamLocation.BODY, default=False, + help="Share with the whole organization"), + Param("shared_users", type=ParamType.INT, location=ParamLocation.BODY, multiple=True, + replace_semantics=True, help="User IDs to share with"), +) + +_cn_update = _ep("update", "PUT", "/connector/{id}/", "Replace a connector instance.", + (_CN_ID, Param("oauth-key", help="Cache key from the OAuth flow"), + *_CN_FIELDS), + subgroup="connector", body=BodyKind.JSON, doc="v1-connectors.mdx", + permission=Permission.READ_WRITE) + +_CONNECTORS: tuple[Endpoint, ...] = ( + _ep("supported", "GET", "/supported_connectors/", + "List connector types supported by the platform.", + (Param("type", choices=["INPUT", "OUTPUT"], help="Filter by direction"), + Param("connector_mode", choices=["FILE_SYSTEM", "DATABASE"], help="Filter by mode")), + subgroup="connector", doc="v1-connectors.mdx", permission=Permission.READ), + _ep("schema", "GET", "/connector_schema/", + "Show the configuration schema for a connector type.", + (Param("id", required=True, help="Connector type identifier"),), + subgroup="connector", doc="v1-connectors.mdx", permission=Permission.READ), + _ep("test", "POST", "/test_connectors/", "Test connector credentials.", + (Param("connector_id", location=ParamLocation.BODY, required=True, + help="Connector type identifier"), + Param("connector_metadata", type=ParamType.JSON, location=ParamLocation.BODY, + required=True, help="Connection configuration to test")), + subgroup="connector", body=BodyKind.JSON, doc="v1-connectors.mdx", + permission=Permission.READ_WRITE), + _ep("list", "GET", "/connector/", "List connector instances.", + (Param("workflow", type=ParamType.UUID, help="Filter by workflow"), + Param("created_by", help="Filter by creator user id"), + Param("connector_type", choices=["INPUT", "OUTPUT"], help="Filter by direction"), + Param("connector_mode", choices=["FILE_SYSTEM", "DATABASE"], help="Filter by mode")), + subgroup="connector", doc="v1-connectors.mdx", permission=Permission.READ, + table_columns=("id", "connector_name", "connector_id", "connector_mode")), + _ep("create", "POST", "/connector/", "Create a connector instance.", + (Param("oauth-key", help="Cache key from the OAuth flow, for OAuth connectors"), + *_CN_FIELDS), + subgroup="connector", body=BodyKind.JSON, doc="v1-connectors.mdx", + permission=Permission.READ_WRITE), + _ep("get", "GET", "/connector/{id}/", "Show one connector instance.", (_CN_ID,), + subgroup="connector", doc="v1-connectors.mdx", permission=Permission.READ), + _cn_update, + derive_patch(_cn_update, summary="Partially update a connector instance."), + _ep("delete", "DELETE", "/connector/{id}/", "Delete a connector instance.", (_CN_ID,), + subgroup="connector", doc="v1-connectors.mdx", permission=Permission.FULL_ACCESS, + description=f"{_DELETE_NOTE} Returns 409 if used by a workflow."), +) + +#: Not org-scoped, unlike everything else in this module. +_OAUTH_CACHE_KEY = Endpoint( + name="oauth-cache-key", + group="docstudio", + subgroup="platform connector", + method="GET", + path="/api/v1/oauth/cache-key/{backend}", + api=ApiGroup.PLATFORM, + summary="Generate a cache key to associate an OAuth flow with a connector.", + params=( + Param("backend", location=ParamLocation.PATH, required=True, + help="OAuth backend identifier, e.g. google-oauth2"), + ), + doc_source=f"{_DOCS}/v1-connectors.mdx", + permission=Permission.READ, + description="This endpoint is not organization-scoped.", + no_trailing_slash=True, +) + + +# --------------------------------------------------------------------------- # +# Groups, users, sharing +# --------------------------------------------------------------------------- # + +#: Group identifiers are plain integers, unlike the UUIDs used elsewhere (P10). +_GROUP_ID = Param("id", type=ParamType.INT, location=ParamLocation.PATH, required=True, + help="Group identifier (an integer, not a UUID)") + +_GROUPS: tuple[Endpoint, ...] = ( + _ep("list", "GET", "/groups/", "List user groups.", subgroup="group", + doc="v1-user-groups.mdx", permission=Permission.READ, + table_columns=("id", "name", "description", "member_count")), + _ep("create", "POST", "/groups/", "Create a user group.", + (Param("name", location=ParamLocation.BODY, required=True, + help="Group name, unique per organization"), + Param("description", location=ParamLocation.BODY, help="Group description")), + subgroup="group", body=BodyKind.JSON, doc="v1-user-groups.mdx", + permission=Permission.READ_WRITE, + description=( + "The response echoes only name and description. Re-list groups to " + "obtain the new id." + )), + _ep("patch", "PATCH", "/groups/{id}/", "Update a group's name or description.", + (_GROUP_ID, + Param("name", location=ParamLocation.BODY, help="New name, unique per organization"), + Param("description", location=ParamLocation.BODY, help="New description")), + subgroup="group", body=BodyKind.JSON, doc="v1-user-groups.mdx", + permission=Permission.READ_WRITE), + _ep("delete", "DELETE", "/groups/{id}/", "Delete a group.", (_GROUP_ID,), + subgroup="group", doc="v1-user-groups.mdx", permission=Permission.FULL_ACCESS, + description=( + f"{_DELETE_NOTE} Removes every resource share referencing the group; " + "member accounts themselves are not deleted." + )), + _ep("list", "GET", "/groups/{id}/members/", "List members of a group.", (_GROUP_ID,), + subgroup="group member", doc="v1-user-groups.mdx", permission=Permission.READ), + _ep("add", "POST", "/groups/{id}/members/", "Add members to a group.", + (_GROUP_ID, + Param("user_ids", type=ParamType.INT, location=ParamLocation.BODY, multiple=True, + required=True, help="User IDs to add; existing members are ignored")), + subgroup="group member", body=BodyKind.JSON, doc="v1-user-groups.mdx", + permission=Permission.READ_WRITE), + # No trailing slash on this path, unlike its siblings. + _ep("remove", "DELETE", "/groups/{id}/members/{user_id}", "Remove a member from a group.", + (_GROUP_ID, Param("user_id", type=ParamType.INT, location=ParamLocation.PATH, + required=True, help="User to remove")), + subgroup="group member", doc="v1-user-groups.mdx", permission=Permission.FULL_ACCESS, + description=_DELETE_NOTE, no_trailing_slash=True), + _ep("resources", "GET", "/groups/{id}/resources/", + "List resources shared with a group.", (_GROUP_ID,), subgroup="group", + doc="v1-user-groups.mdx", permission=Permission.READ), + _ep("list", "GET", "/users/", "List organization members.", subgroup="user", + doc="v1-organization-users.mdx", permission=Permission.READ, + table_columns=("id", "email", "role", "is_admin"), + description=( + "Member ids are returned as strings but must be sent as integers in " + "shared_users; the CLI converts them for you." + )), + Endpoint( + name="share", + group="docstudio", + subgroup="platform", + method="POST", + path=f"{_BASE}/{{resource}}/{{id}}/share/", + api=ApiGroup.PLATFORM, + summary="Share a resource with users, groups, or the whole organization.", + description=( + "Each axis REPLACES its existing list rather than appending. To add " + "one user without dropping the others, read the current shares first " + "and send the combined list." + ), + params=( + _ORG, + Param("resource", location=ParamLocation.PATH, required=True, + choices=SHARE_RESOURCES, + help="Resource type; mapped to the correct URL segment"), + Param("id", location=ParamLocation.PATH, required=True, + help="Resource identifier"), + Param("shared_users", type=ParamType.INT, location=ParamLocation.BODY, + multiple=True, replace_semantics=True, help="User IDs to share with"), + Param("shared_groups", type=ParamType.INT, location=ParamLocation.BODY, + multiple=True, replace_semantics=True, help="Group IDs to share with"), + Param("shared_to_org", type=ParamType.BOOL, location=ParamLocation.BODY, + help="Share with the whole organization"), + ), + body=BodyKind.JSON, + doc_source=f"{_DOCS}/v1-user-groups.mdx", + permission=Permission.READ_WRITE, + examples=( + "unstract platform share --resource api-deployment --id --shared-users 2 --shared-users 5", + ), + ), +) + + +ENDPOINTS: tuple[Endpoint, ...] = ( + *_PROMPT_STUDIO, + *_WORKFLOWS, + *_WORKFLOW_ASSEMBLY, + *_API_DEPLOYMENTS, + *_PIPELINES, + *_ADAPTERS, + *_CONNECTORS, + _OAUTH_CACHE_KEY, + *_GROUPS, +) diff --git a/src/unstract_cli/endpoints/whisper.py b/src/unstract_cli/endpoints/whisper.py new file mode 100644 index 0000000..c030d32 --- /dev/null +++ b/src/unstract_cli/endpoints/whisper.py @@ -0,0 +1,304 @@ +"""LLMWhisperer v2 endpoint records (SPEC.md §6.1). + +Authored from `llmwhisperer-docs/docs/llm_whisperer/apis/`. Base URL is +region-specific (US / EU) and configurable for on-prem; auth is the +`unstract-key` header. +""" + +from __future__ import annotations + +from unstract_cli.core.model import ( + ApiGroup, + BodyKind, + Endpoint, + MutuallyExclusive, + Param, + ParamLocation, + ParamType, + PollSpec, +) + +_DOCS = "llmwhisperer-docs/docs/llm_whisperer/apis" + +#: Parameters shared by the extraction endpoint. Kept as a module constant so the +#: `--wait` flow and the raw command describe exactly the same surface. +_EXTRACT_PARAMS: tuple[Param, ...] = ( + Param( + "file", + type=ParamType.FILE, + location=ParamLocation.BODY, + client_side=True, + help="Path to the document to convert", + ), + Param( + "url", + location=ParamLocation.BODY, + client_side=True, + help="Publicly accessible URL to fetch the document from; sets url_in_post", + ), + # Set automatically when --url is used, but exposed so every documented + # parameter is reachable as a flag (SPEC.md §1.1.3). + Param( + "url_in_post", + type=ParamType.BOOL, + help="Send the source URL in the request body; implied by --url", + ), + Param( + "mode", + default="form", + choices=["native_text", "low_cost", "high_quality", "form", "table"], + help=( + "Processing mode. native_text for digital PDFs, low_cost for clean " + "scans, high_quality for handwriting/noisy scans, form for forms and " + "checkboxes, table for dense tabular documents" + ), + ), + Param( + "output_mode", + default="layout_preserving", + choices=["layout_preserving", "text"], + help="layout_preserving keeps document structure (best for LLMs); text is plain", + ), + # The API's own spelling is `page_seperator`; preserved verbatim on the wire. + Param("page_seperator", default="<<<", flag="--page-separator", + help="String used to separate pages"), + Param("pages_to_extract", help="Pages to extract, e.g. '1-5,7,21-'"), + Param("median_filter_size", type=ParamType.INT, applies_when="mode=low_cost", + help="Median filter size for denoising"), + Param("gaussian_blur_radius", type=ParamType.FLOAT, applies_when="mode=low_cost", + help="Gaussian blur radius for denoising"), + Param("line_splitter_tolerance", type=ParamType.FLOAT, default=0.4, + help="Fraction of average character height before text moves to the next line"), + Param("line_splitter_strategy", default="left-priority", + help="Advanced line-splitting strategy"), + Param("horizontal_stretch_factor", type=ParamType.FLOAT, default=1.0, + help="Horizontal stretch; raise slightly when multi-column layouts merge"), + Param("mark_vertical_lines", type=ParamType.BOOL, default=False, + applies_when="mode is not native_text", + help="Reproduce vertical lines in the output"), + Param("mark_horizontal_lines", type=ParamType.BOOL, default=False, + applies_when="mode is not native_text and mark_vertical_lines=true", + help="Reproduce horizontal lines in the output"), + Param("lang", default="eng", help="Language hint for OCR (currently auto-detected)"), + Param("tag", default="default", help="Audit tag, cross-referenced in usage reports"), + Param("file_name", help="Audit file name, cross-referenced in usage reports"), + Param("use_webhook", help="Name of a registered webhook to call on completion"), + Param("webhook_metadata", help="Metadata forwarded verbatim to the webhook"), + Param("add_line_nos", type=ParamType.BOOL, default=False, + help="Add line numbers and store line metadata for the highlights API"), + Param("allow_rotated_text", type=ParamType.BOOL, default=True, + applies_when="mode in form/high_quality/table", + help="Keep rotated text such as watermarks; false filters it out"), + Param("word_confidence_threshold", type=ParamType.FLOAT, default=0.3, + applies_when="mode in form/high_quality/table", + help="Drop words whose OCR confidence is below this value (0-1)"), +) + + +ENDPOINTS: tuple[Endpoint, ...] = ( + Endpoint( + name="usage", + group="whisper", + method="GET", + path="/get-usage-info", + api=ApiGroup.LLMWHISPERER, + summary="Show usage metrics for your LLMWhisperer account.", + doc_source=f"{_DOCS}/usage_api.md", + examples=("unstract whisper usage",), + ), + Endpoint( + name="extract", + group="whisper", + method="POST", + path="/whisper", + api=ApiGroup.LLMWHISPERER, + summary="Convert a document to LLM-ready text.", + description=( + "Accepts PDFs, scanned documents, images, Office documents and " + "spreadsheets. Returns a whisper_hash immediately; use --wait to poll " + "and retrieve the text in one step." + ), + params=_EXTRACT_PARAMS, + body=BodyKind.BINARY_FILE, + constraints=(MutuallyExclusive(("file", "url")),), + poll=PollSpec( + status_endpoint="whisper.status", + status_field="status", + terminal_success=("processed",), + terminal_failure=("error",), + handle_field="whisper_hash", + handle_param="whisper_hash", + retrieve_endpoint="whisper.retrieve", + one_shot=True, + ), + doc_source=f"{_DOCS}/whisper.md", + raw_field="result_text", + examples=( + "unstract whisper extract --file invoice.pdf --mode form --wait", + "unstract whisper extract --url https://example.com/doc.pdf --mode high_quality", + ), + ), + Endpoint( + name="status", + group="whisper", + method="GET", + path="/whisper-status", + api=ApiGroup.LLMWHISPERER, + summary="Check the status of a conversion.", + description="Statuses: accepted, processing, processed, error, retrieved.", + params=( + Param("whisper_hash", required=True, help="Hash returned by `whisper extract`"), + ), + doc_source=f"{_DOCS}/whisper_status.md", + examples=("unstract whisper status --whisper-hash abc123",), + ), + Endpoint( + name="retrieve", + group="whisper", + method="GET", + path="/whisper-retrieve", + api=ApiGroup.LLMWHISPERER, + summary="Retrieve the extracted text of a completed conversion.", + description=( + "ONE-SHOT: text can be retrieved only once, for privacy reasons. A " + "second call fails and the text cannot be recovered -- pass --save to " + "persist it on the first read." + ), + params=( + Param("whisper_hash", required=True, help="Hash returned by `whisper extract`"), + Param("text_only", type=ParamType.BOOL, default=False, + help="Return only the text, omitting metadata"), + Param("save", client_side=True, + help="Write the result to this path before exiting (recommended)"), + ), + doc_source=f"{_DOCS}/whisper_retrieve.md", + raw_field="result_text", + examples=( + "unstract whisper retrieve --whisper-hash abc123 --save result.json", + "unstract whisper retrieve --whisper-hash abc123 --text-only --output raw", + ), + ), + Endpoint( + name="detail", + group="whisper", + method="GET", + path="/whisper-detail", + api=ApiGroup.LLMWHISPERER, + summary="Show processing details and metadata for a conversion.", + params=( + Param("whisper_hash", required=True, help="Hash returned by `whisper extract`"), + ), + doc_source=f"{_DOCS}/whisper_detail.md", + # The docs *index* lists `/whisper-details` (plural), but the endpoint page + # and the official client both use the singular. Verified against + # llm-whisperer-python-client client_v2.py:349. + doc_conflict=( + "Docs index says /whisper-details (plural); endpoint page and official " + "Python client use /whisper-detail (singular). Singular is correct -- " + "do not 'fix' this from the index page." + ), + examples=("unstract whisper detail --whisper-hash abc123",), + ), + Endpoint( + name="highlights", + group="whisper", + method="GET", + path="/highlights", + api=ApiGroup.LLMWHISPERER, + summary="Retrieve line metadata (bounding boxes) for highlighting.", + description="Requires the conversion to have been run with --add-line-nos.", + params=( + Param("whisper_hash", required=True, help="Hash returned by `whisper extract`"), + Param("lines", required=True, help="Lines to fetch, e.g. '1-5,7,21-'"), + ), + doc_source=f"{_DOCS}/highlighting_api.md", + examples=("unstract whisper highlights --whisper-hash abc123 --lines 1-20",), + ), + Endpoint( + name="usage-by-tag", + group="whisper", + method="GET", + path="/usage", + api=ApiGroup.LLMWHISPERER, + summary="Show usage metrics filtered by audit tag.", + description="Without dates, usage covers the last 30 days.", + params=( + Param("tag", required=True, help="Audit tag to report on"), + Param("from_date", type=ParamType.DATE, help="Start date, YYYY-MM-DD"), + Param("to_date", type=ParamType.DATE, help="End date, YYYY-MM-DD"), + ), + doc_source=f"{_DOCS}/usage_stat.md", + examples=("unstract whisper usage-by-tag --tag invoices --from-date 2026-01-01",), + ), + Endpoint( + name="create", + group="whisper", + subgroup="webhook", + method="POST", + path="/whisper-manage-callback", + api=ApiGroup.LLMWHISPERER, + summary="Register a webhook to be called when a conversion completes.", + description=( + "The URL is verified at registration: LLMWhisperer posts a test payload " + "and refuses to register if it does not return 200." + ), + params=( + Param("webhook_name", required=True, location=ParamLocation.BODY, + help="Name used to reference this webhook in `whisper extract --use-webhook`"), + Param("url", required=True, location=ParamLocation.BODY, + help="URL to call when conversion completes"), + Param("auth_token", location=ParamLocation.BODY, default="", + help="Bearer token for the webhook; empty if unauthenticated"), + ), + body=BodyKind.JSON, + doc_source=f"{_DOCS}/webhook_manage.md", + examples=( + "unstract whisper webhook create --webhook-name prod --url https://example.com/hook", + ), + ), + Endpoint( + name="get", + group="whisper", + subgroup="webhook", + method="GET", + path="/whisper-manage-callback", + api=ApiGroup.LLMWHISPERER, + summary="Show a registered webhook's configuration.", + params=(Param("webhook_name", required=True, help="Name of the webhook"),), + doc_source=f"{_DOCS}/webhook_manage.md", + examples=("unstract whisper webhook get --webhook-name prod",), + ), + Endpoint( + name="update", + group="whisper", + subgroup="webhook", + method="PUT", + path="/whisper-manage-callback", + api=ApiGroup.LLMWHISPERER, + summary="Update a registered webhook.", + params=( + Param("webhook_name", required=True, location=ParamLocation.BODY, + help="Name of the existing webhook"), + Param("url", required=True, location=ParamLocation.BODY, help="New URL"), + Param("auth_token", location=ParamLocation.BODY, default="", + help="New bearer token; empty if unauthenticated"), + ), + body=BodyKind.JSON, + doc_source=f"{_DOCS}/webhook_manage.md", + examples=( + "unstract whisper webhook update --webhook-name prod --url https://example.com/v2", + ), + ), + Endpoint( + name="delete", + group="whisper", + subgroup="webhook", + method="DELETE", + path="/whisper-manage-callback", + api=ApiGroup.LLMWHISPERER, + summary="Delete a registered webhook.", + params=(Param("webhook_name", required=True, help="Name of the webhook"),), + doc_source=f"{_DOCS}/webhook_manage.md", + examples=("unstract whisper webhook delete --webhook-name prod",), + ), +) diff --git a/src/unstract_cli/skill/__init__.py b/src/unstract_cli/skill/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/src/unstract_cli/skill/docdiff.py b/src/unstract_cli/skill/docdiff.py new file mode 100644 index 0000000..4c197f1 --- /dev/null +++ b/src/unstract_cli/skill/docdiff.py @@ -0,0 +1,417 @@ +"""Cross-reference endpoint records against the public API documentation. + +Supports the `update-unstract-cli` Claude Skill (SPEC.md §8): parse the docs, +parse the records, and report drift with citations. + +Two documentation formats: + +* **Markdown** (LLMWhisperer, API deployment, HITL) -- parameters in tables with + ``Parameter | Type | Default | Required | Description`` columns. +* **MDX** (Platform v1) -- parameters as ```` component props. This + is JSX rather than Markdown, so props are parsed, not table cells. + +Findings are *reported*, never auto-applied. Documentation lags implementation, +so an endpoint missing from the docs is usually undocumented rather than removed +(SPEC.md §8.5). +""" + +from __future__ import annotations + +import json +import re +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any + +from unstract_cli.core.model import Endpoint +from unstract_cli.endpoints import ALL_ENDPOINTS + +#: Front matter marking an unstable contract; such pages are excluded entirely. +DRAFT_MARKER = re.compile(r"^draft:\s*true\s*$", re.MULTILINE) + + +@dataclass +class DocParam: + """A parameter as the documentation describes it.""" + + name: str + type: str = "" + default: str = "" + required: bool = False + description: str = "" + + +@dataclass +class DocEndpoint: + """An endpoint as the documentation describes it.""" + + method: str + path: str + params: list[DocParam] = field(default_factory=list) + source: str = "" + heading: str = "" + + @property + def key(self) -> tuple[str, str]: + return (self.method.upper(), self.path) + + +@dataclass +class Finding: + """One drift observation, always carrying a citation.""" + + kind: str # missing_in_cli | missing_in_docs | param_drift + severity: str # info | warning | action + message: str + citation: str = "" + command: str = "" + suggestion: str = "" + + def to_dict(self) -> dict[str, Any]: + return { + "kind": self.kind, + "severity": self.severity, + "message": self.message, + "citation": self.citation, + "command": self.command, + "suggestion": self.suggestion, + } + + +# --------------------------------------------------------------------------- # +# Parsing +# --------------------------------------------------------------------------- # + +_TRUTHY = {"yes", "true", "required"} + + +def _clean(cell: str) -> str: + """Strip Markdown emphasis, links and code ticks from a table cell.""" + text = cell.strip() + text = re.sub(r"\[([^\]]+)\]\([^)]*\)", r"\1", text) + text = text.replace("`", "").replace("*", "").replace("
", " ") + return text.strip() + + +#: Headings that begin a *request parameter* table. Everything after a heading +#: not in this set is response/status documentation, which must not be mistaken +#: for input parameters -- otherwise every response field looks like a missing +#: flag. +_REQUEST_HEADINGS = re.compile( + r"^#{1,4}\s+(request\s+)?(parameters|request\s+body|query\s+parameters|" + r"path\s+parameters|body|arguments|request)\b", + re.IGNORECASE, +) +_HEADING = re.compile(r"^#{1,4}\s+\S") + + +def parse_markdown_params(text: str) -> list[DocParam]: + """Extract request parameters from a Markdown parameter table. + + Only tables under a request-parameter heading are considered. API reference + pages document responses, metadata and status codes in the same table format, + and treating those as inputs produces a flood of false drift. + """ + params: list[DocParam] = [] + seen: set[str] = set() + in_request_section = False + + for line in text.splitlines(): + if _HEADING.match(line): + in_request_section = bool(_REQUEST_HEADINGS.match(line)) + continue + if not in_request_section: + continue + if not line.strip().startswith("|"): + continue + cells = [_clean(c) for c in line.strip().strip("|").split("|")] + if len(cells) < 3: + continue + + name = cells[0] + # Skip header and separator rows, and prose rows that aren't parameters. + if not name or name.lower() in {"parameter", "name", "field"}: + continue + if set(name) <= {"-", ":", " "}: + continue + if not re.fullmatch(r"[a-zA-Z_][a-zA-Z0-9_]*", name): + continue + if name in seen: + continue + + seen.add(name) + required = len(cells) > 3 and any(t in cells[3].lower() for t in _TRUTHY) + params.append( + DocParam( + name=name, + type=cells[1] if len(cells) > 1 else "", + default=cells[2] if len(cells) > 2 else "", + required=required, + description=cells[4] if len(cells) > 4 else "", + ) + ) + return params + + +def parse_markdown_doc(path: Path) -> list[DocEndpoint]: + """Parse a Markdown API reference page.""" + text = path.read_text(encoding="utf-8") + if DRAFT_MARKER.search(text): + return [] # Unstable contract; excluded by policy (SPEC.md §11). + + endpoint = "" + for pattern in ( + r"\|\s*Endpoint\s*\|\s*`([^`]+)`", + r"\|\s*URL\s*\|\s*`https?://[^/]+(/[^`]+)`", + ): + if match := re.search(pattern, text): + endpoint = match.group(1).strip() + break + if not endpoint: + return [] + + methods = re.findall(r"\|\s*Method\s*\|\s*(.+?)\s*\|", text) + found = re.findall(r"\b(GET|POST|PUT|PATCH|DELETE)\b", methods[0]) if methods else ["GET"] + + params = parse_markdown_params(text) + if not endpoint.startswith("/"): + endpoint = "/" + endpoint + + return [ + DocEndpoint(method=m, path=endpoint, params=params, source=str(path)) + for m in (found or ["GET"]) + ] + + +def _prop_entries(block: str, prop: str) -> list[tuple[str, str]]: + """Return ``(name, raw_object)`` pairs from one `` prop array. + + Extracts ``prop={[ {...}, {...} ]}`` by scanning for the matching bracket, so + a nested brace inside a description cannot terminate the array early. + """ + match = re.search(rf"\b{prop}=\{{\[", block) + if not match: + return [] + + depth = 0 + start = match.end() - 1 + end = start + for index in range(start, len(block)): + char = block[index] + if char == "[": + depth += 1 + elif char == "]": + depth -= 1 + if depth == 0: + end = index + break + else: + return [] + + return [ + (m.group(1), m.group(0)) + for m in re.finditer(r'\{\s*name:\s*"([^"]+)"[^}]*\}', block[start : end + 1]) + ] + + +def parse_mdx_doc(path: Path) -> list[DocEndpoint]: + """Parse `` components from a Platform v1 MDX page.""" + text = path.read_text(encoding="utf-8") + if DRAFT_MARKER.search(text): + return [] + + endpoints: list[DocEndpoint] = [] + for block in re.findall(r"", text, re.DOTALL): + method = re.search(r'method="([^"]+)"', block) + path_match = re.search(r'path="([^"]+)"', block) + if not method or not path_match: + continue + + params: list[DocParam] = [] + seen: set[str] = set() + # Only request-side props. `responseBody` describes output: treating it + # as input makes every response field look like a missing flag, which is + # the MDX equivalent of the response-table problem in Markdown. + for prop in ("pathParams", "queryParams", "requestBody"): + for entry in _prop_entries(block, prop): + name = entry[0] + # `members[].id`-style rows document nested response shapes, and + # prose placeholders like "(any writable field)" are descriptions + # of a whole class of fields rather than a named parameter. + if not re.fullmatch(r"[a-zA-Z_][a-zA-Z0-9_-]*", name): + continue + if name in seen: + continue + seen.add(name) + params.append( + DocParam( + name=name, + required="required: true" in entry[1] + or "required={true}" in entry[1], + type=( + t.group(1) + if (t := re.search(r'type:\s*"([^"]+)"', entry[1])) + else "" + ), + ) + ) + + endpoints.append( + DocEndpoint( + method=method.group(1).upper(), + path=path_match.group(1), + params=params, + source=str(path), + ) + ) + return endpoints + + +def parse_docs(root: Path) -> list[DocEndpoint]: + """Parse every documentation page under a docs repository root.""" + results: list[DocEndpoint] = [] + for path in sorted(root.rglob("*.md")): + results.extend(parse_markdown_doc(path)) + for path in sorted(root.rglob("*.mdx")): + results.extend(parse_mdx_doc(path)) + return results + + +# --------------------------------------------------------------------------- # +# Diffing +# --------------------------------------------------------------------------- # + + +def _normalise(path: str) -> str: + """Compare paths by shape, ignoring placeholder spelling and prefixes. + + The same endpoint is written several ways across sources: records use + `{org_id}` while the Markdown docs use ``, and Platform v1 + paths carry an org-scoped prefix that its MDX omits. Only the shape + distinguishes one endpoint from another, so both placeholder syntaxes + collapse to `{}`. + + Note that trailing slashes are normalised **here only**, for matching. + `Endpoint.path` itself remains literal, because the server distinguishes + them (P11). + """ + path = re.sub(r"^/api/v1/unstract/\{org_id\}", "", path) + path = re.sub(r"\{[^}]+\}", "{}", path) + path = re.sub(r"<[^>]+>", "{}", path) + return path.rstrip("/") or "/" + + +def diff( + doc_endpoints: list[DocEndpoint], + cli_endpoints: tuple[Endpoint, ...] = ALL_ENDPOINTS, +) -> list[Finding]: + """Compare documentation against records, three ways (SPEC.md §8.3).""" + findings: list[Finding] = [] + + by_cli: dict[tuple[str, str], Endpoint] = { + (e.method.upper(), _normalise(e.path)): e for e in cli_endpoints + } + by_doc: dict[tuple[str, str], DocEndpoint] = { + (d.method.upper(), _normalise(d.path)): d for d in doc_endpoints + } + + for key, doc in by_doc.items(): + cli = by_cli.get(key) + if cli is None: + findings.append( + Finding( + kind="missing_in_cli", + severity="action", + message=f"{doc.method} {doc.path} is documented but has no CLI command.", + citation=doc.source, + suggestion="Add an Endpoint record to the matching endpoints/*.py module.", + ) + ) + continue + + # A record carrying `doc_conflict` has already been reconciled against the + # docs by hand, and the divergence is the point -- `api-deployment key + # create` deliberately drops the documented `pipeline` body param because + # the path fixes the resource. Reporting it would invite the Skill to + # "restore" the very thing that was removed, so the exemption covers + # parameters as well as the endpoint's existence. + if cli.doc_conflict: + continue + + cli_names = {p.name for p in cli.params} + for param in doc.params: + if param.name in cli_names: + continue + # A parameter mirrored into the body from a path param is present on + # the wire under its documented name, just not as a separate flag. + if any(p.mirrors and p.body_name == param.name for p in cli.params): + continue + # Path parameters often appear under different placeholder names. + if any(param.name in cli.path for _ in (0,)): + continue + findings.append( + Finding( + kind="param_drift", + severity="action", + message=( + f"{cli.dotted_name}: documented parameter " + f"{param.name!r} is not exposed as a flag." + ), + citation=doc.source, + command="unstract " + " ".join(cli.command_path), + suggestion=( + f"Add Param({param.name!r}" + + (", required=True" if param.required else "") + + ") to this record." + ), + ) + ) + + for key, cli in by_cli.items(): + if key in by_doc: + continue + if cli.doc_conflict: + continue # A verified deliberate divergence; never revert it. + findings.append( + Finding( + kind="missing_in_docs", + severity="info", + message=( + f"{cli.dotted_name} ({cli.method} {cli.path}) was not found in " + "the parsed docs. Docs lag implementation -- report only." + ), + citation=cli.doc_source, + command="unstract " + " ".join(cli.command_path), + suggestion="Verify manually. NEVER delete a command for this reason alone.", + ) + ) + + return findings + + +def report(findings: list[Finding]) -> str: + """Render findings as JSON for the Skill to act on.""" + by_kind: dict[str, list[dict[str, Any]]] = {} + for finding in findings: + by_kind.setdefault(finding.kind, []).append(finding.to_dict()) + return json.dumps( + { + "summary": {kind: len(items) for kind, items in by_kind.items()}, + "total": len(findings), + "findings": by_kind, + }, + indent=2, + ) + + +__all__ = [ + "DocEndpoint", + "DocParam", + "Finding", + "diff", + "parse_docs", + "parse_markdown_doc", + "parse_markdown_params", + "parse_mdx_doc", + "report", +] diff --git a/tests/__init__.py b/tests/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/conftest.py b/tests/conftest.py new file mode 100644 index 0000000..502ed2a --- /dev/null +++ b/tests/conftest.py @@ -0,0 +1,70 @@ +"""Shared fixtures. + +Every test runs against an isolated config path and a controlled environment, so +a developer's real `~/.config/unstract/config.toml` can never influence results +(or be written to). +""" + +from __future__ import annotations + +import pytest +from click.testing import CliRunner + +from unstract_cli.app import build_cli + +#: Recognisable so redaction tests can assert this exact string never appears. +FAKE_KEY = "sk-test-SECRETVALUE-0123456789" + +WHISPER_BASE = "https://llmwhisperer-api.us-central.unstract.com/api/v2" +PLATFORM_BASE = "https://us-central.unstract.com" + + +@pytest.fixture(autouse=True) +def isolated_env(tmp_path, monkeypatch): + """Point config at a temp path and clear inherited credentials.""" + for var in ( + "UNSTRACT_PROFILE", "LLMWHISPERER_API_KEY", "LLMWHISPERER_BASE_URL", + "UNSTRACT_PLATFORM_KEY", "UNSTRACT_DEPLOYMENT_KEY", "UNSTRACT_ORG_ID", + "UNSTRACT_BASE_URL", "UNSTRACT_APIHUB_KEY", "UNSTRACT_APIHUB_BASE_URL", + "UNSTRACT_ANTHROPIC_API_KEY", "UNSTRACT_OUTPUT", "NO_COLOR", + ): + monkeypatch.delenv(var, raising=False) + monkeypatch.setenv("UNSTRACT_CONFIG", str(tmp_path / "config.toml")) + return tmp_path + + +@pytest.fixture +def whisper_env(monkeypatch): + """Credentials for LLMWhisperer, supplied only through the environment. + + Deliberately env-only: this is also the assertion that the CLI is fully + usable with no config file, which is how it runs in CI and agent sandboxes. + """ + monkeypatch.setenv("LLMWHISPERER_API_KEY", FAKE_KEY) + return FAKE_KEY + + +@pytest.fixture +def platform_env(monkeypatch): + monkeypatch.setenv("UNSTRACT_PLATFORM_KEY", FAKE_KEY) + monkeypatch.setenv("UNSTRACT_DEPLOYMENT_KEY", FAKE_KEY) + monkeypatch.setenv("UNSTRACT_ORG_ID", "org_test123") + return FAKE_KEY + + +@pytest.fixture +def runner(): + """Click runner that keeps stderr separate, so stdout purity is testable.""" + return CliRunner() + + +@pytest.fixture +def cli(): + return build_cli() + + +@pytest.fixture +def sample_file(tmp_path): + path = tmp_path / "sample.pdf" + path.write_bytes(b"%PDF-1.4 fake document for testing") + return path diff --git a/tests/test_cli.py b/tests/test_cli.py new file mode 100644 index 0000000..cc8a2fd --- /dev/null +++ b/tests/test_cli.py @@ -0,0 +1,732 @@ +"""End-to-end CLI behaviour: the promises an agent actually depends on.""" + +from __future__ import annotations + +import json +import os +import stat +import subprocess +import sys +from pathlib import Path + +import httpx +import pytest +import respx + +from unstract_cli.app import discover +from unstract_cli.endpoints import ALL_ENDPOINTS + +from .conftest import FAKE_KEY, WHISPER_BASE + + +class TestWalkingSkeleton: + """M1.3 - one endpoint, end to end, proving the whole spine.""" + + @respx.mock + def test_whisper_usage_succeeds(self, runner, cli, whisper_env): + respx.get(f"{WHISPER_BASE}/get-usage-info").mock( + return_value=httpx.Response( + 200, json={"subscription_plan": "free", "daily_quota": 100} + ) + ) + result = runner.invoke(cli, ["whisper", "usage", "--output", "json"]) + assert result.exit_code == 0 + assert json.loads(result.stdout)["subscription_plan"] == "free" + + @respx.mock + def test_works_with_env_vars_only(self, runner, cli, whisper_env, isolated_env): + """Zero-config operation: no config file exists at all here.""" + assert not (isolated_env / "config.toml").exists() + respx.get(f"{WHISPER_BASE}/get-usage-info").mock( + return_value=httpx.Response(200, json={"subscription_plan": "free"}) + ) + assert runner.invoke(cli, ["whisper", "usage"]).exit_code == 0 + + +class TestExitCodes: + """SPEC §5.4 end-to-end, not just at the mapping layer.""" + + @respx.mock + @pytest.mark.parametrize( + "status,expected", + [(401, 3), (403, 3), (404, 4), (400, 5), (422, 5), (429, 6), (500, 8)], + ) + def test_http_status_maps_to_exit_code(self, runner, cli, whisper_env, status, expected): + respx.get(f"{WHISPER_BASE}/get-usage-info").mock( + return_value=httpx.Response(status, json={"message": "failure"}) + ) + result = runner.invoke(cli, ["whisper", "usage", "--no-retry"]) + assert result.exit_code == expected + + def test_missing_credentials_is_usage_error(self, runner, cli): + result = runner.invoke(cli, ["whisper", "usage"]) + assert result.exit_code == 2 + assert "LLMWHISPERER_API_KEY" in result.stderr + + def test_constraint_violation_is_usage_error(self, runner, cli, whisper_env, sample_file): + result = runner.invoke( + cli, ["whisper", "extract", "--file", str(sample_file), "--url", "http://x"] + ) + assert result.exit_code == 2 + + +class TestStdoutPurity: + """SPEC §5.1 - `unstract ... | jq` must always parse.""" + + @respx.mock + def test_errors_are_parseable_on_both_streams_when_piped(self, runner, cli, whisper_env): + """DOC 9 - when stdout is not a TTY (the agent/wrapper case), the error + envelope is mirrored to stdout so a stdout->JSON pipeline sees a valid + object instead of an empty stream. The human copy still goes to stderr.""" + respx.get(f"{WHISPER_BASE}/get-usage-info").mock( + return_value=httpx.Response(403, json={"message": "Unauthorized"}) + ) + result = runner.invoke(cli, ["whisper", "usage", "--no-retry"]) + # CliRunner's stdout is not a TTY -- exactly the piped case. + assert json.loads(result.stdout)["error"]["code"] == "auth_error" + assert json.loads(result.stderr)["error"]["code"] == "auth_error" + + @respx.mock + def test_diagnostics_go_to_stderr(self, runner, cli, whisper_env): + respx.get(f"{WHISPER_BASE}/get-usage-info").mock( + return_value=httpx.Response(200, json={"ok": True}) + ) + result = runner.invoke(cli, ["whisper", "usage", "-vv"]) + assert json.loads(result.stdout) == {"ok": True} + + @respx.mock + def test_json_is_default_when_not_a_tty(self, runner, cli, whisper_env): + """CliRunner's stdout is not a TTY, which is the agent's situation.""" + respx.get(f"{WHISPER_BASE}/get-usage-info").mock( + return_value=httpx.Response(200, json={"plan": "free"}) + ) + result = runner.invoke(cli, ["whisper", "usage"]) + assert json.loads(result.stdout) == {"plan": "free"} + + def test_click_usage_error_is_parseable_on_stdout(self): + """DOC 9, real entry point - Click's own parse errors (No such option) + must also emit the envelope on stdout when piped. CliRunner bypasses the + __main__ handler, so drive the actual console entry point in a subprocess + with a captured (non-TTY) stdout.""" + proc = subprocess.run( + [sys.executable, "-m", "unstract_cli", "whisper", "usage", "--nope"], + capture_output=True, + text=True, + ) + assert proc.returncode == 2 + assert json.loads(proc.stdout)["error"]["exit_code"] == 2 + assert "No such option" in proc.stderr # human message survives on stderr + + +class TestOneShotSemantics: + """SPEC §5.6 - the footgun that silently loses an agent's data.""" + + @respx.mock + def test_save_writes_before_exit(self, runner, cli, whisper_env, tmp_path): + respx.get(f"{WHISPER_BASE}/whisper-retrieve").mock( + return_value=httpx.Response(200, json={"result_text": "IMPORTANT DATA"}) + ) + out = tmp_path / "result.txt" + result = runner.invoke( + cli, ["whisper", "retrieve", "--whisper-hash", "h1", "--save", str(out)] + ) + assert result.exit_code == 0 + # raw_field means the saved artefact is the text itself, not the envelope. + assert out.read_text() == "IMPORTANT DATA" + + @respx.mock + def test_second_retrieve_exits_9(self, runner, cli, whisper_env): + respx.get(f"{WHISPER_BASE}/whisper-retrieve").mock( + return_value=httpx.Response(200, json={"message": "Whisper already delivered"}) + ) + result = runner.invoke(cli, ["whisper", "retrieve", "--whisper-hash", "h1"]) + assert result.exit_code == 9 + assert "once" in json.loads(result.stderr)["error"]["hint"].lower() + + @respx.mock + def test_no_retry_on_consumed_result(self, runner, cli, whisper_env): + """A retry here could consume a result the first call already delivered.""" + route = respx.get(f"{WHISPER_BASE}/whisper-retrieve").mock( + return_value=httpx.Response(406, json={"message": "already delivered"}) + ) + runner.invoke(cli, ["whisper", "retrieve", "--whisper-hash", "h1"]) + assert route.call_count == 1 + + @respx.mock + def test_deployment_run_wait_save_persists_one_shot_result( + self, runner, cli, monkeypatch, sample_file, tmp_path + ): + """CAPTURE2 BUG 2 - `run --wait --save` must persist the one-shot result on + its single read: no 406, no data loss.""" + monkeypatch.setenv("UNSTRACT_DEPLOYMENT_KEY", FAKE_KEY) + monkeypatch.setenv("UNSTRACT_ORG_ID", "org_test") + base = "https://us-central.unstract.com" + respx.post(f"{base}/deployment/api/org_test/inv/").mock( + return_value=httpx.Response( + 200, json={"message": {"execution_id": "e-9", "execution_status": "PENDING"}} + ) + ) + respx.get(f"{base}/deployment/api/org_test/inv/").mock( + return_value=httpx.Response( + 200, json={"status": "COMPLETED", + "message": [{"file": "bill.pdf", "result": {"invoice_no": "X-1"}}]} + ) + ) + out = tmp_path / "result.json" + result = runner.invoke( + cli, + ["docstudio", "deployment", "run", "--api-name", "inv", + "--file", str(sample_file), "--wait", "--save", str(out), + "--base-url", base], + ) + assert result.exit_code == 0, result.stderr + assert json.loads(out.read_text())["message"][0]["result"]["invoice_no"] == "X-1" + + +class TestDryRun: + def test_prints_request_and_sends_nothing(self, runner, cli, whisper_env, sample_file): + with respx.mock: + route = respx.post(f"{WHISPER_BASE}/whisper") + result = runner.invoke( + cli, ["whisper", "extract", "--file", str(sample_file), "--dry-run"] + ) + assert result.exit_code == 0 + assert route.call_count == 0, "--dry-run must not send the request" + payload = json.loads(result.stdout) + assert payload["method"] == "POST" + assert payload["query"]["mode"] == "form" + + def test_redacts_secrets(self, runner, cli, whisper_env, sample_file): + result = runner.invoke( + cli, ["whisper", "extract", "--file", str(sample_file), "--dry-run"] + ) + assert FAKE_KEY not in result.stdout + assert "***REDACTED***" in result.stdout + + +class TestNoSecretLeaks: + """SPEC §5.7 - a credential must not reach any stream, ever.""" + + @respx.mock + def test_verbose_error_path_stays_clean(self, runner, cli, whisper_env): + respx.get(f"{WHISPER_BASE}/get-usage-info").mock( + return_value=httpx.Response(401, json={"message": f"bad key {FAKE_KEY}"}) + ) + result = runner.invoke(cli, ["whisper", "usage", "-vv", "--no-retry"]) + assert FAKE_KEY not in result.stdout + assert FAKE_KEY not in result.stderr, "the key leaked through the API's own message" + + +class TestDiscover: + """SPEC §5.3 - the machine-readable index agents discover the CLI through.""" + + def test_valid_json_covering_every_endpoint(self, runner, cli): + # The command list lives at --detail summary (or full); the default is now + # the cheaper groups overview. + result = runner.invoke(cli, ["--discover", "--detail", "summary"]) + assert result.exit_code == 0 + data = json.loads(result.stdout) + endpoints = [c for c in data["commands"] if c["kind"] == "endpoint"] + assert len(endpoints) == len(ALL_ENDPOINTS) + + def test_default_is_groups_overview(self, runner, cli): + """The default --discover is the cheap navigable-groups map, not the flat + command list -- ~1k tokens instead of ~4.5k.""" + result = runner.invoke(cli, ["--discover"]) + assert result.exit_code == 0 + data = json.loads(result.stdout) + assert data["detail"] == "groups" + assert "commands" not in data, "the groups overview must not carry the flat list" + assert data["groups"], "must list navigable groups" + # It stays the entry point, so it keeps the global boilerplate. + assert data["exit_codes"]["9"].startswith("result already consumed") + assert "conventions" in data + # And it is genuinely cheaper than the flat summary. + summary = json.loads( + runner.invoke(cli, ["--discover", "--detail", "summary"]).stdout + ) + assert len(result.stdout) < len(json.dumps(summary)) / 2 + + def test_group_overview_drill_hints_resolve(self, runner, cli): + """The scar this project carries: discovery must never advertise a path the + parser won't honour. Every group's `drill` command must return the count it + advertised.""" + overview = json.loads(runner.invoke(cli, ["--discover"]).stdout) + for g in overview["groups"]: + if g.get("note"): + continue # direct-only lines advertise a partial count by design + filt = g["drill"].removeprefix("unstract --discover --command ") + filt = filt.removesuffix(" --detail summary").strip().strip("'") + got = discover(command=filt) + assert got["count"] == g["commands"], ( + f"{g['group']!r} advertised {g['commands']} but drill returned {got['count']}" + ) + + def test_group_overview_is_drift_free(self, runner, cli): + """Every command reachable in the flat list falls under some overview group, + and the advertised counts sum to the whole surface.""" + overview = json.loads(runner.invoke(cli, ["--discover"]).stdout) + # Counts of non-direct-only group lines partition the command surface. + total = sum(g["commands"] for g in overview["groups"] if not g.get("note")) + # The one direct-only line (docstudio platform's own `share`) adds its leaves. + total += sum(g["commands"] for g in overview["groups"] if g.get("note")) + assert total == overview["command_count"] + + def test_click_to_info_dict_contract(self): + """R3's retirement rests on this exact shape. + + If a Click upgrade reshapes `to_info_dict()`, discovery would silently + degrade -- so the dependency bump must fail here instead. + """ + import click + + info = click.Option( + ["--mode"], type=click.Choice(["a", "b"]), default="a", help="h" + ).to_info_dict() + for key in ("name", "opts", "type", "required", "multiple", "default", "help"): + assert key in info, f"Click no longer reports {key!r}" + assert info["type"]["choices"] == ("a", "b") + + def test_flags_carry_full_metadata(self): + extract = next( + c for c in discover(detail="full")["commands"] + if c["command"] == "unstract whisper extract" + ) + mode = next(f for f in extract["flags"] if f["name"] == "mode") + assert mode["choices"] == ["native_text", "low_cost", "high_quality", "form", "table"] + assert mode["default"] == "form" + assert extract["api"] == {"method": "POST", "path": "/whisper"} + assert extract["supports_wait"] and extract["one_shot"] + + def test_local_commands_flagged(self): + """An agent must be able to tell local operations from API calls.""" + data = discover(detail="full") + local = [c for c in data["commands"] if c["kind"] == "local"] + assert {c["command"] for c in local} >= {"unstract config init", "unstract config use"} + assert all("api" not in c for c in local) + + def test_documents_exit_codes_and_conventions(self): + data = discover() + assert data["exit_codes"]["9"].startswith("result already consumed") + assert "never_interactive" in data["conventions"] + + def test_flag_is_discover_only(self, runner, cli): + """`--dump-commands` was the original name and is fully removed. + + Pinned by a test so it cannot creep back in via a copied example. + """ + assert "--discover" in runner.invoke(cli, ["--help"]).stdout + result = runner.invoke(cli, ["--dump-commands"]) + assert result.exit_code != 0 + + def test_advertised_params_match_the_real_parser(self, cli): + """The index must describe what the parser actually accepts. + + This is the guarantee `--discover` exists to provide. It once failed for + the hand-authored `config` group, which was described by a parallel data + structure rather than introspected: it advertised `--product/--key/--value` + for `config set`, whose parameters are positional. An agent following + that index would build a command line the parser rejects. + """ + mismatches = [] + for entry in discover(detail="full")["commands"]: + node = cli + for part in entry["path"]: + node = node.commands.get(part) if hasattr(node, "commands") else None + if node is None: + break + assert node is not None, f"{entry['command']} is advertised but not registered" + + real = { + p.to_info_dict()["name"]: ( + "argument" + if p.to_info_dict()["param_type_name"] == "argument" + else "option" + ) + for p in node.params + } + for flag in entry["flags"]: + if real.get(flag["name"]) != flag["kind"]: + mismatches.append( + f"{entry['command']}: {flag['name']} advertised as " + f"{flag['kind']}, parser says {real.get(flag['name'])}" + ) + assert not mismatches, "\n".join(mismatches) + + def test_positional_args_are_not_advertised_as_flags(self): + """A positional must never carry a `--flag` spelling.""" + entry = next( + c for c in discover(command="config set", detail="full")["commands"] + ) + arg = next(f for f in entry["flags"] if f["kind"] == "argument") + assert "flags" not in arg, "a positional is not a flag" + # The metavar spells out the real shape; the bare name ("ARGS") would not. + assert entry["usage"] == "unstract config set TARGET... KEY VALUE" + + def test_usage_line_present_for_every_command(self): + """`usage` shows the invocation form, which flags alone cannot convey.""" + for entry in discover(detail="full")["commands"]: + assert entry["usage"].startswith("unstract "), entry["command"] + + def test_choices_reflect_the_real_enum(self): + """Advertised choices must come from the parser, not a copy of it.""" + entry = next( + c for c in discover(command="whisper extract", detail="full")["commands"] + ) + mode = next(f for f in entry["flags"] if f["name"] == "mode") + assert "form" in mode["choices"] and "high_quality" in mode["choices"] + + def test_summary_entries_are_minimal(self): + """At --detail summary each command carries only what's needed to choose + it -- names and one-liners, no flags.""" + data = discover(detail="summary") + assert data["detail"] == "summary" + entry = data["commands"][0] + assert set(entry) <= {"command", "kind", "summary"} + assert "flags" not in entry + + def test_summary_explains_how_to_drill_down(self): + data = discover(detail="summary") + assert "--group" in data["drill_down"]["one_group"] + assert set(data["groups"]) >= {"whisper", "docstudio", "apihub", "config"} + + def test_filter_by_group(self): + data = discover(group="whisper") + assert data["count"] == 11 + assert all(c["command"].startswith("unstract whisper") for c in data["commands"]) + + def test_filter_by_exact_command(self): + data = discover(command="whisper extract", detail="full") + assert data["count"] == 1 + assert data["commands"][0]["api"] == {"method": "POST", "path": "/whisper"} + + def test_filter_by_command_prefix(self): + """A prefix selects a subtree, so `whisper webhook` gets all four.""" + assert discover(command="whisper webhook")["count"] == 4 + + def test_filter_tolerates_unstract_prefix(self): + """`command` values are copied from output, which includes 'unstract '.""" + assert discover(command="unstract whisper extract")["count"] == 1 + + def test_group_and_command_combine(self): + assert discover(group="docstudio", command="docstudio platform adapter")["count"] == 13 + assert discover(group="whisper", command="docstudio platform adapter")["count"] == 0 + + def test_detail_is_independent_of_selection(self): + """Both axes are orthogonal: any combination must work.""" + for group in (None, "whisper"): + for detail in ("summary", "full"): + data = discover(group=group, detail=detail) + assert data["count"] > 0 + assert data["detail"] == detail + + def test_narrow_queries_omit_global_boilerplate(self): + """On a filtered view the exit-code table would outweigh the answer.""" + narrow = discover(group="whisper") + assert "exit_codes" not in narrow + assert "exit_codes" in discover() + + def test_no_match_exits_2_with_parseable_error(self, runner, cli): + """DOC 9 - the error envelope is mirrored to stdout when piped, so a + consumer parsing --discover output sees a valid object, not empty input.""" + result = runner.invoke(cli, ["--discover", "--group", "nope"]) + assert result.exit_code == 2 + assert "hint" in json.loads(result.stdout)["error"] + assert "hint" in json.loads(result.stderr)["error"] + + def test_compact_when_piped(self, runner, cli): + """Pretty-printing costs ~36% more tokens for a consumer that parses it.""" + out = runner.invoke(cli, ["--discover", "--group", "whisper"]).stdout + assert json.loads(out)["count"] == 11 + assert "\n " not in out, "output should be compact when stdout is not a TTY" + + def test_doc_conflict_recorded(self): + """The whisper-detail divergence must be visible, so it is not 'fixed'.""" + detail = next( + c for c in discover(detail="full")["commands"] + if c["command"] == "unstract whisper detail" + ) + assert "singular" in detail["doc_conflict"] + + +class TestHelp: + def test_every_group_renders_help(self, runner, cli): + for group in ("whisper", "docstudio", "apihub", "config"): + result = runner.invoke(cli, [group, "--help"]) + assert result.exit_code == 0, f"{group} --help failed" + + def test_leaf_help_shows_api_and_examples(self, runner, cli): + result = runner.invoke(cli, ["whisper", "extract", "--help"]) + assert "POST /whisper" in result.stdout + assert "Examples:" in result.stdout + + def test_destructive_command_states_permission(self, runner, cli): + result = runner.invoke(cli, ["docstudio", "platform", "workflow", "delete", "--help"]) + assert "full_access" in result.stdout + + def test_one_shot_help_warns(self, runner, cli): + result = runner.invoke(cli, ["whisper", "retrieve", "--help"]) + assert "ONE-SHOT" in result.stdout + + def test_unknown_command_suggests(self, runner, cli): + result = runner.invoke(cli, ["whisperr"]) + assert result.exit_code != 0 + assert "whisper" in result.stderr + + +class TestFlagCoverage: + """SPEC §1.1.3 - every documented parameter must be reachable as a flag.""" + + def test_all_params_surface_as_flags(self, cli): + data = discover(detail="full") + by_command = {c["command"]: c for c in data["commands"]} + for endpoint in ALL_ENDPOINTS: + entry = by_command["unstract " + " ".join(endpoint.command_path)] + exposed = {f["name"] for f in entry["flags"]} + for param in endpoint.params: + assert param.py_name in exposed, ( + f"{endpoint.dotted_name}: {param.name} is not reachable as a flag" + ) + + +class TestConfigCommands: + """M1.5 - hand-authored commands, and never a prompt.""" + + def test_init_writes_0600(self, runner, cli, isolated_env): + result = runner.invoke(cli, ["config", "init"]) + assert result.exit_code == 0 + path = isolated_env / "config.toml" + assert path.exists() + assert stat.S_IMODE(os.stat(path).st_mode) == 0o600 + + def test_init_refuses_to_clobber_without_force(self, runner, cli): + runner.invoke(cli, ["config", "init"]) + result = runner.invoke(cli, ["config", "init"]) + assert result.exit_code == 2 + assert "--force" in json.loads(result.stderr)["error"]["hint"] + + def test_init_force_overwrites(self, runner, cli): + runner.invoke(cli, ["config", "init"]) + assert runner.invoke(cli, ["config", "init", "--force"]).exit_code == 0 + + def test_use_switches_default_profile(self, runner, cli): + runner.invoke(cli, ["config", "init"]) + assert runner.invoke(cli, ["config", "use", "cloud-eu"]).exit_code == 0 + current = json.loads(runner.invoke(cli, ["config", "current"]).stdout) + assert current["active_profile"] == "cloud-eu" + + def test_use_rejects_unknown_profile(self, runner, cli): + runner.invoke(cli, ["config", "init"]) + result = runner.invoke(cli, ["config", "use", "nope"]) + assert result.exit_code == 2 + + def test_starter_config_contains_no_secrets(self, runner, cli, isolated_env): + runner.invoke(cli, ["config", "init"]) + content = (isolated_env / "config.toml").read_text() + assert "env:" in content + assert "sk-" not in content + + def test_get_never_echoes_a_secret(self, runner, cli, whisper_env): + result = runner.invoke(cli, ["config", "get", "llmwhisperer", "api_key"]) + assert result.exit_code == 0 + payload = json.loads(result.stdout) + assert payload["configured"] is True + assert FAKE_KEY not in result.stdout + + @pytest.mark.parametrize( + "target", [["docstudio.platform"], ["docstudio", "platform"]] + ) + def test_both_separators_address_the_same_group(self, runner, cli, target): + """`docstudio platform` is what a shell user types; the dot is canonical.""" + runner.invoke(cli, ["config", "init"]) + assert runner.invoke( + cli, ["config", "set", *target, "org_id", "org_X"] + ).exit_code == 0 + shown = json.loads( + runner.invoke(cli, ["config", "get", *target, "org_id"]).stdout + ) + assert shown["value"] == "org_X" + + @pytest.mark.parametrize("bare", ["platform", "deployment", "hitl", "whisper"]) + def test_unqualified_target_is_rejected(self, runner, cli, bare): + """A group owned by a product must be named through it. + + A bare `platform` hides which product the setting belongs to, so it is + an error rather than a silent guess. + """ + runner.invoke(cli, ["config", "init"]) + result = runner.invoke(cli, ["config", "set", bare, "org_id", "x"]) + assert result.exit_code == 2 + error = json.loads(result.stderr)["error"] + assert "docstudio.platform" in error["hint"], "the hint must list valid targets" + + @pytest.mark.parametrize( + "block", ["[profiles.p.whisper]", "[profiles.p.platform]", "[profiles.p.hitl]"] + ) + def test_non_canonical_blocks_are_ignored(self, runner, cli, isolated_env, block): + """There is exactly one accepted layout -- no aliases, no flat fallback. + + A block written any other way must NOT be silently picked up: a config + that looks applied but is not surfaces later as a missing-credential + error with no obvious cause. + """ + (isolated_env / "config.toml").write_text( + f'default_profile = "p"\n\n{block}\nbase_url = "https://wrong.example/"\n' + ) + current = json.loads(runner.invoke(cli, ["config", "current"]).stdout) + for settings in current["settings"].values(): + assert settings.get("base_url") != "https://wrong.example/" + + def test_canonical_block_is_read(self, runner, cli, isolated_env): + (isolated_env / "config.toml").write_text( + 'default_profile = "p"\n\n' + "[profiles.p.docstudio.platform]\n" + 'org_id = "org_CANON"\n' + ) + shown = json.loads( + runner.invoke(cli, ["config", "get", "docstudio.platform", "org_id"]).stdout + ) + assert shown["value"] == "org_CANON" + + def test_set_writes_where_get_reads(self, runner, cli, isolated_env): + """A write must land in the block the reader consults. + + Config blocks nest by product ([profiles.X.docstudio.platform]); writing + a flat block instead would be silently ignored on read. + """ + runner.invoke(cli, ["config", "init"]) + assert runner.invoke( + cli, ["config", "set", "docstudio.platform", "org_id", "org_ROUNDTRIP"] + ).exit_code == 0 + + shown = json.loads(runner.invoke(cli, ["config", "get", "docstudio.platform", "org_id"]).stdout) + assert shown["value"] == "org_ROUNDTRIP" + + text = (isolated_env / "config.toml").read_text() + assert "[profiles.cloud-us.docstudio.platform]" in text + assert "[profiles.cloud-us.platform]" not in text, "stray flat block written" + + def test_docstudio_owns_three_api_groups(self, runner, cli, isolated_env): + """Document Studio's groups keep separate credentials and hosts.""" + runner.invoke(cli, ["config", "init"]) + text = (isolated_env / "config.toml").read_text() + for api in ("platform", "deployment", "hitl"): + assert f"[profiles.cloud-us.docstudio.{api}]" in text + + def test_config_flag_selects_a_file(self, runner, cli, tmp_path): + """Several config files can coexist; --config picks one per invocation.""" + alt = tmp_path / "alt.toml" + assert runner.invoke(cli, ["--config", str(alt), "config", "init"]).exit_code == 0 + assert alt.exists() + + shown = json.loads( + runner.invoke(cli, ["--config", str(alt), "config", "path"]).stdout + ) + assert shown["path"] == str(alt) + + def test_config_flag_outranks_env(self, runner, cli, monkeypatch, tmp_path): + env_file, flag_file = tmp_path / "env.toml", tmp_path / "flag.toml" + runner.invoke(cli, ["--config", str(flag_file), "config", "init"]) + monkeypatch.setenv("UNSTRACT_CONFIG", str(env_file)) + + shown = json.loads( + runner.invoke(cli, ["--config", str(flag_file), "config", "path"]).stdout + ) + assert shown["path"] == str(flag_file), "flag must outrank $UNSTRACT_CONFIG" + + def test_project_config_found_from_subdirectory(self, runner, cli, tmp_path, monkeypatch): + """A project's `.unstract.toml` applies to anyone working inside it. + + Found by walking upward, the way git and ruff locate their settings, so a + repo can commit its own hosts and org without every caller passing flags. + """ + monkeypatch.delenv("UNSTRACT_CONFIG", raising=False) + project = tmp_path / "proj" + (project / "deep" / "nested").mkdir(parents=True) + (project / ".unstract.toml").write_text( + 'default_profile = "proj"\n\n' + '[profiles.proj.llmwhisperer]\n' + 'base_url = "https://project.example/api/v2"\n' + ) + monkeypatch.chdir(project / "deep" / "nested") + + shown = json.loads(runner.invoke(cli, ["config", "path"]).stdout) + assert shown["path"] == str(project / ".unstract.toml") + + current = json.loads(runner.invoke(cli, ["config", "current"]).stdout) + assert current["active_profile"] == "proj" + assert current["settings"]["llmwhisperer"]["base_url"].startswith( + "https://project.example" + ) + + def test_project_config_search_stops_at_home(self, monkeypatch, tmp_path): + """A stray file above $HOME must not capture every invocation.""" + from unstract_cli.config.loader import find_project_config + + monkeypatch.setattr(Path, "home", classmethod(lambda cls: tmp_path)) + (tmp_path.parent / ".unstract.toml").write_text("") + work = tmp_path / "work" + work.mkdir() + try: + assert find_project_config(work) is None + finally: + (tmp_path.parent / ".unstract.toml").unlink(missing_ok=True) + + def test_profile_selects_region(self, runner, cli, whisper_env): + runner.invoke(cli, ["config", "init"]) + with respx.mock: + route = respx.get( + "https://llmwhisperer-api.eu-west.unstract.com/api/v2/get-usage-info" + ).mock(return_value=httpx.Response(200, json={"plan": "eu"})) + result = runner.invoke(cli, ["whisper", "usage", "--profile", "cloud-eu"]) + assert result.exit_code == 0, result.stderr + assert route.call_count == 1 + + def test_doctor_names_the_unset_env_var(self, runner, cli, isolated_env, monkeypatch): + """DOC 8 - an `env:` ref pointing at a variable absent from THIS process + is the classic silent-auth-failure trap; doctor must name it.""" + monkeypatch.delenv("LLMWHISPERER_API_KEY", raising=False) + (isolated_env / "config.toml").write_text( + 'default_profile = "p"\n\n' + "[profiles.p.llmwhisperer]\n" + 'api_key = "env:LLMWHISPERER_API_KEY"\n' + ) + report = json.loads(runner.invoke(cli, ["config", "doctor", "--no-check"]).stdout) + whisper = next(g for g in report["groups"] if g["target"] == "llmwhisperer") + assert whisper["api_key"]["resolved"] is False + assert "LLMWHISPERER_API_KEY" in whisper["api_key"]["detail"] + + def test_doctor_reports_env_source_when_present(self, runner, cli, isolated_env, whisper_env): + (isolated_env / "config.toml").write_text( + 'default_profile = "p"\n\n' + "[profiles.p.llmwhisperer]\n" + 'api_key = "env:LLMWHISPERER_API_KEY"\n' + ) + report = json.loads(runner.invoke(cli, ["config", "doctor", "--no-check"]).stdout) + whisper = next(g for g in report["groups"] if g["target"] == "llmwhisperer") + assert whisper["api_key"]["resolved"] is True + assert "LLMWHISPERER_API_KEY" in whisper["api_key"]["source"] + + def test_doctor_live_check_reports_auth(self, runner, cli, isolated_env, whisper_env): + (isolated_env / "config.toml").write_text( + 'default_profile = "p"\n\n' + "[profiles.p.llmwhisperer]\n" + 'api_key = "env:LLMWHISPERER_API_KEY"\n' + ) + with respx.mock: + respx.get(f"{WHISPER_BASE}/get-usage-info").mock( + return_value=httpx.Response(200, json={"plan": "free"}) + ) + report = json.loads(runner.invoke(cli, ["config", "doctor"]).stdout) + whisper = next(g for g in report["groups"] if g["target"] == "llmwhisperer") + assert whisper["live_check"]["ok"] is True + + def test_doctor_never_echoes_the_secret(self, runner, cli, isolated_env): + (isolated_env / "config.toml").write_text( + 'default_profile = "p"\n\n' + "[profiles.p.llmwhisperer]\n" + 'api_key = "sk-literal-secret-value"\n' + ) + out = runner.invoke(cli, ["config", "doctor", "--no-check"]).stdout + assert "sk-literal-secret-value" not in out + assert "literal" in out diff --git a/tests/test_exit_codes.py b/tests/test_exit_codes.py new file mode 100644 index 0000000..f1ece56 --- /dev/null +++ b/tests/test_exit_codes.py @@ -0,0 +1,63 @@ +"""Exit codes must reach the shell (SPEC §5.4). + +These run the CLI as a **subprocess**, deliberately. Click's `CliRunner` +invokes commands with `standalone_mode=True` and never executes +`unstract_cli.__main__.main`, so the entire class of "the contract is right but +the entry point drops it" is invisible to it — which is exactly the bug these +were written for: `main()` discarded the code Click returns under +`standalone_mode=False`, so every failure exited 0 while its error envelope +reported `"exit_code": 2`. `set -e` never tripped, and an agent branching on the +status code saw success. +""" + +from __future__ import annotations + +import os +import subprocess +import sys + +import pytest + + +def run_cli(*args: str, **env: str) -> int: + """Run the CLI in a subprocess and return its real process exit code.""" + return subprocess.run( + [sys.executable, "-m", "unstract_cli", *args], + capture_output=True, + env={ + **os.environ, + "UNSTRACT_PLATFORM_KEY": "k", + "UNSTRACT_ORG_ID": "o", + **env, + }, + ).returncode + + +#: (argv, expected code). Each row pins one documented code at the process +#: boundary; `--dry-run` keeps the success cases off the network. +CASES: list[tuple[list[str], int]] = [ + # Usage error (2): a constraint rejected before any request is sent. + ( + [ + "docstudio", "platform", "prompt-studio", "profile", "create", + "--tool-id", "t", "--profile-name", "p", "--llm", "l", + "--x2text", "x", "--chunk-size", "1024", + ], + 2, + ), + # Usage error (2): Click's own parse failure, routed through the same envelope. + (["whisper", "--definitely-not-a-flag"], 2), + (["docstudio", "platform", "prompt-studio", "nonexistent-command"], 2), + # Success (0): the paths that must not regress into a non-zero code. + (["docstudio", "platform", "prompt-studio", "list", "--dry-run"], 0), + (["--help"], 0), + (["--discover"], 0), +] + + +@pytest.mark.parametrize("argv,expected", CASES, ids=lambda v: None) +def test_exit_code_reaches_the_shell(argv: list[str], expected: int) -> None: + assert run_cli(*argv) == expected, ( + f"`unstract {' '.join(argv)}` should exit {expected}. A failure exiting 0 " + "silently breaks `set -e` and any agent branching on the status code." + ) diff --git a/tests/test_gotchas.py b/tests/test_gotchas.py new file mode 100644 index 0000000..e9c493e --- /dev/null +++ b/tests/test_gotchas.py @@ -0,0 +1,245 @@ +"""Regressions for the friction points recorded in GOTCHAS.md. + +Each test pins the *behaviour a user hit*, not the implementation that fixes it, +so a later refactor is free to change the mechanism but cannot quietly restore +the friction. Numbers refer to GOTCHAS.md sections. +""" + +from __future__ import annotations + +import pytest + +from unstract_cli.config.loader import ConfigFile, ResolvedConfig +from unstract_cli.core.errors import hint_for +from unstract_cli.core.http import build_request +from unstract_cli.core.model import RequiredUnless +from unstract_cli.endpoints import get_endpoint + +from .conftest import FAKE_KEY + + +def _config(monkeypatch, **env) -> ResolvedConfig: + for key, value in env.items(): + monkeypatch.setenv(key, value) + return ResolvedConfig(file=ConfigFile(exists=False)) + + +class TestProfileManagerOnPromptCreate: + """#1 - `fetch-response` reads the PROMPT's profile, never the project default.""" + + def test_prompt_create_accepts_a_profile(self): + param = get_endpoint("docstudio.platform.prompt-studio.prompt.create").param( + "profile_manager" + ) + assert param is not None, "prompt create must be able to set the LLM profile" + assert param.location.value == "body" + + def test_profile_reaches_the_body(self, monkeypatch): + config = _config(monkeypatch, UNSTRACT_PLATFORM_KEY=FAKE_KEY, UNSTRACT_ORG_ID="org_1") + plan = build_request( + get_endpoint("docstudio.platform.prompt-studio.prompt.create"), + config, + {"tool_id": "t-1", "prompt_key": "invoice_no", "profile_manager": "p-9"}, + ) + assert plan.json_body["profile_manager"] == "p-9" + # The tool_id mirror must survive alongside it (the older BUG 2 fix). + assert plan.json_body["tool_id"] == "t-1" + + def test_the_misleading_error_gets_a_corrective_hint(self): + hint = hint_for(500, message="Default LLM profile is not configured.") + assert hint and "profile_manager" in hint + # The whole point: stop the reader chasing `profile set-default`. + assert "does NOT fall back" in hint or "not fall back" in hint.lower() + + +class TestChallengeLlm: + """#2 - the 422 that only surfaced at the final `deployment run`.""" + + def test_project_can_set_challenge_llm_before_export(self): + for command in ( + "docstudio.platform.prompt-studio.create", + "docstudio.platform.prompt-studio.patch", + ): + assert get_endpoint(command).param("challenge_llm") is not None + + def test_tool_validation_failure_explains_the_fix(self): + hint = hint_for(422, message="Tool validation failed") + assert hint and "challenge_llm" in hint + assert "set-metadata" in hint + + +class TestVectorStoreStaysRequired: + """#3 - asked for `--vector-store` to be optional at `--chunk-size 0`. + + It cannot be: `ProfileManager.vector_store` and `.embedding_model` are + `null=False` and the serializer is `fields = "__all__"`, so DRF derives + required=True and the server rejects the profile whatever chunk_size says. + Relaxing the local rule would only trade a fast exit-2 for a slow remote 400, + so the requirement stays and the help explains what to pass. These tests pin + that decision so it is not "re-fixed" into a regression. + """ + + @pytest.mark.parametrize( + "command", + [ + "docstudio.platform.prompt-studio.profile.create", + "docstudio.platform.prompt-studio.profile.update", + ], + ) + def test_both_adapters_stay_required(self, command): + for name in ("vector_store", "embedding_model"): + assert get_endpoint(command).param(name).required, ( + f"{name} must stay required: the server rejects a profile without " + "it even when chunk_size=0" + ) + + def test_help_explains_the_chunk_size_zero_case(self): + endpoint = get_endpoint("docstudio.platform.prompt-studio.profile.create") + assert "chunk-size 0" in endpoint.description + # The caller must learn that the value is stored but unused, so that + # supplying "any valid adapter id" reads as intended rather than a bodge. + assert "REQUIRES" in endpoint.description + + +class TestRequiredUnlessPrimitive: + """The conditional-required constraint itself, kept for cases that need it.""" + + def test_sentinel_value_relaxes_the_requirement(self): + constraint = RequiredUnless(("vector_store",), unless="chunk_size", unless_values=(0,)) + assert constraint.check({"chunk_size": 0}) is None + assert constraint.check({"chunk_size": 1024}) is not None + + def test_string_and_int_sentinels_agree(self): + # Click hands ints through typed, but config defaults and tests may not. + constraint = RequiredUnless(("vector_store",), unless="chunk_size", unless_values=(0,)) + assert constraint.check({"chunk_size": "0"}) is None + + def test_supplying_the_param_always_satisfies_it(self): + constraint = RequiredUnless(("vector_store",), unless="chunk_size", unless_values=(0,)) + assert constraint.check({"chunk_size": 1024, "vector_store": "v-1"}) is None + + +class TestSingleJsonDocumentOnStdout: + """#4 - reported as duplicate output; stdout must carry exactly one document.""" + + def test_stdout_parses_as_one_json_object(self, runner, cli, monkeypatch): + import json + + monkeypatch.setenv("UNSTRACT_PLATFORM_KEY", FAKE_KEY) + monkeypatch.setenv("UNSTRACT_ORG_ID", "org_1") + result = runner.invoke( + cli, + ["docstudio", "platform", "prompt-studio", "list", "--dry-run", "-o", "json"], + ) + assert result.exit_code == 0 + json.loads(result.stdout) # raises "Extra data" if the payload were doubled + + +class TestKeyCreateNeedsOneIdentifier: + """#6 - `--api-id` alone must suffice; the body's `api` is the same value.""" + + def test_api_key_create_mirrors_the_path_id(self, monkeypatch): + config = _config(monkeypatch, UNSTRACT_PLATFORM_KEY=FAKE_KEY, UNSTRACT_ORG_ID="org_1") + plan = build_request( + get_endpoint("docstudio.platform.api-deployment.key.create"), + config, + {"api_id": "dep-1"}, + ) + assert plan.json_body["api"] == "dep-1" + assert "dep-1" in plan.url + + def test_pipeline_key_create_mirrors_the_path_id(self, monkeypatch): + config = _config(monkeypatch, UNSTRACT_PLATFORM_KEY=FAKE_KEY, UNSTRACT_ORG_ID="org_1") + plan = build_request( + get_endpoint("docstudio.platform.pipeline.key.create"), + config, + {"pipeline_id": "pipe-1"}, + ) + assert plan.json_body["pipeline"] == "pipe-1" + + def test_no_second_flag_is_demanded(self): + # The old MutuallyExclusive(api, pipeline) rejected `--api-id` on its own. + assert get_endpoint("docstudio.platform.api-deployment.key.create").validate( + {"api_id": "dep-1"} + ) == [] + + +class TestOrgIdFallsBackToPlatform: + """#7 - the deployment/hitl config blocks start empty, but the org is the same.""" + + def test_deployment_run_uses_the_platform_org_id(self, monkeypatch): + config = _config( + monkeypatch, UNSTRACT_DEPLOYMENT_KEY=FAKE_KEY, UNSTRACT_ORG_ID="org_platform" + ) + plan = build_request( + get_endpoint("docstudio.deployment.run"), + config, + {"api_name": "invoice-api"}, + ) + assert "org_platform" in plan.url + + def test_an_explicit_org_id_still_wins(self, monkeypatch): + config = _config( + monkeypatch, UNSTRACT_DEPLOYMENT_KEY=FAKE_KEY, UNSTRACT_ORG_ID="org_platform" + ) + plan = build_request( + get_endpoint("docstudio.deployment.run"), + config, + {"api_name": "invoice-api", "org_id": "org_explicit"}, + ) + assert "org_explicit" in plan.url + + +class TestIndexDocumentSupportsWait: + """#8 - index-document was the only async command without --wait.""" + + def test_it_declares_a_poll_spec(self): + endpoint = get_endpoint("docstudio.platform.prompt-studio.index-document") + assert endpoint.poll is not None + assert endpoint.poll.handle_field == "task_id" + + def test_indexing_has_nothing_to_retrieve(self): + # Indexing writes no Output Manager row, so the terminal status IS the + # result; a retrieve step would fetch an unrelated prompt's output. + assert get_endpoint( + "docstudio.platform.prompt-studio.index-document" + ).poll.retrieve_endpoint is None + + def test_wait_flag_is_exposed(self, runner, cli): + result = runner.invoke( + cli, + ["docstudio", "platform", "prompt-studio", "index-document", "--help"], + ) + assert "--wait" in result.output + + +class TestDocumentedServerLimitations: + """#5, #9, #10 - not CLI-fixable; the help must say so rather than mislead.""" + + def test_export_tool_explains_the_new_registry_id(self): + description = get_endpoint( + "docstudio.platform.prompt-studio.export-tool" + ).description + assert "function_name" in description and "not" in description.lower() + + def test_project_delete_documents_the_registry_cascade(self): + description = get_endpoint("docstudio.platform.prompt-studio.delete").description + assert "cascade" in description.lower() or "registry" in description.lower() + + def test_default_triad_explains_an_empty_object(self): + description = get_endpoint( + "docstudio.platform.adapter.default-triad.get" + ).description + assert "{}" in description + + def test_adapter_choices_points_at_the_working_alternative(self): + description = get_endpoint( + "docstudio.platform.prompt-studio.adapter-choices" + ).description + assert "adapter list" in description + + def test_task_status_says_it_needs_the_tool_id(self): + description = get_endpoint( + "docstudio.platform.prompt-studio.task-status" + ).description + assert "--tool-id" in description diff --git a/tests/test_http.py b/tests/test_http.py new file mode 100644 index 0000000..058f89b --- /dev/null +++ b/tests/test_http.py @@ -0,0 +1,414 @@ +"""HTTP layer: auth injection, exit-code mapping, retries and redaction.""" + +from __future__ import annotations + +import httpx +import pytest +import respx + +from unstract_cli.config.loader import ( + ConfigError, + ConfigFile, + ResolvedConfig, + load_config, +) +from unstract_cli.core.errors import ( + REDACTED, + CLIError, + ExitCode, + exit_code_for_status, + is_retryable, + redact_headers, + scrub, +) +from unstract_cli.core.http import ( + GATEWAY_INJECTED_HEADERS, + auth_headers, + build_request, + execute, + raise_for_status, +) +from unstract_cli.core.model import ApiGroup +from unstract_cli.endpoints import get_endpoint + +from .conftest import FAKE_KEY, WHISPER_BASE + + +def _config(monkeypatch, **env) -> ResolvedConfig: + for key, value in env.items(): + monkeypatch.setenv(key, value) + return ResolvedConfig(file=ConfigFile(exists=False)) + + +class TestExitCodeMapping: + """SPEC §5.4 - the table an agent branches on.""" + + @pytest.mark.parametrize( + "status,expected", + [ + (200, ExitCode.SUCCESS), + (400, ExitCode.VALIDATION), + (401, ExitCode.AUTH), + (403, ExitCode.AUTH), + (404, ExitCode.NOT_FOUND), + (406, ExitCode.ALREADY_CONSUMED), + (409, ExitCode.VALIDATION), + (422, ExitCode.VALIDATION), + (429, ExitCode.RATE_LIMITED), + (500, ExitCode.SERVER_ERROR), + (503, ExitCode.SERVER_ERROR), + ], + ) + def test_status_maps_to_exit_code(self, status, expected): + assert exit_code_for_status(status) is expected + + @pytest.mark.parametrize("status", [429, 500, 502, 503]) + def test_retryable(self, status): + assert is_retryable(status) + + @pytest.mark.parametrize("status", [400, 401, 403, 404, 409, 422]) + def test_not_retryable(self, status): + """Retrying a 4xx re-sends a request already rejected on its merits -- + and for one-shot reads could consume a result the first call delivered.""" + assert not is_retryable(status) + + +class TestAuth: + """SPEC §4.4 - three products, three schemes.""" + + def test_llmwhisperer_uses_unstract_key(self, monkeypatch): + cfg = _config(monkeypatch, LLMWHISPERER_API_KEY=FAKE_KEY) + assert auth_headers(ApiGroup.LLMWHISPERER, cfg) == {"unstract-key": FAKE_KEY} + + def test_platform_uses_bearer(self, monkeypatch): + cfg = _config(monkeypatch, UNSTRACT_PLATFORM_KEY=FAKE_KEY) + assert auth_headers(ApiGroup.PLATFORM, cfg)["Authorization"] == f"Bearer {FAKE_KEY}" + + def test_apihub_uses_apikey(self, monkeypatch): + cfg = _config(monkeypatch, UNSTRACT_APIHUB_KEY=FAKE_KEY) + assert auth_headers(ApiGroup.APIHUB, cfg)["apikey"] == FAKE_KEY + + def test_apihub_never_sends_gateway_headers(self, monkeypatch): + """Kong injects tenancy headers from its own Redis lookup. + + A client-set value would be overwritten at best, and treated as a tenancy + claim at worst, so the CLI must never emit them. + """ + cfg = _config( + monkeypatch, UNSTRACT_APIHUB_KEY=FAKE_KEY, UNSTRACT_APIHUB_BASE_URL="https://hub.test" + ) + plan = build_request( + get_endpoint("apihub.status"), cfg, {"file_hash": "abc"} + ) + for header in GATEWAY_INJECTED_HEADERS: + assert header not in {k.lower() for k in plan.headers} + + +class TestEnumKeyedLookup: + """Callers pass `ApiGroup`; the resolver must accept it, not just strings. + + When only `Product` was normalised, an `ApiGroup` key fell through + unconverted: profile values were silently missed and error messages read + "ApiGroup.LLMWHISPERER.api_key". + """ + + def test_enum_and_string_agree(self, tmp_path, monkeypatch): + config = tmp_path / "c.toml" + config.write_text( + 'default_profile = "p"\n\n' + "[profiles.p.llmwhisperer]\n" + 'base_url = "https://from-file.example/api/v2"\n' + ) + monkeypatch.setenv("UNSTRACT_CONFIG", str(config)) + cfg = ResolvedConfig(file=load_config(config)) + + assert cfg.get(ApiGroup.LLMWHISPERER, "base_url") == cfg.get( + "llmwhisperer", "base_url" + ) == "https://from-file.example/api/v2" + + def test_error_message_uses_the_plain_name(self, monkeypatch): + monkeypatch.delenv("LLMWHISPERER_API_KEY", raising=False) + cfg = ResolvedConfig(file=ConfigFile(exists=False)) + with pytest.raises(ConfigError) as exc: + cfg.require(ApiGroup.LLMWHISPERER, "api_key") + assert "llmwhisperer.api_key" in str(exc.value) + assert "ApiGroup." not in str(exc.value) + + +class TestRedaction: + """SPEC §5.7 - secrets appear in no output stream.""" + + def test_headers_redacted(self): + out = redact_headers({"unstract-key": FAKE_KEY, "Authorization": f"Bearer {FAKE_KEY}"}) + assert FAKE_KEY not in str(out) + assert out["unstract-key"] == REDACTED + + def test_scrub_removes_literals(self): + assert FAKE_KEY not in scrub(f"failed with key {FAKE_KEY}", [FAKE_KEY]) + + def test_dry_run_describe_redacts(self, monkeypatch): + cfg = _config(monkeypatch, LLMWHISPERER_API_KEY=FAKE_KEY) + plan = build_request(get_endpoint("whisper.status"), cfg, {"whisper_hash": "h"}) + assert FAKE_KEY not in str(plan.describe()) + + +class TestRequestBuilding: + def test_query_params_and_url(self, monkeypatch): + cfg = _config(monkeypatch, LLMWHISPERER_API_KEY=FAKE_KEY) + plan = build_request(get_endpoint("whisper.status"), cfg, {"whisper_hash": "abc123"}) + assert plan.url == f"{WHISPER_BASE}/whisper-status" + assert plan.params["whisper_hash"] == "abc123" + + def test_booleans_serialise_lowercase(self, monkeypatch): + """Python's True would be rejected; the wire format is `true`.""" + cfg = _config(monkeypatch, LLMWHISPERER_API_KEY=FAKE_KEY) + plan = build_request( + get_endpoint("whisper.retrieve"), cfg, {"whisper_hash": "h", "text_only": True} + ) + assert plan.params["text_only"] == "true" + + def test_path_params_substituted_from_env(self, monkeypatch): + cfg = _config( + monkeypatch, UNSTRACT_DEPLOYMENT_KEY=FAKE_KEY, UNSTRACT_ORG_ID="org_abc" + ) + plan = build_request( + get_endpoint("docstudio.deployment.status"), + cfg, + {"api_name": "invoice-api", "execution_id": "e1"}, + ) + assert plan.url.endswith("/deployment/api/org_abc/invoice-api/") + + def test_missing_required_path_param_is_usage_error(self, monkeypatch): + cfg = _config(monkeypatch, UNSTRACT_DEPLOYMENT_KEY=FAKE_KEY, UNSTRACT_ORG_ID="o") + with pytest.raises(CLIError) as exc: + build_request(get_endpoint("docstudio.deployment.status"), cfg, {"execution_id": "e"}) + assert exc.value.exit_code is ExitCode.USAGE + + def test_freeform_ext_param(self, monkeypatch): + """P5 - `--ext-param foo=bar` must reach the wire as `ext_foo=bar`.""" + cfg = _config( + monkeypatch, UNSTRACT_APIHUB_KEY=FAKE_KEY, UNSTRACT_APIHUB_BASE_URL="https://hub.test" + ) + plan = build_request( + get_endpoint("apihub.retrieve"), + cfg, + {"file_hash": "h", "ext_param": ["foo=bar", "baz=qux"]}, + ) + # `retrieve` has no ext_param; verify on `extract`, which does. + plan = build_request( + get_endpoint("apihub.extract"), + cfg, + {"vertical": "table", "sub_vertical": "extract_table", + "use_cached_file_hash": "h", "ext_param": ["foo=bar", "baz=qux"]}, + ) + assert plan.params["ext_foo"] == "bar" + assert plan.params["ext_baz"] == "qux" + + def test_freeform_rejects_malformed(self, monkeypatch): + cfg = _config( + monkeypatch, UNSTRACT_APIHUB_KEY=FAKE_KEY, UNSTRACT_APIHUB_BASE_URL="https://hub.test" + ) + with pytest.raises(CLIError) as exc: + build_request( + get_endpoint("apihub.extract"), cfg, + {"vertical": "table", "sub_vertical": "extract_table", + "use_cached_file_hash": "h", "ext_param": ["novalue"]}, + ) + assert exc.value.exit_code is ExitCode.USAGE + + def test_share_resource_maps_to_path_segment(self, monkeypatch): + """P3 - the friendly name is not the URL segment.""" + cfg = _config(monkeypatch, UNSTRACT_PLATFORM_KEY=FAKE_KEY, UNSTRACT_ORG_ID="o") + plan = build_request( + get_endpoint("docstudio.platform.share"), cfg, + {"resource": "api-deployment", "id": "abc"}, + ) + assert "/api/deployment/abc/share/" in plan.url + + def test_apihub_without_base_url_is_usage_error(self, monkeypatch): + """API Hub has no default base URL; inventing one would send documents + to a host we cannot vouch for (SPEC §11.1).""" + cfg = _config(monkeypatch, UNSTRACT_APIHUB_KEY=FAKE_KEY) + with pytest.raises(CLIError) as exc: + build_request(get_endpoint("apihub.status"), cfg, {"file_hash": "h"}) + assert exc.value.exit_code is ExitCode.USAGE + + +class TestJsonParams: + """BUG 1 - ParamType.JSON params must reach the wire as objects, not strings.""" + + def _cfg(self, monkeypatch): + return _config(monkeypatch, UNSTRACT_PLATFORM_KEY=FAKE_KEY, UNSTRACT_ORG_ID="o") + + def test_body_json_param_is_parsed_to_object(self, monkeypatch): + """A BODY+JSON param becomes a dict in json_body, not an escaped string.""" + plan = build_request( + get_endpoint("docstudio.platform.prompt-studio.sync-prompts"), + self._cfg(monkeypatch), + {"tool_id": "t", "data": '{"prompts": [{"prompt_key": "k"}]}'}, + ) + assert plan.json_body["data"] == {"prompts": [{"prompt_key": "k"}]} + assert isinstance(plan.json_body["data"], dict) + + def test_dry_run_renders_json_param_as_object(self, monkeypatch): + plan = build_request( + get_endpoint("docstudio.platform.prompt-studio.sync-prompts"), + self._cfg(monkeypatch), + {"tool_id": "t", "data": '{"a": 1}'}, + ) + assert plan.describe()["body"]["data"] == {"a": 1} + + def test_json_param_accepts_file_reference(self, monkeypatch, tmp_path): + payload = tmp_path / "prompts.json" + payload.write_text('{"prompts": ["x"]}') + plan = build_request( + get_endpoint("docstudio.platform.prompt-studio.sync-prompts"), + self._cfg(monkeypatch), + {"tool_id": "t", "data": f"@{payload}"}, + ) + assert plan.json_body["data"] == {"prompts": ["x"]} + + def test_invalid_json_is_usage_error(self, monkeypatch): + with pytest.raises(CLIError) as exc: + build_request( + get_endpoint("docstudio.platform.prompt-studio.sync-prompts"), + self._cfg(monkeypatch), + {"tool_id": "t", "data": "{not json"}, + ) + assert exc.value.exit_code is ExitCode.USAGE + + def test_missing_json_file_is_usage_error(self, monkeypatch): + with pytest.raises(CLIError) as exc: + build_request( + get_endpoint("docstudio.platform.prompt-studio.sync-prompts"), + self._cfg(monkeypatch), + {"tool_id": "t", "data": "@/no/such/file.json"}, + ) + assert exc.value.exit_code is ExitCode.USAGE + + def test_form_json_param_is_serialized_back_to_string(self, monkeypatch): + """FORM+JSON (deployment run --custom-data) must be a *string* in the + multipart field -- httpx cannot form-encode a dict, and the server + json.loads it. This guards against a double-encode regression.""" + cfg = _config( + monkeypatch, UNSTRACT_DEPLOYMENT_KEY=FAKE_KEY, UNSTRACT_ORG_ID="o" + ) + plan = build_request( + get_endpoint("docstudio.deployment.run"), + cfg, + {"api_name": "inv", "custom_data": '{"k": "v"}'}, + ) + assert plan.data["custom_data"] == '{"k": "v"}' + assert isinstance(plan.data["custom_data"], str) + + +class TestResponseFieldGuards: + """BUG 2 - mirror tool_id into the prompt-create body so it links.""" + + def test_prompt_create_mirrors_tool_id_into_body(self, monkeypatch): + plan = build_request( + get_endpoint("docstudio.platform.prompt-studio.prompt.create"), + _config(monkeypatch, UNSTRACT_PLATFORM_KEY=FAKE_KEY, UNSTRACT_ORG_ID="o"), + {"tool_id": "the-tool", "prompt_key": "k"}, + ) + # The path still carries it... + assert "/the-tool/" in plan.url + # ...and it is also sent in the body, defeating the orphaning defect. + assert plan.json_body["tool_id"] == "the-tool" + + def test_prompt_create_declares_required_response_field(self): + ep = get_endpoint("docstudio.platform.prompt-studio.prompt.create") + assert "tool_id" in ep.require_response_fields + + +class TestExecute: + @respx.mock + def test_success(self, monkeypatch): + respx.get(f"{WHISPER_BASE}/get-usage-info").mock( + return_value=httpx.Response(200, json={"subscription_plan": "free"}) + ) + cfg = _config(monkeypatch, LLMWHISPERER_API_KEY=FAKE_KEY) + plan = build_request(get_endpoint("whisper.usage"), cfg, {}) + assert execute(plan, max_retries=0).payload["subscription_plan"] == "free" + + @respx.mock + def test_concatenated_json_body_becomes_one_array(self, monkeypatch): + """CAPTURE2 DOC 6 - an endpoint that streams several JSON objects back to + back would break `| json`. The CLI recovers them into one array so the + contract 'exactly one JSON document' holds.""" + respx.get(f"{WHISPER_BASE}/get-usage-info").mock( + return_value=httpx.Response( + 200, headers={"content-type": "application/json"}, + content=b'{"a": 1}\n{"b": 2}', + ) + ) + cfg = _config(monkeypatch, LLMWHISPERER_API_KEY=FAKE_KEY) + plan = build_request(get_endpoint("whisper.usage"), cfg, {}) + payload = execute(plan, max_retries=0).payload + assert payload == [{"a": 1}, {"b": 2}] + + @respx.mock + def test_truly_malformed_json_falls_back_to_text(self, monkeypatch): + respx.get(f"{WHISPER_BASE}/get-usage-info").mock( + return_value=httpx.Response( + 200, headers={"content-type": "application/json"}, + content=b'{"a": 1} not json at all', + ) + ) + cfg = _config(monkeypatch, LLMWHISPERER_API_KEY=FAKE_KEY) + plan = build_request(get_endpoint("whisper.usage"), cfg, {}) + assert execute(plan, max_retries=0).payload == '{"a": 1} not json at all' + + @respx.mock + def test_retries_on_500_then_succeeds(self, monkeypatch): + route = respx.get(f"{WHISPER_BASE}/get-usage-info").mock( + side_effect=[httpx.Response(500), httpx.Response(200, json={"ok": True})] + ) + cfg = _config(monkeypatch, LLMWHISPERER_API_KEY=FAKE_KEY) + plan = build_request(get_endpoint("whisper.usage"), cfg, {}) + assert execute(plan, max_retries=2, sleep=lambda _: None).status == 200 + assert route.call_count == 2 + + @respx.mock + def test_does_not_retry_4xx(self, monkeypatch): + route = respx.get(f"{WHISPER_BASE}/get-usage-info").mock( + return_value=httpx.Response(403, json={"message": "Unauthorized"}) + ) + cfg = _config(monkeypatch, LLMWHISPERER_API_KEY=FAKE_KEY) + plan = build_request(get_endpoint("whisper.usage"), cfg, {}) + assert execute(plan, max_retries=3, sleep=lambda _: None).status == 403 + assert route.call_count == 1, "a 4xx must not be retried" + + @respx.mock + def test_already_consumed_maps_to_exit_9(self, monkeypatch): + """SPEC §5.6 - the one-shot footgun gets its own exit code.""" + respx.get(f"{WHISPER_BASE}/whisper-retrieve").mock( + return_value=httpx.Response(406, json={"message": "Whisper already delivered"}) + ) + cfg = _config(monkeypatch, LLMWHISPERER_API_KEY=FAKE_KEY) + endpoint = get_endpoint("whisper.retrieve") + plan = build_request(endpoint, cfg, {"whisper_hash": "h"}) + response = execute(plan, max_retries=0) + with pytest.raises(CLIError) as exc: + raise_for_status(response, endpoint) + assert exc.value.exit_code is ExitCode.ALREADY_CONSUMED + assert "already" in (exc.value.hint or "").lower() + + @respx.mock + def test_error_carries_hint_and_details(self, monkeypatch): + respx.get(f"{WHISPER_BASE}/whisper-status").mock( + return_value=httpx.Response( + 422, + json={"type": "validation_error", + "errors": [{"code": "invalid", "detail": "bad hash", "attr": "whisper_hash"}]}, + ) + ) + cfg = _config(monkeypatch, LLMWHISPERER_API_KEY=FAKE_KEY) + endpoint = get_endpoint("whisper.status") + plan = build_request(endpoint, cfg, {"whisper_hash": "x"}) + with pytest.raises(CLIError) as exc: + raise_for_status(execute(plan, max_retries=0), endpoint) + payload = exc.value.to_dict()["error"] + assert payload["message"] == "bad hash" + assert payload["retryable"] is False + assert payload["exit_code"] == 5 diff --git a/tests/test_model.py b/tests/test_model.py new file mode 100644 index 0000000..ab59eb8 --- /dev/null +++ b/tests/test_model.py @@ -0,0 +1,238 @@ +"""The schema expresses parameter patterns P1-P12. + +This is the load-bearing design work: if any pattern in the endpoint reference +cannot be expressed declaratively, generation degrades into per-command +special-casing and the Skill's diff stops being meaningful. Each test below pins +one pattern to a real record in the shipped surface. +""" + +from __future__ import annotations + +import pytest + +from unstract_cli.core.model import ( + ApiGroup, + AtLeastOneOf, + BodyKind, + Endpoint, + MutuallyExclusive, + Param, + ParamLocation, + ParamType, + derive_patch, + with_params, +) +from unstract_cli.endpoints import ALL_ENDPOINTS, get_endpoint + + +class TestP1MutuallyExclusive: + """P1 - exactly one of a set of flags.""" + + def test_rejects_both(self): + c = MutuallyExclusive(("file", "url")) + assert "mutually exclusive" in (c.check({"file": "a", "url": "b"}) or "") + + def test_rejects_neither_when_required(self): + assert "required" in (MutuallyExclusive(("file", "url")).check({}) or "") + + def test_accepts_exactly_one(self): + assert MutuallyExclusive(("file", "url")).check({"file": "a"}) is None + + def test_optional_variant_allows_neither(self): + assert MutuallyExclusive(("api", "pipeline"), required=False).check({}) is None + + def test_real_endpoint_uses_it(self): + endpoint = get_endpoint("whisper.extract") + assert any(isinstance(c, MutuallyExclusive) for c in endpoint.constraints) + assert endpoint.validate({"file": "a", "url": "b"}) + + +class TestP2AtLeastOneOf: + """P2 - guard against an unfiltered destructive operation.""" + + def test_requires_one(self): + assert AtLeastOneOf(("ids", "status")).check({}) is not None + assert AtLeastOneOf(("ids", "status")).check({"status": "ERROR"}) is None + + def test_file_history_clear_is_guarded(self): + # Without a filter this endpoint would delete every file history. + endpoint = get_endpoint("docstudio.platform.workflow.file-history.clear") + assert any(isinstance(c, AtLeastOneOf) for c in endpoint.constraints) + assert endpoint.validate({"workflow_id": "w"}) + + +class TestP3ChoiceMapping: + """P3 - friendly value maps to a different wire value.""" + + def test_mapping_translates(self): + param = Param("resource", choices={"api-deployment": "api/deployment"}) + assert param.to_wire("api-deployment") == "api/deployment" + + def test_plain_sequence_is_identity(self): + param = Param("mode", choices=["form", "table"]) + assert param.to_wire("form") == "form" + + def test_share_resource_mapping(self): + param = get_endpoint("docstudio.platform.share").param("resource") + assert param.choice_map()["api-deployment"] == "api/deployment" + + +class TestP4Repeatable: + def test_multiple_flag(self): + assert get_endpoint("docstudio.deployment.run").param("files").multiple + + +class TestP5Freeform: + """P5 - escape hatch for parameters newer than the CLI.""" + + def test_ext_param_declares_prefix(self): + param = get_endpoint("apihub.extract").param("ext_param") + assert param.freeform_prefix == "ext_" + assert param.multiple + + +class TestP6PathParamDefaults: + def test_org_id_defaults_from_profile(self): + param = get_endpoint("docstudio.deployment.run").param("org_id") + assert param.location is ParamLocation.PATH + assert param.default_sources[0] == "deployment.org_id" + + def test_org_id_falls_back_to_the_platform_block(self): + # The deployment/hitl config blocks start empty, but the organization is + # the same one the platform block already names (GOTCHAS #7). The block's + # own value must still win, so the fallback comes second. + for name in ("docstudio.deployment.run", "docstudio.hitl.approved.get"): + sources = get_endpoint(name).param("org_id").default_sources + assert sources[-1] == "platform.org_id" + assert len(sources) == 2 + + def test_api_name_has_no_default(self): + # One profile serves many deployments, so this cannot be defaulted. + param = get_endpoint("docstudio.deployment.run").param("api_name") + assert param.required and param.default_from is None + + +class TestP7Locations: + def test_all_locations_representable(self): + used = {p.location for e in ALL_ENDPOINTS for p in e.params} + assert {ParamLocation.QUERY, ParamLocation.BODY, ParamLocation.PATH, + ParamLocation.FORM} <= used + + def test_body_kinds_used(self): + used = {e.body for e in ALL_ENDPOINTS} + assert {BodyKind.NONE, BodyKind.JSON, BodyKind.MULTIPART, + BodyKind.BINARY_FILE} <= used + + +class TestP8DerivedPatch: + """P8 - PATCH derives from PUT so records cannot drift apart.""" + + def test_strips_required_but_keeps_path_params(self): + put = Endpoint( + name="update", group="g", method="PUT", path="/x/{id}/", + api=ApiGroup.PLATFORM, summary="s", + params=( + Param("id", location=ParamLocation.PATH, required=True), + Param("name", location=ParamLocation.BODY, required=True), + ), + ) + patch = derive_patch(put) + assert patch.method == "PATCH" + assert patch.param("id").required, "path params stay required" + assert not patch.param("name").required, "body fields become optional" + + def test_shipped_patches_mirror_their_put(self): + for group in ("docstudio.platform.prompt-studio", "docstudio.platform.workflow"): + put = get_endpoint(f"{group}.update") + patch = get_endpoint(f"{group}.patch") + assert {p.name for p in patch.params} >= {p.name for p in put.params}, ( + "a PATCH missing a PUT field means the records have drifted" + ) + + def test_with_params_appends(self): + base = Endpoint(name="a", group="g", method="PATCH", path="/x", + api=ApiGroup.PLATFORM, summary="s") + extended = with_params(base, Param("active", type=ParamType.BOOL)) + assert extended.param("active") is not None + assert base.param("active") is None, "the source record must be untouched" + + +class TestP9ConditionalApplicability: + def test_documented_not_enforced(self): + param = get_endpoint("whisper.extract").param("median_filter_size") + assert param.applies_when == "mode=low_cost" + # Deliberately no constraint: the server owns this rule, and enforcing it + # locally would guess wrong when the server's behaviour changes. + assert not get_endpoint("whisper.extract").validate({"file": "x"}) + + +class TestP10IdentifierTypes: + def test_group_ids_are_ints_not_uuids(self): + assert get_endpoint("docstudio.platform.group.patch").param("id").type is ParamType.INT + assert get_endpoint("docstudio.platform.workflow.get").param("id").type is ParamType.UUID + + +class TestP11LiteralPaths: + """P11 - upstream paths are inconsistent, and that inconsistency is load-bearing.""" + + @pytest.mark.parametrize( + "name,expected", + [ + ("docstudio.platform.prompt-studio.profile.create", + "/api/v1/unstract/{org_id}/prompt-studio/profilemanager/{tool_id}"), + ("docstudio.platform.prompt-studio.profile.get", + "/api/v1/unstract/{org_id}/prompt-studio/profile-manager/{profile_id}/"), + ("docstudio.platform.group.member.remove", + "/api/v1/unstract/{org_id}/groups/{id}/members/{user_id}"), + ("docstudio.platform.pipeline.postman-collection", + "/api/v1/unstract/{org_id}/pipeline/api/postman_collection/{id}/"), + ("docstudio.platform.api-deployment.postman-collection", + "/api/v1/unstract/{org_id}/api/postman_collection/{id}/"), + ], + ) + def test_exact_paths(self, name, expected): + assert get_endpoint(name).path == expected + + def test_missing_trailing_slash_is_declared(self): + """A path without a trailing slash must say so, so it reads as intent.""" + for endpoint in ALL_ENDPOINTS: + if endpoint.api is not ApiGroup.PLATFORM: + continue + if "{" in endpoint.path.split("/")[-1] and not endpoint.path.endswith("/"): + assert endpoint.no_trailing_slash, ( + f"{endpoint.dotted_name} lacks a trailing slash but does not " + "declare no_trailing_slash -- typo or intent?" + ) + + +class TestP12ReplaceSemantics: + def test_shared_users_marked_replace(self): + assert get_endpoint("docstudio.platform.share").param("shared_users").replace_semantics + + +class TestRegistryIntegrity: + def test_names_unique(self): + names = [e.dotted_name for e in ALL_ENDPOINTS] + assert len(names) == len(set(names)), "duplicate command paths" + + def test_every_endpoint_documented(self): + for endpoint in ALL_ENDPOINTS: + assert endpoint.summary, f"{endpoint.dotted_name} has no summary" + assert endpoint.doc_source, f"{endpoint.dotted_name} has no doc_source" + + def test_every_param_has_help(self): + for endpoint in ALL_ENDPOINTS: + for param in endpoint.params: + assert param.help, f"{endpoint.dotted_name}:{param.name} has no help" + + def test_poll_specs_reference_real_endpoints(self): + for endpoint in ALL_ENDPOINTS: + if not endpoint.poll: + continue + get_endpoint(endpoint.poll.status_endpoint) + if endpoint.poll.retrieve_endpoint: + get_endpoint(endpoint.poll.retrieve_endpoint) + + def test_methods_are_valid(self): + valid = {"GET", "POST", "PUT", "PATCH", "DELETE"} + assert {e.method for e in ALL_ENDPOINTS} <= valid diff --git a/tests/test_output.py b/tests/test_output.py new file mode 100644 index 0000000..d0010cf --- /dev/null +++ b/tests/test_output.py @@ -0,0 +1,105 @@ +"""Output rendering (SPEC.md §5.1). + +The table renderer is the human-facing format, so the property that matters most +is that it is **lossless**: a value shown in a table must be the whole value. +Truncation is the kind of defect you only discover after acting on a half-read +id or URL. +""" + +from __future__ import annotations + +import json + +import pytest + +from unstract_cli.core.output import ( + OutputFormat, + render, + render_table, +) + +LONG_NOTE = ( + "Credentials use env: indirection, so this file holds no secrets. " + "Set the referenced environment variables to authenticate." +) + + +class TestTableIsLossless: + def test_long_value_is_wrapped_not_truncated(self): + out = render_table({"note": LONG_NOTE}, max_width=80) + assert "..." not in out + # Every word survives, even though the cell spans several lines. + flattened = " ".join(out.split()) + for word in LONG_NOTE.split(): + assert word in flattened + + @pytest.mark.parametrize("width", [40, 60, 80, 100, 200]) + def test_lossless_at_any_width(self, width): + out = render_table({"note": LONG_NOTE}, max_width=width) + assert "..." not in out + assert "authenticate" in out, "the tail of the value must always survive" + + def test_unbroken_token_is_hard_wrapped(self): + """A long URL has no spaces to wrap on; it must break rather than overflow.""" + url = "https://ex" + "y" * 120 + ".com/end" + out = render_table({"url": url}, max_width=60) + assert "..." not in out + # Reassembling the value column's lines must reproduce the URL exactly. + value_lines = [ln.split(maxsplit=1)[-1] for ln in out.splitlines()[2:]] + assert "".join(part.strip() for part in value_lines) == url + assert all(len(line) <= 60 for line in out.splitlines()) + + +class TestTableLayout: + def test_respects_terminal_width(self): + out = render_table({"note": LONG_NOTE}, max_width=60) + assert all(len(line) <= 60 for line in out.splitlines()) + + def test_columns_stay_aligned_across_wrapped_rows(self): + rows = [ + {"id": "a1", "description": LONG_NOTE}, + {"id": "b2", "description": "short"}, + ] + lines = render_table(rows, max_width=70).splitlines() + # The id column is narrow, so every continuation line begins with the + # blank gutter that keeps the grid readable. + continuation = [ln for ln in lines[2:] if ln and not ln[0].isalnum()] + assert continuation, "expected wrapped continuation lines" + assert all(ln.startswith(" ") for ln in continuation) + + def test_narrow_column_not_squeezed_for_a_wide_neighbour(self): + """Only the widest column gives up space, so ids stay readable.""" + rows = [{"id": "9c1e-4f2a", "description": LONG_NOTE}] + out = render_table(rows, max_width=60) + assert "9c1e-4f2a" in out, "the narrow id column must not wrap" + + def test_key_value_shape_for_single_object(self): + out = render_table({"a": 1, "b": 2}) + assert out.splitlines()[0].split() == ["key", "value"] + + def test_list_of_objects_uses_keys_as_columns(self): + out = render_table([{"id": "x", "name": "y"}]) + assert out.splitlines()[0].split() == ["id", "name"] + + def test_empty_input(self): + assert render_table([]) == "(no results)" + + def test_booleans_render_lowercase(self): + assert "false" in render_table({"replaced_existing": False}) + + +class TestOtherFormats: + def test_json_is_untouched_by_table_wrapping(self): + """Only the human format wraps; machine formats stay verbatim.""" + payload = {"note": LONG_NOTE} + assert json.loads(render(payload, OutputFormat.JSON)) == payload + + def test_raw_returns_the_payload_alone(self): + out = render({"result_text": "EXTRACTED"}, OutputFormat.RAW, raw_field="result_text") + assert out == "EXTRACTED" + + def test_yaml_round_trips(self): + import yaml + + payload = {"note": LONG_NOTE, "n": 1} + assert yaml.safe_load(render(payload, OutputFormat.YAML)) == payload diff --git a/tests/test_poll.py b/tests/test_poll.py new file mode 100644 index 0000000..d52039d --- /dev/null +++ b/tests/test_poll.py @@ -0,0 +1,343 @@ +"""`--wait` polling, including the 422 defect (R4). + +The single most important test in this suite is +`TestDeployment422Defect::test_422_and_200_resolve_identically`. + +The Unstract deployment API currently returns **HTTP 422** for the in-progress +states `PENDING` and `EXECUTING` -- a documented server-side defect that is +scheduled to be fixed. If the CLI branched on the HTTP status code it would +either break today (reading in-progress as failure) or break the day the defect +is fixed. Branching on the response body's `status` field makes both behave +identically, and that is what these tests pin down. +""" + +from __future__ import annotations + +import httpx +import pytest +import respx + +from unstract_cli.config.loader import ConfigFile, ResolvedConfig +from unstract_cli.core.errors import CLIError, ExitCode +from unstract_cli.core.poll import extract_handle, extract_status, wait_for_completion +from unstract_cli.endpoints import get_endpoint + +from .conftest import FAKE_KEY, PLATFORM_BASE, WHISPER_BASE + + +def _config(monkeypatch, **env) -> ResolvedConfig: + for key, value in env.items(): + monkeypatch.setenv(key, value) + return ResolvedConfig(file=ConfigFile(exists=False)) + + +class TestFieldExtraction: + def test_reads_bare_field(self): + assert extract_status({"status": "processed"}) == "processed" + + def test_reads_through_message_envelope(self): + """The deployment API nests its payload under `message`.""" + payload = {"message": {"execution_status": "COMPLETED"}} + assert extract_status(payload, "execution_status") == "COMPLETED" + + def test_reads_handle_through_envelope(self): + payload = {"message": {"execution_id": "e-123"}} + assert extract_handle(payload, "execution_id") == "e-123" + + +class TestWhisperWait: + @respx.mock + def test_polls_then_retrieves(self, monkeypatch): + respx.get(f"{WHISPER_BASE}/whisper-status").mock( + side_effect=[ + httpx.Response(200, json={"status": "processing"}), + httpx.Response(200, json={"status": "processed"}), + ] + ) + respx.get(f"{WHISPER_BASE}/whisper-retrieve").mock( + return_value=httpx.Response(200, json={"result_text": "EXTRACTED"}) + ) + cfg = _config(monkeypatch, LLMWHISPERER_API_KEY=FAKE_KEY) + result = wait_for_completion( + endpoint=get_endpoint("whisper.extract"), + initial={"whisper_hash": "h1", "status": "processing"}, + config=cfg, + sleep=lambda _: None, + ) + assert result["result_text"] == "EXTRACTED" + + @respx.mock + def test_error_status_raises(self, monkeypatch): + respx.get(f"{WHISPER_BASE}/whisper-status").mock( + return_value=httpx.Response(200, json={"status": "error", "message": "bad scan"}) + ) + cfg = _config(monkeypatch, LLMWHISPERER_API_KEY=FAKE_KEY) + with pytest.raises(CLIError) as exc: + wait_for_completion( + endpoint=get_endpoint("whisper.extract"), + initial={"whisper_hash": "h1"}, + config=cfg, + sleep=lambda _: None, + ) + assert exc.value.exit_code is ExitCode.VALIDATION + + @respx.mock + def test_timeout_exits_7_and_returns_handle(self, monkeypatch): + """On timeout an agent must be able to resume, not reprocess.""" + respx.get(f"{WHISPER_BASE}/whisper-status").mock( + return_value=httpx.Response(200, json={"status": "processing"}) + ) + cfg = _config(monkeypatch, LLMWHISPERER_API_KEY=FAKE_KEY) + clock = iter([0.0, 0.0, 100.0, 200.0, 400.0, 500.0, 600.0]) + + with pytest.raises(CLIError) as exc: + wait_for_completion( + endpoint=get_endpoint("whisper.extract"), + initial={"whisper_hash": "h-resume-me"}, + config=cfg, + timeout=300.0, + sleep=lambda _: None, + now=lambda: next(clock), + ) + assert exc.value.exit_code is ExitCode.TIMEOUT + assert exc.value.to_dict()["error"]["whisper_hash"] == "h-resume-me" + + +class TestDeployment422Defect: + """R4 - the highest-value test in the suite. + + `EXECUTING`/`PENDING` currently arrive with HTTP 422. When that defect is + fixed they will arrive with HTTP 200. Both must behave identically. + """ + + @staticmethod + def _run(monkeypatch, in_progress_status_code: int): + # The real status GET returns a TOP-LEVEL `status`, and its `message` holds + # the result -- NOT the nested `{message: {execution_status}}` shape the run + # POST uses. Testing the true shape is what protects CAPTURE2 BUG 2. + url = f"{PLATFORM_BASE}/deployment/api/org_test/my-api/" + respx.get(url).mock( + side_effect=[ + httpx.Response( + in_progress_status_code, + json={"status": "EXECUTING", "message": None}, + ), + httpx.Response( + 200, + json={"status": "COMPLETED", + "message": [{"file": "a.pdf", "status": "Success"}]}, + ), + ] + ) + cfg = _config( + monkeypatch, UNSTRACT_DEPLOYMENT_KEY=FAKE_KEY, UNSTRACT_ORG_ID="org_test" + ) + return wait_for_completion( + endpoint=get_endpoint("docstudio.deployment.run"), + initial={"message": {"execution_id": "e-1", "execution_status": "PENDING"}}, + config=cfg, + values={"api_name": "my-api"}, + sleep=lambda _: None, + ) + + @respx.mock + def test_current_behaviour_422_in_progress(self, monkeypatch): + """Today: 422 + EXECUTING means 'still running', not 'failed'.""" + result = self._run(monkeypatch, 422) + assert extract_status(result, ("status", "execution_status")) == "COMPLETED" + + @respx.mock + def test_future_behaviour_200_in_progress(self, monkeypatch): + """After the fix: 200 + EXECUTING must behave exactly the same.""" + result = self._run(monkeypatch, 200) + assert extract_status(result, ("status", "execution_status")) == "COMPLETED" + + @respx.mock + def test_both_paths_agree(self, monkeypatch): + a = self._run(monkeypatch, 422) + respx.reset() + b = self._run(monkeypatch, 200) + assert a == b, "the 422 defect must not change the observable outcome" + + @respx.mock + def test_genuine_422_without_status_still_fails(self, monkeypatch): + """A 422 carrying no recognisable state is a real validation failure. + + Reading the body must not become a way to swallow genuine errors. + """ + url = f"{PLATFORM_BASE}/deployment/api/org_test/my-api/" + respx.get(url).mock( + return_value=httpx.Response( + 422, json={"type": "client_error", + "errors": [{"detail": "Pipeline is inactive"}]} + ) + ) + cfg = _config( + monkeypatch, UNSTRACT_DEPLOYMENT_KEY=FAKE_KEY, UNSTRACT_ORG_ID="org_test" + ) + with pytest.raises(CLIError) as exc: + wait_for_completion( + endpoint=get_endpoint("docstudio.deployment.run"), + initial={"message": {"execution_id": "e-1"}}, + config=cfg, + values={"api_name": "my-api"}, + sleep=lambda _: None, + ) + assert exc.value.exit_code is ExitCode.VALIDATION + assert "inactive" in exc.value.message + + @respx.mock + def test_error_status_is_terminal_failure(self, monkeypatch): + url = f"{PLATFORM_BASE}/deployment/api/org_test/my-api/" + respx.get(url).mock( + return_value=httpx.Response( + 422, json={"status": "ERROR", "message": "tool failed"} + ) + ) + cfg = _config( + monkeypatch, UNSTRACT_DEPLOYMENT_KEY=FAKE_KEY, UNSTRACT_ORG_ID="org_test" + ) + with pytest.raises(CLIError) as exc: + wait_for_completion( + endpoint=get_endpoint("docstudio.deployment.run"), + initial={"message": {"execution_id": "e-1"}}, + config=cfg, + values={"api_name": "my-api"}, + sleep=lambda _: None, + ) + assert exc.value.exit_code is ExitCode.VALIDATION + + @respx.mock + def test_first_poll_terminal_returns_result_not_406(self, monkeypatch): + """CAPTURE2 BUG 2 - fast completion: the very first status poll is already + COMPLETED. The status endpoint is the one-shot store, so that read consumes + the result. The loop must recognise the top-level `status` on that read and + return its body (the result) -- not fail to recognise it, discard it, and + 406 on a second poll. A single mocked read enforces 'no second read'.""" + url = f"{PLATFORM_BASE}/deployment/api/org_test/my-api/" + route = respx.get(url).mock( + return_value=httpx.Response( + 200, + json={"status": "COMPLETED", + "message": [{"file": "bill.pdf", "result": {"invoice_no": "X-1"}}]}, + ) + ) + cfg = _config( + monkeypatch, UNSTRACT_DEPLOYMENT_KEY=FAKE_KEY, UNSTRACT_ORG_ID="org_test" + ) + result = wait_for_completion( + endpoint=get_endpoint("docstudio.deployment.run"), + initial={"message": {"execution_id": "e-1", "execution_status": "PENDING"}}, + config=cfg, + values={"api_name": "my-api"}, + sleep=lambda _: None, + ) + assert route.call_count == 1, "must not re-read a one-shot result" + assert result["message"][0]["result"]["invoice_no"] == "X-1" + + +class TestPromptStudioWait: + """IMPROVEMENT 3 - fetch-response is fire-and-forget; --wait must poll + task-status to completion and then read the result from the Output Manager, + which is keyed by the *original request's* tool_id, not the poll handle.""" + + _PS_BASE = f"{PLATFORM_BASE}/api/v1/unstract/org_test/prompt-studio" + + @respx.mock + def test_polls_task_status_then_reads_output(self, monkeypatch): + respx.get(f"{self._PS_BASE}/the-tool/task-status/task-9").mock( + side_effect=[ + httpx.Response(200, json={"task_id": "task-9", "status": "processing"}), + httpx.Response(200, json={"task_id": "task-9", "status": "completed"}), + ] + ) + output = respx.get(f"{self._PS_BASE}/prompt-output/").mock( + return_value=httpx.Response( + 200, json=[{"prompt_id": "p1", "output": "42", "modified_at": "t"}] + ) + ) + cfg = _config( + monkeypatch, UNSTRACT_PLATFORM_KEY=FAKE_KEY, UNSTRACT_ORG_ID="org_test" + ) + result = wait_for_completion( + endpoint=get_endpoint("docstudio.platform.prompt-studio.fetch-response"), + initial={"task_id": "task-9", "run_id": "r1", "status": "accepted"}, + config=cfg, + values={"tool_id": "the-tool", "document_id": "d1", "id": "p1"}, + sleep=lambda _: None, + ) + assert result[0]["output"] == "42" + # The retrieve is narrowed to the exact prompt + document this call ran + # (id->prompt_id, document_id->document_manager), keyed by the original + # tool_id, and does NOT leak the task_id handle into the output-list call. + url = str(output.calls.last.request.url) + assert "tool_id=the-tool" in url + assert "prompt_id=p1" in url + assert "document_manager=d1" in url + assert "task_id" not in url + + @respx.mock + def test_failed_task_is_terminal_failure(self, monkeypatch): + respx.get(f"{self._PS_BASE}/the-tool/task-status/task-9").mock( + return_value=httpx.Response( + 500, json={"task_id": "task-9", "status": "failed", "error": "boom"} + ) + ) + cfg = _config( + monkeypatch, UNSTRACT_PLATFORM_KEY=FAKE_KEY, UNSTRACT_ORG_ID="org_test" + ) + with pytest.raises(CLIError) as exc: + wait_for_completion( + endpoint=get_endpoint("docstudio.platform.prompt-studio.fetch-response"), + initial={"task_id": "task-9", "status": "accepted"}, + config=cfg, + values={"tool_id": "the-tool"}, + sleep=lambda _: None, + ) + assert exc.value.exit_code is ExitCode.VALIDATION + + @respx.mock + def test_single_pass_retrieve_asks_for_single_pass_rows(self, monkeypatch): + respx.get(f"{self._PS_BASE}/the-tool/task-status/task-9").mock( + return_value=httpx.Response(200, json={"status": "completed"}) + ) + output = respx.get(f"{self._PS_BASE}/prompt-output/").mock( + return_value=httpx.Response(200, json=[{"output": "sp"}]) + ) + cfg = _config( + monkeypatch, UNSTRACT_PLATFORM_KEY=FAKE_KEY, UNSTRACT_ORG_ID="org_test" + ) + wait_for_completion( + endpoint=get_endpoint("docstudio.platform.prompt-studio.single-pass"), + initial={"task_id": "task-9", "status": "accepted"}, + config=cfg, + values={"tool_id": "the-tool"}, + sleep=lambda _: None, + ) + assert "is_single_pass_extract=true" in str(output.calls.last.request.url) + + +class TestApiHubWait: + @respx.mock + def test_traverses_three_stage_progression(self, monkeypatch): + base = "https://hub.test" + respx.get(f"{base}/api/v1/status").mock( + side_effect=[ + httpx.Response(200, json={"status": "QUEUED_FOR_WHISPER"}), + httpx.Response(200, json={"status": "QUEUED_FOR_EXTRACTION"}), + httpx.Response(200, json={"status": "COMPLETED"}), + ] + ) + respx.get(f"{base}/api/v1/retrieve").mock( + return_value=httpx.Response(200, json={"tables": [{"rows": 3}]}) + ) + cfg = _config( + monkeypatch, UNSTRACT_APIHUB_KEY=FAKE_KEY, UNSTRACT_APIHUB_BASE_URL=base + ) + result = wait_for_completion( + endpoint=get_endpoint("apihub.extract"), + initial={"file_hash": "fh-1", "status": "QUEUED_FOR_WHISPER"}, + config=cfg, + sleep=lambda _: None, + ) + assert result["tables"][0]["rows"] == 3 diff --git a/tests/test_skill_docdiff.py b/tests/test_skill_docdiff.py new file mode 100644 index 0000000..34572fd --- /dev/null +++ b/tests/test_skill_docdiff.py @@ -0,0 +1,215 @@ +"""Phase 6 acceptance: the Skill's docs-vs-records diff. + +The strongest signal available is the **zero-drift regression**: the records were +authored from exactly these documentation files, so running the diff against the +current docs must report no actionable drift. If it does, either a record is +wrong or the parser is. +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from unstract_cli.endpoints import endpoints_for, get_endpoint +from unstract_cli.skill.docdiff import ( + DocEndpoint, + DocParam, + diff, + parse_docs, + parse_markdown_params, + report, +) + +#: Docs repos are siblings of this one; skip rather than fail when absent, so the +#: suite still runs in a checkout that has only unstract-cli. +_REPO_ROOT = Path(__file__).resolve().parents[2] +_LLMW_DOCS = _REPO_ROOT / "llmwhisperer-docs/docs/llm_whisperer/apis" +_DEPLOY_DOCS = _REPO_ROOT / "unstract-docs/docs/unstract_platform/api_deployment" + +_PLATFORM_DOCS = ( + _REPO_ROOT / "unstract-docs/docs/unstract_platform/api_documentation/versions" +) + +requires_docs = pytest.mark.skipif( + not _LLMW_DOCS.exists(), reason="documentation repos not checked out alongside" +) +requires_platform_docs = pytest.mark.skipif( + not _PLATFORM_DOCS.exists(), reason="unstract-docs not checked out alongside" +) + + +class TestMarkdownParsing: + def test_extracts_request_parameters(self): + text = """ +## Parameters + +| Parameter | Type | Default | Required | Description | +| --- | --- | --- | --- | --- | +| mode | string | `form` | No | The processing mode | +| whisper_hash | string | | Yes | The whisper hash | +""" + params = {p.name: p for p in parse_markdown_params(text)} + assert params["mode"].default == "form" + assert params["whisper_hash"].required + assert not params["mode"].required + + def test_ignores_response_tables(self): + """Response fields are documented in the same format as parameters. + + Without section scoping, every response field reads as a missing flag. + """ + text = """ +## Parameters + +| Parameter | Type | Required | Description | +| --- | --- | --- | --- | +| whisper_hash | string | Yes | The hash | + +## Response + +| Parameter | Type | Description | +| --- | --- | --- | +| result_text | string | The extracted text | +| confidence_metadata | array | Confidence scores | +""" + names = {p.name for p in parse_markdown_params(text)} + assert names == {"whisper_hash"} + + def test_ignores_status_code_tables(self): + text = """ +## Parameters + +| Parameter | Type | Required | Description | +| --- | --- | --- | --- | +| execution_id | string | Yes | Execution id | + +### Possible Execution status + +| Status | Description | +| --- | --- | +| PENDING | Queued | +| COMPLETED | Done | +""" + assert {p.name for p in parse_markdown_params(text)} == {"execution_id"} + + +class TestZeroDriftRegression: + """Exit criterion 1: current docs must produce no actionable drift.""" + + @requires_docs + def test_llmwhisperer_has_no_drift(self): + findings = [ + f + for f in diff(parse_docs(_LLMW_DOCS), endpoints_for("llmwhisperer")) + if f.severity == "action" + ] + assert not findings, "\n".join(f"{f.kind}: {f.message}" for f in findings) + + @requires_docs + def test_deployment_has_no_drift(self): + findings = [ + f + for f in diff(parse_docs(_DEPLOY_DOCS), endpoints_for("deployment")) + if f.severity == "action" + ] + assert not findings, "\n".join(f"{f.kind}: {f.message}" for f in findings) + + @requires_platform_docs + def test_platform_mdx_parser_extracts_the_surface(self): + """Assert positively before asserting absence. + + `missing_in_docs` is only informational, so a parser that silently + extracted nothing would make a "no action findings" assertion pass while + being completely broken. Pin the volume and a known parameter set first. + """ + docs = parse_docs(_PLATFORM_DOCS) + assert len(docs) > 90, f"MDX parser extracted only {len(docs)} endpoints" + + create = next( + d for d in docs if d.method == "POST" and d.path.endswith("/workflow/") + ) + names = {p.name for p in create.params} + assert {"workflow_name", "description", "deployment_type"} <= names + + # responseBody fields must not be mistaken for request parameters. + assert not names & {"id", "created_at", "created_by_email", "status"} + + @requires_platform_docs + def test_platform_has_no_drift(self): + findings = [ + f + for f in diff(parse_docs(_PLATFORM_DOCS), endpoints_for("platform")) + if f.severity == "action" + ] + assert not findings, "\n".join(f"{f.kind}: {f.message}" for f in findings) + + +class TestSyntheticDrift: + """Exit criterion 2: a genuinely new parameter must be detected and cited.""" + + def test_detects_new_parameter(self): + doc = DocEndpoint( + method="GET", + path="/whisper-status", + params=[DocParam(name="whisper_hash", required=True), + DocParam(name="include_progress", type="boolean")], + source="whisper_status.md", + ) + findings = diff([doc], endpoints_for("llmwhisperer")) + drift = [f for f in findings if f.kind == "param_drift"] + assert len(drift) == 1 + assert "include_progress" in drift[0].message + assert drift[0].citation == "whisper_status.md" + assert "Param(" in drift[0].suggestion + + def test_detects_new_endpoint(self): + doc = DocEndpoint(method="POST", path="/whisper-cancel", source="new.md") + findings = diff([doc], endpoints_for("llmwhisperer")) + assert any(f.kind == "missing_in_cli" for f in findings) + + +class TestSafetyRules: + """Exit criterion 3, plus the rules that prevent destructive 'fixes'.""" + + def test_missing_from_docs_is_informational_only(self): + """Docs lag implementation; absence is never grounds for deletion.""" + findings = diff([], endpoints_for("llmwhisperer")) + removals = [f for f in findings if f.kind == "missing_in_docs"] + assert removals, "expected the endpoints to be reported as undocumented" + assert all(f.severity == "info" for f in removals) + assert all("NEVER delete" in f.suggestion for f in removals) + + def test_doc_conflict_endpoints_are_exempt(self): + """`/whisper-detail` diverges from the docs index deliberately. + + It must not be reported as removable, or a future run would 'correct' a + decision that was verified against the official client. + """ + assert get_endpoint("whisper.detail").doc_conflict + reported = { + f.command for f in diff([], endpoints_for("llmwhisperer")) if f.kind == "missing_in_docs" + } + assert "unstract whisper detail" not in reported + + def test_draft_pages_are_excluded(self, tmp_path): + """`draft: true` marks an unstable contract; those endpoints stay out.""" + page = tmp_path / "chat.md" + page.write_text( + "---\nid: chat\ndraft: true\n---\n\n" + "| Endpoint | `/md/chat` |\n| Method | `POST` |\n\n" + "## Parameters\n\n| Parameter | Type | Required | Description |\n" + "| --- | --- | --- | --- |\n| query | string | Yes | The question |\n" + ) + assert parse_docs(tmp_path) == [] + + +class TestReport: + def test_report_is_valid_json(self): + import json + + doc = DocEndpoint(method="POST", path="/whisper-cancel", source="new.md") + payload = json.loads(report(diff([doc], endpoints_for("llmwhisperer")))) + assert payload["total"] >= 1 + assert "missing_in_cli" in payload["summary"] diff --git a/uv.lock b/uv.lock new file mode 100644 index 0000000..2dadcf9 --- /dev/null +++ b/uv.lock @@ -0,0 +1,543 @@ +version = 1 +revision = 3 +requires-python = ">=3.12" +resolution-markers = [ + "python_full_version >= '3.15'", + "python_full_version < '3.15'", +] + +[[package]] +name = "anyio" +version = "4.14.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "idna" }, + { name = "typing-extensions", marker = "python_full_version < '3.13'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/61/cc/a381afa6efea9f496eff839d4a6a1aed3bfafc7b3ab4b0d1b243a12573dd/anyio-4.14.2.tar.gz", hash = "sha256:cfa139f3ed1a23ee8f88a145ddb5ac7605b8bbfd8592baacd7ce3d8bb4313c7f", size = 260176, upload-time = "2026-07-12T20:29:07.082Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/da/35/f2287558c17e29fafc8ef3daf819bb9834061cfa43bff8014f7df7f63bdc/anyio-4.14.2-py3-none-any.whl", hash = "sha256:9f505dda5ac9f0c8309b5e8bd445a8c2bf7246f3ce950121e45ea15bc41d1494", size = 125813, upload-time = "2026-07-12T20:29:05.763Z" }, +] + +[[package]] +name = "ast-serialize" +version = "0.6.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/58/ad/0d70a3a2d6e01968d985415259e8ec7ad3f777903f9b1c1f3c8c44642c60/ast_serialize-0.6.0.tar.gz", hash = "sha256:aadd3ffcf4858c9726bf3515f7b199c7eadbe504f96028e4a87172c0da65a8fe", size = 61489, upload-time = "2026-06-30T20:02:55.555Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/3f/12/3e5f575f156555547c250a8b0d1347517a3a20fc7f4492e9703a69d4f45e/ast_serialize-0.6.0-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:a7520b672827885bafeae7501f684d14d47d17e5f45256f9df547686cca52264", size = 1177640, upload-time = "2026-06-30T20:02:06.708Z" }, + { url = "https://files.pythonhosted.org/packages/a2/a4/921a9e27951627983b0f368859ea00f8330a551dc0bf4c2fdcb11855a98b/ast_serialize-0.6.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:a14191beec7e0c078d2fc1f6edc0aee88bcd4db9f18e1bc9f8052b559c22dddc", size = 1168111, upload-time = "2026-06-30T20:02:08.366Z" }, + { url = "https://files.pythonhosted.org/packages/00/69/950cf404de7b8782cf95e5c1237e25e2aa46177b287f39f9eeddf481fd6f/ast_serialize-0.6.0-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:32ef62ec34cf6be20ad77d4799556638fbdf187f3ae10698dfb20ef9f2c89516", size = 1227656, upload-time = "2026-06-30T20:02:09.843Z" }, + { url = "https://files.pythonhosted.org/packages/4c/a8/46f8f6a6479d9d2273980957bb091a506c55f5b95d3c029ee58518a78407/ast_serialize-0.6.0-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:13b7769970a39983b0adf2f38917b1cd3b8946f76df045756c3d741bc689f089", size = 1227706, upload-time = "2026-06-30T20:02:11.367Z" }, + { url = "https://files.pythonhosted.org/packages/b7/b9/9ac415bda0a40e49eab8fea3b2741c19c98bb84d57d62c4cfc6230eb67be/ast_serialize-0.6.0-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:6f7a408601bb3edaefb3bc67a4c01f5235e3253653b6a5729a2ee2382b35341c", size = 1431705, upload-time = "2026-06-30T20:02:12.737Z" }, + { url = "https://files.pythonhosted.org/packages/e5/06/8807115d441444879f7561b5eede5ac18fc80392f11826d61ccf31f503b1/ast_serialize-0.6.0-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:8670bfa51208a2c0c8d138928e40e998fab158f9200d53bb80c088b5b8eda7b8", size = 1249533, upload-time = "2026-06-30T20:02:14.571Z" }, + { url = "https://files.pythonhosted.org/packages/3e/c0/c2ba82ef9618650357d9421a1fdb27ffec862a7f57e8e2de82a3ccd11e12/ast_serialize-0.6.0-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a4826809eb8597a8cd59fd924b6d7c285b8969a1e0007e2cb652cab62376270f", size = 1252619, upload-time = "2026-06-30T20:02:16.219Z" }, + { url = "https://files.pythonhosted.org/packages/0f/a7/fa31d52dd4102cede29fb9634e98d214129b2783b4f95528c6dc6a8f6587/ast_serialize-0.6.0-cp314-cp314t-manylinux_2_31_riscv64.whl", hash = "sha256:577a6c189068686869f5f1ddc38363f3ae1808a4753b577266f9202071a7bb66", size = 1242983, upload-time = "2026-06-30T20:02:17.813Z" }, + { url = "https://files.pythonhosted.org/packages/b1/20/ddf742b5ad3c4bafd3466f2265037cfd99bc1b9a5ee46a5d58c90d523242/ast_serialize-0.6.0-cp314-cp314t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:085de7f62dc9cc247eb01e965a362707d1d90b1d89a82c5bf78301a60a3c417b", size = 1296148, upload-time = "2026-06-30T20:02:19.146Z" }, + { url = "https://files.pythonhosted.org/packages/24/cb/9f6f217cce8b3b632c5568b478d195a35e79dce4dbe309438cb89ba6ea4f/ast_serialize-0.6.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:9f8a8b78b13173de6a9ec22111d9be674874cd5bdccda04f14ae5ebc2bef403a", size = 1403826, upload-time = "2026-06-30T20:02:20.696Z" }, + { url = "https://files.pythonhosted.org/packages/2d/f8/9d16d4f0107a183924425cc0e7618d8bf76f96b45afa9ff19f924ed1ad57/ast_serialize-0.6.0-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:f2ff3baffc3a29c1f15bc9098aa0c09763410262d5e6cef42116f7356c184554", size = 1502943, upload-time = "2026-06-30T20:02:22.034Z" }, + { url = "https://files.pythonhosted.org/packages/80/dd/bbc1c38756350dddf7e24acae1c9482ef42051c267417e019aecc1ed4075/ast_serialize-0.6.0-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:0067b25fce104eaae5b88383de9ab803faeb671831e14ca698b771b356e2600f", size = 1497632, upload-time = "2026-06-30T20:02:23.517Z" }, + { url = "https://files.pythonhosted.org/packages/42/7e/9daffefcf5b97e6bb4c3e0b3c024c1aee9722f23d3cf7cd2ff80d6fb4a40/ast_serialize-0.6.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:c617417f9cbb0cb144f6283c3cbe0d2e0f01beaf9f608f662b21191058a626ec", size = 1448858, upload-time = "2026-06-30T20:02:24.889Z" }, + { url = "https://files.pythonhosted.org/packages/e5/1f/f9baaab81a677ea0af7d2458cac2f94ebcc85958f8a3c15ba9d9e5dab653/ast_serialize-0.6.0-cp314-cp314t-win32.whl", hash = "sha256:5337cb256dcea3df9288205213d1601581536526b8f4da44b6974f1180f3252a", size = 1052600, upload-time = "2026-06-30T20:02:26.263Z" }, + { url = "https://files.pythonhosted.org/packages/9e/1f/41b535866519512d8cf6669cb2cff7823b7672bb6279c0333b4ff89d7d9f/ast_serialize-0.6.0-cp314-cp314t-win_amd64.whl", hash = "sha256:2d947e45cafc4b09bd7528917fa84c517654a43de173c79785574b7b3068ac24", size = 1095570, upload-time = "2026-06-30T20:02:27.639Z" }, + { url = "https://files.pythonhosted.org/packages/50/64/e472fe3e3a2d33d874b987e8518aedf24562919e3b6161a4fa1797e89c0f/ast_serialize-0.6.0-cp314-cp314t-win_arm64.whl", hash = "sha256:6e15ec740436e1a0d62de848641abe5f3a2f89a7f94907d534795ac91bbacf14", size = 1067267, upload-time = "2026-06-30T20:02:28.949Z" }, + { url = "https://files.pythonhosted.org/packages/52/19/ac8348ae8711c9b5ae834634f635780cab62a0f5e6f988882e048b89c2ae/ast_serialize-0.6.0-cp39-abi3-macosx_10_12_x86_64.whl", hash = "sha256:093cb8bb91b720d8523580498d031791bb1bbaa048599c3d21085d380e11a596", size = 1185367, upload-time = "2026-06-30T20:02:30.427Z" }, + { url = "https://files.pythonhosted.org/packages/c1/f6/ec7ec652c51db77c2f61d8573338e13e4704303265ccc658cb4031d9f354/ast_serialize-0.6.0-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:e61580a69faf47e3689795367ed211f2a10fd741478cc0f36a0f128793360aad", size = 1178657, upload-time = "2026-06-30T20:02:31.964Z" }, + { url = "https://files.pythonhosted.org/packages/6f/02/613a7534a41d0122f37d1e0c64aa8ac78bfb831f8c92f6db057a311abb3c/ast_serialize-0.6.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:305802f2ce2a7c4e87835078ea85c58b586ddda8095b92fe2ead9364ae19c80a", size = 1238620, upload-time = "2026-06-30T20:02:33.664Z" }, + { url = "https://files.pythonhosted.org/packages/4d/21/087957bba486242afc52f49b2d9e21c9dad00289356cf9efe67084015a9d/ast_serialize-0.6.0-cp39-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:c7b8b8f0c42f752ea00b2b7d7c090b3f80d9c1c5c75cadf16423790a0cc74081", size = 1236075, upload-time = "2026-06-30T20:02:34.936Z" }, + { url = "https://files.pythonhosted.org/packages/82/04/78128bbb170071c2c72a210a181f1c00e11cc1cec60a8beef747b07f9201/ast_serialize-0.6.0-cp39-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:cd5b91b9e6f2356ace3a556963b0cd783b395fbbb0bb17b4defc283415466e77", size = 1441348, upload-time = "2026-06-30T20:02:36.245Z" }, + { url = "https://files.pythonhosted.org/packages/64/64/62fb99d6faf199b4c3e5b08a07136e9a0d7664bb249c6de3670e5b63e9b6/ast_serialize-0.6.0-cp39-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:4d6ef91590258ada18909b9caea344dac4de2013906b035473cd674a43f4b790", size = 1258580, upload-time = "2026-06-30T20:02:37.53Z" }, + { url = "https://files.pythonhosted.org/packages/ca/87/b4d6c38e0ccd5e85dc54cecdf933a152c60b28fe5d993a6d8a72fa6d5896/ast_serialize-0.6.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:dcbed41e9386059fc0261d602445ede0976c2ecec2939688bcbcb9ed0b6f28b7", size = 1261693, upload-time = "2026-06-30T20:02:39.123Z" }, + { url = "https://files.pythonhosted.org/packages/0e/4b/3676ca2191f39bafb75f93f99b2f429ec464586158fece2165f3572805dc/ast_serialize-0.6.0-cp39-abi3-manylinux_2_31_riscv64.whl", hash = "sha256:cdc4e6f930b9090c2f92c9036ad12ffb8e6e44d4a5ba06f1458a05d60f203f7b", size = 1252517, upload-time = "2026-06-30T20:02:40.511Z" }, + { url = "https://files.pythonhosted.org/packages/f3/58/494ef8c4b4acb2f4a265ac934caf45f792a08fe27d6b853de35ad991941a/ast_serialize-0.6.0-cp39-abi3-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:897ac47b5637be41c0c07061c8a912fafa967ef1dc73fa115e4bfa70882a093b", size = 1304843, upload-time = "2026-06-30T20:02:41.961Z" }, + { url = "https://files.pythonhosted.org/packages/b1/f2/13736d920ab3d49bbee80ef1a277dd7b7aaf3b3545efd9d2a8114fe05525/ast_serialize-0.6.0-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:c4af9a1386166e40ed01464991806f89038a2d89782576c7774876fa77034e32", size = 1413698, upload-time = "2026-06-30T20:02:44.179Z" }, + { url = "https://files.pythonhosted.org/packages/a8/5a/e046f3899e2acba4677d7427b76431443a1aa1a0e583dfb05b55b69d55cf/ast_serialize-0.6.0-cp39-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:c901adbd750029b9ac4ad3d6aa56853e0ad4875119fbf52b7b8298afc223828b", size = 1512209, upload-time = "2026-06-30T20:02:45.584Z" }, + { url = "https://files.pythonhosted.org/packages/cc/c7/e42aaca7bb2d22a7c06d5a8c7930086c5a334e93d716e6fa5e6647a4515f/ast_serialize-0.6.0-cp39-abi3-musllinux_1_2_i686.whl", hash = "sha256:3ae22a366b752ab4496191525b78b097b5b72d531752e3c1dd7e383a8f2c8a1a", size = 1508464, upload-time = "2026-06-30T20:02:46.942Z" }, + { url = "https://files.pythonhosted.org/packages/95/93/5524a3dc6c3f593de3228ed9cbef73afa047625b7000ec21b7f58e6eb4d4/ast_serialize-0.6.0-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:4ed29121da8b3fdc291002801a1de0f76248fa07dce89157a5f277842cf6126e", size = 1457164, upload-time = "2026-06-30T20:02:48.294Z" }, + { url = "https://files.pythonhosted.org/packages/4f/c0/36a6ffb4d653cf621427b4c4928671f53ad800c453474de2b82564a44ad9/ast_serialize-0.6.0-cp39-abi3-pyemscripten_2026_0_wasm32.whl", hash = "sha256:b1dac4e09d341c1300ba69cdcbe62867b32a8c75d90db9bf4d083bec3b039f0b", size = 863014, upload-time = "2026-06-30T20:02:49.742Z" }, + { url = "https://files.pythonhosted.org/packages/09/c7/7d5ad8b49e1278e1c2a1e0274bd7850560b3f09313aa00c13bc8d5544792/ast_serialize-0.6.0-cp39-abi3-win32.whl", hash = "sha256:82c312a7844d2fdeb4d5c48bd3d215bf940dafd4704e1a9bcf252a99010a99b1", size = 1063165, upload-time = "2026-06-30T20:02:50.98Z" }, + { url = "https://files.pythonhosted.org/packages/47/ae/6710c14ecb276031cf10249f6adf5a59e2d3fdb3b5183bd59f70524067ee/ast_serialize-0.6.0-cp39-abi3-win_amd64.whl", hash = "sha256:113b58346f9ceb664352032770caca817d4a3c86f611c6088e6ef65ddaa70f0e", size = 1101444, upload-time = "2026-06-30T20:02:52.554Z" }, + { url = "https://files.pythonhosted.org/packages/66/40/c53deb2cd0c9b0fb636d24d9f40924cf2e65028e6b20b10cd5c1eeb2c730/ast_serialize-0.6.0-cp39-abi3-win_arm64.whl", hash = "sha256:ccd132fe8db56f61fe743b1f644d01b8d65b83248a8da506f3132bda86d6ed5e", size = 1072965, upload-time = "2026-06-30T20:02:54.097Z" }, +] + +[[package]] +name = "certifi" +version = "2026.7.22" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/a3/c2/24167ea9858356b47a87a50d39908bfdb72ceeefe0041586e704e5376b3a/certifi-2026.7.22.tar.gz", hash = "sha256:741e2c3b351ddf169a738da9f2c048608ff7f2c5cc02f1ebc6b118bb090d5d55", size = 138112, upload-time = "2026-07-22T03:35:12.644Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/0b/a7/71ac2cff56fec219ed242bb11b8efb69fcc4bec75db06fb7bfe35de520e6/certifi-2026.7.22-py3-none-any.whl", hash = "sha256:62f22742b58a1a33014a2b6b706588a8d7e2a88ae7bd1a6ebe8c992928483775", size = 136983, upload-time = "2026-07-22T03:35:11.276Z" }, +] + +[[package]] +name = "click" +version = "8.4.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "colorama", marker = "sys_platform == 'win32'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/76/d4/81420972a676e8ffea40450d8c8c92943e7218a78fe9b64359836cc9876b/click-8.4.2.tar.gz", hash = "sha256:9a6cea6e60b17ebe0a44c5cc636d94f09bd66142c1cd7d8b4cd731c4917a15f6", size = 338000, upload-time = "2026-06-24T17:45:15.148Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/fb/e2/79c688af8b210d232694e31e59da9f6ec747bae31c3f5946e4e9b98860d5/click-8.4.2-py3-none-any.whl", hash = "sha256:e6f9f66136c816745b9d65817da91d61d957fb16e02e4dcd0552553c5a197b76", size = 119243, upload-time = "2026-06-24T17:45:13.73Z" }, +] + +[[package]] +name = "colorama" +version = "0.4.6" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d8/53/6f443c9a4a8358a93a6792e2acffb9d9d5cb0a5cfd8802644b7b1c9a02e4/colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44", size = 27697, upload-time = "2022-10-25T02:36:22.414Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" }, +] + +[[package]] +name = "coverage" +version = "7.15.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/76/d0/55fe630f4cf94e3fcba868240fad8c8cdd1f764e2a932f8926347e6ec4cd/coverage-7.15.2.tar.gz", hash = "sha256:3df60dc267f0a2ca23cb7a9ab1109c62b9335ffbf519fcfe167157c28c09b81d", size = 927741, upload-time = "2026-07-15T18:56:19.558Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/6a/50/eb5bf42e531611a9f8d272556b1ed4de503f84a91413584094487cf69f8f/coverage-7.15.2-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:1adac78e5abc7c5438f7a209c9ca69d06542f0bf481d728b6989ea80b813fdf9", size = 221587, upload-time = "2026-07-15T18:54:18.439Z" }, + { url = "https://files.pythonhosted.org/packages/06/d1/da99af464c335d4e023a6efcd7ec30f63b88a43c93745154ab74ffb31cea/coverage-7.15.2-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:b868acc62aa5de3be7a9d05c2333bf8359ca987e43f9cb30ff8fbda6a024ab73", size = 221943, upload-time = "2026-07-15T18:54:20.062Z" }, + { url = "https://files.pythonhosted.org/packages/5b/8a/13c42723d61ca447eafa18732e8141dd6a63f2732e1c7e1502c182dd88d7/coverage-7.15.2-cp312-cp312-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:6f6966fc30e6f06ca8f98fb0ce51eda6b111b3ee8d066a8b1ec9e77fa06ab55d", size = 253450, upload-time = "2026-07-15T18:54:21.765Z" }, + { url = "https://files.pythonhosted.org/packages/d7/29/99021303f98fbdcb63504b4d07bea4cc025b9b2dd907c4f07c85d50a0dab/coverage-7.15.2-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:68af907f595ab01a78f794932ff3bdf929c316d3000810d38dbc247129e26f8b", size = 256187, upload-time = "2026-07-15T18:54:23.4Z" }, + { url = "https://files.pythonhosted.org/packages/f9/a8/fd503715ed6ca9c5d742923aa5209257340b367a867b2ced0c7d4ba8a0b9/coverage-7.15.2-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:afa29e2eff3d5729267e2cb2fd4ce9d61c952932fb2694e34ccb5d9540c6a296", size = 257301, upload-time = "2026-07-15T18:54:25.183Z" }, + { url = "https://files.pythonhosted.org/packages/da/40/3f4b8fb409810036ebc2857d36adc0498c6e957b5df0290c5036b2e143f1/coverage-7.15.2-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:bbf44513ceb1589e31948e20eafbde9deaface90e1a1afa5f5f77b4423d17ce6", size = 259562, upload-time = "2026-07-15T18:54:27.204Z" }, + { url = "https://files.pythonhosted.org/packages/0b/8a/9bdffbef47db77cce3d6b02a28f7e919b19f0106c4b080c2c2246040f885/coverage-7.15.2-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:9deddf09eecb717b7f980414b43d90a5b22ff3967d2949ab29cb0aa83d9e9098", size = 253841, upload-time = "2026-07-15T18:54:29.134Z" }, + { url = "https://files.pythonhosted.org/packages/1b/1e/9031efde019d31a06646261fce6dfc5c3c74e951e27a71e5c9a424563178/coverage-7.15.2-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:ae901f7e55ba405c84ee1cab3d3e962e4e871e4a2bcb9c90911adbd69b42ac5a", size = 255221, upload-time = "2026-07-15T18:54:31.142Z" }, + { url = "https://files.pythonhosted.org/packages/56/db/787acde872389fc84a9ef9d8cd1ccc658e391ab4cb5b28092a714426a394/coverage-7.15.2-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:a0f47002c6eeb7c280228467a4cb0cc15ca2103a8421b986b2d3ec04a0f9bd8b", size = 253366, upload-time = "2026-07-15T18:54:32.886Z" }, + { url = "https://files.pythonhosted.org/packages/2f/9b/6f57bc4b93c842eef1695f8cdaf2318e35e7ba54f5ba80d84be213ab7858/coverage-7.15.2-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:1cd7a5beb7af3e864a13b1f0fb26efd3695da43ef0daf71e586adfffaf34d5b2", size = 257434, upload-time = "2026-07-15T18:54:34.7Z" }, + { url = "https://files.pythonhosted.org/packages/88/26/b3186a21b2acc83e451118978905c81c7072c3333707804db09a78c096a2/coverage-7.15.2-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:97a5c5457a9fb1d6c4e06cfb5dc835871fbfb6a6a51addc9e925bdeff5ef7440", size = 252935, upload-time = "2026-07-15T18:54:36.548Z" }, + { url = "https://files.pythonhosted.org/packages/20/c2/c9f3376b2e717ea69ed7a6e9a5fcab968fb0b290db6cf4bd9a1fc7541b75/coverage-7.15.2-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:0901cfe6c13bcd2302da4f83e884555d2a22bda6e4c476f09ef204ba20ca536e", size = 254807, upload-time = "2026-07-15T18:54:38.296Z" }, + { url = "https://files.pythonhosted.org/packages/f0/e1/dfc15401f4a8aaeb486e1ba3e9e3c40522a6e38bd0ecf0b3f29cb8082957/coverage-7.15.2-cp312-cp312-win32.whl", hash = "sha256:b171bdd71cb7ff792bf32e376173b0ace7e7963e7e57c58dfc42063a6a7174cd", size = 223641, upload-time = "2026-07-15T18:54:40.103Z" }, + { url = "https://files.pythonhosted.org/packages/91/40/81b6d809d320cd366ec5bdf8176575e897dcb8efe7fb4b489ef9e93e4d13/coverage-7.15.2-cp312-cp312-win_amd64.whl", hash = "sha256:582edc45c2040543fef83341be23c43024a3ab3ae0c2d8bc498a06282905ad40", size = 224172, upload-time = "2026-07-15T18:54:41.882Z" }, + { url = "https://files.pythonhosted.org/packages/ef/28/9f14ec438149f7de557f45518f09b4a7917b795cc37083aa7db482693f8c/coverage-7.15.2-cp312-cp312-win_arm64.whl", hash = "sha256:a638db90c61cd219aeee65e83a24fdaa57269a741ae0cf773309208ac862cee3", size = 223556, upload-time = "2026-07-15T18:54:43.674Z" }, + { url = "https://files.pythonhosted.org/packages/fc/d5/f8c838e6b7282976f7c918884b792df7a0c42c5bba5d99c60ad2d221d56d/coverage-7.15.2-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:1121caa19159a38b5463eaae4b1e1fde81e525b15ecc5e000cd5b1a108f743a8", size = 221606, upload-time = "2026-07-15T18:54:45.448Z" }, + { url = "https://files.pythonhosted.org/packages/bf/37/97c926376364f66298cc44893b89cdf17b8bc406376497c4061ae4b8a8ff/coverage-7.15.2-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:a300c6934e0989c327b9e8a1e110329da4641149f872bbe9f70168be66da76c1", size = 221982, upload-time = "2026-07-15T18:54:47.341Z" }, + { url = "https://files.pythonhosted.org/packages/b7/30/a36050a6e83c2135ee0776f452ca3948224befc6d7f26acecc082d0c106a/coverage-7.15.2-cp313-cp313-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:2617f8799d268fabdeef42a7e89ac3a23e1deee9025427db2df970f99a89a578", size = 252972, upload-time = "2026-07-15T18:54:49.2Z" }, + { url = "https://files.pythonhosted.org/packages/31/d3/06b5f1daf95f0f15ab05bd75f26ba5f3c8b33d0bb72f3aaa3cf41d1bad3a/coverage-7.15.2-cp313-cp313-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:7dc2950a2992cd676d35c20ae63522836deeb034f08874699d14068710af3dc1", size = 255569, upload-time = "2026-07-15T18:54:51.098Z" }, + { url = "https://files.pythonhosted.org/packages/81/1c/9afb3f8de2b8d36960391c48559a2e3ff96594b58099f115921549ea8d0d/coverage-7.15.2-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9e36686f7a442185db2400b3df171aac520869faf9deb59df687d28659eda2a6", size = 256806, upload-time = "2026-07-15T18:54:53.145Z" }, + { url = "https://files.pythonhosted.org/packages/64/d8/b989f96061a5e32d82fddd1b1b9ff48a7c8f8ae7606f0e80fd9de54b1e33/coverage-7.15.2-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:7d29ca7bd67af6e12e74632d65f026eabc1364da5c254494cd914446a28a3ef7", size = 258936, upload-time = "2026-07-15T18:54:55.015Z" }, + { url = "https://files.pythonhosted.org/packages/b8/fa/f99771f5110457c7b511c1935ca49ddf288218eaa84322e028b9334146ae/coverage-7.15.2-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:db9c8438057e5b0f6a22a0af99c0c1d26b57fbbdbd1be5861ddb8f897fcc3a2d", size = 253178, upload-time = "2026-07-15T18:54:57.527Z" }, + { url = "https://files.pythonhosted.org/packages/f6/96/c098a6044d119c751ceede7be91035fa8310170ec24a6523aff72f0a5793/coverage-7.15.2-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:63022c4c8dec1d0342f05c3ede99842fe3d007689acc45e86f123a1746e4a026", size = 254934, upload-time = "2026-07-15T18:54:59.41Z" }, + { url = "https://files.pythonhosted.org/packages/b2/a2/1457b3a7a50c8d77500103b97a046db863e2f59a1cf6d2f814595f349885/coverage-7.15.2-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:6c0be82b4d4aa5b2704e08518e2252f3e3d110164bcca826816801052e48a7aa", size = 252898, upload-time = "2026-07-15T18:55:01.338Z" }, + { url = "https://files.pythonhosted.org/packages/6c/0e/76958874c471ecfcdde0d2b2747bb2c61bdbf34a40636f4ce9db9923e643/coverage-7.15.2-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:4510fb9cdf6bb02dfa6af0be4a534b8102d086e22e4a33f8836df663da3d660d", size = 257056, upload-time = "2026-07-15T18:55:03.243Z" }, + { url = "https://files.pythonhosted.org/packages/7c/7c/3d7c4e3bf58baa40327dc7edc2272b17cf02299366d52763db1b0ca1556a/coverage-7.15.2-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:42ec3d989421b174a2ab607c1539f24127ad362757b7f1c0c0d7a2993f7eb37b", size = 252718, upload-time = "2026-07-15T18:55:05.029Z" }, + { url = "https://files.pythonhosted.org/packages/c8/b8/1cecffed9ce14fb25be9ba42d37b6bb61485c9a3ddd43cd3dde36b6087d8/coverage-7.15.2-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:e8f91bce78e32343af184c3b7fa28fcf5a9e2641f4b6623d392038f804939188", size = 254490, upload-time = "2026-07-15T18:55:06.889Z" }, + { url = "https://files.pythonhosted.org/packages/6c/2c/42984561bc7f4c045dca67516a0c50ee5ef8d84352dbeb5559dc86c4823e/coverage-7.15.2-cp313-cp313-win32.whl", hash = "sha256:434e68d531858205895eb0d74b73d20b84260de426387d53c422a5acda2cf050", size = 223647, upload-time = "2026-07-15T18:55:08.941Z" }, + { url = "https://files.pythonhosted.org/packages/41/9f/39c7c9245efc583beddf89a87683574e663ed93637f3afb6cd7b88405676/coverage-7.15.2-cp313-cp313-win_amd64.whl", hash = "sha256:26c3b04a6377fd7c09800921fa934e3a17c0020439cd59df73e73ae1d4b6a78c", size = 224190, upload-time = "2026-07-15T18:55:10.789Z" }, + { url = "https://files.pythonhosted.org/packages/c7/de/3a2883cf8a213659280ef4b403059e17a9acaeb7fc7fd4105e1226ff2e6d/coverage-7.15.2-cp313-cp313-win_arm64.whl", hash = "sha256:3ed010aa1b69cda8e827aabfca9866216c980e2dca82ab9a78c5f83689964c8b", size = 223583, upload-time = "2026-07-15T18:55:12.678Z" }, + { url = "https://files.pythonhosted.org/packages/81/5f/aed265fd7a3551a394f36dfe41868aee709b7f95db4052205b4ad1563ac3/coverage-7.15.2-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:40f633c5c5fc783732f6312280122e859538fa24461235597c13d803ea9a108a", size = 221650, upload-time = "2026-07-15T18:55:14.527Z" }, + { url = "https://files.pythonhosted.org/packages/6b/2c/222ba12a545189017120f8eddfc1a0bd4616b47d5d4a8d99421edb2fe4c6/coverage-7.15.2-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:075560438765b7a2ef43bf7aa7758661b53d889df47f062a31bda6c1ade553a2", size = 221988, upload-time = "2026-07-15T18:55:16.674Z" }, + { url = "https://files.pythonhosted.org/packages/aa/38/304b5877ab46e6c290b4292cfcf3fe28245f0e5597cad7f6acc91fc7e0a4/coverage-7.15.2-cp314-cp314-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:25fd15dd40a0a2c51a500d664ca29053c09c3259d998407bf982b6e114696138", size = 253029, upload-time = "2026-07-15T18:55:18.856Z" }, + { url = "https://files.pythonhosted.org/packages/6c/58/821b533b8db9e44cf1d8a97bd525149ced40dde1d0093da02cb78e715244/coverage-7.15.2-cp314-cp314-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:b9a6367e4aff723e8ee8190836836124284e8fcd4265e307c844010cfa074f3f", size = 255536, upload-time = "2026-07-15T18:55:21.027Z" }, + { url = "https://files.pythonhosted.org/packages/f1/f2/7aa06604c389d32ea7f0a6a988359a7eafc3cd3f8e7bc2e88cd2fdf0b877/coverage-7.15.2-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9854ca62c152874b2060772503535be2e8f53f70b8aaa7686b094888d872f984", size = 256881, upload-time = "2026-07-15T18:55:23.125Z" }, + { url = "https://files.pythonhosted.org/packages/a2/4f/1ef342339c7916d0096bc5888cc0f653882cc7bc8f897d5cb89143287c9b/coverage-7.15.2-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:913b6c56e110da40e035bbd168353bf7aaa2544a5eaccea5d98a4629aac156c7", size = 259196, upload-time = "2026-07-15T18:55:25.099Z" }, + { url = "https://files.pythonhosted.org/packages/fe/f4/7ed055d7a9c5ec13b161773a115a5ccc6b0081d568c31fad830806306cc7/coverage-7.15.2-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:aaccad4129d735a8a4d526f26929894c9a4e8ef7034566f210b176749d6906e3", size = 253036, upload-time = "2026-07-15T18:55:27.018Z" }, + { url = "https://files.pythonhosted.org/packages/14/79/ea82cca18c242a3a38b6c017da39726aa62dcb64aa635abf79b92009975c/coverage-7.15.2-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:a164b50081fc7357331c4024ef4d17b78ba325f8380d05f5a69599a7e05257ee", size = 254887, upload-time = "2026-07-15T18:55:29.084Z" }, + { url = "https://files.pythonhosted.org/packages/a4/ba/a136db3c0d9562b00e10b72540dbf3a33cd3bc5b95060c9308e247494623/coverage-7.15.2-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:bfd341ccf78128e72c094bc70cc25b3ef309c33c7c2c66ba3ed4309549e02de1", size = 252852, upload-time = "2026-07-15T18:55:31.184Z" }, + { url = "https://files.pythonhosted.org/packages/17/17/ea334246b16b7d059953fad6fdefa11e33c68efbd3fe37b1098120a1fac2/coverage-7.15.2-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:1473b3ba8e7ee0f076117b1a72c23f579a2b9e2bb742f48a8d86ea27ca93f91a", size = 257128, upload-time = "2026-07-15T18:55:33.163Z" }, + { url = "https://files.pythonhosted.org/packages/ed/c3/074fb66d46d607855f710876b117cbda562c5ab08363528e78820449f937/coverage-7.15.2-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:17c432b5f73ad52ef46fb06019f6fa7c66ce381961cf0f7dfd1d3a4bd3a98145", size = 252668, upload-time = "2026-07-15T18:55:35.063Z" }, + { url = "https://files.pythonhosted.org/packages/e1/c1/f620850ada9b36435921c9a3a8057013422b1d964eb4bf37fe138724d192/coverage-7.15.2-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:77f0ef5011df53a4bd1b35211ab122287f8d9b8d7aa1c4553e5c2deb24b1d446", size = 254325, upload-time = "2026-07-15T18:55:37.125Z" }, + { url = "https://files.pythonhosted.org/packages/cc/31/a729ca3689404493af82ef8e6ff70bd88bdda8da89aeef6ca9b387aeb2b4/coverage-7.15.2-cp314-cp314-win32.whl", hash = "sha256:f653e5d7248c1191ec988a85c72edeab46c3ff44f90639a4ed4874ec0be90243", size = 223844, upload-time = "2026-07-15T18:55:39.078Z" }, + { url = "https://files.pythonhosted.org/packages/c6/83/5d809dc808fb1698c671f3e372259bb9158e64b7ea526fc6ab7de64de9fe/coverage-7.15.2-cp314-cp314-win_amd64.whl", hash = "sha256:9911f31aad8906abe337c271343485cf20df5e70df5d2f57f9f136e7b55f26bc", size = 224331, upload-time = "2026-07-15T18:55:41.346Z" }, + { url = "https://files.pythonhosted.org/packages/16/4e/35e488548e952795829e129995c4174df33bf432b591d1aa42c8d9e4e7ad/coverage-7.15.2-cp314-cp314-win_arm64.whl", hash = "sha256:e38def96ad59853824c97953fdcd2c320a84ba3ce99b417db78af8bb6c3db635", size = 223760, upload-time = "2026-07-15T18:55:43.518Z" }, + { url = "https://files.pythonhosted.org/packages/ed/49/dd2c86cd6374038f6e415fb5bfb86db5218553209c081384a020369dee79/coverage-7.15.2-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:835ec4e20b45f0a7f63ed78f94065aca00de033403df8377bfe8b9c6abc0a7be", size = 222384, upload-time = "2026-07-15T18:55:45.569Z" }, + { url = "https://files.pythonhosted.org/packages/d3/74/173ff17a1c0808e5a438f549f6f145d5ac7528f2791310b63523e3200ac7/coverage-7.15.2-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:7466cc7ab6dc0db871d264bf99e8779f0917ee63d40730af0552f71535a6e072", size = 222647, upload-time = "2026-07-15T18:55:47.544Z" }, + { url = "https://files.pythonhosted.org/packages/84/f8/b8cba872162356fb44ac79c10309d987206a4461e32072fc29228dad7331/coverage-7.15.2-cp314-cp314t-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:e370c12133095ff18432de8c044962be85a5a96d90c6fcbce8e17e76236d2328", size = 264013, upload-time = "2026-07-15T18:55:49.768Z" }, + { url = "https://files.pythonhosted.org/packages/ee/67/a807a7586d0b8cae485308ddd55756f0806c92f8e0b411bacbf23c48edf3/coverage-7.15.2-cp314-cp314t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:fe41909c9515c3bfdb5f02c4d1f857dba322d9a9a1178069b91eea77889df63a", size = 266135, upload-time = "2026-07-15T18:55:51.941Z" }, + { url = "https://files.pythonhosted.org/packages/ce/67/cd78771dc985f7e4ebdcc82b1a96d9a932af9e806f01f2f91a89f4c72e80/coverage-7.15.2-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6aa28cfb6488e5453b5b762d65f73aa586380f6693a04d58078ce228a29b06c0", size = 268555, upload-time = "2026-07-15T18:55:54.065Z" }, + { url = "https://files.pythonhosted.org/packages/18/3e/10134cf81275188c58568f324fc74aedff32c63ca4d5bbc513a91944a6f0/coverage-7.15.2-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:bcc0aae933921d03096f53b0b03eeb702129fd406dee59f08d2efacc68681fa5", size = 269674, upload-time = "2026-07-15T18:55:56.066Z" }, + { url = "https://files.pythonhosted.org/packages/75/4a/771b77de446cba985dc414bbc5844bd21604da05dbc044286df8318a48a7/coverage-7.15.2-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:7c63387e21ab21f512c69c9756a8c7dadd322c7275edb064064433c9a09c3743", size = 263101, upload-time = "2026-07-15T18:55:58.107Z" }, + { url = "https://files.pythonhosted.org/packages/5f/b5/70a7011da15f4071943361183aefa27847f3e3aec4fd335f1cb3d3a622b1/coverage-7.15.2-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:0e55510bc98ae943cece9e667a6c0fe94c6a92913720dea34243657a17993d0c", size = 266007, upload-time = "2026-07-15T18:56:00.468Z" }, + { url = "https://files.pythonhosted.org/packages/b4/0d/f9547e804ce7ad49646ffeffac26699510efbe6c0f751b66fdc960c4e825/coverage-7.15.2-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:2ff08701be2d1556fc78b326c80a3e8042da09352ecb3819105f8e386c8a3071", size = 263611, upload-time = "2026-07-15T18:56:02.615Z" }, + { url = "https://files.pythonhosted.org/packages/ac/59/f576a396659c0efd351f5c1544f67c3560e89c7761cabf7f65e412beeda5/coverage-7.15.2-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:38c9518b7103826c403a461544e3c2e77151e8676d06eaed85911a97e962584a", size = 267344, upload-time = "2026-07-15T18:56:04.622Z" }, + { url = "https://files.pythonhosted.org/packages/7c/5d/c2e4fce3579c0cb635024293f1a32bbe26df101b3e3a69f22243d1352b6c/coverage-7.15.2-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:dee88b1ed88587abd8c0269a1fc1f4cc77f7750d1dfde2869e2a123af420e67d", size = 262456, upload-time = "2026-07-15T18:56:06.641Z" }, + { url = "https://files.pythonhosted.org/packages/bb/dd/956287d69436b66094bc4b57ac2da71e43bfd2a5524e958900b9f582fcf8/coverage-7.15.2-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:2fbeeeecea279727f8ac16c8e1133ddfeee793e985c86ae343d6a5ce744eef8c", size = 264771, upload-time = "2026-07-15T18:56:08.795Z" }, + { url = "https://files.pythonhosted.org/packages/2c/5a/6f979530c2734c575de77cf58f5f28d51f7123a94b5030fd9156fe5f363c/coverage-7.15.2-cp314-cp314t-win32.whl", hash = "sha256:cb0fddaa6884be6aae36ced9544b5e90f7d5f03845a2853bf47a14953a4e8688", size = 224151, upload-time = "2026-07-15T18:56:10.856Z" }, + { url = "https://files.pythonhosted.org/packages/54/7e/27f6b2a74d484742f4017553e710b01e396b23d809df3e95ca0bb9a2824b/coverage-7.15.2-cp314-cp314t-win_amd64.whl", hash = "sha256:77f091ea3a9cc611cd29f433565476bc1936c084ac8eee00ea0e7e70c27e4199", size = 224981, upload-time = "2026-07-15T18:56:12.928Z" }, + { url = "https://files.pythonhosted.org/packages/b1/48/284863423aa474240f6842bd00d680da22f4e6ea2e466618ef7c9c9e69a9/coverage-7.15.2-cp314-cp314t-win_arm64.whl", hash = "sha256:6fc448c377d6eeb00a47c673494bd9bae29280ca53987e1869e67ebedfe20658", size = 224294, upload-time = "2026-07-15T18:56:15.156Z" }, + { url = "https://files.pythonhosted.org/packages/ec/82/32e3bd191d498e64f6f911ad55d14006a0861e54869d2d32452326399e65/coverage-7.15.2-py3-none-any.whl", hash = "sha256:eb6bcae8d1a9d305351ecb108232441d11c5cfe9de840a04388ba5d2db8d735c", size = 213375, upload-time = "2026-07-15T18:56:17.305Z" }, +] + +[[package]] +name = "h11" +version = "0.16.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/01/ee/02a2c011bdab74c6fb3c75474d40b3052059d95df7e73351460c8588d963/h11-0.16.0.tar.gz", hash = "sha256:4e35b956cf45792e4caa5885e69fba00bdbc6ffafbfa020300e549b208ee5ff1", size = 101250, upload-time = "2025-04-24T03:35:25.427Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/04/4b/29cac41a4d98d144bf5f6d33995617b185d14b22401f75ca86f384e87ff1/h11-0.16.0-py3-none-any.whl", hash = "sha256:63cf8bbe7522de3bf65932fda1d9c2772064ffb3dae62d55932da54b31cb6c86", size = 37515, upload-time = "2025-04-24T03:35:24.344Z" }, +] + +[[package]] +name = "httpcore" +version = "1.0.9" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "certifi" }, + { name = "h11" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/06/94/82699a10bca87a5556c9c59b5963f2d039dbd239f25bc2a63907a05a14cb/httpcore-1.0.9.tar.gz", hash = "sha256:6e34463af53fd2ab5d807f399a9b45ea31c3dfa2276f15a2c3f00afff6e176e8", size = 85484, upload-time = "2025-04-24T22:06:22.219Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7e/f5/f66802a942d491edb555dd61e3a9961140fd64c90bce1eafd741609d334d/httpcore-1.0.9-py3-none-any.whl", hash = "sha256:2d400746a40668fc9dec9810239072b40b4484b640a8c38fd654a024c7a1bf55", size = 78784, upload-time = "2025-04-24T22:06:20.566Z" }, +] + +[[package]] +name = "httpx" +version = "0.28.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "anyio" }, + { name = "certifi" }, + { name = "httpcore" }, + { name = "idna" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/b1/df/48c586a5fe32a0f01324ee087459e112ebb7224f646c0b5023f5e79e9956/httpx-0.28.1.tar.gz", hash = "sha256:75e98c5f16b0f35b567856f597f06ff2270a374470a5c2392242528e3e3e42fc", size = 141406, upload-time = "2024-12-06T15:37:23.222Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/2a/39/e50c7c3a983047577ee07d2a9e53faf5a69493943ec3f6a384bdc792deb2/httpx-0.28.1-py3-none-any.whl", hash = "sha256:d909fcccc110f8c7faf814ca82a9a4d816bc5a6dbfea25d6591d6985b8ba59ad", size = 73517, upload-time = "2024-12-06T15:37:21.509Z" }, +] + +[[package]] +name = "idna" +version = "3.18" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/cd/63/9496c57188a2ee585e0f1db071d75089a11e98aa86eb99d9d7618fc1edce/idna-3.18.tar.gz", hash = "sha256:ffb385a7e039654cef1ab9ef32c6fafe283c0c0467bba1d9029738ce4a14a848", size = 196711, upload-time = "2026-06-02T14:34:07.794Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1e/5e/d4e9f1a599fb8e573b7b87160658329fbf28d19eac2718f51fc3def3aa5a/idna-3.18-py3-none-any.whl", hash = "sha256:7f952cbe720b688055e3f87de14f5c3e5fdaa8bc3928985c4077ca689de849a2", size = 65455, upload-time = "2026-06-02T14:34:06.319Z" }, +] + +[[package]] +name = "iniconfig" +version = "2.3.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/72/34/14ca021ce8e5dfedc35312d08ba8bf51fdd999c576889fc2c24cb97f4f10/iniconfig-2.3.0.tar.gz", hash = "sha256:c76315c77db068650d49c5b56314774a7804df16fee4402c1f19d6d15d8c4730", size = 20503, upload-time = "2025-10-18T21:55:43.219Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/cb/b1/3846dd7f199d53cb17f49cba7e651e9ce294d8497c8c150530ed11865bb8/iniconfig-2.3.0-py3-none-any.whl", hash = "sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12", size = 7484, upload-time = "2025-10-18T21:55:41.639Z" }, +] + +[[package]] +name = "librt" +version = "0.13.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/dc/2f/3908645ddddab7120b46295e541ead308109fa48dbec7d67d7a778870d60/librt-0.13.0.tar.gz", hash = "sha256:1d2a610c14ac0d0750ee0a3ab8548e83155258387891caaca04def4bf7289781", size = 211402, upload-time = "2026-07-08T12:26:29.834Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f0/f4/b2933ddae222dac338476abb872641169a5cfed2c2bb5444a5b07b32b0c3/librt-0.13.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:30536798f4504c0fad0885b1d371b0539abb081e4570c9d7c641cb51141b49f0", size = 150990, upload-time = "2026-07-08T12:25:02.42Z" }, + { url = "https://files.pythonhosted.org/packages/90/ef/db98f744ca50e6efc9c95c70ee49b77aefac31f6a3fc7c83754a42d6a74f/librt-0.13.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:93d24ebb82aa4420b1409c389e7857bc35bd0b668007ac8172427d5c73cc8cc5", size = 155238, upload-time = "2026-07-08T12:25:03.681Z" }, + { url = "https://files.pythonhosted.org/packages/03/e7/a197e7bc72baf2c61ce7fdc6906a5054dc05bd8da0819aa894e4857bf87e/librt-0.13.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:cb8a1adce42d8b75485a5d56a9623a50bcab995b6079f1dac59fc44034dd93d9", size = 503073, upload-time = "2026-07-08T12:25:05.049Z" }, + { url = "https://files.pythonhosted.org/packages/f8/e7/7887712e27da7c1ab80fcabb1de6eb24243964f6557cae530d4b70706dbd/librt-0.13.0-cp312-cp312-manylinux2014_i686.manylinux_2_17_i686.manylinux_2_28_i686.whl", hash = "sha256:0763ca2ab66058174f9dee426dc64f5e0a89c24a7df8d3fe3f1836c04e25de4b", size = 496528, upload-time = "2026-07-08T12:25:06.26Z" }, + { url = "https://files.pythonhosted.org/packages/94/f0/f2283385bb6b950b26a1410f4ce51ec27231e0b3a4b925c46366d218b198/librt-0.13.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b222493da6e7b6199db9bd79502436cf5a27da3c1f7fa83c7e285444fc93fd03", size = 531786, upload-time = "2026-07-08T12:25:07.658Z" }, + { url = "https://files.pythonhosted.org/packages/36/11/69ac3b54766ffba5fd7e5acebfb048d66dbe1f9f2d14516c2b3edc59cf87/librt-0.13.0-cp312-cp312-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:fadc63331f4388c3dc90090448f682a7e9feafc11481391c1e94f2f907a3976e", size = 524393, upload-time = "2026-07-08T12:25:09.121Z" }, + { url = "https://files.pythonhosted.org/packages/61/5f/d72f95fd444a926a3c14b4e24979474116988dd57a45be242077c45d3c22/librt-0.13.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:70d9c62a4cffd9f23396cd5ef93fc5d11b31596b9b7d6306074abe3d5fcf09bd", size = 543026, upload-time = "2026-07-08T12:25:10.459Z" }, + { url = "https://files.pythonhosted.org/packages/c4/08/dcd9993ad192737a004ba263d549f8ea605b326b952e7d6205c7d4170b76/librt-0.13.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:66c0e7e6b02a155576df2c77ec933a70b72da726e248c494abf690923e624348", size = 546829, upload-time = "2026-07-08T12:25:11.716Z" }, + { url = "https://files.pythonhosted.org/packages/96/d5/6d9bb2f54e4109a956b7128836529653eb9d740f784bc47ed10a02c1000e/librt-0.13.0-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:ac04bcd3328eb91d99dfedf6a60d9c1f15d3434e6f6daf922f0420f7d90b85c7", size = 535700, upload-time = "2026-07-08T12:25:13.144Z" }, + { url = "https://files.pythonhosted.org/packages/8c/f2/10946922503858a359492fa27f13e86228bde702116a740ac7b3cd185f24/librt-0.13.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:db327e7271e653c32040b85ae6188059c924b57d7e1e29f935523fa017cd4e82", size = 573566, upload-time = "2026-07-08T12:25:14.336Z" }, + { url = "https://files.pythonhosted.org/packages/48/a8/94f00e3c99479a18088af3685ea016c42f3c7d5d1964d8dbb40c08d7f1aa/librt-0.13.0-cp312-cp312-win32.whl", hash = "sha256:860bd1d8ba48456ce08feaf8d343a8aaeb2fa086f2bcaa2a923fa3f7a3ff9aa3", size = 106099, upload-time = "2026-07-08T12:25:16.159Z" }, + { url = "https://files.pythonhosted.org/packages/c9/7b/2da9c74c1ed25a89cc4e1c8e007ea2eb4a0f1fafa3e70d757fe3242c5c5c/librt-0.13.0-cp312-cp312-win_amd64.whl", hash = "sha256:e54a315caf843c8d77e388cadc56ea9ded569935ee2d2347d7ea94992e5aa6fa", size = 126934, upload-time = "2026-07-08T12:25:17.275Z" }, + { url = "https://files.pythonhosted.org/packages/d0/65/aead61bbf3b5358593f9d4779d2a0e88eaf6ec191a6342dde36dd1df6371/librt-0.13.0-cp312-cp312-win_arm64.whl", hash = "sha256:c718e99a0992127af84385378460db624103b559ab260435abcfe77a4e4ed1c1", size = 112236, upload-time = "2026-07-08T12:25:18.425Z" }, + { url = "https://files.pythonhosted.org/packages/67/3b/18e7b63255297a2bdc9c25c8d6d4ca8eca9f63aceb1252c0f7427ac7099e/librt-0.13.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:a468951af16155824e88bdd8326ebe5bdb371f3ec0ac04642994b98201d914f3", size = 151027, upload-time = "2026-07-08T12:25:19.638Z" }, + { url = "https://files.pythonhosted.org/packages/4d/68/e2248452c00d1a03b45fee1752cdc8f790a476efd2402b75181da88a9e61/librt-0.13.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:ae01d8512cc17079e53425635327dbf3f7ff57a42c00dec348bf79791c56444c", size = 155152, upload-time = "2026-07-08T12:25:20.851Z" }, + { url = "https://files.pythonhosted.org/packages/0e/16/52b1c99bf19057a062aac39c900cbb81499f6f75d6c537c14463d247ba78/librt-0.13.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:32c26893cd085c1efe83219e78d866da23fb20a066101b8f68210004361d224c", size = 502499, upload-time = "2026-07-08T12:25:22.055Z" }, + { url = "https://files.pythonhosted.org/packages/9f/54/b811151805c795f55e0dedee6ec687b75f9982a8105d240ea3910737a77b/librt-0.13.0-cp313-cp313-manylinux2014_i686.manylinux_2_17_i686.manylinux_2_28_i686.whl", hash = "sha256:5929da1981a46bcf4b28b1b9499905f0ff58e2419da402a048234e9783acbc4b", size = 496108, upload-time = "2026-07-08T12:25:23.296Z" }, + { url = "https://files.pythonhosted.org/packages/8f/f8/094d6b2bd93f3fdaa54db54cc788c4a365333bddad65ab02e04da0b1d004/librt-0.13.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:94b85d664d777bab6c0d709416cb42938251fda9e221b79e3a2215d85df5f4f9", size = 531576, upload-time = "2026-07-08T12:25:24.648Z" }, + { url = "https://files.pythonhosted.org/packages/2e/40/541733d5755824f968f7ec39d78ffbd75d145964157ae5e69a09ec6d7326/librt-0.13.0-cp313-cp313-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:531b2df3e9fe96b1fcf73a6d165921e4656be5f58d631d384ebce344298368db", size = 524390, upload-time = "2026-07-08T12:25:25.898Z" }, + { url = "https://files.pythonhosted.org/packages/c6/b5/255673cfdbf5ba663339d36cd863c897289ab4337577e19f9405ce059f36/librt-0.13.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:109b84a9edf69ad89dc1f66358659e14a031baca95e3e5b0060bd903ede8efd6", size = 543053, upload-time = "2026-07-08T12:25:27.436Z" }, + { url = "https://files.pythonhosted.org/packages/9e/11/ab5005e9c9850710f21e354201bf090646349d3fabf5f951eaf70235729e/librt-0.13.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:1304368a3e7ffc3e9db986796cc5326fdb5943a3567ecc137cff318e4240c0e7", size = 546387, upload-time = "2026-07-08T12:25:28.65Z" }, + { url = "https://files.pythonhosted.org/packages/a2/04/a5d7ce1d1df1afd15ca283dcdf7530ac073e12d69ae8c40879dda96f7868/librt-0.13.0-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:e4f9b472e7d308d94b62c801982065661158c6ed02790d6c7ddb4337cea0f9c1", size = 535970, upload-time = "2026-07-08T12:25:30.171Z" }, + { url = "https://files.pythonhosted.org/packages/5a/76/927e267a6daa290174ac281b23c9804c8829b042ade9c6f24a065f540958/librt-0.13.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:9f836c37478f167a81200d8c8b2c920a22224564bed2c23d7aeec760965c367a", size = 573582, upload-time = "2026-07-08T12:25:31.507Z" }, + { url = "https://files.pythonhosted.org/packages/10/24/b6c5213efe39c19f9e13605644d0cf063b4ddaa33ac2e45b088e23a70e2e/librt-0.13.0-cp313-cp313-pyemscripten_2025_0_wasm32.whl", hash = "sha256:4000d961ff9598ac6ea603c6c836a5ed49bc205ade5fc378b998dfe1e2c36628", size = 82189, upload-time = "2026-07-08T12:25:32.675Z" }, + { url = "https://files.pythonhosted.org/packages/4c/00/d29736be177a906ac0b84a5b04b4fbfa22c776dc2f366de4172b0f968c08/librt-0.13.0-cp313-cp313-win32.whl", hash = "sha256:79e44cff71750d299d61a678e49995b0d5935a9cda238c2574daeca3ba536927", size = 106193, upload-time = "2026-07-08T12:25:33.692Z" }, + { url = "https://files.pythonhosted.org/packages/c8/ac/aff6fb45393cb8912f39dfb156ef6b2d1cadb207ff465fc8f66141054be8/librt-0.13.0-cp313-cp313-win_amd64.whl", hash = "sha256:54dab44a847d5ad1acd05c8a83fe518ae685516ecf4d3f7cc6e3df2a66767650", size = 126962, upload-time = "2026-07-08T12:25:34.769Z" }, + { url = "https://files.pythonhosted.org/packages/d9/3a/d68cb2b334d53fd30fac81d3a489ce4ba0d9506f4df43fcf676b68352b19/librt-0.13.0-cp313-cp313-win_arm64.whl", hash = "sha256:d4cb6fbfdf874340ab5e51450753c0f817b6958a3621125ee695bbc3de866566", size = 112127, upload-time = "2026-07-08T12:25:35.981Z" }, + { url = "https://files.pythonhosted.org/packages/7b/66/f49ae0d592bd45b6941e9a8bafcb6a87cddcd501ee7874707e767f01b585/librt-0.13.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:25218d94b1d2cbc0ba1d8a3f9dc9af578d9646e5ed16443a70cde1dfdcce6d71", size = 149818, upload-time = "2026-07-08T12:25:37.203Z" }, + { url = "https://files.pythonhosted.org/packages/3d/50/51c76d74014d04fb95b6506d286808984b78a2f7a41039094e6b2194ac48/librt-0.13.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:f26629539d4893c2957a16c41bb058e1e135c1f150f6a2e25ed047f64cf3f5c6", size = 154071, upload-time = "2026-07-08T12:25:39.399Z" }, + { url = "https://files.pythonhosted.org/packages/b8/fe/f19b0f5f82d5a1f2da736586bc840abd00ce07d6388136ae80b7333883fc/librt-0.13.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a4517d47b2b8af26975a406fba7d314de9696d864252e0257c6ea90238cfe27f", size = 494168, upload-time = "2026-07-08T12:25:40.641Z" }, + { url = "https://files.pythonhosted.org/packages/94/bc/b8550c75775127fd31a5f20e8775997f7b527ad661fc8ddccd7497c064f7/librt-0.13.0-cp314-cp314-manylinux2014_i686.manylinux_2_17_i686.manylinux_2_28_i686.whl", hash = "sha256:f19e181de5b3a1148bb3420b8c4b0b0ea0fce6950099724ad151d6cea5acc180", size = 491054, upload-time = "2026-07-08T12:25:41.905Z" }, + { url = "https://files.pythonhosted.org/packages/30/14/4d0204867623df3f33f86efd3d3692ba5e01321443f4d6eab35a22697618/librt-0.13.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:22034924f5b42d5a56371cf271771bfeaabf235a7a8b6264bef2d20013f786c6", size = 523006, upload-time = "2026-07-08T12:25:43.327Z" }, + { url = "https://files.pythonhosted.org/packages/19/0a/c45fc9a260934696bace1ac5df1e148ac92bd71767aee3bf7cd7a4534f4c/librt-0.13.0-cp314-cp314-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:c7897db4e95e22468bdda33d8e012ceacd0182abf001e6389d763f0def6286b9", size = 515058, upload-time = "2026-07-08T12:25:44.541Z" }, + { url = "https://files.pythonhosted.org/packages/13/0a/50c5ce45b326854ef8fa6ae4c36cf5142e5c55315eaf9e51d0ae73ac4da3/librt-0.13.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:1ce61b3746545029d4f5c17d6bd74b676254ad98433086c846ffb5e8fa73f007", size = 534025, upload-time = "2026-07-08T12:25:45.825Z" }, + { url = "https://files.pythonhosted.org/packages/89/2d/08c413c8f93fc13b8103624fce38e5caa86cd08cbbc8465870ab287af54b/librt-0.13.0-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:46c330e82565962c761dbce7941be2cff7db674ee807455a8d0cadc5f9b759b0", size = 540557, upload-time = "2026-07-08T12:25:47.059Z" }, + { url = "https://files.pythonhosted.org/packages/b3/c1/93af71fb4a364952210051811dd4e40174e79656b050c89cacac18af3330/librt-0.13.0-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:375f5af8f99cbaa99dd293af986e3d57caabc9ba81a5d3f021603764854197a1", size = 523201, upload-time = "2026-07-08T12:25:48.392Z" }, + { url = "https://files.pythonhosted.org/packages/c1/6e/9766f07b676a4889d9f8bc2864e9ba5fff165653143ef4dda7df6aa34d16/librt-0.13.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:9320d34c3376ae204b2cd176e8d4883a013934e0aef822f1aed9c536490c275d", size = 565740, upload-time = "2026-07-08T12:25:49.678Z" }, + { url = "https://files.pythonhosted.org/packages/a2/1e/664e3472ce2b6e10e9b83f29d4a36eb982ff6b5a169ae7567bba3a4c4ff5/librt-0.13.0-cp314-cp314-pyemscripten_2026_0_wasm32.whl", hash = "sha256:9af313c66157a69dc69ea0059a66961692250e0dc95af9c385a48ffb770a0d16", size = 81611, upload-time = "2026-07-08T12:25:50.857Z" }, + { url = "https://files.pythonhosted.org/packages/2f/d4/8582a4d65e2234673685e07309d02c230b28a85724eb0acbf13f019b7f6e/librt-0.13.0-cp314-cp314-win32.whl", hash = "sha256:f2a7253458e34f33543551394ae4fe104b497ec2a65ac266074de64c1df82e37", size = 100106, upload-time = "2026-07-08T12:25:52.03Z" }, + { url = "https://files.pythonhosted.org/packages/63/ce/0cb99efe6086b46cd985dc26672166fae312a239690e75871f7fafbd3fc5/librt-0.13.0-cp314-cp314-win_amd64.whl", hash = "sha256:a3dfe4edf10e8ed7e55b026a8bfc2c2a8704218b659cd4bffdf604fab966dc39", size = 121209, upload-time = "2026-07-08T12:25:53.166Z" }, + { url = "https://files.pythonhosted.org/packages/26/85/4f3ccb083a3c9b0d42e223acdb3c3f507953324a59cdcab4826e8e2e3b89/librt-0.13.0-cp314-cp314-win_arm64.whl", hash = "sha256:68a5faee4bba381cb93b5961f684a514cf0053cb92308ff9c792c2fea0b174c6", size = 106404, upload-time = "2026-07-08T12:25:54.253Z" }, + { url = "https://files.pythonhosted.org/packages/b2/77/333191499538c8e8189de7a4cba8e6f49ee949fd6d6e6324b21fd1522466/librt-0.13.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:a38fb81d8376dfa2f8963b265fec07637802b0d01e2a127c19c66cb070fb24f5", size = 159231, upload-time = "2026-07-08T12:25:55.432Z" }, + { url = "https://files.pythonhosted.org/packages/7a/9e/2aa83758f22c278b837a1d8025898434ce2b8bff36678d5330ecaef56dff/librt-0.13.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:d4c8d9bd5abce34b2e75edb3bf37ab0f34e49b1f915a40ae8468eb7c85bc5b46", size = 161300, upload-time = "2026-07-08T12:25:56.585Z" }, + { url = "https://files.pythonhosted.org/packages/bb/c0/86791e936553ca763d6b3c2fb4d31d596cd00e14fa631c283a40ba01559a/librt-0.13.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:387e2f1d27e89bffe0d3f520f0da0662c973fd607ca16c1808f8a5085419485e", size = 582056, upload-time = "2026-07-08T12:25:58.144Z" }, + { url = "https://files.pythonhosted.org/packages/a8/d3/a9ec15984a185e000c4d2a16ba28bd623124ad4c38a10974c7ff78e3a893/librt-0.13.0-cp314-cp314t-manylinux2014_i686.manylinux_2_17_i686.manylinux_2_28_i686.whl", hash = "sha256:4f6db193d2e5e0ed60359b9a5a682cd67205d0d3b1e459a867dd4b5c4e7eaa7a", size = 562758, upload-time = "2026-07-08T12:25:59.544Z" }, + { url = "https://files.pythonhosted.org/packages/3c/af/dbe36b78b19c06a55097f99305e4ea9458e2273e6ae16a3cbecaad7ee978/librt-0.13.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0d38604854e8d22faadf683ec6c02bb0f886e2ba56ef981a1c36ee275f21ea22", size = 602095, upload-time = "2026-07-08T12:26:00.991Z" }, + { url = "https://files.pythonhosted.org/packages/2a/a8/2966891b4dd2830f5203fbee92ac2c4947653a2390ba73dfa44244fad025/librt-0.13.0-cp314-cp314t-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:371f7ce73026815dafd51c50ce38416e91428b28c4b2ec97cd39271164b0045c", size = 593452, upload-time = "2026-07-08T12:26:02.352Z" }, + { url = "https://files.pythonhosted.org/packages/61/f5/4df8bfc8405ecf8c0d525b4d69636f694bdd8620b313ec8b76e54a5926cc/librt-0.13.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:3aaedf52171bee90860704c560bc798fe83b76247df47568e0197e9b13c735a0", size = 623729, upload-time = "2026-07-08T12:26:04.294Z" }, + { url = "https://files.pythonhosted.org/packages/d6/13/9ac202dffc8db06f75d06c08c2f9f6ff054be67d21272dcc078fa1cc0c57/librt-0.13.0-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:96bad8725a4f196a798366c25ce075d1f7543a4ec045ffc13e6a7ec095cdab04", size = 617077, upload-time = "2026-07-08T12:26:05.845Z" }, + { url = "https://files.pythonhosted.org/packages/6e/f0/ebe38610716aee5cb28efd95089bb90192096179802779381e1c5dcf239c/librt-0.13.0-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:6bf6a559ffe4a93bbea6cf31ddf01a7fd9ba342ef51f27beb178e318b74acd61", size = 599561, upload-time = "2026-07-08T12:26:07.21Z" }, + { url = "https://files.pythonhosted.org/packages/4f/5c/c2e72e236fff7abc716d5b1753b8b8cd3ea85ac46fe17d2e7c51d4e1c723/librt-0.13.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:301067672387902c55f94b51d5022304b36c966ea9fe1f21caab99a9bef487c9", size = 645511, upload-time = "2026-07-08T12:26:08.562Z" }, + { url = "https://files.pythonhosted.org/packages/0c/99/6203ce619dee940d6bfbe099ec3fe4be00a68e9d60f70abf906cf124fe66/librt-0.13.0-cp314-cp314t-win32.whl", hash = "sha256:5fdcf34f86de8fb66d7dc7589f96ba91c4aa46671200d400e6fd6f109a483f18", size = 104357, upload-time = "2026-07-08T12:26:09.828Z" }, + { url = "https://files.pythonhosted.org/packages/52/dd/843b6314087c41657c7036d7914d8f294bdf9b580aa8513ea0588c8e9a3d/librt-0.13.0-cp314-cp314t-win_amd64.whl", hash = "sha256:260c33e92263fa629b4f6d3c51967a1c2158fe6c33237aaa3ebeac586b085259", size = 126998, upload-time = "2026-07-08T12:26:10.975Z" }, + { url = "https://files.pythonhosted.org/packages/5f/5d/3dcec2884ba1b0806d1408612555c38dd5d68e90156b59f75f6e36435c3a/librt-0.13.0-cp314-cp314t-win_arm64.whl", hash = "sha256:2f281549a4c52ac7bb97997f14353f8bd0e53a34ca0dad1c905cfd0b4a58ae99", size = 110771, upload-time = "2026-07-08T12:26:12.303Z" }, +] + +[[package]] +name = "mypy" +version = "2.3.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "ast-serialize" }, + { name = "librt", marker = "platform_python_implementation != 'PyPy'" }, + { name = "mypy-extensions" }, + { name = "pathspec" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/12/af/4e516a05d3ca2eb9283e9ec45b2c02225c1514dd6da49fd3c9eaa6639370/mypy-2.3.0.tar.gz", hash = "sha256:465965d41cd9a2726694e983e8ce7113259327bec798115d1e1dfa2a52fb666e", size = 3988104, upload-time = "2026-07-13T11:34:53.387Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/dc/94/0e7e592619e2133596a47cdd642534b0456545c218430bd3b9d8fefdd1b1/mypy-2.3.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:2d53fc67b9d28a43c6199077f49fea0f05839e36cf6158500331c9549225e5a5", size = 15026523, upload-time = "2026-07-13T11:34:49.206Z" }, + { url = "https://files.pythonhosted.org/packages/f6/d2/1e1731df090a857df2807177a4626863e5ac0f0256513c35780efe53986f/mypy-2.3.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:fbc00cee7bdbb9291979ddc9d08034a29dfcda4932628c9bbc28c1edd589df0c", size = 14032189, upload-time = "2026-07-13T11:33:57.168Z" }, + { url = "https://files.pythonhosted.org/packages/44/95/cab921f4a806e171f34113e6181dd23c55358ccf6a80741269ef594a410e/mypy-2.3.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:04e617030eca5221909c8b7d8d7fd1c637948199aa2100b2ad9813feb07e1491", size = 14198696, upload-time = "2026-07-13T11:32:12.767Z" }, + { url = "https://files.pythonhosted.org/packages/66/80/e6d008bb19fe446e3662d85e0e2717bf9f2d611a2164fb29d6e067dbf46c/mypy-2.3.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:56c184d2c20ca6b6378d58d1960270a767f41f5e44acbbd27f05effef4f4e1d7", size = 15286904, upload-time = "2026-07-13T11:34:27.594Z" }, + { url = "https://files.pythonhosted.org/packages/db/83/94397c9293608a364aa03e8084fb34ede4ae976a260384b9b52929308135/mypy-2.3.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:3961a4a34b05f7c74b0f05aa51fbfe99a2d1e126038df40318d15c8f558b7ef3", size = 15528342, upload-time = "2026-07-13T11:34:07.819Z" }, + { url = "https://files.pythonhosted.org/packages/cf/96/d8b37d819adec6cfccfb1fd3afc1735d94717ddeafb45536db9c6943e09b/mypy-2.3.0-cp312-cp312-win_amd64.whl", hash = "sha256:b1942b9314d4c784b8ea1dbab4972603290e5dd5630f06675f13aec97526bc4c", size = 11218346, upload-time = "2026-07-13T11:28:27.745Z" }, + { url = "https://files.pythonhosted.org/packages/2b/cd/cd9f725b19b19e5b530a154cf9bcf9e94279c5d55b3c34fb42b3aa48ea1b/mypy-2.3.0-cp312-cp312-win_arm64.whl", hash = "sha256:be51653d7669d7d7955d613b8d0bb57d5b652eaf71a873ddf65ac87254dd2595", size = 10204525, upload-time = "2026-07-13T11:31:02.552Z" }, + { url = "https://files.pythonhosted.org/packages/6e/ae/f7d056eb0294586a572d0d0d89580ec633c064db520f11d37d5a2fb833bd/mypy-2.3.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:91ad22a52ae2c7e621c2f67c94d5a17f66b3209a4cff5cf8a573579835c69e97", size = 14947298, upload-time = "2026-07-13T11:27:47.734Z" }, + { url = "https://files.pythonhosted.org/packages/32/d5/db3e7af01e7844d21662c6ddc1f7825ec7cb4053f0391ac02faf3638396f/mypy-2.3.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:99ac767cc5d3b64c8d0ae226ead10c96694f94e4e7da1668642225dcd4e75aac", size = 13950768, upload-time = "2026-07-13T11:27:57.726Z" }, + { url = "https://files.pythonhosted.org/packages/d9/fb/43c031f0190513d1ec248ed037eceb742ddd2a4d74bbf406658a28173837/mypy-2.3.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:de6d2c484742a4d7b0ed6d07b143375624d3b899c5749c7b3c947f56261f48a6", size = 14151586, upload-time = "2026-07-13T11:29:18.615Z" }, + { url = "https://files.pythonhosted.org/packages/ec/c3/f8b2ffc60883084da91be51af58e88a7ffd4ff9795acb7d902ff88d31eb1/mypy-2.3.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7da939dd335cfd2ad788bdfd081c9f4e47634ab995e5a45eb15fd1e5bc052f8b", size = 15227411, upload-time = "2026-07-13T11:30:29.904Z" }, + { url = "https://files.pythonhosted.org/packages/83/2e/16b917fc7adcf03f1aadddfc93aab804ffb234b1ab09c0ffd6d92a5d34a2/mypy-2.3.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:7247eb2824f996722a949530183394921ca71deb9680052a338cf53cff7925c2", size = 15478790, upload-time = "2026-07-13T11:33:14.686Z" }, + { url = "https://files.pythonhosted.org/packages/c0/88/aaa65a93c73d0cdae7e42f8adb302bf6885bb281302084f99d0290a35347/mypy-2.3.0-cp313-cp313-win_amd64.whl", hash = "sha256:75b0984bb3cbd76bb5c9291a8671f7ae66ca3b51c7584c358fc2e923259f0757", size = 11234919, upload-time = "2026-07-13T11:33:39.28Z" }, + { url = "https://files.pythonhosted.org/packages/35/19/b40de63f1a80e63bc2d40f0679a6a8dbd34e95176c8122119bdf406aa552/mypy-2.3.0-cp313-cp313-win_arm64.whl", hash = "sha256:d78fcf900b59cb7e82cb7e3a235e31b462d9333d92285bd1e4952d355b8ffba1", size = 10201510, upload-time = "2026-07-13T11:31:52.619Z" }, + { url = "https://files.pythonhosted.org/packages/a4/58/fa0ae047da911f540284009b4f44b96fe09d83c076d7c103e9d645f46303/mypy-2.3.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:ea317b060ce83e26050f8f9e4d7d6bf44ed7597c8ff9990bccffbb9d1d8522db", size = 14941909, upload-time = "2026-07-13T11:32:34.332Z" }, + { url = "https://files.pythonhosted.org/packages/15/14/2ba1d61452d7c2a7fe12741e8d374e52b183476b07aa7f9e2a0d02b0720a/mypy-2.3.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:094af99f92638aa92852326188b85a89e50f4a472f44827c03362228482f0762", size = 13967581, upload-time = "2026-07-13T11:30:00.587Z" }, + { url = "https://files.pythonhosted.org/packages/ed/5a/483fb9e5ffbbb1a28dccc7b0a13d141b17ac769b6c9f488c0a0c63698962/mypy-2.3.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:de121747278144fc9ae7caa2e978cf5df12aebc82933182f5b3b86081a30baef", size = 14168807, upload-time = "2026-07-13T11:28:48.6Z" }, + { url = "https://files.pythonhosted.org/packages/ae/77/70d7a10732063beb74ad713682cf871e88f5c5fa39bfc8beff8a524bf9cb/mypy-2.3.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:37fa4de896a84e2dc9200d91e614c22563b43d1a266789d4bbac7b22ebe6192b", size = 15200144, upload-time = "2026-07-13T11:31:25.283Z" }, + { url = "https://files.pythonhosted.org/packages/56/72/766218ac783be4fdfcd699b90037b63017348a3e86fb2c1fbfb18302637d/mypy-2.3.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:f1b3a98dfd21058bc759bb3337d5d1f61d0fdf9f3cf9c00f4291790fb5427bff", size = 15460389, upload-time = "2026-07-13T11:29:29.077Z" }, + { url = "https://files.pythonhosted.org/packages/38/4e/8a9db7411ecb8ec0cb1fd05dba432f28bafffcd38b4e887714a4a0506689/mypy-2.3.0-cp314-cp314-pyemscripten_2026_0_wasm32.whl", hash = "sha256:944c665d984157cb96a679dfb7a4a81dd1d36b24b9c284b699514e6e626b82d4", size = 7753664, upload-time = "2026-07-13T11:29:08.147Z" }, + { url = "https://files.pythonhosted.org/packages/65/4c/c3f8bfd6ed0e5e38b5a244403b27f821d433443df5a15a278417c10a3a3c/mypy-2.3.0-cp314-cp314-win_amd64.whl", hash = "sha256:4359424140d985192c778c1ce2c114a10c1ca58a381ed79cfa70d37df94b299f", size = 11417237, upload-time = "2026-07-13T11:33:47.467Z" }, + { url = "https://files.pythonhosted.org/packages/3c/00/89a32eaf5ccf174bc4f90db0eaea5d70636c01b8d49f384bdab2e8834390/mypy-2.3.0-cp314-cp314-win_arm64.whl", hash = "sha256:3dd0bed92c4bdec57c42505b96416fb9e6a5aa7be84d2809bcd5f2ecec2860d7", size = 10389252, upload-time = "2026-07-13T11:31:43.81Z" }, + { url = "https://files.pythonhosted.org/packages/31/56/104f93d69aa9f339b6b9d3b0a7faa699b8b466c942cf3ae86cc2a2ec0915/mypy-2.3.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:691fdc37132b1ae628d834f672e74de83462d9fb4aff621835767fb43a8dd373", size = 16385495, upload-time = "2026-07-13T11:29:49.818Z" }, + { url = "https://files.pythonhosted.org/packages/d2/03/f1d2123313f55efafdd27706960f43a771c62f1b68426c76043f3ab9ebf3/mypy-2.3.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:aec15d465d477558fd842757b487849007311cf3897849cdda0e3162ac0ac556", size = 15098155, upload-time = "2026-07-13T11:30:40.301Z" }, + { url = "https://files.pythonhosted.org/packages/e5/5d/d5f9200399b445e81726c4f23becee33f233aee81c72680b1ef3a258b641/mypy-2.3.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b352b7e49f5e6576009e8df730e1ff4f915cb565b851b396d2ffe2f5a6f5da88", size = 15514155, upload-time = "2026-07-13T11:34:38.569Z" }, + { url = "https://files.pythonhosted.org/packages/cd/ce/69977c555f08faa3190cfde44189b89dbd56861b1ab97aa18fc5f3a2e4a3/mypy-2.3.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1c6c6bf687b17f90dbfcad95b960d32eaa0154c00da45f03ab50bf8952e047fe", size = 16766351, upload-time = "2026-07-13T11:33:29.195Z" }, + { url = "https://files.pythonhosted.org/packages/bc/92/6648b6caa3ab9e00f9ac0c2a78307805f873dd48139b24a6f6f7c3667bbf/mypy-2.3.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:f4ed18f111bfe2d599bca7468e7f9251042c1c2118f762c8de2766a56d773c60", size = 17043490, upload-time = "2026-07-13T11:30:53.927Z" }, + { url = "https://files.pythonhosted.org/packages/7c/ab/0dc91d80f3f016634c68d451f294a97320fe903a9b6f90b9e57b3f7f1717/mypy-2.3.0-cp314-cp314t-win_amd64.whl", hash = "sha256:0b025a93cffb9781d231f232be07a17912f35f10a313c24f301c81e842870654", size = 12146869, upload-time = "2026-07-13T11:29:38.874Z" }, + { url = "https://files.pythonhosted.org/packages/85/b5/4c964d02634ba81f4d1c84838e5c5b18ab06d13ed568960f5d6318495ccc/mypy-2.3.0-cp314-cp314t-win_arm64.whl", hash = "sha256:adebc76aab4f3495a88b41d48aa4aff0c03f2822501da76625afcca5975f19e5", size = 10965113, upload-time = "2026-07-13T11:28:07.056Z" }, + { url = "https://files.pythonhosted.org/packages/2c/fa/fdc54fe583ba3cafbcedfb70eeeaf03849f75b1827a07096c7bd996f582d/mypy-2.3.0-py3-none-any.whl", hash = "sha256:6b1cdb579446b60432432b2b2403a6201b4b475a004d7f488511c9ba177c9e88", size = 2753292, upload-time = "2026-07-13T11:33:18.48Z" }, +] + +[[package]] +name = "mypy-extensions" +version = "1.1.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/a2/6e/371856a3fb9d31ca8dac321cda606860fa4548858c0cc45d9d1d4ca2628b/mypy_extensions-1.1.0.tar.gz", hash = "sha256:52e68efc3284861e772bbcd66823fde5ae21fd2fdb51c62a211403730b916558", size = 6343, upload-time = "2025-04-22T14:54:24.164Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/79/7b/2c79738432f5c924bef5071f933bcc9efd0473bac3b4aa584a6f7c1c8df8/mypy_extensions-1.1.0-py3-none-any.whl", hash = "sha256:1be4cccdb0f2482337c4743e60421de3a356cd97508abadd57d47403e94f5505", size = 4963, upload-time = "2025-04-22T14:54:22.983Z" }, +] + +[[package]] +name = "packaging" +version = "26.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d7/f1/e7a6dd94a8d4a5626c03e4e99c87f241ba9e350cd9e6d75123f992427270/packaging-26.2.tar.gz", hash = "sha256:ff452ff5a3e828ce110190feff1178bb1f2ea2281fa2075aadb987c2fb221661", size = 228134, upload-time = "2026-04-24T20:15:23.917Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/df/b2/87e62e8c3e2f4b32e5fe99e0b86d576da1312593b39f47d8ceef365e95ed/packaging-26.2-py3-none-any.whl", hash = "sha256:5fc45236b9446107ff2415ce77c807cee2862cb6fac22b8a73826d0693b0980e", size = 100195, upload-time = "2026-04-24T20:15:22.081Z" }, +] + +[[package]] +name = "pathspec" +version = "1.1.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/5a/82/42f767fc1c1143d6fd36efb827202a2d997a375e160a71eb2888a925aac1/pathspec-1.1.1.tar.gz", hash = "sha256:17db5ecd524104a120e173814c90367a96a98d07c45b2e10c2f3919fff91bf5a", size = 135180, upload-time = "2026-04-27T01:46:08.907Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f1/d9/7fb5aa316bc299258e68c73ba3bddbc499654a07f151cba08f6153988714/pathspec-1.1.1-py3-none-any.whl", hash = "sha256:a00ce642f577bf7f473932318056212bc4f8bfdf53128c78bbd5af0b9b20b189", size = 57328, upload-time = "2026-04-27T01:46:07.06Z" }, +] + +[[package]] +name = "pluggy" +version = "1.6.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f9/e2/3e91f31a7d2b083fe6ef3fa267035b518369d9511ffab804f839851d2779/pluggy-1.6.0.tar.gz", hash = "sha256:7dcc130b76258d33b90f61b658791dede3486c3e6bfb003ee5c9bfb396dd22f3", size = 69412, upload-time = "2025-05-15T12:30:07.975Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl", hash = "sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746", size = 20538, upload-time = "2025-05-15T12:30:06.134Z" }, +] + +[[package]] +name = "pygments" +version = "2.20.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/c3/b2/bc9c9196916376152d655522fdcebac55e66de6603a76a02bca1b6414f6c/pygments-2.20.0.tar.gz", hash = "sha256:6757cd03768053ff99f3039c1a36d6c0aa0b263438fcab17520b30a303a82b5f", size = 4955991, upload-time = "2026-03-29T13:29:33.898Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f4/7e/a72dd26f3b0f4f2bf1dd8923c85f7ceb43172af56d63c7383eb62b332364/pygments-2.20.0-py3-none-any.whl", hash = "sha256:81a9e26dd42fd28a23a2d169d86d7ac03b46e2f8b59ed4698fb4785f946d0176", size = 1231151, upload-time = "2026-03-29T13:29:30.038Z" }, +] + +[[package]] +name = "pytest" +version = "9.1.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "colorama", marker = "sys_platform == 'win32'" }, + { name = "iniconfig" }, + { name = "packaging" }, + { name = "pluggy" }, + { name = "pygments" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/e4/47/b9efed96c114afcfa3c9d3fe98a76a1d14c74a9e266d397cf6eb64be5e01/pytest-9.1.1.tar.gz", hash = "sha256:1088fbde8f2b49d95a549a195707afa7a76a3ce9bcadc26b6d71f0ffda5fe313", size = 1636369, upload-time = "2026-06-19T10:58:32.857Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/24/25/1de2678b631f5a49215c6c96fff41ba892b0a34df68d6d80292b1b48aa7f/pytest-9.1.1-py3-none-any.whl", hash = "sha256:37a86b45efb9a47a61a36449063e8e18d0cab3161329fc099eb21783169c4f0c", size = 386536, upload-time = "2026-06-19T10:58:31.347Z" }, +] + +[[package]] +name = "pytest-cov" +version = "7.1.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "coverage" }, + { name = "pluggy" }, + { name = "pytest" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/b1/51/a849f96e117386044471c8ec2bd6cfebacda285da9525c9106aeb28da671/pytest_cov-7.1.0.tar.gz", hash = "sha256:30674f2b5f6351aa09702a9c8c364f6a01c27aae0c1366ae8016160d1efc56b2", size = 55592, upload-time = "2026-03-21T20:11:16.284Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/9d/7a/d968e294073affff457b041c2be9868a40c1c71f4a35fcc1e45e5493067b/pytest_cov-7.1.0-py3-none-any.whl", hash = "sha256:a0461110b7865f9a271aa1b51e516c9a95de9d696734a2f71e3e78f46e1d4678", size = 22876, upload-time = "2026-03-21T20:11:14.438Z" }, +] + +[[package]] +name = "pyyaml" +version = "6.0.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/05/8e/961c0007c59b8dd7729d542c61a4d537767a59645b82a0b521206e1e25c2/pyyaml-6.0.3.tar.gz", hash = "sha256:d76623373421df22fb4cf8817020cbb7ef15c725b9d5e45f17e189bfc384190f", size = 130960, upload-time = "2025-09-25T21:33:16.546Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d1/33/422b98d2195232ca1826284a76852ad5a86fe23e31b009c9886b2d0fb8b2/pyyaml-6.0.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:7f047e29dcae44602496db43be01ad42fc6f1cc0d8cd6c83d342306c32270196", size = 182063, upload-time = "2025-09-25T21:32:11.445Z" }, + { url = "https://files.pythonhosted.org/packages/89/a0/6cf41a19a1f2f3feab0e9c0b74134aa2ce6849093d5517a0c550fe37a648/pyyaml-6.0.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:fc09d0aa354569bc501d4e787133afc08552722d3ab34836a80547331bb5d4a0", size = 173973, upload-time = "2025-09-25T21:32:12.492Z" }, + { url = "https://files.pythonhosted.org/packages/ed/23/7a778b6bd0b9a8039df8b1b1d80e2e2ad78aa04171592c8a5c43a56a6af4/pyyaml-6.0.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9149cad251584d5fb4981be1ecde53a1ca46c891a79788c0df828d2f166bda28", size = 775116, upload-time = "2025-09-25T21:32:13.652Z" }, + { url = "https://files.pythonhosted.org/packages/65/30/d7353c338e12baef4ecc1b09e877c1970bd3382789c159b4f89d6a70dc09/pyyaml-6.0.3-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:5fdec68f91a0c6739b380c83b951e2c72ac0197ace422360e6d5a959d8d97b2c", size = 844011, upload-time = "2025-09-25T21:32:15.21Z" }, + { url = "https://files.pythonhosted.org/packages/8b/9d/b3589d3877982d4f2329302ef98a8026e7f4443c765c46cfecc8858c6b4b/pyyaml-6.0.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ba1cc08a7ccde2d2ec775841541641e4548226580ab850948cbfda66a1befcdc", size = 807870, upload-time = "2025-09-25T21:32:16.431Z" }, + { url = "https://files.pythonhosted.org/packages/05/c0/b3be26a015601b822b97d9149ff8cb5ead58c66f981e04fedf4e762f4bd4/pyyaml-6.0.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:8dc52c23056b9ddd46818a57b78404882310fb473d63f17b07d5c40421e47f8e", size = 761089, upload-time = "2025-09-25T21:32:17.56Z" }, + { url = "https://files.pythonhosted.org/packages/be/8e/98435a21d1d4b46590d5459a22d88128103f8da4c2d4cb8f14f2a96504e1/pyyaml-6.0.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:41715c910c881bc081f1e8872880d3c650acf13dfa8214bad49ed4cede7c34ea", size = 790181, upload-time = "2025-09-25T21:32:18.834Z" }, + { url = "https://files.pythonhosted.org/packages/74/93/7baea19427dcfbe1e5a372d81473250b379f04b1bd3c4c5ff825e2327202/pyyaml-6.0.3-cp312-cp312-win32.whl", hash = "sha256:96b533f0e99f6579b3d4d4995707cf36df9100d67e0c8303a0c55b27b5f99bc5", size = 137658, upload-time = "2025-09-25T21:32:20.209Z" }, + { url = "https://files.pythonhosted.org/packages/86/bf/899e81e4cce32febab4fb42bb97dcdf66bc135272882d1987881a4b519e9/pyyaml-6.0.3-cp312-cp312-win_amd64.whl", hash = "sha256:5fcd34e47f6e0b794d17de1b4ff496c00986e1c83f7ab2fb8fcfe9616ff7477b", size = 154003, upload-time = "2025-09-25T21:32:21.167Z" }, + { url = "https://files.pythonhosted.org/packages/1a/08/67bd04656199bbb51dbed1439b7f27601dfb576fb864099c7ef0c3e55531/pyyaml-6.0.3-cp312-cp312-win_arm64.whl", hash = "sha256:64386e5e707d03a7e172c0701abfb7e10f0fb753ee1d773128192742712a98fd", size = 140344, upload-time = "2025-09-25T21:32:22.617Z" }, + { url = "https://files.pythonhosted.org/packages/d1/11/0fd08f8192109f7169db964b5707a2f1e8b745d4e239b784a5a1dd80d1db/pyyaml-6.0.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:8da9669d359f02c0b91ccc01cac4a67f16afec0dac22c2ad09f46bee0697eba8", size = 181669, upload-time = "2025-09-25T21:32:23.673Z" }, + { url = "https://files.pythonhosted.org/packages/b1/16/95309993f1d3748cd644e02e38b75d50cbc0d9561d21f390a76242ce073f/pyyaml-6.0.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:2283a07e2c21a2aa78d9c4442724ec1eb15f5e42a723b99cb3d822d48f5f7ad1", size = 173252, upload-time = "2025-09-25T21:32:25.149Z" }, + { url = "https://files.pythonhosted.org/packages/50/31/b20f376d3f810b9b2371e72ef5adb33879b25edb7a6d072cb7ca0c486398/pyyaml-6.0.3-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ee2922902c45ae8ccada2c5b501ab86c36525b883eff4255313a253a3160861c", size = 767081, upload-time = "2025-09-25T21:32:26.575Z" }, + { url = "https://files.pythonhosted.org/packages/49/1e/a55ca81e949270d5d4432fbbd19dfea5321eda7c41a849d443dc92fd1ff7/pyyaml-6.0.3-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a33284e20b78bd4a18c8c2282d549d10bc8408a2a7ff57653c0cf0b9be0afce5", size = 841159, upload-time = "2025-09-25T21:32:27.727Z" }, + { url = "https://files.pythonhosted.org/packages/74/27/e5b8f34d02d9995b80abcef563ea1f8b56d20134d8f4e5e81733b1feceb2/pyyaml-6.0.3-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0f29edc409a6392443abf94b9cf89ce99889a1dd5376d94316ae5145dfedd5d6", size = 801626, upload-time = "2025-09-25T21:32:28.878Z" }, + { url = "https://files.pythonhosted.org/packages/f9/11/ba845c23988798f40e52ba45f34849aa8a1f2d4af4b798588010792ebad6/pyyaml-6.0.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:f7057c9a337546edc7973c0d3ba84ddcdf0daa14533c2065749c9075001090e6", size = 753613, upload-time = "2025-09-25T21:32:30.178Z" }, + { url = "https://files.pythonhosted.org/packages/3d/e0/7966e1a7bfc0a45bf0a7fb6b98ea03fc9b8d84fa7f2229e9659680b69ee3/pyyaml-6.0.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:eda16858a3cab07b80edaf74336ece1f986ba330fdb8ee0d6c0d68fe82bc96be", size = 794115, upload-time = "2025-09-25T21:32:31.353Z" }, + { url = "https://files.pythonhosted.org/packages/de/94/980b50a6531b3019e45ddeada0626d45fa85cbe22300844a7983285bed3b/pyyaml-6.0.3-cp313-cp313-win32.whl", hash = "sha256:d0eae10f8159e8fdad514efdc92d74fd8d682c933a6dd088030f3834bc8e6b26", size = 137427, upload-time = "2025-09-25T21:32:32.58Z" }, + { url = "https://files.pythonhosted.org/packages/97/c9/39d5b874e8b28845e4ec2202b5da735d0199dbe5b8fb85f91398814a9a46/pyyaml-6.0.3-cp313-cp313-win_amd64.whl", hash = "sha256:79005a0d97d5ddabfeeea4cf676af11e647e41d81c9a7722a193022accdb6b7c", size = 154090, upload-time = "2025-09-25T21:32:33.659Z" }, + { url = "https://files.pythonhosted.org/packages/73/e8/2bdf3ca2090f68bb3d75b44da7bbc71843b19c9f2b9cb9b0f4ab7a5a4329/pyyaml-6.0.3-cp313-cp313-win_arm64.whl", hash = "sha256:5498cd1645aa724a7c71c8f378eb29ebe23da2fc0d7a08071d89469bf1d2defb", size = 140246, upload-time = "2025-09-25T21:32:34.663Z" }, + { url = "https://files.pythonhosted.org/packages/9d/8c/f4bd7f6465179953d3ac9bc44ac1a8a3e6122cf8ada906b4f96c60172d43/pyyaml-6.0.3-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:8d1fab6bb153a416f9aeb4b8763bc0f22a5586065f86f7664fc23339fc1c1fac", size = 181814, upload-time = "2025-09-25T21:32:35.712Z" }, + { url = "https://files.pythonhosted.org/packages/bd/9c/4d95bb87eb2063d20db7b60faa3840c1b18025517ae857371c4dd55a6b3a/pyyaml-6.0.3-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:34d5fcd24b8445fadc33f9cf348c1047101756fd760b4dacb5c3e99755703310", size = 173809, upload-time = "2025-09-25T21:32:36.789Z" }, + { url = "https://files.pythonhosted.org/packages/92/b5/47e807c2623074914e29dabd16cbbdd4bf5e9b2db9f8090fa64411fc5382/pyyaml-6.0.3-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:501a031947e3a9025ed4405a168e6ef5ae3126c59f90ce0cd6f2bfc477be31b7", size = 766454, upload-time = "2025-09-25T21:32:37.966Z" }, + { url = "https://files.pythonhosted.org/packages/02/9e/e5e9b168be58564121efb3de6859c452fccde0ab093d8438905899a3a483/pyyaml-6.0.3-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:b3bc83488de33889877a0f2543ade9f70c67d66d9ebb4ac959502e12de895788", size = 836355, upload-time = "2025-09-25T21:32:39.178Z" }, + { url = "https://files.pythonhosted.org/packages/88/f9/16491d7ed2a919954993e48aa941b200f38040928474c9e85ea9e64222c3/pyyaml-6.0.3-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c458b6d084f9b935061bc36216e8a69a7e293a2f1e68bf956dcd9e6cbcd143f5", size = 794175, upload-time = "2025-09-25T21:32:40.865Z" }, + { url = "https://files.pythonhosted.org/packages/dd/3f/5989debef34dc6397317802b527dbbafb2b4760878a53d4166579111411e/pyyaml-6.0.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:7c6610def4f163542a622a73fb39f534f8c101d690126992300bf3207eab9764", size = 755228, upload-time = "2025-09-25T21:32:42.084Z" }, + { url = "https://files.pythonhosted.org/packages/d7/ce/af88a49043cd2e265be63d083fc75b27b6ed062f5f9fd6cdc223ad62f03e/pyyaml-6.0.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:5190d403f121660ce8d1d2c1bb2ef1bd05b5f68533fc5c2ea899bd15f4399b35", size = 789194, upload-time = "2025-09-25T21:32:43.362Z" }, + { url = "https://files.pythonhosted.org/packages/23/20/bb6982b26a40bb43951265ba29d4c246ef0ff59c9fdcdf0ed04e0687de4d/pyyaml-6.0.3-cp314-cp314-win_amd64.whl", hash = "sha256:4a2e8cebe2ff6ab7d1050ecd59c25d4c8bd7e6f400f5f82b96557ac0abafd0ac", size = 156429, upload-time = "2025-09-25T21:32:57.844Z" }, + { url = "https://files.pythonhosted.org/packages/f4/f4/a4541072bb9422c8a883ab55255f918fa378ecf083f5b85e87fc2b4eda1b/pyyaml-6.0.3-cp314-cp314-win_arm64.whl", hash = "sha256:93dda82c9c22deb0a405ea4dc5f2d0cda384168e466364dec6255b293923b2f3", size = 143912, upload-time = "2025-09-25T21:32:59.247Z" }, + { url = "https://files.pythonhosted.org/packages/7c/f9/07dd09ae774e4616edf6cda684ee78f97777bdd15847253637a6f052a62f/pyyaml-6.0.3-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:02893d100e99e03eda1c8fd5c441d8c60103fd175728e23e431db1b589cf5ab3", size = 189108, upload-time = "2025-09-25T21:32:44.377Z" }, + { url = "https://files.pythonhosted.org/packages/4e/78/8d08c9fb7ce09ad8c38ad533c1191cf27f7ae1effe5bb9400a46d9437fcf/pyyaml-6.0.3-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:c1ff362665ae507275af2853520967820d9124984e0f7466736aea23d8611fba", size = 183641, upload-time = "2025-09-25T21:32:45.407Z" }, + { url = "https://files.pythonhosted.org/packages/7b/5b/3babb19104a46945cf816d047db2788bcaf8c94527a805610b0289a01c6b/pyyaml-6.0.3-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6adc77889b628398debc7b65c073bcb99c4a0237b248cacaf3fe8a557563ef6c", size = 831901, upload-time = "2025-09-25T21:32:48.83Z" }, + { url = "https://files.pythonhosted.org/packages/8b/cc/dff0684d8dc44da4d22a13f35f073d558c268780ce3c6ba1b87055bb0b87/pyyaml-6.0.3-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a80cb027f6b349846a3bf6d73b5e95e782175e52f22108cfa17876aaeff93702", size = 861132, upload-time = "2025-09-25T21:32:50.149Z" }, + { url = "https://files.pythonhosted.org/packages/b1/5e/f77dc6b9036943e285ba76b49e118d9ea929885becb0a29ba8a7c75e29fe/pyyaml-6.0.3-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:00c4bdeba853cc34e7dd471f16b4114f4162dc03e6b7afcc2128711f0eca823c", size = 839261, upload-time = "2025-09-25T21:32:51.808Z" }, + { url = "https://files.pythonhosted.org/packages/ce/88/a9db1376aa2a228197c58b37302f284b5617f56a5d959fd1763fb1675ce6/pyyaml-6.0.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:66e1674c3ef6f541c35191caae2d429b967b99e02040f5ba928632d9a7f0f065", size = 805272, upload-time = "2025-09-25T21:32:52.941Z" }, + { url = "https://files.pythonhosted.org/packages/da/92/1446574745d74df0c92e6aa4a7b0b3130706a4142b2d1a5869f2eaa423c6/pyyaml-6.0.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:16249ee61e95f858e83976573de0f5b2893b3677ba71c9dd36b9cf8be9ac6d65", size = 829923, upload-time = "2025-09-25T21:32:54.537Z" }, + { url = "https://files.pythonhosted.org/packages/f0/7a/1c7270340330e575b92f397352af856a8c06f230aa3e76f86b39d01b416a/pyyaml-6.0.3-cp314-cp314t-win_amd64.whl", hash = "sha256:4ad1906908f2f5ae4e5a8ddfce73c320c2a1429ec52eafd27138b7f1cbe341c9", size = 174062, upload-time = "2025-09-25T21:32:55.767Z" }, + { url = "https://files.pythonhosted.org/packages/f1/12/de94a39c2ef588c7e6455cfbe7343d3b2dc9d6b6b2f40c4c6565744c873d/pyyaml-6.0.3-cp314-cp314t-win_arm64.whl", hash = "sha256:ebc55a14a21cb14062aa4162f906cd962b28e2e9ea38f9b4391244cd8de4ae0b", size = 149341, upload-time = "2025-09-25T21:32:56.828Z" }, +] + +[[package]] +name = "respx" +version = "0.23.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "httpx" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/43/98/4e55c9c486404ec12373708d015ebce157966965a5ebe7f28ff2c784d41b/respx-0.23.1.tar.gz", hash = "sha256:242dcc6ce6b5b9bf621f5870c82a63997e8e82bc7c947f9ffe272b8f3dd5a780", size = 29243, upload-time = "2026-04-08T14:37:16.008Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1d/4a/221da6ca167db45693d8d26c7dc79ccfc978a440251bf6721c9aaf251ac0/respx-0.23.1-py2.py3-none-any.whl", hash = "sha256:b18004b029935384bccfa6d7d9d74b4ec9af73a081cc28600fffc0447f4b8c1a", size = 25557, upload-time = "2026-04-08T14:37:14.613Z" }, +] + +[[package]] +name = "ruff" +version = "0.15.22" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/3a/06/ae069393fc66e8ff33036d4b368003833bf6e88ccf182e17e7a2f1c754fd/ruff-0.15.22.tar.gz", hash = "sha256:3f15175b1fb580126f58285a5dae6b2ea89000136d980c64499211f116b54809", size = 4785063, upload-time = "2026-07-16T15:14:13.244Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/23/18/ee54b7ae1e121be7a28ea6da4b67564ebb0530e183a54415ab7e3bcd2c4e/ruff-0.15.22-py3-none-linux_armv6l.whl", hash = "sha256:44423e73493737f5e7c5b41d475483898ff37afcdae38bc3da5085e29af1c2d8", size = 10781258, upload-time = "2026-07-16T15:13:19.452Z" }, + { url = "https://files.pythonhosted.org/packages/2f/d2/2520cb14761ddbeaf57642a76942fc36adcbdbe53b4532241995f6fc485c/ruff-0.15.22-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:b82c6482946e9eda7ff2e091d25b8bad3f718684e1916d41bd56873cee05b697", size = 10999477, upload-time = "2026-07-16T15:13:23.318Z" }, + { url = "https://files.pythonhosted.org/packages/c9/10/74e53572aa758dfaa678c2a2646b5c5515d884b7ca56be4d2ce03ca4b560/ruff-0.15.22-py3-none-macosx_11_0_arm64.whl", hash = "sha256:11c1c715af53a09f714e011106bffc419751ec8232fcb5da42173284ea3fec6f", size = 10466716, upload-time = "2026-07-16T15:13:26.162Z" }, + { url = "https://files.pythonhosted.org/packages/1e/cc/44eaaf0844e028182f2d0a8f2190d0f359159aed0a9e5ab861d892f1ae2a/ruff-0.15.22-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:742a29cf29bddb7c8327895d6a10e0e6c5b38a96dd407af9b5d0857f809c0576", size = 10892644, upload-time = "2026-07-16T15:13:29.229Z" }, + { url = "https://files.pythonhosted.org/packages/9f/21/8edf559014d2b0f82beea19cfb713993ad802ccda16868769979c6090a84/ruff-0.15.22-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:72af58b951b0ae395935ae79763dc349bc0eb706319d28f7a33ad2cfb3cfc178", size = 10576719, upload-time = "2026-07-16T15:13:32.35Z" }, + { url = "https://files.pythonhosted.org/packages/bf/1e/3a13abd392a3b50b62e5938a831f9ab6e588358cacad5c18545b716d2182/ruff-0.15.22-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:62d425005c1835eb24e2ee4161cb90e8db263415f4a71c8c72c33abaa6c0c224", size = 11376494, upload-time = "2026-07-16T15:13:35.958Z" }, + { url = "https://files.pythonhosted.org/packages/bf/3e/422d3d95bcf04dd78e1aeac22184d4f9a8fb2c01865d39d44618484a0317/ruff-0.15.22-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:e8b9b3f8779a4f08c969defc3c8c35abffaa757e601ed5ae66d6d1db6519969a", size = 12208370, upload-time = "2026-07-16T15:13:39.185Z" }, + { url = "https://files.pythonhosted.org/packages/1e/91/5d065a0e0a02bf4813f5119ad278462eed081d2b832eb7c021ade0ec9e65/ruff-0.15.22-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:1e0dd1b2e4d3d585f897a0d137cbf4eaf6223bef4e8ce34d6bb12556c5f9249e", size = 11581098, upload-time = "2026-07-16T15:13:42.132Z" }, + { url = "https://files.pythonhosted.org/packages/f6/f9/a0d4871d12fae702eb1f41b686caf05f1f8b124dc6db6f784f53d74918fa/ruff-0.15.22-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:365523eb91d9224e1bcb03b022fbf0facb8f9e23792a2c53d9d4b3924bdbdebb", size = 11399422, upload-time = "2026-07-16T15:13:45.2Z" }, + { url = "https://files.pythonhosted.org/packages/18/80/c843a5176cddbceb0b7e8dd41cf9993490796c1c469348d384f5a5c13c56/ruff-0.15.22-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:fabfd168afdf29fee5be98b831efa9683c94d7c5a3b58b9ce5a2e38444589a74", size = 11381683, upload-time = "2026-07-16T15:13:48.46Z" }, + { url = "https://files.pythonhosted.org/packages/d4/00/8485de0ae92239438a36cfc51350db9b9e85c9ebdfaea91b18e422706662/ruff-0.15.22-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:225dbf095a87f1d9f90f5fd7924d2613ee452a75a4308c63a8f50f761787aa7c", size = 10850295, upload-time = "2026-07-16T15:13:51.655Z" }, + { url = "https://files.pythonhosted.org/packages/fa/91/24977ec2ec72eaf15e4394ace2959fdff2dd1e14f03e005e838023407169/ruff-0.15.22-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:1877d63b9d24ed278744f1523fd11b85540566d54641f97c566d7d9dc5ca5296", size = 10579640, upload-time = "2026-07-16T15:13:54.79Z" }, + { url = "https://files.pythonhosted.org/packages/9c/47/9b51216951974df1f263ac19da550d34252e0ed7218c25f10c5ef9ed7517/ruff-0.15.22-py3-none-musllinux_1_2_i686.whl", hash = "sha256:a1606c510bd7215680d32efab38965f7cdec3ef69f5170a3f4791404ffdd5262", size = 11105077, upload-time = "2026-07-16T15:13:57.915Z" }, + { url = "https://files.pythonhosted.org/packages/c2/47/20e9d4a3b8016778acea5fc32bb50d35d207500a17ddb529ffa6996feef8/ruff-0.15.22-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:630479b18625f5ffc373f77603a22a9f8ac0acd7ff0501178b5db28ec71e9c64", size = 11490980, upload-time = "2026-07-16T15:14:01.032Z" }, + { url = "https://files.pythonhosted.org/packages/4d/76/3f72d8fc38c1cb77b38c56a70da9d0c17700cc1cc50f9649c9d3c8f5ba71/ruff-0.15.22-py3-none-win32.whl", hash = "sha256:e5ba0e4a13fd14abbed2a77b517a3911290c6c6c59ef67784328d1668fab76cf", size = 10789165, upload-time = "2026-07-16T15:14:04.16Z" }, + { url = "https://files.pythonhosted.org/packages/cb/46/4965251734c2b6fcdca1b1b187d20bcac3af0ee5b083b89c910bb961ce3a/ruff-0.15.22-py3-none-win_amd64.whl", hash = "sha256:9be63ba1eb936acd2d1342fb8337c356353706fce233b2a15a09a97037e6acde", size = 11938297, upload-time = "2026-07-16T15:14:07.316Z" }, + { url = "https://files.pythonhosted.org/packages/57/c9/e69b1ff4c8b69093ef08b8919ab767af0569666865b39c30a8795d88d3c6/ruff-0.15.22-py3-none-win_arm64.whl", hash = "sha256:e1168075b72158510839f250027659cdd78476f40507dd517892304c41318661", size = 11298172, upload-time = "2026-07-16T15:14:10.51Z" }, +] + +[[package]] +name = "tomli-w" +version = "1.2.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/19/75/241269d1da26b624c0d5e110e8149093c759b7a286138f4efd61a60e75fe/tomli_w-1.2.0.tar.gz", hash = "sha256:2dd14fac5a47c27be9cd4c976af5a12d87fb1f0b4512f81d69cce3b35ae25021", size = 7184, upload-time = "2025-01-15T12:07:24.262Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c7/18/c86eb8e0202e32dd3df50d43d7ff9854f8e0603945ff398974c1d91ac1ef/tomli_w-1.2.0-py3-none-any.whl", hash = "sha256:188306098d013b691fcadc011abd66727d3c414c571bb01b1a174ba8c983cf90", size = 6675, upload-time = "2025-01-15T12:07:22.074Z" }, +] + +[[package]] +name = "types-pyyaml" +version = "6.0.12.20260518" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/b8/83/4a1afc3fbfcf5b8d46fc390cd95ed6b0dc9010a265f4e9f46314efffa37a/types_pyyaml-6.0.12.20260518.tar.gz", hash = "sha256:d917f83fb38462550338c1297faedd860b3ec83912b96b1e3d73255f7473e466", size = 17850, upload-time = "2026-05-18T06:01:58.675Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/06/a2/c01db32be2ae7d6a1689972f3c492b149ee4e164b12fdfd9f64b50888215/types_pyyaml-6.0.12.20260518-py3-none-any.whl", hash = "sha256:d2150f75a231c9fe9c7463bd29487d93e60bac90400287351384bc2284eba7cd", size = 20312, upload-time = "2026-05-18T06:01:57.368Z" }, +] + +[[package]] +name = "typing-extensions" +version = "4.16.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f6/cc/6253133b5bb138fc3306cebfbda2c520f545d36b5be2c7255cc528bb45d6/typing_extensions-4.16.0.tar.gz", hash = "sha256:dc983d19a509c94dba722ee6abd33940f7c05a89e243c47e907eb4db6f1a43e5", size = 113555, upload-time = "2026-07-02T08:40:05.92Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/49/d3/b8441a820a491ddfc024b0b0cf0393375b75ea13866d9c66727e54c2fc80/typing_extensions-4.16.0-py3-none-any.whl", hash = "sha256:481caa481374e813c1b176ada14e97f1f67a4539ce9cfeb3f350d78d6370c2e8", size = 45571, upload-time = "2026-07-02T08:40:04.659Z" }, +] + +[[package]] +name = "unstract-cli" +version = "0.1.0" +source = { editable = "." } +dependencies = [ + { name = "click" }, + { name = "httpx" }, + { name = "pyyaml" }, + { name = "tomli-w" }, +] + +[package.optional-dependencies] +dev = [ + { name = "mypy" }, + { name = "pytest" }, + { name = "pytest-cov" }, + { name = "respx" }, + { name = "ruff" }, + { name = "types-pyyaml" }, +] + +[package.metadata] +requires-dist = [ + { name = "click", specifier = ">=8.1,<9" }, + { name = "httpx", specifier = ">=0.27" }, + { name = "mypy", marker = "extra == 'dev'", specifier = ">=1.11" }, + { name = "pytest", marker = "extra == 'dev'", specifier = ">=8.0" }, + { name = "pytest-cov", marker = "extra == 'dev'", specifier = ">=5.0" }, + { name = "pyyaml", specifier = ">=6.0" }, + { name = "respx", marker = "extra == 'dev'", specifier = ">=0.21" }, + { name = "ruff", marker = "extra == 'dev'", specifier = ">=0.6" }, + { name = "tomli-w", specifier = ">=1.0" }, + { name = "types-pyyaml", marker = "extra == 'dev'" }, +] +provides-extras = ["dev"]