feat: Unstract CLI — unified interface for Document Studio, LLMWhisperer, and API Hub#1
Open
hari-kuriakose wants to merge 21 commits into
Open
feat: Unstract CLI — unified interface for Document Studio, LLMWhisperer, and API Hub#1hari-kuriakose wants to merge 21 commits into
hari-kuriakose wants to merge 21 commits into
Conversation
Generated command tree from declarative Endpoint records covering LLMWhisperer, deployments, Platform v1, HITL and API Hub. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Completes Phase 6: update-unstract-cli Skill with docs cross-referencing, zero-drift regression tests, GitHub Actions CI, shell completions. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… files param parse_mdx_doc scanned all <ApiEndpoint> props, so responseBody fields were read as request parameters (358 false positives across platform v1). Now reads only pathParams/queryParams/requestBody. This surfaced a real bug: workflow.execute used 'file' where the API expects 'files'. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Credentials intentionally have no CLI flag (shell history/process listings), and the block name in the hint now matches what config init writes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Unfiltered full output was ~63k tokens, too large for an agent to read speculatively. Selection (--group/--command) and verbosity (--detail) are independent axes; default is now names+summaries (~4.5k tokens). Also: compact JSON when piped, global boilerplate omitted from filtered views, exit 2 with clean stdout when no command matches. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Keeps --dump-commands as a hidden deprecated alias that warns on stderr, since it is referenced in the Skill, README and CI. Internal function dump_commands() renamed to discover(). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
--discover is now the only spelling. A test pins the removal so the old flag cannot creep back via a copied example. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…t.toml Config selection was previously env-var-only. Adds a --config flag and upward discovery of .unstract.toml (stopping at $HOME), composing with the existing profile mechanism. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Table mode capped cells at 60 chars and appended '...', silently hiding the tail of notes, URLs and ids. Cells now wrap across lines within the terminal width, so the human format is lossless like json/yaml/raw. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The config commands were described by a hand-maintained parallel data structure (CONFIG_COMMANDS/HandAuthoredCommand) instead of being introspected, so the index claimed --product/--key/--value for 'config set', whose parameters are positional. An agent following the index would build a command line the parser rejects. Now introspected from the real Click commands, like every generated command. Each entry gains kind (argument|option) and a usage line. The duplicate description model is deleted so it cannot drift again, and a test cross-checks every advertised parameter against the real parser. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Unstract is the company and the CLI name. It builds three products: Document Studio (formerly named Unstract), LLMWhisperer and API Hub. - Product enum now has exactly three members; a new ApiGroup enum carries the five API surfaces, with the product derived from it. - platform/deployment/hitl nest under 'unstract docstudio ...'. - Config blocks nest by product: [profiles.X.docstudio.platform]. - --discover exposes product, product_name and api_group per command. Wire paths and UNSTRACT_* env vars are unchanged: the product was renamed, not the API. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
config get/set now accept only docstudio.platform (or 'docstudio platform'), docstudio.deployment, docstudio.hitl, llmwhisperer, apihub. Bare group names and the 'whisper' alias are rejected with exit 2 and a hint listing the valid targets, so a setting always says which product it configures. Also: config current keys its output by the same target names, and the variadic arguments declare metavars so --help and --discover both show 'TARGET... KEY VALUE' rather than an opaque 'ARGS...'. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Profile blocks now have exactly one accepted layout: the API group nested under its product. A block written any other way is ignored rather than half-applied, so a config that looks applied but is not cannot mislead. Fixes a latent bug found while removing them: ResolvedConfig.get/require normalised only Product, so an ApiGroup key fell through unconverted -- profile values were silently missed and errors read 'ApiGroup.LLMWHISPERER'. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The repo tracked 50 `__pycache__/*.pyc` files, so every commit that ran the test suite carried dozens of modified binaries and buried the real diff. They are build output: regenerated on any run, machine- and interpreter-specific (both cpython-312 and -313 copies were committed), and never useful to a reader. `git rm --cached` drops them from the index while leaving them on disk, so nothing needs rebuilding locally. Also adds a .gitignore, which the repo had been missing entirely -- covering caches, build output, virtualenvs, and `.unstract.toml`, which is a local config file that can hold credentials.
Work in progress that predates the gotcha-fix pass; committing it as its own change so the fixes that follow are reviewable in isolation. - `config doctor` reports where each setting resolves from (flag, env var, profile literal, or profile `env:` reference), which answers the question that costs the most time: "the CLI says the key is missing, but I set it -- where is it looking?" Backed by `ResolvedConfig.resolution_source`, which reports the winning source without echoing the secret. - `--discover` gains the navigable-groups overview: a ~1k-token map of the command tree with per-group counts and the exact `drill` command for each subtree, so an agent reads the map and follows one drill rather than pulling all 156 commands into context. - `PollSpec.status_field` accepts a tuple of candidate field names, so a poll can match a status endpoint whose body spells the state differently from the run response. - Drops the unused `typer` dependency and documents why each remaining one is there. `uv.lock` is committed so installs are reproducible and hash-pinned.
The CLI runs Click with `standalone_mode=False` so that Click's own parse errors reach us and can be rendered through the structured error envelope (SPEC §5.5). That mode also changes how `ctx.exit(code)` behaves: instead of raising SystemExit, Click *returns* the code from the invocation. `main()` ignored the return value and unconditionally returned 0. So every documented exit code -- 2 usage, 3 auth, 4 not-found, 7 timeout, 9 already-consumed (SPEC §5.4) -- was reported to the shell as success. A validation error printed `"exit_code": 2` inside its own error envelope while the process exited 0. `set -e` never tripped, `cmd || handle` never fired, and an agent branching on the status code saw every failure as a pass. This defeats the exit-code contract the README advertises as a core promise for agent use. Return the invocation's result when it is an int (a command that falls off the end returns None, which is success). Tested by subprocess, which is the only way to catch this: Click's `CliRunner` invokes with `standalone_mode=True` and never runs `main()`, so the existing suite asserted the right contract against a code path the real binary does not take -- 213 tests passed throughout.
Fixes for the issues hit while driving a document through the CLI end to
end (extract, Prompt Studio, export tool, deploy API, run). Root causes
were traced in the backend source rather than inferred, which changed the
fix in two cases and ruled one out entirely.
`prompt create --profile-manager`. `fetch_response` resolves a prompt's
LLM profile from the prompt's own `profile_manager` field and never falls
back to the project default -- unlike `index_document` and `single_pass`,
which both call `get_default_llm_profile(tool)`. A prompt created without
one is unrunnable, and fails with "Default LLM profile is not configured",
which names the *project* default -- genuinely set, and never read. The
flag makes prompts runnable from birth; the 500 now carries a hint naming
the real fix instead of sending the reader back to `profile set-default`.
`prompt-studio --challenge-llm` / `--monitor-llm`. A deployed run ended in
`ERROR` with a 422 at the very last step. The exported tool's metadata
carries `challenge_llm: ""`, which fails schema validation: the property
is `adapterType: LLM`, so the schema gains an `enum` of real adapter ids
and "" is not among them. (Not the `required` rule, as it first appears --
a present-but-empty key satisfies `required`; verified by running the
validator.) Export already resolves a valid value, so setting it on the
project before `export-tool` keeps the metadata non-empty.
`Param.mirror_as` for the API-key routes. `api-deployment key create`
needed both `--api-id` (URL) and `--api` (body) -- the same value twice.
The path value is now mirrored into the body under its own name, so
`--api-id` alone works. Same fix for the `pipeline key create` sibling.
Both records carry `doc_conflict` so the docs-diff skill does not restore
the flags; the diff also now recognises a mirrored param as present.
`default_from` fallback chains. `deployment` and `hitl` `org_id` fall back
to `platform.org_id` -- the same organization, in a block that starts
empty. Credentials deliberately do not chain: reusing a key across API
groups would be credential confusion, not convenience.
`--wait` on `index-document`, which returns the same `{task_id, status}`
shape as its siblings and was the only async command without it. Indexing
writes no Output Manager row, so the terminal status is the result and
there is no retrieve step.
Not fixed, deliberately: making `--vector-store`/`--embedding-model`
optional at `--chunk-size 0`. Both columns are NOT NULL server-side and
the serializer derives required=True, so relaxing the CLI would trade a
fast local exit-2 for a slower remote 400 -- moving the friction, not
removing it. `RequiredUnless` is kept as a tested primitive for a case
where the server genuinely accepts the omission. Help text now explains
that the values are stored but never queried.
Remaining items are server-side and cannot be fixed here; their help text
now describes the real behaviour instead of implying a workaround exists:
the registry's missing back-reference to the Prompt Studio tool, the
absent DELETE route, `adapter-choices` (routed to a method that does not
exist anywhere in the backend, so it 500s on every call), and the empty
default-triad response meaning "unset" rather than an error.
Additive reconciliation of SPEC, README and the update-unstract-cli Skill against the fixes in the previous commit. Nothing here contradicts what was already written -- the gaps were mechanisms the docs did not yet mention. SKILL.md matters most: it is what a future skill run reads to learn which encodings exist, so a gap causes real errors later. It now covers `mirror_as`, the `default_from` fallback chain, and `RequiredUnless` (listed as available but explicitly *unused*, with the reason, so it is not reached for where the server enforces the field anyway). The `doc_conflict` guidance gains the key-create case as a second worked example beside whisper-detail, so the deliberately dropped `api` and `pipeline` flags are not "restored" on a later sync. SPEC §7.1 generalises the mirroring sentence and adds the fallback chain and the conditional constraint; §8.3/§8.5 record that `doc_conflict` now suppresses param drift too, and that a mirrored parameter is not drift. README gains one line: the org_id fallback, in the resolution-order section. Per-endpoint flags stay out of it -- below its altitude. The exit-code fix needs no doc change: SPEC §5.4 and the README table always specified the correct behaviour, and the fix aligned the implementation to them. A spec describes the contract, not its regressions.
The build-sequence plan has served its purpose now that the CLI is
implemented; SPEC.md remains the living specification. Removing it leaves
six dangling citations, all fixed here so nothing points at a file that
no longer exists:
- README dropped the link to it from the closing "see also" line.
- pyproject, model.py (x3) and output.py cited section labels (R3, §1,
§2, M1.6) that existed only in the plan. The surrounding prose stands
on its own, so the parentheticals are simply removed -- the P1-P12
pattern framework they referenced is defined inline in model.py's own
docstring, so nothing is lost.
- test_model.py's module docstring opened by quoting the plan ("§2 calls
this the load-bearing design work"); reworded to state the point
directly.
Verified: `grep -rn IMPLEMENTATION_PLAN` returns nothing. Tests, ruff and
mypy all green.
A human-runnable end-to-end scenario for the `unstract` CLI: extract from an invoice, build a Prompt Studio project, export it as a tool, deploy an API, and run it -- with setup (temp config) and teardown (delete created resources) steps and a summary-report checklist. This is the same flow whose friction was captured in the recent gotcha-fix pass.
The install step ran `uv pip install --system -e ".[dev]"`, which targets the runner's `/usr` interpreter. `setup-uv` with `python-version` also provisions a project venv and exports `VIRTUAL_ENV`, and that interpreter is externally managed (PEP 668), so `--system` aborts with "externally managed environment" before any test runs. Switch to the project workflow the README documents for CI: `uv sync --frozen --extra dev` installs the exact hash-pinned tree from uv.lock into the venv (and fails if the lock is stale), and every step runs through `uv run`. Reproduced green locally: sync, ruff, mypy, pytest (245), and all three `--discover` index checks.
hari-kuriakose
requested review from
chandrasekharan-zipstack,
johnyrahul,
muhammad-ali-e,
ritwik-g and
vishnuszipstack
July 24, 2026 09:04
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Establishes the
unstractCLI — one interface over Unstract's three products (Document Studio, LLMWhisperer, API Hub), built for agent use: declarative endpoint records, machine-readable--discover, stable exit codes, structured errors, and no interactive prompts.mainis an empty root commit, so this PR's diff is the entire codebase.Architecture
Every command is generated from a declarative
Endpointrecord insrc/unstract_cli/endpoints/. The command tree, flags, help text, validation, and--discoverall derive from those records, so help text cannot drift from behaviour and the bundled Claude Skill has exactly one place to edit.Highlights
.unstract.toml,--config; resolution order flag → env → profile → default;config doctorreports where each setting resolves from without echoing secrets.--discover— a ~1k-token group map an agent drills into, rather than pulling all 156 commands into context.--wait— declarative execute → poll → retrieve, deciding terminal state from the response body (not the HTTP code, which the deployment API returns as 422 mid-run).--savepersists them atomically before exit; a second read exits 9.Fixes from a full end-to-end run
Driving a document through extract → Prompt Studio → export → deploy → run surfaced real friction; root causes were traced in the backend source, which changed two fixes and ruled one out:
standalone_mode=Falsemakes Click return the exit code, andmain()discarded it, so every failure exited 0 while its envelope reported the real code. Highest-impact fix; caught only by subprocess tests, sinceCliRunnernever runsmain().prompt create --profile-manager—fetch_responsereads the prompt's own profile and never falls back to the project default (unlike its siblings), failing with a message that names a setting it never reads.prompt-studio --challenge-llm— an emptychallenge_llmfails anenumcheck at deploy time; setting it before export keeps the metadata valid.Param.mirror_as— the key-create routes needed the same id in both path and body; mirroring lets--api-idalone suffice.org_idfallback chains and--waitonindex-document.Verification
245 tests pass;
ruffandmypyclean;--discoveremits valid JSON (156 commands).🤖 Generated with Claude Code