Release v0.5.0#11
Conversation
Both paths populate them: the hosted durable path reads the pair
GET /v1/runs/{id}/results now returns (unpacked from the runner's
tokens_usages.json artifact — pipelex-platform PR #82), and the blocking
fallback unpacks the same pair from the execute response's
extension-open pipe_output, so result.tokens_usages reads the same
regardless of which path ran. None against older platforms or
pre-artifact runs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`RunResults.tokens_usages` was a `list[dict[str, Any]]` bag. It now validates into `TokensUsageRecord`, a client-side mirror of the wire contract specified in docs/specs/pipelex-mthds-protocol.md — every field optional and `extra="allow"`, so pre-contract durable artifacts (relayed verbatim, never migrated) still parse with `cost`/`pipe_code` None and their legacy `job_metadata`/`unit_costs` kept in `model_extra`. Enum-ish fields stay plain `str` so runtime enum churn is non-breaking. Both paths feed the same typed records: the durable path off the results body, the blocking path lifted out of the execute response's extension-open `pipe_output`. Breaking: `RunResults.pipe_output` is now `DictPipeOutputAbstract | None` rather than `dict[str, Any] | None`. The blocking path already parsed the protocol model and then discarded the types via `.model_dump()`; it now carries the parsed model through, so consumers read the working memory as attributes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mirror the TokensUsageRecord wire contract on RunResults
…mple Review follow-ups raised on the TypeScript mirror of this page; the same three defects were present here, so both SDKs stay in sync. - Drop a dead link to the protocol spec. It pointed at a path that does not exist in the public runtime repo, and the spec's real home is an internal repo, so there is no public target to redirect to — reference the spec and its section by name instead. While there, correct the brand: the record is a Pipelex runtime concept, not part of the MTHDS standard, which says nothing about usage reporting. - Make the example meaningful. `mthds_contents=[...]` parses in Python but passes an `Ellipsis` where a bundle source is expected; a registered `pipe_code` with `inputs` is both valid and the more idiomatic hosted call. - Scope two guarantees that the page stated absolutely while documenting their exception further down. The spec qualifies both with "on a record emitted under this contract": pre-contract artifacts are relayed verbatim, so they do carry a raw `unit_costs` rate table and `job_metadata`. Name the exemption at both claims rather than only at the compatibility section. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Scope the contract-only usage claims and fix the run-usage example
Phase 0 of the hosted input-upload & SDK-preparation project: promote the approved contract into this repo's own docs before the code lands. New docs/input-preparation.md is the Python-flavored mirror of the JS SDK contract — upload_file/prepare_inputs, str/Path/bytes sources, signature-driven asset identification, upload-record guarantees, error/capability outcomes, inherited storage policy, and stability across the future endpoint move. Notes the build_inputs parity gap prepare_inputs closes. Pointer added from architecture.md's storage bullet. Docs-only; no code touched. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y9RPphjd62DywHtok2fX1q
…@pipelex/sdk)
Add the hosted upload-capability surface, the Python counterpart of @pipelex/sdk's
uploadFile/prepareInputs, plus the build_inputs route the signature step needs:
- client.upload_file(source, *, filename=None, content_type=None) -> UploadRecord —
accepts str/Path paths and raw bytes; record (uri, content_type, size, filename)
assembled client-side (mimetypes for MIME).
- client.prepare_inputs(*, files, pipe_ref=None, inputs) -> PreparedInputs — resolves
the pipe's declared signature via build_inputs(explicit=True), template-guided walk
(file signal = a canonical {url} content dict, mirroring the runtime's
input_normalizer), uploads file-bearing values, returns a copy-on-write rewrite
carrying pipelex-storage:// in `url` plus one UploadRecord per prepared asset.
http(s)/pipelex-storage:// pass through; data URLs and local/byte sources upload;
dedup by source identity; all failures before any run. An unrecognized value at a
file position fails as a typed InputPreparationError.
- client.build_inputs(BuildInputsRequest) -> BuildInputsResponse — closes the
/v1/build/* parity gap the Python SDK had. The response is an is_valid discriminated
union parsed through BuildInputsResponseAdapter (a TypeAdapter), mirroring the repo's
PipelexValidationResultAdapter precedent, so a malformed 200 raises a clean
ValidationError rather than being read as a valid verdict.
- Typed exception family mirrors the JS SDK: InputPreparationError base +
InvalidLocalSource / RejectedAsset / UnsupportedUploadCapability /
UploadAuthentication / UploadTransport.
Scope: prepare_inputs takes the closure as inline files; catalog method_id and opt-in
http(s) ingest deferred (additive). Matrix-derived parity tests + real-client wiring
tests. make agent-check + make agent-test green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y9RPphjd62DywHtok2fX1q
…feedback) Addresses the bot-review findings on PR #10 that are genuine correctness or doc-accuracy issues; deliberately defers the rest. Fixed: - Malformed base64 data URL no longer escapes the typed-error contract. Decode with `validate=True` (junk chars are rejected, not silently discarded into corrupted bytes) and map `binascii.Error` to `InputPreparationError`, so all preparation failures stay within the documented exception family. (Codex, Greptile, cubic all flagged this.) - Percent-encoded binary data URLs keep their exact bytes: decode with `urllib.parse.unquote_to_bytes` instead of `unquote(...).encode("utf-8")`, which corrupted any byte >= 0x80. - docs/input-preparation.md: the URL / storage-URI pass-through is a `prepare_inputs`-level behavior, not a feature of `upload_file` (which treats every string as a filesystem path); removed the `checksum` row that listed a field `UploadRecord` does not have. - docs/architecture.md: upload_file/prepare_inputs are now implemented, not a "planned addition". Deferred (not bugs): offloading the file read/base64 off the event loop (perf nicety on an inherently I/O-bound op), a template-required-per-format validator on BuildInputsValidReport (prepare_inputs already raises a clean typed error), and splitting a P3 two-scenario transport-error test. Regression tests: malformed base64 (bad padding + non-alphabet) raises a typed error and uploads nothing; percent-encoded binary round-trips its exact bytes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019cpRcAL7MEQcsx2hHvGxC2
… shape Second pass on PR #10 review-bot feedback — the two remaining substantive items, both promoted from "defer" to "fix now" after independent verification showed each closes a documented JS-parity gap in new code at near-zero cost. - upload_file: offload the synchronous whole-file read via asyncio.to_thread so a large read no longer blocks the event loop before the first await. Matches the JS SDK (which reads off-loop via node:fs/promises) and the repo's async-only invariant. base64 stays inline on purpose — CPython's binascii holds the GIL, so threading it would not free the loop (JS also encodes inline); `_to_asset_bytes`/`_read_path` stay sync. (Greptile P2 + cubic P2.) - BuildInputsValidReport: add a model_validator enforcing the template-present- per-format invariant (inputs for json, inputs_toml for toml, not the other). This makes the adapter's documented malformed-200 guarantee true for the template shape too — an `is_valid: true` body missing its template now raises a clean ValidationError at parse time instead of surfacing one layer down in prepare_inputs with a misleading "got json" message. Mirrors the JS sibling's per-format-required modeling. (cubic P2.) - Tests: assert the read is offloaded (to_thread awaited with _to_asset_bytes, real behavior preserved via wraps); assert the validator rejects the three malformed template shapes and accepts a well-formed toml report. Parametrized the two-scenario transport-error test for per-case attribution (cubic P3). make agent-check + make agent-test green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019cpRcAL7MEQcsx2hHvGxC2
feat: upload_file + prepare_inputs + build_inputs route (parity with @pipelex/sdk)
Greptile SummaryThis PR releases v0.5.0 with hosted input preparation and typed run usage. The main changes are:
Confidence Score: 4/5The upload rejection mapping needs a fix before merging; empty list templates can also bypass file preparation. Some valid asset rejections are exposed as transport failures. Empty list templates can leave raw file values in prepared inputs. The main build-input and run-usage paths otherwise preserve their documented shapes. pipelex_sdk/upload.py and pipelex_sdk/prepare_inputs.py
What T-Rex did
Important Files Changed
Prompt To Fix All With AIFix the following 2 code review issues. Work through them one at a time, proposing concise fixes.
---
### Issue 1 of 2
pipelex_sdk/upload.py:101-103
**Asset Rejections Become Transport Errors**
When the upload endpoint rejects an asset with a client-error status other than 401, 403, 404, or 413, this fallback raises `UploadTransportError`. A validation rejection such as 400 or 422 is therefore reported as a network or server failure instead of the documented `RejectedAssetError`, so callers cannot handle rejected assets by category.
### Issue 2 of 2
pipelex_sdk/prepare_inputs.py:155-160
**Empty List Template Skips Uploads**
If an explicit template represents a multiple file input with an empty list, the truthiness check skips the element walk. Caller values such as a list of raw bytes then remain unprepared, and the later run receives bytes instead of canonical `{"url": ...}` content.
Reviews (1): Last reviewed commit: "Release v0.5.0" | Re-trigger Greptile |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bba5074a29
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Triaged the unresolved SWE-bot comments on the v0.5.0 release PR against the JS SDK (@pipelex/sdk) and the pipelex runtime. Records the one confirmed-but-deferred bug (nested asset under a structured url field is not uploaded — parity-bound, needs an upstream template-contract change) plus the server-side 422-vs-413 upload seam, and dismisses the two false positives. No source changes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019cpRcAL7MEQcsx2hHvGxC2
Release v0.5.0
Bumps version from
0.4.0to0.5.0.Changelog
Added
Input preparation:
upload_fileandprepare_inputs(hosted upload capability). The Python counterpart of@pipelex/sdk'suploadFile/prepareInputs, in parity.client.upload_file(source, *, filename=None, content_type=None)uploads one local asset — a filesystem path (str/Path) or rawbytes— and returns anUploadRecord(uri,content_type,size,filename) assembled client-side.client.prepare_inputs(*, files, pipe_ref=None, inputs)resolves the target pipe's declared signature from the explicit inputs template, interprets the caller's compactinputstop-down against it (the file signal is the canonical Image/Document content shape — a{"url": …}dict — mirroring the runtime'sinput_normalizer), uploads the file-bearing values, and returnsPreparedInputs: a copy-on-write rewrite ofinputswith each asset reference replaced by canonical content carryingpipelex-storage://inurl, plus oneUploadRecordper prepared asset. HTTP(S) URLs and existingpipelex-storage://URIs pass through unchanged; data URLs and local/byte sources are uploaded; the same source referenced twice uploads once (dedup by source identity); all failures are raised before any run is created. Failures are typed per category:InvalidLocalSourceError,RejectedAssetError,UnsupportedUploadCapabilityError,UploadAuthenticationError,UploadTransportError(all extendInputPreparationError).prepare_inputstakes the method closure as inlinefiles; catalogmethod_idresolution and opt-inhttp(s)ingest are deferred and additive. Seedocs/input-preparation.md.build_inputsroute (POST /v1/build/inputs). Closes the/v1/build/*parity gap the Python SDK had —client.build_inputs(BuildInputsRequest)projects a pipe's declared inputs as a fill-in template, returning a 200 verdict discriminated onis_valid(BuildInputsValidReport|CrateInvalidReport); a no-verdict condition throwsApiResponseError. It is the signature sourceprepare_inputsreads (withexplicit=True). Models live inpipelex_sdk/build_models.py.Typed run usage:
RunResults.tokens_usages+RunResults.usage_assembly_error. The per-call usage records a run produces — token counts by category, the server-computedcostin USD, model name and id, the pipe that made the call, job-kind fields and timing, for LLM and img-gen/extract/search calls alike — are now first-class typed fields instead of ridingmodel_extra. Records validate into a newTokensUsageRecordmodel (pipelex_sdk/runs.py) mirroring the wire contract specified in the MTHDS protocol spec. Both paths populate the pair: the hosted durable path reads it offGET /v1/runs/{id}/results(which unpacks the runner'stokens_usages.jsonartifact), and the blocking fallback lifts the same pair out of the execute response's extension-openpipe_output— soresult.tokens_usagesreads the same regardless of which path ran.Note that the rate table (
unit_costs) no longer crosses the wire: a record now carries the computedcostfor the call instead, which isNonewhen the model has no rate table at all (own-GPU, mock, dry run) and0when a rate table priced it at zero. There is no run-level aggregate — sum the records.tokens_usagesisNonewhenever usage assembly produced no list (it was off, it broke, or the run was delivered before the artifact existed) and[]when assembly ran and no inference happened;usage_assembly_erroris the only field separating a broken assembly from an off one.TokensUsageRecordkeeps every field optional and is extension-open, so durable artifacts written before the contract shipped — relayed verbatim, never migrated — still parse:costandpipe_codecome backNone, and the legacyjob_metadata/unit_costssurvive inmodel_extra. Enum-ish fields (model_type,job_category,unit_job_id) are open sets typed as plainstr, so runtime enum churn stays non-breaking.Changed
RunResults.pipe_outputis nowDictPipeOutputAbstract | None, wasdict[str, Any] | None(breaking). The blocking path already parsed the protocol model and then threw the types away with a.model_dump()round-trip; it now carries the parsed model straight through. Read the working memory as attributes —result.pipe_output.working_memory.root["out"].content— rather than nested dict keys. The durable path still leaves itNone.🤖 Generated with Claude Code
Summary by cubic
Release v0.5.0 adds hosted input preparation (
upload_file,prepare_inputs), introducesbuild_inputsfor signature templates, and exposes typed run-usage records on results. It also changesRunResults.pipe_outputto a typed model. Adds a WIP doc with deferred review findings (no code changes).New Features
client.upload_file(source, filename?, content_type?)andclient.prepare_inputs(files, pipe_ref?, inputs)rewrite file inputs topipelex-storage://URLs, pass through http(s)/existing URIs, dedup repeated sources, and raise clear typed errors. Mirrors@pipelex/sdk.build_inputssupport:client.build_inputs(BuildInputsRequest)returns the explicit inputs template with anis_validverdict; used byprepare_inputs.RunResults.tokens_usagesandusage_assembly_errorvalidate intoTokensUsageRecordand are available on both hosted and blocking paths; each record includes per-call token counts and computed USDcost.Migration
RunResults.pipe_outputis nowDictPipeOutputAbstract | None(wasdict). Access working memory via attributes, e.g.result.pipe_output.working_memory.root["out"].content.Written for commit 4ef1d1e. Summary will update on new commits.