Skip to content

Add --arg-secret to pipeline-runs submit for secret-backed inputs#44

Open
arseniy-pplx wants to merge 1 commit into
TangleML:masterfrom
arseniy-pplx:transfer/submit-arg-secret
Open

Add --arg-secret to pipeline-runs submit for secret-backed inputs#44
arseniy-pplx wants to merge 1 commit into
TangleML:masterfrom
arseniy-pplx:transfer/submit-arg-secret

Conversation

@arseniy-pplx

Copy link
Copy Markdown

Summary

Adds a --arg-secret INPUT=SECRET_NAME option to sdk pipeline-runs submit so a pipeline input can be bound to a Tangle secret instead of a plain value. Each reference encodes the standard dynamic-data payload under the input in root_task.arguments:

{"dynamicData": {"secret": {"name": "SECRET_NAME"}}}

The flag is repeatable, and a matching arg_secrets mapping is supported in the submit config file so secret bindings can be expressed the same way as the existing args mapping.

Context

Pipelines frequently need credentials (API keys, DB passwords) that should be resolved from stored secrets rather than passed as literals on the command line or checked into config. Today submit only accepts plain values via --arg, --args-json, and config args. This change lets callers reference a secret by name; the backend resolves it at run time via the existing dynamic-data secret contract. No backend or secret-storage behavior changes — this only teaches the CLI to emit the reference payload the backend already understands.

Behavior details:

  • Validation before I/O. Inputs and secret names must be non-empty after trimming; malformed entries, duplicate inputs, and unsupported config shapes are rejected before any pipeline file is read or any network call is made.
  • Deterministic conflict handling. An input supplied both as a plain value (--arg / --args-json / config args) and as a secret reference (--arg-secret / config arg_secrets) is rejected with an actionable error rather than silently overwritten.
  • Precedence. CLI --arg-secret overrides config arg_secrets, matching the existing CLI-over-config convention.
  • Unchanged paths. --arg, --args-json, config args, hydration/trust, dry-run, and submit-recovery semantics are untouched. The secret payload passes through payload sanitization unchanged.

A secret input must be a declared pipeline input, identical to --arg.

Testing

  • uv run pytest tests/test_pipeline_runs_cli.py
  • new tests cover the exact dynamic-data payload shape, single and multiple secret references, trimming, malformed/duplicate rejection, config mapping parsing and type validation, value-vs-secret conflict rejection, CLI-over-config precedence, dry-run, and that validation errors short-circuit before any file read or network call; payload-contract tests pin the dynamicData.secret.name shape against an in-memory fake client
  • uvx ruff check on the touched files
  • uv lock --check
  • git diff --check
  • uv run tangle sdk pipeline-runs submit --help shows --arg-secret

Support binding a pipeline input to a Tangle secret via
`--arg-secret INPUT=SECRET_NAME` (repeatable) and a matching
`arg_secrets` config mapping. Each reference encodes the OSS
dynamic-data payload {"dynamicData":{"secret":{"name":...}}} under
the input in root_task arguments.

Validation runs before any file read or network call: inputs and
secret names must be non-empty after trimming, duplicates are
rejected, and an input supplied both as a plain value and a secret
reference is rejected rather than silently overwritten. CLI
`--arg-secret` overrides config `arg_secrets`. Existing --arg,
--args-json, config args, hydration, dry-run, and submit-recovery
behavior are unchanged.
@arseniy-pplx
arseniy-pplx marked this pull request as ready for review July 20, 2026 18:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant