feat(tangle-cli): add pipeline compile command and PipelineCompiler#1
Open
Silin144 wants to merge 23 commits into
Open
feat(tangle-cli): add pipeline compile command and PipelineCompiler#1Silin144 wants to merge 23 commits into
Silin144 wants to merge 23 commits into
Conversation
Assisted-By: devx/12bfa114-4ee7-4929-a37c-98018f6b0f41
…attempts-0.1.1 Configure submit recovery attempts
…uler-retries-fix/tangle-cli-release-workflow-inde-f259a7c Allow tangle-cli releases with independent tangle-api version
…uler-retries-fix/release-publish-cli-only Publish only tangle-cli artifacts in release workflow
…uler-retries-fix/release-conditional-publish Publish release artifacts by package version
…uler-retries-fix/release-package-scoped-tags Use package-scoped release tags
Silin144
force-pushed
the
feat/pipeline-compile
branch
2 times, most recently
from
July 9, 2026 21:05
00dc4ea to
a905d75
Compare
Fix generated OpenAPI client request-body handling so mixed simple and complex/object bodies remain lossless, and bump tangle-cli/tangle-api patch versions.
Silin144
force-pushed
the
feat/pipeline-compile
branch
from
July 10, 2026 18:41
a905d75 to
02289db
Compare
feat(tangle-cli): add python_pipeline authoring DSL
Silin144
force-pushed
the
feat/pipeline-compile
branch
from
July 14, 2026 16:23
02289db to
621170b
Compare
Silin144
added a commit
that referenced
this pull request
Jul 14, 2026
…er it Second independent review of TangleML#28 found a blocking correctness bug in the propagate_config broadcast subsystem, plus its total lack of test coverage. Both fixed here, with two doc nits. 1. BLOCKING — broadcast hard-failed through a config-less intermediate. In `_process_subpipeline_children`, an active broadcast (non-empty `ctx.broadcast_stack`) unconditionally called `_read_raw_cfg(child_cfg_path)` for every child edge, and that helper raises `CompileError: config file not found` on a missing file. A pure composing subpipeline (no `cfg` param, no `config=`) has no config.yaml on disk, so the canonical shape `root(propagate_config=True) -> config-less parent -> config-taking child` failed with a spurious "config file not found" for a file the author never wrote. Fix: in the broadcast path, treat a MISSING child config as `{}` (the child declares nothing to overlay; broadcast keys flow PAST it to descendants). A config-DECLARING child whose file is genuinely missing is still caught later, with guidance, in `_load_cfg_and_raw` (verified: the `cfg`-param error message is preserved). 2. Coverage: the entire broadcast subsystem had zero tests (`grep propagate_config tests/` was empty — which is why #1 shipped). Added `tests/fixtures/python_pipeline/broadcast_zone/` (a 3-level tree isolated in its own dir so the config-less intermediate genuinely has no config.yaml — the shared fixtures dir has one that would mask the bug) + a regression test asserting the compile succeeds and the root's broadcast value reaches the grandchild leaf, winning over the grandchild's own config (three distinct values make the source unambiguous). 3. Nit: the "content-hash" wording in a subpipeline test comment is really a compile-key hash (source path + qualname + name + config path + overrides fingerprint), not a hash of emitted content — corrected. 4. Nit: genericized a docstring example name (`options_standardization` / "Options Standardization") that echoed an internal pipeline -> `train_model` / "Train Model". 746 passed (+1); ruff clean on touched files; `uv lock --check` and `git diff --check` clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Run the full pipelines validate authoring validator during pipeline-runs submit and --dry-run after hydration, and route PipelineRunner through the same base hook. Root graph requirements are enforced by the authoring validator, with README and tests updated. Bump tangle-cli patch version to 0.1.3. This is based on master and intentionally excludes PR TangleML#27 skip-validation CLI flag and separate submit-root check. Assisted-By: devx/5490f7cb-c7e4-4cf5-b525-a8c4e2af55b1
Assisted-By: devx/5490f7cb-c7e4-4cf5-b525-a8c4e2af55b1
…ter-pipeline-runner-authoring-3e10657 Validate pipeline specs during submit
Move graph-shape, vendored JSON schema, and component-input validation into tangle-cli so pipeline validation, pipeline-run submit/dry-run, and PipelineRunner share one validation surface. Deduplicate validation so each path validates once, add the vendored schema/refresh helper and jsonschema dependency, preserve tangle_cli.pipelines re-exports for back-compat, and bump tangle-cli to 0.1.4. Assisted-By: devx/5490f7cb-c7e4-4cf5-b525-a8c4e2af55b1
…onsolidate-pipeline-validation-0f07e91 feat: consolidate pipeline validation
Port the Python-pipeline compiler from the internal tangle-deploy package into the OSS CLI so the generic compile logic lives in OSS: - pipeline_compiler.py: PipelineCompiler(TangleCliHandler) executor plus the generic compile_pipeline() free function. The zone-root seam (ZONE_ROOT_MARKERS) is left empty here for downstream injection. - schema_validation.py + schemas/dehydrated_pipeline_schema.json: Draft 2020-12 structural validation, semantic checks, and the no-template-delimiter output contract. - pipelines.compile_pipeline_file facade and a cyclopts `compile` subcommand under `tangle sdk pipelines`, following the hydrate facade. - jsonschema>=4.0.0 runtime dependency (uv.lock updated). - Tests for the compiler, CLI command, and schema validation, with colocated python_pipeline fixtures. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ead code Follow-up to the PipelineCompiler port, addressing an independent review of the PR: - Add compile tests for the headline features the command is named after: subpipeline child-sidecar emission (parent componentRef rewrite, no subpipeline://pending leak), .override_config precedence on a subpipeline edge, and @registered gen_config resolution (omitted -> nearest ancestor; relative rejected when ZONE_ROOT_MARKERS is empty; relative resolved once a marker is appended to the seam). New hermetic fixtures under tests/fixtures/python_pipeline/. - Remove the dead _parse_overrides from pipeline_compiler.py; the live copy lives in pipelines_cli.py and is covered by the CLI bad-override test. - Strip remaining internal-tool references from user-facing surfaces: the config/output-collision CompileError message, the _fragment_for_task docstring, and the _coerce_override docstring in cfg.py. - Widen the DSL no-internal-references guard to scan the sibling OSS compile modules (component_from_func, pipeline_compiler, schema_validation, pipelines, pipelines_cli), not just the DSL package. Full suite: 745 passed. uv lock --check and git diff --check clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…surface Second independent review of TangleML#28 surfaced two verified findings; fixing both here per request: 1. Remove the dead `_resolve_cfg_path(pipeline_fn, module_path)` wrapper in pipeline_compiler.py — a thin, zero-caller shim over `_resolve_cfg_path_in_dir` (the actual resolver, retained). 2. Scrub internal product/infra names from OSS docstrings + examples so the open-source surface never names Oasis / UPI / Comet / areas-ml image paths: - task_env.py: example env `UPI`/`areas-ml-upi:main`/`publish_to_comet`/ `[UPI Clustering]`/"Oasis runner" -> generic `TRAINING`/`python:3.12`/ `train_model`/"downstream runner". - registered.py: relative-path resolution doc referenced the internal `oasis.pipeline_component_root.yaml` marker -> now points at the generic `ZONE_ROOT_MARKERS` seam and states the OSS build registers none by default. - component_from_func.py: 31 example `UPI` env-binding identifiers in the authoring-strip docstrings -> `TRAINING`. - trace.py: CompileError example `return publish_to_comet` -> `return train_model`. Widen `TestNoInternalReferences` to enforce the scrub and prevent regression: alongside the existing tangle_deploy/tangle-deploy substring check it now flags `areas-ml`/`areas/ml` substrings and the standalone words oasis/upi/comet (word-boundary, case-insensitive — so it won't false-positive on "deduping"). 745 passed; `uv lock --check` and `git diff --check` clean; no new ruff findings in touched files. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…er it Second independent review of TangleML#28 found a blocking correctness bug in the propagate_config broadcast subsystem, plus its total lack of test coverage. Both fixed here, with two doc nits. 1. BLOCKING — broadcast hard-failed through a config-less intermediate. In `_process_subpipeline_children`, an active broadcast (non-empty `ctx.broadcast_stack`) unconditionally called `_read_raw_cfg(child_cfg_path)` for every child edge, and that helper raises `CompileError: config file not found` on a missing file. A pure composing subpipeline (no `cfg` param, no `config=`) has no config.yaml on disk, so the canonical shape `root(propagate_config=True) -> config-less parent -> config-taking child` failed with a spurious "config file not found" for a file the author never wrote. Fix: in the broadcast path, treat a MISSING child config as `{}` (the child declares nothing to overlay; broadcast keys flow PAST it to descendants). A config-DECLARING child whose file is genuinely missing is still caught later, with guidance, in `_load_cfg_and_raw` (verified: the `cfg`-param error message is preserved). 2. Coverage: the entire broadcast subsystem had zero tests (`grep propagate_config tests/` was empty — which is why #1 shipped). Added `tests/fixtures/python_pipeline/broadcast_zone/` (a 3-level tree isolated in its own dir so the config-less intermediate genuinely has no config.yaml — the shared fixtures dir has one that would mask the bug) + a regression test asserting the compile succeeds and the root's broadcast value reaches the grandchild leaf, winning over the grandchild's own config (three distinct values make the source unambiguous). 3. Nit: the "content-hash" wording in a subpipeline test comment is really a compile-key hash (source path + qualname + name + config path + overrides fingerprint), not a hash of emitted content — corrected. 4. Nit: genericized a docstring example name (`options_standardization` / "Options Standardization") that echoed an internal pipeline -> `train_model` / "Train Model". 746 passed (+1); ruff clean on touched files; `uv lock --check` and `git diff --check` clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…om docstring, extract cfg-load helper Reviewer follow-ups on the PR: * pipeline_compiler.py — replace ~36 internal design-doc citations (PROPAGATE_CONFIG_DESIGN §N, Decision A–M, NM1/NM2, Fix B, Finding 2, P2/D4/Phase N) with the rule stated inline, so every comment stands on its own in the public repo with no dangling pointer to a doc that isn't published. * schema_validation.py — delete the module-docstring sentence promising a `PipelineHydrator.validate_dehydrated_file` convenience that does not exist. * pipeline_compiler.py — extract the step-1 cfg resolve/load block (plus its warning/override-error branches) out of `_compile_pipeline_fn` into `_resolve_cfg_for_pipeline`, leaving the compile function a thin orchestrator whose remaining steps already delegate to named passes. Comment/docstring-only plus pure code-motion; behavior unchanged (746 tests pass, golden byte-identical fixtures included). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… child Under propagate_config, a broadcast key a child does not declare flows past it as ambient pass-through context, which the parent folds into the child's cycle-check/registry key. The child then recomputed its OWN key (ambient-less) to push onto active_stack, so `child_key in active_stack` never matched: a real cycle escaped precise detection and degraded to the max-depth-32 guard with a misleading "Reduce nesting depth" message. Thread the parent-computed key into _compile_pipeline_fn as `precomputed_key` so the stacked, registry, and cycle-check keys are identical; the root still builds its own (ambient-less) key. Adds the suite's first cycle-detection tests: a direct self-reference and the propagate_config pass-through regression. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Silin144
force-pushed
the
feat/pipeline-compile
branch
from
July 15, 2026 20:13
d1c63c9 to
53acb14
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ports the Python-pipeline compiler from the internal
tangle-deploypackage into the OSS CLI, so the generic compile logic lives in OSS. Follows the existingPipelineHydrator(TangleCliHandler)architecture and the hydrate facade pattern rather than inventing new conventions.Stacked on TangleML#20 (the
python_pipelineauthoring DSL) — this branch imports that package. See the stacking note below.What's included
pipeline_compiler.py—PipelineCompiler(TangleCliHandler)executor plus the genericcompile_pipeline()free function. The zone-root discovery seam is a module-levelZONE_ROOT_MARKERS: list[str] = [], left empty in OSS for downstream injection (e.g.tangle-deployappends its oasis marker).schema_validation.py+ bundledschemas/dehydrated_pipeline_schema.json— JSON-Schema (Draft 2020-12) structural validation, semantic checks (danglingtaskOutput.taskId, undeclaredgraphInput.inputName,outputValues↔outputs, purecomponentRef), and the no-template-delimiter output contract withexempt_paths.pipelines.compile_pipeline_filefunctional facade + a cycloptscompilesubcommand undertangle sdk pipelines, mirroringhydrate.jsonschema>=4.0.0runtime dependency (uv.lockupdated).python_pipelinefixtures.Worked example
Testing
All three CI checks pass locally:
uv run pytest— 731 passed, 0 regressions (97 in the new/touched files)uv lock --check— clean (exit 0)git diff --check— no whitespace errorsStacking note
feat/python-pipeline-dslbranch so the diff shows Phase-2-only changes. Once TangleML#20 merges intomaster, this branch will be rebased ontomasterand retargeted atTangleML/tangle-cli:master.🤖 Generated with Claude Code