Skip to content

feat: add portable WDL module skill #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
claymcleod merged 1 commit into from
Jul 30, 2026
Merged

feat: add portable WDL module skill #1

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 80 additions & 0 deletions .agents/skills/creating-wdl-modules/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
---
name: creating-wdl-modules
description: Use when creating, adding, scaffolding, bootstrapping, or updating a WDL module that wraps a bioinformatics command-line tool in this repository.
---

# Creating WDL Modules

## Overview

Create reusable WDL 1.4 tool modules only after proving upstream license and
container provenance. A module is complete when its typed tasks, documentation,
native Sprocket tests, and repository-wide CI-equivalent checks all pass.

## Required order

1. Inspect `README.md`, `CONTRIBUTING.md`, `.github/pull_request_template.md`,
`sprocket.toml`, CI, and existing modules.
2. Fix the exact upstream tool version and commands in scope.
3. Read [license-review.md](references/license-review.md). Complete the license
decision before creating module files.
4. Resolve a digest-pinned official upstream image. Use BioContainers only
through the documented fallback.
5. Read [module-convention.md](references/module-convention.md). Derive typed
task interfaces from the pinned version's per-subcommand CLI help, and
cross-check source to exclude only hidden developer flags. Represent every
finite public CLI choice set as a WDL enum during interface derivation. Look
up the tool on `bio.tools` before deciding whether the manifest has an ID.
Record upstream tool provenance using the current `tools` object fields.
6. Create or update the manifest, README, root WDL entrypoint, one WDL file per
public subcommand, one adjacent native test YAML per subcommand,
provenance-documented fixtures, and shared `test/fixtures`. Give every WDL
document a two-sentence module documentation comment that states its purpose
and names every task and user-defined type it exports. Fixtures may be
original or deterministically generated with the pinned tool.
7. Read [quality-gates.md](references/quality-gates.md). Run the targeted
module checks, then every repository-wide CI-equivalent check.
8. Report provenance evidence and the factual PR tools-table row.

## Hard gates

Stop rather than improvise when:

- the upstream license is prohibited, missing, or ambiguous;
- neither the upstream project nor BioContainers publishes a suitable image;
- an immutable image digest cannot be verified;
- fixtures are copied or have unknown or undocumented provenance;
- authoritative docs do not support the proposed task interface; or
- formatting, linting, native tests, or CI still fail.

## Non-negotiable rules

| Pressure | Required response |
| ---------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| "Ship now; review the license later" | License review precedes scaffolding. |
| "The tag is specific enough" | WDL container must include the immutable sha256 digest; a bare tag is not pinned. |
| "Use any working image" | Only an official upstream image or BioContainers fallback—searched in that order and digest-pinned—is accepted. |
| "Tests can follow" | Every public task ships with passing `sprocket dev test` coverage before merge. |
| "I'll track tests in a follow-up issue" | No follow-up issue substitutes for `sprocket dev test` passing at merge time. |
| "A smoke test is enough for now" | Native `sprocket dev test` coverage is required; minimal smoke tests are not equivalent. |
| "`extra_args` supports that option" | Every public functional option in the pinned CLI has a typed input; `extra_args` only covers options absent from that CLI. |
| "Silence the lint or change expected output" | Fix the cause; do not weaken the gate without evidence of an incorrect gate. |
| "A `String` is simpler for these choices" | Represent every finite public CLI choice set as a WDL enum with values matching the pinned CLI tokens. |
| "Spell out every enum value" | Omit an explicit enum value when it is identical to the choice name; assign a value only when the pinned CLI token differs. |
| "One WDL file is simpler" | Put each public subcommand in its own folder and WDL file, then selectively re-export it from the module entrypoint. |
| "`meta` and `parameter_meta` are still needed" | Omit them when Sprocket `##` comments document the same task, inputs, and outputs. |
| "The README already describes the module" | Give every WDL document a two-sentence `##` module comment that states its purpose and names every task and user-defined type it exports. |
| "Generated fixtures need no provenance" | Document the pinned tool version, exact generation and transformation commands, seed, and installation method. |
| "Backticks are cosmetic" | Enclose literals in backticks throughout prose documentation, manifest strings, WDL metadata, prose code comments, and runtime error messages. |

## Completion contract

Do not claim completion until the changed module's native tests and all
repository CI-equivalent commands pass. Do not check contributor legal
attestations or maintainer-only PR checkboxes on another person's behalf.

## Cross-client note

The canonical skill lives under `.agents/skills`. Claude uses the committed
`.claude/skills` symlink. Windows checkouts must enable Developer Mode and Git
symlink support or Claude cannot discover this repository skill.
64 changes: 64 additions & 0 deletions .agents/skills/creating-wdl-modules/references/license-review.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
# License and Container Review

## Evidence record

Before writing module files, record:

| Field | Required evidence |
| ------------ | ------------------------------------------------------- |
| Tool | Canonical upstream project name and URL |
| Version | Exact release tag or immutable commit |
| License | Full name and SPDX identifier |
| License file | Direct URL pinned to the selected tag or commit |
| Commands | Authoritative documentation for each wrapped command |
| Container | Registry, immutable digest, and publisher |

Prefer the upstream repository's license file at the selected release. Package
metadata may corroborate it but does not replace available upstream evidence.
Repeat the review for every version update.

## Policy decision

Compare the verified license with `README.md`:

- Continue only when the license is on the approved list.
- Stop for AGPL or another prohibited use-triggered license.
- Stop and present the evidence for maintainer review when the license is
unlisted, unclear, conflicting, or missing.

Do not create a partial module before this decision.

## Container decision

1. Search the upstream project for an official image matching the exact selected
version and supported platform.
2. If no suitable official image exists, search BioContainers for the exact
upstream version and supported platform.
3. Resolve the selected image's OCI digest with:

```bash
docker buildx imagetools inspect <image-tag>
```

4. Put `<image-tag>@sha256:<digest>` in WDL.
5. Verify the image executes the expected tool version through the module's
Sprocket tests.

When BioContainers supplies the fallback, explain why no suitable official
upstream image was used in the module README. Stop if neither source is
available. An image from another third party and a mutable tag are not
fallbacks.

## Pull-request handoff

Provide this factual table:

```markdown
| Tool | Version | License | SPDX | License file | Needs review? |
| ------------------------------- | :---------: | ------------------- | :------: | ------------------------------------------- | :-----------: |
| [<tool>](<project-url>) | `<version>` | <full-license-name> | `<spdx>` | [Link](<version-pinned-license-url>) | ☐ |
```

Leave `Needs review?` as `☐` for an approved license. Change it to `☒` only
for an unlisted license that the maintainer must review. Never check a
contributor's right-to-license attestation or any maintainer-only checkbox.
145 changes: 145 additions & 0 deletions .agents/skills/creating-wdl-modules/references/module-convention.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
# Module Convention

## Layout

Use one root-level lowercase kebab-case directory per upstream tool:

```text
<tool>/
├── module.json
├── README.md
├── <tool>.wdl
├── <subcommand>/
│ ├── <subcommand>.wdl
│ └── test/
│ └── <subcommand>.yaml
└── test/
└── fixtures/
```

Module trees may not contain symlinks. `<tool>.wdl` is the explicit module
entrypoint. For a tool with subcommands, put each public subcommand task in
`<subcommand>/<subcommand>.wdl` and selectively import it into the entrypoint
so consumers can write `import { <subcommand> } from <tool>`. Keep a tool with
no subcommands in the entrypoint. Do not add workflows.

## Manifest

`module.json` follows the schema identified by its `$schema` field. Engines
ignore unrecognized fields for forward compatibility, but repository manifests
use only standardized fields. A manifest must:

- use the module license expression `MIT OR Apache-2.0`;
- set `entrypoint` to `<tool>.wdl`;
- set `repository` to the canonical repository URL;
- use a `tools` array whose entries contain required `name`, `version`, and
SPDX `license` strings, plus optional `url` and `ids` fields;
- search `bio.tools` by tool name and upstream URL before assigning an ID;
- encode identifiers as an `ids` array of registered CURIE strings, such as
`["biotools:<biotoolsID>"]`, only when the `bio.tools` record is an exact
upstream match; omit `ids` when no authoritative identifier exists;
- enclose code-like literals in backticks within prose fields such as
`description`;
- use `{}` for `dependencies` unless WDL imports another module; each declared
dependency uses either a local `path` or a `git` URL with exactly one of
`version`, `tag`, `branch`, or `commit`, plus an optional repository `path`;
- contain no comments, duplicate keys, or trailing commas.

The module's release version and the wrapped tool's `tools[].version` serve
different purposes. Changing the wrapped tool in a way that changes expected
outputs requires a new module Git tag even though `module.json` has no
top-level version field.

## WDL tasks

Every document declares `version 1.4`. For tools with subcommands, name each
task after its subcommand, e.g., `filter`, so the module supports
`import { filter } from fq`. For tools without subcommands, use a concise
snake_case task name.

Begin every WDL document with a Sprocket `##` module documentation comment
immediately before the `version` declaration. Its first sentence states the
document's purpose. Its second sentence names every task and user-defined type
that the document exports. Keep license, container, and fixture provenance in
the README rather than duplicating them in this comment.

Treat the pinned executable's actual per-subcommand CLI help as the authoritative
option surface. Cross-check source code to identify flags hidden from help; README
omissions do not narrow required coverage. Every public functional option must have
an explicit typed input. Exclude only help, version, and hidden developer flags.
`extra_args` supports options absent from the pinned CLI, such as future upstream
additions; it never substitutes for a typed input for a current option.

Every finite public CLI choice set uses a top-level WDL enum declared before all
tasks. Enum values exactly match the pinned CLI tokens. Free-form `String` is
reserved for genuinely open-ended input such as filenames, record identifiers,
and regular expressions. When a choice's value is identical to its name, omit
the redundant explicit value. Assign a value only when the pinned CLI token
differs from the WDL choice name. Extract enum values for use in Bash with
`value()`.

For each task:

- expose stable options as typed inputs;
- place `Array[String] extra_args = []` last;
- provide conservative overridable `cpu`, `memory`, `disk_gib`, and
digest-pinned `container` defaults;
- use `requirements`, never deprecated `runtime` or the `docker` alias;
- use a heredoc command beginning with `set -euo pipefail`;
- use WDL `env` declarations for string, file, directory, and argument-file
values that enter Bash;
- put `# shellcheck disable=SC2154` before `set -euo pipefail`; this is the only
default suppression and reconciles WDL runtime `env` exports with ShellCheck;
- do not begin input identifiers with `input`, which Sprocket rejects;
- serialize `extra_args` with `write_lines`, load them using `mapfile -t`, and
expand the Bash array as `"${extra_args[@]}"`;
- declare deterministic, exact output paths rather than broad globs; and
- use Sprocket `##` comments to document the task, every input, and every output;
do not add redundant `meta` or `parameter_meta` blocks.

Use Sprocket documentation comments beginning with `##`. Place documentation
comments immediately before every task, user-defined type, enum choice or
struct member, input declaration, and output declaration. Use ordinary `#`
prose comments for private declarations because `sprocket dev doc` does not
publish them and Sprocket reports `##` there as `UnusedDocComments`. Keep
`meta` and `parameter_meta` because they remain part of the WDL task interface.
Enable documentation comments for `sprocket dev doc` with
`doc.with_doc_comments = true` in `sprocket.toml`.

In manifest prose, `meta`, `parameter_meta`, documentation comments, prose
comments, and runtime error messages, enclose code-like literals in backticks.
This includes tool names, versions, task names, subcommand names, option flags,
file paths, identifiers, default values, enum values, and format suffixes such
as `.gz`. Quote shell error strings so backticks remain literal rather than
command substitutions. Syntactic directives such as
`# shellcheck disable=SC2154` are exempt where backticks would break the tool.

`extra_args` preserves argument boundaries but remains a trusted-caller escape
hatch: the wrapper cannot decide whether the upstream tool treats an argument
as dangerous or invalid.

## README

Document:

1. the wrapped tool, exact version, and upstream URL;
2. the upstream license and version-pinned license URL;
3. the container URI, digest, and publisher;
4. why BioContainers replaced an official upstream image, when applicable;
5. every public task and its material behavior, including enum types and their
available choices for any finite option set; and
6. a `Fixture provenance` section with the pinned tool version, installation
method, exact deterministic generation and transformation commands, seed,
and an explicit statement that the data is synthetic when fixtures are
generated;
and
7. `sprocket dev test <tool>`.

Enclose code-like literals in backticks throughout README prose: tool names,
version strings, task names, option names, enum choice names and values, file
paths, and format suffixes. Do not backtick ordinary domain terms such as
"FASTQ" or "read pair".

Do not copy upstream prose, code, binaries, or fixtures into the repository.
Fixtures must either be authored for the module or deterministically generated
by the pinned tool with documented provenance.
61 changes: 61 additions & 0 deletions .agents/skills/creating-wdl-modules/references/quality-gates.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
# Quality Gates

## Test placement

Sprocket matches each `<name>.wdl` with an adjacent `test/<name>.yaml`. For
tools with subcommands, keep each native test YAML beside its subcommand WDL
under `<subcommand>/test/<subcommand>.yaml`. Put shared minimal fixtures under
the module's root `test/fixtures`. Fixtures may be authored for the module or
generated deterministically by the pinned tool when the README records the
exact command, seed, version, installation method, transformation commands, and
synthetic-data status. Never use copied or unknown fixtures.
Sprocket writes execution state under ignored `test/runs` directories.

Every public task needs at least one native test. Add cases or an input matrix
for meaningful option branches. Assert every supported semantic property:

- `exit_code` or intentional `should_fail`;
- exact Boolean, string, integer, or float values;
- array/map length or emptiness and first/last array elements;
- file and directory basenames with `Name`; and
- meaningful stdout/stderr regular expressions.

Do not use the schema's `custom` assertion until the repository's minimum
Sprocket release implements it. Do not add test-only WDL outputs merely to
inspect opaque file contents.

## Development loop

Format changed WDL:

```bash
sprocket format overwrite <module-directory>
```

Run the smallest relevant checks:

```bash
sprocket dev module verify --manifest-path <module-directory>
sprocket lint <module-directory>
sprocket dev test <module-directory>
```

Then reproduce CI:

```bash
sprocket format check .
sprocket lint .
sprocket dev doc --check .

shopt -s nullglob
for manifest in ./*/module.json; do
module_dir="$(dirname "$manifest")"
sprocket dev module verify --manifest-path "$module_dir"
sprocket dev test "$module_dir"
done
```

If a gate fails, preserve the evidence, fix the root cause, rerun the smallest
failing command, then rerun the complete gate. Do not add broad exceptions,
remove assertions, change expected output without upstream evidence, or weaken
CI.
17 changes: 17 additions & 0 deletions .agents/skills/creating-wdl-modules/templates/module.json.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
{
"$schema": "https://raw.githubusercontent.com/openwdl/wdl/8199dd7e3f17322e827225d6e6bc886dac139207/modules/schemas/module.schema.json",
"name": "__TOOL_DISPLAY_NAME__",
"license": "MIT OR Apache-2.0",
"description": "WDL tasks for `__TOOL_DISPLAY_NAME__`",
"repository": "https://github.com/stjude-rust-labs/modules",
"entrypoint": "__TOOL__.wdl",
"tools": [
{
"name": "__TOOL_DISPLAY_NAME__",
"version": "__VERSION__",
"license": "__UPSTREAM_SPDX__",
"url": "__UPSTREAM_URL__"
}
],
"dependencies": {}
}
Loading