Skip to content

cli-reference.md

Latest commit

 

History

History
144 lines (115 loc) · 8 KB

cli-reference.md

File metadata and controls

144 lines (115 loc) · 8 KB
Raw

CLI reference

Every command supports --help for its complete, always-current flag list -- this page covers what each one is for and the flags that matter most for choosing between them. Provider-calling commands share a common flag set covered once in Shared provider/governance flags rather than repeated per command.

The pipeline commands

buffdata optimize

The main entry point: runs the full 7-stage pipeline (see architecture.md) with CLI flags overriding a --config YAML file where both are given.

buffdata optimize data.jsonl -o optimized.jsonl --provider openai
buffdata optimize train.csv -o train.parquet --accuracy-contract strict
buffdata optimize data.jsonl -o out.jsonl --classification off
buffdata optimize data.jsonl -o out.jsonl --classification-pii-mode identifiers
buffdata optimize data.jsonl -o out.jsonl \
  --provider gemini --model gemini-3.5-flash-lite \
  --quality-mode sampled --quality-sample-size 40

Input/output format is inferred from the file extension -- .json, .jsonl, .ndjson, .csv, .tsv, .parquet/.pq, .arrow/.feather/.ipc, .yaml/.yml, .txt, their .gz variants, and .hf (Hugging Face save_to_disk) directories -- converting between formats is free (optimize train.csv -o train.parquet). optimize/pipeline don't accept cloud (s3:///gs:///az://) URLs, since their checkpoint/report/rejected sidecar files assume a local filesystem; every other provider-calling command below does.

Produces three files beside the output: <name>.rejected.jsonl (every dropped row with a reason), <name>.report.json (profile, per-stage metrics, provider/model, token usage -- what buffdata audit record and buffdata contract check both read), and, with --observability, <name>.metrics.prom (see observability.md).

Accuracy-gate flags (--validation-file, --require-positive-gain, --accuracy-min-gain, --accuracy-seeds, --accuracy-epochs, --accuracy-max-train-rows) publish the output only when it's proven, against a held-out labeled set, to not regress accuracy -- the same machinery buffdata generate uses (below).

buffdata pipeline <config.yaml> -i input -o output

Same engine as optimize, but every setting comes from a PipelineConfig YAML file (see examples/pipeline_config.yaml) with no per-flag overrides. Use this when a run needs to be checked into source control as config, or wired into an orchestrator that shells out with a config path rather than a long flag list.

buffdata generate <labeled.jsonl> -o generated.jsonl --validation-file held_out.jsonl

Iteratively augments a labeled classification dataset -- generating variations, retraining a proxy classifier, measuring the accuracy gain against --validation-file -- until it clears --min-relative-gain (default 10%) or exhausts --max-iterations (default 5), reporting the shortfall honestly rather than publishing an unproven result either way. --weak-class-count targets retry rounds at whichever classes are dragging accuracy down. Backed by buffdata/optimizers/gated_generator.py and the same buffdata/evaluation/accuracy_gate.py proxy-classifier machinery optimize --require-positive-gain uses.

Focused single-stage commands

Each does one pipeline stage's job standalone, and (unlike optimize/pipeline) every one of these accepts cloud URLs directly:

buffdata score data.jsonl -o scored.jsonl --provider openai        # LLM-as-a-judge quality scoring
buffdata refine data.jsonl -o refined.jsonl --provider anthropic   # clean/expand/de-boilerplate
buffdata evolve data.jsonl -o evolved.jsonl --provider gemini      # Evol-Instruct complexity growth
buffdata dpo data.jsonl -o dpo.jsonl --provider openai             # chosen/rejected preference pairs
buffdata augment labeled.jsonl -o augmented.jsonl --provider gemini --multiplier 1  # label-guarded synthetic variations
buffdata classify data.jsonl -o labeled.jsonl --type multi-label --provider anthropic
buffdata dedup data.jsonl -o deduped.jsonl --method minhash        # exact | minhash | semantic-local | semantic
buffdata scrub data.jsonl -o scrubbed.jsonl                        # local PII redaction, no provider needed
buffdata validate data.jsonl                                       # schema/structural check only
buffdata filter-ppl data.jsonl -o filtered.jsonl                   # local PyTorch perplexity filter, no provider needed
buffdata synthesize source.pdf -o synthetic.jsonl --provider anthropic  # documents -> instruction pairs
buffdata stats data.jsonl                                          # quality/statistics summary
buffdata report data.jsonl -o report.html                          # interactive HTML audit report

Data movement

buffdata push optimized.jsonl user/my-dataset --token $HF_TOKEN   # to the Hugging Face Hub
buffdata export data.jsonl training.json --format unsloth          # to Axolotl/Unsloth training format

Scale and distribution

See scaling.md for the full picture; in short:

buffdata shard data.jsonl -o parts/ -n 8                    # split for orchestrator-level parallelism
# ... N parallel `buffdata optimize` runs, one per partition-NNNN.jsonl (Airflow/Dagster/k8s/xargs -P) ...
buffdata run-ray pipeline.yaml parts/partition-*.jsonl -o out/  # or: distribute those same partitions as Ray tasks instead
buffdata merge out/*.jsonl -o final.jsonl                    # recombine either way

Governance and compliance

See governance.md for the full picture.

buffdata sbom -o sbom.json                                          # CycloneDX 1.5 SBOM of installed packages
buffdata contract check contract.yaml --report run.report.json      # CI gate: exit 0 pass / 1 fail
buffdata audit record run.report.json --command optimize            # ingest a run into the durable audit log
buffdata audit query --command optimize --since 2026-01-01           # list recorded runs
buffdata audit usage-report --pricing my_rates.yaml                  # aggregate token usage (never guesses $ pricing)

Developer productivity

Opt-in, requires pip install -e ".[productivity]":

buffdata productivity graph-build .                                          # local AST-only Graphify knowledge graph
buffdata productivity graph-query "How does optimize reach the quality scorer?"
buffdata productivity docs-search pydantic                                    # find a Context7 library id
buffdata productivity docs-context /pydantic/pydantic "How do model validators work?"

Shared provider/governance flags

Present on every command that can call an LLM (optimize, score, refine, evolve, dpo, augment, classify, filter-ppl where relevant, synthesize, generate, pipeline via YAML):

Flag Purpose
--provider gemini, openai, anthropic, azure_openai, bedrock_anthropic, openai_compatible, or a local server (ollama, lmstudio, vllm, llamacpp) -- see providers.md
--model Provider model override; each provider has a balanced default except the private/local ones, which have none (there's no universal default a customer gateway or a locally-pulled model could safely assume)
--base-url Endpoint for openai_compatible/local-server providers -- another machine on your network, e.g. http://192.168.1.50:11434/v1
--network-policy unrestricted (default), local (only a provider whose endpoint you control, on a loopback/private address), or strict (zero network calls, guaranteed structurally)
--actor / --policy Check this actor's permissions against an access-policy YAML before running -- governance.md
--bearer-token / --oidc-config Verify a real OIDC bearer token instead of trusting a plain --actor string -- governance.md
--concurrency / --rpm Async request concurrency and rate limit