SciPackU

Scientific compression and conversion agent for high-throughput research data.

SciPackU combines a real-execution ReAct compression agent with deterministic scientific-data profiling, validated compression adapters, and a progressive compression skill tree. It is designed for acquisition and data engineering teams working with microscopy, BCI/neural streams, electrophysiology, simulations, astronomy, climate/geospatial data, spectroscopy, tabular lab exports, and mixed experiment bundles.

The default safety posture is strict: ambiguous compression requests route to lossless methods. Near-lossless and lossy workflows require explicit authorization and an explicit error, precision, rate, or quality target.

Core principle

Treat primary scientific acquisition data as irreplaceable. SciPackU plans compression with format awareness, progressive skill loading, and explicit safety gates before any controlled-loss path.

Contents

• Core Capabilities
• Architecture
• Installation
• Quick Start
• CLI
• Streamlit UI
• Compression Skills
• Safety Policy
• Agent Workflow

Core Capabilities

Capability	What It Provides	Primary Interface
ReAct compression execution	Profiles the dataset, senses tools, routes through skills, executes compression, validates results, and saves an audit trace	spk compress
Verified lossless archives	Runs directory archive adapters and verifies outputs by streaming decompression plus per-file SHA-256 comparison	spk compress, spk lossless-benchmark, spk validate-lossless
Controlled-loss skill audits	Tests near-lossless/lossy methods on samples or derived previews with explicit authorization, targets, parameters, metrics, skip reasons, and reports	spk compress --mode lossy, skills/

Architecture

flowchart LR
    A["User data and objective"] --> B["Scientific profiler"]
    B --> C["Skill tree router"]
    C --> D{"Compression policy"}
    D -->|"default"| E["Lossless skills"]
    D -->|"authorized + target"| F["Near-lossless skills"]
    D -->|"explicit derived product"| G["Lossy skills"]
    E --> H["Controlled execution tools"]
    F --> I["Sample / audit tools"]
    G --> I
    H --> J["Validation, trace, and report"]
    I --> J

Loading

The compression skill system is physically flat on disk but logically organized as a progressive-disclosure tree:

scientific-compression-common
  lossless-compression-common
  near-lossless-compression-common
  lossy-compression-common
    method-specific compression skills

Progressive disclosure

Agents should load scientific-compression-common first, then only the selected type-specific common skill, and finally the selected method skill. Long references and scripts remain dormant unless the selected method needs them.

Project Layout

SciPackU/
  agent/                  LangGraph ReAct agent
  core/                   Deterministic profiling, skill loading, routing, planning
  prompts/                Agent system instructions
  tools/                  Agent-callable tools
  skills/                 Lossless, near-lossless, lossy, and unified skill tree packs
  configs/                Model configuration template
  scripts/                Portable command-line helpers
  tests/                  Profiler, router, CLI, and UI helper tests
  main.py                 JSON-first CLI implementation and legacy compatibility entry point
  demo.py                 Streamlit frontend

Installation

git clone https://github.com/SciDataOcean/SciPackU.git
cd SciPackU
python -m venv .venv
source .venv/bin/activate
pip install -e ".[full]"

Editable full installation is the recommended local research workflow. It exposes the short spk command, installs the ReAct/Streamlit/scientific I/O dependencies, and keeps the repo-local skills/ tree available for routing.

For deterministic utilities only, the minimal package is enough:

pip install -e .

If your environment uses a controlled package mirror, the existing requirements file can also be used by administrators, but pyproject.toml is the canonical install definition:

pip install -r requirements.txt

For agent-powered conversion, copy the model configuration template and fill in your provider or proxy settings:

cp configs/model_config.template.yaml configs/model_config.yaml

You can also use environment variables:

export SCIPACKU_API_KEY="..."
export SCIPACKU_BASE_URL="https://your-provider-or-proxy/v1"
export SCIPACKU_MODEL_NAME="gpt-4o"

spk compress uses the ReAct agent and therefore needs a working model configuration. Deterministic utilities such as spk doctor, spk lossless-benchmark, and spk validate-lossless can be used for local checks and batch verification without relying on the interactive agent loop.

Quick Start

First verify the installation and skill tree:

spk doctor --json

For spk compress, make sure a model is configured through configs/model_config.yaml or the SCIPACKU_* environment variables shown above.

Then run the one-command compression workflow:

spk compress \
  --input DATA_PATH \
  --output TARGET_PATH

Use DATA_PATH for the local source file or directory and TARGET_PATH for the final compressed artifact, for example a .tar.zst path for a directory dataset. By default, SciPackU treats the task as strict lossless archival compression. It profiles the input, routes through the scientific compression skill tree, executes the selected method, validates the output, and saves a trace.

After the first run summary, SciPackU enters a same-session follow-up chat. You can refine the compression task without restarting the agent:

SciPackU> use gzip instead
SciPackU> summary
SciPackU> trace
SciPackU> exit

use gzip instead is a natural-language follow-up prompt. summary reprints the current session summary, trace shows the trace file path, and exit ends the chat.

The main run artifacts are:

TARGET_PATH                  final compressed output artifact requested by the user
output/traces/.../trace.jsonl  append-only ReAct event log with user messages, tool calls, tool results, decision summaries, and final status
output/traces/.../sharegpt.json human-readable conversation export for review, replay, or dataset construction
output/traces/.../summary.json  compact machine-readable run summary with status, tool names, paths, and metadata

For controlled-loss audits, method-specific reports are written beside the requested output under near_lossless_audit/ or lossy_audit/.

CLI

The standard user workflow is a single interactive compression command. Use --json only when another program needs machine-readable output.

Task	Command
One-command compression	spk compress --input DATA_PATH --output TARGET_PATH
Explicit method override	spk compress --input DATA_PATH --output TARGET_PATH --mode lossless --method gzip
Automation JSON output	spk compress --input DATA_PATH --output TARGET_PATH --json
Authorized near-lossless audit	spk compress --input DATA_PATH --output TARGET_PATH --mode near-lossless --target quality=60 --authorize-controlled-loss
Authorized lossy audit	spk compress --input DATA_PATH --output TARGET_PATH --mode lossy --target psnr=45 --authorize-controlled-loss
Standalone validation	spk validate-lossless --input DATA_PATH --output TARGET_PATH --method zstd --json
Compare lossless methods	spk lossless-benchmark --input DATA_PATH --output-dir TARGET_PATH --methods all --json

Human mode renders an audit-oriented terminal view with per-round context, streamed progress events, selected skill details, parameters, metrics, and trace location. Result tables are shown only in summaries, after real audit entries exist. Controlled-loss modes keep the same interface but require explicit authorization and a target.

Default lossless compression:

spk compress \
  --input DATA_PATH \
  --output TARGET_PATH

Force a specific lossless method:

spk compress \
  --input DATA_PATH \
  --output TARGET_PATH \
  --mode lossless \
  --method gzip

Automation mode keeps stdout as pure JSON and disables the follow-up prompt:

spk compress \
  --input DATA_PATH \
  --output TARGET_PATH \
  --json

Validate an existing archive:

spk validate-lossless \
  --input DATA_PATH \
  --output TARGET_PATH \
  --method zstd \
  --json

Compare all applicable lossless methods:

spk lossless-benchmark \
  --input DATA_PATH \
  --output-dir TARGET_PATH \
  --methods all \
  --json

Directory archive methods currently include zstd, lz4, gzip, xz, bzip2, brotli, snappy, blosc-bitshuffle, and container-tiff-lossless. zfp-reversible and fpzip are array-only methods and return not_applicable for mixed directory datasets unless a known-shape numeric array is selected.

Run an explicitly authorized lossy skill audit on samples or derived previews:

spk compress \
  --input DATA_PATH \
  --output TARGET_PATH \
  --mode lossy \
  --target psnr=45 \
  --authorize-controlled-loss

In the follow-up prompt, request methods such as test zfp lossy, test JPEG2000/JPEG-LS/JXL, test video codecs, or summary. SciPackU records parameter adjustments, measured metrics, skipped tools, and not-applicable formats under lossy_audit/.

Run an explicitly authorized near-lossless audit:

spk compress \
  --input DATA_PATH \
  --output TARGET_PATH \
  --mode near-lossless \
  --target quality=60 \
  --authorize-controlled-loss

In the follow-up prompt, request methods such as test zfp near-lossless, test JPEG2000/JPEG-LS/JXL, test SZ3/cuSZ, or summary. SciPackU records pass, skipped, blocked, and not-applicable outcomes under near_lossless_audit/.

Streamlit UI

streamlit run demo.py

The frontend provides:

• Home: project scope, supported data classes, and safety boundaries.
• Profile: upload or select a local file and inspect a lightweight scientific data profile.
• Skill Route: enter an objective and inspect the lossless/near-lossless/lossy route.
• Compression Plan: preview command templates, planned validation steps, and warnings.
• Agent Convert: use the original LLM ReAct conversion workflow.
• Reports/Logs: inspect recent local output files.

Near-lossless and lossy routes expose an authorization checkbox and target input. The UI blocks execution-style planning until both are provided.

Compression Skills

SciPackU keeps four portable skill distributions under skills/:

Pack	Purpose
lossless-compression-pack	Exact-preservation compression methods such as zstd, lz4, gzip, Blosc, reversible zfp, and container-aware workflows.
near-lossless-compression-pack	Controlled-error compression methods such as zfp, SZ3/cuSZ, MGARD, fpzip, FITS quantization, and JPEG 2000/JPEG-LS/JXL workflows.
lossy-compression-pack	Intentional-loss workflows for derived products, previews, videos, and explicitly authorized lossy scientific compression.
scientific-compression-skill-tree	Unified router tree that helps agents load only the common and method skills needed for a task.

These skill packs are agent-facing knowledge and command templates. They do not imply that every external compressor is installed locally or enabled for direct execution.

The default skill root is:

skills/scientific-compression-skill-tree/skills

Override it with:

export SCIPACKU_SKILL_ROOT=/path/to/skills

Skill Tree Snapshot

scientific-compression-common
  lossless-compression-common
    compress-with-zstd
    compress-with-lz4
    compress-with-gzip-deflate
    ...
  near-lossless-compression-common
    compress-nearlossless-zfp
    compress-nearlossless-sz3-cusz
    ...
  lossy-compression-common
    compress-lossy-image-codecs
    compress-lossy-video-codecs
    ...

Safety Policy

Policy	Behavior
Original preservation	Never plan a workflow that overwrites the only source copy.
Lossless default	Ambiguous objectives and unknown formats route to lossless methods.
Controlled-loss authorization	Near-lossless and lossy routes require explicit user authorization.
Target requirement	Controlled-loss routes require a concrete error, precision, rate, PSNR, CRF, or quality target.
Primary vs derived data	Image/video codecs are treated as derived-product routes unless explicitly approved for primary data.
Progressive skill loading	Root skill first, type-specific common skill second, method skill last.
Execution boundary	Real execution is enabled for validated lossless directory archive adapters. Array-only and controlled-loss methods remain gated by applicability, authorization, target, and validation support.

Agent Workflow

For compression tasks, the ReAct agent is instructed to:

1. Profile the data with scipacku_profile.
2. Load the root skill through scipacku_list_compression_skills / scipacku_read_compression_skill.
3. Route to lossless, near-lossless, or lossy common skills with scipacku_route_compression_skill.
4. Load only the selected method skill.
5. Execute the selected method through scipacku_execute_compression.
6. Validate strict lossless outputs with the built-in hash validator, or call scipacku_validate_lossless when a separate validation pass is needed.
7. Save the full ReAct trace as JSONL and ShareGPT-compatible JSON for audit and replay.
8. In human mode, render concise progress and allow same-session follow-up prompts without overwriting existing outputs.
9. For lossy audits, test sample or derived-preview artifacts only by default, and record requested parameters, effective parameters, adjustments, metrics, and explicit skip/not-applicable reasons.

spk compress is the primary real-execution ReAct entry point. Deterministic lossless-benchmark and validate-lossless remain available for batch benchmarking and independent verification without an LLM loop.

Status Snapshot

Area	Current State
Compression skills	33 skill folders in the unified scientific compression skill tree
Data detection	Pure standard-library profiler for common scientific containers, text exports, and raw-stream naming conventions
Direct compression execution	Real lossless directory archive execution for zstd, lz4, gzip, xz, bzip2, Brotli, Snappy, Blosc-bitshuffle, and TIFF-sidecar bundle routes
Controlled-loss audit execution	Sample/derived-preview audits for authorized near-lossless and lossy workflows, with parameter records, metrics, skip reasons, and not-applicable status
UI	Lightweight Streamlit frontend for profile, route, plan, convert, and inspect workflows
Safety	Lossless-first defaults with explicit controlled-loss gates

License

MIT