quality-harness

Test software and agents. Turn execution evidence into quality assessments.

中文

quality-harness provides SDKs for API testing, agent evaluation, performance testing, trace analysis, and agent trajectory analysis. Use them to run your project's tests or assess evidence from existing runs, then produce reports and machine-readable verdicts for developers, CI, and agent workflows.

What you can check

Question	Capability	SDKs
Do service APIs behave as expected?	e2e — run cases and check API contracts	Python / Go
How good are an agent's outputs?	eval — evaluate results and compare experiments	Python
How does the system perform under load?	perf — measure latency, throughput, and resource use against declared targets	Python / TypeScript
Where does a call chain show abnormal behavior?	trace — analyze spans, locate anomalies, and investigate causes	Python / TypeScript
Are an agent's decisions and actions effective and efficient?	trajectory — measure cost, detect patterns, and verify behavior	Python

Get started

Choose an example for your scenario:

Example	Use it for
API cases	Data-driven requests and assertions
Python service tests	Tests with setup, multiple operations, and cleanup
Go service tests	Case execution with go test and aggregated verdicts
Agent evaluation	Dataset-driven quality assessment

To run the Python API example, start from a source checkout with Python 3.11+ and uv installed. Configure the example's cases and service URL for your target, then run:

cd sdks/python
uv sync
export WIDGET_BASE_URL=http://localhost:8080
export WIDGET_TOKEN=...
uv run e2e run ../../examples/api-test/cases.yaml \
  --config ../../examples/api-test/config.yaml \
  --runs-dir ../../runs

The target service must be running and expose the endpoints used by the cases. Results are written to a run directory under runs/, including verdict.json. Skipped cases and execution errors remain visible in the result.