Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 22 additions & 17 deletions docs/how-it-works.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ src/
├── pipeline/ # Run orchestration - ADD execution flows here
│ ├── orchestrator.py # End-to-end `eval run` driver
│ ├── runtime.py # Pre-flight checks (deps, creds, endpoints)
│ ├── runtime.py # Evaluator loading and per-row evaluator execution
│ ├── invocations.py # Per-row agent / model invocation strategies
│ ├── thresholds.py # Threshold pass/fail evaluation
│ ├── reporter.py # Markdown report generation
Expand All @@ -73,6 +73,7 @@ src/
│ ├── skills.py # Coding agent skill installation
│ ├── cicd.py # CI/CD workflow generation
│ ├── evidence_pack.py # Release evidence aggregation/writer
│ ├── preflight.py # Pre-flight checks (workspace, auth, Foundry, App Insights)
│ └── trace_promotion.py # Trace export → dataset candidates
├── agent/ # Doctor, Cockpit, and agent server
Expand All @@ -96,7 +97,7 @@ src/
|---|---|
| Add a field to `agentops.yaml` | `core/agentops_config.py` |
| Add a new evaluator preset | `core/evaluators.py` (catalog) |
| Change pre-flight checks | `pipeline/runtime.py` |
| Change pre-flight checks | `services/preflight.py` |
| Add a target kind | `pipeline/invocations.py` + `core/agentops_config.py` |
| Tweak the report layout | `pipeline/reporter.py` |
| Add a publish destination | `pipeline/publisher.py` or `pipeline/cloud_runner.py` |
Expand Down Expand Up @@ -617,21 +618,25 @@ Implementation lives in [src/agentops/pipeline/publisher.py](../src/agentops/pip

## Pre-flight checks

Before any agent invocation, [pipeline/runtime.py](../src/agentops/pipeline/runtime.py)
runs a short series of checks and reports **all** failures at once:

* Required Python packages installed (`azure-identity`,
`azure-ai-evaluation` for AI-assisted evaluators, `azure-ai-projects`
for Foundry invocation, publishing, or `execution: cloud`).
* Required env vars set (`AZURE_AI_FOUNDRY_PROJECT_ENDPOINT`,
`AZURE_OPENAI_*` deployment fields).
* Azure CLI credential acquires a token within 30 s
(`process_timeout=30` is set everywhere `DefaultAzureCredential` is
instantiated to absorb Windows `az.cmd` cold starts).
* For URL agents, the endpoint resolves and accepts a TCP connection.

`agentops eval run --dry-run` runs only the pre-flight phase and exits
`0` (all clear) or `1` (something to fix). Useful for CI gating.
Before any agent invocation, [services/preflight.py](../src/agentops/services/preflight.py)
runs a short series of checks and reports **all** rows at once instead of
stopping at the first problem. It is wired into `agentops doctor` and
`agentops cockpit`:

* **Workspace** — the target directory is a usable AgentOps workspace.
* **Azure authentication** — `DefaultAzureCredential` acquires an ARM token
within 30 s (`process_timeout=30` is set to absorb Windows `az.cmd` cold
starts).
* **Foundry project** — the configured project endpoint is reachable.
* **Application Insights** — a connection string is available, either
auto-discovered from Foundry or set through
`APPLICATIONINSIGHTS_CONNECTION_STRING`.

Each check is a single best-effort attempt with no retries, and a failing
check never raises into the CLI. The default policy is advisory: warnings
print and the command continues. For CI gating, pass `--strict-preflight`
to `agentops doctor` so any failure exits non-zero. Pass `--no-preflight`
to `doctor` or `cockpit` to skip the checks entirely.

## Invocation strategies (target kind → wire call)

Expand Down
Loading