From 5ea7a43da2f7545225e4247e9243d3b25a6c5c20 Mon Sep 17 00:00:00 2001 From: Paulo Lacerda Date: Sun, 9 Aug 2026 07:48:56 -0300 Subject: [PATCH] docs: correct the pre-flight checks section in how-it-works The "Pre-flight checks" section described a subsystem that does not exist as written. Verified against the v0.8.6 tree: * It pointed at `pipeline/runtime.py`. That module only contains evaluator loading and per-row evaluator execution (`load_evaluator`, `run_evaluator`, `_model_config`, `_extract_score`, `_is_transient_credential_error`). The real implementation is `services/preflight.py`, imported only from `cli/app.py`. * The bullet list did not match the four checks that actually run. The real display names are Workspace, Azure authentication, Foundry project and Application Insights. There is no package-presence check, no env-var aggregation check, and no TCP probe for URL agents. `git grep` for `missing_packages|required_env|MISSING_ENV|_check_env` under `src/agentops/pipeline` returns nothing, and the only socket use in the CLI is `_port_in_use`, which checks whether the cockpit port is taken. * It documented `agentops eval run --dry-run` for CI gating. That flag does not exist on `eval run`, whose flags are `--agent --baseline --config/-c --format/-f --output/-o`. The only `--dry-run` in the CLI belongs to `telemetry dashboard deploy`. Anyone copying that line into a pipeline would get a hard failure. The real gate is `--strict-preflight` on `agentops doctor`. The corrected text is confirmed by the sample output already present in `docs/tutorial-prompt-agent.md`, which shows `AgentOps pre-flight 4 ok` covering exactly those four checks. Also adds the missing `services/preflight.py` entry to the documented directory tree and repoints the "Where to Add New Code" row. The same fix landed on the `docs` branch in #406. This commit keeps the in-repo copy read by contributors from `README.md` and `CONTRIBUTING.md` consistent with it. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bcb9c0b6-d506-46dc-90d2-8120413166ee --- docs/how-it-works.md | 39 ++++++++++++++++++++++----------------- 1 file changed, 22 insertions(+), 17 deletions(-) diff --git a/docs/how-it-works.md b/docs/how-it-works.md index cbf914e..f397ac0 100644 --- a/docs/how-it-works.md +++ b/docs/how-it-works.md @@ -58,7 +58,7 @@ src/ │ ├── pipeline/ # Run orchestration - ADD execution flows here │ ├── orchestrator.py # End-to-end `eval run` driver - │ ├── runtime.py # Pre-flight checks (deps, creds, endpoints) + │ ├── runtime.py # Evaluator loading and per-row evaluator execution │ ├── invocations.py # Per-row agent / model invocation strategies │ ├── thresholds.py # Threshold pass/fail evaluation │ ├── reporter.py # Markdown report generation @@ -73,6 +73,7 @@ src/ │ ├── skills.py # Coding agent skill installation │ ├── cicd.py # CI/CD workflow generation │ ├── evidence_pack.py # Release evidence aggregation/writer + │ ├── preflight.py # Pre-flight checks (workspace, auth, Foundry, App Insights) │ └── trace_promotion.py # Trace export → dataset candidates │ ├── agent/ # Doctor, Cockpit, and agent server @@ -96,7 +97,7 @@ src/ |---|---| | Add a field to `agentops.yaml` | `core/agentops_config.py` | | Add a new evaluator preset | `core/evaluators.py` (catalog) | -| Change pre-flight checks | `pipeline/runtime.py` | +| Change pre-flight checks | `services/preflight.py` | | Add a target kind | `pipeline/invocations.py` + `core/agentops_config.py` | | Tweak the report layout | `pipeline/reporter.py` | | Add a publish destination | `pipeline/publisher.py` or `pipeline/cloud_runner.py` | @@ -617,21 +618,25 @@ Implementation lives in [src/agentops/pipeline/publisher.py](../src/agentops/pip ## Pre-flight checks -Before any agent invocation, [pipeline/runtime.py](../src/agentops/pipeline/runtime.py) -runs a short series of checks and reports **all** failures at once: - -* Required Python packages installed (`azure-identity`, - `azure-ai-evaluation` for AI-assisted evaluators, `azure-ai-projects` - for Foundry invocation, publishing, or `execution: cloud`). -* Required env vars set (`AZURE_AI_FOUNDRY_PROJECT_ENDPOINT`, - `AZURE_OPENAI_*` deployment fields). -* Azure CLI credential acquires a token within 30 s - (`process_timeout=30` is set everywhere `DefaultAzureCredential` is - instantiated to absorb Windows `az.cmd` cold starts). -* For URL agents, the endpoint resolves and accepts a TCP connection. - -`agentops eval run --dry-run` runs only the pre-flight phase and exits -`0` (all clear) or `1` (something to fix). Useful for CI gating. +Before any agent invocation, [services/preflight.py](../src/agentops/services/preflight.py) +runs a short series of checks and reports **all** rows at once instead of +stopping at the first problem. It is wired into `agentops doctor` and +`agentops cockpit`: + +* **Workspace** — the target directory is a usable AgentOps workspace. +* **Azure authentication** — `DefaultAzureCredential` acquires an ARM token + within 30 s (`process_timeout=30` is set to absorb Windows `az.cmd` cold + starts). +* **Foundry project** — the configured project endpoint is reachable. +* **Application Insights** — a connection string is available, either + auto-discovered from Foundry or set through + `APPLICATIONINSIGHTS_CONNECTION_STRING`. + +Each check is a single best-effort attempt with no retries, and a failing +check never raises into the CLI. The default policy is advisory: warnings +print and the command continues. For CI gating, pass `--strict-preflight` +to `agentops doctor` so any failure exits non-zero. Pass `--no-preflight` +to `doctor` or `cockpit` to skip the checks entirely. ## Invocation strategies (target kind → wire call)