Skip to content

docs: correct the pre-flight checks section in how-it-works - #407

Merged
Paulo Lacerda (placerda) merged 1 commit into
developfrom
chore/sync-preflight-docs
Aug 9, 2026
Merged

docs: correct the pre-flight checks section in how-it-works#407
Paulo Lacerda (placerda) merged 1 commit into
developfrom
chore/sync-preflight-docs

Conversation

@placerda

Copy link
Copy Markdown
Contributor

Corrects the Pre-flight checks section of docs/how-it-works.md, which
described a subsystem that does not exist as written.

This is the develop counterpart of #406. That PR fixed the copy on the
orphan docs branch, which is what the published site is built from. This
PR fixes the copy that lives alongside the code and is linked from
README.md and CONTRIBUTING.md.

What was wrong

Verified against the v0.8.6 tree.

1. Wrong module. The section pointed at pipeline/runtime.py. That
module only contains evaluator loading and per-row evaluator execution
(load_evaluator, run_evaluator, _model_config, _extract_score,
_is_transient_credential_error). The real implementation is
services/preflight.py, whose run_preflight() is imported only from
cli/app.py.

2. Wrong checks. The bullet list did not describe the checks that
actually run. The real display names are Workspace, Azure
authentication
, Foundry project and Application Insights.

Negative evidence for the three documented checks that do not exist:

$ git grep -n "missing_packages\|required_env\|MISSING_ENV\|_check_env" -- src/agentops/pipeline
(no output, exit 1)

$ git grep -ln "socket\.\|create_connection" -- src/agentops
src/agentops/agent/sources/azure_monitor.py
src/agentops/cli/app.py

The only socket use in the CLI is _port_in_use, which checks whether the
cockpit port is already taken. It has nothing to do with resolving a URL
agent endpoint.

3. A flag that does not exist. The section told readers to use
agentops eval run --dry-run for CI gating. The real flags on eval run
are --agent --baseline --config/-c --format/-f --output/-o. The only
--dry-run in the whole CLI belongs to telemetry dashboard deploy.
Anyone copying that line into a pipeline gets a hard failure. The real gate
is --strict-preflight on agentops doctor.

Corroboration

docs/tutorial-prompt-agent.md already shows the real output:

AgentOps pre-flight  4 ok

covering the Workspace, Azure auth, Foundry project and App Insights
checks. The corrected text now matches that.

Also in this PR

  • Adds the missing services/preflight.py entry to the documented
    directory tree.
  • Repoints the "Where to Add New Code" row from pipeline/runtime.py to
    services/preflight.py.

How this was found

By resolving every documented agentops ... invocation against the real
click command tree and diffing the flags. That check now reports zero
suspect flags across README.md, src/agentops/templates/**/*.md and
docs/**/*.md on both this branch and the docs branch.

The "Pre-flight checks" section described a subsystem that does not exist
as written. Verified against the v0.8.6 tree:

* It pointed at `pipeline/runtime.py`. That module only contains evaluator
  loading and per-row evaluator execution (`load_evaluator`, `run_evaluator`,
  `_model_config`, `_extract_score`, `_is_transient_credential_error`).
  The real implementation is `services/preflight.py`, imported only from
  `cli/app.py`.

* The bullet list did not match the four checks that actually run. The real
  display names are Workspace, Azure authentication, Foundry project and
  Application Insights. There is no package-presence check, no env-var
  aggregation check, and no TCP probe for URL agents. `git grep` for
  `missing_packages|required_env|MISSING_ENV|_check_env` under
  `src/agentops/pipeline` returns nothing, and the only socket use in the
  CLI is `_port_in_use`, which checks whether the cockpit port is taken.

* It documented `agentops eval run --dry-run` for CI gating. That flag does
  not exist on `eval run`, whose flags are `--agent --baseline --config/-c
  --format/-f --output/-o`. The only `--dry-run` in the CLI belongs to
  `telemetry dashboard deploy`. Anyone copying that line into a pipeline
  would get a hard failure. The real gate is `--strict-preflight` on
  `agentops doctor`.

The corrected text is confirmed by the sample output already present in
`docs/tutorial-prompt-agent.md`, which shows `AgentOps pre-flight 4 ok`
covering exactly those four checks.

Also adds the missing `services/preflight.py` entry to the documented
directory tree and repoints the "Where to Add New Code" row.

The same fix landed on the `docs` branch in #406. This commit keeps the
in-repo copy read by contributors from `README.md` and `CONTRIBUTING.md`
consistent with it.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: bcb9c0b6-d506-46dc-90d2-8120413166ee
@placerda
Paulo Lacerda (placerda) merged commit ce5496b into develop Aug 9, 2026
13 checks passed
@placerda
Paulo Lacerda (placerda) deleted the chore/sync-preflight-docs branch August 9, 2026 10:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant