docs: correct the pre-flight checks section in how-it-works - #407
Merged
Conversation
The "Pre-flight checks" section described a subsystem that does not exist as written. Verified against the v0.8.6 tree: * It pointed at `pipeline/runtime.py`. That module only contains evaluator loading and per-row evaluator execution (`load_evaluator`, `run_evaluator`, `_model_config`, `_extract_score`, `_is_transient_credential_error`). The real implementation is `services/preflight.py`, imported only from `cli/app.py`. * The bullet list did not match the four checks that actually run. The real display names are Workspace, Azure authentication, Foundry project and Application Insights. There is no package-presence check, no env-var aggregation check, and no TCP probe for URL agents. `git grep` for `missing_packages|required_env|MISSING_ENV|_check_env` under `src/agentops/pipeline` returns nothing, and the only socket use in the CLI is `_port_in_use`, which checks whether the cockpit port is taken. * It documented `agentops eval run --dry-run` for CI gating. That flag does not exist on `eval run`, whose flags are `--agent --baseline --config/-c --format/-f --output/-o`. The only `--dry-run` in the CLI belongs to `telemetry dashboard deploy`. Anyone copying that line into a pipeline would get a hard failure. The real gate is `--strict-preflight` on `agentops doctor`. The corrected text is confirmed by the sample output already present in `docs/tutorial-prompt-agent.md`, which shows `AgentOps pre-flight 4 ok` covering exactly those four checks. Also adds the missing `services/preflight.py` entry to the documented directory tree and repoints the "Where to Add New Code" row. The same fix landed on the `docs` branch in #406. This commit keeps the in-repo copy read by contributors from `README.md` and `CONTRIBUTING.md` consistent with it. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bcb9c0b6-d506-46dc-90d2-8120413166ee
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Corrects the Pre-flight checks section of
docs/how-it-works.md, whichdescribed a subsystem that does not exist as written.
This is the
developcounterpart of #406. That PR fixed the copy on theorphan
docsbranch, which is what the published site is built from. ThisPR fixes the copy that lives alongside the code and is linked from
README.mdandCONTRIBUTING.md.What was wrong
Verified against the
v0.8.6tree.1. Wrong module. The section pointed at
pipeline/runtime.py. Thatmodule only contains evaluator loading and per-row evaluator execution
(
load_evaluator,run_evaluator,_model_config,_extract_score,_is_transient_credential_error). The real implementation isservices/preflight.py, whoserun_preflight()is imported only fromcli/app.py.2. Wrong checks. The bullet list did not describe the checks that
actually run. The real display names are Workspace, Azure
authentication, Foundry project and Application Insights.
Negative evidence for the three documented checks that do not exist:
The only socket use in the CLI is
_port_in_use, which checks whether thecockpit port is already taken. It has nothing to do with resolving a URL
agent endpoint.
3. A flag that does not exist. The section told readers to use
agentops eval run --dry-runfor CI gating. The real flags oneval runare
--agent --baseline --config/-c --format/-f --output/-o. The only--dry-runin the whole CLI belongs totelemetry dashboard deploy.Anyone copying that line into a pipeline gets a hard failure. The real gate
is
--strict-preflightonagentops doctor.Corroboration
docs/tutorial-prompt-agent.mdalready shows the real output:covering the Workspace, Azure auth, Foundry project and App Insights
checks. The corrected text now matches that.
Also in this PR
services/preflight.pyentry to the documenteddirectory tree.
pipeline/runtime.pytoservices/preflight.py.How this was found
By resolving every documented
agentops ...invocation against the realclickcommand tree and diffing the flags. That check now reports zerosuspect flags across
README.md,src/agentops/templates/**/*.mdanddocs/**/*.mdon both this branch and thedocsbranch.