fix: honour the agent override in official eval and test the release cut - #411
Merged
Merged
Conversation
Closes five audit follow-ups that shared one root cause: behaviour that nothing exercised. #398 - the official evaluation runner ignored the agent override. prepare_official_eval read the agent name and version straight from agentops.yaml, so --agent and AGENTOPS_AGENT were accepted and then discarded. A pipeline pinning a version still evaluated whatever the config carried. Resolution now happens once in resolve_agent_override, official_eval.py applies it, and the generated GitHub Actions and Azure DevOps workflows forward it. An unexpanded CI token such as $(AGENTOPS_AGENT) is treated as absent rather than parsed as a name. #404 - the release cut logic lived as an inline Python heredoc inside cut-release.yml, where no test could import it, so a regression only surfaced while a release was being cut. That is how the 0.8.6 cut broke. The transformation moved to scripts/check_changelog.py cut, the workflow calls it, and the unit tests exercise the same code. The subcommand is idempotent and rejects a changelog with no [Unreleased] marker. #399 - the CHANGELOG gate had no fixture proving Dependabot pull requests pass. They do, but only because the EXEMPT_AUTHORS check runs before file classification. PR #359 touches pyproject.toml, which is shipping code, so reordering those checks would block every dependency bump. Three real payloads now pin that ordering. #395 - the Azure DevOps CI test covered a single eval runner. It is now parametrized over azd-ai-agent-eval, agentops-cloud and agentops-local. #396 - the safety-eval job deliberately omits environment: so it cannot be gated behind a manual approval. Both production templates carry a marker comment explaining why, and two tests read the generated file to keep it. Test plan: 1206 passed, 6 skipped. The one deselected failure, test_list_role_definition_ids_extracts_guid_suffix, is environmental (azure.mgmt is absent from the local venv) and unrelated to this change. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bcb9c0b6-d506-46dc-90d2-8120413166ee
This was referenced Aug 9, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes five audit follow-ups that shared one root cause: behaviour nothing exercised.
#398 - the official evaluation runner ignored the agent override
prepare_official_evalread the agent name and version straight fromagentops.yaml, so--agentandAGENTOPS_AGENTwere accepted and then discarded. A pipeline pinning a specific version still evaluated whatever version the config happened to carry, silently.Resolution now happens once in
resolve_agent_override,official_eval.pyapplies it, and the generated GitHub Actions and Azure DevOps workflows forward it. An unexpanded CI token such as$(AGENTOPS_AGENT)is treated as absent rather than parsed as an agent name, which is what would otherwise happen when a variable is not defined in the pipeline.#404 - the release cut logic was untestable by construction
The transformation lived as an inline Python heredoc inside
cut-release.yml. No test could import it, so a regression in it only surfaced while a release was already being cut. That is exactly how the 0.8.6 cut broke.It moved to
scripts/check_changelog.py cut. The workflow calls the script, and the unit tests exercise the same code path rather than a copy of it. The subcommand is idempotent (re-running it for a version already present is a no-op) and raises on a changelog with no## [Unreleased]marker.#399 - the CHANGELOG gate had no Dependabot fixture
Dependabot pull requests do pass the gate, but only because the
EXEMPT_AUTHORScheck runs before file classification. PR #359 touches onlypyproject.toml, which is shipping code. Reordering those two checks would block every dependency bump and nothing would have caught it. Three real payloads (#362, #358, #359) now pin that ordering.Part 2 of the issue (branch protection on
develop) needs admin rights and is a policy decision, so it stays open.#395 - the Azure DevOps CI test covered one eval runner
Now parametrized over
azd-ai-agent-eval,agentops-cloudandagentops-local.#396 - the missing
environment:looked like an oversightThe
safety-evaljob omits it deliberately, so the job cannot be gated behind a manual approval. Both production templates carry a marker comment explaining why, and two tests read the generated file to keep it there.Test plan
1206 passed, 6 skipped.One test is deselected locally:
test_list_role_definition_ids_extracts_guid_suffixfails withAttributeError: module 'azure' has no attribute 'mgmt'. That is a missing optional dependency in my local venv, not a code change. It is untouched by this PR and should pass in CI.