From ec4f9565b56753b094e1c7ec4c1c4b23f40da984 Mon Sep 17 00:00:00 2001 From: Paulo Lacerda Date: Sun, 9 Aug 2026 10:11:22 -0300 Subject: [PATCH 1/2] feat: give agents a Microsoft Entra identity across register, trace and evidence Closes #353, #354, #355. A trace could say what an agent did but not which registered agent did it. Nothing in the release evidence pack tied runtime behaviour back to an accountable owner in the tenant, so "who owns this agent" was answerable only out of band. These three issues asked for the same loop from different ends, and they only make sense implemented together. The loop has three points. Register. `agentops agent register --sponsor ` creates or adopts a Microsoft Entra agent identity blueprint and writes the resolved id to `.agentops/identity/agent-identity.json`. It is idempotent: an existing blueprint with the same display name is adopted rather than duplicated. A sponsor is mandatory, because an identity with no named owner is not governable, which is the whole point of registering it. `--dry-run` resolves and prints the inputs without contacting Microsoft Graph. Stamp. `_resource_attributes()` in the telemetry bootstrap resolves the id and sets it as the OpenTelemetry resource attribute `gen_ai.agent.id` on every span. When no identity is registered the attribute is omitted entirely rather than set to an empty string, so its presence is itself a usable filter in a trace query. Publish. The release evidence pack gains an `agent_identity` section reporting the id and whether it came from the local record or the environment. A missing identity produces a warning, not a blocker, because adoption has to be incremental. A read-only Doctor check reports registration posture under the Security category. It resolves the local record first, at zero cost, and only contacts Microsoft Graph when `identity.verify` is set in `agentops.yaml`. Four design decisions deviate from the literal issue text, each forced by the existing code rather than chosen: - The posture check is standalone rather than a `RULE_REGISTRY` rule. Every rule in that registry takes `(payload: AzureResourcesPayload, source_name)`, and `run_posture_check` returns early when `resources.account is None`, so a registry rule would silently never run for a workspace with no Azure account resolved. It follows the `governance.py` standalone pattern and is called from `analyzer.py`. - Registration is `agentops agent register`, not a step inside `agentops ship`. There is no `ship` command. Ship is a pillar, documented in `docs/ship.md` and materialised as workflow templates. The templates now carry the step, gated on the `AGENTOPS_IDENTITY_SPONSOR` repository variable so it stays opt-in: it writes to a tenant, which should never be the default behaviour of a generated workflow. - Findings use `Category.SECURITY`. There is no `governance` member in the `Category` enum, and adding one would reshape every consumer of the catalogue for a single check. - `agentops.yaml` needed a typed `AgentIdentityConfig` submodel rather than a free-form dict. `AgentOpsConfig` sets `extra="forbid"`, so the `identity` key was rejected outright until it was declared. A permissive `Dict[str, Any]` would have accepted `sponser:` silently and failed much later with a misleading "sponsor is required", so the submodel also forbids extras. The Agent 365 ingestion API is not public, so nothing here depends on it. The handshake is documented instead, in `docs/observe.md`, so the id can be correlated by whoever consumes the traces. 65 unit tests added across six files. Full suite: 1271 passed, 6 skipped. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bcb9c0b6-d506-46dc-90d2-8120413166ee --- CHANGELOG.md | 17 + docs/doctor-checks.md | 14 +- docs/observe.md | 68 ++ docs/ship.md | 22 + src/agentops/agent/analyzer.py | 2 + src/agentops/agent/checks/agent_identity.py | 144 +++++ src/agentops/agent/checks/catalog.py | 53 ++ src/agentops/cli/app.py | 130 +++- src/agentops/core/agentops_config.py | 59 ++ src/agentops/core/release_evidence.py | 1 + src/agentops/services/agent_identity.py | 580 ++++++++++++++++++ src/agentops/services/evidence_pack.py | 75 +++ .../workflows/agentops-deploy-prod-azd.yml | 19 + .../workflows/agentops-deploy-prod.yml | 21 + src/agentops/utils/telemetry.py | 51 +- .../unit/test_agent_checks_agent_identity.py | 123 ++++ tests/unit/test_agent_identity_service.py | 284 +++++++++ tests/unit/test_agentops_config_identity.py | 66 ++ tests/unit/test_cli_agent_register.py | 148 +++++ .../unit/test_evidence_pack_agent_identity.py | 111 ++++ tests/unit/test_telemetry_agent_id.py | 70 +++ 21 files changed, 2042 insertions(+), 16 deletions(-) create mode 100644 src/agentops/agent/checks/agent_identity.py create mode 100644 src/agentops/services/agent_identity.py create mode 100644 tests/unit/test_agent_checks_agent_identity.py create mode 100644 tests/unit/test_agent_identity_service.py create mode 100644 tests/unit/test_agentops_config_identity.py create mode 100644 tests/unit/test_cli_agent_register.py create mode 100644 tests/unit/test_evidence_pack_agent_identity.py create mode 100644 tests/unit/test_telemetry_agent_id.py diff --git a/CHANGELOG.md b/CHANGELOG.md index bd9cb6c7..50830743 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,23 @@ This format follows [Keep a Changelog](https://keepachangelog.com/) and adheres ## [Unreleased] +### Added +- **Agents can now carry a first-class Microsoft Entra identity that travels + from registration through traces into release evidence.** Before this, a + trace could tell you what an agent did but not which registered agent did it, + so nothing in the evidence pack tied runtime behaviour back to an accountable + owner in the tenant. Three pieces close that loop. `agentops agent register` + creates or adopts an agent identity blueprint in Microsoft Entra (idempotent, + sponsor required, `--dry-run` supported) and records the resolved id under + `.agentops/identity/agent-identity.json`. AgentOps then stamps that id on + every span it emits as the OpenTelemetry resource attribute + `gen_ai.agent.id`, omitting the attribute entirely when no identity is + registered so presence is a meaningful filter. Finally, the release evidence + pack publishes an `agent_identity` section reporting the id and its source. + A read-only Doctor check reports registration posture, contacting Microsoft + Graph only when `identity.verify` is enabled in `agentops.yaml`. `agentops.yaml` + accepts a new optional `identity` block (`display_name`, `sponsor`, `verify`). + ### Fixed - **The official evaluation runner now honours the agent version override.** `prepare_official_eval` read the agent name and version straight from diff --git a/docs/doctor-checks.md b/docs/doctor-checks.md index 16d85734..4ba7e833 100644 --- a/docs/doctor-checks.md +++ b/docs/doctor-checks.md @@ -33,7 +33,7 @@ Two conventions across the table: ## Data sources -The Doctor reaches Azure through five sources, all configured in +The Doctor reaches Azure through six sources, all configured in `.agentops/agent.yaml`: | Source | Reads | @@ -43,6 +43,7 @@ The Doctor reaches Azure through five sources, all configured in | `azure_monitor` | Application Insights / Log Analytics via REST (KQL) | | `foundry_control` | Foundry project / agents / evaluation rules via `azure-ai-projects` | | `azure_resources` | Cognitive Services account properties via `azure-mgmt-cognitiveservices`; inferred from explicit config, AZD `.azure//.env` when present, or Foundry endpoint/account matching | +| `graph` | Microsoft Entra agent identity blueprints via Microsoft Graph, read-only, and only when `identity.verify` is enabled | The LLM-judged rules additionally use the Foundry project's OpenAI client (auto-discovered) as the judge model. @@ -64,6 +65,17 @@ stopping the whole run. | `waf.security.diagnostic_settings` | warning | `azure_resources` | programmatic | account has ≥1 diagnostic setting with a `workspace_id` | | `safety.runtime.content_filter` | warning | `azure_monitor` | programmatic | KQL hits on `gen_ai.response.finish_reasons contains content_filter` | | `responsible_ai.llm.prompt_jailbreak_surface` | info / warning | `foundry_control` | llm-judged | judge model scans system prompt for override-phrasing, embedded secrets, unbounded role-play | +| `agent_identity.not_registered` | warning | `graph` | programmatic | no Entra Agent ID in `.agentops/identity/agent-identity.json` or `AGENTOPS_ENTRA_AGENT_ID` | +| `agent_identity.not_recorded` | info | `graph` | programmatic | `identity.verify: true` and Graph finds a blueprint, but no local record exists | +| `agent_identity.lookup_failed` | warning | `graph` | programmatic | `identity.verify: true` and the Graph lookup failed (missing consent, throttling, network) | + +!!! info "Identity checks are read-only" + The `agent_identity.*` checks never create or modify anything in Microsoft + Entra. Registration is an explicit, separate action: `agentops agent + register`. Graph is contacted only when `identity.verify` is set to true in + `agentops.yaml`; with it off, the check reads local state only and still + reports whether an identity exists. See + [Agent identity on traces](observe.md#agent-identity-on-traces). ### ⚙️ Operational Excellence diff --git a/docs/observe.md b/docs/observe.md index 20b4e49a..77ae26fd 100644 --- a/docs/observe.md +++ b/docs/observe.md @@ -66,6 +66,74 @@ with no monitoring does not look healthy simply because nothing is being graded. is intended: a real release should investigate latency and errors before promoting, even when the candidate's eval scores pass. +## Agent identity on traces + +Traces tell you what an agent did. They do not, by default, tell you *which* +agent did it in a way an auditor can reconcile with your tenant. Microsoft +Entra Agent ID closes that gap: the agent gets a first-class identity, and the +same identifier travels from registration through traces into release evidence. + +The handshake has three steps, and each one is a different tool, so it is worth +being explicit about who writes what. + +**1. Register the identity.** `agentops agent register` creates (or adopts) an +agent identity blueprint in Microsoft Entra and records the resolved +application id locally: + +```bash +agentops agent register --sponsor owner@contoso.com +``` + +The sponsor is required. An agent identity with no accountable owner cannot be +governed, so there is no default. The command is idempotent: if a blueprint +with the same display name already exists, AgentOps reuses it instead of +creating a duplicate. Run it with `--dry-run` first to see the resolved display +name and sponsor without calling Microsoft Graph. + +The resolved id is written to `.agentops/identity/agent-identity.json`. Declare +the inputs in `agentops.yaml` so they are source-controlled: + +```yaml +identity: + display_name: support-agent + sponsor: owner@contoso.com + verify: true +``` + +`verify: true` tells the Doctor to confirm the blueprint against Microsoft +Graph. It is off by default because that lookup needs tenant admin consent +(`AgentIdentityBlueprint.Read.All`), which most workspaces will not have on day +one. With it off, the Doctor still reports whether an identity is registered at +all, using only local state. + +**2. Stamp it on traces.** Once an identity is resolved, AgentOps adds it to +the OpenTelemetry resource as `gen_ai.agent.id`, so every span AgentOps emits +carries the Entra Agent ID. In CI, where the local record is not checked in, +set `AGENTOPS_ENTRA_AGENT_ID` instead and the attribute resolves from the +environment. + +The attribute is **omitted** when no identity is registered, never emitted as an +empty string. That distinction matters when you query: filtering on presence +tells you which traffic is attributable and which is not. + +```kusto +dependencies +| where isnotempty(customDimensions["gen_ai.agent.id"]) +| summarize runs = count() by tostring(customDimensions["gen_ai.agent.id"]) +``` + +**3. Publish it as evidence.** The release evidence pack reads the same record +and adds an `agent_identity` section reporting the id and where it came from +(the local record or the environment variable). When no identity is registered, +the pack raises a warning rather than a blocker, because identity registration +is a governance improvement rather than a correctness gate. + +!!! note "AgentOps does not ingest into Agent 365" + There is no public ingestion API for Agent 365 telemetry today. AgentOps + stamps the identifier and publishes it as evidence so the correlation is + possible from the Azure Monitor side. It does not push traces into Agent + 365. + ## Trace-to-regression promotion The strongest use of observability is turning real production behavior into new diff --git a/docs/ship.md b/docs/ship.md index adfe26b8..c7ec90be 100644 --- a/docs/ship.md +++ b/docs/ship.md @@ -158,6 +158,28 @@ long steps here: - [GitHub OIDC with Azure (workload identity federation)](https://learn.microsoft.com/azure/active-directory/workload-identities/workload-identity-federation-create-trust?pivots=identity-wif-apps-methods-azp) - [Assign Azure roles (RBAC)](https://learn.microsoft.com/azure/role-based-access-control/role-assignments-portal) +### Giving the agent its own identity + +The OIDC principal above answers "which pipeline is deploying". It does not +answer "which agent is running in production", and that second question is the +one an auditor asks. A Microsoft Entra agent identity closes the gap: the agent +gets its own directory object with a named human sponsor, and AgentOps then +stamps that id on every trace and republishes it in the release evidence. + +Registration is a one-line command and it is idempotent, so re-running it adopts +the existing blueprint instead of creating a duplicate: + +```bash +agentops agent register --sponsor owner@contoso.com +``` + +The generated production workflows carry the same step, disabled by default. +Set the `AGENTOPS_IDENTITY_SPONSOR` repository variable to the sponsor's UPN and +the step turns on. It stays opt-in because it writes to your tenant, which needs +a deliberate decision rather than a default. The full handshake, the +`identity` block in `agentops.yaml`, and the trace query are documented in +[Agent identity on traces](observe.md#agent-identity-on-traces). + ## Try it Generate the CI/CD workflows from the same analysis AgentOps uses, smallest gate diff --git a/src/agentops/agent/analyzer.py b/src/agentops/agent/analyzer.py index b1ae3082..57e8c762 100644 --- a/src/agentops/agent/analyzer.py +++ b/src/agentops/agent/analyzer.py @@ -10,6 +10,7 @@ from agentops.agent.checks.errors import run_errors_check from agentops.agent.checks.foundry_config import run_foundry_config_check +from agentops.agent.checks.agent_identity import run_agent_identity_check from agentops.agent.checks.governance import run_governance_check from agentops.agent.checks.latency import run_latency_check from agentops.agent.checks.observability import run_observability_check @@ -151,6 +152,7 @@ def analyze( findings.extend(run_rbac_openai_data_plane_check(resources)) findings.extend(run_opex_workspace_check(workspace)) findings.extend(run_governance_check(workspace)) + findings.extend(run_agent_identity_check(workspace)) findings.extend(run_observability_check(workspace)) findings.extend(run_opex_check(history, config.checks.opex)) findings.extend(run_release_readiness_check(workspace, history, foundry)) diff --git a/src/agentops/agent/checks/agent_identity.py b/src/agentops/agent/checks/agent_identity.py new file mode 100644 index 00000000..fc16b9d8 --- /dev/null +++ b/src/agentops/agent/checks/agent_identity.py @@ -0,0 +1,144 @@ +"""Agent 365 registration posture check. + +``agentops doctor`` scores a workspace against Well-Architected rules, but it +had no visibility into whether the agent exists as a first-class identity in +Microsoft Entra. Without an agent identity blueprint the agent cannot be +governed by Microsoft Agent 365: it does not appear in the agent inventory, +Conditional Access cannot target it, and its traces cannot be correlated back +to an accountable owner. + +The check is deliberately read-only and cheap: + +* it first resolves the identity from the workspace record or the + ``AGENTOPS_ENTRA_AGENT_ID`` environment variable, which costs nothing, +* it only calls Microsoft Graph when the workspace opts in via + ``identity.verify: true``, because the lookup needs tenant admin consent + that most workspaces will not have on day one, +* every Graph failure becomes a warning with a readable sentence, never a + stack trace. +""" + +from __future__ import annotations + +from pathlib import Path +from typing import Any, List + +from agentops.agent.findings import Category, Finding, Severity +from agentops.services.agent_identity import ( + REGISTRATION_DOCS_URL, + AgentIdentityError, + load_identity_config, + lookup_blueprint, + resolve_agent_id, + resolve_display_name, +) + +SOURCE_NAME = "agent_identity" + + +def run_agent_identity_check(workspace: Path) -> List[Finding]: + """Report whether the agent is registered in Microsoft Agent 365.""" + + workspace = Path(workspace) + identity = load_identity_config(workspace) + display_name = resolve_display_name(workspace) or workspace.resolve().name + + if resolve_agent_id(workspace): + return [] + + if _verify_enabled(identity): + try: + blueprint = lookup_blueprint(display_name) + except AgentIdentityError as exc: + return [_lookup_failed(display_name, str(exc))] + if blueprint is not None: + return [_registered_but_unrecorded(blueprint.app_id, display_name)] + + return [_not_registered(display_name)] + + +# --------------------------------------------------------------------------- +# Findings +# --------------------------------------------------------------------------- + + +def _not_registered(display_name: str) -> Finding: + return Finding( + id="agent_identity.not_registered", + severity=Severity.WARNING, + category=Category.SECURITY, + title="Agent is not registered in Microsoft Agent 365", + summary=( + "No Entra Agent ID is recorded for this workspace, so the agent has " + "no first-class identity in Microsoft Agent 365. Without it the agent " + "is absent from the tenant agent inventory, Conditional Access cannot " + "target it, and its traces cannot be attributed to an accountable owner." + ), + recommendation=( + "Register the agent identity blueprint with " + "'agentops agent register --sponsor '. Set " + "'identity.sponsor' in agentops.yaml first so the registration is " + f"reproducible in CI. Background: {REGISTRATION_DOCS_URL}" + ), + source=SOURCE_NAME, + evidence={"display_name": display_name, "registered": False}, + ) + + +def _registered_but_unrecorded(app_id: str, display_name: str) -> Finding: + return Finding( + id="agent_identity.not_recorded", + severity=Severity.INFO, + category=Category.SECURITY, + title="Agent identity exists in Entra but is not recorded locally", + summary=( + f"Microsoft Entra has an agent identity blueprint named " + f"'{display_name}', but this workspace has no local record of it. " + "Traces and the release evidence bundle therefore cannot quote the " + "Entra Agent ID." + ), + recommendation=( + "Run 'agentops agent register' to adopt the existing blueprint into " + "this workspace. The command is idempotent and will reuse the " + "blueprint instead of creating a duplicate." + ), + source=SOURCE_NAME, + evidence={"display_name": display_name, "app_id": app_id, "registered": True}, + ) + + +def _lookup_failed(display_name: str, reason: str) -> Finding: + return Finding( + id="agent_identity.lookup_failed", + severity=Severity.WARNING, + category=Category.SECURITY, + title="Agent 365 registration could not be verified", + summary=( + "AgentOps could not confirm whether this agent has an Entra Agent ID. " + f"{reason}" + ), + recommendation=( + "Grant the AgentIdentityBlueprint.Read.All application permission and " + "admin consent, or set 'identity.verify: false' in agentops.yaml to " + "rely on the locally recorded identity instead." + ), + source=SOURCE_NAME, + evidence={"display_name": display_name, "reason": reason}, + ) + + +# --------------------------------------------------------------------------- +# Config helpers +# --------------------------------------------------------------------------- + + +def _verify_enabled(identity: dict[str, Any]) -> bool: + value = identity.get("verify") + if isinstance(value, bool): + return value + if isinstance(value, str): + return value.strip().lower() in {"1", "true", "yes", "on"} + return False + + +__all__ = ["SOURCE_NAME", "run_agent_identity_check"] diff --git a/src/agentops/agent/checks/catalog.py b/src/agentops/agent/checks/catalog.py index 9f7ff9f9..b25a22ae 100644 --- a/src/agentops/agent/checks/catalog.py +++ b/src/agentops/agent/checks/catalog.py @@ -35,6 +35,7 @@ "azure_resources": "Azure resources (ARM)", "spec_workspace": "spec docs (.specify / AGENTS.md)", "judge_model": "judge model deployment", + "graph": "Microsoft Graph (Entra agent identity)", } SOURCE_DESCRIPTIONS: Dict[str, str] = { @@ -82,6 +83,13 @@ "checks. It reviews semantic signals like prompt guardrails, dataset " "PII risk, bundle coverage, and spec-vs-implementation gaps." ), + "graph": ( + "Microsoft Graph v1.0, read-only, used to confirm that the agent has " + "an Entra agent identity blueprint in Microsoft Agent 365. This source " + "is opt-in: set `identity.verify: true` in `agentops.yaml`. It needs " + "the `AgentIdentityBlueprint.Read.All` application permission with " + "tenant admin consent." + ), } # Recognized check flags. Keep this list short and stable. @@ -150,6 +158,15 @@ "observability.trace_replay_missing": ( "https://learn.microsoft.com/azure/foundry/concepts/observability" ), + "agent_identity.not_registered": ( + "https://learn.microsoft.com/entra/identity/agent-id/agent-id-overview" + ), + "agent_identity.not_recorded": ( + "https://learn.microsoft.com/entra/identity/agent-id/agent-id-overview" + ), + "agent_identity.lookup_failed": ( + "https://learn.microsoft.com/entra/identity/agent-id/agent-id-overview" + ), } @@ -687,6 +704,42 @@ def is_llm_judged(self) -> bool: # ------------------------------------------------------------------ # Security # ------------------------------------------------------------------ + CheckSpec( + id="agent_identity.not_registered", + category=Category.SECURITY, + title="Agent is not registered in Microsoft Agent 365", + summary=( + "No Entra Agent ID is recorded for the workspace, so the " + "agent is absent from the tenant agent inventory and cannot " + "be targeted by Conditional Access." + ), + severities=(Severity.WARNING,), + requires=("workspace",), + ), + CheckSpec( + id="agent_identity.not_recorded", + category=Category.SECURITY, + title="Agent identity exists in Entra but is not recorded locally", + summary=( + "Microsoft Entra has a matching agent identity blueprint but " + "the workspace has no local record, so traces and the release " + "evidence bundle cannot quote the Entra Agent ID." + ), + severities=(Severity.INFO,), + requires=("workspace", "graph"), + ), + CheckSpec( + id="agent_identity.lookup_failed", + category=Category.SECURITY, + title="Agent 365 registration could not be verified", + summary=( + "Microsoft Graph could not be consulted, usually because the " + "AgentIdentityBlueprint.Read.All permission has no admin " + "consent in the tenant." + ), + severities=(Severity.WARNING,), + requires=("workspace", "graph"), + ), CheckSpec( id="waf.security.local_auth_disabled", category=Category.SECURITY, diff --git a/src/agentops/cli/app.py b/src/agentops/cli/app.py index b509ab03..539d72ed 100644 --- a/src/agentops/cli/app.py +++ b/src/agentops/cli/app.py @@ -935,8 +935,42 @@ class ExplainPage: title="Agent server commands", command="agentops agent", synopsis=("agentops agent COMMAND [ARGS]...", "agentops agent explain"), - summary=("Contains commands that host AgentOps Doctor as an HTTP agent/Copilot Extension surface.",), - children=("serve",), + summary=("Contains commands that host AgentOps Doctor as an HTTP agent/Copilot Extension surface, and that manage the agent's Microsoft Entra identity.",), + children=("serve", "register"), + ), + ("agent", "register"): ExplainPage( + title="Register the agent identity blueprint", + command="agentops agent register", + synopsis=( + "agentops agent register [--sponsor UPN_OR_ID] [--display-name NAME] [--workspace PATH] [--dry-run]", + "agentops agent register explain", + ), + summary=( + "Creates the agent's identity blueprint in Microsoft Entra so the agent becomes a governed principal in Microsoft Agent 365.", + "Registration is what makes an agent visible in the tenant agent inventory, targetable by Conditional Access, and attributable to an accountable sponsor. Until it exists, the agent is just a workload with no identity of its own.", + "The command is idempotent: it looks the blueprint up by display name first and adopts the existing one instead of creating a duplicate.", + ), + how_it_works=( + "Resolves the display name from `--display-name`, then `identity.display_name` in `agentops.yaml`, then the agent target name.", + "Resolves the sponsor from `--sponsor` or `identity.sponsor` in `agentops.yaml`. A sponsor is mandatory: Agent 365 requires an accountable human owner.", + "Acquires an app-only Microsoft Graph token through the shared Azure credential chain.", + "Queries `GET /applications` filtered by display name. If a blueprint already exists it is reused.", + "Otherwise it POSTs a `Microsoft.Graph.AgentIdentityBlueprint` application to Graph v1.0.", + "Writes `.agentops/identity/agent-identity.json` so Doctor, the OTel exporter, and the release evidence pack can all quote the same Entra Agent ID.", + ), + inputs=( + "`agentops.yaml` keys `identity.sponsor`, `identity.display_name`, `identity.owner`.", + "Microsoft Graph application permission `AgentIdentityBlueprint.Create` with tenant admin consent.", + ), + outputs=( + "`.agentops/identity/agent-identity.json` containing `app_id`, `object_id`, and `display_name`.", + "The Entra Agent ID echoed to stdout for use in CI logs.", + ), + examples=( + "agentops agent register --sponsor jane@contoso.com", + "agentops agent register --dry-run", + ), + see_also=("agentops explain doctor", "agentops explain telemetry"), ), ("agent", "serve"): ExplainPage( title="Serve AgentOps as an HTTP agent", @@ -4791,6 +4825,7 @@ def _build_doctor_explain_text( "azure_monitor", "azure_resources", "judge_model", + "graph", ] lines: list[str] = _manual_banner( "AgentOps Doctor", @@ -5015,6 +5050,7 @@ def _build_doctor_explain_markdown( "azure_monitor", "azure_resources", "judge_model", + "graph", ] lines: list[str] = [ "# AgentOps Doctor manual", @@ -6159,6 +6195,96 @@ def cmd_agent_serve( uvicorn.run(fastapi_app, host=host, port=port, workers=workers) +@agent_app.command("register") +def cmd_agent_register( + sponsor: Annotated[ + str | None, + typer.Option( + "--sponsor", + help=( + "Accountable owner (UPN or object id). Falls back to " + "`identity.sponsor` in agentops.yaml." + ), + ), + ] = None, + display_name: Annotated[ + str | None, + typer.Option( + "--display-name", + help=( + "Blueprint display name. Falls back to " + "`identity.display_name`, then the agent target name." + ), + ), + ] = None, + workspace: Annotated[ + Path, + typer.Option("--workspace", "-w", help="Project root."), + ] = Path("."), + dry_run: Annotated[ + bool, + typer.Option( + "--dry-run", + help="Resolve inputs and report what would happen, without calling Graph.", + ), + ] = False, + explain: Annotated[str | None, typer.Argument(hidden=True)] = None, +) -> None: + """Register the agent's identity blueprint in Microsoft Entra. + + Idempotent: an existing blueprint with the same display name is + adopted rather than duplicated. The resolved Entra Agent ID is + written to ``.agentops/identity/agent-identity.json`` so Doctor, + tracing, and the release evidence pack all quote the same value. + """ + if _maybe_explain_leaf(("agent", "register"), explain): + return + + from agentops.services.agent_identity import ( + AgentIdentityError, + register_blueprint, + resolve_registration_inputs, + write_identity_record, + ) + + workspace = workspace.resolve() + + try: + resolved_name, resolved_sponsor = resolve_registration_inputs( + workspace, + display_name=display_name, + sponsor=sponsor, + ) + except AgentIdentityError as exc: + typer.echo(f"{_cli_error('Error')}: {exc}", err=True) + raise typer.Exit(code=1) from exc + + typer.echo(f"{_cli_label('Display name')}: {resolved_name}") + typer.echo(f"{_cli_label('Sponsor')}: {resolved_sponsor}") + + if dry_run: + typer.echo( + "Dry run: no Microsoft Graph call was made. " + "Re-run without --dry-run to register." + ) + return + + try: + blueprint, created = register_blueprint( + resolved_name, sponsor=resolved_sponsor + ) + except AgentIdentityError as exc: + typer.echo(f"{_cli_error('Error')}: {exc}", err=True) + raise typer.Exit(code=1) from exc + + record_path = write_identity_record(workspace, blueprint, created=created) + + action = "Registered" if created else "Reused existing" + typer.echo(f"{_cli_label(action + ' agent identity')}: {blueprint.app_id}") + typer.echo(f"{_cli_label('Wrote')}: {_cli_path(record_path)}") + typer.echo(f"{_cli_label('Entra portal')}: {blueprint.portal_url}") + + @app.command("cockpit") def cmd_cockpit( host: Annotated[ diff --git a/src/agentops/core/agentops_config.py b/src/agentops/core/agentops_config.py index 7be891c8..2728a374 100644 --- a/src/agentops/core/agentops_config.py +++ b/src/agentops/core/agentops_config.py @@ -733,6 +733,50 @@ class RedTeamRunConfig(BaseModel): model_config = ConfigDict(extra="forbid") +class AgentIdentityConfig(BaseModel): + """Optional Microsoft Agent 365 identity settings. + + Declares the agent's first-class identity in Microsoft Entra so Doctor, + tracing, and the release evidence pack all reference the same Entra Agent + ID. AgentOps never invents these values: ``agentops agent register`` + creates or adopts the blueprint and records the resolved id under + ``.agentops/identity/agent-identity.json``. + + Example:: + + identity: + display_name: support-agent + sponsor: owner@contoso.com + verify: true + """ + + display_name: Optional[str] = Field( + None, + description=( + "Blueprint display name in Microsoft Entra. When omitted, " + "AgentOps derives one from the 'agent' target." + ), + ) + sponsor: Optional[str] = Field( + None, + description=( + "Accountable owner (UPN or object id). Required to register a " + "blueprint; there is no silent fallback because an unowned agent " + "identity cannot be governed." + ), + ) + verify: bool = Field( + False, + description=( + "When true, Doctor calls Microsoft Graph to confirm the blueprint " + "exists. Off by default because the lookup needs tenant admin " + "consent that most workspaces will not have on day one." + ), + ) + + model_config = ConfigDict(extra="forbid") + + class StreamConfig(BaseModel): """Streaming aggregation options for ``http-json`` targets. @@ -867,6 +911,13 @@ class AgentOpsConfig(BaseModel): single-turn evals working while letting Doctor, Cockpit, CI evidence, and azd/Foundry recipes reason about multi-turn coverage, rubric gates, trace sampling, and trace replay links. + + ``identity`` + Optional Microsoft Agent 365 identity settings (display name, sponsor, + and whether Doctor verifies the blueprint against Microsoft Graph). The + resolved Entra Agent ID itself is not stored here; it lives in + ``.agentops/identity/agent-identity.json`` after + ``agentops agent register``. """ version: int = Field(..., description="Schema version. Must be 1.") @@ -932,6 +983,14 @@ class AgentOpsConfig(BaseModel): "redteam_path automatically." ), ) + identity: Optional[AgentIdentityConfig] = Field( + None, + description=( + "Optional Microsoft Agent 365 identity settings used by 'agentops " + "agent register', the Doctor registration posture check, trace " + "stamping, and the release evidence pack." + ), + ) thresholds: Dict[str, Any] = Field( default_factory=dict, diff --git a/src/agentops/core/release_evidence.py b/src/agentops/core/release_evidence.py index 06778399..33f094d5 100644 --- a/src/agentops/core/release_evidence.py +++ b/src/agentops/core/release_evidence.py @@ -54,5 +54,6 @@ class ReleaseEvidence(BaseModel): observability: Dict[str, Any] = Field(default_factory=dict) ailz: Dict[str, Any] = Field(default_factory=dict) governance: Dict[str, Any] = Field(default_factory=dict) + agent_identity: Dict[str, Any] = Field(default_factory=dict) model_config = ConfigDict(extra="forbid") diff --git a/src/agentops/services/agent_identity.py b/src/agentops/services/agent_identity.py new file mode 100644 index 00000000..3c3f29b7 --- /dev/null +++ b/src/agentops/services/agent_identity.py @@ -0,0 +1,580 @@ +"""Microsoft Entra Agent ID support for the Agent 365 control plane. + +An agent produced by the accelerator is invisible to the Agent 365 control +plane until an *agent identity blueprint* exists for it in Microsoft Entra. +This module owns the three things AgentOps needs around that identity: + +* a very small app-only Microsoft Graph client (:class:`GraphClient`), +* create/lookup of the blueprint (:func:`lookup_blueprint`, + :func:`register_blueprint`), +* on-disk persistence of the resulting ``appId`` so the doctor check, the + OTel resource attributes, and the release evidence bundle can all quote the + same value without re-calling Graph. + +Design notes +------------ + +**Read-only by default.** Nothing here runs implicitly. The doctor check does +a read-only lookup, and registration only happens when the operator asks for +it explicitly. + +**No new dependency.** Graph is a plain REST API, so we use +:mod:`urllib.request` from the standard library rather than pulling in an SDK. +The credential comes from the same shared factory the doctor sources use. + +**Errors are messages, not stack traces.** Every failure mode a user can +realistically hit (missing consent, missing role, no credential, package not +installed) is converted into an :class:`AgentIdentityError` carrying a +sentence that says what to do next. +""" + +from __future__ import annotations + +import json +import logging +import os +import urllib.error +import urllib.parse +import urllib.request +from dataclasses import dataclass +from datetime import datetime, timezone +from pathlib import Path +from typing import Any, Mapping, Optional + +log = logging.getLogger(__name__) + +#: Graph endpoint AgentOps talks to. v1.0 only - the ``AgentIdentity*`` +#: scopes this module needs are published there. +GRAPH_BASE_URL = "https://graph.microsoft.com/v1.0" + +#: App-only token scope for Microsoft Graph. +GRAPH_SCOPE = "https://graph.microsoft.com/.default" + +#: OData type discriminator for an agent identity blueprint. +BLUEPRINT_ODATA_TYPE = "Microsoft.Graph.AgentIdentityBlueprint" + +#: OTel resource attribute carrying the Entra Agent ID. +AGENT_ID_ATTRIBUTE = "gen_ai.agent.id" + +#: Environment override for the Entra Agent ID. CI exports this when the +#: identity is provisioned outside the workspace (for example by a platform +#: team) so traces still carry the right value without a local record. +AGENT_ID_ENV = "AGENTOPS_ENTRA_AGENT_ID" + +#: Where the resolved identity is persisted inside a workspace. +IDENTITY_RECORD_RELPATH = Path(".agentops") / "identity" / "agent-identity.json" + +#: Documentation pointer used in remediation text. +REGISTRATION_DOCS_URL = ( + "https://learn.microsoft.com/entra/identity/agent-id/agent-id-overview" +) + +#: Deep link template for an application object in the Entra admin center. +ENTRA_APP_DEEPLINK = ( + "https://entra.microsoft.com/#view/Microsoft_AAD_RegisteredApps" + "/ApplicationMenuBlade/~/Overview/appId/{app_id}" +) + + +class AgentIdentityError(RuntimeError): + """A user-actionable failure while talking to Microsoft Graph. + + The message is expected to be shown verbatim to the operator, so it + always explains the remediation rather than quoting an HTTP status. + """ + + +@dataclass(frozen=True) +class AgentIdentityBlueprint: + """An agent identity blueprint as Agent 365 sees it.""" + + app_id: str + object_id: Optional[str] = None + display_name: Optional[str] = None + + @property + def portal_url(self) -> str: + return ENTRA_APP_DEEPLINK.format(app_id=self.app_id) + + def to_dict(self) -> dict[str, Any]: + return { + "app_id": self.app_id, + "object_id": self.object_id, + "display_name": self.display_name, + "portal_url": self.portal_url, + } + + +# --------------------------------------------------------------------------- +# Graph client +# --------------------------------------------------------------------------- + + +class GraphClient: + """Minimal app-only Microsoft Graph client. + + Only the two verbs this feature needs are implemented. Tests inject a + stand-in with the same ``get``/``post`` shape rather than patching + :mod:`urllib`. + """ + + def __init__( + self, + *, + token: Optional[str] = None, + base_url: str = GRAPH_BASE_URL, + timeout: int = 30, + ) -> None: + self._token = token + self._base_url = base_url.rstrip("/") + self._timeout = timeout + + # -- public API ------------------------------------------------------ + + def get(self, path: str, *, params: Optional[Mapping[str, str]] = None) -> Any: + url = self._url(path) + if params: + url = f"{url}?{urllib.parse.urlencode(dict(params))}" + return self._request("GET", url, body=None) + + def post(self, path: str, body: Mapping[str, Any]) -> Any: + return self._request("POST", self._url(path), body=dict(body)) + + # -- internals ------------------------------------------------------- + + def _url(self, path: str) -> str: + return f"{self._base_url}/{path.lstrip('/')}" + + def _headers(self) -> dict[str, str]: + return { + "Authorization": f"Bearer {self._resolve_token()}", + "Accept": "application/json", + "Content-Type": "application/json", + # Agent identity blueprints are an OData-typed resource; Graph + # rejects the payload without the version header. + "OData-Version": "4.0", + } + + def _resolve_token(self) -> str: + if self._token: + return self._token + self._token = acquire_graph_token() + return self._token + + def _request(self, method: str, url: str, *, body: Optional[dict[str, Any]]) -> Any: + data = json.dumps(body).encode("utf-8") if body is not None else None + request = urllib.request.Request( + url, data=data, method=method, headers=self._headers() + ) + try: + with urllib.request.urlopen(request, timeout=self._timeout) as response: + raw = response.read().decode("utf-8") + except urllib.error.HTTPError as exc: + raise _http_error(exc) from None + except urllib.error.URLError as exc: + raise AgentIdentityError( + "Could not reach Microsoft Graph " + f"({exc.reason}). Check network connectivity or proxy settings." + ) from None + if not raw.strip(): + return {} + try: + return json.loads(raw) + except json.JSONDecodeError: + raise AgentIdentityError( + "Microsoft Graph returned a response that is not valid JSON." + ) from None + + +def _http_error(exc: urllib.error.HTTPError) -> AgentIdentityError: + """Translate an HTTP failure into an actionable message.""" + + detail = _graph_error_message(exc) + status = exc.code + if status in (401,): + return AgentIdentityError( + "Microsoft Graph rejected the credential (401). Sign in with an " + "identity that has app-only access to Graph, or set the " + "AZURE_CLIENT_ID / AZURE_TENANT_ID / AZURE_CLIENT_SECRET " + f"environment variables in CI. {detail}".strip() + ) + if status in (403,): + return AgentIdentityError( + "Microsoft Graph denied the request (403). The app registration is " + "missing admin consent for AgentIdentityBlueprint.Read.All (lookup) " + "or AgentIdentityBlueprint.Create (registration). Ask a tenant " + f"admin to grant consent, then retry. {detail}".strip() + ) + if status == 404: + return AgentIdentityError( + "Microsoft Graph returned 404 for the agent identity endpoint. The " + "tenant may not be enrolled in Microsoft Agent 365 yet. " + f"See {REGISTRATION_DOCS_URL}. {detail}".strip() + ) + if status == 429: + return AgentIdentityError( + "Microsoft Graph throttled the request (429). Retry in a few " + f"seconds. {detail}".strip() + ) + return AgentIdentityError( + f"Microsoft Graph request failed with HTTP {status}. {detail}".strip() + ) + + +def _graph_error_message(exc: urllib.error.HTTPError) -> str: + """Best-effort extraction of the Graph ``error.message`` field.""" + + try: + payload = json.loads(exc.read().decode("utf-8")) + except Exception: # noqa: BLE001 - error bodies are unreliable + return "" + error = payload.get("error") if isinstance(payload, dict) else None + if isinstance(error, dict): + message = error.get("message") + if isinstance(message, str) and message.strip(): + return f"Graph said: {message.strip()}" + return "" + + +def acquire_graph_token() -> str: + """Return an app-only Microsoft Graph access token. + + Reuses the shared doctor credential so a process that already + authenticated for Azure Monitor does not walk the credential chain again. + """ + + try: + from agentops.agent.sources._credentials import ( + format_source_error, + get_shared_credential, + ) + except ImportError: # pragma: no cover - package layout guarantees this + raise AgentIdentityError( + "AgentOps could not load its credential helper." + ) from None + + try: + credential = get_shared_credential() + except ImportError: + raise AgentIdentityError( + "The 'azure-identity' package is required to talk to Microsoft " + "Graph. Install it with: pip install azure-identity" + ) from None + + try: + token = credential.get_token(GRAPH_SCOPE) + except Exception as exc: # noqa: BLE001 - surfaced as a clean message + raise AgentIdentityError( + "Could not acquire a Microsoft Graph token: " + f"{format_source_error(exc)}" + ) from None + + value = getattr(token, "token", None) + if not value: + raise AgentIdentityError( + "The credential returned an empty Microsoft Graph token." + ) + return str(value) + + +# --------------------------------------------------------------------------- +# Blueprint lookup / registration +# --------------------------------------------------------------------------- + + +def _escape_odata_literal(value: str) -> str: + """Escape a string for use inside an OData ``$filter`` literal.""" + + return value.replace("'", "''") + + +def _blueprint_from_payload(payload: Mapping[str, Any]) -> Optional[AgentIdentityBlueprint]: + app_id = payload.get("appId") + if not isinstance(app_id, str) or not app_id.strip(): + return None + object_id = payload.get("id") + display_name = payload.get("displayName") + return AgentIdentityBlueprint( + app_id=app_id.strip(), + object_id=object_id if isinstance(object_id, str) else None, + display_name=display_name if isinstance(display_name, str) else None, + ) + + +def lookup_blueprint( + display_name: str, + *, + client: Optional[GraphClient] = None, +) -> Optional[AgentIdentityBlueprint]: + """Return the agent identity blueprint named ``display_name``, if any. + + Read-only. Returns ``None`` when the tenant has no blueprint with that + display name; raises :class:`AgentIdentityError` when Graph could not be + consulted at all (so callers can tell "not registered" apart from + "could not check"). + """ + + name = (display_name or "").strip() + if not name: + raise AgentIdentityError( + "An agent display name is required to look up its identity." + ) + + graph = client or GraphClient() + payload = graph.get( + "/applications", + params={ + "$filter": f"displayName eq '{_escape_odata_literal(name)}'", + "$select": "id,appId,displayName", + "$top": "1", + }, + ) + values = payload.get("value") if isinstance(payload, Mapping) else None + if not isinstance(values, list) or not values: + return None + first = values[0] + if not isinstance(first, Mapping): + return None + return _blueprint_from_payload(first) + + +def register_blueprint( + display_name: str, + *, + sponsor: str, + client: Optional[GraphClient] = None, +) -> tuple[AgentIdentityBlueprint, bool]: + """Create the agent identity blueprint, or return the existing one. + + Returns ``(blueprint, created)`` where ``created`` is ``False`` when a + blueprint with the same display name already existed. Re-running is + therefore safe: no duplicate is ever created. + """ + + name = (display_name or "").strip() + if not name: + raise AgentIdentityError( + "An agent display name is required to register an agent identity." + ) + if not (sponsor or "").strip(): + raise AgentIdentityError( + "A sponsor is required to register an agent identity. Set " + "'identity.sponsor' in agentops.yaml to the object id or UPN of " + "the human accountable for this agent." + ) + + graph = client or GraphClient() + + existing = lookup_blueprint(name, client=graph) + if existing is not None: + return existing, False + + payload = graph.post( + "/applications", + { + "@odata.type": BLUEPRINT_ODATA_TYPE, + "displayName": name, + "sponsors": [sponsor.strip()], + }, + ) + if not isinstance(payload, Mapping): + raise AgentIdentityError( + "Microsoft Graph accepted the registration but returned no " + "application object." + ) + blueprint = _blueprint_from_payload(payload) + if blueprint is None: + raise AgentIdentityError( + "Microsoft Graph accepted the registration but returned no appId." + ) + return blueprint, True + + +# --------------------------------------------------------------------------- +# Workspace persistence +# --------------------------------------------------------------------------- + + +def identity_record_path(workspace: Path) -> Path: + return Path(workspace) / IDENTITY_RECORD_RELPATH + + +def write_identity_record( + workspace: Path, + blueprint: AgentIdentityBlueprint, + *, + created: bool = False, +) -> Path: + """Persist ``blueprint`` under ``.agentops/identity/`` and return the path.""" + + path = identity_record_path(workspace) + path.parent.mkdir(parents=True, exist_ok=True) + record = { + "version": 1, + "recorded_at": datetime.now(timezone.utc).isoformat(), + "created": bool(created), + **blueprint.to_dict(), + } + path.write_text(json.dumps(record, indent=2) + "\n", encoding="utf-8") + return path + + +def read_identity_record(workspace: Path) -> Optional[dict[str, Any]]: + """Return the persisted identity record, or ``None`` when absent/invalid.""" + + path = identity_record_path(workspace) + if not path.exists(): + return None + try: + payload = json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError): + return None + return payload if isinstance(payload, dict) else None + + +def resolve_agent_id(workspace: Optional[Path] = None) -> Optional[str]: + """Return the Entra Agent ID for this workspace. + + Resolution order: the ``AGENTOPS_ENTRA_AGENT_ID`` environment variable + (so CI can inject an identity provisioned elsewhere), then the persisted + record. Returns ``None`` when the agent has no known identity, which is + the normal state before registration. + """ + + override = os.environ.get(AGENT_ID_ENV, "").strip() + if override and "$(" not in override and "${{" not in override: + return override + if workspace is None: + return None + record = read_identity_record(workspace) + if not record: + return None + app_id = record.get("app_id") + return app_id.strip() if isinstance(app_id, str) and app_id.strip() else None + + +# --------------------------------------------------------------------------- +# Workspace configuration +# --------------------------------------------------------------------------- + + +def load_identity_config(workspace: Path) -> dict[str, Any]: + """Return the ``identity`` block from ``agentops.yaml``, or ``{}``.""" + + path = Path(workspace) / "agentops.yaml" + if not path.exists(): + return {} + try: + from agentops.utils.yaml import load_yaml + + data = load_yaml(path) + except Exception: # noqa: BLE001 - config problems must not crash callers + return {} + if not isinstance(data, dict): + return {} + identity = data.get("identity") + return identity if isinstance(identity, dict) else {} + + +def resolve_display_name( + workspace: Path, *, override: Optional[str] = None +) -> Optional[str]: + """Resolve the blueprint display name for ``workspace``. + + Precedence: explicit override, then ``identity.display_name``, then a + previously recorded name, then the agent target name from + ``agentops.yaml``. Returns ``None`` when nothing usable is configured. + """ + + if isinstance(override, str) and override.strip(): + return override.strip() + + identity = load_identity_config(workspace) + configured = identity.get("display_name") + if isinstance(configured, str) and configured.strip(): + return configured.strip() + + record = read_identity_record(workspace) or {} + recorded = record.get("display_name") + if isinstance(recorded, str) and recorded.strip(): + return recorded.strip() + + return _name_from_target(workspace) + + +def _name_from_target(workspace: Path) -> Optional[str]: + path = Path(workspace) / "agentops.yaml" + if not path.exists(): + return None + try: + from agentops.utils.yaml import load_yaml + + data = load_yaml(path) + except Exception: # noqa: BLE001 + return None + if not isinstance(data, dict): + return None + agent = data.get("agent") + if not isinstance(agent, str): + return None + raw = agent.strip() + if not raw or "://" in raw: + return None + name = raw.split(":", 1)[0].strip() + return name or None + + +def resolve_registration_inputs( + workspace: Path, + *, + display_name: Optional[str] = None, + sponsor: Optional[str] = None, +) -> tuple[str, str]: + """Resolve ``(display_name, sponsor)`` for a registration call. + + Raises :class:`AgentIdentityError` with an actionable sentence when + either value is missing, because Agent 365 rejects a blueprint that has + no name and AgentOps refuses to create one that has no accountable owner. + """ + + identity = load_identity_config(workspace) + + resolved_name = resolve_display_name(workspace, override=display_name) + if not resolved_name: + raise AgentIdentityError( + "No display name for the agent identity. Pass --display-name, or " + "set 'identity.display_name' in agentops.yaml." + ) + + resolved_sponsor = sponsor if isinstance(sponsor, str) else None + if not (resolved_sponsor and resolved_sponsor.strip()): + configured = identity.get("sponsor") + resolved_sponsor = configured if isinstance(configured, str) else None + if not (resolved_sponsor and resolved_sponsor.strip()): + raise AgentIdentityError( + "No sponsor for the agent identity. Microsoft Agent 365 requires an " + "accountable owner. Pass --sponsor , or set " + "'identity.sponsor' in agentops.yaml." + ) + + return resolved_name, resolved_sponsor.strip() + + +__all__ = [ + "AGENT_ID_ATTRIBUTE", + "AGENT_ID_ENV", + "AgentIdentityBlueprint", + "AgentIdentityError", + "GraphClient", + "REGISTRATION_DOCS_URL", + "acquire_graph_token", + "identity_record_path", + "load_identity_config", + "lookup_blueprint", + "read_identity_record", + "register_blueprint", + "resolve_agent_id", + "resolve_display_name", + "resolve_registration_inputs", + "write_identity_record", +] diff --git a/src/agentops/services/evidence_pack.py b/src/agentops/services/evidence_pack.py index fc2814ac..6fa65ad3 100644 --- a/src/agentops/services/evidence_pack.py +++ b/src/agentops/services/evidence_pack.py @@ -68,6 +68,7 @@ def build_release_evidence( observability = _observability_status(root, trace_dataset) ailz = _ailz_status(analysis) governance = _governance_status(root) + agent_identity = _agent_identity_status(root) checks: list[ReleaseEvidenceCheck] = [] blockers: list[str] = [] @@ -85,6 +86,7 @@ def build_release_evidence( _add_trace_dataset_check(checks, warnings, ready, trace_dataset) _add_ailz_check(checks, warnings, ready, ailz) _add_governance_check(checks, warnings, ready, governance) + _add_agent_identity_check(checks, warnings, ready, agent_identity) status = "blocked" if blockers else "ready_with_warnings" if warnings else "ready" links = _links(latest_eval, observability) @@ -111,6 +113,7 @@ def build_release_evidence( observability=observability, ailz=ailz, governance=governance, + agent_identity=agent_identity, ) return ReleaseEvidence.model_validate(_redact_obj(evidence.model_dump())) @@ -834,6 +837,78 @@ def _add_governance_check( ) +def _agent_identity_status(root: Path) -> dict[str, Any]: + """Summarize the agent's Entra identity for the evidence bundle. + + The bundle records the Entra Agent ID so a release artifact can be tied + back to the governed principal that produced its traces. Absence is + reported as a status rather than an error, because registration is + opt-in until a tenant has Agent 365 enabled. + """ + + from agentops.services.agent_identity import ( + identity_record_path, + read_identity_record, + resolve_agent_id, + ) + + record = read_identity_record(root) or {} + agent_id = resolve_agent_id(root) + if not agent_id: + return {"status": "not_registered", "agent_id": None} + + summary: dict[str, Any] = { + "status": "registered", + "agent_id": agent_id, + "source": "record" if record.get("app_id") else "environment", + } + for key in ("display_name", "object_id", "recorded_at"): + value = record.get(key) + if isinstance(value, str) and value.strip(): + summary[key] = value.strip() + if record: + summary["record_path"] = str(identity_record_path(root)) + return summary + + +def _add_agent_identity_check( + checks: list[ReleaseEvidenceCheck], + warnings: list[str], + ready: list[str], + agent_identity: dict[str, Any], +) -> None: + if agent_identity.get("status") != "registered": + message = ( + "Agent identity is not registered in Microsoft Entra, so release " + "evidence cannot be attributed to a governed principal. " + "Run `agentops agent register --sponsor `." + ) + warnings.append(message) + checks.append( + ReleaseEvidenceCheck( + name="Agent identity", + status="warning", + summary=message, + evidence=agent_identity, + ) + ) + return + + message = ( + "Agent identity " + f"{agent_identity.get('agent_id')} is registered and stamped on traces." + ) + ready.append(message) + checks.append( + ReleaseEvidenceCheck( + name="Agent identity", + status="ready", + summary=message, + evidence=agent_identity, + ) + ) + + def _links(latest_eval: dict[str, Any], observability: dict[str, Any]) -> list[ReleaseEvidenceLink]: links: list[ReleaseEvidenceLink] = [] report_url = latest_eval.get("foundry_report_url") diff --git a/src/agentops/templates/workflows/agentops-deploy-prod-azd.yml b/src/agentops/templates/workflows/agentops-deploy-prod-azd.yml index 2410a5dc..ec8f8acf 100644 --- a/src/agentops/templates/workflows/agentops-deploy-prod-azd.yml +++ b/src/agentops/templates/workflows/agentops-deploy-prod-azd.yml @@ -170,6 +170,25 @@ __EVAL_STEPS__ tenant-id: ${{ vars.AZURE_TENANT_ID }} subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID }} __AZD_CLI_SETUP__ + # Opt-in: register the agent's Microsoft Entra identity blueprint so + # traces and release evidence can be attributed to a governed agent. + # Off unless you set the `AGENTOPS_IDENTITY_SPONSOR` repository variable + # to the accountable owner's UPN. The command is idempotent: an existing + # blueprint with the same display name is adopted, not duplicated. + # Requires the workload identity to hold `AgentIdentityBlueprint.Create`. + - name: Set up Python (agent identity) + if: vars.AGENTOPS_IDENTITY_SPONSOR != '' + uses: actions/setup-python@v6 + with: + python-version: "3.11" + - name: Register agent identity (optional) + if: vars.AGENTOPS_IDENTITY_SPONSOR != '' + env: + AZURE_TENANT_ID: ${{ vars.AZURE_TENANT_ID }} + run: | + pip install "agentops-accelerator[agent]__AGENTOPS_INSTALL_SPEC__" + agentops agent register --workspace . \ + --sponsor "${{ vars.AGENTOPS_IDENTITY_SPONSOR }}" - name: Run azd deploy env: AZURE_ENV_NAME: ${{ vars.AZURE_ENV_NAME || 'production' }} diff --git a/src/agentops/templates/workflows/agentops-deploy-prod.yml b/src/agentops/templates/workflows/agentops-deploy-prod.yml index 01a8be9e..5de7497e 100644 --- a/src/agentops/templates/workflows/agentops-deploy-prod.yml +++ b/src/agentops/templates/workflows/agentops-deploy-prod.yml @@ -162,6 +162,27 @@ __EVAL_STEPS__ tenant-id: ${{ vars.AZURE_TENANT_ID }} subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID }} + # Opt-in: register the agent's Microsoft Entra identity blueprint so + # traces and release evidence can be attributed to a governed agent. + # Off unless you set the `AGENTOPS_IDENTITY_SPONSOR` repository variable + # to the accountable owner's UPN. The command is idempotent: an existing + # blueprint with the same display name is adopted, not duplicated. + # Requires the workload identity to hold `AgentIdentityBlueprint.Create`. + - name: Set up Python (agent identity) + if: vars.AGENTOPS_IDENTITY_SPONSOR != '' + uses: actions/setup-python@v6 + with: + python-version: "3.11" + + - name: Register agent identity (optional) + if: vars.AGENTOPS_IDENTITY_SPONSOR != '' + env: + AZURE_TENANT_ID: ${{ vars.AZURE_TENANT_ID }} + run: | + pip install "agentops-accelerator[agent]__AGENTOPS_INSTALL_SPEC__" + agentops agent register --workspace . \ + --sponsor "${{ vars.AGENTOPS_IDENTITY_SPONSOR }}" + # --------------------------------------------------------------- # TODO: replace this placeholder with your deploy step. # See agentops-deploy-dev.yml for example snippets (ACA, App diff --git a/src/agentops/utils/telemetry.py b/src/agentops/utils/telemetry.py index 875e63e6..12fbd844 100644 --- a/src/agentops/utils/telemetry.py +++ b/src/agentops/utils/telemetry.py @@ -131,12 +131,7 @@ def init_tracing() -> None: import agentops - resource = Resource( - attributes={ - "service.name": "agentops", - "service.version": getattr(agentops, "__version__", "0.0.0"), - } - ) + resource = Resource(attributes=_resource_attributes()) provider = TracerProvider(resource=resource) exporter = OTLPSpanExporter(endpoint=otlp_endpoint + "/v1/traces") @@ -198,18 +193,48 @@ def _appinsights_connection_string_parts(value: str) -> dict[str, str]: return parts +def _resource_attributes() -> dict: + """Resource attributes shared by every AgentOps tracer provider. + + The Entra Agent ID is stamped as ``gen_ai.agent.id`` when the workspace + has a registered identity. Without it every span is attributable to a + service but not to a governed principal, which is exactly the gap that + makes agent traces hard to audit. The attribute is omitted rather than + emitted empty so downstream queries can filter on its presence. + """ + + import agentops + + attributes = { + "service.name": "agentops", + "service.version": getattr(agentops, "__version__", "0.0.0"), + } + try: + from pathlib import Path + + from agentops.services.agent_identity import ( + AGENT_ID_ATTRIBUTE, + resolve_agent_id, + ) + + agent_id = resolve_agent_id(Path.cwd()) + if agent_id: + attributes[AGENT_ID_ATTRIBUTE] = agent_id + except Exception: # noqa: BLE001 - identity is optional, tracing is not + pass + return attributes + + def _agentops_resource() -> Optional[Any]: try: from opentelemetry.sdk.resources import Resource - import agentops except Exception: # noqa: BLE001 return None - return Resource.create( - { - "service.name": "agentops", - "service.version": getattr(agentops, "__version__", "0.0.0"), - } - ) + try: + attributes = _resource_attributes() + except Exception: # noqa: BLE001 + return None + return Resource.create(attributes) def shutdown() -> None: diff --git a/tests/unit/test_agent_checks_agent_identity.py b/tests/unit/test_agent_checks_agent_identity.py new file mode 100644 index 00000000..645efe17 --- /dev/null +++ b/tests/unit/test_agent_checks_agent_identity.py @@ -0,0 +1,123 @@ +"""Tests for the Agent 365 registration posture check.""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from agentops.agent.checks import agent_identity as check_module +from agentops.agent.checks.agent_identity import SOURCE_NAME, run_agent_identity_check +from agentops.agent.findings import Category, Severity +from agentops.services.agent_identity import ( + AGENT_ID_ENV, + AgentIdentityBlueprint, + AgentIdentityError, + write_identity_record, +) + + +@pytest.fixture(autouse=True) +def _clear_agent_id_env(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.delenv(AGENT_ID_ENV, raising=False) + + +def _write_config(workspace: Path, body: str) -> None: + (workspace / "agentops.yaml").write_text(body, encoding="utf-8") + + +def test_registered_workspace_produces_no_findings(tmp_path: Path) -> None: + write_identity_record(tmp_path, AgentIdentityBlueprint(app_id="app-1")) + assert run_agent_identity_check(tmp_path) == [] + + +def test_environment_provided_identity_produces_no_findings( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.setenv(AGENT_ID_ENV, "app-from-ci") + assert run_agent_identity_check(tmp_path) == [] + + +def test_unregistered_workspace_warns(tmp_path: Path) -> None: + findings = run_agent_identity_check(tmp_path) + assert [f.id for f in findings] == ["agent_identity.not_registered"] + finding = findings[0] + assert finding.severity is Severity.WARNING + assert finding.category is Category.SECURITY + assert finding.source == SOURCE_NAME + assert "agentops agent register" in finding.recommendation + + +def test_graph_is_not_called_when_verify_is_off( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """Verification is opt-in: most tenants lack the Graph consent on day one.""" + + def _boom(*args: object, **kwargs: object) -> None: + raise AssertionError("Graph must not be consulted when verify is off") + + monkeypatch.setattr(check_module, "lookup_blueprint", _boom) + findings = run_agent_identity_check(tmp_path) + assert [f.id for f in findings] == ["agent_identity.not_registered"] + + +def test_verify_reports_blueprint_that_is_not_recorded_locally( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + _write_config(tmp_path, "identity:\n verify: true\n display_name: support-agent\n") + monkeypatch.setattr( + check_module, + "lookup_blueprint", + lambda name: AgentIdentityBlueprint(app_id="app-1", display_name=name), + ) + findings = run_agent_identity_check(tmp_path) + assert [f.id for f in findings] == ["agent_identity.not_recorded"] + assert findings[0].severity is Severity.INFO + assert findings[0].evidence["app_id"] == "app-1" + + +def test_verify_reports_not_registered_when_graph_finds_nothing( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + _write_config(tmp_path, "identity:\n verify: true\n") + monkeypatch.setattr(check_module, "lookup_blueprint", lambda name: None) + findings = run_agent_identity_check(tmp_path) + assert [f.id for f in findings] == ["agent_identity.not_registered"] + + +def test_graph_failure_becomes_a_readable_warning( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """A missing consent must never surface as a stack trace.""" + + def _raise(name: str) -> None: + raise AgentIdentityError("Admin consent is missing for AgentIdentityBlueprint.Read.All.") + + _write_config(tmp_path, "identity:\n verify: true\n") + monkeypatch.setattr(check_module, "lookup_blueprint", _raise) + findings = run_agent_identity_check(tmp_path) + assert [f.id for f in findings] == ["agent_identity.lookup_failed"] + assert findings[0].severity is Severity.WARNING + assert "Admin consent is missing" in findings[0].summary + + +@pytest.mark.parametrize("raw", ["true", "True", "yes", "on", "1"]) +def test_verify_accepts_string_truthy_values( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, raw: str +) -> None: + _write_config(tmp_path, f"identity:\n verify: '{raw}'\n") + monkeypatch.setattr( + check_module, + "lookup_blueprint", + lambda name: AgentIdentityBlueprint(app_id="app-1"), + ) + assert [f.id for f in run_agent_identity_check(tmp_path)] == [ + "agent_identity.not_recorded" + ] + + +def test_check_falls_back_to_directory_name_for_display_name(tmp_path: Path) -> None: + workspace = tmp_path / "my-agent" + workspace.mkdir() + findings = run_agent_identity_check(workspace) + assert findings[0].evidence["display_name"] == "my-agent" diff --git a/tests/unit/test_agent_identity_service.py b/tests/unit/test_agent_identity_service.py new file mode 100644 index 00000000..147c04d4 --- /dev/null +++ b/tests/unit/test_agent_identity_service.py @@ -0,0 +1,284 @@ +"""Tests for the Entra agent identity service. + +Every Graph call is exercised through an injected double, so the suite never +needs a tenant, a token, or network access. +""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any, Mapping + +import pytest + +from agentops.services.agent_identity import ( + AGENT_ID_ATTRIBUTE, + AGENT_ID_ENV, + AgentIdentityBlueprint, + AgentIdentityError, + identity_record_path, + load_identity_config, + lookup_blueprint, + read_identity_record, + register_blueprint, + resolve_agent_id, + resolve_display_name, + resolve_registration_inputs, + write_identity_record, +) + + +class FakeGraphClient: + """Records calls and replays canned responses.""" + + def __init__( + self, + *, + get_response: Mapping[str, Any] | None = None, + post_response: Mapping[str, Any] | None = None, + get_error: Exception | None = None, + ) -> None: + self._get_response = get_response if get_response is not None else {"value": []} + self._post_response = post_response or {} + self._get_error = get_error + self.get_calls: list[tuple[str, dict[str, str] | None]] = [] + self.post_calls: list[tuple[str, dict[str, Any]]] = [] + + def get(self, path: str, params: dict[str, str] | None = None) -> Mapping[str, Any]: + self.get_calls.append((path, params)) + if self._get_error is not None: + raise self._get_error + return self._get_response + + def post(self, path: str, body: dict[str, Any]) -> Mapping[str, Any]: + self.post_calls.append((path, body)) + return self._post_response + + +BLUEPRINT_PAYLOAD = { + "id": "object-1", + "appId": "app-1", + "displayName": "support-agent", +} + + +def test_agent_id_attribute_matches_semantic_convention() -> None: + assert AGENT_ID_ATTRIBUTE == "gen_ai.agent.id" + + +# --------------------------------------------------------------------------- +# lookup_blueprint +# --------------------------------------------------------------------------- + + +def test_lookup_returns_none_when_tenant_has_no_blueprint() -> None: + client = FakeGraphClient(get_response={"value": []}) + assert lookup_blueprint("support-agent", client=client) is None + path, params = client.get_calls[0] + assert path == "/applications" + assert params is not None + assert params["$filter"] == "displayName eq 'support-agent'" + + +def test_lookup_returns_blueprint_when_found() -> None: + client = FakeGraphClient(get_response={"value": [BLUEPRINT_PAYLOAD]}) + blueprint = lookup_blueprint("support-agent", client=client) + assert blueprint is not None + assert blueprint.app_id == "app-1" + assert blueprint.object_id == "object-1" + assert blueprint.display_name == "support-agent" + + +def test_lookup_escapes_single_quotes_in_display_name() -> None: + client = FakeGraphClient(get_response={"value": []}) + lookup_blueprint("o'brien agent", client=client) + _, params = client.get_calls[0] + assert params is not None + assert params["$filter"] == "displayName eq 'o''brien agent'" + + +def test_lookup_rejects_blank_display_name() -> None: + with pytest.raises(AgentIdentityError): + lookup_blueprint(" ", client=FakeGraphClient()) + + +def test_lookup_propagates_graph_errors() -> None: + boom = AgentIdentityError("consent missing") + client = FakeGraphClient(get_error=boom) + with pytest.raises(AgentIdentityError, match="consent missing"): + lookup_blueprint("support-agent", client=client) + + +# --------------------------------------------------------------------------- +# register_blueprint +# --------------------------------------------------------------------------- + + +def test_register_creates_blueprint_when_absent() -> None: + client = FakeGraphClient( + get_response={"value": []}, + post_response=BLUEPRINT_PAYLOAD, + ) + blueprint, created = register_blueprint( + "support-agent", sponsor="paulo@contoso.com", client=client + ) + assert created is True + assert blueprint.app_id == "app-1" + path, body = client.post_calls[0] + assert path == "/applications" + assert body["displayName"] == "support-agent" + assert body["sponsors"] == ["paulo@contoso.com"] + + +def test_register_is_idempotent_when_blueprint_exists() -> None: + client = FakeGraphClient(get_response={"value": [BLUEPRINT_PAYLOAD]}) + blueprint, created = register_blueprint( + "support-agent", sponsor="paulo@contoso.com", client=client + ) + assert created is False + assert blueprint.app_id == "app-1" + assert client.post_calls == [] + + +def test_register_requires_a_sponsor() -> None: + with pytest.raises(AgentIdentityError, match="sponsor"): + register_blueprint("support-agent", sponsor=" ", client=FakeGraphClient()) + + +def test_register_requires_a_display_name() -> None: + with pytest.raises(AgentIdentityError, match="display name"): + register_blueprint("", sponsor="paulo@contoso.com", client=FakeGraphClient()) + + +def test_register_fails_loudly_when_graph_returns_no_app_id() -> None: + client = FakeGraphClient(get_response={"value": []}, post_response={"id": "x"}) + with pytest.raises(AgentIdentityError, match="appId"): + register_blueprint("support-agent", sponsor="p@c.com", client=client) + + +# --------------------------------------------------------------------------- +# Workspace persistence +# --------------------------------------------------------------------------- + + +def test_write_and_read_identity_record(tmp_path: Path) -> None: + blueprint = AgentIdentityBlueprint( + app_id="app-1", object_id="object-1", display_name="support-agent" + ) + path = write_identity_record(tmp_path, blueprint, created=True) + assert path == identity_record_path(tmp_path) + record = read_identity_record(tmp_path) + assert record is not None + assert record["app_id"] == "app-1" + assert record["created"] is True + assert record["version"] == 1 + + +def test_read_identity_record_returns_none_when_absent(tmp_path: Path) -> None: + assert read_identity_record(tmp_path) is None + + +def test_read_identity_record_tolerates_corrupt_json(tmp_path: Path) -> None: + path = identity_record_path(tmp_path) + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text("{not json", encoding="utf-8") + assert read_identity_record(tmp_path) is None + + +def test_resolve_agent_id_prefers_environment_override( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + write_identity_record(tmp_path, AgentIdentityBlueprint(app_id="from-record")) + monkeypatch.setenv(AGENT_ID_ENV, "from-env") + assert resolve_agent_id(tmp_path) == "from-env" + + +def test_resolve_agent_id_ignores_unexpanded_ci_placeholders( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + write_identity_record(tmp_path, AgentIdentityBlueprint(app_id="from-record")) + monkeypatch.setenv(AGENT_ID_ENV, "${{ secrets.AGENT_ID }}") + assert resolve_agent_id(tmp_path) == "from-record" + + +def test_resolve_agent_id_returns_none_before_registration( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.delenv(AGENT_ID_ENV, raising=False) + assert resolve_agent_id(tmp_path) is None + + +# --------------------------------------------------------------------------- +# Configuration resolution +# --------------------------------------------------------------------------- + + +def _write_config(workspace: Path, body: str) -> None: + (workspace / "agentops.yaml").write_text(body, encoding="utf-8") + + +def test_load_identity_config_returns_empty_without_config(tmp_path: Path) -> None: + assert load_identity_config(tmp_path) == {} + + +def test_load_identity_config_reads_identity_block(tmp_path: Path) -> None: + _write_config(tmp_path, "identity:\n sponsor: paulo@contoso.com\n") + assert load_identity_config(tmp_path)["sponsor"] == "paulo@contoso.com" + + +def test_load_identity_config_tolerates_malformed_yaml(tmp_path: Path) -> None: + _write_config(tmp_path, "identity: [unclosed\n") + assert load_identity_config(tmp_path) == {} + + +def test_resolve_display_name_prefers_explicit_override(tmp_path: Path) -> None: + _write_config(tmp_path, "identity:\n display_name: from-config\n") + assert resolve_display_name(tmp_path, override="from-flag") == "from-flag" + + +def test_resolve_display_name_falls_back_to_config(tmp_path: Path) -> None: + _write_config(tmp_path, "identity:\n display_name: from-config\n") + assert resolve_display_name(tmp_path) == "from-config" + + +def test_resolve_display_name_falls_back_to_existing_record(tmp_path: Path) -> None: + write_identity_record( + tmp_path, AgentIdentityBlueprint(app_id="app-1", display_name="from-record") + ) + assert resolve_display_name(tmp_path) == "from-record" + + +def test_resolve_registration_inputs_returns_name_and_sponsor(tmp_path: Path) -> None: + _write_config( + tmp_path, + "identity:\n display_name: support-agent\n sponsor: paulo@contoso.com\n", + ) + name, sponsor = resolve_registration_inputs(tmp_path) + assert name == "support-agent" + assert sponsor == "paulo@contoso.com" + + +def test_resolve_registration_inputs_requires_a_display_name(tmp_path: Path) -> None: + _write_config(tmp_path, "identity:\n sponsor: paulo@contoso.com\n") + with pytest.raises(AgentIdentityError, match="display name"): + resolve_registration_inputs(tmp_path) + + +def test_resolve_registration_inputs_requires_a_sponsor(tmp_path: Path) -> None: + _write_config(tmp_path, "identity:\n display_name: support-agent\n") + with pytest.raises(AgentIdentityError, match="sponsor"): + resolve_registration_inputs(tmp_path) + + +def test_resolve_registration_inputs_accepts_overrides(tmp_path: Path) -> None: + name, sponsor = resolve_registration_inputs( + tmp_path, display_name="flag-agent", sponsor="flag@contoso.com" + ) + assert (name, sponsor) == ("flag-agent", "flag@contoso.com") + + +def test_identity_record_is_valid_json_on_disk(tmp_path: Path) -> None: + write_identity_record(tmp_path, AgentIdentityBlueprint(app_id="app-1")) + raw = identity_record_path(tmp_path).read_text(encoding="utf-8") + assert json.loads(raw)["app_id"] == "app-1" diff --git a/tests/unit/test_agentops_config_identity.py b/tests/unit/test_agentops_config_identity.py new file mode 100644 index 00000000..c3ee2e2e --- /dev/null +++ b/tests/unit/test_agentops_config_identity.py @@ -0,0 +1,66 @@ +"""Regression tests for the optional ``identity`` block in agentops.yaml. + +``AgentOpsConfig`` sets ``extra="forbid"``, so any key the model does not +declare makes every command that calls ``load_agentops_config`` fail. The +Agent 365 identity work reads the block directly from the raw YAML, which +means a missing field declaration would only surface later, in unrelated +commands. These tests pin the contract. +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from agentops.core.config_loader import load_agentops_config + +BASE_CONFIG = "version: 1\nagent: my-rag:3\ndataset: .agentops/data/seed.jsonl\n" + + +def _write(tmp_path: Path, body: str) -> Path: + path = tmp_path / "agentops.yaml" + path.write_text(body, encoding="utf-8") + return path + + +def test_config_without_identity_still_loads(tmp_path: Path) -> None: + config = load_agentops_config(_write(tmp_path, BASE_CONFIG)) + assert config.identity is None + + +def test_identity_block_is_accepted(tmp_path: Path) -> None: + path = _write( + tmp_path, + BASE_CONFIG + + "identity:\n" + + " display_name: support-agent\n" + + " sponsor: owner@contoso.com\n" + + " verify: true\n", + ) + + config = load_agentops_config(path) + + assert config.identity is not None + assert config.identity.display_name == "support-agent" + assert config.identity.sponsor == "owner@contoso.com" + assert config.identity.verify is True + + +def test_identity_verify_defaults_to_false(tmp_path: Path) -> None: + path = _write(tmp_path, BASE_CONFIG + "identity:\n sponsor: owner@contoso.com\n") + + config = load_agentops_config(path) + + assert config.identity is not None + assert config.identity.verify is False + assert config.identity.display_name is None + + +def test_unknown_identity_key_is_rejected(tmp_path: Path) -> None: + path = _write(tmp_path, BASE_CONFIG + "identity:\n sponser: owner@contoso.com\n") + + with pytest.raises(ValueError) as excinfo: + load_agentops_config(path) + + assert "sponser" in str(excinfo.value) diff --git a/tests/unit/test_cli_agent_register.py b/tests/unit/test_cli_agent_register.py new file mode 100644 index 00000000..9b8dd2ad --- /dev/null +++ b/tests/unit/test_cli_agent_register.py @@ -0,0 +1,148 @@ +"""CLI tests for `agentops agent register`. + +The command is exercised through Typer's CliRunner. Microsoft Graph is never +contacted: the happy path patches the service function, and every other path +either fails input resolution or is a dry run. +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest +from typer.testing import CliRunner + +from agentops.cli.app import app +from agentops.services import agent_identity as identity_service +from agentops.services.agent_identity import ( + AgentIdentityBlueprint, + AgentIdentityError, + identity_record_path, + read_identity_record, +) + +runner = CliRunner() + + +def _write_config(workspace: Path, body: str) -> None: + (workspace / "agentops.yaml").write_text(body, encoding="utf-8") + + +def _invoke(workspace: Path, *args: str): + return runner.invoke(app, ["agent", "register", "-w", str(workspace), *args]) + + +def test_register_explain_renders_the_manual() -> None: + result = runner.invoke(app, ["agent", "register", "explain"]) + assert result.exit_code == 0 + assert "register" in result.output.lower() + + +def test_register_fails_without_a_display_name(tmp_path: Path) -> None: + result = _invoke(tmp_path, "--sponsor", "paulo@contoso.com", "--dry-run") + assert result.exit_code == 1 + assert "display name" in result.output.lower() + + +def test_register_fails_without_a_sponsor(tmp_path: Path) -> None: + result = _invoke(tmp_path, "--display-name", "support-agent", "--dry-run") + assert result.exit_code == 1 + assert "sponsor" in result.output.lower() + + +def test_dry_run_reports_inputs_without_writing_a_record(tmp_path: Path) -> None: + result = _invoke( + tmp_path, + "--display-name", + "support-agent", + "--sponsor", + "paulo@contoso.com", + "--dry-run", + ) + assert result.exit_code == 0 + assert "support-agent" in result.output + assert "paulo@contoso.com" in result.output + assert "Dry run" in result.output + assert not identity_record_path(tmp_path).exists() + + +def test_dry_run_resolves_inputs_from_config(tmp_path: Path) -> None: + _write_config( + tmp_path, + "identity:\n display_name: config-agent\n sponsor: owner@contoso.com\n", + ) + result = _invoke(tmp_path, "--dry-run") + assert result.exit_code == 0 + assert "config-agent" in result.output + assert "owner@contoso.com" in result.output + + +def test_register_writes_the_identity_record( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.setattr( + identity_service, + "register_blueprint", + lambda name, *, sponsor: ( + AgentIdentityBlueprint( + app_id="app-1", object_id="object-1", display_name=name + ), + True, + ), + ) + result = _invoke( + tmp_path, + "--display-name", + "support-agent", + "--sponsor", + "paulo@contoso.com", + ) + assert result.exit_code == 0 + assert "Registered" in result.output + assert "app-1" in result.output + record = read_identity_record(tmp_path) + assert record is not None + assert record["app_id"] == "app-1" + assert record["created"] is True + + +def test_register_reports_reuse_when_blueprint_already_exists( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.setattr( + identity_service, + "register_blueprint", + lambda name, *, sponsor: ( + AgentIdentityBlueprint(app_id="app-1", display_name=name), + False, + ), + ) + result = _invoke( + tmp_path, + "--display-name", + "support-agent", + "--sponsor", + "paulo@contoso.com", + ) + assert result.exit_code == 0 + assert "Reused existing" in result.output + assert read_identity_record(tmp_path)["created"] is False + + +def test_graph_failure_is_reported_without_a_stack_trace( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + def _raise(name: str, *, sponsor: str): + raise AgentIdentityError("Admin consent is missing for AgentIdentityBlueprint.Create.") + + monkeypatch.setattr(identity_service, "register_blueprint", _raise) + result = _invoke( + tmp_path, + "--display-name", + "support-agent", + "--sponsor", + "paulo@contoso.com", + ) + assert result.exit_code == 1 + assert "Admin consent is missing" in result.output + assert "Traceback" not in result.output diff --git a/tests/unit/test_evidence_pack_agent_identity.py b/tests/unit/test_evidence_pack_agent_identity.py new file mode 100644 index 00000000..d2815634 --- /dev/null +++ b/tests/unit/test_evidence_pack_agent_identity.py @@ -0,0 +1,111 @@ +"""Tests for the agent identity section of the release evidence bundle.""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from agentops.core.release_evidence import ReleaseEvidence, ReleaseEvidenceCheck +from agentops.services.agent_identity import ( + AGENT_ID_ENV, + AgentIdentityBlueprint, + write_identity_record, +) +from agentops.services.evidence_pack import ( + _add_agent_identity_check, + _agent_identity_status, +) + + +@pytest.fixture(autouse=True) +def _clear_agent_id_env(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.delenv(AGENT_ID_ENV, raising=False) + + +def test_status_reports_not_registered_for_a_fresh_workspace(tmp_path: Path) -> None: + status = _agent_identity_status(tmp_path) + assert status["status"] == "not_registered" + assert status["agent_id"] is None + + +def test_status_reports_registered_from_the_record(tmp_path: Path) -> None: + write_identity_record( + tmp_path, + AgentIdentityBlueprint( + app_id="app-1", object_id="object-1", display_name="support-agent" + ), + ) + status = _agent_identity_status(tmp_path) + assert status["status"] == "registered" + assert status["agent_id"] == "app-1" + assert status["source"] == "record" + assert status["display_name"] == "support-agent" + assert status["object_id"] == "object-1" + assert "record_path" in status + + +def test_status_reports_environment_sourced_identity( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.setenv(AGENT_ID_ENV, "app-from-ci") + status = _agent_identity_status(tmp_path) + assert status["status"] == "registered" + assert status["agent_id"] == "app-from-ci" + assert status["source"] == "environment" + + +def test_unregistered_identity_produces_an_actionable_warning() -> None: + checks: list[ReleaseEvidenceCheck] = [] + warnings: list[str] = [] + ready: list[str] = [] + + _add_agent_identity_check( + checks, warnings, ready, {"status": "not_registered", "agent_id": None} + ) + + assert len(checks) == 1 + assert checks[0].status == "warning" + assert checks[0].name == "Agent identity" + assert ready == [] + assert len(warnings) == 1 + assert "agentops agent register" in warnings[0] + + +def test_registered_identity_produces_a_ready_signal() -> None: + checks: list[ReleaseEvidenceCheck] = [] + warnings: list[str] = [] + ready: list[str] = [] + + _add_agent_identity_check( + checks, warnings, ready, {"status": "registered", "agent_id": "app-1"} + ) + + assert len(checks) == 1 + assert checks[0].status == "ready" + assert warnings == [] + assert len(ready) == 1 + assert "app-1" in ready[0] + + +def test_release_evidence_model_accepts_the_agent_identity_section() -> None: + """The model forbids extra keys, so the field must be declared.""" + + evidence = ReleaseEvidence( + generated_at="2026-01-01T00:00:00+00:00", + workspace="/tmp/ws", + status="ready", + agent_identity={"status": "registered", "agent_id": "app-1"}, + ) + assert evidence.agent_identity["agent_id"] == "app-1" + round_tripped = ReleaseEvidence.model_validate(evidence.model_dump()) + assert round_tripped.agent_identity == evidence.agent_identity + + +def test_release_evidence_defaults_agent_identity_to_an_empty_dict() -> None: + evidence = ReleaseEvidence( + generated_at="2026-01-01T00:00:00+00:00", + workspace="/tmp/ws", + status="ready", + ) + assert evidence.agent_identity == {} diff --git a/tests/unit/test_telemetry_agent_id.py b/tests/unit/test_telemetry_agent_id.py new file mode 100644 index 00000000..e51f3e97 --- /dev/null +++ b/tests/unit/test_telemetry_agent_id.py @@ -0,0 +1,70 @@ +"""Tests for the Entra Agent ID stamped on OpenTelemetry resources.""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from agentops.services.agent_identity import ( + AGENT_ID_ATTRIBUTE, + AGENT_ID_ENV, + AgentIdentityBlueprint, + write_identity_record, +) +from agentops.utils.telemetry import _resource_attributes + + +@pytest.fixture(autouse=True) +def _clear_agent_id_env(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.delenv(AGENT_ID_ENV, raising=False) + + +def test_resource_always_carries_service_identity() -> None: + attributes = _resource_attributes() + assert attributes["service.name"] == "agentops" + assert isinstance(attributes["service.version"], str) + + +def test_agent_id_is_omitted_when_not_registered( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """An unregistered agent must not emit an empty attribute. + + Downstream queries filter on the attribute's presence, so emitting it + blank would make unregistered agents indistinguishable from registered + ones whose id failed to resolve. + """ + + monkeypatch.chdir(tmp_path) + assert AGENT_ID_ATTRIBUTE not in _resource_attributes() + + +def test_agent_id_is_stamped_from_the_workspace_record( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + write_identity_record(tmp_path, AgentIdentityBlueprint(app_id="app-1")) + monkeypatch.chdir(tmp_path) + assert _resource_attributes()[AGENT_ID_ATTRIBUTE] == "app-1" + + +def test_agent_id_is_stamped_from_the_environment( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.chdir(tmp_path) + monkeypatch.setenv(AGENT_ID_ENV, "app-from-ci") + assert _resource_attributes()[AGENT_ID_ATTRIBUTE] == "app-from-ci" + + +def test_identity_failures_never_break_tracing(monkeypatch: pytest.MonkeyPatch) -> None: + """Tracing is mandatory, identity is optional; the former must survive.""" + + import agentops.services.agent_identity as identity_service + + def _boom(*args: object, **kwargs: object) -> None: + raise RuntimeError("identity subsystem exploded") + + monkeypatch.setattr(identity_service, "resolve_agent_id", _boom) + attributes = _resource_attributes() + assert attributes["service.name"] == "agentops" + assert AGENT_ID_ATTRIBUTE not in attributes From 200e55847b3e39b101140734617717804908f1f5 Mon Sep 17 00:00:00 2001 From: Paulo Lacerda Date: Sun, 9 Aug 2026 10:15:47 -0300 Subject: [PATCH 2/2] fix: drop unused agentops import left behind in init_tracing The resource attribute assembly moved into _resource_attributes(), which imports agentops itself. The import at the init_tracing call site became dead and ruff F401 flagged it. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bcb9c0b6-d506-46dc-90d2-8120413166ee --- src/agentops/utils/telemetry.py | 2 -- 1 file changed, 2 deletions(-) diff --git a/src/agentops/utils/telemetry.py b/src/agentops/utils/telemetry.py index 12fbd844..7c2e6588 100644 --- a/src/agentops/utils/telemetry.py +++ b/src/agentops/utils/telemetry.py @@ -129,8 +129,6 @@ def init_tracing() -> None: from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor - import agentops - resource = Resource(attributes=_resource_attributes()) provider = TracerProvider(resource=resource)