diff --git a/CHANGELOG.md b/CHANGELOG.md index bd9cb6c7..50830743 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,23 @@ This format follows [Keep a Changelog](https://keepachangelog.com/) and adheres ## [Unreleased] +### Added +- **Agents can now carry a first-class Microsoft Entra identity that travels + from registration through traces into release evidence.** Before this, a + trace could tell you what an agent did but not which registered agent did it, + so nothing in the evidence pack tied runtime behaviour back to an accountable + owner in the tenant. Three pieces close that loop. `agentops agent register` + creates or adopts an agent identity blueprint in Microsoft Entra (idempotent, + sponsor required, `--dry-run` supported) and records the resolved id under + `.agentops/identity/agent-identity.json`. AgentOps then stamps that id on + every span it emits as the OpenTelemetry resource attribute + `gen_ai.agent.id`, omitting the attribute entirely when no identity is + registered so presence is a meaningful filter. Finally, the release evidence + pack publishes an `agent_identity` section reporting the id and its source. + A read-only Doctor check reports registration posture, contacting Microsoft + Graph only when `identity.verify` is enabled in `agentops.yaml`. `agentops.yaml` + accepts a new optional `identity` block (`display_name`, `sponsor`, `verify`). + ### Fixed - **The official evaluation runner now honours the agent version override.** `prepare_official_eval` read the agent name and version straight from diff --git a/docs/doctor-checks.md b/docs/doctor-checks.md index 16d85734..4ba7e833 100644 --- a/docs/doctor-checks.md +++ b/docs/doctor-checks.md @@ -33,7 +33,7 @@ Two conventions across the table: ## Data sources -The Doctor reaches Azure through five sources, all configured in +The Doctor reaches Azure through six sources, all configured in `.agentops/agent.yaml`: | Source | Reads | @@ -43,6 +43,7 @@ The Doctor reaches Azure through five sources, all configured in | `azure_monitor` | Application Insights / Log Analytics via REST (KQL) | | `foundry_control` | Foundry project / agents / evaluation rules via `azure-ai-projects` | | `azure_resources` | Cognitive Services account properties via `azure-mgmt-cognitiveservices`; inferred from explicit config, AZD `.azure//.env` when present, or Foundry endpoint/account matching | +| `graph` | Microsoft Entra agent identity blueprints via Microsoft Graph, read-only, and only when `identity.verify` is enabled | The LLM-judged rules additionally use the Foundry project's OpenAI client (auto-discovered) as the judge model. @@ -64,6 +65,17 @@ stopping the whole run. | `waf.security.diagnostic_settings` | warning | `azure_resources` | programmatic | account has ≥1 diagnostic setting with a `workspace_id` | | `safety.runtime.content_filter` | warning | `azure_monitor` | programmatic | KQL hits on `gen_ai.response.finish_reasons contains content_filter` | | `responsible_ai.llm.prompt_jailbreak_surface` | info / warning | `foundry_control` | llm-judged | judge model scans system prompt for override-phrasing, embedded secrets, unbounded role-play | +| `agent_identity.not_registered` | warning | `graph` | programmatic | no Entra Agent ID in `.agentops/identity/agent-identity.json` or `AGENTOPS_ENTRA_AGENT_ID` | +| `agent_identity.not_recorded` | info | `graph` | programmatic | `identity.verify: true` and Graph finds a blueprint, but no local record exists | +| `agent_identity.lookup_failed` | warning | `graph` | programmatic | `identity.verify: true` and the Graph lookup failed (missing consent, throttling, network) | + +!!! info "Identity checks are read-only" + The `agent_identity.*` checks never create or modify anything in Microsoft + Entra. Registration is an explicit, separate action: `agentops agent + register`. Graph is contacted only when `identity.verify` is set to true in + `agentops.yaml`; with it off, the check reads local state only and still + reports whether an identity exists. See + [Agent identity on traces](observe.md#agent-identity-on-traces). ### ⚙️ Operational Excellence diff --git a/docs/observe.md b/docs/observe.md index 20b4e49a..77ae26fd 100644 --- a/docs/observe.md +++ b/docs/observe.md @@ -66,6 +66,74 @@ with no monitoring does not look healthy simply because nothing is being graded. is intended: a real release should investigate latency and errors before promoting, even when the candidate's eval scores pass. +## Agent identity on traces + +Traces tell you what an agent did. They do not, by default, tell you *which* +agent did it in a way an auditor can reconcile with your tenant. Microsoft +Entra Agent ID closes that gap: the agent gets a first-class identity, and the +same identifier travels from registration through traces into release evidence. + +The handshake has three steps, and each one is a different tool, so it is worth +being explicit about who writes what. + +**1. Register the identity.** `agentops agent register` creates (or adopts) an +agent identity blueprint in Microsoft Entra and records the resolved +application id locally: + +```bash +agentops agent register --sponsor owner@contoso.com +``` + +The sponsor is required. An agent identity with no accountable owner cannot be +governed, so there is no default. The command is idempotent: if a blueprint +with the same display name already exists, AgentOps reuses it instead of +creating a duplicate. Run it with `--dry-run` first to see the resolved display +name and sponsor without calling Microsoft Graph. + +The resolved id is written to `.agentops/identity/agent-identity.json`. Declare +the inputs in `agentops.yaml` so they are source-controlled: + +```yaml +identity: + display_name: support-agent + sponsor: owner@contoso.com + verify: true +``` + +`verify: true` tells the Doctor to confirm the blueprint against Microsoft +Graph. It is off by default because that lookup needs tenant admin consent +(`AgentIdentityBlueprint.Read.All`), which most workspaces will not have on day +one. With it off, the Doctor still reports whether an identity is registered at +all, using only local state. + +**2. Stamp it on traces.** Once an identity is resolved, AgentOps adds it to +the OpenTelemetry resource as `gen_ai.agent.id`, so every span AgentOps emits +carries the Entra Agent ID. In CI, where the local record is not checked in, +set `AGENTOPS_ENTRA_AGENT_ID` instead and the attribute resolves from the +environment. + +The attribute is **omitted** when no identity is registered, never emitted as an +empty string. That distinction matters when you query: filtering on presence +tells you which traffic is attributable and which is not. + +```kusto +dependencies +| where isnotempty(customDimensions["gen_ai.agent.id"]) +| summarize runs = count() by tostring(customDimensions["gen_ai.agent.id"]) +``` + +**3. Publish it as evidence.** The release evidence pack reads the same record +and adds an `agent_identity` section reporting the id and where it came from +(the local record or the environment variable). When no identity is registered, +the pack raises a warning rather than a blocker, because identity registration +is a governance improvement rather than a correctness gate. + +!!! note "AgentOps does not ingest into Agent 365" + There is no public ingestion API for Agent 365 telemetry today. AgentOps + stamps the identifier and publishes it as evidence so the correlation is + possible from the Azure Monitor side. It does not push traces into Agent + 365. + ## Trace-to-regression promotion The strongest use of observability is turning real production behavior into new diff --git a/docs/ship.md b/docs/ship.md index adfe26b8..c7ec90be 100644 --- a/docs/ship.md +++ b/docs/ship.md @@ -158,6 +158,28 @@ long steps here: - [GitHub OIDC with Azure (workload identity federation)](https://learn.microsoft.com/azure/active-directory/workload-identities/workload-identity-federation-create-trust?pivots=identity-wif-apps-methods-azp) - [Assign Azure roles (RBAC)](https://learn.microsoft.com/azure/role-based-access-control/role-assignments-portal) +### Giving the agent its own identity + +The OIDC principal above answers "which pipeline is deploying". It does not +answer "which agent is running in production", and that second question is the +one an auditor asks. A Microsoft Entra agent identity closes the gap: the agent +gets its own directory object with a named human sponsor, and AgentOps then +stamps that id on every trace and republishes it in the release evidence. + +Registration is a one-line command and it is idempotent, so re-running it adopts +the existing blueprint instead of creating a duplicate: + +```bash +agentops agent register --sponsor owner@contoso.com +``` + +The generated production workflows carry the same step, disabled by default. +Set the `AGENTOPS_IDENTITY_SPONSOR` repository variable to the sponsor's UPN and +the step turns on. It stays opt-in because it writes to your tenant, which needs +a deliberate decision rather than a default. The full handshake, the +`identity` block in `agentops.yaml`, and the trace query are documented in +[Agent identity on traces](observe.md#agent-identity-on-traces). + ## Try it Generate the CI/CD workflows from the same analysis AgentOps uses, smallest gate diff --git a/src/agentops/agent/analyzer.py b/src/agentops/agent/analyzer.py index b1ae3082..57e8c762 100644 --- a/src/agentops/agent/analyzer.py +++ b/src/agentops/agent/analyzer.py @@ -10,6 +10,7 @@ from agentops.agent.checks.errors import run_errors_check from agentops.agent.checks.foundry_config import run_foundry_config_check +from agentops.agent.checks.agent_identity import run_agent_identity_check from agentops.agent.checks.governance import run_governance_check from agentops.agent.checks.latency import run_latency_check from agentops.agent.checks.observability import run_observability_check @@ -151,6 +152,7 @@ def analyze( findings.extend(run_rbac_openai_data_plane_check(resources)) findings.extend(run_opex_workspace_check(workspace)) findings.extend(run_governance_check(workspace)) + findings.extend(run_agent_identity_check(workspace)) findings.extend(run_observability_check(workspace)) findings.extend(run_opex_check(history, config.checks.opex)) findings.extend(run_release_readiness_check(workspace, history, foundry)) diff --git a/src/agentops/agent/checks/agent_identity.py b/src/agentops/agent/checks/agent_identity.py new file mode 100644 index 00000000..fc16b9d8 --- /dev/null +++ b/src/agentops/agent/checks/agent_identity.py @@ -0,0 +1,144 @@ +"""Agent 365 registration posture check. + +``agentops doctor`` scores a workspace against Well-Architected rules, but it +had no visibility into whether the agent exists as a first-class identity in +Microsoft Entra. Without an agent identity blueprint the agent cannot be +governed by Microsoft Agent 365: it does not appear in the agent inventory, +Conditional Access cannot target it, and its traces cannot be correlated back +to an accountable owner. + +The check is deliberately read-only and cheap: + +* it first resolves the identity from the workspace record or the + ``AGENTOPS_ENTRA_AGENT_ID`` environment variable, which costs nothing, +* it only calls Microsoft Graph when the workspace opts in via + ``identity.verify: true``, because the lookup needs tenant admin consent + that most workspaces will not have on day one, +* every Graph failure becomes a warning with a readable sentence, never a + stack trace. +""" + +from __future__ import annotations + +from pathlib import Path +from typing import Any, List + +from agentops.agent.findings import Category, Finding, Severity +from agentops.services.agent_identity import ( + REGISTRATION_DOCS_URL, + AgentIdentityError, + load_identity_config, + lookup_blueprint, + resolve_agent_id, + resolve_display_name, +) + +SOURCE_NAME = "agent_identity" + + +def run_agent_identity_check(workspace: Path) -> List[Finding]: + """Report whether the agent is registered in Microsoft Agent 365.""" + + workspace = Path(workspace) + identity = load_identity_config(workspace) + display_name = resolve_display_name(workspace) or workspace.resolve().name + + if resolve_agent_id(workspace): + return [] + + if _verify_enabled(identity): + try: + blueprint = lookup_blueprint(display_name) + except AgentIdentityError as exc: + return [_lookup_failed(display_name, str(exc))] + if blueprint is not None: + return [_registered_but_unrecorded(blueprint.app_id, display_name)] + + return [_not_registered(display_name)] + + +# --------------------------------------------------------------------------- +# Findings +# --------------------------------------------------------------------------- + + +def _not_registered(display_name: str) -> Finding: + return Finding( + id="agent_identity.not_registered", + severity=Severity.WARNING, + category=Category.SECURITY, + title="Agent is not registered in Microsoft Agent 365", + summary=( + "No Entra Agent ID is recorded for this workspace, so the agent has " + "no first-class identity in Microsoft Agent 365. Without it the agent " + "is absent from the tenant agent inventory, Conditional Access cannot " + "target it, and its traces cannot be attributed to an accountable owner." + ), + recommendation=( + "Register the agent identity blueprint with " + "'agentops agent register --sponsor '. Set " + "'identity.sponsor' in agentops.yaml first so the registration is " + f"reproducible in CI. Background: {REGISTRATION_DOCS_URL}" + ), + source=SOURCE_NAME, + evidence={"display_name": display_name, "registered": False}, + ) + + +def _registered_but_unrecorded(app_id: str, display_name: str) -> Finding: + return Finding( + id="agent_identity.not_recorded", + severity=Severity.INFO, + category=Category.SECURITY, + title="Agent identity exists in Entra but is not recorded locally", + summary=( + f"Microsoft Entra has an agent identity blueprint named " + f"'{display_name}', but this workspace has no local record of it. " + "Traces and the release evidence bundle therefore cannot quote the " + "Entra Agent ID." + ), + recommendation=( + "Run 'agentops agent register' to adopt the existing blueprint into " + "this workspace. The command is idempotent and will reuse the " + "blueprint instead of creating a duplicate." + ), + source=SOURCE_NAME, + evidence={"display_name": display_name, "app_id": app_id, "registered": True}, + ) + + +def _lookup_failed(display_name: str, reason: str) -> Finding: + return Finding( + id="agent_identity.lookup_failed", + severity=Severity.WARNING, + category=Category.SECURITY, + title="Agent 365 registration could not be verified", + summary=( + "AgentOps could not confirm whether this agent has an Entra Agent ID. " + f"{reason}" + ), + recommendation=( + "Grant the AgentIdentityBlueprint.Read.All application permission and " + "admin consent, or set 'identity.verify: false' in agentops.yaml to " + "rely on the locally recorded identity instead." + ), + source=SOURCE_NAME, + evidence={"display_name": display_name, "reason": reason}, + ) + + +# --------------------------------------------------------------------------- +# Config helpers +# --------------------------------------------------------------------------- + + +def _verify_enabled(identity: dict[str, Any]) -> bool: + value = identity.get("verify") + if isinstance(value, bool): + return value + if isinstance(value, str): + return value.strip().lower() in {"1", "true", "yes", "on"} + return False + + +__all__ = ["SOURCE_NAME", "run_agent_identity_check"] diff --git a/src/agentops/agent/checks/catalog.py b/src/agentops/agent/checks/catalog.py index 9f7ff9f9..b25a22ae 100644 --- a/src/agentops/agent/checks/catalog.py +++ b/src/agentops/agent/checks/catalog.py @@ -35,6 +35,7 @@ "azure_resources": "Azure resources (ARM)", "spec_workspace": "spec docs (.specify / AGENTS.md)", "judge_model": "judge model deployment", + "graph": "Microsoft Graph (Entra agent identity)", } SOURCE_DESCRIPTIONS: Dict[str, str] = { @@ -82,6 +83,13 @@ "checks. It reviews semantic signals like prompt guardrails, dataset " "PII risk, bundle coverage, and spec-vs-implementation gaps." ), + "graph": ( + "Microsoft Graph v1.0, read-only, used to confirm that the agent has " + "an Entra agent identity blueprint in Microsoft Agent 365. This source " + "is opt-in: set `identity.verify: true` in `agentops.yaml`. It needs " + "the `AgentIdentityBlueprint.Read.All` application permission with " + "tenant admin consent." + ), } # Recognized check flags. Keep this list short and stable. @@ -150,6 +158,15 @@ "observability.trace_replay_missing": ( "https://learn.microsoft.com/azure/foundry/concepts/observability" ), + "agent_identity.not_registered": ( + "https://learn.microsoft.com/entra/identity/agent-id/agent-id-overview" + ), + "agent_identity.not_recorded": ( + "https://learn.microsoft.com/entra/identity/agent-id/agent-id-overview" + ), + "agent_identity.lookup_failed": ( + "https://learn.microsoft.com/entra/identity/agent-id/agent-id-overview" + ), } @@ -687,6 +704,42 @@ def is_llm_judged(self) -> bool: # ------------------------------------------------------------------ # Security # ------------------------------------------------------------------ + CheckSpec( + id="agent_identity.not_registered", + category=Category.SECURITY, + title="Agent is not registered in Microsoft Agent 365", + summary=( + "No Entra Agent ID is recorded for the workspace, so the " + "agent is absent from the tenant agent inventory and cannot " + "be targeted by Conditional Access." + ), + severities=(Severity.WARNING,), + requires=("workspace",), + ), + CheckSpec( + id="agent_identity.not_recorded", + category=Category.SECURITY, + title="Agent identity exists in Entra but is not recorded locally", + summary=( + "Microsoft Entra has a matching agent identity blueprint but " + "the workspace has no local record, so traces and the release " + "evidence bundle cannot quote the Entra Agent ID." + ), + severities=(Severity.INFO,), + requires=("workspace", "graph"), + ), + CheckSpec( + id="agent_identity.lookup_failed", + category=Category.SECURITY, + title="Agent 365 registration could not be verified", + summary=( + "Microsoft Graph could not be consulted, usually because the " + "AgentIdentityBlueprint.Read.All permission has no admin " + "consent in the tenant." + ), + severities=(Severity.WARNING,), + requires=("workspace", "graph"), + ), CheckSpec( id="waf.security.local_auth_disabled", category=Category.SECURITY, diff --git a/src/agentops/cli/app.py b/src/agentops/cli/app.py index b509ab03..539d72ed 100644 --- a/src/agentops/cli/app.py +++ b/src/agentops/cli/app.py @@ -935,8 +935,42 @@ class ExplainPage: title="Agent server commands", command="agentops agent", synopsis=("agentops agent COMMAND [ARGS]...", "agentops agent explain"), - summary=("Contains commands that host AgentOps Doctor as an HTTP agent/Copilot Extension surface.",), - children=("serve",), + summary=("Contains commands that host AgentOps Doctor as an HTTP agent/Copilot Extension surface, and that manage the agent's Microsoft Entra identity.",), + children=("serve", "register"), + ), + ("agent", "register"): ExplainPage( + title="Register the agent identity blueprint", + command="agentops agent register", + synopsis=( + "agentops agent register [--sponsor UPN_OR_ID] [--display-name NAME] [--workspace PATH] [--dry-run]", + "agentops agent register explain", + ), + summary=( + "Creates the agent's identity blueprint in Microsoft Entra so the agent becomes a governed principal in Microsoft Agent 365.", + "Registration is what makes an agent visible in the tenant agent inventory, targetable by Conditional Access, and attributable to an accountable sponsor. Until it exists, the agent is just a workload with no identity of its own.", + "The command is idempotent: it looks the blueprint up by display name first and adopts the existing one instead of creating a duplicate.", + ), + how_it_works=( + "Resolves the display name from `--display-name`, then `identity.display_name` in `agentops.yaml`, then the agent target name.", + "Resolves the sponsor from `--sponsor` or `identity.sponsor` in `agentops.yaml`. A sponsor is mandatory: Agent 365 requires an accountable human owner.", + "Acquires an app-only Microsoft Graph token through the shared Azure credential chain.", + "Queries `GET /applications` filtered by display name. If a blueprint already exists it is reused.", + "Otherwise it POSTs a `Microsoft.Graph.AgentIdentityBlueprint` application to Graph v1.0.", + "Writes `.agentops/identity/agent-identity.json` so Doctor, the OTel exporter, and the release evidence pack can all quote the same Entra Agent ID.", + ), + inputs=( + "`agentops.yaml` keys `identity.sponsor`, `identity.display_name`, `identity.owner`.", + "Microsoft Graph application permission `AgentIdentityBlueprint.Create` with tenant admin consent.", + ), + outputs=( + "`.agentops/identity/agent-identity.json` containing `app_id`, `object_id`, and `display_name`.", + "The Entra Agent ID echoed to stdout for use in CI logs.", + ), + examples=( + "agentops agent register --sponsor jane@contoso.com", + "agentops agent register --dry-run", + ), + see_also=("agentops explain doctor", "agentops explain telemetry"), ), ("agent", "serve"): ExplainPage( title="Serve AgentOps as an HTTP agent", @@ -4791,6 +4825,7 @@ def _build_doctor_explain_text( "azure_monitor", "azure_resources", "judge_model", + "graph", ] lines: list[str] = _manual_banner( "AgentOps Doctor", @@ -5015,6 +5050,7 @@ def _build_doctor_explain_markdown( "azure_monitor", "azure_resources", "judge_model", + "graph", ] lines: list[str] = [ "# AgentOps Doctor manual", @@ -6159,6 +6195,96 @@ def cmd_agent_serve( uvicorn.run(fastapi_app, host=host, port=port, workers=workers) +@agent_app.command("register") +def cmd_agent_register( + sponsor: Annotated[ + str | None, + typer.Option( + "--sponsor", + help=( + "Accountable owner (UPN or object id). Falls back to " + "`identity.sponsor` in agentops.yaml." + ), + ), + ] = None, + display_name: Annotated[ + str | None, + typer.Option( + "--display-name", + help=( + "Blueprint display name. Falls back to " + "`identity.display_name`, then the agent target name." + ), + ), + ] = None, + workspace: Annotated[ + Path, + typer.Option("--workspace", "-w", help="Project root."), + ] = Path("."), + dry_run: Annotated[ + bool, + typer.Option( + "--dry-run", + help="Resolve inputs and report what would happen, without calling Graph.", + ), + ] = False, + explain: Annotated[str | None, typer.Argument(hidden=True)] = None, +) -> None: + """Register the agent's identity blueprint in Microsoft Entra. + + Idempotent: an existing blueprint with the same display name is + adopted rather than duplicated. The resolved Entra Agent ID is + written to ``.agentops/identity/agent-identity.json`` so Doctor, + tracing, and the release evidence pack all quote the same value. + """ + if _maybe_explain_leaf(("agent", "register"), explain): + return + + from agentops.services.agent_identity import ( + AgentIdentityError, + register_blueprint, + resolve_registration_inputs, + write_identity_record, + ) + + workspace = workspace.resolve() + + try: + resolved_name, resolved_sponsor = resolve_registration_inputs( + workspace, + display_name=display_name, + sponsor=sponsor, + ) + except AgentIdentityError as exc: + typer.echo(f"{_cli_error('Error')}: {exc}", err=True) + raise typer.Exit(code=1) from exc + + typer.echo(f"{_cli_label('Display name')}: {resolved_name}") + typer.echo(f"{_cli_label('Sponsor')}: {resolved_sponsor}") + + if dry_run: + typer.echo( + "Dry run: no Microsoft Graph call was made. " + "Re-run without --dry-run to register." + ) + return + + try: + blueprint, created = register_blueprint( + resolved_name, sponsor=resolved_sponsor + ) + except AgentIdentityError as exc: + typer.echo(f"{_cli_error('Error')}: {exc}", err=True) + raise typer.Exit(code=1) from exc + + record_path = write_identity_record(workspace, blueprint, created=created) + + action = "Registered" if created else "Reused existing" + typer.echo(f"{_cli_label(action + ' agent identity')}: {blueprint.app_id}") + typer.echo(f"{_cli_label('Wrote')}: {_cli_path(record_path)}") + typer.echo(f"{_cli_label('Entra portal')}: {blueprint.portal_url}") + + @app.command("cockpit") def cmd_cockpit( host: Annotated[ diff --git a/src/agentops/core/agentops_config.py b/src/agentops/core/agentops_config.py index 7be891c8..2728a374 100644 --- a/src/agentops/core/agentops_config.py +++ b/src/agentops/core/agentops_config.py @@ -733,6 +733,50 @@ class RedTeamRunConfig(BaseModel): model_config = ConfigDict(extra="forbid") +class AgentIdentityConfig(BaseModel): + """Optional Microsoft Agent 365 identity settings. + + Declares the agent's first-class identity in Microsoft Entra so Doctor, + tracing, and the release evidence pack all reference the same Entra Agent + ID. AgentOps never invents these values: ``agentops agent register`` + creates or adopts the blueprint and records the resolved id under + ``.agentops/identity/agent-identity.json``. + + Example:: + + identity: + display_name: support-agent + sponsor: owner@contoso.com + verify: true + """ + + display_name: Optional[str] = Field( + None, + description=( + "Blueprint display name in Microsoft Entra. When omitted, " + "AgentOps derives one from the 'agent' target." + ), + ) + sponsor: Optional[str] = Field( + None, + description=( + "Accountable owner (UPN or object id). Required to register a " + "blueprint; there is no silent fallback because an unowned agent " + "identity cannot be governed." + ), + ) + verify: bool = Field( + False, + description=( + "When true, Doctor calls Microsoft Graph to confirm the blueprint " + "exists. Off by default because the lookup needs tenant admin " + "consent that most workspaces will not have on day one." + ), + ) + + model_config = ConfigDict(extra="forbid") + + class StreamConfig(BaseModel): """Streaming aggregation options for ``http-json`` targets. @@ -867,6 +911,13 @@ class AgentOpsConfig(BaseModel): single-turn evals working while letting Doctor, Cockpit, CI evidence, and azd/Foundry recipes reason about multi-turn coverage, rubric gates, trace sampling, and trace replay links. + + ``identity`` + Optional Microsoft Agent 365 identity settings (display name, sponsor, + and whether Doctor verifies the blueprint against Microsoft Graph). The + resolved Entra Agent ID itself is not stored here; it lives in + ``.agentops/identity/agent-identity.json`` after + ``agentops agent register``. """ version: int = Field(..., description="Schema version. Must be 1.") @@ -932,6 +983,14 @@ class AgentOpsConfig(BaseModel): "redteam_path automatically." ), ) + identity: Optional[AgentIdentityConfig] = Field( + None, + description=( + "Optional Microsoft Agent 365 identity settings used by 'agentops " + "agent register', the Doctor registration posture check, trace " + "stamping, and the release evidence pack." + ), + ) thresholds: Dict[str, Any] = Field( default_factory=dict, diff --git a/src/agentops/core/release_evidence.py b/src/agentops/core/release_evidence.py index 06778399..33f094d5 100644 --- a/src/agentops/core/release_evidence.py +++ b/src/agentops/core/release_evidence.py @@ -54,5 +54,6 @@ class ReleaseEvidence(BaseModel): observability: Dict[str, Any] = Field(default_factory=dict) ailz: Dict[str, Any] = Field(default_factory=dict) governance: Dict[str, Any] = Field(default_factory=dict) + agent_identity: Dict[str, Any] = Field(default_factory=dict) model_config = ConfigDict(extra="forbid") diff --git a/src/agentops/services/agent_identity.py b/src/agentops/services/agent_identity.py new file mode 100644 index 00000000..3c3f29b7 --- /dev/null +++ b/src/agentops/services/agent_identity.py @@ -0,0 +1,580 @@ +"""Microsoft Entra Agent ID support for the Agent 365 control plane. + +An agent produced by the accelerator is invisible to the Agent 365 control +plane until an *agent identity blueprint* exists for it in Microsoft Entra. +This module owns the three things AgentOps needs around that identity: + +* a very small app-only Microsoft Graph client (:class:`GraphClient`), +* create/lookup of the blueprint (:func:`lookup_blueprint`, + :func:`register_blueprint`), +* on-disk persistence of the resulting ``appId`` so the doctor check, the + OTel resource attributes, and the release evidence bundle can all quote the + same value without re-calling Graph. + +Design notes +------------ + +**Read-only by default.** Nothing here runs implicitly. The doctor check does +a read-only lookup, and registration only happens when the operator asks for +it explicitly. + +**No new dependency.** Graph is a plain REST API, so we use +:mod:`urllib.request` from the standard library rather than pulling in an SDK. +The credential comes from the same shared factory the doctor sources use. + +**Errors are messages, not stack traces.** Every failure mode a user can +realistically hit (missing consent, missing role, no credential, package not +installed) is converted into an :class:`AgentIdentityError` carrying a +sentence that says what to do next. +""" + +from __future__ import annotations + +import json +import logging +import os +import urllib.error +import urllib.parse +import urllib.request +from dataclasses import dataclass +from datetime import datetime, timezone +from pathlib import Path +from typing import Any, Mapping, Optional + +log = logging.getLogger(__name__) + +#: Graph endpoint AgentOps talks to. v1.0 only - the ``AgentIdentity*`` +#: scopes this module needs are published there. +GRAPH_BASE_URL = "https://graph.microsoft.com/v1.0" + +#: App-only token scope for Microsoft Graph. +GRAPH_SCOPE = "https://graph.microsoft.com/.default" + +#: OData type discriminator for an agent identity blueprint. +BLUEPRINT_ODATA_TYPE = "Microsoft.Graph.AgentIdentityBlueprint" + +#: OTel resource attribute carrying the Entra Agent ID. +AGENT_ID_ATTRIBUTE = "gen_ai.agent.id" + +#: Environment override for the Entra Agent ID. CI exports this when the +#: identity is provisioned outside the workspace (for example by a platform +#: team) so traces still carry the right value without a local record. +AGENT_ID_ENV = "AGENTOPS_ENTRA_AGENT_ID" + +#: Where the resolved identity is persisted inside a workspace. +IDENTITY_RECORD_RELPATH = Path(".agentops") / "identity" / "agent-identity.json" + +#: Documentation pointer used in remediation text. +REGISTRATION_DOCS_URL = ( + "https://learn.microsoft.com/entra/identity/agent-id/agent-id-overview" +) + +#: Deep link template for an application object in the Entra admin center. +ENTRA_APP_DEEPLINK = ( + "https://entra.microsoft.com/#view/Microsoft_AAD_RegisteredApps" + "/ApplicationMenuBlade/~/Overview/appId/{app_id}" +) + + +class AgentIdentityError(RuntimeError): + """A user-actionable failure while talking to Microsoft Graph. + + The message is expected to be shown verbatim to the operator, so it + always explains the remediation rather than quoting an HTTP status. + """ + + +@dataclass(frozen=True) +class AgentIdentityBlueprint: + """An agent identity blueprint as Agent 365 sees it.""" + + app_id: str + object_id: Optional[str] = None + display_name: Optional[str] = None + + @property + def portal_url(self) -> str: + return ENTRA_APP_DEEPLINK.format(app_id=self.app_id) + + def to_dict(self) -> dict[str, Any]: + return { + "app_id": self.app_id, + "object_id": self.object_id, + "display_name": self.display_name, + "portal_url": self.portal_url, + } + + +# --------------------------------------------------------------------------- +# Graph client +# --------------------------------------------------------------------------- + + +class GraphClient: + """Minimal app-only Microsoft Graph client. + + Only the two verbs this feature needs are implemented. Tests inject a + stand-in with the same ``get``/``post`` shape rather than patching + :mod:`urllib`. + """ + + def __init__( + self, + *, + token: Optional[str] = None, + base_url: str = GRAPH_BASE_URL, + timeout: int = 30, + ) -> None: + self._token = token + self._base_url = base_url.rstrip("/") + self._timeout = timeout + + # -- public API ------------------------------------------------------ + + def get(self, path: str, *, params: Optional[Mapping[str, str]] = None) -> Any: + url = self._url(path) + if params: + url = f"{url}?{urllib.parse.urlencode(dict(params))}" + return self._request("GET", url, body=None) + + def post(self, path: str, body: Mapping[str, Any]) -> Any: + return self._request("POST", self._url(path), body=dict(body)) + + # -- internals ------------------------------------------------------- + + def _url(self, path: str) -> str: + return f"{self._base_url}/{path.lstrip('/')}" + + def _headers(self) -> dict[str, str]: + return { + "Authorization": f"Bearer {self._resolve_token()}", + "Accept": "application/json", + "Content-Type": "application/json", + # Agent identity blueprints are an OData-typed resource; Graph + # rejects the payload without the version header. + "OData-Version": "4.0", + } + + def _resolve_token(self) -> str: + if self._token: + return self._token + self._token = acquire_graph_token() + return self._token + + def _request(self, method: str, url: str, *, body: Optional[dict[str, Any]]) -> Any: + data = json.dumps(body).encode("utf-8") if body is not None else None + request = urllib.request.Request( + url, data=data, method=method, headers=self._headers() + ) + try: + with urllib.request.urlopen(request, timeout=self._timeout) as response: + raw = response.read().decode("utf-8") + except urllib.error.HTTPError as exc: + raise _http_error(exc) from None + except urllib.error.URLError as exc: + raise AgentIdentityError( + "Could not reach Microsoft Graph " + f"({exc.reason}). Check network connectivity or proxy settings." + ) from None + if not raw.strip(): + return {} + try: + return json.loads(raw) + except json.JSONDecodeError: + raise AgentIdentityError( + "Microsoft Graph returned a response that is not valid JSON." + ) from None + + +def _http_error(exc: urllib.error.HTTPError) -> AgentIdentityError: + """Translate an HTTP failure into an actionable message.""" + + detail = _graph_error_message(exc) + status = exc.code + if status in (401,): + return AgentIdentityError( + "Microsoft Graph rejected the credential (401). Sign in with an " + "identity that has app-only access to Graph, or set the " + "AZURE_CLIENT_ID / AZURE_TENANT_ID / AZURE_CLIENT_SECRET " + f"environment variables in CI. {detail}".strip() + ) + if status in (403,): + return AgentIdentityError( + "Microsoft Graph denied the request (403). The app registration is " + "missing admin consent for AgentIdentityBlueprint.Read.All (lookup) " + "or AgentIdentityBlueprint.Create (registration). Ask a tenant " + f"admin to grant consent, then retry. {detail}".strip() + ) + if status == 404: + return AgentIdentityError( + "Microsoft Graph returned 404 for the agent identity endpoint. The " + "tenant may not be enrolled in Microsoft Agent 365 yet. " + f"See {REGISTRATION_DOCS_URL}. {detail}".strip() + ) + if status == 429: + return AgentIdentityError( + "Microsoft Graph throttled the request (429). Retry in a few " + f"seconds. {detail}".strip() + ) + return AgentIdentityError( + f"Microsoft Graph request failed with HTTP {status}. {detail}".strip() + ) + + +def _graph_error_message(exc: urllib.error.HTTPError) -> str: + """Best-effort extraction of the Graph ``error.message`` field.""" + + try: + payload = json.loads(exc.read().decode("utf-8")) + except Exception: # noqa: BLE001 - error bodies are unreliable + return "" + error = payload.get("error") if isinstance(payload, dict) else None + if isinstance(error, dict): + message = error.get("message") + if isinstance(message, str) and message.strip(): + return f"Graph said: {message.strip()}" + return "" + + +def acquire_graph_token() -> str: + """Return an app-only Microsoft Graph access token. + + Reuses the shared doctor credential so a process that already + authenticated for Azure Monitor does not walk the credential chain again. + """ + + try: + from agentops.agent.sources._credentials import ( + format_source_error, + get_shared_credential, + ) + except ImportError: # pragma: no cover - package layout guarantees this + raise AgentIdentityError( + "AgentOps could not load its credential helper." + ) from None + + try: + credential = get_shared_credential() + except ImportError: + raise AgentIdentityError( + "The 'azure-identity' package is required to talk to Microsoft " + "Graph. Install it with: pip install azure-identity" + ) from None + + try: + token = credential.get_token(GRAPH_SCOPE) + except Exception as exc: # noqa: BLE001 - surfaced as a clean message + raise AgentIdentityError( + "Could not acquire a Microsoft Graph token: " + f"{format_source_error(exc)}" + ) from None + + value = getattr(token, "token", None) + if not value: + raise AgentIdentityError( + "The credential returned an empty Microsoft Graph token." + ) + return str(value) + + +# --------------------------------------------------------------------------- +# Blueprint lookup / registration +# --------------------------------------------------------------------------- + + +def _escape_odata_literal(value: str) -> str: + """Escape a string for use inside an OData ``$filter`` literal.""" + + return value.replace("'", "''") + + +def _blueprint_from_payload(payload: Mapping[str, Any]) -> Optional[AgentIdentityBlueprint]: + app_id = payload.get("appId") + if not isinstance(app_id, str) or not app_id.strip(): + return None + object_id = payload.get("id") + display_name = payload.get("displayName") + return AgentIdentityBlueprint( + app_id=app_id.strip(), + object_id=object_id if isinstance(object_id, str) else None, + display_name=display_name if isinstance(display_name, str) else None, + ) + + +def lookup_blueprint( + display_name: str, + *, + client: Optional[GraphClient] = None, +) -> Optional[AgentIdentityBlueprint]: + """Return the agent identity blueprint named ``display_name``, if any. + + Read-only. Returns ``None`` when the tenant has no blueprint with that + display name; raises :class:`AgentIdentityError` when Graph could not be + consulted at all (so callers can tell "not registered" apart from + "could not check"). + """ + + name = (display_name or "").strip() + if not name: + raise AgentIdentityError( + "An agent display name is required to look up its identity." + ) + + graph = client or GraphClient() + payload = graph.get( + "/applications", + params={ + "$filter": f"displayName eq '{_escape_odata_literal(name)}'", + "$select": "id,appId,displayName", + "$top": "1", + }, + ) + values = payload.get("value") if isinstance(payload, Mapping) else None + if not isinstance(values, list) or not values: + return None + first = values[0] + if not isinstance(first, Mapping): + return None + return _blueprint_from_payload(first) + + +def register_blueprint( + display_name: str, + *, + sponsor: str, + client: Optional[GraphClient] = None, +) -> tuple[AgentIdentityBlueprint, bool]: + """Create the agent identity blueprint, or return the existing one. + + Returns ``(blueprint, created)`` where ``created`` is ``False`` when a + blueprint with the same display name already existed. Re-running is + therefore safe: no duplicate is ever created. + """ + + name = (display_name or "").strip() + if not name: + raise AgentIdentityError( + "An agent display name is required to register an agent identity." + ) + if not (sponsor or "").strip(): + raise AgentIdentityError( + "A sponsor is required to register an agent identity. Set " + "'identity.sponsor' in agentops.yaml to the object id or UPN of " + "the human accountable for this agent." + ) + + graph = client or GraphClient() + + existing = lookup_blueprint(name, client=graph) + if existing is not None: + return existing, False + + payload = graph.post( + "/applications", + { + "@odata.type": BLUEPRINT_ODATA_TYPE, + "displayName": name, + "sponsors": [sponsor.strip()], + }, + ) + if not isinstance(payload, Mapping): + raise AgentIdentityError( + "Microsoft Graph accepted the registration but returned no " + "application object." + ) + blueprint = _blueprint_from_payload(payload) + if blueprint is None: + raise AgentIdentityError( + "Microsoft Graph accepted the registration but returned no appId." + ) + return blueprint, True + + +# --------------------------------------------------------------------------- +# Workspace persistence +# --------------------------------------------------------------------------- + + +def identity_record_path(workspace: Path) -> Path: + return Path(workspace) / IDENTITY_RECORD_RELPATH + + +def write_identity_record( + workspace: Path, + blueprint: AgentIdentityBlueprint, + *, + created: bool = False, +) -> Path: + """Persist ``blueprint`` under ``.agentops/identity/`` and return the path.""" + + path = identity_record_path(workspace) + path.parent.mkdir(parents=True, exist_ok=True) + record = { + "version": 1, + "recorded_at": datetime.now(timezone.utc).isoformat(), + "created": bool(created), + **blueprint.to_dict(), + } + path.write_text(json.dumps(record, indent=2) + "\n", encoding="utf-8") + return path + + +def read_identity_record(workspace: Path) -> Optional[dict[str, Any]]: + """Return the persisted identity record, or ``None`` when absent/invalid.""" + + path = identity_record_path(workspace) + if not path.exists(): + return None + try: + payload = json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError): + return None + return payload if isinstance(payload, dict) else None + + +def resolve_agent_id(workspace: Optional[Path] = None) -> Optional[str]: + """Return the Entra Agent ID for this workspace. + + Resolution order: the ``AGENTOPS_ENTRA_AGENT_ID`` environment variable + (so CI can inject an identity provisioned elsewhere), then the persisted + record. Returns ``None`` when the agent has no known identity, which is + the normal state before registration. + """ + + override = os.environ.get(AGENT_ID_ENV, "").strip() + if override and "$(" not in override and "${{" not in override: + return override + if workspace is None: + return None + record = read_identity_record(workspace) + if not record: + return None + app_id = record.get("app_id") + return app_id.strip() if isinstance(app_id, str) and app_id.strip() else None + + +# --------------------------------------------------------------------------- +# Workspace configuration +# --------------------------------------------------------------------------- + + +def load_identity_config(workspace: Path) -> dict[str, Any]: + """Return the ``identity`` block from ``agentops.yaml``, or ``{}``.""" + + path = Path(workspace) / "agentops.yaml" + if not path.exists(): + return {} + try: + from agentops.utils.yaml import load_yaml + + data = load_yaml(path) + except Exception: # noqa: BLE001 - config problems must not crash callers + return {} + if not isinstance(data, dict): + return {} + identity = data.get("identity") + return identity if isinstance(identity, dict) else {} + + +def resolve_display_name( + workspace: Path, *, override: Optional[str] = None +) -> Optional[str]: + """Resolve the blueprint display name for ``workspace``. + + Precedence: explicit override, then ``identity.display_name``, then a + previously recorded name, then the agent target name from + ``agentops.yaml``. Returns ``None`` when nothing usable is configured. + """ + + if isinstance(override, str) and override.strip(): + return override.strip() + + identity = load_identity_config(workspace) + configured = identity.get("display_name") + if isinstance(configured, str) and configured.strip(): + return configured.strip() + + record = read_identity_record(workspace) or {} + recorded = record.get("display_name") + if isinstance(recorded, str) and recorded.strip(): + return recorded.strip() + + return _name_from_target(workspace) + + +def _name_from_target(workspace: Path) -> Optional[str]: + path = Path(workspace) / "agentops.yaml" + if not path.exists(): + return None + try: + from agentops.utils.yaml import load_yaml + + data = load_yaml(path) + except Exception: # noqa: BLE001 + return None + if not isinstance(data, dict): + return None + agent = data.get("agent") + if not isinstance(agent, str): + return None + raw = agent.strip() + if not raw or "://" in raw: + return None + name = raw.split(":", 1)[0].strip() + return name or None + + +def resolve_registration_inputs( + workspace: Path, + *, + display_name: Optional[str] = None, + sponsor: Optional[str] = None, +) -> tuple[str, str]: + """Resolve ``(display_name, sponsor)`` for a registration call. + + Raises :class:`AgentIdentityError` with an actionable sentence when + either value is missing, because Agent 365 rejects a blueprint that has + no name and AgentOps refuses to create one that has no accountable owner. + """ + + identity = load_identity_config(workspace) + + resolved_name = resolve_display_name(workspace, override=display_name) + if not resolved_name: + raise AgentIdentityError( + "No display name for the agent identity. Pass --display-name, or " + "set 'identity.display_name' in agentops.yaml." + ) + + resolved_sponsor = sponsor if isinstance(sponsor, str) else None + if not (resolved_sponsor and resolved_sponsor.strip()): + configured = identity.get("sponsor") + resolved_sponsor = configured if isinstance(configured, str) else None + if not (resolved_sponsor and resolved_sponsor.strip()): + raise AgentIdentityError( + "No sponsor for the agent identity. Microsoft Agent 365 requires an " + "accountable owner. Pass --sponsor , or set " + "'identity.sponsor' in agentops.yaml." + ) + + return resolved_name, resolved_sponsor.strip() + + +__all__ = [ + "AGENT_ID_ATTRIBUTE", + "AGENT_ID_ENV", + "AgentIdentityBlueprint", + "AgentIdentityError", + "GraphClient", + "REGISTRATION_DOCS_URL", + "acquire_graph_token", + "identity_record_path", + "load_identity_config", + "lookup_blueprint", + "read_identity_record", + "register_blueprint", + "resolve_agent_id", + "resolve_display_name", + "resolve_registration_inputs", + "write_identity_record", +] diff --git a/src/agentops/services/evidence_pack.py b/src/agentops/services/evidence_pack.py index fc2814ac..6fa65ad3 100644 --- a/src/agentops/services/evidence_pack.py +++ b/src/agentops/services/evidence_pack.py @@ -68,6 +68,7 @@ def build_release_evidence( observability = _observability_status(root, trace_dataset) ailz = _ailz_status(analysis) governance = _governance_status(root) + agent_identity = _agent_identity_status(root) checks: list[ReleaseEvidenceCheck] = [] blockers: list[str] = [] @@ -85,6 +86,7 @@ def build_release_evidence( _add_trace_dataset_check(checks, warnings, ready, trace_dataset) _add_ailz_check(checks, warnings, ready, ailz) _add_governance_check(checks, warnings, ready, governance) + _add_agent_identity_check(checks, warnings, ready, agent_identity) status = "blocked" if blockers else "ready_with_warnings" if warnings else "ready" links = _links(latest_eval, observability) @@ -111,6 +113,7 @@ def build_release_evidence( observability=observability, ailz=ailz, governance=governance, + agent_identity=agent_identity, ) return ReleaseEvidence.model_validate(_redact_obj(evidence.model_dump())) @@ -834,6 +837,78 @@ def _add_governance_check( ) +def _agent_identity_status(root: Path) -> dict[str, Any]: + """Summarize the agent's Entra identity for the evidence bundle. + + The bundle records the Entra Agent ID so a release artifact can be tied + back to the governed principal that produced its traces. Absence is + reported as a status rather than an error, because registration is + opt-in until a tenant has Agent 365 enabled. + """ + + from agentops.services.agent_identity import ( + identity_record_path, + read_identity_record, + resolve_agent_id, + ) + + record = read_identity_record(root) or {} + agent_id = resolve_agent_id(root) + if not agent_id: + return {"status": "not_registered", "agent_id": None} + + summary: dict[str, Any] = { + "status": "registered", + "agent_id": agent_id, + "source": "record" if record.get("app_id") else "environment", + } + for key in ("display_name", "object_id", "recorded_at"): + value = record.get(key) + if isinstance(value, str) and value.strip(): + summary[key] = value.strip() + if record: + summary["record_path"] = str(identity_record_path(root)) + return summary + + +def _add_agent_identity_check( + checks: list[ReleaseEvidenceCheck], + warnings: list[str], + ready: list[str], + agent_identity: dict[str, Any], +) -> None: + if agent_identity.get("status") != "registered": + message = ( + "Agent identity is not registered in Microsoft Entra, so release " + "evidence cannot be attributed to a governed principal. " + "Run `agentops agent register --sponsor `." + ) + warnings.append(message) + checks.append( + ReleaseEvidenceCheck( + name="Agent identity", + status="warning", + summary=message, + evidence=agent_identity, + ) + ) + return + + message = ( + "Agent identity " + f"{agent_identity.get('agent_id')} is registered and stamped on traces." + ) + ready.append(message) + checks.append( + ReleaseEvidenceCheck( + name="Agent identity", + status="ready", + summary=message, + evidence=agent_identity, + ) + ) + + def _links(latest_eval: dict[str, Any], observability: dict[str, Any]) -> list[ReleaseEvidenceLink]: links: list[ReleaseEvidenceLink] = [] report_url = latest_eval.get("foundry_report_url") diff --git a/src/agentops/templates/workflows/agentops-deploy-prod-azd.yml b/src/agentops/templates/workflows/agentops-deploy-prod-azd.yml index 2410a5dc..ec8f8acf 100644 --- a/src/agentops/templates/workflows/agentops-deploy-prod-azd.yml +++ b/src/agentops/templates/workflows/agentops-deploy-prod-azd.yml @@ -170,6 +170,25 @@ __EVAL_STEPS__ tenant-id: ${{ vars.AZURE_TENANT_ID }} subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID }} __AZD_CLI_SETUP__ + # Opt-in: register the agent's Microsoft Entra identity blueprint so + # traces and release evidence can be attributed to a governed agent. + # Off unless you set the `AGENTOPS_IDENTITY_SPONSOR` repository variable + # to the accountable owner's UPN. The command is idempotent: an existing + # blueprint with the same display name is adopted, not duplicated. + # Requires the workload identity to hold `AgentIdentityBlueprint.Create`. + - name: Set up Python (agent identity) + if: vars.AGENTOPS_IDENTITY_SPONSOR != '' + uses: actions/setup-python@v6 + with: + python-version: "3.11" + - name: Register agent identity (optional) + if: vars.AGENTOPS_IDENTITY_SPONSOR != '' + env: + AZURE_TENANT_ID: ${{ vars.AZURE_TENANT_ID }} + run: | + pip install "agentops-accelerator[agent]__AGENTOPS_INSTALL_SPEC__" + agentops agent register --workspace . \ + --sponsor "${{ vars.AGENTOPS_IDENTITY_SPONSOR }}" - name: Run azd deploy env: AZURE_ENV_NAME: ${{ vars.AZURE_ENV_NAME || 'production' }} diff --git a/src/agentops/templates/workflows/agentops-deploy-prod.yml b/src/agentops/templates/workflows/agentops-deploy-prod.yml index 01a8be9e..5de7497e 100644 --- a/src/agentops/templates/workflows/agentops-deploy-prod.yml +++ b/src/agentops/templates/workflows/agentops-deploy-prod.yml @@ -162,6 +162,27 @@ __EVAL_STEPS__ tenant-id: ${{ vars.AZURE_TENANT_ID }} subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID }} + # Opt-in: register the agent's Microsoft Entra identity blueprint so + # traces and release evidence can be attributed to a governed agent. + # Off unless you set the `AGENTOPS_IDENTITY_SPONSOR` repository variable + # to the accountable owner's UPN. The command is idempotent: an existing + # blueprint with the same display name is adopted, not duplicated. + # Requires the workload identity to hold `AgentIdentityBlueprint.Create`. + - name: Set up Python (agent identity) + if: vars.AGENTOPS_IDENTITY_SPONSOR != '' + uses: actions/setup-python@v6 + with: + python-version: "3.11" + + - name: Register agent identity (optional) + if: vars.AGENTOPS_IDENTITY_SPONSOR != '' + env: + AZURE_TENANT_ID: ${{ vars.AZURE_TENANT_ID }} + run: | + pip install "agentops-accelerator[agent]__AGENTOPS_INSTALL_SPEC__" + agentops agent register --workspace . \ + --sponsor "${{ vars.AGENTOPS_IDENTITY_SPONSOR }}" + # --------------------------------------------------------------- # TODO: replace this placeholder with your deploy step. # See agentops-deploy-dev.yml for example snippets (ACA, App diff --git a/src/agentops/utils/telemetry.py b/src/agentops/utils/telemetry.py index 875e63e6..7c2e6588 100644 --- a/src/agentops/utils/telemetry.py +++ b/src/agentops/utils/telemetry.py @@ -129,14 +129,7 @@ def init_tracing() -> None: from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor - import agentops - - resource = Resource( - attributes={ - "service.name": "agentops", - "service.version": getattr(agentops, "__version__", "0.0.0"), - } - ) + resource = Resource(attributes=_resource_attributes()) provider = TracerProvider(resource=resource) exporter = OTLPSpanExporter(endpoint=otlp_endpoint + "/v1/traces") @@ -198,18 +191,48 @@ def _appinsights_connection_string_parts(value: str) -> dict[str, str]: return parts +def _resource_attributes() -> dict: + """Resource attributes shared by every AgentOps tracer provider. + + The Entra Agent ID is stamped as ``gen_ai.agent.id`` when the workspace + has a registered identity. Without it every span is attributable to a + service but not to a governed principal, which is exactly the gap that + makes agent traces hard to audit. The attribute is omitted rather than + emitted empty so downstream queries can filter on its presence. + """ + + import agentops + + attributes = { + "service.name": "agentops", + "service.version": getattr(agentops, "__version__", "0.0.0"), + } + try: + from pathlib import Path + + from agentops.services.agent_identity import ( + AGENT_ID_ATTRIBUTE, + resolve_agent_id, + ) + + agent_id = resolve_agent_id(Path.cwd()) + if agent_id: + attributes[AGENT_ID_ATTRIBUTE] = agent_id + except Exception: # noqa: BLE001 - identity is optional, tracing is not + pass + return attributes + + def _agentops_resource() -> Optional[Any]: try: from opentelemetry.sdk.resources import Resource - import agentops except Exception: # noqa: BLE001 return None - return Resource.create( - { - "service.name": "agentops", - "service.version": getattr(agentops, "__version__", "0.0.0"), - } - ) + try: + attributes = _resource_attributes() + except Exception: # noqa: BLE001 + return None + return Resource.create(attributes) def shutdown() -> None: diff --git a/tests/unit/test_agent_checks_agent_identity.py b/tests/unit/test_agent_checks_agent_identity.py new file mode 100644 index 00000000..645efe17 --- /dev/null +++ b/tests/unit/test_agent_checks_agent_identity.py @@ -0,0 +1,123 @@ +"""Tests for the Agent 365 registration posture check.""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from agentops.agent.checks import agent_identity as check_module +from agentops.agent.checks.agent_identity import SOURCE_NAME, run_agent_identity_check +from agentops.agent.findings import Category, Severity +from agentops.services.agent_identity import ( + AGENT_ID_ENV, + AgentIdentityBlueprint, + AgentIdentityError, + write_identity_record, +) + + +@pytest.fixture(autouse=True) +def _clear_agent_id_env(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.delenv(AGENT_ID_ENV, raising=False) + + +def _write_config(workspace: Path, body: str) -> None: + (workspace / "agentops.yaml").write_text(body, encoding="utf-8") + + +def test_registered_workspace_produces_no_findings(tmp_path: Path) -> None: + write_identity_record(tmp_path, AgentIdentityBlueprint(app_id="app-1")) + assert run_agent_identity_check(tmp_path) == [] + + +def test_environment_provided_identity_produces_no_findings( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.setenv(AGENT_ID_ENV, "app-from-ci") + assert run_agent_identity_check(tmp_path) == [] + + +def test_unregistered_workspace_warns(tmp_path: Path) -> None: + findings = run_agent_identity_check(tmp_path) + assert [f.id for f in findings] == ["agent_identity.not_registered"] + finding = findings[0] + assert finding.severity is Severity.WARNING + assert finding.category is Category.SECURITY + assert finding.source == SOURCE_NAME + assert "agentops agent register" in finding.recommendation + + +def test_graph_is_not_called_when_verify_is_off( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """Verification is opt-in: most tenants lack the Graph consent on day one.""" + + def _boom(*args: object, **kwargs: object) -> None: + raise AssertionError("Graph must not be consulted when verify is off") + + monkeypatch.setattr(check_module, "lookup_blueprint", _boom) + findings = run_agent_identity_check(tmp_path) + assert [f.id for f in findings] == ["agent_identity.not_registered"] + + +def test_verify_reports_blueprint_that_is_not_recorded_locally( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + _write_config(tmp_path, "identity:\n verify: true\n display_name: support-agent\n") + monkeypatch.setattr( + check_module, + "lookup_blueprint", + lambda name: AgentIdentityBlueprint(app_id="app-1", display_name=name), + ) + findings = run_agent_identity_check(tmp_path) + assert [f.id for f in findings] == ["agent_identity.not_recorded"] + assert findings[0].severity is Severity.INFO + assert findings[0].evidence["app_id"] == "app-1" + + +def test_verify_reports_not_registered_when_graph_finds_nothing( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + _write_config(tmp_path, "identity:\n verify: true\n") + monkeypatch.setattr(check_module, "lookup_blueprint", lambda name: None) + findings = run_agent_identity_check(tmp_path) + assert [f.id for f in findings] == ["agent_identity.not_registered"] + + +def test_graph_failure_becomes_a_readable_warning( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """A missing consent must never surface as a stack trace.""" + + def _raise(name: str) -> None: + raise AgentIdentityError("Admin consent is missing for AgentIdentityBlueprint.Read.All.") + + _write_config(tmp_path, "identity:\n verify: true\n") + monkeypatch.setattr(check_module, "lookup_blueprint", _raise) + findings = run_agent_identity_check(tmp_path) + assert [f.id for f in findings] == ["agent_identity.lookup_failed"] + assert findings[0].severity is Severity.WARNING + assert "Admin consent is missing" in findings[0].summary + + +@pytest.mark.parametrize("raw", ["true", "True", "yes", "on", "1"]) +def test_verify_accepts_string_truthy_values( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, raw: str +) -> None: + _write_config(tmp_path, f"identity:\n verify: '{raw}'\n") + monkeypatch.setattr( + check_module, + "lookup_blueprint", + lambda name: AgentIdentityBlueprint(app_id="app-1"), + ) + assert [f.id for f in run_agent_identity_check(tmp_path)] == [ + "agent_identity.not_recorded" + ] + + +def test_check_falls_back_to_directory_name_for_display_name(tmp_path: Path) -> None: + workspace = tmp_path / "my-agent" + workspace.mkdir() + findings = run_agent_identity_check(workspace) + assert findings[0].evidence["display_name"] == "my-agent" diff --git a/tests/unit/test_agent_identity_service.py b/tests/unit/test_agent_identity_service.py new file mode 100644 index 00000000..147c04d4 --- /dev/null +++ b/tests/unit/test_agent_identity_service.py @@ -0,0 +1,284 @@ +"""Tests for the Entra agent identity service. + +Every Graph call is exercised through an injected double, so the suite never +needs a tenant, a token, or network access. +""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any, Mapping + +import pytest + +from agentops.services.agent_identity import ( + AGENT_ID_ATTRIBUTE, + AGENT_ID_ENV, + AgentIdentityBlueprint, + AgentIdentityError, + identity_record_path, + load_identity_config, + lookup_blueprint, + read_identity_record, + register_blueprint, + resolve_agent_id, + resolve_display_name, + resolve_registration_inputs, + write_identity_record, +) + + +class FakeGraphClient: + """Records calls and replays canned responses.""" + + def __init__( + self, + *, + get_response: Mapping[str, Any] | None = None, + post_response: Mapping[str, Any] | None = None, + get_error: Exception | None = None, + ) -> None: + self._get_response = get_response if get_response is not None else {"value": []} + self._post_response = post_response or {} + self._get_error = get_error + self.get_calls: list[tuple[str, dict[str, str] | None]] = [] + self.post_calls: list[tuple[str, dict[str, Any]]] = [] + + def get(self, path: str, params: dict[str, str] | None = None) -> Mapping[str, Any]: + self.get_calls.append((path, params)) + if self._get_error is not None: + raise self._get_error + return self._get_response + + def post(self, path: str, body: dict[str, Any]) -> Mapping[str, Any]: + self.post_calls.append((path, body)) + return self._post_response + + +BLUEPRINT_PAYLOAD = { + "id": "object-1", + "appId": "app-1", + "displayName": "support-agent", +} + + +def test_agent_id_attribute_matches_semantic_convention() -> None: + assert AGENT_ID_ATTRIBUTE == "gen_ai.agent.id" + + +# --------------------------------------------------------------------------- +# lookup_blueprint +# --------------------------------------------------------------------------- + + +def test_lookup_returns_none_when_tenant_has_no_blueprint() -> None: + client = FakeGraphClient(get_response={"value": []}) + assert lookup_blueprint("support-agent", client=client) is None + path, params = client.get_calls[0] + assert path == "/applications" + assert params is not None + assert params["$filter"] == "displayName eq 'support-agent'" + + +def test_lookup_returns_blueprint_when_found() -> None: + client = FakeGraphClient(get_response={"value": [BLUEPRINT_PAYLOAD]}) + blueprint = lookup_blueprint("support-agent", client=client) + assert blueprint is not None + assert blueprint.app_id == "app-1" + assert blueprint.object_id == "object-1" + assert blueprint.display_name == "support-agent" + + +def test_lookup_escapes_single_quotes_in_display_name() -> None: + client = FakeGraphClient(get_response={"value": []}) + lookup_blueprint("o'brien agent", client=client) + _, params = client.get_calls[0] + assert params is not None + assert params["$filter"] == "displayName eq 'o''brien agent'" + + +def test_lookup_rejects_blank_display_name() -> None: + with pytest.raises(AgentIdentityError): + lookup_blueprint(" ", client=FakeGraphClient()) + + +def test_lookup_propagates_graph_errors() -> None: + boom = AgentIdentityError("consent missing") + client = FakeGraphClient(get_error=boom) + with pytest.raises(AgentIdentityError, match="consent missing"): + lookup_blueprint("support-agent", client=client) + + +# --------------------------------------------------------------------------- +# register_blueprint +# --------------------------------------------------------------------------- + + +def test_register_creates_blueprint_when_absent() -> None: + client = FakeGraphClient( + get_response={"value": []}, + post_response=BLUEPRINT_PAYLOAD, + ) + blueprint, created = register_blueprint( + "support-agent", sponsor="paulo@contoso.com", client=client + ) + assert created is True + assert blueprint.app_id == "app-1" + path, body = client.post_calls[0] + assert path == "/applications" + assert body["displayName"] == "support-agent" + assert body["sponsors"] == ["paulo@contoso.com"] + + +def test_register_is_idempotent_when_blueprint_exists() -> None: + client = FakeGraphClient(get_response={"value": [BLUEPRINT_PAYLOAD]}) + blueprint, created = register_blueprint( + "support-agent", sponsor="paulo@contoso.com", client=client + ) + assert created is False + assert blueprint.app_id == "app-1" + assert client.post_calls == [] + + +def test_register_requires_a_sponsor() -> None: + with pytest.raises(AgentIdentityError, match="sponsor"): + register_blueprint("support-agent", sponsor=" ", client=FakeGraphClient()) + + +def test_register_requires_a_display_name() -> None: + with pytest.raises(AgentIdentityError, match="display name"): + register_blueprint("", sponsor="paulo@contoso.com", client=FakeGraphClient()) + + +def test_register_fails_loudly_when_graph_returns_no_app_id() -> None: + client = FakeGraphClient(get_response={"value": []}, post_response={"id": "x"}) + with pytest.raises(AgentIdentityError, match="appId"): + register_blueprint("support-agent", sponsor="p@c.com", client=client) + + +# --------------------------------------------------------------------------- +# Workspace persistence +# --------------------------------------------------------------------------- + + +def test_write_and_read_identity_record(tmp_path: Path) -> None: + blueprint = AgentIdentityBlueprint( + app_id="app-1", object_id="object-1", display_name="support-agent" + ) + path = write_identity_record(tmp_path, blueprint, created=True) + assert path == identity_record_path(tmp_path) + record = read_identity_record(tmp_path) + assert record is not None + assert record["app_id"] == "app-1" + assert record["created"] is True + assert record["version"] == 1 + + +def test_read_identity_record_returns_none_when_absent(tmp_path: Path) -> None: + assert read_identity_record(tmp_path) is None + + +def test_read_identity_record_tolerates_corrupt_json(tmp_path: Path) -> None: + path = identity_record_path(tmp_path) + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text("{not json", encoding="utf-8") + assert read_identity_record(tmp_path) is None + + +def test_resolve_agent_id_prefers_environment_override( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + write_identity_record(tmp_path, AgentIdentityBlueprint(app_id="from-record")) + monkeypatch.setenv(AGENT_ID_ENV, "from-env") + assert resolve_agent_id(tmp_path) == "from-env" + + +def test_resolve_agent_id_ignores_unexpanded_ci_placeholders( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + write_identity_record(tmp_path, AgentIdentityBlueprint(app_id="from-record")) + monkeypatch.setenv(AGENT_ID_ENV, "${{ secrets.AGENT_ID }}") + assert resolve_agent_id(tmp_path) == "from-record" + + +def test_resolve_agent_id_returns_none_before_registration( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.delenv(AGENT_ID_ENV, raising=False) + assert resolve_agent_id(tmp_path) is None + + +# --------------------------------------------------------------------------- +# Configuration resolution +# --------------------------------------------------------------------------- + + +def _write_config(workspace: Path, body: str) -> None: + (workspace / "agentops.yaml").write_text(body, encoding="utf-8") + + +def test_load_identity_config_returns_empty_without_config(tmp_path: Path) -> None: + assert load_identity_config(tmp_path) == {} + + +def test_load_identity_config_reads_identity_block(tmp_path: Path) -> None: + _write_config(tmp_path, "identity:\n sponsor: paulo@contoso.com\n") + assert load_identity_config(tmp_path)["sponsor"] == "paulo@contoso.com" + + +def test_load_identity_config_tolerates_malformed_yaml(tmp_path: Path) -> None: + _write_config(tmp_path, "identity: [unclosed\n") + assert load_identity_config(tmp_path) == {} + + +def test_resolve_display_name_prefers_explicit_override(tmp_path: Path) -> None: + _write_config(tmp_path, "identity:\n display_name: from-config\n") + assert resolve_display_name(tmp_path, override="from-flag") == "from-flag" + + +def test_resolve_display_name_falls_back_to_config(tmp_path: Path) -> None: + _write_config(tmp_path, "identity:\n display_name: from-config\n") + assert resolve_display_name(tmp_path) == "from-config" + + +def test_resolve_display_name_falls_back_to_existing_record(tmp_path: Path) -> None: + write_identity_record( + tmp_path, AgentIdentityBlueprint(app_id="app-1", display_name="from-record") + ) + assert resolve_display_name(tmp_path) == "from-record" + + +def test_resolve_registration_inputs_returns_name_and_sponsor(tmp_path: Path) -> None: + _write_config( + tmp_path, + "identity:\n display_name: support-agent\n sponsor: paulo@contoso.com\n", + ) + name, sponsor = resolve_registration_inputs(tmp_path) + assert name == "support-agent" + assert sponsor == "paulo@contoso.com" + + +def test_resolve_registration_inputs_requires_a_display_name(tmp_path: Path) -> None: + _write_config(tmp_path, "identity:\n sponsor: paulo@contoso.com\n") + with pytest.raises(AgentIdentityError, match="display name"): + resolve_registration_inputs(tmp_path) + + +def test_resolve_registration_inputs_requires_a_sponsor(tmp_path: Path) -> None: + _write_config(tmp_path, "identity:\n display_name: support-agent\n") + with pytest.raises(AgentIdentityError, match="sponsor"): + resolve_registration_inputs(tmp_path) + + +def test_resolve_registration_inputs_accepts_overrides(tmp_path: Path) -> None: + name, sponsor = resolve_registration_inputs( + tmp_path, display_name="flag-agent", sponsor="flag@contoso.com" + ) + assert (name, sponsor) == ("flag-agent", "flag@contoso.com") + + +def test_identity_record_is_valid_json_on_disk(tmp_path: Path) -> None: + write_identity_record(tmp_path, AgentIdentityBlueprint(app_id="app-1")) + raw = identity_record_path(tmp_path).read_text(encoding="utf-8") + assert json.loads(raw)["app_id"] == "app-1" diff --git a/tests/unit/test_agentops_config_identity.py b/tests/unit/test_agentops_config_identity.py new file mode 100644 index 00000000..c3ee2e2e --- /dev/null +++ b/tests/unit/test_agentops_config_identity.py @@ -0,0 +1,66 @@ +"""Regression tests for the optional ``identity`` block in agentops.yaml. + +``AgentOpsConfig`` sets ``extra="forbid"``, so any key the model does not +declare makes every command that calls ``load_agentops_config`` fail. The +Agent 365 identity work reads the block directly from the raw YAML, which +means a missing field declaration would only surface later, in unrelated +commands. These tests pin the contract. +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from agentops.core.config_loader import load_agentops_config + +BASE_CONFIG = "version: 1\nagent: my-rag:3\ndataset: .agentops/data/seed.jsonl\n" + + +def _write(tmp_path: Path, body: str) -> Path: + path = tmp_path / "agentops.yaml" + path.write_text(body, encoding="utf-8") + return path + + +def test_config_without_identity_still_loads(tmp_path: Path) -> None: + config = load_agentops_config(_write(tmp_path, BASE_CONFIG)) + assert config.identity is None + + +def test_identity_block_is_accepted(tmp_path: Path) -> None: + path = _write( + tmp_path, + BASE_CONFIG + + "identity:\n" + + " display_name: support-agent\n" + + " sponsor: owner@contoso.com\n" + + " verify: true\n", + ) + + config = load_agentops_config(path) + + assert config.identity is not None + assert config.identity.display_name == "support-agent" + assert config.identity.sponsor == "owner@contoso.com" + assert config.identity.verify is True + + +def test_identity_verify_defaults_to_false(tmp_path: Path) -> None: + path = _write(tmp_path, BASE_CONFIG + "identity:\n sponsor: owner@contoso.com\n") + + config = load_agentops_config(path) + + assert config.identity is not None + assert config.identity.verify is False + assert config.identity.display_name is None + + +def test_unknown_identity_key_is_rejected(tmp_path: Path) -> None: + path = _write(tmp_path, BASE_CONFIG + "identity:\n sponser: owner@contoso.com\n") + + with pytest.raises(ValueError) as excinfo: + load_agentops_config(path) + + assert "sponser" in str(excinfo.value) diff --git a/tests/unit/test_cli_agent_register.py b/tests/unit/test_cli_agent_register.py new file mode 100644 index 00000000..9b8dd2ad --- /dev/null +++ b/tests/unit/test_cli_agent_register.py @@ -0,0 +1,148 @@ +"""CLI tests for `agentops agent register`. + +The command is exercised through Typer's CliRunner. Microsoft Graph is never +contacted: the happy path patches the service function, and every other path +either fails input resolution or is a dry run. +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest +from typer.testing import CliRunner + +from agentops.cli.app import app +from agentops.services import agent_identity as identity_service +from agentops.services.agent_identity import ( + AgentIdentityBlueprint, + AgentIdentityError, + identity_record_path, + read_identity_record, +) + +runner = CliRunner() + + +def _write_config(workspace: Path, body: str) -> None: + (workspace / "agentops.yaml").write_text(body, encoding="utf-8") + + +def _invoke(workspace: Path, *args: str): + return runner.invoke(app, ["agent", "register", "-w", str(workspace), *args]) + + +def test_register_explain_renders_the_manual() -> None: + result = runner.invoke(app, ["agent", "register", "explain"]) + assert result.exit_code == 0 + assert "register" in result.output.lower() + + +def test_register_fails_without_a_display_name(tmp_path: Path) -> None: + result = _invoke(tmp_path, "--sponsor", "paulo@contoso.com", "--dry-run") + assert result.exit_code == 1 + assert "display name" in result.output.lower() + + +def test_register_fails_without_a_sponsor(tmp_path: Path) -> None: + result = _invoke(tmp_path, "--display-name", "support-agent", "--dry-run") + assert result.exit_code == 1 + assert "sponsor" in result.output.lower() + + +def test_dry_run_reports_inputs_without_writing_a_record(tmp_path: Path) -> None: + result = _invoke( + tmp_path, + "--display-name", + "support-agent", + "--sponsor", + "paulo@contoso.com", + "--dry-run", + ) + assert result.exit_code == 0 + assert "support-agent" in result.output + assert "paulo@contoso.com" in result.output + assert "Dry run" in result.output + assert not identity_record_path(tmp_path).exists() + + +def test_dry_run_resolves_inputs_from_config(tmp_path: Path) -> None: + _write_config( + tmp_path, + "identity:\n display_name: config-agent\n sponsor: owner@contoso.com\n", + ) + result = _invoke(tmp_path, "--dry-run") + assert result.exit_code == 0 + assert "config-agent" in result.output + assert "owner@contoso.com" in result.output + + +def test_register_writes_the_identity_record( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.setattr( + identity_service, + "register_blueprint", + lambda name, *, sponsor: ( + AgentIdentityBlueprint( + app_id="app-1", object_id="object-1", display_name=name + ), + True, + ), + ) + result = _invoke( + tmp_path, + "--display-name", + "support-agent", + "--sponsor", + "paulo@contoso.com", + ) + assert result.exit_code == 0 + assert "Registered" in result.output + assert "app-1" in result.output + record = read_identity_record(tmp_path) + assert record is not None + assert record["app_id"] == "app-1" + assert record["created"] is True + + +def test_register_reports_reuse_when_blueprint_already_exists( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.setattr( + identity_service, + "register_blueprint", + lambda name, *, sponsor: ( + AgentIdentityBlueprint(app_id="app-1", display_name=name), + False, + ), + ) + result = _invoke( + tmp_path, + "--display-name", + "support-agent", + "--sponsor", + "paulo@contoso.com", + ) + assert result.exit_code == 0 + assert "Reused existing" in result.output + assert read_identity_record(tmp_path)["created"] is False + + +def test_graph_failure_is_reported_without_a_stack_trace( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + def _raise(name: str, *, sponsor: str): + raise AgentIdentityError("Admin consent is missing for AgentIdentityBlueprint.Create.") + + monkeypatch.setattr(identity_service, "register_blueprint", _raise) + result = _invoke( + tmp_path, + "--display-name", + "support-agent", + "--sponsor", + "paulo@contoso.com", + ) + assert result.exit_code == 1 + assert "Admin consent is missing" in result.output + assert "Traceback" not in result.output diff --git a/tests/unit/test_evidence_pack_agent_identity.py b/tests/unit/test_evidence_pack_agent_identity.py new file mode 100644 index 00000000..d2815634 --- /dev/null +++ b/tests/unit/test_evidence_pack_agent_identity.py @@ -0,0 +1,111 @@ +"""Tests for the agent identity section of the release evidence bundle.""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from agentops.core.release_evidence import ReleaseEvidence, ReleaseEvidenceCheck +from agentops.services.agent_identity import ( + AGENT_ID_ENV, + AgentIdentityBlueprint, + write_identity_record, +) +from agentops.services.evidence_pack import ( + _add_agent_identity_check, + _agent_identity_status, +) + + +@pytest.fixture(autouse=True) +def _clear_agent_id_env(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.delenv(AGENT_ID_ENV, raising=False) + + +def test_status_reports_not_registered_for_a_fresh_workspace(tmp_path: Path) -> None: + status = _agent_identity_status(tmp_path) + assert status["status"] == "not_registered" + assert status["agent_id"] is None + + +def test_status_reports_registered_from_the_record(tmp_path: Path) -> None: + write_identity_record( + tmp_path, + AgentIdentityBlueprint( + app_id="app-1", object_id="object-1", display_name="support-agent" + ), + ) + status = _agent_identity_status(tmp_path) + assert status["status"] == "registered" + assert status["agent_id"] == "app-1" + assert status["source"] == "record" + assert status["display_name"] == "support-agent" + assert status["object_id"] == "object-1" + assert "record_path" in status + + +def test_status_reports_environment_sourced_identity( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.setenv(AGENT_ID_ENV, "app-from-ci") + status = _agent_identity_status(tmp_path) + assert status["status"] == "registered" + assert status["agent_id"] == "app-from-ci" + assert status["source"] == "environment" + + +def test_unregistered_identity_produces_an_actionable_warning() -> None: + checks: list[ReleaseEvidenceCheck] = [] + warnings: list[str] = [] + ready: list[str] = [] + + _add_agent_identity_check( + checks, warnings, ready, {"status": "not_registered", "agent_id": None} + ) + + assert len(checks) == 1 + assert checks[0].status == "warning" + assert checks[0].name == "Agent identity" + assert ready == [] + assert len(warnings) == 1 + assert "agentops agent register" in warnings[0] + + +def test_registered_identity_produces_a_ready_signal() -> None: + checks: list[ReleaseEvidenceCheck] = [] + warnings: list[str] = [] + ready: list[str] = [] + + _add_agent_identity_check( + checks, warnings, ready, {"status": "registered", "agent_id": "app-1"} + ) + + assert len(checks) == 1 + assert checks[0].status == "ready" + assert warnings == [] + assert len(ready) == 1 + assert "app-1" in ready[0] + + +def test_release_evidence_model_accepts_the_agent_identity_section() -> None: + """The model forbids extra keys, so the field must be declared.""" + + evidence = ReleaseEvidence( + generated_at="2026-01-01T00:00:00+00:00", + workspace="/tmp/ws", + status="ready", + agent_identity={"status": "registered", "agent_id": "app-1"}, + ) + assert evidence.agent_identity["agent_id"] == "app-1" + round_tripped = ReleaseEvidence.model_validate(evidence.model_dump()) + assert round_tripped.agent_identity == evidence.agent_identity + + +def test_release_evidence_defaults_agent_identity_to_an_empty_dict() -> None: + evidence = ReleaseEvidence( + generated_at="2026-01-01T00:00:00+00:00", + workspace="/tmp/ws", + status="ready", + ) + assert evidence.agent_identity == {} diff --git a/tests/unit/test_telemetry_agent_id.py b/tests/unit/test_telemetry_agent_id.py new file mode 100644 index 00000000..e51f3e97 --- /dev/null +++ b/tests/unit/test_telemetry_agent_id.py @@ -0,0 +1,70 @@ +"""Tests for the Entra Agent ID stamped on OpenTelemetry resources.""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from agentops.services.agent_identity import ( + AGENT_ID_ATTRIBUTE, + AGENT_ID_ENV, + AgentIdentityBlueprint, + write_identity_record, +) +from agentops.utils.telemetry import _resource_attributes + + +@pytest.fixture(autouse=True) +def _clear_agent_id_env(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.delenv(AGENT_ID_ENV, raising=False) + + +def test_resource_always_carries_service_identity() -> None: + attributes = _resource_attributes() + assert attributes["service.name"] == "agentops" + assert isinstance(attributes["service.version"], str) + + +def test_agent_id_is_omitted_when_not_registered( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """An unregistered agent must not emit an empty attribute. + + Downstream queries filter on the attribute's presence, so emitting it + blank would make unregistered agents indistinguishable from registered + ones whose id failed to resolve. + """ + + monkeypatch.chdir(tmp_path) + assert AGENT_ID_ATTRIBUTE not in _resource_attributes() + + +def test_agent_id_is_stamped_from_the_workspace_record( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + write_identity_record(tmp_path, AgentIdentityBlueprint(app_id="app-1")) + monkeypatch.chdir(tmp_path) + assert _resource_attributes()[AGENT_ID_ATTRIBUTE] == "app-1" + + +def test_agent_id_is_stamped_from_the_environment( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.chdir(tmp_path) + monkeypatch.setenv(AGENT_ID_ENV, "app-from-ci") + assert _resource_attributes()[AGENT_ID_ATTRIBUTE] == "app-from-ci" + + +def test_identity_failures_never_break_tracing(monkeypatch: pytest.MonkeyPatch) -> None: + """Tracing is mandatory, identity is optional; the former must survive.""" + + import agentops.services.agent_identity as identity_service + + def _boom(*args: object, **kwargs: object) -> None: + raise RuntimeError("identity subsystem exploded") + + monkeypatch.setattr(identity_service, "resolve_agent_id", _boom) + attributes = _resource_attributes() + assert attributes["service.name"] == "agentops" + assert AGENT_ID_ATTRIBUTE not in attributes