docs: build the published site from develop (#408)

.github/workflows/deploy-docs.yml

@@ -1,23 +1,38 @@
 name: Deploy Documentation
 
-# IMPORTANT: this file must exist on the DEFAULT branch, not only on `docs`.
+# The published site is built from `develop`, which is the branch every fix PR
+# lands on. That is deliberate: a documentation-only change reaches the site as
+# soon as its PR is squashed, with no release, tag, or changelog entry needed.
 #
+# `main` still receives `docs/` through the normal release merge, so README,
+# CONTRIBUTING, and the in-product links that point at `docs/` on the default
+# branch keep resolving.
+#
+# IMPORTANT: this file must exist on the DEFAULT branch, not only on `develop`.
 # GitHub Actions resolves workflows for non-`push` events (such as `release`)
-# from the default branch only. While this file lived exclusively on the
-# orphan `docs` branch, the `release` trigger below was inert and never fired,
-# so the published site kept serving stale release metadata until someone ran
-# the workflow by hand.
+# from the default branch only. While this file lived exclusively on the orphan
+# `docs` branch, the `release` trigger below was inert and never fired, so the
+# published site kept serving stale release metadata until someone ran the
+# workflow by hand.
 #
-# Keeping a copy here does NOT cause duplicate runs. For `push` events GitHub
-# uses the workflow file from the pushed commit, so pushing to `docs` still
-# runs the copy on `docs`, and pushing to this branch matches no trigger.
-# Every job below checks out `docs` explicitly, so the site content is always
-# built from that branch regardless of which ref started the run.
+# Keeping a copy on the default branch does NOT cause duplicate runs. For `push`
+# events GitHub uses the workflow file from the pushed commit, and the `push`
+# trigger below only matches `develop`. Every job checks out `develop`
+# explicitly, so the site content is always built from that branch regardless of
+# which ref started the run.
 
 on:
   push:
     branches:
-      - docs
+      - develop
+    # Only rebuild when something that affects the rendered site changes.
+    paths:
+      - 'docs/**'
+      - 'mkdocs.yml'
+      - 'main.py'
+      - 'requirements-docs.txt'
+      - 'overrides/**'
+      - '.github/workflows/deploy-docs.yml'
 
   # Rebuild when a release or pre-release is published, since the home page
   # shows release metadata baked at build time.
@@ -30,6 +45,12 @@ on:
   # Allow triggering a rebuild manually from the Actions UI.
   workflow_dispatch:
 
+# Let a newer commit supersede an in-flight build instead of racing it to the
+# gh-pages branch.
+concurrency:
+  group: deploy-docs
+  cancel-in-progress: true
+
 permissions:
   contents: write
 
@@ -42,7 +63,7 @@ jobs:
         uses: actions/checkout@v4
         with:
           fetch-depth: 0
-          ref: docs
+          ref: develop
 
       - name: Set up Python
         uses: actions/setup-python@v5
@@ -52,6 +73,9 @@ jobs:
       - name: Install MkDocs and dependencies
         run: pip install -r requirements-docs.txt
 
+      - name: Build the site
+        run: mkdocs build --strict
+
       - name: Deploy to GitHub Pages
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

.gitignore

@@ -464,3 +464,6 @@ e2e-runs/
 tmp-artifacts/
 
 tmp/
+
+# MkDocs local build output
+site/

README.md

@@ -155,7 +155,8 @@ Cockpit sections, in display order:
 ## Documentation
 
 - [Foundry Prompt Agent tutorial](docs/tutorial-prompt-agent.md) - use this when the Foundry target is `agent: name:version`. Walks the sandbox to dev journey with a PR gate.
-- [Hosted or HTTP Agent tutorial](docs/tutorial-hosted-agent-quickstart.md) - use this when the target is a Foundry hosted or HTTP endpoint URL. Same sandbox to dev journey for endpoint-based agents.
+- [Hosted Agent tutorial](docs/tutorial-hosted-agent.md) - use this when Foundry runs your agent code as a managed hosted runtime behind a stable endpoint.
+- [HTTP Agent tutorial](docs/tutorial-http-agent.md) - use this when the target is an HTTP service you operate behind your own URL.
 - [End-to-end tutorial](docs/tutorial-end-to-end.md) - extends either of the above with the full sandbox to dev to qa to prod promotion, Foundry red-team scans, and trace-to-regression promotion.
 - [Evaluation paths](docs/evaluation.md) - choose static dataset, grey-box HTTP, or telemetry/trace import.
 - [Core concepts](docs/concepts.md)

docs/ci-github-actions.md

@@ -1,3 +1,7 @@
+---
+render_macros: false
+---
+
 # AgentOps GenAIOps GitFlow on GitHub Actions
 
 This guide shows how to wire AgentOps into a complete GenAIOps CI/CD
@@ -18,38 +22,55 @@ workflow is available separately when you explicitly generate `--kinds doctor`.
 
 | File | Trigger | GitHub Environment | Purpose |
 |---|---|---|---|
-| `agentops-pr.yml` | PRs to `develop`, `release/**`, `main` | `dev` | Eval gate + Doctor gate (default blocks on critical findings; configurable via `--doctor-gate`) + PR comment |
+| `agentops-pr.yml` | PRs to `develop`, `release/**` | `sandbox` for prompt-agent PR candidates, `dev` for generic PR gates | Eval PR candidate + Doctor gate (default blocks on critical findings; configurable via `--doctor-gate`) + PR comment |
 | `agentops-deploy-dev.yml` | push to `develop` | `dev` | Eval → build → deploy DEV |
 | `agentops-deploy-qa.yml` | push to `release/**` | `qa` | Eval → build → deploy QA |
-| `agentops-deploy-prod.yml` | push to `main` | `production` | Safety eval → evidence → build → deploy PROD |
+| `agentops-deploy-prod.yml` | push to `main` | `production` | Deploy PROD → smoke test |
 | `agentops-doctor.yml` | daily cron | `dev` | Optional scheduled Doctor + release evidence |
 
 ## GitFlow assumed
 
 ```mermaid
 flowchart LR
-    feat["feature/*"] -->|PR| prGate1{{"agentops-pr.yml<br/>(gate)"}}
-    prGate1 -->|merge| dev["develop"]
-    dev --> deployDev["agentops-deploy-dev.yml"]
-    deployDev --> DEV(["DEV"])
-
-    rel["release/*"] -->|push| deployQa["agentops-deploy-qa.yml"]
-    deployQa --> QA(["QA"])
-
-    rel -->|PR| prGate2{{"agentops-pr.yml<br/>(gate)"}}
-    prGate2 -->|merge| main["main"]
-    main --> deployProd["agentops-deploy-prod.yml"]
-    deployProd --> PROD(["PROD<br/>(required reviewers)"])
-
-    classDef gate fill:#fff3cd,stroke:#856404,color:#000;
+    feature["feature/*"] --> prDev["PR eval<br/>candidate"]
+    prDev --> sandbox["sandbox"]
+    prDev --> develop["develop"]
+    develop --> devDeploy["Eval + deploy<br/>agentops-deploy-dev"]
+    devDeploy --> devEnv["dev"]
+
+    develop --> release["release/*"]
+    release --> qaDeploy["Eval + deploy<br/>agentops-deploy-qa"]
+    qaDeploy --> qaEnv["qa"]
+
+    release --> prProd["PR: release to main<br/>manual approval"]
+    prProd --> main["main"]
+    main --> prodDeploy["Prod release process<br/>deploy + smoke test<br/>agentops-deploy-prod"]
+    prodDeploy --> prodEnv["production"]
+
+    classDef branch fill:#e7f0fd,stroke:#1f4e79,color:#000;
+    classDef pipeline fill:#ede7f6,stroke:#4527a0,color:#000;
     classDef env fill:#d1ecf1,stroke:#0c5460,color:#000;
-    class prGate1,prGate2 gate;
-    class DEV,QA,PROD env;
+    class feature,develop,release,main branch;
+    class prDev,devDeploy,qaDeploy,prProd,prodDeploy pipeline;
+    class sandbox,devEnv,qaEnv,prodEnv env;
 ```
 
+Legend:
+<span style="display:inline-block;width:0.9em;height:0.9em;background:#e7f0fd;border:1px solid #1f4e79;vertical-align:-0.1em;"></span> Git branch
+<span style="display:inline-block;width:0.9em;height:0.9em;background:#ede7f6;border:1px solid #4527a0;vertical-align:-0.1em;"></span> PR or workflow gate
+<span style="display:inline-block;width:0.9em;height:0.9em;background:#d1ecf1;border:1px solid #0c5460;vertical-align:-0.1em;"></span> deployed environment
+
 If you are on trunk-based development, generate only the templates you
 need: `agentops workflow generate --kinds pr,dev,prod`.
 
+The PR gate validates candidates before they enter `develop` or `release/**`. It
+is not a dev deployment. HTTP agent tutorials point that candidate at the
+sandbox endpoint; prompt-agent workflows stage and evaluate the candidate prompt
+version in sandbox. The PR from `release/**` to `main` is a manual approval gate
+with static checks only. It does not call agents. After it merges,
+`agentops-deploy-prod` runs the production release process: deploy and smoke
+test.
+
 ## Quick start
 
 ```bash
@@ -253,7 +274,8 @@ az ad app federated-credential list --id "$APP_ID" \
 | Error | Cause | Fix |
 |---|---|---|
 | `AADSTS700213: No matching federated identity record found for presented assertion subject` | The credential `subject` is not byte-identical to the subject GitHub sent. Usually the immutable-ID prefix above. Also caused by the wrong environment name, or a `ref:refs/heads/...` subject on a job that uses `environment:`. | Copy the subject quoted in the error, compare it against `az ad app federated-credential list`, and add the missing credential. |
-| `AADSTS53003: Access has been blocked by Conditional Access policies` | Usually `AZURE_TENANT_ID` points at a tenant that cannot see the app registration, not an actual CA policy. | Set `AZURE_TENANT_ID` to the tenant that owns the app registration and the federated credential, not a subscription `managedByTenants` entry. |
+| `AADSTS53003: Access has been blocked by Conditional Access policies` | A Conditional Access policy blocked the token. Workload identities are in scope of CA, so a policy that requires MFA, a compliant device, or a named location will block a GitHub-hosted runner. | Open the sign-in in Entra ID > Sign-in logs > Service principal sign-ins, read the Conditional Access tab to find the policy that applied, then exclude the workload identity or scope the policy so it does not target it. |
+| `AADSTS700016: Application with identifier '<id>' was not found in the directory` | `AZURE_CLIENT_ID` or `AZURE_TENANT_ID` is wrong. The app registration exists in a different tenant than the one being authenticated against. | Set `AZURE_TENANT_ID` to the tenant that owns the app registration and the federated credential, not a subscription `managedByTenants` entry. Confirm with `az ad app show --id "$AZURE_CLIENT_ID" --query appId`. |
 | `AuthorizationFailed` on `azd provision` | The principal has no role at the scope the ARM deployment targets. | Check the template's target scope, then assign at that scope. See below. |
 
 `azd` templates commonly declare `targetScope = 'subscription'` in
@@ -288,6 +310,45 @@ Resource-group scope is enough only when the template is
 from the Foundry roles below, which stay scoped to the Foundry project and the
 AI Services account.
 
+#### `azd` in CI
+
+`azure/login@v3` authenticates the Azure CLI. It does not authenticate `azd`,
+which keeps a separate credential store and never falls back to the `az`
+session. CI jobs that call `azd` need three extra things, all of which the
+generated workflows already do:
+
+1. **Install the `azure.ai.agents` extension explicitly and pin it.** azd
+   refuses to auto-install extensions on CI runners. AgentOps pins
+   `1.0.0-beta.9`, the same version the eval gate uses, and reads an override
+   from `AGENTOPS_AZD_AI_AGENTS_EXTENSION_VERSION`.
+
+   ```bash
+   azd extension install azure.ai.agents --version "1.0.0-beta.9"
+   ```
+
+2. **Log azd in on GitHub Actions**, using the same federated credential as the
+   `az` login:
+
+   ```bash
+   azd auth login \
+     --client-id "$AZURE_CLIENT_ID" \
+     --tenant-id "$AZURE_TENANT_ID" \
+     --federated-credential-provider github
+   ```
+
+3. **On Azure DevOps, reuse the `az` session instead.** The steps run inside an
+   `AzureCLI@2` inline script, so the service connection has already produced an
+   authenticated CLI session that azd can borrow:
+
+   ```bash
+   azd config set auth.useAzCliAuth "true"
+   ```
+
+Skipping step 1 surfaces as `ERROR: no extensions found` or
+`Auto-installation is not supported in CI/CD environments`. Skipping step 2 or 3
+surfaces as an azd authentication failure in a job where `az` commands work
+fine.
+
 For Foundry prompt-agent gates, the same app registration / service principal
 needs **two** Azure RBAC roles before the first workflow run. Both are required
 and the eval step fails silently (every metric returns `null`) if only one is
@@ -330,9 +391,6 @@ In Settings → Environments, create three:
 - Override env-specific variables for QA infra.
 
 #### `production`
-- **Required reviewers**: at least one. Deploys to PROD pause until
-  approved.
-- Optional: **Wait timer** for an extra cool-down.
 - Optional: **Deployment branches**: restrict to `main`.
 - Override env-specific variables for production infra.
 
@@ -359,7 +417,7 @@ prompt.
 ### 4. Choose deployment mode
 
 AgentOps is azd-first for deployment: AgentOps runs the evaluation gate,
-while Azure Developer CLI manages infrastructure, packaging, deployment, and
+while Azure Developer CLI owns infrastructure, packaging, deployment, and
 hooks declared in `azure.yaml`.
 
 Before choosing manually, run:
@@ -529,11 +587,11 @@ agentops workflow analyze --format markdown --out agentops-workflow-plan.md
 
 Use the output as the plan for your coding agent:
 
-1. AgentOps handles repo-side eval gates, Doctor readiness checks, artifacts, and
+1. AgentOps owns repo-side eval gates, Doctor readiness checks, artifacts, and
    Cockpit visibility.
-2. `azd` manages `provision`, `deploy`, and hooks for app/infra lifecycle when
+2. `azd` owns `provision`, `deploy`, and hooks for app/infra lifecycle when
    `azure.yaml` is present or can be added.
-3. Foundry manages hosted agents, evaluations, traces, and operations.
+3. Foundry owns hosted agents, evaluations, traces, and operations.
 4. Project-specific steps such as indexing data, seeding search, building
    containers, updating app config, or running private-network post-provision
    work stay in the accelerator's azd hooks or existing deployment tooling.
@@ -583,8 +641,8 @@ contract to gate deploys:
 | `2` | Eval ran, one or more thresholds failed | ❌ fail (deploy never runs) |
 | `1` | Runtime / config error | ❌ fail |
 
-For prompt-agent cloud eval, Foundry runs the managed evaluation and
-AgentOps enforces the CI exit code. A threshold failure exits `2`, so the PR/deploy
+For prompt-agent cloud eval, Foundry owns the managed evaluation run and
+AgentOps owns the CI exit code. A threshold failure exits `2`, so the PR/deploy
 gate fails with the failing threshold rows in `report.md`.
 
 ## Artifacts

docs/concepts.md

@@ -27,6 +27,35 @@ regression data.
 The short version is: **Foundry operates the agent; AgentOps turns that operating
 signal into repo-side release proof.**
 
+## What AgentOps produces
+
+AgentOps Accelerator turns every readiness run into outputs that people and CI
+can both use.
+
+| Output | Use it for |
+|---|---|
+| `report.md` | PR review: what passed, what failed, and what changed from the baseline. |
+| `results.json` | CI automation: stable metrics, thresholds, exit status, and target details. |
+| `evidence.md` / `evidence.json` | Release approval: the proof package for the ship/no-ship decision. |
+| Cockpit | Local review: latest evals, Doctor findings, evidence, and next actions in one view. |
+
+## Reference architecture
+
+Use this as the mental model for the AgentOps loop: build and learn in a
+sandbox, commit the release contract to source control, promote through
+environments with evidence, then feed production learning back into the next
+evaluation set.
+
+![AgentOps Accelerator reference architecture](media/agentops-architecture.png){ .agentops-reference-architecture }
+
+| Area | What it owns |
+|---|---|
+| **Sandbox inner loop** | Create, evaluate, and improve the candidate agent in a safe Foundry project before it is promoted. |
+| **AgentOps Accelerator** | Keep release readiness close to the repo: config, datasets, evaluation gates, Doctor diagnostics, Cockpit views, CI workflows, thresholds, and release evidence. |
+| **Foundry** | Hosts managed agent projects, Prompt Agent and Hosted Agent runtime options, traces, operate views, guardrails, and evaluations where applicable. |
+| **Outer loop delivery** | Move the same reviewed candidate through dev, QA or staging, and production. Production release should be gated by reviewable evidence, not memory or a manual spot check. |
+| **Operate and improve** | Watch telemetry, dashboards, alerts, cost, success rate, compliance, quota, security posture, and data governance. Turn production traces into the next regression cases. |
+
 ## How an Evaluation Works
 
 ```mermaid
@@ -96,10 +125,8 @@ Common `agent:` values:
 | `"model:gpt-4o-mini"` | Direct model deployment |
 
 HTTP targets can add top-level mapping fields such as `request_field`,
-`response_fields`, `tool_calls_field`, `auth_header_env`, and `extra_fields`.
-Use `response_fields.response` for the final answer and
-`response_fields.context` for retrieved context. Use `response_source: dataset`
-when each dataset row already contains the response to evaluate.
+`response_field`, `tool_calls_field`, `auth_header_env`, and
+`extra_fields`.
 
 ### Dataset
 
@@ -157,7 +184,7 @@ evidence outputs into a release gate.
 | Target | Foundry server-side eval through AgentOps | AgentOps local runner | Recommended default |
 |---|---|---|---|
 | Foundry Prompt Agent (`name:version`) | Yes, with `execution: cloud` | Yes | Use cloud for official Foundry-hosted runs; use local for fast feedback or fallback. |
-| Foundry Hosted Agent URL | No | Yes | Use local runner; optionally publish local metrics to Foundry with `publish: true`. |
+| Foundry Hosted Agent URL | Yes, with `execution: cloud`, when the URL contains `/agents/<name>/versions/<version>` | Yes | Use cloud when the endpoint carries the versioned agent path; otherwise use the local runner and optionally `publish: true`. |
 | Generic HTTP/JSON endpoint | No | Yes | Use local runner. |
 | Raw model deployment (`model:<name>`) | No | Yes | Use local runner. |
 
@@ -198,17 +225,12 @@ AgentOps auto-selects common evaluation patterns from the dataset:
 | **Agent workflow** | `tool_calls` + `tool_definitions` | Tool-use quality |
 | **Content safety** | Safety evaluators | Responsible AI checks |
 
-Use one of the three hands-on tutorials for scenario coverage:
+Use the hands-on tutorials for scenario coverage:
 
-- [Evaluation paths](evaluation.md) explains when to use a static dataset,
-  grey-box HTTP response mapping, or telemetry/trace import.
-- [Foundry Prompt Agent tutorial](tutorial-prompt-agent-quickstart.md) for Foundry
+- [Prompt agent tutorial](tutorial-prompt-agent.md) for Foundry
   prompt agents referenced as `name:version`.
-- [Hosted or HTTP Agent tutorial](tutorial-hosted-agent-quickstart.md) for Foundry
-  hosted endpoints, generic HTTP agents, RAG services, and code-based workflows.
-- [End-to-end tutorial](tutorial-end-to-end.md) for the complete Foundry +
-  AgentOps loop, including CI/CD, observability, red-team follow-through,
-  Doctor, release evidence, and trace regression.
+- [HTTP agent tutorial](tutorial-http-agent.md) for HTTP agents,
+  RAG services, and code-based workflows behind a JSON endpoint.
 
 ## Configuration Model
 
@@ -219,13 +241,9 @@ the fields your target needs:
 version: 1
 agent: "https://api.example.com/chat"
 dataset: .agentops/data/support.jsonl
-response_source: agent
 
-protocol: http-json
 request_field: message
-response_fields:
-  response: text
-  context: retrieved_context
+response_field: text
 
 thresholds:
   coherence: ">=3"