netops-agent

An AI agent that triages ISP/network incidents by reasoning over a configurable fleet of MCP servers and producing evidence-cited diagnoses. The fleet is config-driven ([[mcp.server]] entries; see docs/mcp-fleet.md) — mcp-client is a generic spawn-based registry with no hardcoded MCP list. The fleet doc is the canonical roster, with per-MCP status, transport, safety posture, and the incident-class coverage matrix.

Status

Pre-alpha. Workspace scaffold. 9 fleet MCPs total: 8 product MCPs (4 ready — nautobot, openwifi, andrena, unifi; 4 stub/bootstrap — aaa, mikrotik, junos, kubectl) plus 1 reference scaffold (template-mcp, not spawned at runtime). 2 MCPs were dropped (csp-mcp folded into andrena-mcp; kb-mcp folded into the rag crate) — ADRs preserved, see docs/mcp-fleet.md for per-MCP status, tool counts, and the canonical roster. Priority ranks are speculative until eval/incidents/ is populated with real ticket data.

The runtime source-of-truth for what the agent actually spawns is apps/agent/config/mcp_presets.toml (currently lists all 8 product MCPs as --enable <id> presets).

Layout

crates/
  agent-core/      ReAct loop, evidence chains, iteration budget
  agent-build/     Single-source-of-truth `Agent` builder shared by both binaries
  llm/             Anthropic client, prompt caching, tool-use schema, CanonicalUsage
  mcp-client/      rmcp client multiplexer + tool registry
  guardrails/      Action allowlist, approval flow, loop detection, dry-run sink
  security/        Input/output validation, instruction hierarchy, PII redaction
  rag/             Knowledge corpus loader (Stage 1: long-context)
  context/         Cost-aware planner hint pipeline, topology cache, summarizer
  eval/            Offline eval runner, LLM-as-judge, rubric scoring
  observability/   OTel GenAI tracing, span structure, audit + run-record streams
  cost/            Token accounting and cost rollups (consumes netops_llm::CanonicalUsage)
  types/           Shared domain types (Claim, Evidence, Citation, ToolCallId, IterationBudget)
  config/          TOML + env-var configuration loader
apps/
  agent/           `netops-agent diagnose "..."` CLI
  eval-runner/     `netops-eval run --suite eval/incidents/` CLI
prompts/           Versioned system prompts, few-shot examples, judge rubric
eval/              Ground-truth incident set + retrieval test set
docs/              Architecture, threat model, cost model

Planned crates

No additional crates are scaffolded ahead of need. New crates land only when an ADR pins the architectural commit — see docs/adr/0002-react-not-planner.md for the precedent that pre-empted a speculative crates/planner. Future scaffolds (keyed to milestones in docs/architecture.md and survey-driven issues) will be listed here as they are accepted.

Design docs

docs/architecture.md — component map and dataflow.
docs/mcp-fleet.md — canonical MCP roster, per-MCP posture, incident-class coverage, build-order priority.
docs/threat-model.md — adversary models, trust boundaries, sanitize-seam defense-in-depth, tool-policy gates. Design input for crates/security and every MCP author's redaction layer.
docs/cost-model.md — token budget per query, per-MCP cost contribution, caching tiers, operator knobs, monthly cost projection. Design input for crates/cost and crates/llm caching.

Installing `netops-agent`

Operators use the packaged binary; developers use cargo. See docs/installing.md for the four install channels (Homebrew tap, GH release tarball, Docker, cargo install) and docs/quickstart.md for the first-query walk-through. Issue #225 is the packaging-foundation tracker.

Quick start

cargo check --workspace
cargo test  --workspace

Sync the KB snapshot

The agent's RAG layer reads from a snapshot of the canonical andrena-io/devops-kb repo, not from a live editor / Notion / a single laptop. To populate the snapshot on a fresh checkout:

eval "$(scripts/sync-devops-kb.sh)"        # clones into ./.cache/devops-kb
netops-agent diagnose "..."                # picks up NETOPS_RAG_ROOT from the eval

The script clones the repo on first run and fast-forward-pulls thereafter, honoring a KB_DEST override (default ./.cache/devops-kb). It prints both NETOPS_RAG_ROOT=<path> and NETOPS_DEVOPS_KB=<path> on stdout so the calling shell can eval it directly — the first powers the agent at runtime (see docs/architecture.md), the second gates the crates/rag/tests/devops_kb_smoke.rs loader regression test.

Snapshot vs live. The default snapshot is deterministic — the agent sees exactly what was at origin/main the last time you ran the script. If you want the agent to follow live edits (e.g. while authoring a runbook), point KB_DEST at a working clone you're editing:

KB_DEST=~/dev/andrena-io/devops-kb eval "$(scripts/sync-devops-kb.sh)"

The devops-kb-smoke CI job (see .github/workflows/devops-kb-smoke.yml) runs the loader against the same upstream on every push to main and nightly, so loader regressions surface against the real corpus shape — not just synthetic fixtures. Fork PRs skip (the upstream is private).

Backfill frontmatter on operational runbooks

If you add a new folder of .md runbooks to andrena-io/devops-kb and they lack the canonical bold-list frontmatter block (**Category**, **Tags**, **Severity**, **Visibility**, **Last Updated**), the agent's RAG loader treats them as un-tagged blobs and the devops-kb-smoke test in CI will fail (the post-#187 floor is corpus.len() / 2 AND >= 200 docs with canonical frontmatter). To backfill defaults across a checkout:

# Walk + report; exit non-zero if any files need backfill.
python3 scripts/seed-frontmatter.py --check --root ~/dev/andrena-io/devops-kb

# Rewrite missing files in place (idempotent).
python3 scripts/seed-frontmatter.py --apply --root ~/dev/andrena-io/devops-kb

# Preview without writing.
python3 scripts/seed-frontmatter.py --dry-run --root ~/dev/andrena-io/devops-kb

The script picks Category from the folder, escalates Severity to High when the filename contains outage / down / fail* / crash / emergency / postmortem, defaults Visibility to public, and pulls Last Updated from the file's git-tracked commit date when available. It intentionally leaves **Tags**: blank — that's human work (Pass-2). Once Pass-2 lands, opt into the > 0.7 * corpus.len() tag-coverage assertion by setting NETOPS_DEVOPS_KB_TAG_THRESHOLD=0.7 in the smoke job env.

See docs/kb-conventions.md in andrena-io/devops-kb for the authoring contract this enforces.

Eval CI integration

Two CI workflows surface eval signal on every PR and every night, closing the loop on the eval-driven-development discipline documented in docs/eval-driven-development.md:

eval-anchor (issues #92, #103, #445) — runs on every PR that touches crates/agent-core, crates/llm, crates/mcp-client, crates/rag, crates/context, crates/guardrails, crates/security, crates/observability, crates/eval, or prompts/. Replays the 16 anchor incidents in eval/anchors.toml via netops-eval anchor. The gate is variance-aware (issue #445): an anchor passes when any of up to --max-attempts samples (default 3) meets its unchanged expected_score_floor; only failing anchors are retried (under retry-N/ in the output root), and an anchor that scores below its floor on every attempt fails the workflow. Every attempt persists; gate-attempts.json and anchor-gate-summary.txt label the accepted aggregate per anchor. Cost envelope: ~$4/PR baseline, plus only the retried anchors (bounded at 3 samples per anchor worst-case). --max-attempts 1 reproduces the pre-#445 one-shot gate for diagnostics.
eval-nightly (issue #103) — runs at 03:00 UTC via cron and on-demand via workflow_dispatch. Replays the full eval/incidents/INC-*.yaml corpus via netops-eval run, uploads scores.csv as a build artifact (30-day retention), and on regression opens (or comments on the open) eval-regression-labeled GitHub issue with a link to the run.

Both workflows skip cleanly on forks / unconfigured repos when ANTHROPIC_API_KEY is absent; a validate-* job still gates the schema so a malformed eval/anchors.toml or eval/incidents/INC-*.yaml cannot slip through.

Local-dev parity

Replay the per-PR gate before pushing — same code path as CI, just sourcing the Anthropic key from ~/.config/netops-agent/secrets.toml or $ANTHROPIC_API_KEY:

# Per-PR proxy: 16 anchor incidents, ~$4 baseline (failing anchors are
# retried up to --max-attempts, default 3; pass --max-attempts 1 for the
# one-shot diagnostic mode).
cargo run -p netops-eval -- anchor \
  --anchors eval/anchors.toml --suite eval/incidents --output target/eval-anchor

# OAuth-backed local parity when no API key is installed. This still drives
# the normal Agent + Judge pipeline, but CLI wrapper token/cache accounting is
# not wire-equivalent to the Messages API.
cargo run -p netops-eval -- anchor \
  --backend claude-cli \
  --anchors eval/anchors.toml --suite eval/incidents --output target/eval-anchor

# Full nightly suite locally (mirrors `eval-nightly`).
cargo run -p netops-eval -- run \
  --suite eval/incidents --out target/eval-nightly

Override path: `eval-exempt`

When a PR deliberately retunes a prompt or rewrites a code path that the current anchor floors can't accommodate (e.g. a planned regression-then-fix pair), a maintainer can add the eval-exempt label with a rationale in a PR comment. This is the documented escape hatch — there is no other bypass. Branch protection on main treats eval-anchor as a required check unless the label is present.

Installing fleet MCPs — readonly by default

netops-agent spawns each MCP server as a subprocess via the binary name declared in apps/agent/config/mcp_presets.toml. The target convention is two binaries per release (a readonly default and a separately named admin build):

Binary on `$PATH`	Built with	Surface
`<mcp>-mcp` (default)	no extra features	Read-only tools only. The agent's presets invoke this name.
`<mcp>-mcp-admin` (opt-in)	`--features admin`	Adds write/destructive tools (e.g. `provision_site`, `nautobot_update`).

Release enforcement of that convention is mid-rollout and tracked in #400. Current status for the bundled fleet presets is:

Preset	Binary	Release-workflow status
`andrena`	`andrena-mcp`	Partial: `release.yml` exists, but builds the readonly binary only.
`nautobot`	`nautobot-mcp`	No release workflow; only `ci.yml` is present.
`unifi`	`unifi-mcp`	No release workflow; only `ci.yml` is present.
`openwifi`	`openwifi-mcp`	No release workflow; only `ci.yml` is present.
`aaa`	`aaa-mcp`	No CI or release workflow is present.
`kubectl`	`kubectl-mcp`	No release workflow; only `ci.yml` is present.
`mikrotik`	`mikrotik-mcp`	No release workflow; only `ci.yml` is present.
`junos`	`junos-mcp`	No release workflow; only `ci.yml` is present.

zabbix is a documented deviation: the preset runs uvx around a pinned third-party server rather than a Rust <mcp>-mcp binary. The reference template-mcp scaffold has the two-binary release workflow, but the fleet rollout is not complete.

Install with --locked. It is required, not stylistic: an unlocked install re-resolves dependencies instead of honoring the committed Cargo.lock, which can carry an rmcp/rmcp-macros mismatch onto an operator laptop. See docs/mcp-fleet.md for the fleet dependency policy.

netops-agent operators do have protections in force today:

Spawn-time env forcing. For every preset whose MCP supports a <SERVER>_READONLY env switch, the runner forces it to true at spawn (apps/agent/src/presets.rs). Even if an operator installed the -admin binary, the binary still runs in readonly mode unless that operator explicitly overrides the env via a [[mcp.server]] block in config.toml.
CLI write-gate. The bundle selection flag is named --enable-all-readonly. Writes require the separate --unsafe-allow-writes flag, which is TTY-gated, prints a banner, and waits 3 seconds before proceeding. The legacy --enable-all is kept as a deprecated alias and never gates writes.

The CI readonly-contract job (see .github/workflows/ci.yml) runs the integration tests that verify both of those layers continue to hold on every PR.

Observability — `terminated_by` SLI

Every diagnose() call brackets its run with two Session RunRecord events (start + end). The session-end record carries a terminated_by field — the snake_case wire encoding of the ReAct loop's exit reason: natural_stop, budget_exhausted, cap_exceeded, max_tool_calls_exceeded, max_input_tokens_exceeded, or error (issue #114, error variant added in issue #289). The error variant fires when the loop short-circuits on an upstream failure (e.g. LlmError); the matching error field on the session-end record carries the stringified upstream error.

When RunRecords are appended to a JSONL ledger (via FileJsonlRunSink), the breakdown of session-end labels and the budget-exhausted rate both fall out of jq one-liners.

Label breakdown:

jq -r 'select(.span_kind=="session") | .terminated_by // empty' runs.jsonl \
  | sort | uniq -c

Budget-exhausted rate (the SLI defined in issue #114; counts budget_exhausted, max_tool_calls_exceeded, and max_input_tokens_exceeded together — every exhaustion class, but not cap_exceeded which is the per-query USD kill-switch, and not error which is a separate SRE signal class for failed sessions):

jq -r 'select(.span_kind=="session") | .terminated_by // empty' runs.jsonl \
  | awk 'BEGIN{n=0; e=0}
         {n++; if ($1=="budget_exhausted" || $1=="max_tool_calls_exceeded" || $1=="max_input_tokens_exceeded") e++}
         END{if(n) printf "budget_exhaustion_rate=%.3f (%d/%d)\n", e/n, e, n}'

A healthy fleet expects this rate well under 10 % of all session-end records. A sustained crossing of that threshold means the agent is hitting IterationBudget (or the call/token caps) before it can resolve the ticket — raise the budget, fix the prompt that's wasting turns, or split the ticket. The session-start record omits terminated_by entirely (Option<String> with skip_serializing_if = "Option::is_none"), so the select(.span_kind=="session") filter combined with the // empty default cleanly drops bookend starts without false positives.

The same threshold is enforced in-process: build_run_emitter wraps every sink in BudgetExhaustionSli<dyn RunEmitter> (default window 50, threshold 10 %), and a tracing::warn!(target = "guardrail.budget_exhaustion_sli") fires on the upward crossing — no metrics infra required. See docs/cost-model.md for the daily/weekly operator runbook.

Observability — tool I/O byte attribution (issue #108)

Every ToolCall record carries tool_input_bytes (the JSON-serialised args dispatched to the MCP tool) and tool_output_bytes (the raw response from the MCP server, measured BEFORE any truncation or redaction pass). A combined tool_io_bytes field stays present for back-compat. The surrounding Iteration record aggregates iteration_tool_io_bytes across every dispatch in that turn.

This is the "why did this query cost more than expected?" diagnostic — the single biggest predictor of an iteration's input-token bill is the total bytes the tools returned the prior turn. Rank tools by total bytes returned with one jq:

jq -r 'select(.span_kind=="tool_call") | [.tool_name, .tool_output_bytes] | @tsv' runs.jsonl \
  | sort | uniq -c | sort -rn

A tool whose tool_output_bytes dominates the ranking is a candidate for response trimming, pagination, or a narrower default filter at the MCP layer.

Agent-side trimming is available now (issue #385). Pass --max-tool-output-bytes <BYTES> to diagnose to project any single oversized successful tool response down before it reaches the model:

netops-agent diagnose --enable unifi --max-tool-output-bytes 8000 \
  "which APs at site X are disconnected"

The elision is announced in-band — the projected payload carries a _netops_trimmed envelope naming the dropped fields and telling the planner to use a targeted per-item lookup instead of citing absent data — and tool_output_bytes keeps reporting the raw pre-trim count, so the ranking above stays honest. Each trim also emits one context.tool_output_trimmed warn event carrying the tool name, both byte counts, and how many array elements were dropped. The flag is off by default.

Observability — per-class cost SLI (`incident_class`)

When the operator (or the eval harness) supplies an --incident-class hint, every RunRecord carries the kebab-case wire form on incident_class (the canonical-15 playbook slugs — slow-wifi, ap-unreachable, whole-site-outage, billing-disconnect, ap-firmware-stale, mpsk-rotation, controller-failure, ipam-vlan-conflict, site-provisioning, ap-swap, bandwidth-validation, tech-dispatch, cert-rotation, alert-tuning, config-drift — plus other). Prod runs that omit the flag emit records without the field, so existing JSONL consumers see no change.

The eval-runner stamps every record automatically from the INC-*.yaml fixture's incident_class, so per-class p50/p95 cost dashboards fall out of one jq pipeline over runs.jsonl:

jq -r 'select(.span_kind=="session" and .incident_class == "slow-wifi") | .cost_usd' runs.jsonl \
  | datamash p50 1 p95 1

Swap the literal for ap-unreachable, mpsk-rotation, whole-site-outage, billing-disconnect, or any other canonical class slug to fence the roll-up by class. A budget overshoot on one class without a matching shift on the others points at a class-specific routing bug — the opposite signal of terminated_by (see previous section) which catches loop-shape problems regardless of incident type. Issue #106.

Observability — per-tool latency p50/p95 (issue #109)

Every ToolCall record carries latency_ms — the wall-clock time the tool dispatch took, measured via tokio::time::Instant deltas at the agent's dispatch_one seam (so it covers the MCP subprocess round-trip end-to-end). Pair it with mcp_server (the qualified-name prefix the agent splits out) to get per-MCP latency distributions with one jq + datamash one-liner:

jq -r 'select(.span_kind=="tool_call") | [.mcp_server, .latency_ms] | @tsv' runs.jsonl \
  | datamash -s -g 1 perc:50 2 perc:95 2 \
  | awk 'BEGIN{printf "%-20s %8s %8s\n", "mcp_server", "p50_ms", "p95_ms"}
         {printf "%-20s %8.0f %8.0f\n", $1, $2, $3}'

A junos server whose p95 climbs past 2000 ms is a NETCONF SSH session issue; an openwifi p95 climb usually traces back to a slow upstream subservice. The signal lands in the runs ledger before any human notices the agent feels sluggish.

Observability — OTel sink (opt-in)

The JSONL stream above is the prod-grade observability path. Operators on Grafana Tempo, Honeycomb, Jaeger, or Datadog APM can additionally export the same stream as OpenTelemetry spans by enabling the otel feature on netops-observability and configuring an OTLP endpoint:

[observability.otlp]
endpoint = "http://otel-collector.observability:4318"
service_name = "netops-agent"

cargo build --release -p netops-agent --features otel

Spans are mapped onto the OpenTelemetry GenAI semantic conventions (gen_ai.system = "anthropic", gen_ai.usage.input_tokens, gen_ai.tool.call, …). See docs/observability.md for the attribute-stability table (stable vs experimental vs custom) and per-backend wiring examples for Tempo / Honeycomb / Jaeger. The "OTel is a sink, not the primary path" decision is recorded in ADR 0006.

License

MIT

Name		Name	Last commit message	Last commit date
.github		.github
Formula		Formula
apps		apps
container		container
crates		crates
docs		docs
eval		eval
helm/netops-agent		helm/netops-agent
prompts		prompts
scripts		scripts
tests/redact_fixtures		tests/redact_fixtures
.dockerignore		.dockerignore
.gitignore		.gitignore
CLAUDE.md		CLAUDE.md
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
Dockerfile		Dockerfile
README.md		README.md
deny.toml		deny.toml
docker-compose.example.yml		docker-compose.example.yml
fleet.lock.toml		fleet.lock.toml
rust-toolchain.toml		rust-toolchain.toml

Navigation Menu

Repository files navigation

netops-agent

Status

Layout

Planned crates

Design docs

Installing `netops-agent`

Quick start

Sync the KB snapshot

Backfill frontmatter on operational runbooks

Eval CI integration

Local-dev parity

Override path: `eval-exempt`

Installing fleet MCPs — readonly by default

Observability — `terminated_by` SLI

Observability — tool I/O byte attribution (issue #108)

Observability — per-class cost SLI (`incident_class`)

Observability — per-tool latency p50/p95 (issue #109)

Observability — OTel sink (opt-in)

License

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

netops-agent

Status

Layout

Planned crates

Design docs

Installing netops-agent

Quick start

Sync the KB snapshot

Backfill frontmatter on operational runbooks

Eval CI integration

Local-dev parity

Override path: eval-exempt

Installing fleet MCPs — readonly by default

Observability — terminated_by SLI

Observability — tool I/O byte attribution (issue #108)

Observability — per-class cost SLI (incident_class)

Observability — per-tool latency p50/p95 (issue #109)

Observability — OTel sink (opt-in)

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

Installing `netops-agent`

Override path: `eval-exempt`

Observability — `terminated_by` SLI

Observability — per-class cost SLI (`incident_class`)