Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion internal/cli/monit_query.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ import (
)

func newMonitQueryCmd() *cobra.Command {
cmd := newGroupCmd("monit-query", "Probe monit-backed datasources (9 types via data; diagnose/rows support prometheus|victorialogs|loki|mysql)")
cmd := newGroupCmd("monit-query", "Probe monit-backed datasources (9 types via data; diagnose: loki|victorialogs log patterns, prometheus metric trends)")
cmd.AddCommand(newMonitQueryDiagnoseCmd())
cmd.AddCommand(newMonitQueryDataCmd())
cmd.AddCommand(newMonitQueryRowsCmd())
Expand Down
8 changes: 6 additions & 2 deletions skills/flashduty/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,11 +69,15 @@ Some asks span several commands. For those the skill ships a script that fetches
| post-mortem / postmortem 复盘 / 复盘报告 / 复盘模板 / post-incident review / RCA report | **`reference/postmortem.md`** |
| alert / 告警 / dedup 去重 / alert fields 告警字段 / alert pipeline 告警管道 | **`reference/alert.md`** |
| change / 变更 / deployment 部署 / release 发布 / correlated change 变更关联 / what changed | **`reference/change.md`** |
| monitor / 监控 / alert rule 告警规则 / datasource 数据源 / inspection 巡检 / rule config 规则配置 | **`reference/monit.md`** |
| monitor / 监控 / inspection 巡检 — unsure which Flashmonit surface | **`reference/monit.md`** (index; routes to the four below) |
| alert rule 告警规则 / rule config 规则配置 / rule folder 规则文件夹 / rule export 规则导出 | **`reference/monit-rule.md`** |
| datasource 数据源 = a system Flashmonit queries (Prometheus / Loki / SQL / SLS) / connect a datasource 连接数据源 / SLS project / logstore | **`reference/monit-datasource.md`** |
| service map 服务地图 / topology 拓扑 / service dependencies 服务依赖 / agent fleet 探针队列 | **`reference/monit-servicemap.md`** |
| store ruleset 规则集 / 规则模板库 | **`reference/monit-ruleset.md`** |
| automation / 自动化 / 定时 AI SRE / scheduled AI task / daily brief / weekly report / webhook trigger / POST trigger / chat-created automation | **`reference/automation.md`** |
| metric/log query / 指标查询 / 日志查询 / PromQL / LogsQL / SQL / trend 趋势 / log clustering 日志聚类 / datasource RCA 数据源排查 | **`reference/monit-query.md`** |
| host diagnostics / 主机诊断 / on-box / process 进程 / load 负载 / lock 锁 / slow query 慢查询 / mysql / reachability 可达性 | **`reference/monit-agent.md`** |
| channel / 协作空间 / collaboration space / 频道 / integration 集成 / alert grouping 告警分组 | **`reference/channel.md`** |
| channel / 协作空间 / collaboration space / 频道 / integration 集成 / 告警来源 alert source / alert grouping 告警分组 | **`reference/channel.md`** |
| dispatch rule 分派策略 / 分派规则 / escalation rule 升级规则 / notify layers 通知层级 / who gets paged | **`reference/escalation.md`** |
| silence 静默 / 屏蔽 / inhibit 抑制 / drop rule 丢弃 / noise reduction 降噪 / maintenance silence 维护窗口静默 | **`reference/noise.md`** |
| enrichment / 数据加工 / 富化 / label mapping 字段映射 / extraction 提取 / mapping schema 集成 schema | **`reference/enrichment.md`** |
Expand Down
80 changes: 80 additions & 0 deletions skills/flashduty/reference/monit-datasource.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
# fduty monit — datasources

Prereq: `SKILL.md` + `reference/monit.md` read. Datasources are what every other Flashmonit surface points at: a rule evaluates one, a probe queries one.

## Route here when

"数据源 / 连接数据源 / SLS project / logstore" or "datasource / connect a datasource / SLS discovery" → this card, **when the datasource is something Flashmonit queries** (Prometheus, Loki, VictoriaLogs, SQL engines, SLS).

**"Datasource" is two unrelated things in this product — check which one the user means.** This card is `POST /monit/datasource/*`: the Flashmonit config surface, i.e. the systems Flashmonit *queries*. On-call has its own, older use of the word: the top-level `datasource` group is `POST /datasource/*` and holds IM-integration plumbing (`fduty datasource im-war-room-enabled-list`, `fduty datasource im-person-try-link`), while the On-call **integrations** that *receive* alerts into a channel live in `reference/channel.md`. So 接入告警 / 集成 / 告警来源 → On-call, not here; "连一个 Prometheus / Loki / MySQL 上来查" → here.

**Mutating:** `datasource-create`, `datasource-update`, `datasource-delete` — confirm before running. **`datasource-delete` is irreversible**; confirm the target with `datasource-info` first.

## Intent → verb

| want | verb |
|---|---|
| list all datasources (by type) | `datasource-list` |
| datasource detail | `datasource-info` |
| create / update a datasource | `datasource-create` / `datasource-update` |
| delete a datasource | `datasource-delete` |
| SLS project/logstore discovery | `datasource-sls-projects` / `datasource-sls-logstores` |

## Gotchas

- **Datasource name is not guessable.** A `can not find datasource` 400 means the name is wrong — re-run `datasource-list` and copy the exact `Name`. Never invent variants.
- **`datasource-info` (and the `datasource-create`/`datasource-update` responses) return credentials exactly as configured — nothing is masked.** The `payload` object includes whatever passwords, API keys, tokens, and similar fields were set, in the clear. Treat the response as sensitive: don't dump it into logs or chat, don't echo it back beyond what the task needs, and don't pass it on to another tool.

<!-- GENERATED:monit[datasource] START · 由 fduty __dump-commands 同步 · 勿手改 fence 内 -->

### datasource-create
Create datasource
- `--address` string — Connection address. For Prometheus/Loki/VictoriaLogs: HTTP URL. For MySQL/Oracle/Postgres/ClickHouse: 'host:port'. For SLS: endpoint without http/https prefix. Not required for Elasticsearch cloud deployment.
- `--edge-cluster-name` string (required) — Monitors edge cluster name responsible for evaluating rules using this datasource.
- `--id` int64 — Datasource ID. Required for update; omit for create.
- `--name` string (required) — Datasource display name. This is the name referenced as 'ds_name' in query and diagnose APIs.
- `--note` string — Optional description.
- `--type-ident` string (required) — Datasource type identifier. Allowed: 'prometheus', 'loki', 'mysql', 'oracle', 'postgres', 'clickhouse', 'elasticsearch', 'sls', 'victorialogs'.
- body-only (`--data`): payload (object) (required)
- response: single object (`data` unwrapped to the top level) — fields: account_id (integer); address (string); edge_cluster_name (string); enabled (boolean); id (integer); name (string); note (string); payload (object); type_ident (string); updated_at (string)

### datasource-delete
Delete datasource
- `--id` int64 (required) — Numeric ID of the target resource; the exact meaning depends on the API being called (e.g. datasource ID, ruleset ID).

### datasource-info
Get datasource detail
- `--id` int64 (required) — Numeric ID of the target resource; the exact meaning depends on the API being called (e.g. datasource ID, ruleset ID).
- response: same shape as `datasource-create` above

### datasource-list
List datasources
- `--type` string — Filter by datasource type identifier. Omit to return all types. Allowed values: 'prometheus', 'loki', 'mysql', 'oracle', 'postgres', 'clickhouse', 'elasticsearch', 'sls', 'victorialogs'.
- response: TOP-LEVEL array — pipe `--json | jq '.[]'` (NOT `.items[]`) — fields: account_id (integer); address (string); edge_cluster_name (string); enabled (boolean); id (integer); name (string); note (string); payload (object); type_ident (string); updated_at (string)

### datasource-sls-logstores
List SLS logstores
- `--id` int64 — ID of an SLS-type datasource. Obtainable via 'POST /monit/datasource/list'.
- `--offset` int64 — Pagination offset.
- `--project` string — SLS project name. Obtainable via 'POST /monit/datasource/sls/projects'.
- `--size` int64 — Page size. Defaults to 200 server-side when 0.

### datasource-sls-projects
List SLS projects
- `--id` int64 — ID of an SLS-type datasource. Obtainable via 'POST /monit/datasource/list'.
- `--offset` int64 — Pagination offset.
- `--query` string — Fuzzy filter on project description (maps to the 'description' parameter of Aliyun SLS ListProject). Leave empty to return all.
- `--size` int64 — Page size. Defaults to 200 server-side when 0.

### datasource-update
Update datasource
- `--address` string — Connection address. For Prometheus/Loki/VictoriaLogs: HTTP URL. For MySQL/Oracle/Postgres/ClickHouse: 'host:port'. For SLS: endpoint without http/https prefix. Not required for Elasticsearch cloud deployment.
- `--edge-cluster-name` string (required) — Monitors edge cluster name responsible for evaluating rules using this datasource.
- `--id` int64 — Datasource ID. Required for update; omit for create.
- `--name` string (required) — Datasource display name. This is the name referenced as 'ds_name' in query and diagnose APIs.
- `--note` string — Optional description.
- `--type-ident` string (required) — Datasource type identifier. Allowed: 'prometheus', 'loki', 'mysql', 'oracle', 'postgres', 'clickhouse', 'elasticsearch', 'sls', 'victorialogs'.
- body-only (`--data`): payload (object) (required)
- response: same shape as `datasource-create` above

<!-- GENERATED:monit[datasource] END -->
115 changes: 115 additions & 0 deletions skills/flashduty/reference/monit-probe.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
# fduty monit — probing a datasource or a target

Prereq: `SKILL.md` + `reference/monit.md` read. These are the runtime verbs: ask a datasource a question, or ask a monitored target about itself. Everything else under `monit` is configuration.

## Route here when

"指标查询 / 日志查询 / PromQL / 诊断 / 监控目标 / 主机工具" or "metric query / log query / diagnose / monitored host / tools catalog" → this card.

**Mutating:** `tools-invoke` runs code on the target — confirm before running. The query verbs are read-only.

## Intent → verb

| want | verb |
|---|---|
| run ad-hoc PromQL / SQL / LogQL | `query-data` here, or the curated `monit-query data` — see `reference/monit-query.md` |
| log-pattern / metric-trend RCA evidence | `query-diagnose` |
| list monitored hosts/targets | `targets` |
| what tools a target exposes | `tools-catalog` |
| run host/db diagnostic tools | `tools-invoke` |

## Hot flow — ad-hoc query + diagnose

```bash
# 1. discover the real datasource name — NEVER guess
fduty monit datasource-list --output-format toon
fduty monit datasource-list --type prometheus --output-format toon

# 2a. point-in-time query (PromQL/SQL/LogQL); ALL time range goes INSIDE --expr
# (the curated 'monit-query data' — see the monit-query card)
fduty monit-query data --ds-type prometheus --ds-name <ds-name> \
--expr 'rate(http_requests_total{job="api"}[5m])' --output-format toon

# 2b. log pattern RCA over last 15 min (time_range via --data; omit = last 15 min default)
fduty monit query-diagnose --ds-type loki --ds-name <ds-name> \
--data '{"input":{"query":"{app=\"payment\"} |= \"error\""}}'

# 2c. metric trend analysis with explicit window
fduty monit query-diagnose --ds-type prometheus --ds-name <ds-name> \
--data '{"input":{"query":"rate(http_errors_total[5m])"},"time_range":{"start":1718780000,"end":1718783600}}'
```

## Hot flow — host diagnostics

```bash
# 1. find the target locator (prefix search; --keyword is prefix-only)
fduty monit targets --keyword prod-web --output-format toon

# 2. discover what tools the target exposes
fduty monit tools-catalog --target-locator <hostname-or-ip> --output-format toon

# 3. invoke tools (up to 8 concurrently); use heredoc to avoid shell quoting hell
fduty monit tools-invoke --target-locator <hostname-or-ip> --output-format toon --data - <<'EOF'
{"tools":[{"tool":"os.overview"},{"tool":"os.top_processes","params":{"top_n":10}}]}
EOF
```

## Key concepts

**`operation` on `query-diagnose`**: `log_patterns` (loki / victorialogs) or `metric_trends` (prometheus); inferred from `--ds-type` when omitted — only pass it explicitly for ambiguous source types.

**`query-diagnose` output**: results are versioned evidence, not the former summary-only pattern/series lists. Read `pattern_evidence` for logs or `series_evidence` for metrics; their optional comparison fields are absent when the edge has no evidence. Log output also includes `data_handling`, which declares redaction coverage and paths carrying untrusted observed data.

**`targets`**: `updated_at` means "last seen", not "online now".

## Gotchas

- **`monit-query data` has no time flags.** There is no `--time-start` / `--time-end` / `--operation`. Embed all time range and bucketing inside `--expr`. Passing those flags is a silent no-op or error.
- **`query-diagnose` time window via `--data`**, not flags. Pass `{"time_range":{"start":<unix>,"end":<unix>},...}`. Window wider than 6 hours is rejected server-side. Omitting `time_range` defaults to the last 15 minutes.
- **`tools-catalog` / `tools-invoke` `--target-locator` is required and not guessable.** If the user has not provided a host or IP, ask — do not invent one. Tool names in `invoke` must come from the `tools-catalog` response — never hallucinate them.

<!-- GENERATED:monit[query,targets,tools] START · 由 fduty __dump-commands 同步 · 勿手改 fence 内 -->

### query-data
Query structured data
- `--account-id` int64 — Optional consistency check. Must equal the authenticated account when supplied; mismatched values are rejected. Business execution always uses the authenticated account.
- `--delay-seconds` int64 — Look-back offset in seconds applied to point-in-time queries (Prometheus, Loki stats, VictoriaLogs stats). Ignored for raw / detail queries.
- `--ds-name` string (required) — Data source name; must match a configured data source under the tenant.
- `--ds-type` string (required) — Data source type; must match a configured data source under the tenant. Examples: 'prometheus', 'loki', 'victorialogs', 'sls', 'elasticsearch', 'mysql', 'postgres', 'oracle', 'clickhouse'.
- `--expr` string (required) — Query expression. Syntax depends on 'ds_type' and is interpreted by the corresponding monit-edge client (PromQL for Prometheus, LogQL for Loki, SQL for SQL sources, etc.).
- body-only (`--data`): args (object)
- response: single object (`data` unwrapped to the top level) — fields: format (string); result (object)

### query-diagnose
Diagnose data source
- `--account-id` int64 — Optional consistency check. Must equal the authenticated account when supplied.
- `--ds-name` string (required) — Data source name configured under the tenant.
- `--ds-type` string (required) — Data source type. 'log_patterns' supports 'loki' and 'victorialogs'; 'metric_trends' supports 'prometheus'.
- `--operation` string — Diagnostic operation. When omitted, inferred from 'ds_type' (loki / victorialogs → 'log_patterns', prometheus → 'metric_trends'). Other sources must specify explicitly. · enum: log_patterns | metric_trends
- body-only (`--data`): input (object) (required); methods (array<object>); options (object); time_range (object)
- response: single object (`data` unwrapped to the top level) — fields: data_handling (object); ds_name (string); ds_type (string); operation (string); query (string); results (array<object>); schema_version (string); window (object)

### targets
List monitored targets
- `--account-id` int64 — Optional consistency check. Must equal the authenticated account when supplied.
- `--cursor` string — Opaque pagination cursor from the previous response's 'next_cursor'. Omit / pass empty string for the first page. Reset whenever 'keyword', 'limit', or tenant changes.
- `--keyword` string — Prefix match against 'target_locator'. ASCII only, no whitespace, no '|', max 256 bytes. Substring search is not supported.
- `--limit` int64 — Page size. Default 50, max 200. (max 200)
- response: `{items: [...], next_cursor, total}` page wrapper — pipe `--json | jq '.items[]'` (NOT top-level `.[]`) — items fields: agent_version (string); cluster_name (string); edge_ipport (string); target_kind (string); target_locator (string); updated_at (string)

### tools-catalog
List target tool catalog
- `--account-id` int64 — Optional consistency check. Must equal the authenticated account when supplied.
- `--target-kind` string — Optional target kind. When omitted, webapi infers it from current target routing. If the call returns 'ambiguous_target_kind', retry with a value from 'target_kinds'.
- `--target-locator` string (required) — Target identifier (host name, MySQL address, …). Max 256 bytes; no whitespace, control characters, or '|'.
- response: single object (`data` unwrapped to the top level) — fields: error (object); target (object); tools (array<object>)

### tools-invoke
Invoke target tools
- `--account-id` int64 — Optional consistency check. Must equal the authenticated account when supplied.
- `--target-kind` string — Optional target kind; auto-inferred when omitted.
- `--target-locator` string (required) — Target identifier. Same validation rules as '/monit/tools/catalog'.
- body-only (`--data`): tools (array<object>) (required)
- response: single object (`data` unwrapped to the top level) — fields: error (object); results (array<object>); target (object)

<!-- GENERATED:monit[query,targets,tools] END -->
Loading