Modernize Netbase to .zeek, fix dormant modules, add Phase 2 enrichment + Redis store foundation

.github/workflows/ci.yml

@@ -0,0 +1,24 @@
+name: ci
+
+on:
+  push:
+  pull_request:
+
+jobs:
+  parse-and-test:
+    runs-on: ubuntu-latest
+    container: zeek/zeek:latest
+    steps:
+      - uses: actions/checkout@v4
+
+      # Fail the build on any parse/load error in the package. With no input
+      # interface or trace, Zeek runs zeek_init/zeek_done and exits.
+      - name: Parse-check the Netbase package
+        run: zeek -C .
+
+      # Run the btest smoke test(s). btest ships with the zeek image.
+      - name: Run btests
+        run: |
+          export ZEEKPATH="$(pwd)/..:$(zeek-config --zeekpath)"
+          cd testing
+          btest -d

ROADMAP.md

@@ -0,0 +1,137 @@
+# Netbase Roadmap
+
+This roadmap tracks the work to harden Netbase and grow it from a data-generation
+framework into one that closes the loop with tuning and detection workflows.
+
+Legend: ✅ done · 🔜 planned · 🧪 needs validation
+
+## Deployment architecture (drives the design)
+
+Netbase runs as **several independent Zeek worker processes load-balanced over the
+same traffic (AF_PACKET fanout), with no cluster**. Implications:
+
+- There is no Broker/proxy tier. Every worker takes the `! Cluster::is_enabled()`
+  path and aggregates its own slice in-process, then writes its own `netbase.log`.
+  Per-host observables for an interval are therefore **split across workers and must
+  be merged downstream** (sum counts, union cardinality sets) — this is unchanged by
+  the work here and already how the deployment consumes the logs.
+- Anything needing a single, consistent cross-worker view — **rolling baselines
+  (Phase 3)** and **beaconing series (Phase 2)** — cannot live in worker memory.
+- The sensor has a **local Redis** instance. That is the shared store. `store.zeek`
+  (✅, opt-in) wraps Zeek 7.2+'s Storage framework Redis backend so each worker opens
+  its own connection to the one Redis dataset. Note: get/put is a plain key→value
+  interface with no atomic list ops, so series that multiple workers append to are
+  designed to avoid read-modify-write races (per-worker keys + aggregation, not
+  shared mutable lists).
+
+---
+
+## Phase 0 — Modernize & package ✅
+
+Foundation work so the package installs and validates cleanly on current Zeek.
+
+- ✅ Migrate `*.bro` → `*.zeek` and `bro_init` → `zeek_init`.
+- ✅ `__load__.zeek` now loads every working module (main, labels, flow, software,
+  dns, http, ssh, ftp, weird, stats).
+- ✅ `zkg.meta` added so the package installs via `zkg install`.
+- ✅ `testing/` btest smoke test + `.github/workflows/ci.yml` that parse-checks the
+  package on every push (this is the regression gate for the fixes below).
+
+## Phase 1 — Correctness ✅
+
+Repair the modules that were dormant because they did not compile or silently
+dropped data.
+
+- ✅ **dns.zeek** — rewritten: valid event handler, real DNS observables
+  (client/server roles, authoritative vs recursive answers, NXDOMAIN sent/received,
+  rejected queries, internal/external unique-RR cardinality). Previously
+  non-compilable.
+- ✅ **labels.zeek** — replaced the broken, externally-dependent draft with a
+  self-contained static CIDR labeling implementation (`cidr_labels` redef + optional
+  `cidr_labels_file` via the Input framework). Emits `ip_labels` per host — the hook
+  Phase 3 peer-group comparison depends on.
+- ✅ **ssh.zeek** — fixed field mapping: `ssh_as_client`/`ssh_as_server` now
+  increment their own counters instead of the auth-fail counters.
+- ✅ **flow.zeek** — wired up three dropped observables (`out_orig_conns`,
+  `int_succ_conns`, `inbound_server_conns`) and removed trailing `fallthrough`
+  statements at the end of switches.
+- ✅ **weird.zeek** — fixed the orig/resp mix-up and added the missing `SEND()`
+  calls (weird observables were never delivered before).
+- ✅ **http/ssh/ftp/weird** — monitoring gate aligned to `Netbase::is_monitored()`,
+  matching flow/dns and preventing proxy-side runtime errors on non-monitored IPs.
+- ✅ **stats.zeek** — `CLUSTER_NODE` now falls back to `standalone`.
+- ✅ Removed the orphaned, buggy `utils.zeek` (dead duplicate of `numstats`).
+
+## Phase 2 — Enrich observables (worker-local) ✅ / beaconing 🔜
+
+Additive, high-signal fields that detection keys on. The first four aggregate
+downstream exactly like existing observables (no shared state needed) and shipped
+together:
+
+- ✅ **flow.zeek** — connection **duration** numstats (internal/external,
+  `int_dur_*` / `out_dur_*`) + a `long_conns` counter over the `long_conn_threshold`
+  (`&redef`, default 1h).
+- ✅ **geo.zeek** (new) — cardinality of unique **countries** (`ext_country_cnt`) and
+  **ASNs** (`ext_asn_cnt`) for external peers (`lookup_location` /
+  `lookup_autonomous_system`; degrades to no-op without GeoIP DBs). Enables
+  "new country / new ASN" detection in Phase 3.
+- ✅ **ssl.zeek** (new) — **SNI** cardinality (`tls_sni_cnt`), deprecated-TLS-version
+  count (`tls_old_version_conns`), and certificate **validation-failure** count
+  (`tls_validation_failures`). Base-only (no JA3/JA4 package dependency).
+- ✅ **dns.zeek** — query-name **length** stats (`dns_qname_len_*`) and **TXT**-query
+  count (`dns_txt_queries`) for DGA / tunneling signal. (NXDOMAIN ratio is derivable
+  downstream from the existing `dns_nxdomain_*` counters.)
+
+Deferred — needs the shared store, so it lands with Phase 3:
+
+- 🔜 **beacon.zeek** (new) — per `(src → dst:port)` inter-arrival regularity
+  (coefficient of variation) to flag periodic beaconing. **Cannot** be done per-worker:
+  flow-hash fanout scatters a host's repeat connections across workers, so no single
+  worker sees the full series. Will key per-tuple series in **Redis via `store.zeek`**
+  (per-worker contribution keys to avoid append races), then score periodicity.
+  - **Opt-out:** disabled by default behind `Netbase::enable_beaconing = F &redef`.
+  - **Memory / cost controls:** `&redef` caps on tracked tuples per host and samples
+    per tuple, an idle expiry on the tracking state, an option to restrict tracking to
+    external destinations only, and a Redis-key TTL so abandoned series self-clean.
+
+### JA3/JA4 (optional follow-up)
+TLS fingerprint cardinality (JA3/JA4) is high value but requires the external
+`zeek/ja3` (and JA4) packages. Add as an optional module guarded on those packages so
+the core stays dependency-free.
+
+## Phase 3 — Detection & tuning integration 🔜
+
+The feedback loop the README describes but never implemented. Ships in
+**learning / suppressed mode by default** so it is safe to deploy. Built on the
+**Redis shared store** (`store.zeek` ✅) so all parallel workers read/write one
+consistent baseline.
+
+- ✅ **store.zeek** (foundation) — opt-in Redis backend wrapper (Zeek 7.2+ Storage
+  framework) giving every worker a connection to the one local Redis dataset.
+- 🔜 **baseline.zeek** (new) — running per-host and per-peer-group stats persisted in
+  Redis via `store.zeek` (Welford mean/variance + observed categorical sets: ports,
+  ASNs, software, roles). Exposes `Netbase::is_anomalous(ip, field, value)` and
+  `Netbase::zscore(...)`. Cross-worker writes use per-worker keys + periodic merge to
+  avoid read-modify-write races on shared values.
+- 🔜 **detect.zeek** (new) — on `log_observation`, raise Zeek **Notices**:
+  - numeric: z-score over a `&redef` threshold (conn counts, ext bytes, durations);
+  - categorical "first seen": new external ASN/country, new listening port, new
+    software version, first-time SMB/RDP/SSH **server** role.
+- 🔜 **Tuning loop** — a `&redef` learning window during which baselines build but
+  Notices stay suppressed, plus an analyst **allowlist** (`approved_observables.tsv`,
+  Input framework) to suppress known-good per host/group. That file is the tuning
+  artifact; combined with Zeek's native Notice suppression.
+- 🔜 **Peer-group cold-start** — use Phase 1 labels (role/OS) so a new host is scored
+  against its peer group until it has its own history.
+- 🔜 **SIEM export** — documented ECS field mapping for `netbase.log` + a JSON
+  log-policy toggle, so baselines and anomaly Notices correlate downstream.
+
+---
+
+## Sequencing
+
+```
+Phase 0 ─┬─ Phase 1 (correctness)   ← gate: CI green, all modules load
+         └─ Phase 2 (enrichment)    ← overlaps; each field independently shippable
+                       └─ Phase 3 (detection) ← needs labels (P1) + geo/ssl (P2)
+```

__load__.zeek

@@ -0,0 +1,12 @@
+@load ./main
+@load ./labels
+@load ./flow
+@load ./software
+@load ./dns
+@load ./http
+@load ./ssh
+@load ./ftp
+@load ./weird
+@load ./geo
+@load ./ssl
+@load ./stats