Skip to content

Modernize Netbase to .zeek, fix dormant modules, add Phase 2 enrichment + Redis store foundation - #1

Merged
CyberTaoFlow merged 2 commits into
masterfrom
claude/serene-mendel-vqkfhw
Aug 6, 2026
Merged

Modernize Netbase to .zeek, fix dormant modules, add Phase 2 enrichment + Redis store foundation#1
CyberTaoFlow merged 2 commits into
masterfrom
claude/serene-mendel-vqkfhw

Conversation

@CyberTaoFlow

Copy link
Copy Markdown

Summary

Two-phase pass over Netbase: get the package compiling and installable on current
Zeek, fix the modules that were silently dormant, then add high-signal observables
and the shared-store foundation for detection work.

CI (parse-check + btest on zeek/zeek:latest, Zeek 8) is green.

Phase 0 - modernize & package

  • Migrate *.bro -> *.zeek, bro_init -> zeek_init
  • __load__.zeek now loads every working module (dns/http/ssh/ftp/weird/labels were
    commented out because they did not compile)
  • zkg.meta so the package installs via zkg
  • testing/ btest smoke test + a GitHub Actions workflow that parse-checks the
    package on every push

Phase 1 - correctness

  • dns.zeek: rewrite the non-compilable handler; real DNS observables (client/server
    roles, authoritative vs recursive answers, NXDOMAIN sent/recvd, rejected queries,
    internal/external unique-RR cardinality)
  • labels.zeek: replace the broken, externally-dependent draft (undefined
    profiles/conn_fields/flow_labeled) with a self-contained static CIDR labeler;
    emits ip_labels per host - the peer-grouping hook Phase 3 needs
  • ssh.zeek: fix field mapping so ssh_as_client/server increment their own counters
  • flow.zeek: wire up three dropped observables (out_orig_conns, int_succ_conns,
    inbound_server_conns); remove trailing fallthrough at end of switches
  • weird.zeek: fix orig/resp mix-up and add the missing SEND() calls
  • http/ssh/ftp/weird: gate on is_monitored() to prevent worker-side runtime
    errors on non-monitored IPs
  • stats.zeek: standalone fallback for CLUSTER_NODE; drop val_size() (removed in
    modern Zeek)
  • remove dead, buggy utils.zeek
  • main.zeek: fix &default on a global set (rejected by Zeek 8); geo.zeek: rename
    as (reserved word). Both surfaced by the new CI.

Phase 2 - enrichment (worker-local)

  • flow.zeek: connection duration numstats (internal/external) + long_conns counter
  • geo.zeek (new): unique external country and ASN cardinality for outbound flows
  • ssl.zeek (new): SNI cardinality, deprecated-TLS-version count, cert validation
    failures (base SSL only, no JA3/JA4 package dependency)
  • dns.zeek: query-name length stats + TXT-query count (DGA/tunneling signal)

Shared store (foundation)

  • store.zeek (new, opt-in, not in the default load): wraps the Zeek 7.2+ Storage
    framework Redis backend so the parallel non-cluster workers share one local-Redis
    dataset. Requires a Zeek build with the Redis backend (hiredis); the stock CI image
    is built without it, so store.zeek is excluded from the default load and CI by
    design. It is the foundation for Phase 3 baselines and Phase 2 beaconing.

Architecture note

Deployment is parallel AF_PACKET workers with no cluster. Per-host observables are
split across workers and merged downstream (unchanged by this PR). Anything needing a
consistent cross-worker view - Phase 3 baselines, beaconing - goes through Redis via
store.zeek.

See ROADMAP.md for Phase 3 (baseline.zeek + detect.zeek, shipping in
learning/suppressed mode) and the deferred beaconing module.

Phase 0 - modernize & package:
- Rename all .bro -> .zeek and bro_init -> zeek_init
- __load__.zeek now loads every working module (dns/http/ssh/ftp/weird/labels
  were previously commented out because they did not compile)
- Add zkg.meta so the package installs via zkg
- Add testing/ btest smoke test and a GitHub Actions workflow that parse-checks
  the package on every push

Phase 1 - correctness:
- dns.zeek: rewrite the non-compilable handler; implement real DNS observables
  (client/server roles, authoritative vs recursive answers, NXDOMAIN sent/recvd,
  rejected queries, internal/external unique-RR cardinality)
- labels.zeek: replace the broken, externally-dependent draft (undefined
  profiles/conn_fields/flow_labeled) with a self-contained static CIDR labeler
  (cidr_labels redef + optional Input-framework file); emits ip_labels per host
- ssh.zeek: fix field mapping so ssh_as_client/server increment their own
  counters instead of the auth-fail counters
- flow.zeek: wire up three dropped observables (out_orig_conns, int_succ_conns,
  inbound_server_conns) and remove trailing fallthrough at end of switches
- weird.zeek: fix orig/resp mix-up and add the missing SEND() calls
- http/ssh/ftp/weird: align monitoring gate to is_monitored() to prevent
  worker-side runtime errors on non-monitored IPs
- stats.zeek: fall back to "standalone" when CLUSTER_NODE is unset
- remove orphaned, buggy utils.zeek (dead duplicate of numstats)

Add ROADMAP.md tracking Phase 0-3.
Worker-local-safe observables (aggregate downstream like existing fields):
- flow.zeek: internal/external connection duration numstats (int_dur_*/out_dur_*)
  and a long_conns counter over long_conn_threshold (&redef, default 1h)
- geo.zeek (new): unique external country and ASN cardinality for outbound flows
  (lookup_location / lookup_autonomous_system; degrades to no-op without GeoIP DBs)
- ssl.zeek (new): SNI cardinality, deprecated-TLS-version count, and certificate
  validation-failure count (base SSL only, no JA3/JA4 package dependency)
- dns.zeek: query-name length numstats and TXT-query count (DGA/tunneling signal)

Shared-store foundation for non-cluster parallel workers:
- store.zeek (new, opt-in, not in default __load__): wraps Zeek 7.2+ Storage
  framework Redis backend so each worker opens its own connection to the one local
  Redis dataset. Foundation for Phase 3 baselines and Phase 2 beaconing.

ROADMAP: document the no-cluster/AF_PACKET + local-Redis architecture and its
implications; mark Phase 2 worker-local fields done; move beaconing to land with
the shared store; note get/put has no atomic list ops so cross-worker series
avoid read-modify-write races.
@CyberTaoFlow
CyberTaoFlow merged commit dfc6df8 into master Aug 6, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant