Modernize Netbase to .zeek, fix dormant modules, add Phase 2 enrichment + Redis store foundation - #1
Merged
Conversation
Phase 0 - modernize & package: - Rename all .bro -> .zeek and bro_init -> zeek_init - __load__.zeek now loads every working module (dns/http/ssh/ftp/weird/labels were previously commented out because they did not compile) - Add zkg.meta so the package installs via zkg - Add testing/ btest smoke test and a GitHub Actions workflow that parse-checks the package on every push Phase 1 - correctness: - dns.zeek: rewrite the non-compilable handler; implement real DNS observables (client/server roles, authoritative vs recursive answers, NXDOMAIN sent/recvd, rejected queries, internal/external unique-RR cardinality) - labels.zeek: replace the broken, externally-dependent draft (undefined profiles/conn_fields/flow_labeled) with a self-contained static CIDR labeler (cidr_labels redef + optional Input-framework file); emits ip_labels per host - ssh.zeek: fix field mapping so ssh_as_client/server increment their own counters instead of the auth-fail counters - flow.zeek: wire up three dropped observables (out_orig_conns, int_succ_conns, inbound_server_conns) and remove trailing fallthrough at end of switches - weird.zeek: fix orig/resp mix-up and add the missing SEND() calls - http/ssh/ftp/weird: align monitoring gate to is_monitored() to prevent worker-side runtime errors on non-monitored IPs - stats.zeek: fall back to "standalone" when CLUSTER_NODE is unset - remove orphaned, buggy utils.zeek (dead duplicate of numstats) Add ROADMAP.md tracking Phase 0-3.
Worker-local-safe observables (aggregate downstream like existing fields): - flow.zeek: internal/external connection duration numstats (int_dur_*/out_dur_*) and a long_conns counter over long_conn_threshold (&redef, default 1h) - geo.zeek (new): unique external country and ASN cardinality for outbound flows (lookup_location / lookup_autonomous_system; degrades to no-op without GeoIP DBs) - ssl.zeek (new): SNI cardinality, deprecated-TLS-version count, and certificate validation-failure count (base SSL only, no JA3/JA4 package dependency) - dns.zeek: query-name length numstats and TXT-query count (DGA/tunneling signal) Shared-store foundation for non-cluster parallel workers: - store.zeek (new, opt-in, not in default __load__): wraps Zeek 7.2+ Storage framework Redis backend so each worker opens its own connection to the one local Redis dataset. Foundation for Phase 3 baselines and Phase 2 beaconing. ROADMAP: document the no-cluster/AF_PACKET + local-Redis architecture and its implications; mark Phase 2 worker-local fields done; move beaconing to land with the shared store; note get/put has no atomic list ops so cross-worker series avoid read-modify-write races.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two-phase pass over Netbase: get the package compiling and installable on current
Zeek, fix the modules that were silently dormant, then add high-signal observables
and the shared-store foundation for detection work.
CI (parse-check + btest on
zeek/zeek:latest, Zeek 8) is green.Phase 0 - modernize & package
*.bro->*.zeek,bro_init->zeek_init__load__.zeeknow loads every working module (dns/http/ssh/ftp/weird/labels werecommented out because they did not compile)
zkg.metaso the package installs viazkgtesting/btest smoke test + a GitHub Actions workflow that parse-checks thepackage on every push
Phase 1 - correctness
dns.zeek: rewrite the non-compilable handler; real DNS observables (client/serverroles, authoritative vs recursive answers, NXDOMAIN sent/recvd, rejected queries,
internal/external unique-RR cardinality)
labels.zeek: replace the broken, externally-dependent draft (undefinedprofiles/conn_fields/flow_labeled) with a self-contained static CIDR labeler;emits
ip_labelsper host - the peer-grouping hook Phase 3 needsssh.zeek: fix field mapping sossh_as_client/serverincrement their own countersflow.zeek: wire up three dropped observables (out_orig_conns,int_succ_conns,inbound_server_conns); remove trailingfallthroughat end of switchesweird.zeek: fix orig/resp mix-up and add the missingSEND()callshttp/ssh/ftp/weird: gate onis_monitored()to prevent worker-side runtimeerrors on non-monitored IPs
stats.zeek: standalone fallback forCLUSTER_NODE; dropval_size()(removed inmodern Zeek)
utils.zeekmain.zeek: fix&defaulton a global set (rejected by Zeek 8);geo.zeek: renameas(reserved word). Both surfaced by the new CI.Phase 2 - enrichment (worker-local)
flow.zeek: connection duration numstats (internal/external) +long_connscountergeo.zeek(new): unique external country and ASN cardinality for outbound flowsssl.zeek(new): SNI cardinality, deprecated-TLS-version count, cert validationfailures (base SSL only, no JA3/JA4 package dependency)
dns.zeek: query-name length stats + TXT-query count (DGA/tunneling signal)Shared store (foundation)
store.zeek(new, opt-in, not in the default load): wraps the Zeek 7.2+ Storageframework Redis backend so the parallel non-cluster workers share one local-Redis
dataset. Requires a Zeek build with the Redis backend (hiredis); the stock CI image
is built without it, so
store.zeekis excluded from the default load and CI bydesign. It is the foundation for Phase 3 baselines and Phase 2 beaconing.
Architecture note
Deployment is parallel AF_PACKET workers with no cluster. Per-host observables are
split across workers and merged downstream (unchanged by this PR). Anything needing a
consistent cross-worker view - Phase 3 baselines, beaconing - goes through Redis via
store.zeek.See
ROADMAP.mdfor Phase 3 (baseline.zeek+detect.zeek, shipping inlearning/suppressed mode) and the deferred beaconing module.