adaptive_export: capture ppid/pcomm in dc_snoop (process-forest seed) - #94
Open
ConstanzeTU wants to merge 12 commits into
Open
adaptive_export: capture ppid/pcomm in dc_snoop (process-forest seed)#94ConstanzeTU wants to merge 12 commits into
ConstanzeTU wants to merge 12 commits into
Conversation
ConstanzeTU
force-pushed
the
fix/ae-protocol-export-pxexport
branch
from
August 7, 2026 18:32
00d654e to
2e05ed2
Compare
…cess-forest seed) Step 1 of the per-node process-forest / ancestry provenance work. dc_snoop's kprobe now walks curtask->real_parent inline (supported in kprobes — see pxbeta/vfs_snoop) and emits, per dcache event: pid_start, ppid, ppid_start, pcomm alongside the existing pid/comm. pid_start/ppid_start are group_leader start_time (ns since boot) — a pid-reuse-stable identity for the process and its parent, so (pid,pid_start)->(ppid,ppid_start) is the forest edge dx walks. - schema.sql: dc_snoop gains pid_start/ppid/ppid_start (Int64) + pcomm (String). Columns() derives from schema.sql, so the strict sink picks them up with no separate Go list to sync. - dc_snoop.pxl (retention) selects the 4 new columns; steered queryfor.go path auto-carries them (no column restriction) — both write paths stay column-matched. - Deliberately NO comm-based parent filter in the AE: the exclude set includes containerd-shim/runc, which parent every container process INCLUDING kubectl-exec attacks, so a naive pcomm drop would suppress real attacks. Pod-rooted ancestry filtering + multi-level correlation is dx's job against the forest. Deploy note: UpsertTracepoint is create-if-absent, so an existing dc_snoop tracepoint must be deleted once for the new program (extra columns) to take effect; fresh deploys are unaffected.
…export
The lookup_fast/d_lookup kprobe fires once per path component, so one file
access lands as N suffix rows (sh, bin/sh, ..., opt/bitnami/common/bin/sh) —
a ~3-4x row multiplier and a big chunk of dc_snoop volume.
Collapse to the full path (longest fragment) per (pid, comm, t, basename,
50us window) in the retention export, before the pod merge:
- px.length(file) picks the full path; px.replace('.*/', file, '') is the
basename group key so two DISTINCT files never merge (a walk's fragments all
share one basename); px.bin(time_, 50us) separates rare same-basename repeats.
- A walk straddling a bin boundary only UNDER-collapses (safe, never drops a
real access). Distinct repeats of the same file are preserved as separate rows;
folding those is a separate (pid,comm,file,window)->count pass (see the NSS
nsswitch.conf re-open case).
Uses only proven primitives (px.length/socket_size, px.bin/cql_flow_graph,
px.replace/differential — pattern-first arg order). Needs a live rig run to
validate the composition (px auth pending).
Fast (go test ~2ms) invariants verifying each piece without the rig loop: - TestDcSnoopExportColumnsMatchSchema: export df[[...]] projection (+event_time) == clickhouse.Columns(dc_snoop) — the strict-sink coupling adding ppid could have broken (missing/unknown column -> INSERT fails). - TestDcSnoopTracepointCapturesParent: bpftrace emits ppid/pcomm + group_leader starts via real_parent in BOTH probe blocks. - TestDcSnoopCollapseKeepsRepeats: collapse keyed by basename, filters to max-length rows with NO count aggregation — repeats survive with exact timestamps.
Quantifies exactly how much the ppid addition loads into memory, for the node-crash RCA: - wire (export JSON): +104 bytes/row (239 -> 343) - columnar (PEM table-store): +40 bytes/event (3xInt64 + comm16) Per 1M dcache events: +40 MB columnar / +104 MB wire. Conclusion the RCA needs: the +40B/event is bounded — the PEM table-store is CAPPED, so a bigger row fills the cap faster (shorter lookback) but never raises PEACE memory; and the collapse removes ~3.5x rows so net EXPORTED data drops. So the ppid payload is not a peak-memory regression. The remaining suspect is the collapse's per-export groupby+merge in the PEM Carnot (unbounded per-query), which this Go benchmark can't measure — needs live kubectl-top during export.
…estry filter Two coupled fixes for the dc_snoop process-forest work: 1. Tracepoint capture bug (ppid/pcomm parsed as 0 across 49k rows on the rig). The printf used the 64-bit 'lld' verb (not in Pixie's tracepoint printf subset — it misaligns every field after it), ->real_parent, and a $tk intermediate. Rewrite both probe blocks to the exec_snoop/vfs_snoop-proven form: %d ints, curtask->parent inline, group_leader->start_time/10000000 (clock ticks, unit-consistent with exec_snoop for a future forest join). Each dcache event now carries its parent's identity + a pid-reuse-stable start for both the process and its parent. 2. ppid ancestry filter (the noise cut). Resolve the PARENT's namespace via a process_stats join keyed on ppid, and drop events whose parent lives in an own-stack namespace — even when the child's own pod is blank (transient process exec'd by an infra pod). Pod/namespace-rooted, not comm-matched, so real attack children of shared blank-namespace parents (containerd-shim/runc) survive. Drops injected from env at __DC_SNOOP_PARENT_EXCLUSION__ using the same DC_SNOOP_EXCLUDE_NAMESPACES list as the self filter; parent_namespace is a temp column dropped by the final projection, so the schema/sink coupling is unchanged. Multi-level ancestry stays dx's job on the process forest. Contract tests updated: tracepoint must use ->parent (not real_parent) in both probe blocks and must not reintroduce the 64-bit verb; the ancestry join must resolve parent_namespace and never leak it to the sink; presets_test asserts the parent-exclusion injection + env override.
UpsertTracepoint is create-if-absent — on an already-deployed tracepoint it no-ops and never refreshes a changed bpftrace program. That is exactly why the dc_snoop ppid printf fix could not reach a cluster already running the old (broken %lld) program: swapping the image installed new export/schema code but left the stale RUNNING tracepoint, so ppid/pcomm stayed 0. deployDesiredTracepoints now DELETES each desired tracepoint first, waits (polls its output table until it stops compiling = gone), then upserts — so the RUNNING program always matches the code in this image. The delete mutation returns the same benign "stream: unimplemented type" error as the upsert and applies server-side; if the table never clears we proceed anyway (no worse than the prior plain-upsert behavior). Brief teardown/redeploy gap on restart is acceptable (restarts are rare; capture is continuous otherwise). This removes the recurring "changed tracepoint program doesn't refresh" foot-gun for every AE-owned bpftrace (dc_snoop, creds_change, …), not just this fix.
…apture)
RCA from the rig (ae_reconcile): the previous image regressed dc_snoop capture
from 49132 read-rows to 0 — the tracepoint reported RUNNING but emitted nothing
and flapped FAILED<->RUNNING. Cause: the printf grew to 9 args (added pid_start,
ppid, ppid_start, AND pcomm), exceeding Pixie's tracepoint bpftrace budget
("printf: Too many arguments for format string"), so the program never compiled.
The proven exec_snoop tracepoint ships exactly 8 args (5 numeric + 3 strings).
Drop the 9th — parent comm (the 4th string) — to match that profile. The ancestry
filter needs only ppid (it joins process_stats on ppid for the parent's namespace,
never the parent comm), so nothing of value is lost; dx resolves parent comm from
ppid on the process forest.
Removed pcomm from the tracepoint printf (both probe blocks), schema.sql, the
dc_snoop.pxl export projection, and the bench row/cols. Contract test now asserts
the 8-arg budget (<=8 conversion specifiers per printf) and that parent comm is
NOT captured, so the regression cannot recur.
…u (not a rewrite) My earlier "fix to the proven exec_snoop form" (inline curtask casts, ->parent, %d, /10000000 divide) REGRESSED dc_snoop capture to zero: ae_reconcile read_count went 49132 -> 0. RCA via git: image 9fd0ca4 CAPTURED 49132 rows with a 9-arg printf ($tk intermediate, real_parent, raw group_leader->start_time, %lld starts, pcomm) — so neither the 9-arg count (my prior wrong RCA) nor pcomm was the problem. The capture-killer was the structural rewrite (the /10000000 divide and/or ->parent/inline-cast changes), which made the bpftrace program emit nothing while still reporting RUNNING (the AE RUNNING check only confirms the table compiles in PxL metadata, not that the bpftrace loaded). This reverts dc_snoop_deploy.pxl to 9fd0ca4's exact capturing structure and makes the ONE change needed to fix ppid=0: the two u64 start fields use %llu, not %lld. The signed 64-bit verb is not handled by Pixie's tracepoint printf output parser and misaligns every field after it (why ppid parsed 0 despite being a correct %d); %llu is the proven u64 verb (time_ uses it). pcomm is restored (9 args capture fine) in schema.sql, the dc_snoop.pxl projection, and the bench. Contract test now pins the capturing form: real_parent in both probes, %llu (not %lld) on the start fields, no /10000000 divide, and pcomm present. Reverts the incorrect 8-arg-budget assertion from the prior commit.
…n-upsert)" This reverts commit 5b419fd.
… ancestry
Now that ppid capture is verified working (100% populated), replace the whack-a-mole
comm blocklists with the ppid-based ancestry cut, in BOTH dc_snoop paths:
- queryfor.go (dx-steered dark path): DELETE darkExcludeCommsDefault (the big
own-stack/host/CNI comm list) + darkExcludeCommSubstrings (kernel-thread families)
+ darkVectorHasComm + darkCommExclusion. Add darkParentAncestryExclusion: resolve
the parent's namespace via a process_stats join on ppid and drop own-stack parents,
gated to tables that actually capture ppid (darkVectorHasPpid = {dc_snoop}). Drops
parent_namespace before display so the sink projection is unchanged. creds_change
and the dx_* tables (no ppid) get the child-namespace exclusion only.
- presets.go (retention path): DELETE defaultExcludeComms + the comm-drop generation
in dcSnoopExclusion; the ancestry filter (dcSnoopParentExclusion) already lands the
parent-namespace cut. The child + parent namespace lists share one resolver so they
can never drift.
Noise is now cut structurally: own-stack PODS by their resolved namespace (child +
parent), their transient blank-ns children by the parent-ancestry join. Host-level
runtime + kernel threads (containerd-shim/systemd/k3s/kworker — blank namespace, blank/
kernel parent that 1-level ancestry can't resolve) intentionally enter the always-on
retention look-back, to be relevance-filtered by dx's multi-level process forest, not
by a hardcoded list.
Comprehensive tests: dark path has NO comm drops + the ancestry join/drop (dc_snoop);
ancestry applies ONLY to ppid tables (creds_change/dx_* excluded); the shared namespace
env override drives both child + parent drops; and TestQueryFor_CleanupNoCollateral
proves every native/protocol table (redis/dns/http/conn_stats/pgsql/mysql/stack_trace)
is untouched — no ancestry, no parent_namespace, no comm drops, still pod-scoped.
ConstanzeTU
force-pushed
the
feat/dc-snoop-ppid-ancestry
branch
from
August 10, 2026 13:24
b5731a7 to
0d9ee64
Compare
…dcoded infra exclusion socdemo/socdemo-ch are the sovereignsocdemo lab namespaces — demo-specific, not generic infra, and must never be hardcoded in the AE. Removing them makes darkExcludeNamespacesDefault match presets.go defaultExcludeNamespaces exactly (generic own-stack/kube infra only). A deployment that needs to exclude a demo namespace can set DC_SNOOP_EXCLUDE_NAMESPACES.
entlein
approved these changes
Aug 10, 2026
This was referenced Aug 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Step 1 of the per-node process-forest / ancestry provenance work (pairs with the dx forest PR stacked on entlein/dx#129).
What: the dc_snoop kprobe now walks
curtask->real_parentinline and emits, per dcache event,pid_start,ppid,ppid_start,pcommalongsidepid/comm.pid_start/ppid_startaregroup_leader->start_time(ns since boot) — a pid-reuse-stable identity, so(pid,pid_start) → (ppid,ppid_start)is the forest edge dx walks.Why here (AE) vs dx: the AE owns the tracepoint + the process data; dx owns the forest cache, pod-rooted ancestry filtering, and steering. This PR is only the data source.
Files
internal/clickhouse/schema.sql— dc_snoop gainspid_start/ppid/ppid_start(Int64) +pcomm(String).Columns()derives from schema.sql, so the strictfastencodesink picks them up with no separate Go list.presets/dc_snoop_deploy.pxl— tracepoint emits the 4 new fields (both kprobe blocks).presets/dc_snoop.pxl— retention export selects them; steeredqueryfor.goauto-carries them (no column restriction) so both write paths stay column-matched.Deliberately NOT done: a comm-based parent filter in the AE. The exclude set includes
containerd-shim/runc, which parent every container process includingkubectl execattacks, so a naivepcommdrop would suppress real attacks. Pod-rooted, multi-level ancestry filtering is dx (against the forest).Deploy note:
UpsertTracepointis create-if-absent — an existing dc_snoop tracepoint must be deleted once for the new (wider) program to take effect; fresh deploys are unaffected.🤖 Generated with Claude Code