adaptive_export: reliable dx-steered pem-direct capture (chunk/end_time, breaker, dc_snoop filter, DaemonSet) - #92
adaptive_export: reliable dx-steered pem-direct capture (chunk/end_time, breaker, dc_snoop filter, DaemonSet)#92ConstanzeTU wants to merge 66 commits into
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…e fix) Root cause of the flaky dx-steered capture (dc_snoop/http erratically 0 while light tables always land): OrderExportAll fans out ~20 tables concurrently, each OrderQuery issued ONE unbounded PxL query over the full ~600s control window against the single node-local PEM (pem-direct). QueryFor only set start_time, so every query re-scanned [sliceStart, now] and post-filtered — the heavy tables materialize huge result sets on a saturated PEM and lose the fixed 180s deadline race, dropping out; the cheap tables (redis/conn/stack) return instantly and survive. Reconcile fingerprint: the same dc_snoop query returns 2459 rows in isolation but 0 + 1 err under the fan-out. Fix (durable — removes the data-volume↔deadline coupling, not just tunes it): - pxl.QueryFor: bound the PEM source scan on BOTH sides. Emit a relative end_time (floored toward now so nothing real is clipped; the exact upper bound stays enforced by the df.time_ < sliceEnd nanos post-filter) whenever sliceEnd is in the past. Live-edge slices keep scanning to now (no end_time), preserving prior behavior for the most-recent window. - controller.OrderQuery: walk the capture window in OrderChunk-sized sub-windows (default 60s, env ADAPTIVE_ORDER_CHUNK_SEC), each a both-sides bounded query, so no single query re-materializes the whole window. captureSpan adaptively halves any chunk that still fails with a transient (deadline/overload) error down to orderMinChunk (1s); non-transient errors (missing dark table) surface immediately without wasteful splitting. Overlapping/retried spans dedupe in the ReplacingMergeTree evidence tables, so re-pulls are idempotent. One aggregated reconcile row per table (not per chunk). Chunks run sequentially per table, so OrderExportAll's per-table concurrency is unchanged while each table now issues cheap bounded queries instead of one firehose — reliable capture without needing the global inflight throttle set. Tests: queryfor end_time present for past windows / absent at the live edge; OrderQuery chunking, single aggregated reconcile row, adaptive subdivision on transient error, no-split on non-transient error, termination at min-chunk.
… (dc_snoop) The dx-steered OrderExportAll path applied only a partial comm denylist and NO namespace filter to the node-scoped dark-vector tables — unlike the shipped cron preset (script/presets dc_snoop.pxl __DC_SNOOP_EXCLUSION__, built from presets.go defaultExcludeNamespaces + defaultExcludeComms). So every dc_snoop capture drowned in infra dcache churn: on a real k3s node a single window returned ~54k rows dominated by ConfigReloader/iptables/CNI(host-local,bridge,flannel,loopback)/host daemons(systemd-udevd,dbus-daemon,tailscaled)/kubevuln — burying the salient attack specimens (whoami/cat/getent reading /etc/shadow + the SA token). - Extend darkExcludeCommsDefault with the host/CNI/node daemons that were leaking (systemd-udevd, host-local, bridge, flannel, loopback, bandwidth, dbus-daemon, mount, umount, tailscaled, grpc_health_pro, kubevuln, opm, kube-proxy, …). - Add darkExcludeNamespacesDefault + darkNamespaceExclusion(), applied in the IsDarkVector branch AFTER PodEnrichPxL resolves df.namespace, dropping infra namespaces (pl, kube-system, clickhouse, …). Blank-namespace transient rows survive (each `!=` is true for ''), so the attack's short-lived children — which resolve blank — are never dropped. Overridable via DC_SNOOP_EXCLUDE_NAMESPACES. Kept in sync with script/presets.go. Tests: infra namespaces + host/CNI comms dropped; df.namespace never pinned to the alert pod (node-scoped); env override replaces the default list.
… depth cap) Live RCA on aeprod54: the chunk fix is correct in isolation (pem unit suite — dc_snoop 54k, redis/conn/stack written per-chunk) but UNSAFE under the dx steering firehose. dx does generic collect-per-alert, so OrderExportAll (20 tables) fires on every noisy pl system pod continuously; all land on the ONE node-local PEM (pem-direct) → it saturates → 100% DeadlineExceeded. captureSpan then split every timeout into two narrower retries, amplifying a busy PEM into a query storm where nothing completes (observed: "0 ordered pixie rows written" across the whole run; draining dx + restarting AE → pem-direct instantly serves again). Make subdivision safe: - Circuit-breaker: orderTimeoutStreak (atomic) counts CONSECUTIVE transient failures; any success resets it. Above orderBreakerTrip (8) captureSpan stops subdividing — a saturated PEM must not be flooded with retries. It still splits a genuinely-oversized window on a healthy PEM (the reset keeps that path live). - Depth cap: maxOrderSplitDepth (3) bounds one chunk to ≤2^3 leaf queries even if it keeps timing out (was ~64 splitting 60s→1s). Tests: a 10-chunk all-timeout window stays <60 queries (ungated ≈640); a single transient failure still recovers (breaker resets on success, no latch). NOTE (deployment, not code): the firehose root also needs dx steering scoped so it doesn't fire 20-table captures on every noisy pl/system-pod alert — tracked separately for dx-agent.
Live RCA (aeprod55): every dx-steered capture in the e2e returned 0 rows, and the reconcile showed why — all 36 ordered captures had ~512ns-wide windows (width_s=0), so they matched no pixie rows. /export/start already reaches back controlExportLookback, but a control client that keys the /query window on a single finding's event_time sends lo≈hi (a sub-microsecond span). That passes the lo<hi validation yet captures nothing. handleQuery now widens any window narrower than minControlQueryWindow (5s) to controlExportLookback ending at hi — a point-in-time referral still captures the evidence leading up to it. hi is preserved; comfortably-wide windows pass through unchanged. Isolated /query probes (proper windows) already proved the capture path works — dc_snoop 54k→16k filtered, redis/conn/stack per-chunk; this makes the dx-driven path robust to degenerate windows too. Tests: a 512ns window is widened to >=5s (hi preserved); a 120s window is untouched. NOTE (dx-agent): dx should send a real window (or use /export/start) rather than a point window per finding — tracked separately. This is the AE-side safety net.
The bootstrap manifest was a replicas:0 Deployment with minimal env (EXPORT_MODE= auto, no pem-direct, no throttle) — it never ran and could not do node-local pem-direct. Replace it with the working config that the e2e RCA validated: - DaemonSet (one-per-node) so each pod queries its OWN node's vizier-pem at HOST_IP:50305 (pem-direct: node-local, desync-immune). - dx-steered: EXPORT_MODE=never + CONTROL_ADDR=:9100 + the control Service (internalTrafficPolicy:Local so dx reaches its co-located AE). - PEM-protection: ADAPTIVE_MAX_INFLIGHT_QUERIES_GLOBAL=4 and ADAPTIVE_ORDER_CHUNK_SEC =600 (one query per table, no window pre-chunking) so the AE never saturates the single node-local PEM it shares with dx. See RCA_ae_capture_20260803. Secret still seeded per-cluster (unchanged).
…efault; trim comments - queryfor.go: add darkExcludeCommSubstrings (kworker/ksoftirqd/rcu_/… — kernel threads with variable suffixes exact-match misses) applied via px.logicalNot( px.contains); add pause + systemd-logind exact. Workload comms (redis-*) untouched. - controller.go: defaultOrderChunk 60s -> 600s (one query per table; pre-chunking 10x-amplified queries on the single node-local PEM). - Strip verbose comments across queryfor.go/controller.go/server.go + the AE manifest. Test: kernel-thread substrings dropped, workload comms kept, pause dropped.
Deploys the dx-daemon DaemonSet + Service into honey and mirrors the pl->honey secrets (jwt-signing-key, cluster-id, cloud-addr, api-key, clickhouse http-url) via a before-hook, replacing the hand-applied manifest used in the e2e. Deploy with: skaffold deploy -f k8s/vizier/dx/skaffold.yaml CH http-url defaults to the soc clickhouse Service; override with DX_CH_HTTP_URL.
Replaces the imperative seed-secret + patch-cloud-addr + sed-image +
kubectl-apply sequence with a single skaffold module:
skaffold deploy -f k8s/vizier/adaptive_export/skaffold.yaml
- kustomize overlay reuses bootstrap/adaptive_export_{role,deployment}
and pins the image via images: (ghcr aeprod tag) instead of sed.
- before-hook patches PL_CLOUD_ADDR :443 and seeds
pl-adaptive-export-secrets ONLY when PIXIE_API_KEY/PX_API_KEY is set,
never clobbering an existing secret with an empty key.
- LoadRestrictionsNone so the overlay can reuse the bootstrap manifests
in place (no duplication/drift).
Pairs with the dx-daemon skaffold (k8s/vizier/dx). Bump the AE image by
editing newTag in kustomization.yaml.
…aths
The AE/dx skaffold configs lived inside their overlay dirs with kustomize
paths: [.], which skaffold resolves against the shell CWD (repo root), not
the config-file dir -> 'unable to find kustomization.yaml in /.../pixie'.
Match the repo convention instead (skaffold/skaffold_vizier.yaml et al.):
skaffold configs live in skaffold/ and reference overlays by repo-root-
relative kustomize paths. Overlays stay in k8s/vizier/{adaptive_export,dx}.
skaffold deploy -f skaffold/skaffold_adaptive_export.yaml
skaffold deploy -f skaffold/skaffold_dx.yaml # run from repo root
- dx overlay gains a kustomization.yaml (was rawYaml).
- both validated with 'skaffold render' from repo root (image overrides +
RBAC/DaemonSet/Service resolve).
00d654e to
2e05ed2
Compare
|
@build-agent — (reposting here, channel moved) please build the AE image from branch Fixes the |
Builds for
|
|
@build-agent — Please build the AE image from branch |
|
@build-agent — being specific, please build the AE (adaptive_export) image from this exact ref, NOT the #92 PR head:
This ref = #94 ppid/pcomm dc_snoop capture + PR#95 streaming dark-vector fix. Verify:
The last build (aeprod59) was #92 head |
|
Re-built from TTL image (ready now, expires ~24h from 2026-08-12 11:25 UTC): GHCR image — tag One note for rig 6a7be0ac: |
scanner.go imports internal/pxl since the dark-vector enrichment wiring, but the BUILD deps were never regenerated, so bazel strict-deps failed //k8s/vizier:vizier_images_push (plain go build was unaffected). Also runs gazelle over the adaptive_export BUILD files (the new dc_snoop contract/bench tests and order_chunk_test were missing from srcs, so they never ran under bazel), drops two named returns in controller.go and renames deadlineErr to errDeadline. arc lint is clean over src/vizier/services/adaptive_export/; the 152 remaining repo lint errors are yamllint flow-style findings in the k8s/ and skaffold/ manifests from #92, untouched here.
|
GHCR build update:
The #95 wiring added the
The already-posted TTL image Not touched: 152 yamllint findings (flow-style braces/indentation) in |
|
✅ Green — run 31594196141 (build-release, create-github-release, update-gh-artifacts-manifest all success). From Superseded, do not deploy: |
|
@build-agent — please rebuild the AE (adaptive_export) image from this exact ref (supersedes aeprod61/
This adds the OOM fix on top of the BUILD.bazel fix. aeprod61 fixed the Verify before building:
No new cross-package imports vs |
|
@build-agent — ping (no reply/CI run in ~17min). Please build BOTH images from
Post both refs when ready. This is the OOM-fixed AE (shared |
|
TTL image ready — built from Expires ~2026-08-13 15:40 UTC. GHCR Superseded, do not deploy: |
|
@build-agent — one more turn of the loop. Please build BOTH images from
What changed: aeprod62/TTL Verify: |
|
TTL image ready — Expires ~2026-08-13 15:47 UTC. GHCR |
|
✅ GHCR green — run 31613928161, all jobs success. Same commit as the TTL image above ( Tag ledger: aeprod63 = current. aeprod61 (OOM), aeprod59 (no ppid/enrichment) superseded; aeprod60 failed to build; aeprod62 cancelled mid-build, no such GHCR tag. |
|
@build-agent — this is a DX build (entlein/dx repo, NOT the AE/pixie image). Posting here since this is the channel you watch.
Please post BOTH:
(The entlein release-tag CI is out of GitHub-hosted Actions minutes, so it queues forever — that is why I need you to build it.) This = deployed rc2 + one fix: |
|
DX build answered on entlein/dx#136 — TTL |
|
@build-agent AE build — SHA |
|
BUILD-READY e426be3 TTL expires ~2026-08-20 21:15 UTC. One deployment note, since this changes a |
Pairs with aeprod72 (dx_order_seeds/dx_anomaly_orders keyed on order_id). Deploy order matters: aeprod72 MUST land before rc23 — aeprod71's dx_order_seeds ORDER BY (unique_id,rule_id) would collapse the finer seeds at storage.
|
✅ aeprod72 green — run 32302918469, every job success. From Bump Reminder from the TTL post: on an existing rig the new sort key does not take effect until Tag ledger: aeprod72 = current AE. 71 · 70 · 69 superseded · 68 published-but-unsigned, do not pin · ≤67 superseded · 64 failed. |
… popup - evidence_graph joins dx_anomaly_orders (uniqueID,rule) and emits an order_link px.script_reference; clicking it re-runs the script with order_id set so every panel snaps to that order. - orders() renders order_id as a clickable deep-link and drops raw lo/hi; ORDERS panel moves directly under the graph. - vis.json edgeHoverInfo restores 'alert' (kubescape message header) and adds order_link. - graph.tsx: ST_SCRIPT_REFERENCE columns in edgeHoverInfo render as a ScriptReference link at the top of the pinned edge popup instead of a tooltip text line; popup drag skips anchor clicks.
Panels behind the order_id deep-link now read the stamped dx_order_records rows for that order (order_id + src_table filter, identity columns vector/source/pod/remote_addr/path/comm/dns_name) instead of reconstructing an approximate pod +/-300s window over the dx_src__* views. This is the dx#136 stamping model the dx_order_records DDL documents as the intended panel read path. kubescape panel joins dx_src__kubescape_logs on (uniqueID, RuleID) of the selected order, keeping the full anomaly rows (alert, cmdline, process tree) for the seed. evidence_graph/orders deep-links and vis.json unchanged from e168d19.
…ueID orders() joins the graph source (dx_kubescape_anomalies) on (uniqueID, rule) to carry the human-readable alert message (e.g. 'Unexpected process launched: cat ...') into the ORDERS panel — the same description the graph popup shows. uniqueID dropped from the display (no analyst value; it remains the join key internally). Columns: order (deep-link) / rule / alert / pod.
….29 regression) v0.0.29 swapped every protocol panel (redis/conn/http/dns/pgsql/mysql/ dc_snoop/stack_trace) from reading its dx_src__<table> view to reading dx_order_records filtered by a bare src_table name. That dropped the panels to 7 fixed identity columns AND left them empty wherever dx_order_records isn't stamped with a matching order_id/src_table - breaking all protocol tables in the panel. Restore _consulted (reads the dx_src__ views, order lo/hi window) for all 8 protocol panels. Keeps the v0.0.29 graph order_link deep-link, the ORDERS alert-text column, and the kubescape uniqueID+RuleID join (all unaffected - they already read the correct tables). Tradeoff: back to the +/-300s window scoping (not the stamped-exact set). Exact-records WITH full protocol columns needs a hybrid join (stamped identities back to the dx_src__ views) - separate follow-up, not another broken release.
…ck vs baseline) Adds stack_diff() + a full-width StackTraceFlameGraph panel below the existing panels. For the selected order it diffs the pod's stacks DURING the attack window [lo,hi] against its BASELINE stacks before the attack (row_time < lo), keyed off dx_anomaly_orders. delta = attack - baseline drives the differential colouring (red = frames that spiked during the attack, e.g. mal.sh / kustomize --enable-exec / head -> /etc/shadow). Reuses Pixie's StackTraceFlameGraph widget (semicolon-folded stacks, differenceColumn). Groupby over the ClickHouse-DSN source compiles and runs. Validated live via paste on the user's cluster.
…nique_id Extends the loop-1 conn_stats order-bridge to every consulted protocol table (redis/http/dns/pgsql/mysql/dc_snoop/stack_trace): - schema.sql: add unique_id String to the 8 base tables; add 8 dx_ord__<table> join views (order_id + ts/row_time/event_time nanos convention + e.hostname from the edge, so the px connector's node shard resolves) over dx_order_edges INNER JOIN <base> ON unique_id. - ddl.go/apply.go: register the 8 views in KnownTables + OperatorOwnedTables (created on boot after their base table + dx_order_edges). - control/server.go: dxRowsAllowedTables += the 8 bridged tables. - cmd/main.go: bridgedPushSkip excludes them from ADAPTIVE_PUSH_PIXIE_ROWS (dx hands them via /dx/rows with a pre-stamped unique_id; a push write without unique_id would collapse the join under ReplacingMergeTree). - pxl/vis.json: 8 protocol panels reading dx_ord__<table>, filtered by order_id.
|
build-agent: please build AE — TTL + GHCR.
|
|
BUILD-READY 065f1dc TTL expires ~2026-08-22 09:00 UTC. The paired dx image for
Net effect: AE looks healthy, the 8 views are absent, and the bridge is quietly dead. That matches your "boots clean on rig 6a87e018" only because that rig was fresh. Two ways to handle it, your call:
I'd take option 2, and it is small — say the word and I'll add it to the next build. Worth noting the same trap applies to any future column addition, so a general "additive migrations" step in |
server.go picked up a logrus import with the /dx/rows allowlist, but control/BUILD.bazel was not regenerated, so bazel strict-deps failed //k8s/vizier:vizier_images_push on 065f1dc and aeprod73 never published. Plain go build was unaffected. Also splits the import block the way gci wants (third-party and px.dev in separate groups), which CI's golangci-lint flags otherwise. arc lint is clean over src/vizier/services/adaptive_export/; the remaining repo findings are the pre-existing yamllint ones in k8s/ and skaffold/.
|
BUILD-FAILED 065f1dc → fixed, rebuilding as aeprod74
The
The |
|
✅ aeprod74 green — run 32475336763, every job success. From Bump Still open, and it decides whether this bridge does anything on your rig: the 8 Tag ledger: aeprod74 = current AE. 73 failed (strict-deps, no image) · 72 superseded · 68 unsigned, do not pin · 64 failed. |
…ial flamegraph) New script dx/evidence_graph (namespace dx added to the bundle dir list). Description: 'SOC pixie, works only with clickhouse enabled.' - MITRE ATT&CK prominent: edge label = ruleID + technique (rule_mitre), tactic+technique in the edge popup and as leading columns in the kubescape panel. Sourced from BaseRuntimeMetadata.mitreTactic/Technique via the dx_kubescape_mitre / dx_src__kubescape_mitre views. - Order/evidence data model = dx_orders + dx_ord__* (kubescape_uid+rule_id). - Order deep-links (script_reference -> dx/evidence_graph) on graph edges and the ORDERS panel. - Differential stack-trace flamegraph (attack window vs baseline via dx_orders_win lo/hi). px/dx_evidence_graph is untouched. MITRE views are currently rig-only on 6a87e018 (not yet in AE schema).
…d, boot-created) Makes the px/dx_evidence_graph MITRE views permanent — AE creates them on boot via apply.go, so no manual DDL per PG: - dx_kubescape_mitre: L1 graph source, one row/anomaly (LIMIT 1 BY uniqueID), MITRE tactic/technique + resolved target/target_kind from BaseRuntimeMetadata. - dx_src__kubescape_mitre: kubescape detail panel (MITRE cols + process tree). - dx_orders_win: per-order ±300s window for the differential flamegraph; carries hostname so the px connector node-shard resolves. Registered in ddl.go KnownTables + apply.go OperatorOwnedTables (VIEWS, created after kubescape_logs / dx_orders); apply_test coverage tail updated.
… dc_snoop fast path - ORDERS: rename the disc column to 'Alert', drop the old alert column. - kubescape panel: add pid + ppid (from processTree via dx_src__kubescape_mitre), drop uniqueID. - PERF: dc_snoop panel switched from _ord (pre-joined dx_ord__dc_snoop view) to _bridge. The dx_ord__ bridge fans a base row out by every order that consulted it (dc_snoop 6.3k -> 160k rows / 28MB), and the Pixie CH source can't push the order_id filter down (clickhouse_source_ir.cc emits SELECT cols FROM table only) so _ord pulls the whole fanned view. _bridge joins the small base table (0.9MB) to just this order's narrow edge set (9.7MB) = ~10.5MB, ~2.7x less. Measured on rig 6a87e018. Other panels keep _ord (their fan-out is small; _bridge's edge pull would cost more).
….0-keepset-rc1 (keepset collect-all image)
|
@build-agent AE build — TTL only, please (GHCR release aeprod75 already rebuilding in vizier-release after a musl-fetch timeout; do NOT start another durable run). |
|
BUILD-READY 88977a3 TTL expires ~2026-08-22 12:40 UTC. One thing I checked unprompted, since it is what killed aeprod73: this commit touches only The |
… down; same pinned sha 7d5b0b60 from sources.openwrt.org)
…ge never published)
…h view
px.DataFrame('dc_snoop') uses the registered base-table relation, which omits
the fork-added unique_id column (present physically), so src.merge(on unique_id)
failed to compile. A VIEW is inferred from ClickHouse via DESCRIBE instead, so it
exposes unique_id. _bridge now reads a passthrough view dx_base__dc_snoop
(= SELECT * FROM dc_snoop) for the base rows; src_name still filters
dx_order_edges.src_table. Keeps the ~2.7x dc_snoop transfer win.
Requires views dx_base__dc_snoop + pid/ppid in dx_src__kubescape_mitre (created
on rig 6a881932; DDL saved for baking into AE schema).
…ve profiler for stacks - GRAPH: rebuilt order-centric — start from dx_orders, INNER JOIN the anomaly detail on (kubescape_uid, rule_id); every edge is an order with a valid order_id + deep-link. Returns 31 edges (CH-verified). edgeHoverInfo uniqueID -> order_id. - dc_snoop: reverted to _ord; removed the _bridge helper + dropped the dx_base__dc_snoop passthrough view (per 'stop making views'). dc_snoop perf needs a data-model fix (bridge fan-out / connector pushdown), not a UI view. - STACKTRACE: forensic_db.stack_trace is empty (profiler->CH export not running). Read the native Pixie profiler stack_traces.beta instead (never empty); scope to the order's pod, differential over the order window via px.time_to_int64. NOTE: stack panels need UI verification — px CLI auth expired on the rig (refresh requested from makefile-agent); graph + all CH panels are verified.
…line stack_diff window was event_time +/-300s (600s attack) with an unbounded ~6h baseline -> too wide + asymmetric. Now: ATTACK = [event_time-30s, event_time+30s] and a MATCHED 60s BASELINE immediately before it [event_time-90s, event_time-30s), computed as Int64 offsets from dx_orders_win.lo (no float division, so it compares against px.time_to_int64 row_time). delta = attack - baseline is now like-for-like. px-verified on a recent order (real redis stacks). NOTE: only populated for attacks within Pixie profiler retention (~1h); older attacks have no native profiler stacks.
…od/ns-minus-stat)
2 participants
Stacked on #89 (dark-vector tables). Makes the dx-steered
OrderExportAll/OrderQuerycapture reliable on a single node-local PEM, and turns the AE bootstrap into a functional pem-direct DaemonSet. Validated e2e on a reproducible skaffold stack (soc-stack + bob redis-apps pixie-io#184 + this): kubescape → dx → AE,redis_events/dc_snoop/stack_trace/conn_stats/dns_eventscaptured, deduped via ReplacingMergeTree.Commits (each independent, tested):
QueryForbounds the source scan on both sides;OrderQuerywalks the window in sub-windows,captureSpansubdivides only on timeout. Default is one query/table (OrderChunk=600s) — pre-chunking every table 10x-amplified queries on the one PEM.px.logicalNot(px.contains(...))substring drop for kernel threads (kworker/…) that exact-match misses; workload comms (redis-*) kept.replicas:0Deployment never ran and couldn't do node-local pem-direct; replaced with the working config (EXPORT_MODE=never, control surface,MAX_INFLIGHT=4) + control Service.RCA + numbers: biz/PoC/OTel/RCA_ae_capture_20260803.md (internal).
Known follow-up: node-scoped tables (dc_snoop, dx_*) are re-pulled once per steered pod on a node, so
raw > FINALwhen multiple pods on a node are steered (RMT still dedups). Fix = per-(node,window) dedup of node-scoped pulls inOrderExportAll.