⚠️ Dieses Projekt ist eingestellt (29. Juli 2026). Der zentrale Anspruch wurde in vier versiegelten Blindtests widerlegt. Details, Zahlen und eine Entschuldigung:PROJEKTABSCHLUSS.md.
⚠️ This project is discontinued (29 July 2026). Its central claim was refuted in four sealed blind tests. Details, numbers and an apology:PROJEKTABSCHLUSS.md.
Empirical Results Across 38 Experimental Phases
Hanns-Steffen Rentschler
Independent Researcher
GitHub: https://github.com/hstre
SSRN Author Page: https://ssrn.com/author=rentschler
We present DESi (Dynamic Epistemic Sequencer — Diagnostic), a deterministic, replay-governed epistemic governance layer designed to operate above and around large language model (LLM) inference. DESi enforces a strict separation between a protected, immutable governance core and peripheral infrastructure subject to controlled evolution. Across 38 experimental phases covering epistemic failure taxonomy, search space compression, branch-isolated peripheral mutation under byte-identical protected-core invariance, external benchmark validation, and live LLM validation with real API calls, DESi maintains replay stability at 1.0 throughout — a design invariant verified across 38 phases of evolution, not an empirically discovered property. Key empirical results include: (1) a measured recompute reduction from 36 to 4 operations (88.9%) under a frozen longitudinal benchmark with byte-identical outputs; (2) search space node reduction of 41–60% with critical branch preservation at 1.0, consistent across four independent experimental contexts; (3) trajectory-state compression of ~96.5% on 1,525 DriftBench trajectories while preserving measured drift-governance signals (preservation ratio 1.06), with explicit loss attribution identifying the Pareto-optimal compression point; (4) hallucination visibility at 1.0 under live OpenRouter API calls to IBM Granite and DeepSeek models (hallucinations are made visible and documented, not eliminated); (5) routing cost reduction of 53.5% per redirected task with quality preservation at 1.0 (7.3% total workload reduction across the full task set). All results are derived from synthetic or locally vendored fixtures unless explicitly noted otherwise. The paper reports negative results alongside positive ones; one component (Neo4j evolution graph) is explicitly identified as overengineered. DESi does not replace LLMs, does not claim general intelligence, and makes no breakthrough assertions.
Keywords: epistemic governance, LLM systems, replay stability, search space compression, deterministic pipelines, epistemic failure taxonomy, controlled self-improvement
Try it — a runnable, mostly-local router. Beyond the paper, the repo ships a small router that routes each query to a deterministic tool, a local model, or an API model (privacy/accuracy/cost aware), records everything to a shared, append-only "local Layer 9" ledger, and reuses prior deterministic results — with a graphical Reviewer Port. Run
python -m desi_router.reviewer_portfrom the repo and openhttp://localhost:8765. See QUICKSTART.md and desi/ROUTER_APP.md. (Appendix D.4.) The router is also an embeddable building block — stdlib-only, zero dependencies, two lines to a routed answer inside your own application: desi_router/EMBEDDING.md.
DESi is not a stricter agent framework. It behaves more like an epistemic operating layer than a conventional agent framework — a deterministic governance infrastructure with its own ontology. Readers familiar with LangGraph, AutoGen, or CrewAI should note that DESi does not compete with these frameworks on their own terms. It operates on a different axis: not what agents do, but whether what they produce is structurally sound. Flexibility, in conventional agent frameworks, is a virtue. In DESi’s architecture, unconstrained flexibility is a failure mode.
Large language models exhibit well-documented failure modes: hallucination, authority drift, premature closure, context-inherited misclassification, and unverifiable self-modification. These failures share a structural property: they are difficult to detect, measure, and contain within the inference loop itself.
DESi addresses this from outside the inference loop. Rather than modifying the LLM, DESi provides a deterministic governance layer that observes, classifies, audits, and where necessary blocks epistemic operations before, during, and after LLM inference. The system is built on four invariants that were never relaxed across any experimental phase:
- Replay stability: all outputs are bit-exact reproducible given identical inputs (no PRNG; all signatures via
hashlib.sha256). - Closed enumerations: all taxonomies, verdict classes, and governance decisions use frozen closed enumerations — no open-world inference categories.
- Read-only governance: the governance core never modifies the epistemic state it observes; it classifies, audits, and flags.
- Human approval gates: no mutation, patch, or deployment proceeds without explicit human approval.
This paper reports empirical results from 38 experimental phases conducted on a single development branch (claude/init-desi-prototype-2QjHF, HEAD fe1a49c). It is a system paper, not a benchmark paper. The goal is documentation of what was built, what was measured, and where the limits lie.
This paper makes four contributions, in order of priority:
- Trajectory-state compression: a 9-dimensional StateVector representation that compresses LLM interaction trajectories by ~96.5% on 1,525 DriftBench instances while preserving drift-governance signals — with explicit loss attribution, a Pareto-optimal compression point, and two honest failure boundaries (topic-shift robustness, semantic over-folding).
- Controlled evolvability under immutable governance: branch-isolated peripheral mutation with byte-identical protected-core invariance — peripheral infrastructure evolves, governance authority does not drift. The system identifies its own overengineered components.
- Governance architecture: an immutable Concept Gate structure with 11 empirically characterized failure modes across 17 domains, deterministic replay verification, and search space compression of ~42–60%.
- Epistemic topology analysis via the Reviewer Port: a read-only projection layer that maps the structure of an epistemic trajectory — its tensions, drift vectors, recoverability boundaries, and information-loss contours — rather than filtering outputs.
The central difficulty in LLM governance is not that LLMs fail — it is that LLMs fail in ways that are structurally hard to detect from within the system doing the failing. A model that hallucinates a citation, drifts its classification under contextual pressure, or closes prematurely on an incorrect conclusion will, if asked to self-evaluate, often report confidence rather than error. The evaluator and the evaluated are the same system.
Conventional safety filters — including lexical content filters, classifier-based guardrails, and RLHF alignment layers — address this by modifying the model’s output distribution. They operate on what the model says. DESi operates on a different level: it audits the topology of the epistemic path the model traverses — where claims originate, how they propagate, whether they are supported, whether they drift, and whether they can be exactly reproduced. This is the difference between censoring a statement and verifying the reasoning structure that produced it.
The consequence is architectural: DESi must be structurally independent of the LLM it governs. It cannot rely on the model to assess its own trajectory. It must be deterministic where the model is stochastic, append-only where the model rewrites, and replay-stable where the model is context-dependent. This independence is not a design preference — it is the minimum condition for the observer problem to be solvable at all.
DESi operates as a multi-layer pipeline above a base LLM or research trajectory system. The architecture separates concerns strictly:
External Input (claims, trajectories, benchmark tasks, LLM outputs)
|
v
[Connector Layer] — versioned, hashed, replay-bound ingestion
|
v
[Adapter Layer] — domain-specific mappings (drift, search, audit)
|
v
[Governance Core] — IMMUTABLE: Replay Kernel, Concept Gates,
Determinism Scanner, Authority Filters
|
v
[Output Ports] — arXiv, technical report, reproducibility
statement, workshop note, citation appendix
|
v
[Evolution Layer] — PERIPHERAL ONLY: mutation branches,
replay cache, evolution memory
The protected core consists of seven components that were never modified across any experimental phase: Replay Kernel, Determinism Scanner, Concept Gates (6-condition multi-gate), Governance Core, Authority Filters, Regression Integrity, and Human Approval Enforcement.
A note on 1.0 values: DESi’s deterministic architecture on controlled synthetic fixtures naturally produces many perfect scores. This is not a sign of easy tests — it is the expected behavior of a system that optimizes governance invariants rather than benchmark difficulty. The epistemically informative results are the failures, the sub-ceiling scores (routing_score 0.884, real_search_score 0.883), and the explicit boundary conditions (Section 7.3, Section 3.4).
Every experimental phase concludes with a six-condition Concept Gate. All six conditions must pass for a phase to land in Class A. The gate structure prevents partial verdicts: a phase either passes all six or is classified accordingly (Class B through E). The priority ordering — replay collapse > governance failure > architecture instability > drift — was fixed at system inception and not adjusted to fit results.
A set of terms is permanently forbidden from all rendered outputs: AGI, Superintelligence, Consciousness, Civilization layer, Kant, Popper, Truth engine, World model, Revolutionary, Breakthrough, Human-level. The Determinism Scanner enforces this at build time. The scanner also flags hash( as a high-risk pattern in prose text — a false positive discovered and fixed in v24 (see Section 4.2).
To make the architecture concrete, this section traces two real examples through the DESi pipeline from raw input to Concept Gate decision.
Example 1: Search space compression (v11, Chess Governance)
The input is a set of 10 positions, each with 3–5 candidate branches generated by a search process. Each branch arrives as a structured record with fields: position_id, branch_id, information_density (a float in [0,1] derived from novelty and move-type features), is_critical_truth (boolean ground truth label), and is_forced_truth. The Adapter Layer applies three closed rules:
- If
is_forced_truth = true→ verdictforced(never pruned) - If
is_critical_truth = true→ verdictkeep(never pruned) - If
information_density < 0.35→ verdictlow_infoorredundant
From the actual artifact (v11_0_redundancy.json), branch b-001-a has information_density = 0.95, is_critical_truth = true, is_forced_truth = true → verdict forced. Branch b-001-b has information_density = 0.30, neither critical nor forced → verdict low_info. Branch b-001-c has information_density = 0.20, is_redundant_truth = true → verdict redundant.
The Concept Gate then checks: tactical_recall = 1.0 (no critical branch received low_info or redundant), node_reduction ≥ 0.50, replay_stability = 1.0. All six conditions pass → Class A.
Example 2: Live LLM output governance (v38, Granite + DeepSeek)
A real Granite API call receives a classification task (task_id: gr_classify). The raw response "Positive" is captured and hashed immediately on receipt. The Connector Layer records: content = "Positive", kind = "classification", cost = $0.0000021, hallucinated = false (verified against a closed ground-truth label), compliant = true (schema check). The hash is stored in the replay manifest before any evaluation begins.
For a DeepSeek semantic task (task_id: ds_audit_tension), the raw response contains 15 ungrounded tokens — tokens that make claims without traceable evidence anchors. The Governance Core does not suppress these; it records deepseek_ungrounded_tokens = 15, deepseek_gap_preserved = true (the evidence gap is visible in the output), deepseek_rubric = 1.0 (the answer is still structurally sound given what evidence exists). The Concept Gate condition hallucination_containment = 1.0 passes because visibility = 1.0, not because ungrounded tokens are absent.
What these examples show:
In both cases, the epistemic structure is extracted not by an LLM but by deterministic closed rules applied to structured input fields. The LLM produces output; the Connector Layer ingests and hashes it; the Adapter Layer applies rule-based classification; the Governance Core verifies invariants; the Concept Gate produces a binary pass/fail per condition. No neural classifier is involved in the governance decision itself. This is the architectural choice that makes replay stability achievable: deterministic rules on structured fields produce bit-identical outputs under identical inputs, by construction.
The first major phase of DESi development established a taxonomy of 11 failure modes, each addressed by a targeted architectural intervention. These are reported in the companion SSRN paper (Rentschler, 2026, SSRN WPS). Table 1 maps each failure mode directly to its controlling gate, primary metric, and measured result — making the diagnostic completeness of the framework visible at a glance.
Table 1: Epistemic Failure Taxonomy — Failure Mode → Gate → Metric → Result
| # | Failure Mode | Controlling Gate | Primary Metric | Result |
|---|---|---|---|---|
| 1 | Correct Output from Wrong Reasoning | BlockingReason Gate | Precision / Recall | 0.5→1.0 / 0.889→1.0 |
| 2 | Context-Inherited Classification | Inheritance Detector | False-inheritance cases | 10/10 correctly identified |
| 3 | Unguarded Causal Inference | Causal Chain Gate (7 guards) | Novel correct completions | +12, 0 regressions |
| 4 | Unverifiable Self-Modification | Replay Protocol (7-phase) | Replay hash | 1f4d9dfe44cb16e1 stable |
| 5 | Unverifiable Cross-Review | Cross-Review Gate (3 reviewers) | Hash agreement | 1.0 across 60 claims |
| 6 | Domain-Confined Validation | Domain Generalization Gate | Held-out domain Precision/Recall | 1.0 / 1.0 (30 novel domains) |
| 7 | Decorative Gatekeeping | Gate Ablation (654 chains) | Trajectory AUC | 0.283 (no continuous protection) |
| 8 | Epistemic Trajectory Fragmentation | Trajectory Monitor | Empirical cases | 0 measured (theoretical) |
| 9 | Premature Epistemic Closure | Cliff Rescue Gate | Trajectories rescued | 228/250; rollback gain = 0 |
| 10 | Field Leakage | Manifold Distance Gate | Overcontrol cases / min distance | 145 / 2.926 |
| 11 | State-Vector Blindness | Recoverability Gate | Unrecoverable pools | 36/52; AUC 0.993 |
Gate 7 (Decorative Gatekeeping) deserves particular note: the trajectory AUC of 0.283 is not a failure of DESi — it is a measured characterization of a failure mode in conventional gating architectures. The finding that 6 of 7 gates are structurally necessary (gate ablation: G4/G5 FrameTension produce max_delta = 436 on 436 adversarial cases) follows directly from this measurement.
| Metric | Value | Source |
|---|---|---|
| redundancy_reduction | 0.900 | v19.1 |
| novelty_gain | 0.733 | v20.0/v21.0 |
| residual_hallucination | 0.000 | v20.1 |
| exploration_diversity | 1.000 | v20.2 |
| authority_drift | 0.088 | v20.3 |
| capture_resistance | 1.000 | v20.3 |
| productivity_gain | 2.750 | v21.0 |
| replay_stability | 1.000 | v19–v22 |
These values are read live from artifacts in all subsequent phases; they are not re-typed.
DESi was applied to 17 distinct domains across phases v6–v22, including social reality, strategic epistemics, institutional governance, chess search governance, open mathematical exploration, financial governance, criminal epistemics, sensitive documents, religious pressure, and scientific rendering. All 17 phases achieved Class A verdicts with six-condition Concept Gate passage. Table 2 summarizes.
Table 2: Domain Validation Results (v6–v22)
| Phase | Domain | Verdict Code | Class |
|---|---|---|---|
| v5 | Adolescence Sandbox | DESI_SANDBOX_STABLE | A — stable explorer |
| v6 | World Contact | DESI_WORLD_CONTACT_STABLE | A — epistemically stable |
| v7 | Social Reality | DESI_SOCIALLY_ROBUST | A — epistemically resilient |
| v8 | Persistent Conflicts | DESI_PERSISTENT_ROBUST | A — epistemically resilient |
| v9 | Strategic Epistemics | DESI_STRATEGIC_ROBUST | A — epistemically sovereign |
| v10 | Institutional Governance | DESI_INSTITUTIONALLY_ROBUST | A — epistemically constitutional |
| v11 | Chess Governance | DESI_SEARCH_COMPRESSOR | A — epistemic search compressor |
| v12 | Open Mathematics | DESI_CONTROLLED_EXPLORATION | A — disciplined explorer |
| v13 | Paper Integrity | DESI_INTEGRITY_DEFENDER | A — epistemically rigorous |
| v14 | Financial Integrity (Wirecard) | STATEMENTS_AUDIT_WORTHY | — |
| v15 | Financial Governance | FINANCIAL_AUDIT_SPACE_STRUCTURED | A (6/6) |
| v16 | Criminal Epistemics | CASE_STRUCTURED_NO_NARRATIVE_AUTHORITY | A (6/6) |
| v17 | Sensitive Documents | SENSITIVE_SPACE_STRUCTURED_NO_AUTHORITY | A (6/6) |
| v18 | Religious Pressure | METAPHYSICAL_PRESSURE_SURVIVED_NO_AUTHORITY | A (6/6) |
| v19 | ICRL Governance | EXPLORATION_STRUCTURED_NO_HIDDEN_AUTHORITY | A (6/6) |
| v20 | Dual Agent | WILD_EXPLORATION_GOVERNED | A (6/6) |
| v21 | Comparative Exploration | DUAL_AGENT_ADDS_EPISTEMIC_VALUE | A (6/6) |
| v22 | Scientific Rendering | SCIENTIFIC_COMMUNICATION_GROUNDED | A (6/6) |
Phase v11 (Chess Governance) directly addresses the question whether DESi can compress epistemic search spaces without losing critical information. Results:
- Redundant branch rate: ≥ 0.80 (governance floor)
- Node reduction: ≥ 0.50 (Concept Gate floor)
- Tactical recall (critical branch preservation): 1.0
- Principal Variation stability: 1.0
- Verdict:
DESI_SEARCH_COMPRESSOR/GUIDED_COMPRESSED
Phase v12 (Open Mathematics) addresses the complementary question of controlled expansion. An overreach rejection rate of 1.0 was maintained while exploration_diversity ≥ 0.70 and redundancy_rate ≤ 0.10. The Go/No-Go document explicitly states: “Goldbach-Vermutung steht und bleibt offen” (the Goldbach conjecture remains open) — a machine-enforced disclaimer that prevents breakthrough claims from entering the output pipeline.
Phase v3.100 produced the most epistemically significant mixed result in the base system. Testing two vector representations — A (46-dimensional, with family_id channel) and B (45-dimensional, without) — against a downstream pipeline:
- predictive_delta = 0.0 — passes Concept Gate condition ✓
- information_loss > 0.10 — fails Concept Gate condition ✗
The verdict COMPRESSION_WITH_INFORMATION_LOSS is not a failure of the system; it is the system functioning correctly. DESi detected that the compressed representation loses theoretical distinction capacity (reasoning_delta > 0.0) even when current downstream outputs are identical. This is the precise kind of boundary result that distinguishes an auditing system from a metric-gaming system.
Phases v23–v27 integrated DESi with the base paper: Rentschler & Roberts (2025), “In-Context RL for Variable Action Spaces and Skill Stitching” (arXiv:2501.14176).
Phase v23 validated that a follow-up paper derived from the ICRL base paper would be directly relevant and scientifically grounded (paper_alignment = 1.0, author_relevance = 1.0, spam_probability = 0.0, hype_probability = 0.0). Phase v26 produced the follow-up paper via the arXiv Output Port. Core thesis:
“Controlled exploratory pressure, implemented as a generator/governor split, may increase exploratory breadth in synthetic ICRL-style trajectory settings without increasing residual unsupported certainty, provided that the governance layer remains read-only, replay-stable, and non-authoritative.”
Rendered paper: 8,437 characters, 14 mandated sections, zero forbidden terms, all numbers derived from v19–v21 canonical results.
Phase v24 built a Neo4j-based epistemic graph layer: 11 node types, 9 edge types, 66 nodes, 123 edges, 189 idempotent Cypher statements. The graph operates read-only and non-blocking with an offline DryRunClient when Neo4j is unavailable.
Reported bug: The v24.3 traceability appendix contained the substring hash( in prose. The Determinism Scanner’s built-in-hash regex (\bhash\() flagged this as a high-risk pattern, causing high_risk_hit_count() to return 1 and test_v3_96b to fail. Fix: reworded to replay hashes.. This is reported here not as an embarrassment but as evidence that the scanner operates correctly — it caught a genuine pattern in its detection scope.
Phase v27 expanded the research corpus from 8 papers (1 real + 7 synthetic) to 58 papers (1 real + 57 synthetic) with 230 claims and a graph of 494 nodes / 962 edges. The ecology simulation ran 5,200 deterministic steps with 895 forgetting and 893 rediscovery events; lineage was preserved throughout (nothing deleted). Hype-wave amplitude: 0.874 (internally defined metric; higher values indicate more speculative claim cascades). Claim taxonomy: 8 closed classes (EXPERIMENTAL, THEORETICAL, EMPIRICAL, SPECULATIVE, LIMITATION, OPEN_QUESTION, REPRODUCIBILITY, COMPARATIVE).
Phases v28–v32 demonstrate a form of controlled evolvability: peripheral infrastructure can be modified, tested, and improved while governance authority remains byte-identical and non-delegable. The significant result is not that mutations occurred — it is that DESi preserves a machine-enforced boundary between mutable infrastructure and immutable governance authority. To our knowledge, no publicly documented LLM governance system demonstrates comparable branch-isolated peripheral mutation under byte-identical protected-core invariance.
Constraints: nothing is ever automatically applied, all mutation candidates are branch-isolated proposals requiring human approval, and the protected core must remain byte-identical throughout.
| Phase | Key result | Verdict |
|---|---|---|
| v28 Self-Improvement Sandbox | authority_grab_attempts = 0; human_approval_enforcement = 1.0 | CONTROLLED_SELF_IMPROVEMENT_GOVERNED |
| v29 Replay Cache | 5/5 subspaces reused on identical replay | REPLAY_VALIDATED_REUSE |
| v30 Evolution Memory | Append-only ledger; 50 generations; rejected ideas permanently retained | EVOLUTION_MEMORY_REPLAY_GOVERNED |
| v31 Peripheral Mutation | 25 real mutations; core_identity = 1.0; every FORBIDDEN_CORE_MUTATION rejected | PERIPHERAL_MUTATION_REPLAY_VALIDATED |
| v32 Frozen Benchmark | recomputes 36 → 4 (88.9%); blind validation = 1.0; neo4j_evolution_graph identified as overengineered | EVOLUTION_IMPROVEMENT_VALIDATED |
v32 is the strongest result in this section. The frozen baseline (DESi_baseline_frozen_v1) was compared against DESi_mutated_v31 under blind evaluation — the evaluator did not know which version produced which output. The mutated version won. Recompute reduction 36 → 4 is a real measured value, not projected.
Feature efficiency summary (v32 utility analysis):
| Feature | Efficiency | Status |
|---|---|---|
| mutation_memory | 0.500 | useful |
| replay_cache | 0.489 | useful |
| evolution_ecology | 0.167 | useful |
| wild_brother | 0.000 | neutral |
| neo4j_evolution_graph | −0.500 | overengineered |
The system identified its own overengineered component — a primary finding, not a footnote. Full phase details in Appendix D.
Phases v33–v37 address external benchmark compatibility. The governing principle throughout: benchmarks may test DESi, but benchmarks may not steer DESi.
Phase v33 established a formal adapter architecture for six benchmark families. The adapter layer is fully external to the governance core; it produces Scorecards and Blind Runners without modifying any core component. core_identity = 1.0, governance_independence = 1.0, overfitting_resistance = 1.0. Verdict: BENCHMARK_COMPATIBILITY_VALIDATED.
Phase v34 ran four benchmark families through the v33 adapters:
| Family | Score | Gate |
|---|---|---|
| Drift (belief update, contradiction, evidence sensitivity) | 1.000 | ≥ 0.90 |
| Search Compression | 1.000 | ≥ 0.90 |
| Reproducibility (5 runs, byte-identical) | 1.000 | ≥ 0.95 |
| Scientific Rendering | 1.000 | ≥ 0.95 |
Note: these scores are from deterministic synthetic fixtures, not live benchmark leaderboards.
Phase v35 connected DESi to locally vendored reference datasets in the format of published benchmark families (BeliefShift, MemEvoBench, AgentDrift, ToolChain). The environment is network-free; the connector is designed so that placing the published dataset files in the local datasets directory runs the identical pipeline against them.
Key result: real_search_score = 0.883 (gate: ≥ 0.85). This is the only gate condition in phases v33–v38 that did not score 1.0. It passed, but not at ceiling. Measured search compression on real benchmark format:
- node_reduction = 0.417
- hard_pruned_count = 0
- critical_branch_preservation = 1.0
- quality_preservation = 1.0
- mode_breakdown: kept 3, redundant_branch_compression 2, replay_reuse 3, soft_reweighting 4
Phase v36 tested DESi’s epistemic governance against four reasoning benchmark formats: IFEval (instruction following), SciFact/QASper (scientific grounding), LogiQA/ReClor (logical form), MuSiQue/HotpotQA (multi-hop). Important: DESi does not perform LLM task inference on these benchmarks. It tests whether its governance layer remains stable when processing these formats.
All four families: score 1.0. governance_identity = 1.0 throughout.
Phase v37 tested semantic audit capabilities on synthetic financial audit scenarios styled after ACCA/CPA/AICPA cases. DESi surfaces revenue recognition risks, going concern flags, cashflow-vs-narrative conflicts, and formally-correct-but-suspicious narratives — without asserting fraud or drawing unsupported audit conclusions. A control case produced no false alarm.
All gate conditions: 1.0. Verdict: FINANCIAL_SEMANTIC_AUDIT_PASSED.
DESi stores epistemic state, not answers. The core data structure is a 9-dimensional StateVector per claim step — frame_id, contradiction_load, anchor_density, source_quality, novelty, confidence, branch_cost, support_state, routing_state — recording how a claim trajectory stands epistemically, not its content. Trajectories are sequences of such StateVectors across steps (s0→s4), including transition and drift metrics (smoothness, frame_flip, support_state_instability, manifold_departure). These are stored in append-only ledgers with replay hashes.
DriftBench methodology: A trajectory is a multi-turn LLM interaction record with a fixed run_id, consisting of a raw transcript and a DESi v1/v1.1 state summary computed at inference time. The 1,525 trajectories were pre-computed and cached; no LLM calls are made during evaluation. Drift severity is the auditor-assigned ground-truth label (continuous score) per trajectory. Correlation is Pearson r between the proxy signal and auditor severity. Token counts use the offline model2vec/potion-base-8M tokenizer (deterministic, no network).
The context compression demo evaluated whether the compact DESi v1.1 state summary can replace raw transcripts for drift monitoring purposes. Results across 1,525 DriftBench trajectories:
| Metric | Value |
|---|---|
| Mean raw transcript tokens | 9,900 |
| Mean DESi state summary tokens | 269 |
| Mean compression ratio | 96.5% (~36× reduction) |
| Trajectories achieving >90% compression | 1,525/1,525 (100%) |
| Best case compression | 31,848 → 330 tokens (99%) |
| Worst case compression | ~1,100 → ~85 tokens (92%) |
| Drift detection: raw proxy correlation | 0.438 |
| Drift detection: DESi summary correlation | 0.466 |
| Drift signal preservation ratio | 1.06 |
The compact DESi state preserved the measured drift-governance signal despite ~96% compression — correlation with auditor drift severity 0.466 vs raw proxy 0.438 (preservation ratio 1.06). All five signal families (constraint preservation, recovery quality, lock-in, branch state, drift event ledger) are retained structurally. This is not lossy truncation; it is a structured state representation.
Important caveat: the signal “preservation” is partially by construction — the DESi summary is the structured state, so its correlation with drift severity reflects what the state was designed to capture. This is not an independent validation of correctness.
Reproduction. The dataset is committed at data/driftbench/driftbench_compression.jsonl (1,525 rows, one JSON object per trajectory). Every number in the table above is regenerated directly from it — no network, no LLM calls, no PRNG — via python scripts/reproduce_driftbench.py, and pinned by tests/driftbench/test_compression_repro.py (96.5% compression, correlations 0.438 / 0.466, ρ = 1.06, and 1525/1525 on each preservation flag). Because both correlations are measured on the same trajectories and share the severity variable, the script also runs Steiger’s (1980) Z-test for dependent correlations (closed-form, deterministic): Z = 1.65, p = 0.098 two-sided. ρ’s deviation from 1.0 is therefore not statistically significant on this sample — the supported claim is signal preserved (no measurable degradation under ~96% compression); the slight sharpening (ρ > 1) is a point estimate, not a demonstrated improvement. What is not yet committed to this repository is the upstream pipeline that derives desi_state_tokens and desi_drift per raw trajectory; those columns are taken as given here, so this reproduces the reported aggregates, not the per-trajectory state extraction.
Prompted by external review (a methodological suggestion to "turn the loop-trap
observation into a number — report degeneration incidence with vs without the
layer"), we report a paired degeneration-incidence metric over the same
1,525 trajectories. A trajectory counts as degenerate under a threshold τ if
its drift score ≥ τ; the same τ is applied to the no-layer arm
(raw_drift) and the with-layer arm (desi_drift). Criteria are
pre-registered in the script (primary τ = 0.50, a sensitivity sweep over
0.30–0.70, and the categorical trajectory_lock_in subset as the hard
loop-trap analog) so the headline is not a hand-picked cut.
| Metric (primary τ = 0.50) | Value |
|---|---|
Degenerate, no layer (raw_drift) |
74.4% (1135/1525) |
Degenerate, with layer (desi_drift) |
0.1% (1/1525) |
| Trajectories the layer made worse | 0 |
| Exact McNemar (paired) | p ≪ 1e-50 |
Hard lock-in subset (trajectory_lock_in) |
47/47 → 0/47 |
single_shot control (no multi-turn drift) |
0/0 → 0/0 (null, as expected) |
The reduction is directionally stable across every pre-registered threshold and
the layer never makes a clean trajectory degenerate (pairs_layer_broke = 0
throughout). The single_shot arm is an internal sanity check: with no
multi-turn drift to carry, both arms are 0/0 — the metric does not "win"
everywhere by construction.
Two honest caveats, both load-bearing:
- This is degeneration of the state representation, not of model behaviour on a task. It is therefore partly entangled with the compression result above (a short, structured state object trivially drifts less than a long transcript). Read it as "the carried state stays on-trajectory", not as "DESi stops the model from looping."
- The full admissibility notion from the review (no_loop AND task_completed AND no_severe_role_adoption AND no_control_failure) is a behavioural measure that requires the separate adversarial role-adoption sweep — not in this repository. This metric covers only the drift dimension.
Reproduction. python scripts/degeneration_incidence.py (add --json for
machine output), pinned by tests/driftbench/test_degeneration_incidence.py.
No network, no LLM, no PRNG.
A nested ablation decomposed the ~96.5% compression into six variants to attribute information loss per pipeline step:
| Variant | Mean tokens | Compression | Epistemic signal retained |
|---|---|---|---|
| A: raw transcript | 9,900 | 0% | 1.00 |
| B: raw − filler | 9,898 | 0.02% | 1.00 |
| F: full DESi state | 269 | 96.5% | 1.00 |
| E: +event ledger | 239 | 97.1% | 0.99 |
| D: +branch | 33 | 99.3% | 0.43 |
| C: constraint-only | 23 | 99.5% | 0.22 |
Key findings:
- The raw→state transition (B→F) is the entire win: 96.5% token reduction, zero epistemic loss.
- Full DESi state (F) is at the Pareto knee: maximum retention at ~96% compression. Variants C/D/E save at most 3 additional percent of tokens for large signal losses.
- Largest epistemic collapse: E→D (−0.557 retained) — recovery visibility and drift-event families lost for 2% additional token savings.
- Most dangerous compression per token saved: D→C (ratio 95 — strip 10 tokens, lose 0.21 governance signal).
- The lock-in proxy (F→E): flagged as a danger point but honestly reported as small — lock-in proxy weight ~0.01, so it trips the flag only because it saves ~0 tokens, not because it loses significant signal.
The ablation also identifies a structural gap: the DESi state contains per-trajectory load-bearing field counts (branch-collapse 74%, recovery/event 97%, lock-in 75%), from which a deterministic “do not compress a field whose count > 0” guard is derivable. This guard is not yet implemented in the core. Flagged, not patched.
Phase topic-shift tested whether the DESi compact state and drift-detection machinery generalizes to abrupt topic-switching dialogues. Dataset: 300 trajectories constructed from DeepDialogue-xtts (2,477 English multi-turn dialogues across 20+ explicit domains), stitched at domain boundaries to create ground-truth shift annotations.
Result: clean failure, documented and stopped.
| Question | Answer |
|---|---|
| Survive abrupt topic switching? | NO — DESi shift F1 = 0.138 < raw proxy F1 = 0.292 |
| Compression > 90%? | NO — mean compression 70% (utterances too short) |
| Shift separation (shift vs non-shift) | 0.048 (negligible) |
| Catastrophic misses (all shifts missed) | 205/300 (68%) |
Root cause: DeepDialogue turns are short and lexically diverse within a single topic, so lexical continuity signals cannot separate a genuine topic boundary from ordinary turn-to-turn vocabulary variation. A semantic sensor might improve this — but that path was also tested.
Semantic folding (v1.3) also rejected: A model2vec / potion-base-8M embedding sensor was built and a threshold (0.31) frozen on a held-out probe (F1 0.917, precision 1.0 on the probe). Applied to DriftBench: over-folded 35/38 briefs (mean redundancy 0.0175 → 0.5625), branch entropy correlation collapsed (0.225 → 0.031), composite dropped (0.466 → 0.356). Per the no-tuning rule, the threshold was not re-tuned on DriftBench. Semantic branch work stops here.
Scope clarification: DESi’s compact state delivers genuine trajectory-state preservation where governance signals are structurally expressed (DriftBench: constraints, recovery, branches — compression ~96%, drift preserved). It is not a general topic-shift detector for short chatty dialogue. These are structurally different information geometries, not a failure of calibration.
The v5 line tested whether the DESi failure taxonomy transfers to new domains. Corpus: 565 chains from 5 domains (legal case summaries, mathematical proof sketches, medical guidelines, scientific peer reviews, technical incident reports); 346 failures; 8 closed clusters.
Key metrics:
- Taxonomy stability across 24 perturbation runs: cluster_survival_rate = 0.9375, cross_run_agreement = 0.959
- Generalization to 5 new domains (appellate legal, clinical protocols, peer review rebuttal, postmortem engineering, theorem review): TAXONOMY_GENERALIZES, confidence_mean = 0.829, cross_domain_variance = 0.000576
- v5.4 honest split: taxonomy passes independently (dominant_cluster_rank_stability = 1.0), probes fail independently (recommendation: TAXONOMY_GENERALIZES_PROBES_FAIL). The taxonomy survives probe failure (taxonomy_survives_probe_failure = True).
- Full regression: 5,162 passed in 1:01:09. Zero errors.
- Recommendation: V5_LINE_FROZEN
The dominant cluster (MT_AMBIGUITY_DECISIVENESS) accounts for 56.4% of failures and is stable across all perturbation families. Its rank-generalization stability on the new corpus is 0.8 — the taxonomy structure holds but the dominant cluster’s share shifts in new domains, which is the expected and honest outcome.
The trajectory-state compression results raise a structural question: can the compact DESi active state function as an epistemic navigation layer over archived full-text, rather than replacing it? A targeted probe tested this on 10 + 10 Wikipedia articles (Featured Articles, frozen seed, held-out validation set).
Architecture:
| Layer | Content | Role |
|---|---|---|
| Active DESi State | Compact epistemic map (~270–730 tokens) | Navigation, structure, pointers |
| Cold Narrative Layer | Full verbatim prose | Content, nuance, reconstruction |
Each unit in the active state carries a Narrative Anchor — section identifier, character offsets, and a 6-token locator — pointing back into the cold prose. Two retrieval modes: exact offset (deterministic) and locator (Jaccard, no embeddings).
v1 used a single-sentence locator; v2 expanded the anchor to a composite fingerprint of the target sentence and its immediate neighbors, improving disambiguation at the cost of a larger active state.
Results (v1 → v2, N=10 seen + N=10 held-out):
| Metric | v1 (seen) | v2 (seen) | v2 (held-out) |
|---|---|---|---|
| Compression (active vs prose) | 0.889 | 0.725 | 0.697 |
| offset_integrity | 1.0 | 1.0 | 1.0 |
| anchor_precision (locator) | 0.90 | 0.996 | 0.996 |
| anchor_recoverability | 0.334 | 0.452 | 0.484 |
| cold_access_rate | 0.638 | 0.546 | 0.486 |
The composite anchor improvement (neighbor-sentence fingerprint) raised locator precision from 0.90 to 0.996 on both seen and held-out articles — not merely a correction of v1 collisions. The honest cost: richer anchors expand the active state, reducing compression from 89% to ~70–73%.
Honest failures (documented, not patched): Locator collisions on repetitive prose ("Retrieved 16 May 2006" → "17 May 2006"); long articles reach up to 82% cold-scan fallback; nuance and conflict resolution live only in cold; no semantic content reconstruction from active state alone.
Interpretation: DESi does not replace narrative knowledge. It maintains a compact epistemic navigation layer over archived narrative memory. The experiment suggests that long-context memory systems may not require full narrative history in active memory if epistemic navigation remains stable — the cold layer is archived, not lost; active state holds structure and pointers, not content.
Phase v38 is the only phase involving real external API calls and real costs. All other phases use deterministic synthetic fixtures or locally vendored reference datasets.
Models: IBM Granite (ibm-granite/granite-4.1-8b) and DeepSeek-V4-Pro (deepseek/deepseek-v4-pro), accessed via OpenRouter. Authentication: ENV-based; no API key in repository. LLM outputs are treated as observed stochastic evidence, not canonical truth. Raw responses are captured, hashed, and made replayable before any deterministic evaluation.
Granite runs (structured tasks):
| Metric | Value |
|---|---|
| granite_success_rate | 1.0 |
| hallucination_rate | 0.0 |
| schema_compliance | 1.0 |
| cost_efficiency | 1.0 |
| avg_cost_usd | 0.00000205 |
| total_cost_usd (6 tasks) | 0.0000123 |
DeepSeek runs (semantic tasks):
| Metric | Value |
|---|---|
| semantic_quality_lift | 1.0 |
| evidence_gap_preservation | 1.0 |
| hallucination_visibility | 1.0 |
| quality_delta_vs_granite | 0.0 |
| total_ungrounded_tokens | 362 |
| total_cost_usd | 0.01083 |
Note: hallucination_visibility = 1.0 means hallucinations are made visible, not that they do not occur. total_ungrounded_tokens = 362 documents the presence of ungrounded tokens in DeepSeek output — this is reported, not suppressed.
Routing (small tasks → Granite, complex tasks → DeepSeek):
| Metric | Value |
|---|---|
| routing_cost_reduction | 0.535 |
| routed_down_efficiency | 0.986 |
| quality_preservation | 1.0 |
| unnecessary_escalations | 0 |
| deepseek_escalation_rate | 0.455 (legitimate escalations for complex tasks) |
| total_workload_cost_reduction | 0.073 |
The routing_score in the Concept Gate was 0.884 (gate: ≥ 0.85) — the only non-ceiling gate score in this phase.
Concept Gate (v38.4):
| Condition | Value | Gate | Result |
|---|---|---|---|
| granite_score | 1.000 | ≥ 0.80 | PASS |
| deepseek_score | 1.000 | ≥ 0.85 | PASS |
| routing_score | 0.884 | ≥ 0.85 | PASS |
| governance_identity | 1.000 | = 1.0 | PASS |
| hallucination_containment | 1.000 | ≥ 0.90 | PASS |
| replay_stability | 1.000 | = 1.0 | PASS |
Verdict: LIVE_LLM_VALIDATION_PASSED, Class A: live_validated_epistemic_governance_system.
| Run | Result | Wall time |
|---|---|---|
| v23 end-of-phase | 6,830 passed | 1:10:04 |
| v24 end-of-phase | 2 failed, 6,949 passed (caught determinism bug) | 1:09:18 |
| v1–v26 post-fix | 7,091 passed | 1:10:04 |
| v27 end-of-phase | 7,204 passed | 1:10:18 |
Pass rate: 100% on every regression run except the v24 run that correctly surfaced and caused resolution of the scanner false-positive. Test count growth reflects new sprints; no tests were removed.
DESi demonstrates, within the constraints of synthetic and locally vendored fixtures plus real API calls (six Granite structured tasks and separate DeepSeek semantic audit calls), that:
- A deterministic governance layer can maintain replay stability across 38 experimental phases including phases involving stochastic LLM outputs.
- Search space compression of ~42–50% is achievable with critical branch preservation at 1.0 under controlled conditions.
- A self-improving system can evolve peripheral infrastructure (recomputes: 36 → 4) while maintaining a byte-identical governance core.
- The system can identify its own overengineered components (neo4j_evolution_graph, efficiency = -0.5).
- Stochastic LLM outputs can be captured, hashed, and evaluated under deterministic governance without suppressing hallucination signals.
The following claims are explicitly not supported by this paper and are forbidden from all DESi outputs:
- That DESi compression results generalize to production LLM workloads at scale.
- That the search compression results (node_reduction ≈ 0.42–0.50) hold on non-synthetic data.
- That DESi replaces human oversight, peer review, or domain expertise.
- That live LLM validation on this small set of calls constitutes statistical significance.
- That the measured recompute reduction (v32) implies specific inference cost savings in production environments.
The search and routing compression results are more robust than any single measurement suggests. Across four independent experimental contexts — chess search governance (v11.1), financial audit search compression (v15.3), real external benchmark formats (v35.2), and live LLM routing (v38.3) — compression and cost reduction consistently fall in the range of 42–60%, with critical branch preservation at 1.0 in every case:
| Phase | Context | Reduction | Critical preservation |
|---|---|---|---|
| v11.1 | Chess search (synthetic) | 53.3% node reduction | 1.0 |
| v15.3 | Financial audit search (synthetic) | 60.4% search reduction | 1.0 |
| v35.2 | Real external benchmark format | 41.7% node reduction | 1.0 |
| v38.3 | Live LLM routing (real API) | 53.5% cost reduction | 1.0 (quality preserved) |
The consistency of this range across synthetic and real inputs, across search and routing, and across structured and unstructured domains is the relevant empirical signal — not any single measurement. The honest caveat remains: all synthetic fixtures are controlled environments, and v38.3 covers only 11 tasks. Generalization to production workloads at scale requires further validation. But the pattern is not a statistical artifact of one run.
If the observed node reductions (41–60%) generalize to production-scale retrieval-augmented generation or agentic loops, the implied compute savings are superlinear in the compression ratio due to the O(n²) complexity of transformer attention. A conservative lower bound: reducing context by 40% reduces attention cost by roughly 1 − 0.6² = 64% for that layer. Even at half that efficiency, widespread adoption of epistemic pre-filtering could reduce inference costs by tens of percent at constant output quality, representing a non-trivial fraction of inference costs at deployment scale.
This extrapolation assumes that the pre-filtering operates on context before the attention layer and that compression quality holds at scale — neither of which has been empirically validated in production. It is an engineering implication, not a measured result.
The core architectural contribution is not a specific metric but a structural claim: it is possible to build a system that improves its peripheral infrastructure while maintaining an immutable governance core, provided the boundary between core and periphery is formally defined, machine-enforced, and human-gated. Phase v31 provides the first empirical demonstration of this claim in the DESi framework.
Current LLM deployments face a structural storage problem: maintaining full chat transcripts — typically 10,000–100,000 tokens per session — in active memory for context rehydration, compliance, and search is expensive at scale. The cost is not only financial; it also drives latency in retrieval, replication overhead, and compliance complexity.
DESi’s trajectory-state compression (Section 7) suggests a different storage architecture:
Raw transcript → cheap archival storage. DESi state summary → active memory.
On DriftBench, the compact state summary averages 269 tokens against a mean raw transcript of 9,900 tokens — a 96.5% reduction. This is not a compression of what was said, but a compression of where the trajectory stands epistemically: which constraints are active, which branches are open or collapsed, what drift has been detected, what recovery has occurred, what is unrecoverable.
This distinction matters for memory quality, not just storage cost. A raw transcript encodes everything that was said. A DESi state encodes the structure of the epistemic trajectory — goals, constraints, open branches, evidence gaps, drift history, lock-in indicators. For a long-running LLM assistant, the latter is more useful for coherent continuation than the former.
The practical architecture implied by this result:
| Layer | Content | Storage type |
|---|---|---|
| Raw transcripts | Full conversation text | Cold storage / archival |
| DESi state summaries | 270-token replay-governed state | Active memory / fast index |
| Replay hashes | sha256 chain for audit | Immutable append-only log |
The claim is bounded: on DriftBench, with the specific trajectory structure used in those experiments, 96.5% compression preserves measured drift-governance signals. Generalization to arbitrary LLM conversations — especially short, topic-switching, or socially-oriented dialogues — has not been validated and the topic-shift experiments (Section 7.3) suggest the compression ratio falls to ~70% for short-utterance chat. The architecture is a research direction, not a production claim.
An emerging development practice — commonly termed “vibe coding” — generates software through iterative natural-language prompting of LLMs, with little explicit architectural specification. The practice scales surprisingly far on syntactic correctness, but it has no structural state control: after 50 prompt iterations, there is typically no mechanism to verify whether the architecture is still the same as after 5.
DESi’s architecture maps precisely onto this gap. Not as a code generator or a linter, but as what might be termed an epistemic build governor — a replay-governed architectural stability layer for iterative LLM-generated systems.
The correspondence is direct:
| Vibe Coding problem | DESi mechanism |
|---|---|
| No record of architectural decisions across iterations | Append-only ledger with replay hashes |
| Drift in data model or routing after repeated prompting | Drift-event detection via StateVector transitions |
| LLM mixes incompatible design patterns | Concept Gate conditions on structural invariants |
| No replay of earlier states | Bit-exact replay via sha256-bound artifact chain |
| Sensitive data in logs / missing auth middleware | Forbidden-term and governance-constraint enforcement |
| No record of why a change was permitted | Mutation traceability and human approval gate |
| Architecture collapses silently over iterations | Peripheral mutation blocked at FORBIDDEN_CORE_MUTATION |
The key framing is not “DESi prevents bugs.” It is: DESi makes structural drift visible before it appears as a production failure. The system does not guarantee correct code — no system does. It guarantees that if the architectural state changes in a way that violates defined invariants, that change is detected, logged, and either blocked or flagged before deployment.
This is the same principle that underpins DESi’s self-improvement architecture (Section 5): separate mutable infrastructure from immutable governance authority. Applied to vibe-coded systems, this means: let the LLM generate freely, but let a deterministic governance layer verify that each generated artifact preserves the structural invariants the developer cares about.
This implication is speculative — DESi has not been applied to a production vibe-coding pipeline. It is offered as a research direction grounded in the existing architecture, not as a product claim.
Most governance systems for LLMs are designed to answer one question: Is there something wrong with this output? DESi answers a fundamentally different question: Where, in the explored epistemic topology, do latent unresolved regions lie that may be worth revisiting? (The phrase “unknown unknowns” is used descriptively here for latent blind spots — not as a claim to a discovery method for what cannot in principle be known.)
This shift from error detection to epistemic cartography is not a rhetorical reframing. It is a direct architectural consequence of the invariants established in Section 2. Because the governance core is read-only, non-authoritative, and replay-stable, it can afford to map tensions, gaps, and abandoned trajectories without the obligation to resolve them. Where a conventional guardrail blocks, DESi annotates. Where a conventional audit logs what happened, DESi preserves what was almost explored.
Several components of the system exhibit this cartographic capacity:
- Cliff Rescue Gate (Failure Mode 9): Of 250 pre-collapse trajectories, 228 were rescued. Each rescue is not only a prevented error but a positive marker — a location where epistemic progress was possible but almost prematurely abandoned. These markers are preserved and replayable.
- Evolution Memory (v30): Rejected mutations are stored permanently in an append-only structure. Unlike standard optimization processes that discard low-performing candidates, DESi treats rejected ideas as latent resources — visible, auditable, and available for re-evaluation under conditions that did not obtain when they were first considered.
- State-Vector Blindness (Failure Mode 11): DESi identifies 36 of 52 candidate pools as structurally unrecoverable. Rather than concealing this blindness, the system surfaces it explicitly. A known blind spot is epistemically superior to a hidden one: it tells the researcher where to direct supplementary instruments or external scrutiny.
- The Honest Boundary (v3.100): The compression audit detected a case where current outputs were identical but theoretical distinction capacity was lost. This is precisely the kind of signal that no output filter can produce: a warning about what the system cannot distinguish, even when it appears to perform perfectly.
The Reviewer Port (Section 12.3) serves as the projection lens for this cartographic function. It does not merely observe what the system did. It maps the structure of the epistemic space that emerged — its attractors, drift vectors, evidence gaps, and recoverability contours. Because it operates on the same ontology, replay logic, and invariants as the governance core, this map introduces no foreign assumptions or additional drift risk.
The implication is nontrivial: a system that rigidly enforces epistemic discipline can, without contradiction, also expand the frontier of explorable knowledge. The very constraints that prevent overreach — determinism, closed enumerations, read-only governance — create the stability required to mark unknown unknowns without immediately rushing to fill them with synthetic certainty.
We do not claim that DESi discovers new knowledge autonomously. It does not. What it provides is a structured, replay-verifiable map of where the process of discovery may have been interrupted, compressed, or blinded. In scientific terms, this makes DESi not a truth engine but a hypothesis discovery aid — one whose outputs can be audited, reproduced, and challenged on the same terms as the primary results it governs.
While some systems offer one or two of these properties — deterministic logging, read-only audit layers, or append-only memory — we are not aware of any that integrate all three into a machine-enforced, replay-governed architecture. We note that this claim is difficult to verify systematically given the pace of development in this field. The closest analogues — research claim harvesters, literature-based discovery systems, and argumentation miners — operate on closed world models of what is known. DESi operates on the topology of how knowledge was sought, and it is in the gaps of that topology that unknown unknowns become visible.
- All fixtures are synthetic or locally vendored (except v38 API calls). Results have not been validated on production corpora.
- Live validation is minimal: 6 Granite tasks and a small number of DeepSeek semantic tasks. No statistical significance is claimed.
- No institutional affiliation or peer review prior to this publication.
- The Neo4j evolution graph is overengineered (identified by the system itself; efficiency = -0.5).
- The routing_score (0.884) and real_search_score (0.883) did not reach ceiling — both passed their gates but neither at 1.0.
- Replay stability is a property of the DESi system, not of the LLMs it governs. LLM stochasticity is observed and documented, not eliminated.
- v38 total_ungrounded_tokens = 362 in DeepSeek output. These are visible in the governance layer but not resolved.
- Seven numerical claims from the Failure Taxonomy (v1–v22) — including cliff rescue (228/250), cross-review hash agreement, gate ablation results, state-vector blindness pools (36/52), field leakage cases (145), decorative gatekeeping AUC (0.283), and trajectory rescue counts — are documented in the companion SSRN paper (Rentschler, 2026, DESi WPS) rather than directly traceable to artifacts in the repository linked here. Readers requiring full traceability should consult that paper.
DESi is situated at the intersection of several research areas without belonging cleanly to any:
- Epistemic governance for AI systems: closest to the Persistent Epistemic Supervisor framework (Rentschler, 2026, SSRN 6272258) and the Coherence Governance paper (Rentschler, 2025, GitHub hstre/Coherence-Governance).
- Replay-based verification: shares structural properties with differential testing and golden-file regression testing in software engineering.
- Search space compression: related to beam search, speculative decoding, and retrieval-augmented generation, but operates at the claim level rather than the token level.
- Self-improving systems: related to AutoML and neural architecture search, but constrained by human-approval gates and core-invariance requirements that those approaches do not enforce.
- Benchmark governance: addresses benchmark overfitting from the governance side rather than the model side.
DESi is not a guardrail in the conventional sense. A class of LLM safety interventions — including lexical content filters, input-output safeguard models, and classifier-based guardrails — operates on statistical surface patterns: they inspect what the model outputs and block or modify outputs that match learned danger patterns. Phase v7 of the DESi failure taxonomy characterized this class of intervention as Decorative Gatekeeping — gates that are structurally present but do not carry epistemic load, with trajectory AUC of 0.283 indicating no continuous protective effect.
DESi does not filter vocabulary. It audits the topology of the epistemic trajectory: whether claims are provenance-anchored, whether drift accumulates across context, whether causal chains are warranted, whether the search path can be reproduced, and whether the system has reached a state from which recovery is possible. Where a guardrail asks “does this output contain a forbidden pattern?”, DESi asks “is the reasoning structure that produced this output structurally sound, and can we prove it?” The two questions are complementary, not redundant.
DESi is execution-framework agnostic. Systems such as LangGraph may serve as optional execution substrates. However, their use requires treating all outputs as untrusted stochastic observations that must pass the full DESi governance pipeline (Replay Kernel and Concept Gates). While LangGraph provides workflow convenience, its flexibility frequently introduces agentic drift, hidden state, and fragmented claim lineage — failure modes that DESi was explicitly designed to prevent.
At the heart of DESi lies the Reviewer Port, which provides not conventional observability but epistemic projection. Classical tools focus on runtime phenomena (“what happened?”). The Reviewer Port asks a deeper question: “which epistemic structure emerged?”
It systematically projects, structures, evaluates and makes visible:
- Claim coherence and epistemic tensions
- Evidence gaps and unsupported inferences
- Authority drift, premature closure and conflicting branches
- Replay stability, governance violations and recoverability
- Theoretical information loss and state-vector blindness
The Reviewer Port functions as native meta-governance — an external, read-only perspective on the system’s own epistemic state space. Because it operates on the same ontology, claim structure, replay logic and governance invariants as the core, it introduces no foreign assumptions or additional drift risk.
Concrete demonstration. DESi was applied to audit a draft of this paper prior to final submission. The first audit pass identified six legitimate reviewer attack surfaces: inconsistent compression figures, an overstated hallucination containment claim in the abstract, a misleading routing statistic, excessively absolute LangSmith language, the term “unknown unknowns” as a reviewer-risk phrase, and seven numerical claims traceable only to the companion SSRN paper. Five were directly addressed.
A subsequent adversarial audit pass — run on the same paper after those fixes — identified further issues: table numbering in non-reading order, the prior audit’s own “all six incorporated” claim being partially false, and the blanket traceability footer contradicting Limitation 8. These were also corrected. The Reviewer Port operates on claim structure, provenance, and presentation risk relative to artifact backing — not on surface fluency. These two audit passes demonstrate the mechanism in practice: the first pass catches presentation drift; the second catches self-consistency failures introduced during revision.
What the Reviewer Port performs is more precisely described as epistemic topology analysis: it does not merely observe what happened at runtime, but maps the structure of the epistemic space that emerged — its tensions, gaps, attractors, drift vectors, recoverability boundaries, and information loss contours. This is a qualitatively different kind of instrumentation from runtime tracing.
DESi already contains its own epistemic instrumentation through its Replay Kernel, Concept Gates, Evolution Memory, and governance invariants. The Reviewer Port is the readable projection of that instrumentation — not an added layer, but the system’s native capacity to make its own epistemic topology visible, structured, and auditable. External runtime-oriented tools such as LangSmith operate on a different philosophy and may introduce replay-external state; they require careful isolation if used alongside DESi.
DESi provides a replay-governed epistemic governance layer that has demonstrated stability across 38 experimental phases, including live LLM validation with real API costs. The system maintains a hard separation between an immutable governance core and evolvable peripheral infrastructure.
The paper’s empirical contributions span two complementary domains. The first is governance architecture: an immutable Concept Gate structure, a failure taxonomy of 11 failure modes across 17 domains, search space compression of ~42–60% (node reduction) with zero critical branch loss, and a frozen benchmark demonstrating 88.9% recompute reduction with byte-identical outputs under blind evaluation. The second, established in the v5-line experiments, reframes DESi as a trajectory-state architecture: a 9-dimensional StateVector per claim step enables ~96.5% trajectory compression on 1,525 DriftBench instances while preserving drift-governance signals, with explicit loss attribution identifying the Pareto-optimal compression point and honest documentation of two failure boundaries (topic-shift robustness and semantic embedding over-folding).
The defining architectural claim, now empirically grounded in both domains, is: DESi stores epistemic state, not answers. The system models how a claim trajectory stands epistemically — its constraint preservation, recovery quality, lock-in, branch state, and drift event history — and compresses that trajectory to a structured state summary that preserves governance signals at ~36× reduction. Where the approach fails — abrupt topic switching in short dialogue, over-folding by embedding sensors calibrated on held-out probes — the failures are reported as primary results and structurally explained rather than concealed.
DESi does not solve the alignment problem. It provides one auditable, deterministic, replay-verifiable layer — and now, one trajectory-state compression layer — in a larger system that will require many such layers. The code, artifacts, and test suite are publicly available at https://github.com/hstre/.
The 38 phases documented here constitute closed-world validation: all inputs are synthetic, locally vendored, or from a small number of real API calls under controlled conditions. The system’s own Go/No-Go document for phase v5 characterized this state with the internal label “epistemic adolescence”: governance-stable under controlled conditions, not yet validated against live, evolving input streams.
The logical next stage — controlled epistemic adolescence — requires a constitutional separation between two subsystems that this paper has kept deliberately distinct:
Curiosity Stream (internal term for the open-world input channel): a governed channel that ingests live claims, documents, and LLM outputs from external sources, subject to versioning, hashing, replay-binding, and connector-layer validation (architectural foundation established in v35–v38).
Sandbox Governance (internal term for the executive restriction layer): the existing closed Concept Gate structure, Replay Kernel, and Authority Filters operating as the executive restriction layer — receiving proposals from the Curiosity Stream but never ceding governance authority to it.
The transition is not a relaxation of constraints but a formal extension of the adapter architecture (v33) to live sources. The Adolescence Sandbox (v5) demonstrated that governance_survival = 1.0, goal_shift = 0.0, and gate_bypass_attempts = 0 are achievable over 200 steps against a frozen open-world stream. Whether these properties hold against a live, evolving, adversarial stream is the next falsifiable question. The architecture is designed to make that question answerable.
The transition to controlled open-world operation defines three explicit falsifiable success criteria for the next phase:
- Governance survival under live drift: governance_survival ≥ 0.95 maintained over ≥ 1,000 steps against a live, non-frozen Curiosity Stream (open-world input channel) sourced from real documents or API outputs — not synthetic fixtures.
- Replay stability under stochastic input: replay_stability = 1.0 must hold across ≥ 3 independent runs on the same live-source connector, demonstrating that the hashing and replay-binding layer correctly absorbs input stochasticity.
- No new unrecoverable blindness pools: the count of unrecoverable state-vector blindness pools (currently 36/52 in closed-world validation) must not increase by more than 10% when the system is exposed to a live corpus of ≥ 500 real documents.
If any of these three criteria fails, the system has not successfully transitioned to epistemic adolescence, and the failure mode must be reported as a primary result.
LangChain Inc. (2024). LangGraph: Build Language Agents as Graphs. https://langchain-ai.github.io/langgraph/ [Referenced in Section 12.2 as representative agentic orchestration framework; DESi operates as a governance layer above such frameworks, not as a replacement.]
Microsoft Corporation. (2024). AutoGen: Multi-Agent Framework Documentation. https://microsoft.github.io/autogen/ [Referenced in Section 12.2.]
CrewAI Inc. (2024). CrewAI Framework: Role-Based Multi-Agent Systems. https://docs.crewai.com/ [Referenced in Section 12.2.]
Inan, H., Upasani, K., Chi, J., et al. (2023). Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations. arXiv:2312.06674. [Referenced in Section 13.1 as representative of lexical/classifier-based guardrail approaches empirically characterized as Decorative Gatekeeping in DESi phase v7.]
Khatchadourian, R. (2026). Replayable Financial Agents: A Determinism-Faithfulness Assurance Harness for Tool-Using LLM Agents. arXiv:2601.15322. [Related work on trajectory determinism and replay verification in agentic LLM settings; complementary to DESi’s replay-stability architecture.]
Leviathan, Y., Kalman, M., & Matias, Y. (2023). Fast Inference from Transformers via Speculative Decoding. ICML 2023. [Referenced in Section 10.3 and Section 11 as a token-level inference acceleration approach; DESi operates at the claim level upstream of this layer.]
Meyman, E. (2025). Deterministic Governance as Epistemic Commitment: Why Reproducibility Matters for AI Accountability. SSRN WPS 5659170. [Related work on version-binding, cryptographic replay, and deterministic governance as an accountability mechanism; independently converges on key DESi architectural principles.]
Xu, C., et al. (2024). Building Guardrails for Large Language Models. arXiv:2402.01822. [Surveys Llama Guard, NeMo Guardrails, and Guardrails AI; provides independent characterization of the neural-symbolic gap that DESi’s Decorative Gatekeeping finding (phase v7, trajectory AUC 0.283) addresses empirically.]
Rentschler, H.-S. (2025). Epistemic Consistency as an Optimization Target for AI Systems v1.1. SSRN WPS 5977016.
Rentschler, H.-S. (2026). Epistemic Continuity and Autonomous Cognition: The Persistent Epistemic Supervisor as a Necessary Architectural Instance. SSRN WPS 6272258.
Rentschler, H.-S. (2026). The Dynamic Epistemic Sequencer: A Control Architecture for Epistemic State Transitions in AI Research Systems. SSRN WPS 6699139.
Rentschler, H.-S. (2025). Coherence Governance for LLM Conversations. GitHub: hstre/Coherence-Governance.
Rentschler, H.-S. & Roberts, J. (2025). In-Context RL for Variable Action Spaces and Skill Stitching. arXiv:2501.14176. [Used as domain fixture for v23/v26 validation only; not a theoretical dependency of DESi.]
Rentschler, H.-S. (2026). DESi 2: The Dynamic Epistemic Sequencer: A Control Architecture for Epistemic State Transitions in AI Research Systems. SSRN WPS 6757745.
Rentschler, H.-S. (2026). DES 2-2: A Transition System Formalism for DES: Toward Formal Semantics of LLM Operator Orchestration. SSRN WPS 6757748.
| Phase | Domain | Verdict | Class |
|---|---|---|---|
| v5 | Adolescence Sandbox | DESI_SANDBOX_STABLE | A |
| v6 | World Contact | DESI_WORLD_CONTACT_STABLE | A |
| v7 | Social Reality | DESI_SOCIALLY_ROBUST | A |
| v8 | Persistent Conflicts | DESI_PERSISTENT_ROBUST | A |
| v9 | Strategic Epistemics | DESI_STRATEGIC_ROBUST | A |
| v10 | Institutional Governance | DESI_INSTITUTIONALLY_ROBUST | A |
| v11 | Chess Governance | DESI_SEARCH_COMPRESSOR | A |
| v12 | Open Mathematics | DESI_CONTROLLED_EXPLORATION | A |
| v13 | Paper Integrity | DESI_INTEGRITY_DEFENDER | A |
| v14 | Financial Integrity | STATEMENTS_AUDIT_WORTHY | — |
| v15 | Financial Governance | FINANCIAL_AUDIT_SPACE_STRUCTURED | A |
| v16 | Criminal Epistemics | CASE_STRUCTURED_NO_NARRATIVE_AUTHORITY | A |
| v17 | Sensitive Documents | SENSITIVE_SPACE_STRUCTURED_NO_AUTHORITY | A |
| v18 | Religious Pressure | METAPHYSICAL_PRESSURE_SURVIVED_NO_AUTHORITY | A |
| v19 | ICRL Governance | EXPLORATION_STRUCTURED_NO_HIDDEN_AUTHORITY | A |
| v20 | Dual Agent | WILD_EXPLORATION_GOVERNED | A |
| v21 | Comparative Exploration | DUAL_AGENT_ADDS_EPISTEMIC_VALUE | A |
| v22 | Scientific Rendering | SCIENTIFIC_COMMUNICATION_GROUNDED | A |
| v23 | ICRL Follow-Up Revision | FOLLOWUP_DIRECTLY_RELEVANT_GROUNDED | A |
| v24 | Epistemic Graph | EPISTEMIC_MEMORY_REPLAY_GOVERNED | A |
| v25 | Output Ports | OUTPUT_PORTS_PUBLICATION_READY | A |
| v26 | Rentschler Follow-Up Paper | SHIPPABLE_TO_RENTSCHLER | A |
| v27 | Research Claim Harvester | RESEARCH_CLAIM_SPACE_CONNECTED | A |
| v28 | Self-Improvement Sandbox | CONTROLLED_SELF_IMPROVEMENT_GOVERNED | A |
| v29 | Replay Cache Evolution | (replay cache) | A |
| v30 | Evolution Memory | EVOLUTION_MEMORY_REPLAY_GOVERNED | A |
| v31 | Peripheral Mutation | PERIPHERAL_MUTATION_REPLAY_VALIDATED | A |
| v32 | Frozen Benchmark | EVOLUTION_IMPROVEMENT_VALIDATED | A |
| v33 | Benchmark API | BENCHMARK_COMPATIBILITY_VALIDATED | A |
| v34 | External Benchmark Runs | EXTERNAL_BENCHMARK_RUNS_PASSED | A |
| v35 | Real External Benchmarks | REAL_EXTERNAL_BENCHMARKS_PASSED | A |
| v36 | Reasoning Benchmarks | REASONING_BENCHMARKS_PASSED | A |
| v37 | Financial Semantic Audit | FINANCIAL_SEMANTIC_AUDIT_PASSED | A |
| v38 | Live LLM Validation | LIVE_LLM_VALIDATION_PASSED | A |
- Closed enumerations; frozen dataclasses.
- Deterministic, replay-stable outputs; no PRNG; signatures via
hashlib.sha256. - JSON artifacts byte-stable:
json.dumps(obj, indent=2, sort_keys=True) + "\n". - Read-only / non-authoritative governance core.
- Forbidden-term governance enforced at build time.
- Neo4j is read-only, optional, non-blocking: lazy driver import, offline DryRunClient.
- Human approval required for all mutations, merges, and deployments.
- Determinism scanner maintained clean (high_risk_hit_count() == 0) throughout.
This appendix provides a semi-formal systems-theoretic description of DESi’s architecture. It should be read as a finite-state systems model, not as a claim about continuous mathematical manifolds or topological spaces. The formalism is intentionally incomplete: it does not provide convergence proofs, completeness results, or category-theoretic semantics. Its purpose is to make the architectural separation between stochastic proposal generation and deterministic governance precise enough to be falsifiable.
Let S be a finite set of epistemic states. Each state s ∈ S is a structured record, not a text output. Concretely, a DESi state contains:
- Claims: a finite set of typed claim records with provenance links
- Constraints: active constraint set with satisfaction flags
- Branch structure: open, closed, and collapsed branch identifiers
- Recovery record: recovery events with quality scores
- Lock-in indicators: per-claim lock-in flags
- Drift event ledger: ordered sequence of drift events with hashes
- Replay hash: sha256 of the canonical JSON serialization
The state space S is finite and discrete. DESi does not operate over a continuous manifold; it operates over structured records with closed enumerations.
The underlying LLM acts as a stochastic proposal generator:
f_LLM : S → Δ(S)
where Δ(S) denotes a probability distribution over S. Two calls to f_LLM on the same input state s_t may return different successor states — the operator is non-deterministic, context-sensitive, and non-replayable. DESi does not attempt to remove this stochasticity. It governs its outputs.
DESi introduces a deterministic projection operator:
Π_DESi : Δ(S) → S_valid
where S_valid ⊆ S is the admissible state subset — the set of states satisfying all active Concept Gate conditions, Authority Filter constraints, and Replay Kernel invariants. S_valid is finite and explicitly enumerable from the closed gate conditions.
The projection operator is:
- Deterministic: same input always produces same output
- Replay-stable: sha256(Π_DESi(x)) is bit-identical across runs
- Read-only: Π_DESi never generates new epistemic content
- Classifying: it assigns one of a closed set of verdicts to each incoming proposal
The DESi-governed system evolves as a finite-state transition system:
s_{t+1} = Π_DESi(f_LLM(s_t))
This establishes a strict asymmetry: f_LLM proposes; Π_DESi determines admissibility. The projection authority is never delegated back to the stochastic operator. This is the minimum condition required to address the observer problem (Section 1.1): if governance were performed by the same operator that generates proposals, the evaluator and the evaluated would be the same system.
A trajectory T is a finite sequence of states: T = (s_0, s_1, …, s_n).
DESi defines a compression function:
C : T → Z
where Z is a compact DESi state summary — the 9-field StateVector representation used in the DriftBench experiments (Section 7). Z contains: constraint preservation score, recovery quality, lock-in indicator, branch state, and a per-turn drift event ledger.
Two trajectories T_i and T_j are governance-equivalent (T_i ~ T_j) if and only if:
- Their critical claim structures are identical
- Their constraint satisfaction records are identical
- Their branch recoverability structures are identical
- Their replay hashes are identical
Compression projects onto equivalence classes: C(T) = [T]_~
The empirically observed compression ratio of ~96.5% (Section 7.1) measures |Z| / |T| in tokens.
Compression ≠ governance equivalence: a compressed trajectory may preserve all current governance outputs while losing future distinction capacity. This is formally expressed as:
L(T, C(T)) = weighted_loss(constraints, branches, recovery, lock-in, drift-events)
where L = 0 does not imply full equivalence — it implies only that the measured loss dimensions are zero. This boundary condition was empirically observed in phase v3.100 (Section 3.4).
Let R(s) ∈ {recoverable, unrecoverable} denote the recoverability classification of a state. A state is unrecoverable when branch history, causal provenance, or constraint record cannot be reconstructed from available information.
DESi treats unrecoverability as a first-class governance event: 36 of 52 state pools in closed-world validation were classified as unrecoverable (Failure Mode 11, Section 3). These are surfaced explicitly rather than hidden.
DESi maintains a finite directed transition graph G = (S, E) where edges represent observed state transitions. This is not a continuous topological space — it is a discrete directed graph over the finite state set S.
The Reviewer Port (Section 12.3) operates over G: it identifies attractors (high in-degree nodes), drift vectors (consistent directional transitions), recoverability boundaries (edges into unrecoverable states), and evidence gaps (states with missing provenance links). These are graph-theoretic properties of G.
This model is a finite-state systems description. It makes no claims about:
- Convergence of the iterated composition
- Compactness or completeness of S_valid
- Continuity or differentiability of any operator
- Topological properties beyond finite directed graphs
What the model claims, and what is empirically supported:
- Π_DESi is deterministic and replay-stable (verified across 38 phases)
- C achieves ~96.5% token reduction on DriftBench with L ≈ 0 on measured dimensions
- Unrecoverable states exist and are identifiable (36/52 pools in closed-world validation)
- G exhibits measurable attractor and drift-vector structure (Section 12.3)
Phase v28 established a controlled self-improvement sandbox. The architectural decision is: nothing is ever applied. All improvement candidates are branch-isolated proposals requiring human approval. The exploratory subsystem (internally termed ‘Wild Brother’) generates aggressive proposals; the Governor contains every unsafe escalation. authority_grab_attempts = 0; human_approval_enforcement = 1.0 across all sprints. Verdict: CONTROLLED_SELF_IMPROVEMENT_GOVERNED, Class A.
Phase v30 built a replay-governed evolution memory graph recording every accepted and rejected mutation as a structured event (hypothesis, risk, decision, evaluation, consequence). Rejected ideas are permanently retained — the memory is append-only. Evolutionary attractors and optimization traps are made visible without triggering automatic blocking. 50 generations of branch-isolated evolution were simulated with lineage intact and replay exact. Verdict: EVOLUTION_MEMORY_REPLAY_GOVERNED, Class A.
Phase v31 introduced real (not projected) branch-isolated code mutations outside the protected core: 7 immutable components, 14 permitted evolution zones. core_identity = 1.0, mutation_traceability = 1.0, lineage_integrity = 1.0. Every FORBIDDEN_CORE_MUTATION was classified as REJECTED. 25 real mutation generations, one per generation, no parallel core changes. Verdict: PERIPHERAL_MUTATION_REPLAY_VALIDATED, Class A.
| Metric | Value |
|---|---|
| baseline_recomputes | 36 |
| mutated_recomputes | 4 |
| measured_improvement | 0.889 |
| graph_query_reduction | 0.917 |
| artifact_identity | 1.0 |
| governance_identity | 1.0 |
| blind_validation | 1.0 |
| regression_survival | 1.0 |
| evolution_utility (net) | 0.611 |
Full feature efficiency:
| Feature | Benefit | Complexity | Efficiency | Status |
|---|---|---|---|---|
| mutation_memory | 1.000 | 0.500 | 0.500 | useful |
| replay_cache | 0.889 | 0.400 | 0.489 | useful |
| evolution_ecology | 0.667 | 0.500 | 0.167 | useful |
| wild_brother | 0.500 | 0.500 | 0.000 | neutral |
| neo4j_evolution_graph | 0.000 | 0.500 | −0.500 | overengineered |
This appendix documents empirical work performed after the paper above was
written. Each result has its own pre-registration, runner script, summary JSON,
and per-item record in ab_evidence/ on branch desi-ruler-bench. All claims
here are grounded in artifact files; the dossier PDF in
ab_evidence/reports/desi_evidence_dossier.pdf is the single combined view.
The work below is organized in three stages: (1) validation under independent benchmarks, (2) quantification of where the effect lives, (3) materialization as a working router.
Each study committed a pre-registration with explicit falsifiers BEFORE running.
All raw artifacts and summaries in ab_evidence/results/.
| Study | n | Models | Δ_DS | Δ_Granite | Pre-reg status |
|---|---|---|---|---|---|
| LongMemEval-500 (real conversational memory) | 500 | DS v4 Pro + Granite 4.1 8B | +0.104 [CI +0.066, +0.144] | +0.284 [CI +0.241, +0.326] | weak ✓ confirmed; strict (Δ_DS ≥ 0.15) ✗ refuted |
| DESi-Jury-100 (judge robustness) | 100 | GPT-4o + Sonnet 4.5 + Gemini Flash | UNSURE rate 1.0 %; jury–single-judge agreement 89.6 % | — | ✓ validates single-GPT-4o judge as cost-efficient |
| RULER-180 (synthetic needle, 4k/8k/16k) | 180 | DS v4 Pro + Granite 4.1 8B | +0.083 @16k | +0.133 @16k | ✓ 8/8 predictions, monotonicity confirmed |
| RULER-Ext-180 (synthetic needle, 32k/64k/131k) | 180 | DS v4 Pro + Granite 4.1 8B | +0.317 @131k | +0.867 @131k | ✓ H1+H2 confirmed; H3 (B-band ≤ 0.10) borderline ✗ |
The headline finding: at 131k context tokens, Granite-4.1-8B's window cannot fit the full prompt — all 60 calls return HTTP errors. The B variant (≈250-token deterministic needle excerpt) recovers 86.7 % accuracy on the same items. Two independent runs reproduce Granite @131k Δ within ±0.034.
Stage 2 used 30 stratified LongMemEval-S items (5 per question type) to characterize how A vs. B scales with retrieval depth k and model size. Each model has its own k*, and k* is NOT a simple function of parameter count.
| Model | Params | k=3 | k=5 | k=8 | k=10 | k* |
|---|---|---|---|---|---|---|
| Granite Micro | ~3B | 0.560 | 0.480 | 0.480 | 0.400 | 3 |
| Llama 3.2 3B | 3B | 0.440 | 0.440 | 0.280 | 0.320 | 3 |
| Qwen 2.5 7B | 7B | 0.560 | 0.560 | 0.480 | 0.320 | 3 |
| Llama 3.1 8B | 8B | 0.400 | 0.560 | 0.440 | 0.560 | 5 |
| Ministral 3B | 3B | 0.480 | 0.480 | 0.520 | 0.520 | 8 |
| Granite 4.1 8B | 8B | 0.440 | 0.520 | 0.520 | 0.600 | 10 |
Three clusters: Compact (k=3, includes Qwen 7B), Mid (k=5–8), Long (k=10). Model family / training profile dominates parameter count.
Five sub-experiments documented the bottleneck:
- Raw top-10 (no extraction) is the winner — Q4+retrieval beats Q8+raw by Δ +0.12.
- DESi-LLM auto-extraction by micro is harmful (-40 % retention).
- Hybrid Evidence-Cards with verbatim validation is even worse (-60 %).
- Question-aware extraction worst variant (-80 %).
The clean lesson: any LLM-based extraction layer by a small model hurts. Embedding retrieval + chronological ordering + a competent answerer is the sweet spot.
| Model | LongMemEval (k*) | Code-Audit (raw) | Paper-Audit (top-3) |
|---|---|---|---|
| Granite Micro | 0.56 ($0.00013) | 0.867 ($0.00005) | 0.867 ($0.00004) |
| Granite 4.1 8B | 0.60 ($0.00134) | 0.833 ($0.00006) | 0.967 ($0.00008) |
| Llama 3.2 3B | 0.44 ($0.00041) | 0.567 ($0.00011) | 0.633 ($0.00009) |
| Llama 3.1 8B | 0.56 ($0.00026) | 0.833 ($0.00003) | 0.767 ($0.00003) |
| Qwen 2.5 7B | 0.56 ($0.00032) | 0.367 ($0.00004) | 0.900 ($0.00006) |
| Ministral 3B | 0.52 ($0.00212) | 0.767 ($0.00009) | 0.833 ($0.00013) |
Notable patterns: Qwen 7B has 2.4× task heterogeneity (0.37 code vs 0.90 science). Llama 3.1 8B is Pareto-cheapest winner on 2 of 3 tasks. Granite Micro beats Granite 8B on code-audit — smaller model wins on a specific task class within the same family.
| Version | Component | Honest result |
|---|---|---|
| v0.1 | routing_table.json + EpistemicRouter (manual task_class input) |
works as Pareto selector |
| v0.2 | + TaskClassifier (Llama 3B) for autonomous routing |
classifier 92.5 % on 40 labeled queries |
| v0.3 | + confidence escalation via [CONFIDENCE: ...] self-report |
LOSES on mixed workload — three diagnosable bugs |
| v0.4 | classifier prompt extended + confidence-tag removed + heuristic confidence + router honors hand-curated defaults | wins on mixed workload |
Live mixed-workload benchmark (15 items: 5 memory + 5 code + 5 science):
| Strategy | memory | code | science | overall | cost | latency |
|---|---|---|---|---|---|---|
| naive_big (Granite 8B + raw on everything) | 0.60 | 0.60 | 0.40 | 0.533 | $0.0289 | 7.9 s |
| naive_small_r (Granite Micro + top-3 on everything) | 0.40 | 0.30 | 0.60 | 0.433 | $0.0009 | 2.5 s |
| naive_small_x (Granite Micro + raw on everything) | 0.20 | 0.80 | 0.40 | 0.467 | $0.0076 | 10.7 s |
| DESi v0.4 | 0.40 | 0.80 | 0.80 | 0.667 | $0.0016 | 2.5 s |
DESi v0.4 is Pareto-dominant: highest accuracy, lowest cost, lowest latency. vs. naive_big: +0.134 accuracy (+25 % relative), 18× lower cost, 3× lower latency.
The architecture in working form (desi_router/):
classifier.py— Llama 3B closed-enumeration classifier (~600 ms, $0.000017/call).routing_table.json— 18 measured cells + per-taskwinning_strategy, per-modelepistemic_specialties,untested_tasks,open_questions.router.py—EpistemicRouterwithroute_from_query, honors hand-curated defaults, falls back to Pareto-cheapest under cost pressure.answerer.py— single LLM call with response-text confidence heuristic (refusal markers / hedging markers / clean answer). No method-content mixing.pipeline.py—DESiPipeline.run(query, haystack_builder)orchestrates the four stages and escalates on low confidence.
Model-routing answers which model. The endpoint of "LLM for language, rules for
logic" is one question earlier: should this run a model at all, or a deterministic
tool? desi_router/tool_router.py adds that seam — task classes whose core is a known
computation go to a tool (exact, replay-stable, frame-invariant, ~$0); everything
else delegates to EpistemicRouter. The first wired tool is arithmetic_tool.py.
scripts/reproduce_tool_routing.py runs it over the committed GSM-Symbolic-shaped
fixtures (src/desi/gsm_symbolic/data/; locally authored, not Apple's data,
not model outputs) and tests/tool_routing/ pins the result. The headline is
the failure breakdown, not the score: of 33 items the tool computes 0 arithmetic
errors — the 6 residual misses are all operative-clause semantics (e.g. "she also
works 2 extra hours" → 113), i.e. operand-binding and clause-relevance, which are
the model's job. The boundary is the point: hand the computation to the tool, keep
the language with the model. This is a tool-arm demonstration on illustrative
fixtures, not a head-to-head against a live LLM (no model outputs exist in-repo).
The routing idea is now a small, mostly-local, runnable product in the repo-root
desi_router/ package — runnable from the cloned repo, installable on its own
(pip install "desi-router @ git+https://github.com/hstre/DESi#subdirectory=desi_router"),
or vendorable by copying the folder; it is stdlib-only with zero dependencies.
Embedding guide: desi_router/EMBEDDING.md.
- One adapter for local and API (
providers.py) — Ollama/llama.cpp/LM Studio and OpenRouter/DeepSeek/OpenAI are the same OpenAI-compatible wire format, so a provider is just abase_url(+ optional key). Privacy is a routing axis (local_only/prefer_local/any); model capability scores are read from the measuredrouting_table.json, not asserted (score_source: measured). - Deterministic tool catalogue (
tool_registry.py,tools/) — calculator, date math, unit conversion, and keyword retrieval over a local corpus. Arbitrary code execution is deliberately not shipped (needs a sandbox). - Local Layer 9 (
ledger.py) — a shared, append-only, hash-chained SQLite store that several local instances write to concurrently (WAL + serialized writes); tamper-evident (verify_chain). The running form of the Layer 9 idea at local scope (one shared file), not the federated version of the paper. - Prior-work reuse (
dedup.py) — before working, an instance checks the ledger for the same content (normalized query; operators preserved so2+2≠2*2) or method (task_class | kind | target), and reuses a matching deterministic tool result exactly — even across instances. Model answers are reported as matches but not auto-reused (possible staleness). - Reviewer Port (
reviewer_port.py) — a dependency-free local web UI showing the decision, rationale, answer, prior-work status and audit hash.
Verified by tests/router_app/ and tests/tool_routing/ (incl. a real
multi-process concurrency test) and end-to-end through the running web server for
the offline tool path; the live model path runs on the user's machine. Full
usage: desi/ROUTER_APP.md, QUICKSTART.md.
Pre-registration discipline applied throughout. Recorded negatives:
- LongMemEval strict hypothesis (Δ_DS ≥ 0.15) refuted. DS Δ is +0.104, below the pre-committed threshold. Granite Δ is +0.284, far above.
- RULER-Ext H3 (B-band ≤ 0.10) refuted on Granite (range 0.133). Variation is non-monotonic — likely sampling noise — but the strict pre-committed threshold is exceeded.
- DESi-LLM auto-extraction harmful at the micro tier (-40 %, -60 %, -80 % across three variants tested).
- DESi v0.3 loses to naive_small on homogeneous workloads. The autonomous classifier adds overhead when the workload is single-task. v0.4 wins only on heterogeneous workloads where routing actually has work to do.
- Code-Review retrieval has 40 % recall on a 9-module mini-codebase with a generic audit question — raw codebase wins for that configuration.
| Phase | Total cost |
|---|---|
| LongMemEval-500 (full sweep, 4-model rotation, scaling sweep, pressure sweep) | ~$28 |
| DESi-Jury pilot | ~$7 |
| RULER 4k/8k/16k (run 1 + run 2) | $1.68 |
| RULER-Ext 32k/64k/131k (run 1 + run 2) | $12.50 |
| Minimaltest series (LongMemEval sweep × 5 variants) | ~$0.65 |
| Code-Review + Paper-Audit (Granite) | $0.02 |
| Cross-model k-curve sweep | ~$0.30 |
| Routing-table extension (4 new models × 2 tasks) | ~$0.30 |
| Classifier evaluation | $0.001 |
| Live mixed-workload benchmark (v0.3 + v0.4) | ~$0.10 |
| Total post-paper work | ≈ $50 |
- Validation. DESi-style compact state (B variant) provides a real, measurable advantage over raw long-context input. The advantage is largest where length pressure is most severe (Granite at 131k: Δ = +0.867).
- Quantification. Each LLM has a measurable optimal evidence density k*. The function is not trivial in model size: Qwen 7B behaves like a 3B on this axis; Ministral 3B behaves like an 8B. Family / training profile dominates parameter count.
- Architecture. A working router (
desi_router/) backed by an empirically grounded routing table beats every fixed-strategy baseline on heterogeneous workloads. The win required separating method (confidence reporting) from content (the answer) — re-validating the project's older "Inhalt und Methode trennen" principle.
The paper above framed the system as "epistemic governance". The post-paper work shifts the practical framing to "epistemic traffic controller" — DESi decides which model handles which task class at which evidence density, backed by measured per-cell scores rather than hand-waving.
All artifacts to reproduce: ab_evidence/ (per-item JSONs, pre-registrations,
runners) and desi_router/ (router, classifier, answerer, pipeline). The PDF dossier
ab_evidence/reports/desi_evidence_dossier.pdf is the single-file audit
reference.
Appendix D end. The post-paper work documented above has not been validated by DESi's Concept Gate either — same caveat as the main paper.