Prepare browser DSP and bound 64-track startup work

.github/ISSUE_SPECS/719-prepare-browser-dsp-and-bound-pcm-feed.md

@@ -0,0 +1,52 @@
+# Prepare browser DSP off-thread and bound first-callback PCM feed work
+
+Base: engine origin/main 0df770b0f1f002b563246db70b71a3991fe8a8c7; public engine SDK 0.2.1. The user authorizes Astra xhigh to continue the scalable 64-track effort and iterate the cold-start overrun before release. No legacy engine source, DSP algorithm changes, storage/download changes, or package publication is in this implementation tranche.
+
+## Measured problem and scope
+
+Three separate Chromium processes with the actual 64-track app EQ/compressor graph, current unbatched canonical storage, all 737280000 PCM bytes independently verified, physical HTTP concurrency four, meters and telemetry each show exactly one cold first-window render overrun (8/5/8ms). Warm inner-render telemetry is clean. Trace attribution identifies 49 synchronous Wasm lazy-compilation events inside the first DSP render totaling4.322ms. The complete feed-plus-render callback also costs about8ms warm: the existing feed greedily fills internal queues from complete shared runways in one callback.
+
+Copied-current-package experiments preserve full verification and feed oracles. A disposed/terminated scratch rehearsal still misses5ms live. Retaining its compiled asset through live-host construction removes live synchronous compilation. Combining that with at most two successful fresh submissions per source per callback gives first complete callbacks1.516ms cold/2.679ms warm, with zero feed failures. The warm trace remains12microseconds above the128-frame/48k deadline; preserve that residual and the effect of instrumentation, rather than rounding it into a perfect result. Local evidence and failed variants: app docs/analysis/mixer-64-cold-start-results.md and e2e/results/mixer-startup-combined-{poc,attribution}-0.json.
+
+Smallest product slice: move default adapter-consumed first-use DSP preparation to a disposable worker, strongly own and reuse its exact compiled module in the live host, and bound the PCM work that precedes every render. Both components are necessary to address the named first-callback budget. Existing APIs remain compatible; no second benchmark framework is needed.
+
+## Design and compatibility
+
+1. Add a browser preparation API returning the validated session shape and a strongly owned WebAssembly.Module. Boot the exact document/policy in the existing bounded worker, rehearse64 fixed quanta with bounded reusable synthetic nonzero source planes and metering when available, dispose the scratch instance, return the module by structured clone, and physically terminate before completion is observable. No real stem is read or marked ready by rehearsal. There is no compiled track-count cap: source count remains constrained by the existing declared resource policy; rehearsal duration is bounded by64 quanta and the existing worker deadline.
+2. Expose the compiled module from its existing verified asset owner without recompiling, instantiating or changing its bytes. The live host accepts this prepared module and passes the same module into its AudioWorklet; it must not fetch/recompile another module on that path. The live instance starts at sample0 with fresh DSP state. Preserve existing URL-only host construction, shape-only scratch APIs and custom scratch/host overrides. Module ownership is explicit rather than an assumption about browser code-cache lifetime.
+3. In the current SDK PCM feed, admit at most two successful fresh submissions per source per process call; scan at most that source ring's capacity. Count only accepted fresh chunks against the submission allowance. Retain unconsumed backpressure slots, source generations, stale release, exact short-tail PCM/EOF flags and error counters. Existing internal queue capacity is unchanged and fills gradually; the all-source shared runway remains available.
+4. Adapter adoption is a downstream tranche after this engine capability is reviewed: retain the preparation result through complete source verification and pass its module into live host construction. Do not copy current engine worklet code into adapter runtime. Root coordinates exact version pins and the engine-to-adapter-to-app release pipeline after packed qualification.
+
+## Objective gates and review
+
+Focused SDK tests prove one compilation and exact module identity through worker/host handoff, shape/policy equality, scratch disposal and physical worker termination on success/failure/abort/deadline, compatibility of old scratch requests and URL-only host calls, and live sample0/fresh state. Rehearsal covers short/zero-length tails without fabricating real readiness, different source/channel counts, console-disabled policies and errors. The feed gate proves per-source admission bounds, capacity-bounded scanning, eventual prior internal queue depth, independent sources, actual backpressure retention, generation-safe seek/stale retirement, exact PCM and EOF, and unchanged continuous playback.
+
+Run proportional SDK type/generated/source/browser PCM and package gates, then a fresh packed consumer against the exact artifact. Root adapter qualification preserves initial/paused/running exact-first-output, full contiguous generation runways and all191 lifecycle tests. Final64-track app effects playback crosses full source EOF and repeats seeks cold/warm, with physical HTTP concurrency four, full independent canonical verification and zero underruns/refused/torn/errors. Measure the complete feed-and-render callback using the existing high-resolution trace path in addition to inner telemetry; do not infer a deadline pass from a1ms clock. No perfect cross-browser claim: physical iPad/Safari remains a separate qualification limit.
+
+A fresh Astra xhigh reviewer verifies the spec before implementation and each coherent green checkpoint. Keep local checkpoints on one batch branch; no publication or deployment until exact packed artifacts and root review are ready. Research helpers are evidence, not production package files.
+
+## Status
+
+Spec prepared before implementation. The existing adapter opening-attachment cancellation blocker is already fixed and independently reviewed in adapter6dbc3c8; full191 tests and packed healthy/fault gates pass. Its v2 archive is retained separately. Engine production implementation has not started.
+
+Fresh Astra xhigh spec review approves beginning the bounded tranche. Clarifications: module identity is local identity at each send boundary, not JavaScript reference equality across structured-clone realms; prove one worker compilation and no host refetch/recompile, then real-browser no-live-lazy trace after worker termination. Snapshot document bytes and nested policy words before async work so caller mutation cannot skew scratch and live boot. Exercise same-turn reply/abort ordering and module-clone/post failures, with physical termination before either public outcome. The 2.679ms instrumented warm residual remains explicit until final artifact qualification.
+
+First coherent checkpoint implements the per-source two-accepted submission cap and capacity-bounded scan in the SDK-owned feed. Focused browser PCM gate passes13 tests against freshly built pinned Wasm (`/tmp/miso-engine-startup-artifacts`, log `/tmp/miso-engine-startup-feed-tests.log`). The new queue model starts from complete shared runways, enforces each source's admission allowance, reaches the prior eight-quantum internal capacity gradually, exercises actual capacity backpressure without consuming its pending shared slot, and preserves zero underruns/refusals. Existing seek, generation, tail, odd mono/stereo PCM and allocation-mutation tests remain green. Prepared-module implementation and final packed startup qualification remain pending.
+
+Prepared-module checkpoint f393c549 adds the explicit retained module API, bounded64-quantum disposable rehearsal, live host no-refetch handoff, owned document/policy snapshots and old scratch compatibility. Astra low implementation passes typecheck and52 focused browser/default tests; the full headless SDK gate passes187 tests (`/tmp/miso-engine-startup-sdk-check.log`). Stateful EQ/compressor/delay fresh-instance output and meter comparisons cover8 quanta, short1/129-frame tails and canonical zero-length refusal, module clone faults, same-turn abort/reply and physical worker termination. Independent Astra xhigh source review and final packed browser startup qualification remain pending. Adapter adoption is separately specified by misofm/engine-web-adapter#52.
+
+Fresh-state follow-up f011e363 activates EQ+6dB at1kHz and5ms wet feedback delay, compares eight live/fresh quanta after an initial excitation, proves nontrivial delayed output and exact four meter-window sample origins, and prepares four independently routed mixed mono/stereo sources with unequal short tails. All37 focused defaults tests pass (`/tmp/prepared-active-tests.log`). Local release preparation updates SDK/package lock and existing OIDC workflow identity gates to0.2.2; registry still reports0.2.1 and no publish workflow is active. This prepares reviewable artifacts only, and does not authorize or claim publication. Upstream/source audit and full callback browser qualification remain the release gates.
+
+Independent Astra xhigh implementation review finds no source blocker in the feed, prepared-module/state tests or adapter3daeaef consumption. The existing direct createEngine default remains shape-only; callers composing preparedModule must also reuse the saved shape through scratchBoot to avoid a separate shape worker compilation. The adapter does this explicitly, and the README documents the pair. Production trace-only packaged qualification is being prepared without copied runtime patches.
+
+## Final exact-package qualification
+
+Frozen engine0.2.2 source `cf7e695b29043abcae9f92d5172b719e33764f03`, archive `/tmp/miso-engine-0.2.2-reviewed/misofm-engine-0.2.2.tgz`, SHA256 `30639cf1f0f9707534020d584736339da3ef08981af5b6c5148f3ac310261e58`; all79 payloads match the source/build receipt. Pinned current-main Wasm is `5695fbc4d72fae4a78b5acd1cf8970c489163703a11ac5351974ce05a90b1574`. Full final headless gate passes188 tests and publishable-tarball gate passes (`/tmp/miso-engine-0.2.2-headless-final.log`, `/tmp/miso-engine-0.2.2-package-check.log`). Adapter0.3.5 exact consumer passes191 tests and real packed initial/resumed/running first-output/full-runway and terminal read-reject/deadline/crash cleanup gates.
+
+Production qualification uses exact package extraction, current unbatched writes, unchanged full737280000-byte independent source proof, HTTP concurrency four and actual64-track app EQ/compressor graph with64 meters. It makes no render-source patches, retained-worker substitution or extra rehearsal. A weak module-reference receipt proves the normal native worker terminate call returns before the live worklet receives the exact prepared module, including29.33/13.59 seconds of cold/warm verified-ingestion retention.
+
+Three fresh processes each pass cold/warm startup: complete first/full trace maxima1.327/2.410ms,1.351/1.828ms,1.371/2.317ms. All6785 traced full callbacks fit the128-frame/48k budget. No realtime Wasm lazy/baseline compilation remains; two JS-to-Wasm wrapper compile events per thread occur outside the first callback, so this is not a blanket zero-compilation claim. Raw reports and independent attribution are app `e2e/results/mixer-production-startup-{qualification,attribution}-{0,1,2}.json`.
+
+The long production run crosses EOF for62 seconds then performs8 two-second seeks in each cache phase; feed and transport pass across64 sources. One warm inner-telemetry window227 reports3 coarse-clock misses (peak4ms), about1.35 seconds into seek8. The initial trace did not cover it. A separate unchanged long-history run tracing that eighth seek passes all456 telemetry windows, all source/feed gates, and1510 complete late callbacks with cold/warm maxima1.086/1.104ms. No realtime compilation or GC occurs in those late traces; the earlier observation did not reproduce and its cause is unproven. The worklet clock falls back to Date.now at1ms resolution; do not infer an actual monotonic callback duration or clock-adjustment cause from that alone. Preserve both reports: `mixer-production-late-seek-{qualification,attribution}-0.json`.
+
+Independent Astra xhigh source review finds no implementation blocker. Nine research-helper synthetic tests and ESLint pass. The app report `docs/analysis/mixer-64-cold-start-results.md` retains exact evidence, failed PoCs and limits. The named startup defect and sustained feed architecture are qualified in this Chromium fixture; physical iPad/Safari and arbitrary scheduler-stall immunity are not claimed. Root owns main merge, OIDC qualification/publication, registry artifact verification, adapter exact adoption/publication and required app integration checks. No publication or deployment was performed by this implementation agent.

.github/workflows/npm-publish.yml

@@ -29,14 +29,14 @@ concurrency:
 
 env:
   PACKAGE_NAME: "@misofm/engine"
-  PACKAGE_VERSION: "0.2.1"
-  EXPECTED_WORKLET_SHA256: "54dcf7dd5f6199cf3ceeab77afefe09067e18b730c9e0a6ef9df73fbfd3afc69"
+  PACKAGE_VERSION: "0.2.2"
+  EXPECTED_WORKLET_SHA256: "5695fbc4d72fae4a78b5acd1cf8970c489163703a11ac5351974ce05a90b1574"
   RUSTUP_TOOLCHAIN: "1.97.1"
   NPM_CONFIG_REGISTRY: "https://registry.npmjs.org"
 
 jobs:
   release:
-    name: ${{ inputs.mode }} @misofm/engine 0.2.1
+    name: ${{ inputs.mode }} @misofm/engine 0.2.2
     runs-on: ubuntu-24.04
     steps:
       - name: Refuse non-main dispatches before checkout
@@ -124,7 +124,7 @@ jobs:
           const jobs = JSON.parse(fs.readFileSync(process.argv[3], "utf8"));
           const artifacts = JSON.parse(fs.readFileSync(process.argv[4], "utf8"));
           const expectedArtifact = `engine-sdk-qualify-${process.env.EXPECTED_SHA}`;
-          const qualificationJobs = Array.isArray(jobs.jobs) ? jobs.jobs.filter((job) => job.name === "qualify @misofm/engine 0.2.1" && job.conclusion === "success") : [];
+          const qualificationJobs = Array.isArray(jobs.jobs) ? jobs.jobs.filter((job) => job.name === "qualify @misofm/engine 0.2.2" && job.conclusion === "success") : [];
           const namedArtifacts = Array.isArray(artifacts.artifacts) ? artifacts.artifacts.filter((artifact) => artifact.name === expectedArtifact && artifact.expired === false) : [];
           if (run.conclusion !== "success" || run.event !== "workflow_dispatch" || run.head_sha !== process.env.EXPECTED_SHA || run.name !== "Publish @misofm/engine" || run.path !== ".github/workflows/npm-publish.yml" || qualificationJobs.length !== 1 || namedArtifacts.length !== 1) {
             throw new Error("qualification run is not a successful manual npm-publish run for expected_sha");
@@ -200,7 +200,7 @@ jobs:
           const entries = JSON.parse(fs.readFileSync(packJson, "utf8"));
           if (!Array.isArray(entries) || entries.length !== 1) throw new Error("npm pack did not report exactly one archive");
           const item = entries[0];
-          if (item.name !== "@misofm/engine" || item.version !== "0.2.1") throw new Error("packed package identity differs");
+          if (item.name !== "@misofm/engine" || item.version !== "0.2.2") throw new Error("packed package identity differs");
           if (typeof item.integrity !== "string" || !item.integrity.startsWith("sha512-")) throw new Error("npm did not report SHA-512 integrity");
           const bytes = fs.readFileSync(archive);
           const sha1 = crypto.createHash("sha1").update(bytes).digest("hex");
@@ -336,7 +336,7 @@ jobs:
           let statement;
           try { statement = JSON.parse(Buffer.from(envelope.payload, "base64").toString("utf8")); } catch { throw new Error("verified attestation DSSE payload is not valid base64 JSON"); }
           if (statement?._type !== "https://in-toto.io/Statement/v1" || statement?.predicateType !== "https://slsa.dev/provenance/v1") throw new Error("DSSE statement is not SLSA provenance v1");
-          const subjects = Array.isArray(statement.subject) ? statement.subject.filter((subject) => subject?.name === "pkg:npm/%40misofm/engine@0.2.1" && subject?.digest?.sha512 === local.sha512) : [];
+          const subjects = Array.isArray(statement.subject) ? statement.subject.filter((subject) => subject?.name === "pkg:npm/%40misofm/engine@0.2.2" && subject?.digest?.sha512 === local.sha512) : [];
           if (subjects.length !== 1) throw new Error("DSSE statement does not uniquely bind the package PURL and tarball SHA-512");
           const workflow = statement?.predicate?.buildDefinition?.externalParameters?.workflow;
           if (workflow?.repository !== "https://github.com/misofm/engine" || workflow?.path !== ".github/workflows/npm-publish.yml" || workflow?.ref !== "refs/heads/main") throw new Error("DSSE workflow identity differs from the trusted publisher");

hosts/host-web/web/miso-engine-v1-audio-worklet-host.d.ts

@@ -659,6 +659,7 @@ export interface CreateMisoAudioWorkletHostOptions {
   document: Uint8Array;
   options: MisoWebBootOptions;
   simd128ModuleUrl: string;
+  preparedModule?: WebAssembly.Module;
   workletModuleUrl: string;
 }
 

hosts/host-web/web/miso-engine-v1-audio-worklet-host.js

@@ -826,7 +826,8 @@ class MisoAudioWorkletHost {
 
 export async function createMisoAudioWorkletHost(options) {
   const quantumFrames = options?.context?.renderQuantumSize ?? 128;
-  if (!hasExactFields(options, OPTION_FIELDS)
+  if (!hasExactFields(options, options?.preparedModule === undefined ? OPTION_FIELDS : [...OPTION_FIELDS, "preparedModule"])
+      || (options.preparedModule !== undefined && !(options.preparedModule instanceof WebAssembly.Module))
       || options.context?.state !== "suspended"
       || !validU32(quantumFrames) || quantumFrames === 0
       || !validU32(options.context?.sampleRate) || options.context.sampleRate === 0
@@ -843,7 +844,7 @@ export async function createMisoAudioWorkletHost(options) {
   try {
     const selected = {
       backend: SHIPPING_BACKEND,
-      module: await fetchModule(options.simd128ModuleUrl),
+      module: options.preparedModule ?? await fetchModule(options.simd128ModuleUrl),
     };
     await options.context.audioWorklet.addModule(options.workletModuleUrl);
     node = new AudioWorkletNode(options.context, PROCESSOR_NAME, {

sdk/README.md

@@ -210,6 +210,20 @@ and `createDefaultHost` imports and invokes the shipped host with `toWebBootOpti
 `scratchBootInWorker` remains the low-level primitive for custom Worker entries. The browser
 helpers install no PCM feed or storage service.
 
+`prepareBrowserSessionWithWorker` additionally rehearses 64 fixed quanta of synthetic PCM in the
+throwaway instance and returns `{ shape, module }`. The instance is disposed and Worker terminated
+before the result resolves. Retain `module` while resolving and fully verifying real source data,
+then pass it as `preparedModule` to `createEngine`, with `scratchBoot: async () => shape`. The live
+host reuses that compiled module without a second Wasm fetch or compilation, and boots fresh DSP
+state at sample 0. Keep the same owned document and policy snapshot across preparation and live
+creation; the helpers copy inputs at each call boundary. `prepareBrowserSessionInWorker` is the
+corresponding primitive for custom Worker entries. This preparation proves no real source ready
+and performs no source delivery. The existing shape-only defaults and URL-only host remain valid.
+
+The SDK PCM feed accepts at most two fresh submissions per source per render callback and scans
+at most that source ring's capacity. Shared source runways remain available while internal queues
+fill gradually, avoiding a first-callback burst proportional to every queued source quantum.
+
 `await engine.console()` binds the same semantic console shown above to the shipped browser host.
 It resolves the browser session map once, then submits the same whole-batch edits over MessagePort.
 All eleven live command kinds are available without numeric rack, channel, parameter, or tap IDs;