feat(editor): imported audio as a first-class timeline region - #569
Open
EtienneLescot wants to merge 79 commits into
Open
feat(editor): imported audio as a first-class timeline region#569EtienneLescot wants to merge 79 commits into
EtienneLescot wants to merge 79 commits into
Conversation
Phase 1 of issue #350 (import voiceover / BGM / SFX). Adds the document model for timeline audio tracks without any UI, IPC, or export wiring yet. - Widen assetSchema.kind to enum(["video","audio"]) so an imported audio file (no video stream) has its own kind. Additive — every existing doc holds "video", which still validates, so no schemaVersion bump. - Add audioTrackSchema: a timeline-global track addressed in OUTPUT (post-trim/post-speed) timeline seconds, the same domain the compositor's concatenated programme PCM lives in. That invariant is what will keep the live preview and the export in sync in later phases. - Add document.audioTracks[] (defaulted, so pre-#350 docs load unchanged), the AxcutAudioTrack type, and a createAudioTrack factory. - Tests cover defaults, trim/position validation, factory round-trip, the kind widening, and that a document omitting audioTracks defaults to []. - Fixture fallout: 15 test files + browserShim build full AxcutDocument literals and now carry audioTracks: [] alongside their zoomRanges: []. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase 2 of issue #350. Wires up picking an external audio file (voiceover / BGM / SFX) and adding it to a project as a kind:"audio" asset. No timeline placement, preview, or export yet. - IPC: open-audio-file-picker mirrors the video picker but approves against a dedicated audio extension set (mp3/wav/m4a/aac/flac/ogg/opus). Factor the path approver into a shared approveReadableMediaPath so the audio and video approvers differ only by their extension gate — an audio picker must not approve a video path or vice versa. - document-service.addAsset takes a kind; an audio import validates against audio extensions and never claims the empty primaryAssetId slot, so a BGM file dropped into a fresh project can't become its primary (video) asset. Threaded kind through the bridge chain (contracts, client, nativeBridge, aiEditionService) and the browser shim. - projectStore.addAudioAsset imports the file, skips the camera-sidecar lookup addAsset does, and probes the real duration up front (new probeAudioDuration, the <audio> counterpart of probeVideoDuration) so the timeline can size the track in a later phase. - i18n: selectAudio / audioFiles dialog strings across all 13 locales (English placeholders for the untranslated 12). - Tests: document-service audio branch (kind, primary guard, extension routing), probeAudioDuration (shared harness, driven per media tag), and addAudioAsset (bridge kind arg, no camera lookup, duration stamping). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase 3 of issue #350. Adds the mutations that place and edit imported audio tracks on the timeline. Still no UI or preview/export — that's next. - New pure module document/audioTracks.ts: append / remove / move / trim / gain / mute, each taking an AxcutDocument and returning a new one. Audio tracks aren't clip-anchored (they float over the assembled programme in output-timeline seconds), so these are plain array edits with schema-valid guards — negatives floored, trimEnd pulled up to trimStart, NaN → 0. - useTimeline wraps them: addAudioTrack looks up the audio asset, places the head at the playhead (output time) by default, and returns the new track id for the UI to select; move/resize/gain/mute/remove each commit one history step. Refuses a non-audio or unknown asset. - Tests: the pure ops (immutability, guards, isolation) and the hook wiring (asset lookup, playhead placement, save, undo). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase 4a of issue #350 — the first user-visible slice. Import an external audio file and it lands on a timeline lane you can select and adjust. Drag-to-move and edge-trim are deferred to a follow-up (4b); reposition is via a numeric offset field in the inspector until then. - Media panel gains an "Import audio" button next to "Import media" (openAudioFilePicker -> store.importAudioAsset), which adds the asset and places a track at the playhead in one action. - Selection lives in the project store, not useTimeline's local state, because the media panel and the inspector are in different subtrees and both touch it; region/clip selection stays hook-local. The hook delegates addAudioTrack to the store and reads selection from it. - V4Timeline renders an audio lane (shown once a track exists) with a teal pill per track: the ClipWaveform reused as a background, windowed to the track's trim and scaled by its own gain, plus a label and mute glyph. Click selects. - The inspector shows an AudioTrackPane (volume / mute / start-offset / remove) in place of the facet when a track is selected, the same precedence a region selection gets. - i18n: importAudio / couldNotAddAudio (editor) and an audioTrack block (settings) across all 13 locales. - documentWriteAudit gains rows for the seven new save sites (two store, five hook), each classified by trigger; this audit should have been run in phases 2-3 and now is. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase 4b of issue #350 — the audio lane is now interactive. Grab the pill body to slide the track, the edge handles to trim: left moves the in-point (and the head, so the right edge stays put), right moves the out-point, capped at the source length. - New setAudioTrackPlacement pure op writes position and both trim points in one shot, so a left-edge drag (which changes timelineStartSec AND trimStartSec together) commits as a single undo step. Hook wrapper placeAudioTrack; documentWriteAudit row added. - startAudioDrag mirrors the region pills' drag: a local preview during the gesture, the same PILL_SNAP_PX magnet to clip boundaries and timeline ends, and one document write on pointerup. AudioLanePill grows two resize handles and moves on a body grab; selection happens on pointer-down. - Tests: the placement op's guards, and the drag itself (pointer→second math, single commit, in/out-point semantics) driven through the geometry harness. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase 5 of issue #350 — imported audio is now audible while editing. Each track plays over the video, positioned on the RAW virtual timeline where it was placed, at its own level. - VirtualPreview mounts one <audio> per track and syncs it in the existing 60Hz rAF loop: position from resolveTimelineAudioPlayback (playhead − timelineStart, offset by the trim in-point), play only inside the track's window, pause outside or when muted, and match the video's playbackRate so a speed region keeps A/V together. Level is the track gain × the global output gain via element.volume — deliberately NOT a WebAudio node, so the delicate primary/supplemental graph is untouched; a boost past 0 dB clamps in the preview but is still written to the export. - Threaded audioTracks + audioSources through Preview → PreviewCanvas → VirtualPreview. videoSources already resolves a URL for every asset, so it doubles as the audio source list (looked up by assetId); both props default to empty, so a project with no imported audio is unchanged. - A note on the coordinate system: tracks live on the RAW/document timeline (where addAudioTrack seeds timelineStartSec from the playhead), not the trim-compressed output timeline — corrected the Phase 1 comment's claim. The export will mix on the same RAW positions (Phase 6). - Tests: the sync math (window, trim offset, mute, untrimmed tail) and an rAF-driven integration test that the loop seeks + plays/pauses the element. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase 6 of issue #350 — imported audio now lands in the export, not just the preview. The native compositor mixes each track over the assembled programme. - audio.rs::mix_external_tracks overlays each track between assemble_concatenated_pcm and finish_audio: its trim window is decoded through the same decode_clip_audio path a clip's audio uses (48 kHz stereo), scaled by the per-track gain (the same 10^(dB/20) law as finish_audio), and summed in at its startSec offset. A track past the video end is truncated so audio and video stay the same length. The placement/gain/clamp math is split into overlay_track_pcm and unit-tested without ffmpeg (cargo test, verified on Linux). - scene.rs gains SceneAudioTrack + Scene.audio_tracks (a separate field, so SceneAudio stays Copy and the pipelines keep copying it out of a borrow). Wired into all three pipeline_{linux,macos,windows}.rs. - buildSceneDescription resolves each track to { path, startSec, gainDb, trimStartSec, trimEndSec, mute }. startSec is the raw timeline position — exact without trims/speed, an accepted approximation otherwise (the preview approximates trims the same way); trimEndSec is always concrete (the compositor preallocates the decode window from it). resolveSceneAssetPaths round-trips the JSON so the new field reaches the addon untouched. - Corrected the Phase 1 schema comment (tracks live on the RAW timeline, not output time) and documented the mix step in export-pipeline.md. Verified: compositor builds + `cargo test --lib audio` (14) pass on Linux; tsc (app+test), biome, and the scene-description tests (95) pass. NOT yet verified: the addon (.node) must be rebuilt with build:native:compositor:linux and an actual export listened to — the manual E2E this phase requires. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Addresses review feedback on #350: a separate "Import audio" button was both undiscoverable (added to only one of three media surfaces) and worse UX than just letting "Import media" take audio too. - open-video-file-picker is now a combined media picker: it offers video AND audio, approves whichever was chosen (video path first, then audio), and returns `kind` so the renderer routes an audio file to importAudioAsset (asset + timeline track) and a video file to addAsset (clip). - All three import surfaces route by kind: MediaStage (the main media view), MediaPane (chat side panel), and EditorEmptyState. The standalone "Import audio" button and its handler are removed. - Dropped the now-dead open-audio-file-picker IPC, its preload method/type, and the importAudio / couldNotAddAudio / selectAudio strings; added a mediaFiles dialog string across all 13 locales. approveReadableAudioPath and the audio extension set stay — the combined picker uses them. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two issues from testing the #350 import: - Jitter: the preview re-seeked each imported track whenever it drifted >25 ms from the playhead. The primary/supplemental audio can use that tight leash because it syncs to the <video>'s own authoritative clock; an imported track syncs to virtualTimeSec, which is DERIVED from that clock each frame and slightly noisy, so at 25 ms it re-seeked most frames and each seek briefly stalled the element — the jitter. Widen the leash to 300 ms while the element is playing (it free-runs in sync from the right offset; the wide leash only catches real scrubs / trim jumps), keeping the 25 ms leash for the paused/seek case. Music beds don't need frame-tight sync — that's the video's job. - Audio shown "along the recording": handleDropAsset (the media stage's "Add to timeline" button and drag) ran insertClipAt for ANY asset, so adding an imported audio asset built a video-style clip in the clip row on top of its lane track. An audio asset has no video and must never become a clip: route it to addAudioTrack instead, and reuse its existing track so the same file can't stack duplicate lanes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reworks the #350 import UX per testing feedback: the media tab arranges video CLIPS (it chains them), which is the wrong model for an audio overlay. Audio is now added the way an annotation is — a timeline action. - New "Add audio" tool in the timeline toolbar (music icon, next to zoom/speed/camera): opens an audio-only picker and places a track at the playhead via importAudioAsset. - The media tab is video-only again: open-video-file-picker reverts to video extensions, restored the dedicated open-audio-file-picker for the toolbar, and MediaStage / MediaPane / EditorEmptyState import video only. - Audio assets are hidden from the media lists (MediaStage + MediaPane) — they're managed on the timeline lane (select the pill to edit/remove), so they never appear as chainable clips. - i18n: audioTrack.add / importFailed, restored selectAudio, dropped the now-unused mediaFiles, across all 13 locales. The handleDropAsset guard (audio → track, never a clip) stays as a backstop. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Testing feedback on the #350 audio UI: - Move the "Add audio" toolbar button left of the first divider (grouped with auto-enhance, ahead of the region tools) instead of isolated at the end. - AudioTrackPane: header is now the generic "Audio track"; the file name moves into the body. Drop the mute button and the start-offset field (position and mute are handled on the lane), leaving volume + delete. The delete button now matches the region panes' danger-outlined style. - Remove the now-unused audioTrack.offset / mute / unmute strings across all 13 locales and correct the help text. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
More #350 UI feedback: - Move the "Add audio" toolbar button to directly right of "Add annotation" (the comment tool), rendered inside the tool row via a Fragment. - Rename the inspector's "Remove track" to "Delete track" and make the button full-width, matching the region panes' delete button. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per #350 feedback: label the slider "Output level" (reusing audio.outputGain, the same string the global Audio pane shows) and add a "Reset audio" button that zeroes the track's gain, styled like the global pane's reset. Drop the now-unused audioTrack.volume string. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
moveAudioTrack and resizeAudioTrack lost their only callers when the inspector's offset field was removed; the lane's edge-drag commits position and trim together through placeAudioTrack (setAudioTrackPlacement), so the separate position-only and trim-only ops were dead. Remove the two hook wrappers, the two pure document ops (moveAudioTrack, setAudioTrackTrim), their tests, and their document-write-audit rows. placeAudioTrack / setAudioTrackPlacement stay and still cover both edges. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The mute button was removed during UI review, leaving mute reachable nowhere — a working-but-unsettable flag with a dead branch in the Rust mixer. It was added on this branch and never shipped, so it comes out cleanly with no schema migration. Volume (down to -12 dB) plus delete cover the need for a simple audio overlay; a mute+solo pass can come back as its own feature. Removed end-to-end: audioTrackSchema.mute, setAudioTrackMute / toggleAudioTrackMute, the pill's mute glyph + .laneAudioMuted, the preview's mute gate, the scene's mute field (TS + scene.rs), the mixer's mute skip, and every test that exercised it. Rust (cargo test --lib audio, 14) and TS (1049 across ai-edition + native) pass; compositor addon rebuilt. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase 7 leftover for issue #350: the top-level-shape table enumerated every other document array but not the new audioTracks[]. Add the row. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Correctness / data integrity: - audio.rs: bound the decode window to the programme remainder and skip a track that starts past the end, so a long track pinned near a short export can't buffer hours of PCM (then discard it). - document-service.removeAsset: pass primary to the next VIDEO asset, never an audio overlay, and drop audioTracks that referenced the removed asset. - useTimeline: backfill a missing audio duration on load (a failed import-time probe otherwise leaves durationSec 0 → a zero-length, never- playing window), mirroring the video-dimension backfill. - projectStore.importAudioAsset: report failure when track placement fails, instead of claiming a successful one-shot import with no track. - V4Timeline: clamp the drag so a track's head/tail stay within the programme (no pill past 100%, matching the export's truncation). - useTimeline: clear region/clip selection after a successful audio-track insert (no concurrent selections); clear the inspector selection only AFTER a delete commits. Tests / quality: - Remove a new `any` cast in projectStore.test (use vi.mocked). - Translate the audioTrack / selectAudio / audioFiles strings in all 12 non-English locales. - Add coverage: removeAsset audio cases, the duration backfill, browser-shim audio import, and the decode-bound skip (Rust). Deferred with rationale (noted on the PR): schema v8 bump (audioTracks follows the repo's additive-no-bump precedent, transcriptionFailure); positive-gain preview via WebAudio nodes (heavy, risks the audio graph); moving audioTracksRef to an effect (matches the file's existing render-ref idiom). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`element.volume` is spec-clamped to [0, 1], so a track pushed above 0 dB played at unity in the preview while the export mixed it at full boost — the preview under-represented exactly the tracks a user deliberately turned up. Route each mounted track element through its own WebAudio gain node (source → trackGain → the existing output gain → destination), the same node type the primary/supplemental sum already uses to boost past 0 dB. The rAF sets each track's gain live, so a slider drag is picked up without rebuilding the graph; the effect re-routes only on a real mount/unmount (keyed on the resolved-track set, not the gains). The `.volume` path stays as the fallback for when WebAudio is unavailable (jsdom, a denied audio policy), where a boost still caps — audible, just not amplified. Effective level is trackGain × outputGain, matching the exporter's order (mix_external_tracks applies the track gain, finish_audio the output gain). Adds a preview test that stubs AudioContext and proves a +6.0206 dB track (×2) drives its gain node to 2 rather than clamping element.volume to 1. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The audio lane was the only timeline lane hidden until it had content, on
the premise (in a now-stale comment) that audio is imported from the media
panel "not a keystroke". Audio is a toolbar peer of the region tools now, so
give it what they have: the lane always renders and, when empty, advertises
the shortcut that fills it ("Press M to add audio") — exactly like the zoom,
trim, annotation, speed, and camera lanes.
Register `addAudio` on M (a free, mnemonic key) in the shortcut config so it
shows in the Shortcuts dialog and is user-rebindable, and handle it in the
editor shell. Unlike its neighbours it opens a file picker rather than
dropping a sized region at the playhead, so it takes no duration.
Lift the picker→import flow out of the timeline toolbar into
`useTimeline.addAudio` so the button and the shortcut share one path; the
toolbar button now calls `tl.addAudio()`.
i18n: add pressAudio / actions.addAudio across all 13 locales.
Tests: addAudio wiring (picker → import, cancel → no-op); shortcut-label
parity still holds.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…350) An imported audio track stores its head in RAW timeline seconds (seeded from the playhead), but the exporter mixes it onto the trim-COMPRESSED programme. The scene builder passed the raw head straight through as the output offset, so every cut ahead of a track delayed it in the render by exactly the removed duration — the reported "the following audio track was delayed by the trim duration once rendered". The preview never showed this because its playhead jumps across a trim, landing the track on time. Project the raw head onto the programme before handing it to the compositor: output(T) = T − (trimmed span before T), a new pure `projectRawTimelineSecToPlayback` that walks clips+trims with the same source-time model as `resolvePlaybackSegments`. Exact for trims; speed regions remain the pre-existing approximation. Regions with no trim — a project with no clips included — pass through unchanged. Corrects the two stale comments that claimed the raw positions "agree" with the export (they only did without trims). Adds unit coverage for the projector (before/after/inside a cut, multi-clip) and an end-to-end scene test pinning a track's head onto the compressed programme. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three findings on the recent commits: - timeline.ts (Major): `projectRawTimelineSecToPlayback` accumulated each matching trim independently, so OVERLAPPING trims were double-counted and RAW GAPS between clips were ignored — e.g. trims [2,5]+[3,4] mapped raw 6 to 2 instead of 3, and a track after an inter-clip gap landed late. Rebuild the projection from the SAME kept intervals as `resolvePlaybackSegments` (per-clip `subtractInterval`, shared output cursor), so the union of trims is counted once and gaps are removed exactly as the programme removes them. A raw head past the last kept frame still carries its overhang through, keeping the no-clips case an identity. Adds overlapping-trim and gap regressions. - useTimeline.addAudio (Minor): the import goes through the store's importAudioAsset, bypassing the hook's selection reset, so a region/clip selection could survive an import. Clear selection/multiSelection/ clipSelection on success (the same exclusivity addAudioTrack keeps). - useTimeline.addAudio (Minor): move the openAudioFilePicker await inside the try so a rejected picker reaches the localized toast instead of an unhandled rejection; a cancel stays a silent early return. Adds tests for the selection reset and the picker-rejection toast. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…, not output Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… export `resolveTimelineAudioPlayback` derived the track's source position from the RAW playhead, which JUMPS across an interior trim — so the `<audio>` element skipped that much of the file and the track ended early. The exporter's `audio::mix_external_tracks` instead overlays the decoded window `[trimStart, trimEnd]` contiguously at its projected offset, cutting nothing. The two disagreed silently: a trim inside a track's span desynced preview from export by the removed duration (Etienne's 10s-clip / raw-2..4-cut example: 2s). Move the function to OUTPUT-programme space — it now takes the playhead and the track head already projected through the trims, and the rAF projects both with `projectRawTimelineSecToPlayback`, the same function the export uses in sceneDescription. `local` is then continuous, so the track plays as one block exactly as the export mixes it. Adds a regression test on Etienne's scenario asserting source position 3 (not 5) at output time 3. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…aveforms survive a reopen
`get-audio-peaks`, `read-binary-file`, `get-readable-file-info` and
`read-file-chunk` gated on `approveReadableVideoPath`, whose extension allowlist
is video-only. On first import the picker approves the exact path, so the
waveform draws; but after a project reopen `approvedPaths` is empty, an imported
audio file outside RECORDINGS_DIR (e.g. ~/Music/bgm.mp3) fails
`hasAllowedImportVideoExtension`, the handler returns `{success:false}`, and
`useAudioPeaks` caches that as "no audio" — losing the waveform for good.
These four handlers serve whichever media the document points at, so gate them
on a combined `hasAllowedImportMediaExtension` (video OR audio) via a new
`approveReadableAvPath`. The type-specific import pickers are untouched and stay
honest.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Speed regions no longer retime imported audio in the preview. The rAF forced each track's playbackRate to the video's, so a voiceover under a 2× region played pitched-up and finished early — but `mix_external_tracks` sums the track at 1× (speed stretches clip PCM only). Pin imported elements to 1×. - The voiceover play path now resumes a suspended AudioContext, as the primary loop already does. A track starting while the primary element is silent (span over, or a recording with no separate audio) otherwise routed into a suspended context and played nothing, with no error. - The raw→output projection for a track's head now walks the same path-filtered clips the programme is assembled from (`resolveVisibleClips`'s filter). Walking the full `document.timeline.clips` counted a relinked-away clip the programme omits, landing every following track past the real programme end. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Deleting an audio track now also drops its orphaned asset. An imported audio asset is only reachable through its track (audio never becomes a clip), so a delete left it in the document forever, invisible in every asset list. - Clear a stale `selectedAudioTrackId`. An undo can remove the selected track without going through `removeAudioTrack`, leaving the inspector open on an empty AudioTrackPane recoverable only by clicking a facet; a small effect resets it. - Reset the volume pane's `liveGain` when the selected track changes, so an uncommitted drag on one track doesn't display as the next track's gain. - Reset `audioDragRef` at the start of a lane drag, so a select-click landing while a prior drag's `placeAudioTrack` is still in flight can't re-commit it. Adds removeAudioTrack asset-cleanup tests (orphan dropped; shared asset kept). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Dedupe probeVideoDuration/probeAudioDuration into one tag-parametrized probeMediaDuration. They were byte-identical but for the element tag; every property touched is on HTMLMediaElement, so a settle/cleanup/timeout fix can no longer drift between them. The two exports stay, so callers/tests are unchanged. - Mark every audio-backfill candidate probed BEFORE the first await. Marking each only as its turn came let a document change that re-entered the effect mid-probe re-probe the still-queued assets. - Scale the audio-lane waveform by the track gain AND the project output gain, as the export's finish_audio does (it applies both and clamps). Scaling by the track gain alone under-drew a boosted output, hiding clipping the file will have. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Brings #526's clip-anchoring model onto #502, per review on #526. #502 keeps its document shape, its native mixer and its output-space preview; audio tracks stop floating at an absolute raw second and travel with the content they were placed over through reorder, trim and delete. - `audioTrackSchema` restated on the v5 clip-anchor contract: `{startMs, endMs, ...clipAnchorShape, offsetMs, gainDb, …}`. `offsetMs` replaces the `trimStartSec`/`trimEndSec` pair — the track's own span already says where it stops, so the tail trim no longer needs storing twice. - `audioTracks` joins `mapAllRegionCollections`, `RegionKind` and `removeRegion`, so every structural clip edit re-derives audio the way it already re-derives zoom and annotation, and a track can be copied and pasted like any other pill. - `document/audioTracks.ts` drops its hand-rolled array ops for the shared pill helpers; the lane renders `collapseTracksToPills` instead of one row per stored track. The fragment problem, which the review called out as unsolved in both PRs: `anchorRawRegionsToClips` copies a region's payload verbatim into each fragment. That is right for value-per-span effects — both halves of a split zoom are still "depth 3" — and wrong for continuous media: two fragments each holding `offsetMs: 2000` both restart the file two seconds in, so a bed spanning a cut audibly restarts at the boundary. `anchorAudioTrackFragments` advances each fragment's `offsetMs` by the source time its predecessors consumed, so the pieces play as one continuous take. Fragments share a `trackId`: the lane collapses them to one pill, the inspector edits the group, and delete takes the group. Also folds the path-resolvability rule that decides which clips make the programme into one predicate shared by `resolveVisibleClips` and the audio projection, rather than two copies that could disagree — the review's `audioLayerTimeline.ts` point, landed one level down from where it pointed: `resolveVisibleClips` returns trim-COMPRESSED segments, and `projectRawTimelineSecToPlayback` subtracts the trims itself, so feeding it those would apply them twice. Claude-Session: https://claude.ai/code/session_01DvB2GJXp18fPr79DHfk356
Adds the payload the anchored track schema was missing: `fadeInMs`, `fadeOutMs`, `loop`, `muted` and a `kind` discriminator, wired through the inspector, the preview and the native mixer. - The inspector gains fade-in, fade-out, mute and loop next to the existing volume slider. Fades cap at 5s there; past that a fade reads as a level change rather than a fade. - `anchorAudioTrackFragments` keeps `fadeInMs` on the first fragment and `fadeOutMs` on the last, so a track split by a cut fades once at each real edge instead of at every boundary. Looping tracks are exempt from the offset advance: they fold within `duration - offset`, which every fragment shares, so advancing would drift them out of phase with the mix. - The scene builder drops muted tracks and emits one mix entry per repeat for a looping one, carrying the fades only on the pieces that touch the track's real edges. - Fades reach the compositor as `fadeInSec`/`fadeOutSec` and are applied by a new envelope in `overlay_track_pcm`, measured against the DECODED length so a track truncated at the programme end does not ramp down over audio the render never reaches. - `resolveFadeSecs` reduces fades that do not fit their span — in proportion rather than clamping each independently, which would turn an asymmetric pair symmetric. An unreduced fade-in longer than the span is worse than cosmetic: it holds the gain at zero for the whole track. Mirrored by `resolve_fade_samples` in `audio.rs` so the preview and the render cannot disagree about how a fade gets shortened. Two things found while wiring this up: - `mix_external_tracks` clamped per-track gain at ±12 dB, but that is the project OUTPUT trim's range; the track schema's own is -60..+12. Every quiet bed was floored at a tenth of the attenuation asked for. Widened, with a test — the existing clamp test covers `finish_audio`, a different path, still bounded at ±12. - Two `SceneAudioTrack` fixtures construct the struct literally and needed the new fields to keep compiling. Claude-Session: https://claude.ai/code/session_01DvB2GJXp18fPr79DHfk356
The one audio gesture #502 has no equivalent of: it is import-only. Music and other files keep coming in through the toolbar's `addAudio` file import, so this dialog exists only for recording, which has live state to show. - `save-recorded-voiceover` IPC writes the MediaRecorder blob under the recordings dir, so a take outlives the session like any other asset. Capped at 512 MB — this writes renderer-supplied bytes straight to disk, and an hour of Opus is a few tens of MB, so the cap refuses a runaway payload without ever being reachable by a real take. - `V` records a voiceover from the playhead. The video plays while the take runs so the user can narrate what they see, and recording stops itself at the end of the timeline. Two bugs the review found in this flow on #526, both fixed rather than carried over: - Every take landed one full take-length to the right of where it was spoken. Recording plays the video, so the live playhead advances for the whole take, and placement read it at the END. The shell now captures the playhead when recording STARTS — and re-captures on Record, since the user may scrub after opening the dialog. - Tearing the dialog down mid-take (project close, shell unmount) stopped the microphone stream but never the recorder, so `onstop` never fired: the take was dropped and the video element left playing. The cleanup now stops the recorder and discards the blob, since nobody is left to place it. Claude-Session: https://claude.ai/code/session_01DvB2GJXp18fPr79DHfk356
A voiceover is speech. It has words, it sits on the timeline, and the transcript
tab could not see any of it — the tab was wired to `timeline.clips`, so the one
lane it could read was the one that came out of the camera. Importing narration
meant losing every tool the transcript gives you over it: trims, word edits, the
agent's grounding, captions.
The aggregation is now parameterised by lane. `TranscriptPlacement` is the unit
it actually runs over — one stretch of ONE asset's source time, laid somewhere on
the ruler. `AxcutClip` is one provider of that shape and was for a long time the
only one, which is why everything downstream is still named after clips;
`voiceoverPlacements` is the second.
Deliberately structural rather than a union of the two record types. Nothing
below the aggregator needs to know which lane a section came from, and the moment
it could ask, something would start behaving differently per lane — which is the
one thing this parameterisation exists to prevent. Sections, word ids, trim runs
and cue lookup are byte-for-byte what they were.
Three things the voiceover provider gets right that a naive map would not:
- It windows the source by the FRAGMENT's offset. A track spanning a cut is
ventilated into a fragment per clip, each with `offsetMs` advanced by what
its predecessors consumed. Collapsing them back into one pill here would
re-read the file from its head on the far side of every cut.
- It leaves music out entirely, rather than filtering it downstream. Music is
not transcribed at all, so a music placement could only ever produce an empty
section that reads as a failed transcription.
- It ignores `loop`. A transcript that says the same sentence three times is
not a transcript of anything.
The switch shows up only when there is a voiceover to switch TO — a one-sided
control asks a question about a lane the project does not have. It renders in
both of the pane's states, and the active lane is DERIVED rather than reset in an
effect: deleting the last voiceover pill while reading it must not leave the pane
addressing a lane that is gone, and an effect would render that empty state once
before correcting it. `hasAnyTranscript` moves from the document to the lane, so
"imported a voiceover, not transcribed yet" says so instead of showing a blank.
Captions still render the recording lane. Making them follow the choice is the
rest of #560 phase 1, and it needs somewhere shared to put the choice — this one
is local because it has exactly one reader today.
Refs #560.
…ndently of language tag
`setWordText` gave the transcript a way to be corrected, but nothing said a word had been. Two consequences: a transcription run REPLACES the asset's transcript, so a user who fixed twenty proper nouns and regenerated lost all twenty without a word about it; and there was no revert, because nothing kept what the transcriber had originally said. `wordSchema` gains the pair that fixes both — `originalText`, captured the first time a user rewrites the word and never overwritten afterwards, and `source`, which also reserves the `"synth"` value a word with no audio behind it will need. Both optional and absent from every document written before them, like `cameraTrack.width`: additive, so no schema bump. `document/transcript.ts` is their only writer and keeps them consistent — typing the original back clears the pair, which IS the revert. `carryOverWordEdits` then re-applies the corrections onto a fresh transcript, and the transcription store calls it on the one path where a run lands. The match is deliberately strict: same original text, overlapping span, one new word per correction. A correction is carried only when the run reproduced the very same mistake at the very same moment, so re-transcribing in another language carries nothing rather than stamping the old language's corrections onto the new words. What could not be carried is counted, not guessed at — saying so to the user needs a string in thirteen locales and belongs with the editing UI, so for now it is a warning in the log. `withTranscript` moves here from `transcribe.ts` and gains `setDocumentWordText` beside it. The document carries the same transcript twice (the per-asset entry and the legacy `transcript` mirror), and writing one without the other leaves the mirror serving pre-edit text forever — the failure that closed #469. It is a pure document operation; the Whisper adapter was not the place to look for it.
The transcript pane looked like a document and behaved like one only halfway: Backspace on a word wrote a trimRange and cut the media, while typing was blocked outright — `handleBeforeInput` said so in a comment, and correcting a mis-transcribed word had no in-app path at all. The two gestures now share the one word stream, with no mode to remember and no second tab: Backspace still cuts, and a double-click opens the word for editing in place. Enter or a click away commits, Escape abandons. A correction writes `transcript.words[].text` and nothing else — the captions follow it, the timeline does not move. Everything the field raises is stopped at the field, so a Backspace inside it types instead of cutting the clip out from under the caret. A corrected word says so: the accent colour, a dotted underline, and a tooltip naming what the transcriber actually heard. Hovering it offers the revert, which writes the original back through the same path that clears the provenance pair — there is no second "unedit" operation that could fall out of step with the first. It sits where the bin sits on a cut word, in the accent rather than the danger colour, and the two never appear together. Emptying a word is how a junk token leaves the captions without the audio going with it, and the caption pipeline already drops empty words. Rendered as its own text, though, such a word is a bare space: invisible, un-clickable, impossible to undo. It gets a chip instead, so it keeps a place in the stream. Writes go through `setDocumentWordText` on the same serialised queue as the trims, so correcting a word and cutting the next one cannot overwrite each other's save. Verified end to end in the browser preview: the edit lands on the word, rebuilds its segment, reaches the legacy `document.transcript` mirror, and the revert takes all three back.
The third gesture on the word stream, and the one that had to get past a guard. Put the caret between two words, type, and a field opens beside the word you were on; Enter turns it into a real word, in amber, marked `source: "synth"`. It has no audio, so its own control deletes it rather than trimming — there is nothing for a trim to remove — and Backspace over a run of nothing but inserts does the same. The guard it got past did not work. React 18 builds `onBeforeInput` from the legacy `textInput` event, whose `TextEvent` has no `inputType`, so the block's handler threw `Cannot read properties of undefined (reading 'startsWith')` on every character typed, never reached its own `preventDefault`, and let the character land in the contentEditable — the exact desynchronisation between the DOM and `words` it was written to prevent. Confirmed in the browser before touching it. It now listens to the native `beforeinput`, where the event really is an `InputEvent`, ignoring what the two nested fields raise on their own. An inserted word takes the silence it is dropped into and nothing else: from the word it follows, up to what its text needs to be read at subtitle pace, and never past the word that comes next. Between two words that run into each other it has no duration at all and rides their caption line, which is where it reads correctly anyway. Dropped into a pause it shortens the `[silence]` pill by exactly what it took. That span is also the slot a synthesized voice will have to fit, when there is one. `wordsInRange` had to learn about a word with no span — an overlap test excludes a point at either edge of the range, which silently lost every word inserted at the very start of a clip — and `withSilenceGaps` now breaks the resulting tie on array order rather than leaving it to sort stability nobody had written down. A run also carries inserts forward now, not just corrections. They have no original text to recognise, so time places them: each goes back after whatever the new transcript ends last before it. `carryOverWordEdits` counts what it could not place, as it already did for corrections. Verified end to end in the browser preview: typing leaves the block's own text untouched, the word lands in `words`, in `segment.wordIds`, in the rebuilt segment text and in the legacy mirror, an insert into the silence shrinks the pill from 2.6s to 2.2s, and deleting it puts the pill back.
…frame
Adding a word only borrowed whatever silence happened to be free, so a word
dropped between two words that run into each other got no time at all. It now
creates the time it needs: the clip splits at the word's edge and a held-frame
clip carries the deficit, so the timeline grows and everything downstream
shifts. Screen and webcam freeze together — both are derived from the one asset
source clock the freeze stops advancing — and the decoder is paused for the
duration rather than free-running past the held frame into what comes after.
That created span is the slot a synthesized voice will speak in.
The gesture is gated to dev builds until there is a voice to put in it. A silent
freeze frame is not a feature, and captions are not always on, so a release
build offers no way to add a word at all — the pane does not even advertise it.
Drop the gate in `openInsertion` when TTS lands.
Three things the split broke, found by running it rather than by reading it:
The captions went DARK over the pause. A line straddling the split was
ventilated once per half, each half carrying the whole line, so the caption
played, blinked out for exactly the pause the word exists for, then played again
from the top. A line covering the held moment now covers the pause too, and
spans that meet on the ruler coalesce — measured before and after: two cues of
"Bonjour on va parler de vraiment Kubernetes" became one, 0→3.9s.
The word rendered TWICE in the pane. The freeze claims it and the half that
starts at the same moment matched it as well. The freeze owns it: it is the
section the playhead is inside while the pause plays.
And one word turned one recording into three headed blocks, each announcing the
same filename over a sliver of timecode ("Clip 2 · 0:02.5—0:02.5"). Sections
that continue the one before — same media, meeting on both clocks — now flow
inline under a single header spanning the whole run. Two clips over one media
still get a header each, which is the case the header exists for.
The pane also says what its gestures are now: double-click corrects, Backspace
cuts, and (in dev) typing between two words adds one. They were invisible until
tried.
Tests cover the freeze end to end: the document split, playback keeping the
created time, the source clock held still through it, the pane showing the word
once, the header run, and the caption playing once straight through.
Splitting the clip to make room for an inserted word did not survive the rest of the app. A project saved after two inserts came back with one clip named `clip_..._fzA_fzA` — split twice, both freezes and both after-halves gone, the full source range restored onto the mangled id — and zero synthesized words. The inserted text was lost on save, and the held-frame clips that did exist were skipped in playback. That is not a bug to chase. Clip surgery for a caption-only feature puts the timeline's shape under the transcript's control, and every other writer of `timeline.clips` — the duration probe, the recording import, resequencing — is entitled to disagree with it. `frozenSec` goes with it, and so do the readers that had to special-case a clip whose source window is a single point: the playback segments, the raw span, the native position clamp, the decoder pause, the caption ventilation, and the pane's split-clip header run. What stays is the part that was always true on its own: an inserted word is a word in the transcript with `source: "synth"`, it borrows whatever silence is free where it lands, and it reaches the captions. The gesture stays dev-gated — now for a second reason, since without created time an inserted word can only speak inside a pause that already exists. Creating time is still the right answer once there is a voice to put in it. It belongs in a record of its own, beside `trimRanges`, which is the inverse operation and the one shape the timeline already threads everywhere.
… new An added word was visible only in the transcript pane. On the timeline — where you decide what the film does — nothing said a moment carried text with no audio behind it. Each one now gets a thin amber tick on its clip's own track, at the moment it sits on, carrying its text in the tooltip and seeking to it on click. Amber is the colour the pane gives the same word, so the two read as one thing. Derived, never stored: the mark is computed from the transcript's `synth` words on every render, so there is no second record to fall out of step with the first, and nothing for another writer of `timeline.clips` to lose. Positioning is a percentage inside the clip's own box rather than an absolute ruler offset, so a mark travels with its clip through a reorder without arithmetic of its own.
The agent could read the transcript and cut it, and that was all. Asked to fix a misheard name — the request the whole transcript-editing feature exists for — it had exactly one tool that touched a word, `addTrim`, which removes the audio along with it. It would either do the destructive thing or say it had no tool, while the app had had the operation for three commits. `setWordText` writes `transcript.words[].text` through the same document function the pane uses, so the captions follow and the timeline does not move. Empty text blanks the word, which is how a junk token leaves the captions without cutting the speech around it. It is registered as a mutating tool: it writes the document, so it passes the consent gate like every other edit. It needed a read to address anything. `getTranscript` answers in SEGMENTS, whose ids live in a different namespace than the words and are refused by name — the trap a model would fall into first, so the refusal says which read hands out the right ones. `getTranscriptWords` is that read: id, text, span, and — only when the word is not plain transcription — where it came from and what the transcriber had originally heard. It takes a span, because a half-hour transcript is ~70k tokens and fixing one name should cost one phrase. Inserting a word is deliberately NOT exposed. The gesture is dev-gated in the UI until a voice can be synthesized for it, and handing the model a tool for something a release build refuses to do would be the same dead affordance the pane was careful not to advertise.
Adding a word only ever borrowed the silence that happened to be free where it landed, so a word dropped between two words that run into each other got no time at all — nothing to see on the timeline, nothing for a voice to speak into. The pause is now a record of its own: `timeline.insertRanges`, anchored in source time, shaped like a trim and stored for the same reason. It is the inverse operation, and a region is the one shape this timeline already carries safely from end to end. The first attempt made CLIPS for it and lost both the pauses and the words they belonged to, because every other writer of `timeline.clips` — the duration probe, the recording import, resequencing — is entitled to disagree with a clip it did not make. Stored, but with one writer and one invariant. `withInsertRangesForWords` runs after every word write and is the only thing that touches the array: it adds the pause an added word needs, resizes one whose text changed length, and drops the ones whose word is gone. `insertRangesMatchWords` is that same rule read back, so a test holds the writer to it rather than trusting it. A pause under 50ms is not stored at all — a few frames of held image is a stutter, not a slot. `timeline/inserted-time.ts` is the arithmetic the readers will need, pure and on its own: where a pause lands on the ruler once projected through the clip that plays its moment, and the pair that converts between stored raw seconds and the seconds the user actually scrubs. They are inverses everywhere except inside a pause, where a stretch of ruler stands for one held source moment — so the collapse answers with that moment AND says it is being held, which is what a caller driving a decoder needs to park rather than seek. Nothing reads it yet. The ruler, playback and the captions come next; this is the record and the arithmetic they will share.
The record and the arithmetic existed; nothing consulted them, so the added time was real in the document and invisible everywhere else. Now every reader that measures the timeline measures the same expanded ruler. Playback holds the frame. `resolvePlaybackSegments` cuts the kept span open at each pause's moment and puts a held segment between the halves — the stream plays up to that frame, stays on it, then carries on, which is what makes the film longer. It carries `heldSec` on `PlaybackSegment`, a type that exists only on the derived shape: nothing can write the field to a stored clip, which is the whole structural difference from the attempt that made clips for this. A pause whose moment a trim removed is never emitted — that moment is not in the film any more, so neither is its pause. The decoder holds with it. `resolveNativePosition` clamps the source clock inside a held segment, and the transport pauses the native side for its duration; free-running would play what comes after while the app clock, which does traverse the pause, re-seeks on the drift and stutters. Screen and webcam hold together because both derive from the one asset source clock the pause stops advancing. The ruler counts it. Total, ticks, clip boxes, lane pills and the playhead are all placed through `expandRawSec`, so nothing drifts from anything else by the added time. The amber mark on the clip becomes a BAND exactly as wide as the time it bought — a word that fitted in silence already there adds nothing and stays the hairline it was. Stored clip geometry is never rewritten for any of this; only what is drawn moves. The captions follow. Expanding both ends of a line's ruler span does the whole job: a line after a pause slides along by it, and a line covering the held moment has only its end pushed out, so it stays on screen through the pause instead of going dark over the one moment an added word exists for.
#570 (word edits) and #569 (imported audio) both rewrite the transcript pane, and #560's lane selector sits on top of both: the tab has to know which lane it reads before a double-click can correct a word in it. Testing either alone stopped being useful. Merged rather than rebased. This branch already carries merge commits, so a rebase flattens them and replays conflicts that were resolved weeks ago — which is exactly what a first attempt did, re-adding a `MediaList` to LeftPanel that `feat(editor): add audio from the timeline toolbar, not the media tab` had deliberately taken out. Where the two met: - The transcript pane. #570's `<Pane>` structure wins — it owns the gesture hint and the word-edit callbacks. #569 contributes the caption-settings action and the lane switch, which goes ABOVE the hint so the hint always describes the stream directly under it. The empty-state guard becomes `placements.length`, so it answers for the lane being read rather than for the recording. - The locale files. Merged as OBJECTS, not as text: both branches append keys to the same blocks, so every one of those 26 conflicts was a union git could not see, and hand-editing them is how a dropped comma or a doubled key gets in. `transcript.help` takes #570's copy — it is the one that mentions double-click, which now exists. - Fixtures. Each branch's pane tests learn the other's props, and the audio lane fixture gains `transcripts: []` for the amber added-word marks.
Everything that reads the timeline answered that question its own way, and the
answers disagreed. The transcript pane asked it by IDENTITY — does a trim name
this clip — which is a question a voiceover placement can never answer yes to:
it carries an audio fragment id and an audio asset, while every trim carries a
video clip. So the voiceover lane read every word as kept, including words whose
moment had been cut out of the film, and a cut authored from that lane removed
nothing at all.
The fix is not a better identity test. A trim is a removed span of the RAW
RULER, and both lanes lie on that one ruler: a word is removed if and only if
the raw moment it occupies is. `programme-time.ts` is that reading —
`keptRawSpans` / `removedRawSpans` / `removalAt` / `subtractRemoved`, derived and
never stored. Storage does not move: a trim stays source-anchored to a clip.
`keptRawSpans` is LIFTED out of `projectRawTimelineSecToPlayback`, which now
calls it, rather than reimplemented beside it — so agreement with playback is by
construction. Deriving it from `trimToTimelineSpan` instead would have been the
trap: that function's un-anchored branch resolves a pre-v7 trim through the
first clip whose source range contains its START, while the playback walk cuts on
OVERLAP across every clip of the asset. A primitive built on it would have left
the second clip's words reading kept over film that is gone — a new bug on the
lane that is correct today. There is a test named after exactly that.
Two boundaries do not follow from the definition and are pinned:
- The programme ends at the last CLIP's raw end, not the last KEPT span's. A
trimmed tail is genuinely removed; raw time PAST every clip is not removed
but unfilmed, because the projection is the identity there. That is what
lets a voiceover hang off the end and keep playing.
- A gap between clips is removed with NO trim ids. Nothing plays there, so a
word over it is not in the film — but there is no pill to restore, and a
caller offering that affordance must key it on `trimIds` being non-empty.
`Interval` and `subtractInterval` move to `timeline/intervals.ts` and are
re-exported, because the dependency runs document/ → timeline/ and importing
back would close a cycle. A second copy of those twelve lines was the
alternative, and two implementations of "what survives a cut" is the shape of
bug this change exists to remove.
The randomised fixture checks the SUM against `resolvePlaybackSegments`, which is
blind to order — so it also projects every span's head and expects the output
length of everything before it, which only holds if the walk yields them in
playback order. Reversing the output fails three assertions.
Refs #560. Step 1 of 7.
The transcript pane decided kept-or-removed by asking whether a trim NAMED the
placement — `trimAppliesToClip`, matching a clip id, else an asset id. A
voiceover placement carries an audio fragment id and an audio asset; every trim
carries a video clip. They never matched, so the voiceover lane read every word
as kept, including words whose moment had been cut out of the film.
`buildClipSection` now takes the removed set from `removedRawSpans` instead of
the trim rows, and tags each word by the RAW moment its centre occupies. Centre,
not overlap — mirroring the rule the identity filter used, so the recording
lane's answers do not shift. Both lanes read one set, so a cut on either greys
the other, and the existing shared-media guarantees are unchanged: the same
per-clip walk decides both.
`ClipWord.trimId` and `TrimRun.trimId` become `trimIds: string[]`. Two
consequences worth naming:
- Overlapping trims merge into one hole carrying both ids, where they used to
split into one run per attributed trim. Restoring still takes one click per
trim, as before — step 4 replaces the single-id op with one that drops the
whole pill.
- A gap between clips is removed with an EMPTY set. It is missing from the
film, so its words are struck through; nothing took it, so there is no bin
icon to click. The affordance is keyed on the set being non-empty.
`findCueWordId` takes a raw second instead of a clip id plus a source second. A
clip id is something only the recording lane has, so the voiceover lane never
highlighted anything at all. Raw time also settles the case the clip id was
introduced for — two clips over one media have identical source ranges but
different raw extents — and `locateVirtualPosition` drops out of the pane.
A LOOPING voiceover now contributes no placement. `anchorAudioTrackFragments`
deliberately does not advance `offsetMs` across a looping track's fragments, so
their words map to raw moments the words do not occupy: the lane would read
kept-or-removed on false evidence and, after step 4, author a cut in the wrong
place.
Forcing `removalAt` to answer null fails 8 assertions across the two aggregator
suites, so the fixtures are exercising the rule rather than agreeing with it by
construction.
Refs #560. Step 2 of 7.
…tail Step 2 made the transcript pane strike through a voiceover word whose moment the film had lost. The mix went on playing it anyway — shifted earlier, because an audio track is laid on the finished programme as one contiguous block — so the red was a lie in both the preview and the export. A voiceover is now SLICED by the cuts: `subtractRemoved` over its raw span gives one mix entry per surviving piece, each reading the source seconds it actually covers, so the two seconds a cut took are never heard. Fades stay on the TAKE's outer edges rather than reappearing at every piece. Music deliberately keeps the old path, byte for byte. A bed plays through a cut and ends early — slicing it at every edit is a musical regression the current code avoids on purpose (see the comment it already carries), and a bed has no words whose redness has to be true. This kind-dependence is the one place the design departs from a single shared projection, and it is the reason the departure is worth it. A LOOPING voiceover also keeps the old path. Step 6 refuses that combination outright, and inventing semantics for something about to be banned would be the worse answer. `resolveVoiceoverPlayback` is extracted next to `resolveTimelineAudioPlayback` rather than left inline in the rAF, because a decision that has to agree with the export is a decision worth testing. It asks the question in RAW seconds, which is both simpler and exact: no projection to invert, and the same question `removedRawSpans` answers for the words themselves, so the two cannot drift. Preview and export are held to each other by walking the take frame by frame and collecting the contiguous runs of source time the preview would play, then asserting they are the entries the scene description emits, piece for piece. Letting the take play through the cut fails two of those assertions. Every existing audio fixture is `kind: "music"`, which is why none of them moved. Refs #560. Step 3 of 7.
Backspace in the transcript pane handed the write site the asset and clip the words belonged TO. On the voiceover lane that is an audio asset and an audio fragment, so the row it wrote matched no clip: `resolvePlaybackSegments` removed nothing, and the word turned red while the film, the preview and the export were all unchanged. The lie step 3 made audible is now impossible to write. `TrimTarget` is gone. Both cut gestures emit a RAW span, and `ventilateTimelineSpanToTrims` resolves the clips actually under it at the write site — the same primitive a zoom straddling a boundary uses, so one gesture becomes several rows and stays one pill. The span is CLAMPED to the placement's own raw extent before it leaves the pane. `wordsInRange` admits a word by overlap and consecutive fragments have touching source windows, so a word straddling an edge would otherwise reach past its placement — and ventilation walks every clip a span touches, so the overspill would cut the head of a neighbouring clip the user never pointed at. A span over no film REFUSES, with a reason. Falling back to the nearest clip — which is what `trimToTimelineSpan` does for an un-anchored row — would remove something nobody asked to remove. Both handlers moved off `applyTimelineOp` onto `enqueueTimelineWrite`, and read the document INSIDE the chain. `tl.setTrimEntries` reads `useProjectStore.getState().document` unqueued, so correcting a word and immediately cutting the next one would let the word edit overwrite the cut — the exact failure the chain exists to prevent. They are declared in the write audit with the trigger they carry. `applyTimelineOp` has no callers left here; the `add_trim_range` / `remove_trim_range` ops stay for the agent, which addresses clips rather than moments. Restoring drops every row of the pill through `dropTrimPillsByIds`, instead of the one id a run happened to carry first. The insert gesture is refused on the voiceover lane, with its reason shown: `insertRangeSchema` has no `clipId` and the pause an added word buys is a held CLIP frame, so a voiceover placement has nothing to hold. Refusing beats writing a record nothing can read. Refs #560. Step 4 of 7.
Which lane the transcript is read from was React state in the pane. Captions
could not follow it from there: `buildSceneDescription` takes the document as its
only input, and one of its callers is the headless CLI exporter. A lane living in
a component would have captioned the preview from one lane and the exported file
from the other, with nothing anywhere to notice.
It becomes `captionLane` on `CaptionSettings`, stored through the same façade as
`language` — which is the precedent, and stored for the same reason: it decides
the text burnt into the file. `legacyEditorSchema` is passthrough, so this is
additive: no schema bump, no migration. It is read through the enum guard, so a
hand-edited blob cannot inject a lane the placements would not recognise.
`resolveCaptionLane` carries the fallback and lives in the PURE layer. That
fallback was React-local, and the export path never runs React: a project that
stored "voiceover" after its last take was deleted would have exported zero
captions while the pane calmly showed the recording's.
Two things deliberately did NOT move:
- `rulerInserts` stays CLIPS-derived on both lanes. A pause is a held clip
frame — it lengthens the ruler under everything, including a take laid over
it. Feeding it placements would measure the pause against the take and land
every voiceover cue early. There is a test that only passes because the
inserts still resolve through the recording.
- No removed-word filter in the cue stream. A cue inside a cut maps to frames
playback never emits, so it is invisible already; dropping the words would
re-flow every line boundary on any project with a trim — a visible change to
output, bought for nothing.
`transcriptRelevantAssetIds` widens to the UNION of both lanes: "can this project
be transcribed" is not a per-lane question, and narrowing it would report "no
transcript" on a project whose other lane is full of words.
Also fixed here: the lane switch has been rendering UNSTYLED since it landed. The
script that added its CSS threw on an unrelated edit before reaching the
stylesheet, and every assertion I had was on roles and text, so nothing saw it.
The switch now carries its styles and a line saying out loud that the choice
decides the exported captions — a user must never be surprised by which lane
their subtitles came from.
Refs #560. Step 5 of 7.
"The voiceover" has to name ONE thing for the transcript tab's lane switch to mean anything. It did not: `packAudioTrackRows` packs by overlap regardless of kind, and seven hand-rolled `[...others, ...fragments]` splices could each put a second take on top of the first — two takes recorded from the same playhead landed exactly there. `placeAudioTrackInDocument` is the one door now. The mode is the gesture: "resize" stops the dragged EDGE at the neighbour, "move" keeps the DURATION and parks the pill against the wall, "create" queues behind what is already there. A take must never be silently cropped because it was dragged somewhere crowded — the user asked to move it, not to shorten it. Deliberately NOT `regionIdentityKey`: `assetId` is in `NON_IDENTITY_FIELDS`, so two different voiceover files with matching payload hash the same and would MERGE — splicing two takes into one pill. There is a test named after it. Only same-kind pills clamp. A voiceover over a music bed is the normal case. Rows are per kind, with the packer UNCHANGED inside each kind: that keeps a document written before this rule legible instead of stacking its pills on each other, and a kind with no tracks takes no row, so the common single-bed project stays exactly as tall as it was. Loop is refused on a voiceover. `anchorAudioTrackFragments` does not advance `offsetMs` across a looping track's fragments, so its words map to raw moments they do not occupy — step 2 already drops it from the lane, and step 3 has no slicing branch for it. Music loops; narration does not. The loop FILL now stops at the next pill of its own kind rather than the programme end, so a bed filling the timeline cannot swallow a second bed behind it. The repair lives inside `mapAllRegionCollections`, not at its four call sites, so no structural edit can skip it. `reanchorAudioTracks` runs first — the generic pipeline copies `offsetMs` verbatim into every fragment, which corrupts a split take's offsets, a live bug unrelated to lanes that this walk was already causing — then `separateAudioLanes` pushes any same-kind pill whose head landed inside its predecessor forward. Deterministic, order-preserving, idempotent, and it cannot lose audio: a pill pushed past the programme end still plays, because removal is defined by trims and gaps only. Repair, never refusal. A schema refine here would turn an ordinary clip drag into a thrown save and make every existing document with overlapping same-kind pills unloadable. Refs #560. Step 6 of 7.
Commit `b9e0f1ff` let the transcript pane author a cut from the voiceover lane while still anchoring it on whatever the words belonged to — an audio asset and an audio fragment. `resolvePlaybackSegments` matches no clip for such a row, so it removed nothing from the film, the preview or the export. All it did was strike the word through. Those rows are now dropped once, at load, in `migrateRawDocumentToCurrent` — the single funnel every read path crosses. Now that both lanes read one removed set, leaving them behind would keep striking words through for a cut that never happened. The test is exact and needs no clip lookup: an audio asset is never a clip's `assetId`, because audio is filtered out of the lists that make clips. So a trim naming one can only have come from that build. An un-anchored pre-v7 trim names a VIDEO asset and is untouched; so is a trim whose clip was deleted, which in-session undo can still bring back. No `schemaVersion` bump: nothing about the format changed, and no output moves — these rows were already inert. What changes is that words the user "deleted" on that build come back as kept. That is the correction, and it belongs in the release note. `audioTrackPills` goes with it: exported, commented, and never called. Refs #560. Step 7 of 7.
… one An added word buys itself time. On the RECORDING lane that time is film: the clip holds a frame and the ruler grows. On the VOICEOVER lane it is silence inside the take — the picture is not touched at all, and the narration that follows lands later against the same image. The previous plan had a voiceover insert holding the picture on screen; that was wrong and is gone. `resolveInsertPlacement` reads the lane from the ASSET's kind, which is already the discriminator the transcript tab uses. Nothing stores a container id, and that is deliberate: every candidate is ephemeral. A voiceover fragment id is re-minted by `reanchorAudioTracks` on the first clip drag, and a clip id does not survive a split — which in this repo is `duplicateClip` plus two `setClipSourceRange` calls that move `atSec` out of the half that was named. The two lanes come back in DIFFERENT shapes on purpose. A recording insert can be given its raw moment immediately, through the clip that plays that source second. A voiceover insert cannot: its ruler position depends on the insertions before it inside the same take AND on the cuts under it, and only the take's own walk resolves that. Handing back an unprojected result is what stops a caller inventing a projection that would then disagree with the walk. `insertRangeSchema` is untouched. The `media` enum the earlier plan proposed is dead schema: nothing would read it, the lane is derivable, and on a voiceover row it would store a fact about itself that is false — the species the comment above that schema warns about. No `schemaVersion` bump, no migration. What did need fixing is user-visible: the writer hard-coded "Held frame for the added word", which is a lie on a take. It is lane-dependent now. The keying stays lane-agnostic — one row per word per asset — so `insertRangesMatchWords` is unchanged and still holds. A voiceover insert is inert on the film today by ACCIDENT: `rulerInserts` and `holdAt` both match on `clip.assetId === insert.assetId`, and an audio asset is never a clip's. That accident is the only reason writing one is currently harmless, so it is now a rule with a test — each paired with a positive control so the assertion cannot pass by testing nothing. Refs #560. Step 1 of 7.
…g it to a word
A take is subject to two opposite forces and they must not be two passes. A cut
under it takes time away — step 3 already slices the mix by `removedRawSpans`. An
insertion inside it adds time: the voice stops and resumes on the same word.
Resolved in two passes, an insertion's raw moment would be computed WITHOUT the
holds before it. Two insertions in one take is enough to show it: the second would
map to a moment inside the first one's hold. `takeProgramme` walks sequentially
with two cursors, and there is a test named after exactly that.
Two things the walk deliberately does NOT do, both settled by the maintainer
after the plan was written:
- It does not react to an insertion in the RECORDING lane. A word added to the
film freezes the picture; the take has its own audio and keeps talking,
finishing that much earlier against a picture that has slid. A take is as long
as the audio it holds, and nothing underneath changes that. The earlier plan
had a film hold parking the voice too — that was mine, not asked for, and it
is gone along with the `filmHolds` argument it needed.
- It does not lengthen the programme. The clips decide the length. A take
insertion pushes the take's later content later inside the SAME timeline, and
whatever that pushes past the last frame is lost at export. That is visible in
the tests: a ten-second take with a one-second pause consumes nine seconds of
its file.
It runs on the PILL, never on a stored fragment. The document stores one fragment
per clip a take covers; growing one leaves its successor's head where it was, and
the mixer sums with `+=` at an absolute offset — a fragment-wise walk ships a take
playing on top of itself.
`rawSpanForOutDuration` is the inverse of `outputDurationOfRawSpan`: a voice plays
at 1x in the mix, so a pause is D seconds of the take's own clock, which under a
2x region is two raw seconds and not one.
The loop carries a bounded-pass guard rather than an argument that it terminates.
Mutating the boundary arithmetic did not fail an assertion — it span forever,
which in a renderer is the worst failure there is. The first guard I wrote for it
reset its own counter every pass and caught nothing; counting passes against the
boundary count does, and cannot false-positive on a zero-length insertion, which
makes real progress without moving the cursor.
Refs #560. Step 2 of 7.
…ng early `projectRawTimelineSecToPlayback` builds the programme clock by accumulating raw lengths, so it could only ever compress. A pause occupies ZERO raw seconds and D OUTPUT seconds — the one thing a flat kept-interval list cannot express, which is why it was left out. The consequence has been shipping: every audio track after a recording pause lands D seconds early, in the preview and in the export. The walk now interleaves the pauses with the kept spans rather than adding them afterwards, because where a pause sits inside a span decides which side of it the playhead falls on. A pause whose moment a trim removed is in no kept span and is never reached — the same rule `resolvePlaybackSegments` already follows. Landing exactly on a pause's moment is the BEGINNING of the hold, matching `expandRawSec`'s strict edge: a track whose head sits there starts with the pause, not after it. `filmInserts` is REQUIRED, and all eight non-test call sites were migrated in this commit. Optional would have left the bug alive at every site nobody had touched yet, which is the exact failure the argument exists to fix — and the kind that resurfaces months later as "the music starts a beat early". The type checker found all eight; that is what paying for it buys. Under a speed region the film either side compresses and the pause does not: a voice plays at 1x, so the second it bought is still a second. Zeroing the accumulation fails three of the new assertions. Refs #560. Step 3 of 7.
…nothing A word added to the recording lengthens the ruler and freezes the preview, and produced no frames at all in the exported file. `walk_composited_timeline` skips every clip whose source window is empty (`timeline_walk.rs:250`), and a held segment is exactly that shape — it has no source to decode and exists only for the frames it holds. `hold_sec` threads through five mirrors, because one missed site would drop it silently on one platform with no compile error: the TS contract, the napi input object and BOTH of its mapping sites, `ClipSource` on Windows, macOS and Linux, and `SceneClip` as `#[serde(default)]` — mirroring `has_audio` three lines above it, so a document written before the field loads at 0.0. The held segment stays its OWN clip. Folding it onto its predecessor was the earlier plan and it was wrong: scene regions are keyed by clip INDEX, so removing an entry points every region after it at the wrong clip, and `resolveVisibleClips` has four consumers rather than the two that would have been folded. The held frames are emitted AFTER the speed segments and deliberately outside them. Inside, `stretch_pcm_to_length` would WSOLA the clip's real voice across the freeze — audible. Outside, the audio slot is longer than the decoded PCM and `min(source.len())` leaves zeros, so the silence costs nothing: no silent file to decode, no extra mix entry. `ceil`, not `round`: rounding loses up to half a frame, taken from the end of the narration the pause exists to make room for. The last composed pair is hoisted out of the segment loop, because `'clip_frames` can break and a hold-only clip never enters it — there would be no frame in scope to hold. A clip that yields no frame at all warns and emits none, rather than composing nothing. `resolveVisibleClips` returns `PlaybackSegment[]` now: widening it to `AxcutClip[]` threw `heldSec` away, which is why the pause could never reach the compositor in the first place. Verified here, not left to CI: `cargo check --all-targets` clean on the compositor and the napi bridge, and all 181 Rust unit tests pass — the test binary needs the bundled ffmpeg DLLs on PATH, which is what the earlier "cannot build in this worktree" note was really about. Refs #560. Step 4 of 7.
The export sliced a take with `subtractRemoved` and the transcript placed it one per stored FRAGMENT. Both were right while a take could only lose time and wrong the moment it can gain some: a fragment's source window knows nothing about a pause before it, so every word after an insertion would be struck through — and highlighted — at a moment it is not heard. The export now walks the PILL, once, from its head fragment. The document keeps one fragment per clip a take covers, and walking them separately would emit overlapping entries into a mixer that sums with `+=` at an absolute offset: the file would contain the take playing on top of itself. `voiceoverPlacements` emits one placement per PLAY PIECE. A ventilated take with no cuts folds back into ONE placement, which is what it always should have been — the fragments exist because the take spans a cut in the film, not because the narration is in two pieces. With cuts it splits at the cuts; with an insertion the piece after it carries the ruler moment the words actually occupy. That also settles the fourth copy of the affine map. `findCueWordId` carries its own inverse, and the plan expected it to need an insertion term. It does not — provided every placement is affine, which walking by play pieces is exactly what buys. Asserted rather than assumed: the cue follows a word pushed later by a pause, and goes quiet while the voice is parked, because no word is being said there. The caption deriver and the transcript pane are fed the same two arguments, so the burnt subtitles and the strikethrough cannot disagree with the mix. One test changed its mind rather than its expectation: it asserted a ventilated take keeps one placement per fragment, on the reasoning that collapsing would re-read the file from its head at every cut. The walk recomputes the source advance, so collapsing is safe now — and necessary. Refs #560. Step 5 of 7.
The rAF drove each stored FRAGMENT of a take independently. That was harmless while a take could only lose time; with an insertion it puts the take on top of itself, exactly as it would in the export — the document keeps one fragment per clip a take covers, and each would play its own slice from its own head. Every fragment but the group's head is now paused, and the head is driven by one walk over the PILL, memoised per take and recomputed only when the cuts or the insertions move. Inside a pause the walk returns the PARKED source second and `shouldPlay: false` — one value for the whole hold. A drifting target would re-seek a paused element every frame, and resuming from the post-insert source would restart the narration on the wrong word. Asserted both ways: the target is a single value across the pause, and the resume lands on the second it stopped on. Fades measure against the SOURCE the walk consumes, never the take's ruler extent. An insertion grows the extent without adding a second of file, so a fade-out measured on it would start early here and nowhere else — a preview that disagrees with `resolve_fade_samples` and nothing to say why. `resolveVoiceoverPlayback` is deleted rather than adapted: `takePlaybackAt` is the same question asked of the walk, and keeping both would be two answers to it. The picture is untouched. Nothing in the film holds for a voiceover insertion — the take stops, the film runs on underneath. One test had to change its detector, not its expectation: it broke runs on a jump in source time, and across a pause the source is deliberately CONTINUOUS, because the voice resumes on the word it stopped on. It breaks on silence now. Refs #560. Step 6 of 7.
The gesture is lifted on the voice-over lane. Its stated reason — "a voiceover placement has no clip to hold" — was the model the maintainer corrected: nothing is held there, the voice simply stops and the film runs on underneath. The `import.meta.env.DEV` gate stays and gates the GESTURE only; the record, the readers and the rendering are live in both builds, so a project made in dev opens and plays in production. The mark finally has a width. `.tlClipInsert::before` was pinned at `width: 3px` while the button carried the pause's real width inline, so a word that bought two seconds and one that bought nothing drew the same tick — and the wide case was a multi-second INVISIBLE column that swallowed clip drags. It fills its button now, with a floor so a word that borrowed existing silence stays clickable. Two defects lived in one ternary. A word WITH a pause was placed on the expanded ruler and one WITHOUT at a fraction of the clip's SOURCE span — two clocks, and the box is drawn in only one of them. Both edges were inclusive, so a word whose pause sits on a split boundary painted a mark in BOTH halves, which is routine because a pause sits at the END of the word it follows. `insertedWordMarks` is extracted so that is testable at all: the suite has no render harness for this component, and the alternative was building one for a single assertion. Half-open at every clip's far edge except the last, or the final word of a project disappears. Removing the boundary rule fails an assertion. Audio pills move from `pctOf` to `pctAt`. The clip boxes are drawn on the EXPANDED ruler and the pills were drawn on the stored one, so any pause in the film slid the two lanes apart — a bug that has been there since the ruler learned to expand, visible the moment anyone adds a word. The take keeps its own length; only its head follows the ruler. Refs #560. Step 7 of 7.
A take that holds somewhere is drawn in pieces: one waveform per stretch it plays, and a hatched amber column where the voice stops. Inside ONE outline, on purpose — the take is still one take, one draggable and slippable object, and the eye should read "the voice stops here", not "two takes". Opposite polarity to the clip lane's band. That one means the picture freezes here; this one means the voice stops here and the film runs on underneath, which is the whole difference between the two insertion lanes. Positioned from the same walk the preview and the export read, so a notch cannot appear where the voice does not actually stop. A take with no insertion — and every music bed, and every looping take — keeps the single unbroken waveform it has always had, so the common pill is untouched. The notch takes no hit target of its own: clicking still selects the take, and the word is deleted from the transcript, which is where it was created. Refs #560.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Imported audio, whole — built on @olamide226's branch rather than beside it.
This PR was originally an independent convergence of #502 and #526. It has been rebuilt: #561's branch is the base, and the four things it did not have were ported onto it. The reason is credit, and it is not a gesture. Ola did the pivot he was asked for on 2026-08-29 and opened #561 on 09-01, while a second implementation of the same feature was landing in parallel. Rebasing onto his work is what makes the authorship on
mainreflect who wrote it:@Beetix)@olamide226)Merging this lands #502, #526 and #561. All three should close with it.
What the base already does
Imported audio is a clip-anchored region on voiceover and music lanes, so a track travels with its clip through reorder, trim and delete. Fragments of one track share a
trackId, andanchorAudioTrackFragmentsadvances each fragment'soffsetMsso a bed spanning a cut does not restart at it. Voiceover recording against the timeline, in a docked bar rather than a modal, with the timeline's own tracks silenced for the take. Fades, loop and mute, applied by an envelope inoverlay_track_pcmand mirrored in the preview through one sharedresolveFadeSecs. Native mixing on all three platforms.Read #561 for the full account, including the correctness work found while testing it on a device — a track inside a trimmed stretch still playing, speed regions dragging audio with them, the ±12 dB clamp on per-track gain.
What was ported onto it
feat(ai): the agent can see and place audio. #561 landed the feature with no agent surface at all — absent fromdocumentSnapshotForModel, absent from the tool roster.addAudio/setAudio, plusremoveModifierresolving an audio track, built on this branch's owntrackGroupId/collapseTracksToPills/patchAudioTrack/anchorAudioTrackFragmentsrather than a second set of helpers.fix(timeline): keyboard activation on lane pills. Every pill carriesrole="button"andtabIndex={0}and nothing answered Enter or Space — pre-existing onmain, and it affects all seven lane kinds, not only the audio ones.fix(transcription): stop the background pass transcribing music. The auto pass queues every asset in the document. Measured with a four-minute bed: 35 seconds of whisper inference at editor open, 164 segments of transcribed music.assetCanCarrySpeechanswers from the timeline, becauseAxcutAsset.kindonly knowsvideo | audioand the voiceover/music distinction lives on the track. This is the rule settled in #560.perf(transcription): extract audio natively, off the UI thread. The freeze at editor open was never the inference — that runs inwhisper-stt-server, its own process, at 6.9× real time. It wasextractMono16kFromVideoUrl, which runs in the renderer: whole file into memory, anarrayBuffer()copy, aslice(0)copy, then a resample loop on the UI thread. ~86 MB of decoded float32 for that same four-minute bed, against 15.7 MB now that ffmpeg does it in the main process. Reuses the peaks path'sresolveFfmpeg, which is what carries the Windows subtlety that the packaged binary is the shared build and not the staticffmpeg.exethe installer excludes.feat(timeline): show what an audio pill crops, and let it slip. An audio pill is the only timeline object that edits media you cannot see. Two thirds of this turned out to be here already, in better form: the edges stop at the file's bounds vialowerLeft/maxEnd, with a loop exception that lets a repeating track outrun its source. What was added is the part that makes the stop legible — a dimmed ghost of the rest of the file around the pill (clamped to the programme, so a four-minute bed under a five-second view is a bounded element and not one tens of screens wide), anin → out / lengthreadout while an edge is pulled, andAlt-drag to slip: the media slides under a span that does not move, at a rate derived from the file rather than the timeline, because an edge drag at timeline scale means dragging three minutes of ruler to reach 3:00 inside a four-minute bed.Related issue
Closes #350. Supersedes #502, #526, #561. Refs #560, #540.
Type of change
Release impact
Desktop impact
Testing
npm run test— 2398 passed, 4 skipped, 0 failed (191 files)npx tsc --noEmitandnpx tsc -p tsconfig.test.json --noEmit— cleannpm run lint(Biome) — clean (16 warnings, all pre-existing on the base)Not done: the export smoke test on real macOS/Windows per AGENTS.md. #561 was built and exercised on macOS (arm64); the ports above were written and run on Windows. The native extraction has not been exercised at runtime on Linux — the path is platform-agnostic (spawn + pipes) and shares its ffmpeg resolution with a peaks path already in production on all three, but that is an argument, not a run.
🤖 Generated with Claude Code
Summary by CodeRabbit