AutoEdit AI

Turns raw footage into an edited video from a brief written in plain language. You drop in the clips, describe what you want ("a 90-second highlight reel, keep the parts where I'm explaining the product, cut the retakes"), and the app analyzes every second of footage locally, has Claude reason over that metadata to produce an edit plan, and renders it with FFmpeg. You refine the result by chatting with it.

Desktop app: React/TypeScript frontend, Python/FastAPI backend, local ML pipeline, FFmpeg assembly.

Status. This is a personal project, published as a working prototype rather than a packaged release. The Rust/Tauri shell that wraps the app is not in this repository — see Running it for how to run the backend and frontend directly.

---

Why it's built this way

The hard part of automated editing isn't the rendering — it's giving a model enough grounded information about the footage to make defensible cuts, without shipping gigabytes of video to an API.

Local-first analysis. Every clip is decomposed into segments, and each segment gets a metadata row computed on your machine: word-level transcript, filler-word ratio, whether the sentence completes, silence ratio, RMS energy, audio quality score, camera stability, shake detection, and optionally pose (person count, gesture intensity, whether the face is toward camera). Nothing leaves the machine for any of this.

Routing by modality. Each analysis job goes to whatever tool is actually right for it rather than to one general-purpose model: faster-whisper for speech, PySceneDetect for cuts, librosa for audio, OpenCV frame-diff for motion, YOLO11-pose for bodies, Gemini Flash for visual description, Claude for the edit plan itself. Frame-diff instead of optical flow for motion, for instance, is roughly 10x faster and sufficient to detect shake.

Cost is visible and opt-in. The only stage that sends media anywhere is visual analysis, and it is off by default. When you turn it on you choose frames mode (one thumbnail per segment, ~$0.002/segment) or clips mode (the video itself, ~$0.01-0.05/segment), with the per-segment cost stated in the UI before you commit.

Structured output, not parsed output. The edit plan comes back from Claude through tool_use with a strict JSON schema, so there is no JSON-parsing retry loop and no partially-valid plan to defend against. Segment references are stable UUIDs, and the assembler validates every ID before it invokes FFmpeg — positional references would silently break the moment a plan reorders clips.

Everything is resumable and nothing is overwritten. Each pipeline stage writes a checkpoint to project.json. Kill the app mid-transcription and it resumes without re-normalizing. Edit plans are immutable and versioned (plan_v001.json, plan_v002.json, ...), renders are versioned alongside them, so undo is real and no past render is ever clobbered.

---

Stack

Layer	Tech
Desktop shell	Tauri 2.x (Rust + WebView) — not in this repository
Frontend	React 18, TypeScript, Tailwind, Vite, Zustand
Backend	Python 3.11+, FastAPI, uvicorn (single worker — the SSE queue is in-memory)
Video	FFmpeg (always Path.as_posix() for Windows safety)
Transcription	faster-whisper (local, word timestamps)
Scene detection	PySceneDetect
Audio analysis	librosa (silence ratio, RMS energy, quality score)
Motion analysis	OpenCV frame-diff (stability, shake)
Pose	YOLO11-pose at 1 fps — optional, see Licensing
Edit planning	Claude via tool_use with a strict JSON schema
Vision	Gemini 2.5 Flash, frames or clips mode — off by default
Media library	Freesound (OAuth2), Jamendo, Pexels search

Pipeline

created -> uploaded -> normalizing -> transcribing -> scene_detecting -> analyzing
        -> pose_analyzing -> vision_analyzing -> planned -> assembling -> assembled
        -> refining -> exported | error

1. Ingest — native file dialog hands paths to the backend, which copies into raw/. No HTTP upload buffering, so multi-GB files are fine.
2. Normalize — FFmpeg re-encodes to H.264/AAC, parsing time= off stderr for progress.
3. Transcribe — faster-whisper with word-level timestamps; derives filler_ratio and sentence_complete.
4. Scene detect — PySceneDetect, sub-splitting anything over 30 s at silence gaps.
5. Audio + motion + pose — three local analyzers; YOLO runs at 1 fps on CPU to stay tractable.
6. Retake detection — transcript heuristics ("let me try again", truncated sentences, high filler) produce is_likely_retake and a confidence score.
7. Vision (optional) — Gemini Flash over frames or clips, throttled by an asyncio.Semaphore. Skipped entirely when vision is off.
8. Plan — pre-filters to the top 120 segments, builds a descriptor per segment from all of the above, and calls Claude with a tool_use schema. Returned segment IDs are validated before the plan is written.
9. Assemble — see below.
10. Chat refinement — multi-turn over the last 10 turns, returning plan deltas with a preview-then-confirm step.
11. Export — the same assembler at slow / crf 18, to a versioned filename.

Every stage checkpoints as done / pending / skipped; the orchestrator skips done stages on restart.

The assembler

backend/pipeline/assembler.py is the densest part of the codebase — a five-pass FFmpeg pipeline:

1. Pre-process clips needing zoompan or speed changes, which can't be applied in a post-pass.
2. Concat demuxer with per-clip in/out points, expanding cut-outs (one entry splits into several ranges).
3. Effects pass — filter_complex for fades, text overlays (with emoji-stripping for drawtext), color grading, plus afade on audio.
4. Audio mix — amix across global tracks and per-clip overlays, with adelay, volume, and fades.
5. Video overlay — overlay / blend=screen for particles and light leaks, looping shorter overlays via -stream_loop.

The concat demuxer is the default rather than filter_complex because filter_complex chokes at roughly 30+ inputs on Windows.

API

All routes under /api/v1. Long operations stream progress over Server-Sent Events from per-project asyncio.Queue instances.

GET    /setup/check                                     POST   /projects/{id}/plan
POST   /setup/download-model            (SSE)           GET    /projects/{id}/plan
POST   /setup/validate-api-key                          GET    /projects/{id}/plan/history
                                                        POST   /projects/{id}/plan/undo
POST   /projects                                        POST   /projects/{id}/assemble
GET    /projects                                        GET    /projects/{id}/assemble/progress (SSE)
GET    /projects/{id}                                   POST   /projects/{id}/chat
DELETE /projects/{id}                                   GET    /projects/{id}/chat/history
                                                        POST   /projects/{id}/export
POST   /projects/{id}/ingest                            GET    /projects/{id}/export/progress (SSE)
POST   /projects/{id}/pipeline/run                      GET    /projects/{id}/files/normalized/{f}  (range)
GET    /projects/{id}/pipeline/progress (SSE)           GET    /projects/{id}/files/output/{f}      (range)
GET    /projects/{id}/segments                          GET    /projects/{id}/files/thumbnails/{f}
PUT    /projects/{id}/brief

Full design notes, data shapes, and the reasoning behind each decision are in docs/ARCHITECTURE.md.

---

Running it

Prerequisites

• FFmpeg on your PATH — sudo apt install ffmpeg, brew install ffmpeg, or ffmpeg.org
• Python 3.11+
• Node.js 20+
• An Anthropic API key (required — this is what plans the edit)
• A Gemini API key (optional — only if you enable visual analysis)

Setup

python3 -m venv .venv
.venv/bin/pip install -r backend/requirements.txt
npm install

On Windows use .venv\Scripts\pip instead. Copy .env.example to .env and add your keys, or enter them in the app's setup wizard.

Run

The Tauri shell normally spawns the backend and serves the frontend. Without it, run the two halves yourself — backend first:

.venv/bin/python backend/main.py

then the frontend:

npm run dev

The backend listens on 127.0.0.1:8765. Two frontend call sites depend on the Tauri runtime and won't work in a plain browser: the native file dialog in src/components/upload/DropZone.tsx and key storage in src/lib/keyStore.ts. Running this way, ingest files via POST /api/v1/projects/{id}/ingest and set keys in .env.

On first run the app checks FFmpeg, then downloads the Whisper base model (~150 MB, one time).

Optional: pose analysis

.venv/bin/pip install -r backend/requirements-pose.txt

Without it the pose stage returns empty metrics and the rest of the pipeline is unaffected. The YOLO weights download automatically on first use.

Project data

Everything a project produces lives in projects/{project-id}/ and is gitignored:

project.json      state + checkpoints
raw/              originals
normalized/       H.264/AAC re-encodes
transcripts/      whisper output
analysis/         scenes + segments JSON
thumbnails/       per-segment JPEGs
edit_plans/       plan_v001.json ... (immutable)
output/           preview_v001.mp4, export_1080p_v001.mp4 (versioned)

---

Licensing

This project is MIT licensed — see LICENSE.

The pose analysis stage uses Ultralytics YOLO, which is AGPL-3.0. To keep this repository free of AGPL obligations, it is deliberately not a required dependency: it lives in backend/requirements-pose.txt, is imported lazily, and the pipeline degrades gracefully without it. If you install it, the AGPL terms apply to your use of that component.

Privacy

Footage never leaves your machine unless you explicitly enable visual analysis, which is off by default. When enabled, only the segments being analyzed are sent, and the per-segment cost is shown before you start. API keys are stored via tauri-plugin-store (OS-level storage) or a local .env — never in project files.