Hypermuse

Hypermuse is a browser-based audio and VJ visualization project built with Three.js. It turns audio energy into animated 3D geometry around a sphere and can map uploaded video frames onto those geometries for live performance visuals.

Just want to run the show, not read code? Jump to Using Hypermuse (plain-language guide). An illustrated PDF covering every mode lives at artifacts/hypermuse-modes-guide.pdf (regenerate with npm run guide:pdf; refresh its screenshots with npm run guide:stills).

Using Hypermuse (plain-language guide)

This section is for performing with Hypermuse without touching any code. If someone technical set it up for you, you can ignore everything else in this README and just follow along here.

The big picture: two windows

Hypermuse runs as two windows:

• The controller (controller.html) — your "remote control." This is where you click buttons. The audience never sees this.
• The visual (sonicsphere.html) — the actual art that reacts to music. This is what you put on the big screen / projector.

The controller opens the visual for you and sends it commands. Think of it like a lighting desk (controller) driving a stage (visual).

The Controller style dropdown (top row) recolors the control panel itself — crt green (phosphor + scanlines, matches the Ikegami terminals), green mono (flat terminal), amber terminal, or red alert. It only themes this window, never the outputs, and the choice persists across reloads.

Step 1: Start it up

If the app is already running on the computer, open a web browser (Chrome works best) and go to:

http://localhost:8080/controller.html

If nothing loads, someone needs to start the app once. A technical helper can do this by opening a terminal in the project folder and running npm install (first time only) then npm start. After that, the link above will work.

Step 2: Open the visual

On the controller, click open/reopen visual (top of the page). A second window opens — that's your art. Drag it onto your projector/second screen and make it fullscreen.

Tip: the easiest way to start with something good-looking is to click one of the Presets buttons (for example cells1+cells2 autoplay (with logo)). It opens the visual already configured.

Step 3: Play music

The visuals react to sound. You have two options:

• Easiest: just play music out loud / through the venue — but the visual needs the audio fed into it. For a reliable signal, load an audio file directly: in the visual window there's a Music file picker — choose an .mp3/.wav and it starts reacting.
• Press h in the visual window to hide or show its control panel.

Step 4: Drive the look from the controller

Here's what each section of the controller does, in plain terms:

• FX Mode — the overall "style" of the visuals. Click one anytime to switch:
  • classic — the original glowing geometric shapes and lines
  • life — cells being born and dying (classic "game of life" feel)
  • hier life — layered living patterns
  • hex life — honeycomb cellular patterns that sweep down the screen
  • kuramoto — pulsing waves that fall in and out of sync
  • gray-scott — organic, blobby coral/spot patterns
  • physarum — slime-mold-like flowing trails
  • molecule — rotating 3D molecule structures
  • next mode — just jump to the next style
• Hex CA — fine-tunes the honeycomb "hex life" style:
  • speed — how fast the pattern evolves (1.0 is normal)
  • sync audio — when checked, the pattern speeds up/down with the music
  • rule — the "recipe" for the pattern (try bloom, maze, coral for different looks)
  • cycle rule — flips to the next recipe automatically
  • aperiodic — adds non-repeating randomness (0 = clean, higher = more chaotic)
  • palette — the color scheme (aurora, magma, violet, mono, neon)
  • apply hex — switches to hex mode and applies your settings
• Basic Video Mode — show your loaded video clips fullscreen with little/no effects. Good for a clean break.
• Bridge — fade to black and fade in. Your safety buttons for transitions.
• SynBioBeta Logo — show/hide a logo overlay and set its position/opacity.
• Triangle (mosaic) vids — tiles video clips into a mosaic; triangle + fx adds effects on top.
• Loop Rating — like loop keeps a clip around; dislike & skip drops it and moves on. Use this to curate what plays during the show.
• Folder loops (playlist) — tick/untick which folders of clips are allowed to play.
• Color board / Color cube board — filter or display clips by color family.

A typical live flow

1. Click a Preset to open the visual, then make it fullscreen on the projector.
2. Start your music (load an audio file in the visual window).
3. Pick an FX Mode that fits the song (e.g. hex life for builds, gray-scott for ambient moments).
4. For hex mode, leave sync audio on so it moves with the beat; tweak palette to match the mood.
5. Use next mode or click another FX button to change the vibe between sections.
6. Use fade to black / fade in for clean transitions.
7. like / dislike & skip clips as they play to shape the set.

That's the whole job — pick a mode, match the palette to the music, and use fades for transitions. Everything below is for people who want to build sets, export videos, or modify the code.

What the project does

• Analyzes incoming audio with the Web Audio API (AnalyserNode).
• Splits the frequency spectrum into bands and tracks per-band energy.
• Maps those bands to points distributed on a sphere.
• Creates reactive shapes, lines, and polygons when band thresholds are exceeded.
• Optionally textures geometry with video frames for VJ-style visuals.
• Exposes many live controls (thresholds, hue, camera/light, rotation, density) through a controller window.

How it is structured

The app is mostly multi-page HTML + inline script, with shared helpers in js/.

• controller.html -> opens sonicsphere.html in a second window and sends control updates with postMessage.
• sonicsphere.html -> main audio-reactive sphere renderer.
• colorsphere.html + colorcontroller.html -> color-focused variant.
• poetsphere.html + poetcontroller.html -> poetry-related variant.
• polysphere.html, videosphere.html, venus.html, kurasphere.html -> alternate visualizer experiments.
• synergetics-fold.html -> Fuller's Synergetics 100.41 fold: a wireframe triangle folds its three corners up into a tetrahedron, tumbles, unfolds, and cycles through shape presets (?shape=equilateral|scalene|right|cycle, ?speed=, ?color=, ?spin=; space pauses, s skips to the next shape). Export with npm run export:fold. Add ?mode=associate for the Synergetics 108.01-03 demonstration: two open triangular spirals (positive + negative helix, ?color2= for the second strand) tremble apart as unstable separate actions, then associate into the tetrahedron's six edges. The two tinted "event" faces bloom first; the two white complementary faces converge in from outside ("from the rest of the Universe") and a pulse marks the enclosed center — inside vs outside, 1 + 1 = 4. Export with npm run export:fold:associate.
• prototypes/ -> older or experimental variants.

Shared utility scripts:

• js/video_processor.js -> video queueing, frame capture, texture updates.
• js/message_controller.js -> message handling for live parameter changes.
• js/audio_controller.js -> audio queue playback helpers and MIDI playback logic.
• js/sonic_geometries.js -> reusable geometry-building utilities.
• js/note_analyzer.js -> note/frequency mapping and peak detection helpers.

Layered simulation scaffold:

• js/layers/audio_engine.js -> emits normalized audio frames (low, mid, high, beat, barPhase) from analyser data.
• js/layers/sim_compositor.js -> combines one or more simulation plugins into a texture layer for Three.js materials.
• js/simulations/life_sim.js -> Conway-style cellular plugin with audio-reactive speed/rules and beat-triggered spawning.
• js/simulations/hierarchical_life_sim.js -> multi-layer Life variant with coupled low/mid/high lanes.
• js/simulations/hex_life_sim.js -> hexagonal cellular automata with selectable B/S rules, color palettes, top-to-bottom reveal sweep, music sync, and an aperiodic modulation option.
• js/simulations/kuramoto_sim.js -> coupled-oscillator "communication" plugin (Kuramoto-inspired local neighbor sync).
• js/simulations/gray_scott_sim.js -> Gray-Scott reaction-diffusion plugin (organic coral/spot patterns with presets).
• js/simulations/physarum_sim.js -> Physarum (slime-mold) agent-trail plugin.
• js/simulations/molecule_graph_sim.js -> molecule communication graph plugin (SDF atom/bond topology with audio-driven diffusion + PubChem fetch support).

Run locally

Because browsers restrict file/media APIs on file://, run a local server.

Option 1: npm scripts

npm install
npm start

Then open:

• http://localhost:8080/controller.html (main controller -> opens visualizer)
• or http://localhost:8080/colorcontroller.html
• or http://localhost:8080/poetcontroller.html

Hypermoon output & kiosk mode

The controller's Hypermoon panel opens hypermoon.html with ?kiosk=1: drag the window to the target display and the first click/keypress makes it fullscreen and hides the cursor. For a true kiosk (its own Chrome instance, fullscreen from the start, autoplay allowed with no click needed):

npm start                 # serve on :8080 (if not already running)
npm run kiosk:hypermoon   # Chrome kiosk pointed at hypermoon.html

The word field in the controller applies live — the letter mosaic redraws in place, no reload (the mosaic font covers a–z, 0–9, and space). Sliders (speed, brightness, bleed, anchor nudges, orbiting vajras, …) and the window presets are live too — CRT, vajra cave, incantation, the bucky repertoire (below), the baked sonicsphere loop, slideshows, any image/video path, or another window (screen capture), which pipes any browser window or app into the moon: pick the preset, click once in the hypermoon window, choose what to share (needs the output opened via localhost). Switching content never reloads; only changing the moon clip itself does.

Window size × is the master control for how big the dark-side bleed-through is: it multiplies the auto-measured footprint live (drag right and the window/word swells to fill the shadow; ?winscale=1.6 pins it in exports). The word width/height sliders still set the exact footprint if you want manual control.

Bucky repertoire (window presets, all live-switchable)

Fuller's structures run as generative animations inside the moon window — hypermoon.html?content=<key> or the controller's window preset menu. All fold modes are wordless: the letter panel stays off so the geometry is never covered by text (same as incant):

key	what happens	Synergetics
fold	a wireframe triangle's corners hinge up into a tetrahedron, hold, unfold	100.41
foldsonic	same fold, but the faces are video panels playing the sonicsphere loop	100.41
foldhelix	positive + negative open triangular spirals tremble apart, then associate into the six edges of a tetrahedron; the two complementary faces converge in from "the rest of Universe" (1 + 1 = 4)	108.01–03
foldjitter	the jitterbug: the vector equilibrium's eight triangles, joined at their vertices, rotate and contract through the icosahedron to the octahedron and spring back open	460
foldgeo	geodesic bloom: an icosahedron carrying a frequency-4 grid bulges until every vertex hits the circumsphere — the geodesic dome — then relaxes flat	985 / geodesics
foldivm	octet truss: the isotropic vector matrix (alternating tetrahedra + octahedra, all struts equal) crystallizes outward from a nucleus, breathes, dissolves	420

The standalone synergetics-fold.html page still renders fold/associate full-frame for exports (npm run export:fold, npm run export:fold:associate).

Pre-rendered fold loops. npm run export:fold:loops bakes every mode into artifacts/fold-loops/ as square 1080×1080 clips (one full animation period each), plus red variants of the five wireframe modes (fold-red.mp4, foldhelix-red.mp4, …). Use them anywhere plain video works — dark-side bleed-through (content=artifacts/fold-loops/fold-red.mp4), the any-video picker, projector feeds, CRTs, other machines without the live makers. Under the hood this uses two new hypermoon.html params that also work live: ?solo=1 renders any keyword content full-frame instead of on the moon, and ?ink=ff3b30&ink2=ff8a5c recolors the wireframe repertoire (red triangles on demand). Run npm run manifest:videos afterwards so the clips show up in the controller's video picker.

Hour program (rotation sequencer)

hypermoon.html?program=hour (or the program dropdown in the controller's Hypermoon panel) runs a long-form arc measured in moon rotations instead of seconds. The pattern: two rotations show only the hyperstition word on its dark patch; on every third rotation the window opens wordless on the current act's content and rides around with the rotation. The acts progress — CRT terminal → incantation mantras → vajra cave → the six bucky folds → the sonicsphere loop — over 204 rotations, which is roughly an hour at the default speed (1/3 × the 6 s moon loop ≈ 18 s per rotation), then the whole program loops. Picking any manual content preset stops the program; the dropdown restarts it. The speed slider stretches or compresses the hour (half speed = two hours).

The program editor below the sliders gives full control over the arc from the controller: each row is an act (content keyword or any video path — the box autocompletes across the whole manifest — plus cycles / dark / reveal counts), with reorder and remove buttons, a live duration estimate in rotations and minutes at the current speed, and a running "rotation N / total" readout. "load hour" fills in the default program as a starting point; "apply program (live)" sends the whole arc to the output without a reload; "stop" returns to manual control.

Audio deck (bed loop + interjections)

The controller has an Audio deck panel that plays from the controller window itself — route that machine's audio output to the PA. One bed loop plays forever (searchable across every audio file the manifest found — mp3/wav/m4a/aac/ogg/flac under audio/, loops/, artifacts/, assets/, external/, infinitestreams/). Three interjection slots (A/B/C) each take a file, a period in seconds, an optional ± jitter, and a volume; while enabled they fire on schedule, and duck drops the bed to 25% for the duration of the interjection, then restores it. The period counts from when the interjection starts, jitter randomizes each occurrence (e.g. every 300 ± 60 s), and the "fire now" buttons trigger a slot immediately. Swapping the bed file while playing restarts with the new track. npm run manifest:videos refreshes the audio list along with the video one (audio-manifest.json).

Music sync (audio deck → hypermoon)

Everything the audio deck plays is tapped into a WebAudio analyser and the band levels are broadcast to the hypermoon output ~20×/s (the tap is transparent — the speakers get the same signal). On the moon, scaled by the intensity slider (0–2, default 1; 0 = deaf):

• vajras flare on bass beats — a kick-onset detector fires a beat envelope; each orbiting vajra swells (+22% size) and brightens (up to +90% opacity), staggered per lane so they don't pop in unison, and sustained bass adds a constant glow
• earthshine breathes with the bass — the faint light on the shadowed hemisphere pulses with the low end
• the window glow shimmers with the highs — hi-hats and cymbals ripple the brightness of whatever is showing through the window
• the letter mosaic shakes faster when it's loud — tile reshuffle rate scales with overall level

The music → moon row in the Audio deck panel has the on/off toggle, the intensity slider, a live ▁▄▇ band meter with a ● beat light, and moon uses its mic: instead of listening to the deck, the hypermoon window opens its own microphone (grant permission once) — use this when the music comes from a DJ, a PA, or anything not played through the controller. Boot params on hypermoon.html: ?react= (intensity) and ?mic=1. Levels decay to zero within a second if the feed stops, so nothing freezes mid-flare.

Any video on the dark side

npm run manifest:videos scans loops/, artifacts/, assets/, external/, and infinitestreams/ for every Chrome-playable clip (mp4/webm/m4v/ogv) and writes video-manifest.json (~700 clips). The controller's any video box in the Hypermoon panel loads it: type to search (native autocomplete over all paths), pick an entry and it swaps into the moon window live; random picks one at random (the search text acts as a filter, e.g. type "underwater" then hit random). Rerun the manifest script after adding new clips.

Real CRT monitors (composite / BNC)

crt-terminal.html is the page to feed broadcast CRTs (Ikegami etc.). Chain: computer HDMI → HDMI-to-AV downscaler → yellow RCA → RCA-to-BNC adapter → VIDEO A IN; select input A on the front. Pass ?fx=0 on a real tube (it makes its own scanlines), pick phosphor with ?color=green|amber|purple|cyan|white, set the announced time with ?when=, override the script with ?text=LINE|LINE~NEXT SCREEN. ?safe= reserves margin for overscan (default 10%). Bake to video with npm run export:crt.

Live streaming to other devices (LAN)

Any device on the same network can watch a live video stream of an output page in its browser. The video travels peer-to-peer over WebRTC; a tiny signaling server just brokers the connections:

npm start                # static server on :8080 (binds all interfaces)
npm run stream:server    # signaling on :8081 — prints the viewer URL

Then:

• Broadcast the moon: open hypermoon.html?stream=1 on the host machine — it captures its own canvas and streams it (add &streamfps= to change the frame rate, &kiosk=1 still works alongside).
• Broadcast anything else: open stream-broadcast.html on the host and pick any window, tab, or screen (sonicsphere, the projection rig, a video player…). Must be opened via localhost — screen capture requires it.
• Watch: on the other device, open http://<host-ip>:8080/stream-view.html (the stream server prints this URL on startup). It autoplays muted with zero clicks; a tap goes fullscreen. Viewers can join and leave at any time, wait before the broadcast starts, and auto-reconnect if it restarts. One broadcaster at a time; a new one takes over.

Option 2: Python server with CORS headers

python3 pyserver.py

Then open http://localhost:8000/controller.html.

Basic usage

1. Open a controller page.
2. Click Start.
3. Load audio files (audio/*, .mid, .midi) and optional videos.
4. Adjust threshold sliders and other controls to shape the reaction.
5. Use keyboard shortcuts in visualizer pages:
   • h toggle control panel
   • v toggle video list panel (where supported)

VJ set preload workflow

To preload a folder of loops (for example loops/bio1) and reuse it as a startup set:

1. Put your loops in loops/bio1 (nested folders are supported).
2. Build a manifest:

npm run build:vj-set

3. In sonicsphere.html, use Video Set Manifest (sets/bio1.json) and click Load Set.

The manifest can also be loaded automatically by the sample export script.

Each loop entry includes transition controls:

{
  "url": "loops/bio1/clip01.mp4",
  "label": "clip01.mp4",
  "transition": {
    "type": "fade",
    "durationMs": 900,
    "holdMs": 8000
  }
}

Set-level playback can be pingpong (back-and-forth) or loop:

{
  "playbackMode": "pingpong"
}

You can tune defaults when building:

VJ_HOLD_MS=10000 VJ_TRANSITION_MS=1200 VJ_TRANSITION_TYPE=fade npm run build:vj-set

Useful set-build options:

VJ_MAX_LOOPS=20 VJ_PLAYBACK_MODE=pingpong npm run build:vj-set

Set manifests can also carry an effect schedule:

{
  "effectTimeline": {
    "enabled": true,
    "phases": [
      { "name": "classic", "durationSec": 16 },
      { "name": "life", "durationSec": 16 },
      { "name": "kuramoto", "durationSec": 16 },
      { "name": "molecule", "durationSec": 16 },
      { "name": "stacked", "durationSec": 16 }
    ]
  }
}

When you click Load Set, this schedule is applied automatically.

Set manifests can optionally specify a real molecule source:

{
  "moleculeGraph": {
    "name": "caffeine",
    "names": ["caffeine", "serotonin", "dopamine", "glucose"],
    "cycleOnPhaseChange": true
  }
}

When present, the molecule source is loaded from PubChem automatically. If names is provided, the set can rotate through the list as phases change.

You can also edit timeline behavior live in sonicsphere.html controls:

• Effect phases (comma-separated)
• sec/phase
• timeline on
• Apply FX Timeline

classic preserves the original triangle/polygon visual style before layered variants.

Sample video export

Generate a short sample output recording:

npm run export:sample

By default it tries to preload sets/bio1.json, injects generated test audio, and writes artifacts/sample-sonicsphere.webm.

To target a different set manifest:

VJ_SET_MANIFEST=sets/another-set.json npm run export:sample

To force a specific real molecule during export:

MOLECULE_NAME=caffeine npm run export:sample

For the 6.5ft x 4.5ft wall target (13:9 aspect ratio), use the dedicated preset:

npm run export:sample:13x9

Or override with explicit frame size:

EXPORT_WIDTH=1872 EXPORT_HEIGHT=1296 npm run export:sample

Exporter note:

• If source is an audio file (.wav, .mp3, etc.), audio is muxed into the output video.
• If source is MIDI (.mid/.midi), visuals still render, but export stays silent unless you provide an audio render of that MIDI.

You can also export directly from the UI using Start Export / Stop Export in sonicsphere.html (uses browser MediaRecorder from the render canvas).

Core visualization idea

Frequency bands are mapped across a sphere using a golden-ratio-style angular step. When a band crosses its threshold, the corresponding point(s) activate and geometry is generated between active points, producing a live "music topology" effect.

Layered simulation mode

sonicsphere.html runs a layered simulation pass:

• Audio analysis -> HypermuseAudioEngine
• Simulation plugin updates -> Life, Hierarchical-Life, Hex-Life, Kuramoto, Gray-Scott, Physarum, and Molecule-Graph plugins
• Texture composition -> HypermuseSimulationCompositor
• Rendered as an additive "simulation shell" mesh in the scene

This is intended as the base architecture for integrating Morpholib-style and other cellular systems with the same music sync pipeline.

The molecule plugin directly adapts the parseSDF approach used in fractastical/metajargon and turns bonds into a communication network for signal propagation.

You can load real molecule data live from controls in sonicsphere.html:

• Molecule (PubChem name) input (examples: caffeine, serotonin, dopamine)
• Load Molecule button
• status text showing loaded atom/bond counts

sonicsphere.html also runs a timed effect scheduler (while audio is active) that cycles through effect profiles (about every 16 seconds by default, configurable via the timeline controls). Available profiles:

• classic -> original triangle/polygon geometry look
• life -> Conway-style cellular look
• hierarchical-life -> multi-layer Life look
• hex-life -> hexagonal cellular automata (rules/palettes/aperiodic; music-synced)
• kuramoto -> oscillator-dominant sync look
• gray-scott -> reaction-diffusion (coral/spots) look
• physarum -> slime-mold trail look
• molecule -> molecule-graph diffusion look
• rewrite -> molecule-graph rewrite variant
• word-cloud -> floating phrase overlay
• stacked -> all simulation layers combined

Hex cellular automata (hex-life)

The hex-life profile is a 6-neighbor hexagonal CA you can control live from controller.html (the FX Mode + Hex CA panels) or via the VJ command API:

• Rule presets: hexlife (B2/S34), bloom (B2/S3), maze (B3/S12345), pulse (B13/S24), coral (B2/S2); plus cycle rule to advance automatically.
• speed, sync audio (sync evolution to the beat), sweep rows (top-to-bottom reveal rate), aperiodic (non-repeating modulation), and palette (aurora/magma/violet/mono/neon).
• URL boot params: hexpalette, hexspeed, hexsync, hexsweeprows, hexrule, hexaperiodic (e.g. sonicsphere.html?mode=hex-life&hexrule=maze&hexpalette=violet).

VJ command API

The controller drives the visual by sending postMessage({ type: 'vj', ... }) to sonicsphere.html (window.vjControl(cmd) does the same in-page). Common fields:

• mode -> effect profile or alias (classic, life, hierarchical-life, hex-life/hex, kuramoto, gray-scott, physarum, molecule/morphospace, rewrite, word-cloud, or next to advance)
• hexRule, hexRuleCycle, hexSpeed, hexSync, hexSweepRows, hexAperiodic, palette -> hex CA controls
• speed, hue, intensity, paused/play/pause
• basicVideo, mosaic, mosaicFx, blackout/blackoutFadeMs
• logo, logoOpacity, logoPosition
• loopPreference (like/dislike), wordList

Notes

• This repo contains multiple experimental pages with overlapping logic.
• Some scripts are in-progress or partially wired; controller.html + sonicsphere.html is the primary path.
• Live control mapping doc: LIVE_CONTROLS_MIXER.md

Related work

The theoretical background is in the Hypermusic repository: https://github.com/fractastical/hypermusic