alignecho

Cross-modal audio-text alignment & retrieval, in one small toolkit.

Contrastive two-tower encoders · InfoNCE training · DTW forced alignment · retrieval benchmark

---

Overview

alignecho encodes audio (log-mel or self-supervised features) and text (transformer embeddings) into a shared space, trains them to agree with a CLIP-style contrastive objective, and then lets you:

• Retrieve captions from audio and audio from captions, with the standard Recall@k / median-rank metrics.
• Forced-align an utterance to its transcript by walking a monotonic DTW path over the cross-modal similarity matrix, recovering rough token time-stamps.
• Benchmark a checkpoint over a manifest with a single command and get a tidy results table back.

The numpy-only pieces (alignecho.align, alignecho.retrieval) import without torch, so you can drop the DTW aligner or the retrieval metrics into an existing pipeline with almost no dependency cost.

Architecture

        audio (log-mel / SSL frames)            text ("a dog barks twice")
                    │                                       │
        ┌───────────▼───────────┐               ┌───────────▼───────────┐
        │  ConvSubsampler (4x)   │               │  HF transformer        │
        │  Transformer encoder   │               │  (DistilBERT, ...)     │
        │  masked mean-pool      │               │  mean / CLS pool       │
        └───────────┬───────────┘               └───────────┬───────────┘
                    │  proj → R^d                            │  proj → R^d
                    ▼                                        ▼
              L2-normalise  ──────►  cosine sim  ◄──────  L2-normalise
                                        │
                          ┌─────────────┴──────────────┐
                          │  τ · (A · Tᵀ)  → InfoNCE    │   (training)
                          │  DTW over (1 − S)           │   (alignment)
                          │  argsort ranks → R@k, MedR  │   (retrieval)
                          └────────────────────────────┘

Both towers project to the same dimension d (default 256). A learnable temperature τ (parameterised in log-space, à la CLIP) scales the logits.

Installation

git clone https://github.com/ella-brown629/alignecho.git
cd alignecho
pip install -e ".[dev]"

Or just the runtime dependencies:

pip install -r requirements.txt

Usage

Train

alignecho train --config configs/base.yaml --output-dir runs/exp1

The training loop does AdamW + linear-warmup-cosine decay, gradient clipping, and per-epoch validation by global (not in-batch) retrieval rsum, saving the best checkpoint to runs/exp1/best.pt.

Benchmark

alignecho bench --manifest examples/toy_manifest.tsv \
    --checkpoint runs/exp1/best.pt --output-dir bench_out

Writes bench_out/report.json and bench_out/report.csv, and prints the results table to stdout.

Forced alignment

alignecho align --audio frames.npy --text tokens.npy --band 12 --hop 0.02

Python API

import numpy as np
from alignecho import cosine_sim_matrix, dtw_align, evaluate_bidirectional

# forced alignment over a similarity matrix
sim = cosine_sim_matrix(audio_frames, text_tokens)   # (T_a, T_t)
result = dtw_align(sim, band=12)
print(result.cost, result.to_frame_token_map())

# bidirectional retrieval metrics
a2t, t2a = evaluate_bidirectional(audio_emb, text_emb, ks=(1, 5, 10))
print(a2t.as_row())

Two runnable demos live under examples/:

python examples/align_demo.py
python examples/retrieval_demo.py

Results

Numbers below are on EchoCaps-mini, a 2k-pair held-out split of short environmental-sound captions (16 kHz, ≤ 10 s clips). Audio tower is a 4-layer transformer over 80-dim log-mels; text tower is DistilBERT. Reported as percentages (higher is better) except median rank (lower is better).

Audio → Text

Model	R@1	R@5	R@10	MedR
Random init (no training)	0.1	0.5	1.0	998
Triplet (hard-neg)	18.6	44.2	57.9	7
InfoNCE	24.3	53.1	66.8	5
InfoNCE + hybrid (α=0.7)	26.1	55.7	69.4	4

Text → Audio

Model	R@1	R@5	R@10	MedR
Triplet (hard-neg)	17.9	43.0	56.4	8
InfoNCE	23.8	52.0	65.5	5
InfoNCE + hybrid (α=0.7)	25.4	54.6	68.1	5

Aggregate rsum (sum of all six recalls) for the hybrid model: 299.3.

Changelog

Version	Date	Highlights
v0.1	2024-10	Two-tower encoders, InfoNCE loss, first end-to-end train loop.
v0.2	2024-12	DTW forced-alignment module + monotonic backtrace; numpy-only.
v0.3	2025-02	Retrieval metrics (R@k, MedR, MRR) vectorised over the sim matrix.
v0.4	2025-04	Manifest dataset + padding collate; benchmark runner & reports.
v0.5	2025-06	Hybrid InfoNCE/triplet loss; Sakoe-Chiba band; config validation.
v0.6	2025-08	CLI polish, lazy imports, docs & examples, packaging.

Citation

If you find this toolkit useful, please cite:

@misc{wu2025alignecho,
  title        = {AlignEcho: A Contrastive Two-Tower Toolkit for
                  Audio-Text Alignment and Cross-Modal Retrieval},
  author       = {Wu, Yunling},
  year         = {2025},
  howpublished = {\url{https://github.com/ella-brown629/alignecho}},
  note         = {Version 0.6.0}
}

License

Released under the MIT License. © 2024 Wu Yunling.