HyperGRAM: Hyperbolic Gramian Volumes for Multimodal Alignment

Code for the CVPR 2026 paper "Hyperbolic Gramian Volumes for Multimodal Alignment".

HyperGRAM is a hybrid-geometry multimodal alignment framework. It measures the alignment of a set of modality embeddings (text, video, audio, subtitle, depth) by the Gramian volume of the parallelotope they span, and combines a Euclidean and a Lorentz-hyperbolic volume through a learnable mixing weight. Euclidean geometry provides discriminative stability; hyperbolic geometry preserves semantic variance and avoids the volume collapse that L2-normalized Euclidean volumes suffer under.

The alignment loss

This is the core of the method. Given a text anchor t and the other modalities of a candidate sample (video v, audio a, and optionally subtitle s, depth d), we form the m × m Gram matrix G of their L2-normalized embeddings, G_ij = ⟨x_i, x_j⟩, and take the volume of the spanned parallelotope:

V(t, v, a, …) = sqrt( | det(G) | )

Well-aligned tuples are nearly collinear, so their volume is small; mismatched tuples span a larger volume. Implemented in utils/volume.py (volume_computation3/4/5 for 3/4/5 modalities).

Hybrid geometry (HyperGRAM). We compute the volume twice — once in Euclidean space and once in the Lorentz hyperbolic model with a learnable curvature κ — rescale the hyperbolic volume to the Euclidean scale, and mix them with learnable, normalized weights:

w_e = |ω_e| / (|ω_e| + |ω_h|),   w_h = |ω_h| / (|ω_e| + |ω_h|)
V_hybrid = w_e · V_euclidean + w_h · V_hyperbolic

Implemented in utils/hyperbolic_volume.py (hybrid_volume3/4/5; the pure hyperbolic volumes are hyperbolic_volume3/4/5).

Contrastive objective. Volumes are turned into logits by a learnable temperature τ (contra_temp, initialized to 0.07). Since a smaller volume means better alignment, we use the negative volume and apply symmetric cross-entropy with label smoothing over the (distributed) batch, whose diagonal is the set of matched pairs:

logits   = -V / τ
L_align  = ½ · [ CE(-V/τ, y) + CE(-Vᵀ/τ, y) ],   label_smoothing = 0.1

A matching (ITM) loss with hard negatives mined from softmax(-V) is added with weight itm_ratio (default 0.1). The full assembly lives in GRAMModel.forward_ret in model/gram.py.

geometry_mode in the model config selects the space: euclidean · hyperbolic · hybrid (the paper's main model) · and the pmrl / pmrl_volume / hybrid_pmrl variants (utils/pmrl_loss.py).

Where the loss lives in the code:

what	file	symbol
loss expression (contrastive + ITM)	model/gram.py	forward_ret() — the F.cross_entropy(-volume, …) around gram.py:650; pretraining path is forward_ret_vast27m() (gram.py:1226)
Euclidean Gramian volume	utils/volume.py	volume_computation3/4/5
hyperbolic + hybrid volume	utils/hyperbolic_volume.py	hyperbolic_volume3/4/5, hybrid_volume3/4/5
PMRL loss variants	utils/pmrl_loss.py	PMRLLoss

Repository layout

code/
├── run.py                 # training / testing entry point
├── model/                 # GRAM/HyperGRAM model + vision/audio/text encoders
├── utils/
│   ├── volume.py              # Euclidean Gramian volume
│   ├── hyperbolic_volume.py   # hyperbolic + hybrid Gramian volume
│   └── pmrl_loss.py           # PMRL loss variants
├── data/                  # dataset indexing, mappers, loaders
├── evaluation/            # retrieval / classification evaluation
├── configs/
│   ├── default_run_cfg.json   # base run config (merged first)
│   ├── default_model_cfg.json # base model config (merged first)
│   ├── pretrain/              # Stage-1 pretrain configs (VAST150K)
│   └── finetune/              # Stage-2 finetune configs (MSR-VTT)
└── scripts/
    ├── download_vast.py       # download the VAST150K subset
    ├── stage1/                # Stage-1 pretraining launchers
    └── stage2/                # Stage-2 finetuning launchers

A config file is layered: configs/default_{run,model}_cfg.json is loaded first and then overridden by the experiment config's run_cfg / model_cfg, and finally by any command-line flags. When --pretrain_dir is given, encoder fields are inherited from that checkpoint's log/hps.json.

Setup

Dependencies are pinned in pyproject.toml; the launchers use uv:

git clone https://github.com/uta-smile/HyperGram.git
cd HyperGram
uv sync

Data and checkpoints

• VAST150K (Stage-1 pretraining subset): uv run scripts/download_vast.py
• Place evaluation datasets (MSR-VTT, VATEX, …) under datasets/ and their annotations under datasets/annotations/, following the paths in the configs.
• Base checkpoints (e.g. pretrained_models/pretrain_vast) go under pretrained_models/.

Training

All launchers wrap torch.distributed.launch over 8 GPUs; adjust --nproc_per_node and --master_port as needed.

Stage 1 — pretrain on VAST150K:

bash scripts/stage1/baseline/train_stage1_pretrain_vatex_val.sh          # learnable hybrid
bash scripts/stage1/curvature/train_stage1_pretrain_curve_1.2.sh         # fixed curvature κ=1.2
bash scripts/stage1/fixed_weights/train_stage1_pretrain_fixed_0.7.sh     # fixed 0.7/0.3 mix

Stage 2 — finetune on MSR-VTT:

bash scripts/stage2/msrvtt/train_stage2_finetune_msrvtt_pre.sh

Each script writes checkpoints and logs to output2/<experiment>/.

Key config fields

Model (model_cfg):

field	meaning
geometry_mode	euclidean / hyperbolic / hybrid / pmrl*
curvature_init	initial hyperbolic curvature κ
learn_curvature	whether κ is trainable
initial_euclidean_weight / initial_hyperbolic_weight	initial mixing weights ω_e, ω_h
learn_hybrid_weights	whether the mixing weights are trainable
gradient_clip_hyperbolic	gradient clip on the hyperbolic branch

Run (run_cfg): learning_rate, train_epoch, valid_freq, save_steps, fp16, grad_norm, first_eval, save_best.

Citation

@InProceedings{Na_2026_CVPR,
    author    = {Na, Saiyang and Jiang, Feng and Zhou, Qifeng and Zhong, Wenliang and Dang, Thao M. and Guo, Yuzhi and Ma, Hehuan and Li, Chunyuan and An, Weizhi and Huang, Junzhou},
    title     = {Hyperbolic Gramian Volumes for Multimodal Alignment},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2026},
    pages     = {37756-37765}
}

HyperGRAM builds on GRAM and the VAST codebase.