RIFT: Representation Inconsistency Forensics on Trajectories

Official implementation for the paper "Mind the Rift: Cross-Scale Coupling Mismatch for AI-Generated Video Detection" (ACM Multimedia 2026).

Overview

RIFT detects AI-generated videos by measuring the cross-scale coupling mismatch between macro-level temporal dynamics and micro-level residual patterns. In natural videos, these two scales are coupled by unified imaging physics. AI generators synthesize each scale independently, breaking this coupling.

Architecture

Video Frames
    |
    v
[Frozen Encoders] -- DINOv2 ViT-S/14 (CLS + patch tokens)
                  -- RAFT-Large (optical flow + residuals)
    |
    v
[Orthogonal Decoupling] -- z_macro, z_micro (Gram-Schmidt guarantee)
    |                |
    v                v
[Macro Stream]   [Micro Stream]
 Dynamic          Sensitive
 Baseline         Forensic Probe
 - Manifold       - SRM 30 kernels
   embedding      - Bayar 16 kernels
 - Differential   - BiGRU temporal
   geometry       - Consistency
 - Persistent       metrics
   homology
    |                |
    v                v
[Coupling Divergence] -- P(micro|macro) via NLL + MINE
    |
    v
[Gated Fusion] --> Real / Fake

Key Results

Metric	Value
Accuracy	99.22%
F1 Score	99.33%
AUC	99.96%
Unseen generator (LOGO)	97.87%
Encoder sensitivity (ViT-S vs ViT-L)	< 0.1%

Installation

# Create environment
conda create -n rift python=3.10
conda activate rift

# Install PyTorch (adjust CUDA version as needed)
pip install torch==2.5.1 torchvision==0.20.1 --index-url https://download.pytorch.org/whl/cu124

# Install dependencies
pip install -r requirements.txt

Dataset Preparation

RIFT is trained on videos from two public sources:

Source	Type	Videos	Description
DVSC2023	Real	~60K	Meta AI Video Similarity Challenge reference set
VidProM	Fake	~60K	7 generators: Pika, ModelScope, VideoCrafter2, ZeroScope, CogVideo, T2V-Zero, LaVie

Automatic Download

python data/download_data.py

This downloads videos into:

data/
  videos/
    real/       # DVSC2023 reference videos
    fake/       # VidProM AI-generated videos

Feature Precomputation

To avoid recomputing frozen encoder outputs during training:

# Step 1: Extract DINOv2 + RAFT features (~170KB per video)
python scripts/precompute_features.py --batch_size 1 --workers 4

# Step 2 (after Stage 1 training): Compute topology cache
python scripts/precompute_topology.py --checkpoint checkpoints/best_stage1.pt

Features are saved to data/precomputed_features/ as .npz files.

Training

RIFT uses a two-stage training strategy:

• Stage 1 (30 epochs, lr=1e-3): Orthogonal decoupling pretraining with higher manifold weight
• Stage 2 (50 epochs, lr=3e-4): Full module joint training with early stopping

# Full training (Stage 1 + Stage 2)
python scripts/train.py --config configs/default.yaml

# Resume from Stage 1 checkpoint
python scripts/train.py --stage2_only --resume_stage1 checkpoints/best_stage1.pt

Checkpoints are saved to checkpoints/.

Evaluation

# Evaluate on validation split
python scripts/evaluate.py \
    --checkpoint checkpoints/best_stage2.pt \
    --split checkpoints/val_indices.json

# Evaluate on all data
python scripts/evaluate.py \
    --checkpoint checkpoints/best_stage2.pt

Demo

Inference on Individual Videos

# Single video
python scripts/demo_inference.py \
    --video demo/seedance2/snow_forest.mp4 \
    --checkpoint checkpoints/best_stage2.pt

# All Seedance 2.0 demo videos
python scripts/demo_inference.py \
    --video_dir demo/seedance2/ \
    --checkpoint checkpoints/best_stage2.pt

Seedance 2.0 Case Study

The demo/seedance2/ directory contains 26 AI-generated videos from Seedance 2.0 (ByteDance, Feb 2026), covering diverse content categories: nature, urban, sports, animals, food, technology, etc. These serve as a zero-shot out-of-distribution test case, as Seedance 2.0 was not seen during training.

Project Structure

code/
├── configs/
│   └── default.yaml              # Model and training configuration
├── models/
│   ├── rift.py                    # Main RIFT model
│   ├── backbone/                  # Frozen feature extractors
│   │   ├── dino_extractor.py      #   DINOv2 ViT-S/14
│   │   └── raft_extractor.py      #   RAFT-Large optical flow
│   ├── decoupling/                # Orthogonal decomposition
│   │   └── orthogonal_decomp.py   #   Macro/micro projectors + Gram-Schmidt
│   ├── macro_stream/              # Dynamic baseline
│   │   ├── macro_analyzer.py      #   Stream orchestrator
│   │   ├── manifold_encoder.py    #   R^128 -> R^24 embedding
│   │   ├── curvature.py           #   Differential geometry (6 quantities)
│   │   ├── trajectory_temporal.py #   Transformer temporal encoder
│   │   └── topology.py            #   Persistent homology (H0/H1)
│   ├── micro_stream/              # Sensitive forensic probe
│   │   ├── micro_analyzer.py      #   Stream orchestrator
│   │   ├── noise_extractors.py    #   SRM + Bayar constrained convolutions
│   │   ├── residual_analyzer.py   #   CNN + statistics + frequency features
│   │   └── temporal_consistency.py#   BiGRU temporal modeling
│   ├── conditional/               # Coupling divergence
│   │   └── conditional_dependency.py  # NLL + MINE mutual information
│   └── fusion/                    # Classification
│       └── gated_fusion.py        #   Independent sigmoid gates + MLP
├── data/
│   ├── dataset.py                 # VideoDataset + FeatureDataset
│   ├── transforms.py              # Forensic augmentation pipeline
│   └── download_data.py           # Dataset download script
├── training/
│   ├── trainer.py                 # Two-stage trainer
│   └── losses.py                  # Focal + ortho + recon + manifold + cond
├── evaluation/
│   └── evaluator.py               # Evaluation metrics
├── scripts/
│   ├── train.py                   # Training entry point
│   ├── evaluate.py                # Evaluation entry point
│   ├── precompute_features.py     # DINOv2 + RAFT feature extraction
│   ├── precompute_topology.py     # Persistent homology cache
│   └── demo_inference.py          # Single-video inference demo
├── demo/
│   └── seedance2/                 # 26 Seedance 2.0 example videos
├── checkpoints/                   # Model weights (released upon acceptance)
├── configs/
│   └── default.yaml               # Default configuration
├── requirements.txt
├── LICENSE                        # Apache 2.0
└── README.md

Model Components

Module	Input	Output	Description
DINOv2 Extractor	(B,T,3,H,W)	CLS (B,T,384), Patch (B,T,N,384)	Frozen ViT-S/14
RAFT Extractor	(B,T,3,H,W)	Flow (B,T-1,2,H,W), Residual (B,T-1,3,H,W)	Frozen optical flow
Orthogonal Decoupling	CLS + Flow + Patch	z_macro, z_micro in R^128	Gram-Schmidt enforced
Macro Stream	z_macro (B,T,128)	h_macro (B,100)	Geometry + Temporal
Topology	z_macro trajectory	h_topo (B,128)	Vietoris-Rips -> Persistence landscape
Micro Stream	Residual features	h_micro (B,67)	SRM/Bayar + GRU + stats
Coupling Divergence	z_macro, z_micro, h_micro	h_cond (B,17)	NLL + MINE
Gated Fusion	All streams	logits (B,2)	Sigmoid gates + 3-layer MLP

Configuration

Key hyperparameters in configs/default.yaml:

Parameter	Value	Description
d_z	128	Orthogonal embedding dimension
d_manifold	24	Manifold target dimension
d_topo	128	Topology feature dimension
segment_length	32	Frames per video segment
spatial_size	224	Input spatial resolution
focal_gamma	2.0	Focal loss focusing parameter
gs_interval	100	Gram-Schmidt interval (steps)
batch_size	64	Training batch size

Hardware Requirements

• Training: 1x GPU with >= 10GB VRAM (tested on RTX 3080)
• Inference: ~41ms per video (precomputed features), ~461ms end-to-end
• Feature precomputation: ~120K videos in ~24 hours on single GPU

License

This project is licensed under the Apache License 2.0. See LICENSE for details.

Citation

@inproceedings{anonymous2026rift,
  title={Mind the Rift: Cross-Scale Coupling Mismatch for AI-Generated Video Detection},
  author={Anonymous},
  booktitle={Proceedings of the ACM International Conference on Multimedia (MM)},
  year={2026}
}