RGDTrack: Radar-Guided Dual-Level Tracking with End-to-End 4D Radar-Camera Fusion for Unmanned Surface Vehicles

About RGDTrack

Abstract

RGDTrack is an end-to-end 4D radar-camera tracker that converts temporally aggregated radar returns into learnable tokens and injects them into a MixFormerV2 tracker at two complementary levels. Radar Guided Tokenization (RGT) maps radar returns from a local temporal window into the current visual search crop and represents each return using normalized crop coordinates and a clipped signed time offset. Radar Search Fusion (RSF) lets dense visual search tokens query the radar tokens before box regression. Radar Regression Fusion (RRF) then lets four regression tokens query a joint search-radar memory to refine the final box prediction. The complete model is optimized end to end with the same supervised tracking losses as the visual baseline.

On the official USVTrack test split, RGDTrack improves the MixFormerV2-Base baseline from 0.6363 to 0.7008 AUC and from 0.6086 to 0.6770 HOTA. The efficient RGDTrack-S variant also improves its matched MixFormerV2-S baseline while running at 64.52 inference FPS.

Contributions

• Radar Guided Tokenization (RGT), a module that converts asynchronous radar point streams into transformer-compatible tokens aligned with the visual search crop.
• Radar Search Fusion (RSF), a cross-attention module that lets visual search tokens query radar tokens, injecting complementary geometric evidence into search-feature formation before box regression.
• Radar Regression Fusion (RRF), a joint search-radar memory that lets four regression tokens attend to the enhanced search representation and radar tokens, thereby refining box prediction with radar-conditioned localization cues.

Method overview

Repository layout

RGDTrack/
├── configs/
│   ├── rgdtrack_base.json    default Base configuration
│   └── rgdtrack_small.json   optional Small configuration
├── rgdtrack/
│   ├── model.py              compact RGDTrack fusion module
│   ├── radar_tokens.py       public radar-token utilities
│   ├── descriptors.py        strict descriptor definitions
│   ├── train.py              config-driven training entry point
│   ├── eval.py               validation/test entry point
│   ├── test_best.py          evaluate validation-selected checkpoint
│   ├── all_metrics.py        SOT, HOTA, and MOT metric aggregation
│   └── runners/              complete MixFormerV2-compatible backend
├── scripts/                  shell wrappers
└── weights/                  downloaded trained checkpoints (not tracked by Git)
├── MixFormerV2/              bundled MixFormerV2 source and Base/Small YAMLs

Datasets, trained checkpoints, experiment outputs, and logs are intentionally excluded from Git. Download the released checkpoints as described below and place them in weights/.

Model variants

Variant	Config	MixFormerV2 architecture	Template / search
RGDTrack (default)	configs/rgdtrack_base.json	depth 8, MLP ratio 4	128 / 288
RGDTrack-S	configs/rgdtrack_small.json	depth 4, MLP ratio 1	112 / 224

Both variants use temporal radius 2 (at most five radar frames), 128 radar tokens, and the three-dimensional descriptor [crop_u, crop_v, dt]. The time offset is clipped to [-0.5, 0.5] seconds and normalized by 0.5. The internal feature-mode identifier base_3d_crop_uv_dt names this descriptor and is shared by both backbone sizes.

Environment

The reported experiments used:

Python 3.11.15
PyTorch 2.11.0+cu130
torchvision 0.26.0+cu130
CUDA 13.0 / cuDNN 91900
NumPy 2.4.4
OpenCV 4.13.0
timm 1.0.26
einops 0.8.2

Install a CUDA-enabled PyTorch build first, then install the remaining dependencies:

python -m pip install -r requirements.txt
python -m pip install -e .

The exact CUDA wheel/index depends on the host system. The version pins in requirements.txt document the reproduction environment and may be relaxed when using another compatible CUDA stack.

Data preparation

Download USVTrack train/validation/test data from the dataset release: Baidu Netdisk.

Use this local layout, or edit the corresponding paths in the JSON configs:

data/USVTrack/
├── images/
│   ├── train/
│   ├── val/
│   └── test/
└── radaruv/
    └── <sequence_id>.csv

Radar CSV files must contain:

timestamp,u,v,range,doppler,power,x,y,z,comp_velocity

Place the official visual initialization checkpoints at:

checkpoints/mixformerv2_base.pth.tar
checkpoints/mixformerv2_small.pth.tar

The architecture and checkpoint must match. Base uses MixFormerV2/experiments/mixformer2_vit_online/288_depth8_score.yaml; Small uses 224_depth4_mlp1_score.yaml.

Training

Run the default RGDTrack experiment with the Base backbone:

python -m rgdtrack.train --config configs/rgdtrack_base.json
# or
scripts/train.sh

Run RGDTrack-S:

python -m rgdtrack.train --config configs/rgdtrack_small.json
# or
scripts/train.sh configs/rgdtrack_small.json

The paper protocol uses AdamW, batch size 64, 60,000 samples per epoch, at most 100 epochs, seed 42, and early stopping after 30 validation rounds without AUC improvement. Validation starts at epoch 30 and is then performed at one-epoch intervals. Most of the visual backbone is frozen; the final Transformer block and normalization layer, regression tokens, prediction heads, and radar modules are trainable.

Training creates the following under the configured output_dir:

best.pth.tar              validation-AUC-selected checkpoint
last.pth.tar              latest checkpoint
validation_history.json
experiment_config.json
logs/
eval/

Evaluation

Evaluate a checkpoint on the test split:

# RGDTrack (Base, default)
python -m rgdtrack.eval \
  --config configs/rgdtrack_base.json \
  --checkpoint weights/rgdtrack_base.pth.tar \
  --split test \
  --out-dir outputs/rgdtrack-base/test

# RGDTrack-S
python -m rgdtrack.eval \
  --config configs/rgdtrack_small.json \
  --checkpoint weights/rgdtrack_small.pth.tar \
  --split test \
  --out-dir outputs/rgdtrack-small/test

Evaluate the validation-selected checkpoint directly:

python -m rgdtrack.test_best --config configs/rgdtrack_base.json
# Small:
python -m rgdtrack.test_best --config configs/rgdtrack_small.json

The evaluator writes metrics.json and per-trajectory predictions under tracks/. To compute the complete paper metrics:

python -m rgdtrack.all_metrics \
  --config configs/rgdtrack_base.json \
  --eval-dir outputs/rgdtrack-base/test \
  --split test

The merged report contains AUC, P@20, mIoU, HOTA, DetA, AssA, LocA, MOTA, MOTP, IDF1, IDSW, TP, FP, and FN.

Main results

The following results were measured on the official USVTrack test split. Each RGDTrack variant is compared with the visual-only MixFormerV2 model using the same backbone scale. Latency is the mean runtime measured serially on a selected sequence at batch size one after a 30-frame warm-up. Lower latency and IDSW are better, while all other metrics are higher-is-better.

Method	Modality	AUC	P@20	mIoU	HOTA	MOTA	IDF1	IDSW	Latency (ms)
MixFormerV2-S	Camera	0.5985	0.5407	0.6005	0.5792	0.4371	0.7087	367	12.28
RGDTrack-S	4D Radar + Camera	0.6226	0.5760	0.6260	0.5880	0.4898	0.7401	196	15.50
MixFormerV2	Camera	0.6363	0.6301	0.6397	0.6086	0.5036	0.7290	799	17.38
RGDTrack	4D Radar + Camera	0.7008	0.7114	0.7093	0.6770	0.6735	0.8334	194	21.29

Compared with MixFormerV2, RGDTrack gains +0.0645 AUC, +0.0813 P@20, +0.0696 mIoU, +0.0684 HOTA, +0.1699 MOTA, and +0.1044 IDF1, while reducing ID switches by 605. RGDTrack-S gains +0.0241 AUC, +0.0353 P@20, +0.0255 mIoU, +0.0088 HOTA, +0.0527 MOTA, and +0.0314 IDF1, while reducing ID switches by 171.

Released checkpoints

The fully trained, validation-selected checkpoints for the two formal RGDTrack variants are released for direct evaluation and reproduction of the reported test results. Download them from Baidu Netdisk (extraction code: 4nsx).

After downloading, place the checkpoint files in weights/ using the names shown below:

Model	Checkpoint	Selected epoch	AUC	HOTA
RGDTrack	weights/rgdtrack_base.pth.tar	85	0.7008	0.6770
RGDTrack-S	weights/rgdtrack_small.pth.tar	48	0.6226	0.5880

The reported AUC and HOTA values are measured on the official USVTrack test split. The checkpoints use the final three-dimensional radar descriptor [crop_u, crop_v, dt]. RGDTrack must be evaluated with configs/rgdtrack_base.json, while RGDTrack-S must be evaluated with configs/rgdtrack_small.json. These are trained RGDTrack evaluation checkpoints, not the official MixFormerV2 initialization checkpoints used to start RGDTrack training.

Reproducibility notes

• Use the same software environment documented above when reproducing the reported results.
• Keep backbone_size, the MixFormerV2 YAML, and the visual checkpoint consistent.
• Do not change the Base/Small crop sizes independently of the selected YAML.
• The default public configuration is the final 3D [crop_u, crop_v, dt] version.
• The necessary MixFormerV2 implementation is included in MixFormerV2/ so a developer does not need to clone another repository.

Citation

If this repository is useful, please cite the RGDTrack paper.