feature(6d): add lightweight RGB coarse pose initialization for SRT3D

[Som5ra]

Summary

Add a lightweight coarse 6D pose initializer for the existing RGB-only SRT3D tracking pipeline so tracking can start—and later recover—without a hard-coded first-frame pose.

This should begin as a research and benchmarking spike, then implement the smallest reliable path suitable for eventual mobile deployment.

Current state

The current pipeline can successfully track a known object from RGB input and an uncolored CAD mesh once a usable initial pose is supplied. SRT3D then performs local pose refinement and frame-to-frame tracking.

The missing part is global/coarse initialization:

• the first frame currently depends on a manually supplied/hard-coded 4×4 pose (approximately 0.8 m away);
• there is no coarse pose estimator, PnP initializer, or learned/template-based global search;
• the lost-track path restores the last pose rather than globally reacquiring the object;
• therefore the system is a working local tracker, but not yet a complete end-to-end 6D object tracking system.

Previous LINE2D/LINEMOD attempt

A shape-based template initializer was attempted with approximately:

• 640×480 templates;
• pyramid strides {4, 8};
• 128 features;
• more than 1,000 rendered poses, expanded by scale to roughly 10,000 templates;
• synthetic objects rendered on a black background.

It responded to rendered training images and renders pasted into test frames, but generally failed on the real object captured by a webcam. Increasing the number of templates also hurt runtime.

Likely causes to test explicitly:

• synthetic-to-real appearance and edge/gradient gap;
• incomplete viewpoint coverage;
• camera-intrinsics mismatch between rendering and inference;
• lighting, blur, antialiasing, exposure, background, and occlusion differences;
• scale augmentation being used where metric translation should instead come from calibrated geometry;
• insufficient confidence calibration and no robust top-K refinement stage.

More LINE2D template count alone is not expected to solve these issues reliably.

Requirements and constraints

• RGB-only inference; depth must not be required.
• Known, potentially untextured/uncolored CAD mesh.
• Coarse first-frame pose is sufficient because SRT3D can refine and track it.
• Offline template rendering and object onboarding on a PC are acceptable.
• Runtime should ultimately be deployable on a phone.
• Returning top-K pose hypotheses is acceptable and preferable for ambiguous or symmetric objects.
• Real-camera performance matters; success on synthetic renders alone is insufficient.
• Object symmetries must be represented as equivalent poses or multiple hypotheses.

Candidate direction

Recommended architecture: GigaPose-inspired mobile initializer + SRT3D

Use GigaPose's decomposition as the design reference rather than immediately integrating the full research stack unchanged:

1. obtain an object ROI/mask;
2. encode the RGB crop with a compact learned encoder;
3. retrieve top-K out-of-plane pose templates from a small precomputed codebook;
4. estimate the remaining in-plane rotation, scale, and 2D translation from patch correspondences;
5. convert candidates to metric 6D poses using calibrated camera intrinsics and mesh geometry;
6. run short SRT3D refinement/scoring for each candidate;
7. accept the best sufficiently confident hypothesis and begin tracking.

This avoids a dense scale-expanded LINE2D database and lets the existing tracker do the precise local optimization it is already good at.

GigaPose CPU/GPU caveat

GigaPose is not conceptually restricted to a GPU, but its official implementation and published performance are GPU-oriented. The reported approximately 48 ms per detection is a GPU result, not evidence of fast CPU or phone-CPU inference. Its dense DINOv2/ViT feature extraction is likely to dominate CPU latency, although the rendered template features can be precomputed.

Therefore:

• benchmark the official model on CPU before making any performance claim;
• do not make full GigaPose a required production dependency yet;
• treat it as an accuracy/reference baseline;
• investigate a distilled MobileNetV3/MobileViT-style encoder and ONNX/Core ML/TFLite/NNAPI deployment;
• report latency, memory, size, energy, and thermal behavior on the actual target device.

An Augmented Autoencoder-style learned codebook is a useful lower-complexity P0 baseline. Full GigaPose-style patch correspondence can follow if the baseline lacks sufficient viewpoint recall or robustness.

Proposed work

P0 — Research spike and minimum initializer

• Define a CoarsePoseInitializer interface returning top-K {pose, confidence, metadata} hypotheses.
• Record the current hard-coded-pose baseline and SRT3D convergence basin experimentally.
• Build a small real-camera evaluation set with ground truth or manually validated poses, including lighting, clutter, blur, partial occlusion, and distance variation.
• Benchmark LINE2D, an AAE-style learned template codebook, and official GigaPose where feasible.
• Benchmark full GigaPose by component on available x86/ARM CPU and GPU:
  • query feature extraction;
  • template retrieval;
  • patch matching/4DoF prediction;
  • RANSAC/geometric recovery;
  • detector/segmenter overhead.
• Implement the smallest learned codebook prototype using an externally supplied ROI/mask.
• Return top-K candidates and use short SRT3D refinement plus a common score to select the winner.
• Remove the hard-coded first-frame pose from the normal initialization path.

P1 — GigaPose-style geometric head

• Replace scale-expanded templates with a compact out-of-plane template set.
• Add patch-level correspondences for in-plane rotation, scale, and image translation.
• Add RANSAC or another robust estimator for noisy correspondences.
• Add symmetry-aware hypothesis generation and evaluation.
• Add calibrated confidence thresholds and failure behavior.
• Reuse the initializer for global relocalization after tracking is lost.

P2 — Mobile optimization

• Distill or replace the desktop feature encoder with a mobile backbone.
• Export through ONNX and evaluate Core ML and/or TFLite/NNAPI paths.
• Evaluate FP16 and INT8 quantization with accuracy regression limits.
• Add or integrate a lightweight detector/segmenter if an ROI cannot be supplied externally.
• Measure warm/cold latency, peak RAM, binary/model size, energy use, and sustained thermal performance on target phones.
• Review licenses and redistribution terms for code, pretrained weights, and training data.

Research questions

• What is the actual CPU latency of official GigaPose on our crop sizes and hardware? There is currently no official CPU-fast result to rely on.
• Which parts require code changes to run without CUDA?
• How much accuracy is lost when replacing/distilling the DINOv2/ViT features?
• What is the smallest template count that preserves top-K recall within SRT3D's convergence basin?
• Can SRT3D's own residual/region score reliably rank coarse hypotheses, or is a separate verification score required?
• How sensitive is metric translation to camera calibration and crop/resize transforms?
• Is an external bounding box/mask acceptable for P0, and what detector/segmenter budget is available later?
• How should object symmetries and visually indistinguishable views be encoded?
• Should mobile deployment optimize for CPU-only, or may it use the phone GPU/NPU through Core ML/NNAPI?
• What frame latency is acceptable for one-shot initialization versus continuous relocalization?

Validation and tests

• Unit-test crop, resize, intrinsics, scale, and translation conversions.
• Test mismatched rendering/inference intrinsics explicitly.
• Test deterministic viewpoint coverage and template indexing.
• Use symmetry-aware pose metrics where applicable.
• Report top-1 and top-K coarse recall before refinement.
• Report success after SRT3D refinement, not only raw estimator error.
• Include real-camera integration tests; do not accept synthetic-paste-only results.
• Verify failure cases do not silently start tracking from a low-confidence pose.
• Verify relocalization does not simply restore a stale last pose.

Initial acceptance criteria

P0 is complete when:

• an RGB frame, calibrated camera, known CAD mesh, and supplied ROI/mask can produce top-K pose hypotheses without a manually entered initial pose;
• at least one candidate falls inside the measured SRT3D convergence basin on the agreed real-camera evaluation set;
• SRT3D refinement selects/validates a usable pose and begins tracking;
• latency, model size, and peak memory are measured on both the development machine and at least one representative mobile target or mobile-class runtime;
• known failure cases and confidence behavior are documented;
• the implementation remains modular so the AAE/codebook encoder can later be replaced by the GigaPose-style geometric initializer.

Non-goals for P0

• Reproducing the entire GigaPose/CNOS/MegaPose research ecosystem inside this repository.
• Solving category-level or unseen-object pose estimation.
• Requiring depth input.
• Guaranteeing a unique single-frame pose for symmetric/visually ambiguous objects.

References

• GigaPose paper: https://arxiv.org/abs/2311.14155
• GigaPose code: https://github.com/nv-nguyen/gigapose
• GigaPose project page: https://nv-nguyen.github.io/gigapose/
• Augmented Autoencoders: https://arxiv.org/abs/1804.03177
• OpenCV real-time textureless object pose tutorial: https://docs.opencv.org/3.4/dc/d2c/tutorial_real_time_pose.html
• OpenCV LINEMOD detector: https://docs.opencv.org/4.x/d7/d07/classcv_1_1linemod_1_1Detector.html