[Roadmap] WebAR on jsfeatNext: two-layer architecture & high-level feature set

[kalwalt]

Umbrella / roadmap issue. Captures the architecture and the full feature set for building a WebAR solution on top of jsfeatNext, from the "Libreria realtà aumentata con jsfeatNext" design discussion. Individual components will spin off into their own issues (in a dedicated high-level AR repo once it exists).

Goal

Natural-feature-tracking WebAR for planar (image) targets — the classic ARToolKit-NFT / Vuforia-image-target use case: detect a reference image in the camera feed and anchor a 3D object on it. (Not full SLAM, not arbitrary surfaces.)

Two-layer architecture

The decision from the discussion: jsfeatNext stays a generic, stateless image-processing / CV library. Everything stateful and AR-specific lives in a separate high-level, stateful AR project (TypeScript), which talks to the CV backend only through the minimum CvBackend interface (#96).

┌─────────────────────────────────────────────────────────────┐
│  High-level AR project — TypeScript, STATEFUL                │
│  target training · tracking loop · pose refinement ·         │
│  geometric validation · temporal filter · renderer adapters  │
└───────────────────────────┬─────────────────────────────────┘
                            │  CvBackend interface (#96)
                            │  detect · describe · match ·
                            │  estimateHomography · poseFromHomography
        ┌───────────────┬───────────────┬──────────────┬──────────────┐
  standard          WASM          WebARKitLib     jsartoolkitNFT
  (jsfeatNext)      (PureCV)      (OpenCV/WASM)   (KPM/WASM)
  pure TS/JS        Rust→WASM     ← already       ← future candidate
  portable          fast          implements      (needs rewrite)
                                  this pipeline

The layer is backend-agnostic and "elastic": pick standard (jsfeatNext), WASM (PureCV), or — with the rewrite noted in #96 — WebARKitLib / jsartoolkitNFT, all behind the same contract.

Readiness map (state of the code)

✅ In jsfeatNext (stateless CV primitives) — the vision half is essentially done:

• Grayscale + image pyramid (imgproc, pyramid_t)
• Keypoint detection (fast_corners, yape, yape06)
• ORB descriptors (orb.describe)
• Homography + RANSAC (homography2d, motion_estimator)
• Optical flow (optical_flow_lk) — the building block for frame-to-frame tracking

🟡 Prototyped (promote to jsfeatNext modules) — tracked in #83:

• bfmatcher — brute-force Hamming matcher (was inline in sample_orb_pinball.html)
• pose_estimator — H → (R, t) planar pose decomposition (camera frame only)

⭐ Prior art: WebARKitLib already implements this pipeline (OpenCV/WASM)

Important context — the high-level layer is NOT green-field. WebARKitLib (C++ → emscripten/WASM, driven by webarkit-testing, an emscripten port of artoolkitX's OCVT planar-tracking design) is a working implementation of exactly this pipeline, built on OpenCV:

• Detection/description: cv::ORB and cv::AKAZE
• Matching: cv::BFMatcher
• Tracking loop: cv::calcOpticalFlowPyrLK + TrackingPointSelector — i.e. the tracker_t detect↔track↔recover machinery already exists there
• Homography: WebARKitHomographyInfo
• Renderer adapters: getPoseMatrixGL() and getCameraProjectionMatrix() — the modelViewGL/projectionGL equivalents, already solved
• JS side: webarkit-testing's WebARKitController.js (init_raw() / process_raw(), getMarker event, pose/matrix getters) is a real high-level controller

What this means for this roadmap: every component below should be designed with WebARKitLib as the reference, not invented from scratch — it is the proven design (and a numeric/behavioural comparison target). The open question is not how to build a planar tracker, but how to factor it so the same orchestration works over a pluggable backend (#96) instead of being fused to one CV implementation.

It also plugs into #96 in two distinct ways:

1. As a granular CvBackend — needs the "valuable rewrite" to expose primitives individually (its public API is processFrameData() → pose, a bundled pipeline).
2. As a whole-pipeline tracker — it could satisfy a coarser TrackerBackend seam (frame in → pose out) as-is, with no rewrite, letting the high-level layer delegate wholesale instead of composing primitives. Worth evaluating before writing tracker_t from zero.

🔴 To build — the high-level AR layer (this roadmap):

• Target training — pattern_t. Reference image (+ its real dimensions) → reusable multi-scale descriptor set with associated model-plane coordinates. Today this is throwaway inline code in the sample; needed to make "targets" first-class objects.
• Tracking loop — tracker_t ⭐ the real bottleneck. A detect → track → recover state machine: once the target is locked, don't re-run ORB every frame — follow the inlier points frame-to-frame with optical_flow_lk and re-estimate the homography from the tracked points; only re-run the heavy detection when tracking is lost or confidence drops. This is what turns a stuttering demo into a 30–60 fps tracker (likely why the sample "didn't work well"). All the bricks exist (optical_flow_lk, pyramid_t, motion_estimator) — the orchestration is what's missing.
• Renderer adapters — modelViewGL + projectionGL. modelViewGL(pose): column-major 4×4 for three.js/WebGL, CV→GL via diag(1,-1,-1). projectionGL(K, w, h, near, far): the GL projection consistent with the same intrinsics K — without it, even a perfect pose won't line up. These are rendering glue, so they live here, not in jsfeatNext (keeps the CV lib renderer-agnostic). (Moved out of jsfeatNext scope — see the Add bfmatcher and pose_estimator modules to complete the natural-feature AR pipeline #83 correction.)
• Geometric validation. Reject implausible RANSAC homographies (projected-corner convexity, determinant sign, aspect-ratio sanity) to stop jitter/flipping.
• Pose refinement. Non-linear minimization of reprojection error (small Gauss-Newton / Levenberg-Marquardt); linalg gives the algebra, the solver is missing.
• Temporal filter. Smooth the pose across frames to kill jitter.
• Undistort (accuracy). Apply known distortion coefficients (could be an imgproc helper in jsfeatNext, or high-level). Full calibration is done offline; pose_estimator.intrinsics() bootstraps without it.

⚪ App layer (out of the library entirely): camera capture (getUserMedia → canvas → RGBA), the engine loop stitching capture → CV → pose → rendering, and the three.js bridge. Running the CV pipeline in a WebWorker keeps the main thread free — and heavy frames are exactly where the WASM (PureCV) backend beats pure JS.

Open question worth settling early

Given WebARKitLib already works: is the goal (a) a new TS orchestration layer that can drive any backend — with jsfeatNext as the portable no-WASM option and WebARKitLib/PureCV as the fast ones — or (b) jsfeatNext-based AR specifically, as a dependency-free alternative to the WASM stack? The components below are the same either way, but it changes whether tracker_t is written fresh or wraps an existing tracker.

Suggested sequencing

1. pattern_t (target training) + projectionGL — closes the end-to-end loop so you can see something anchored.
2. tracker_t — the detect↔track↔recover loop; the single biggest lever from "demo" to "usable".
3. Then robustness: geometric validation → temporal filter → pose refinement.

What stays in jsfeatNext vs here

In jsfeatNext (stateless, generic)	High-level AR project (stateful)
bfmatcher, pose (R,t), detect/ORB, estimateHomography (#83, #96)	pattern_t, tracker_t, pose refinement, geometric validation, temporal filter, renderer adapters (modelViewGL, projectionGL)

Non-goals

• Full SLAM / arbitrary-surface tracking (planar image targets only).
• Real camera calibration workflow (done offline; only bootstrap intrinsics here).
• Creating the high-level AR repo in this issue — this is the roadmap; the repo + per-component issues come when the work starts.

Related

• CV backend contract: Define a minimum CvBackend interface (pluggable, stateless CV contract for a high-level WebAR layer) #96
• CV primitives to promote: Add bfmatcher and pose_estimator modules to complete the natural-feature AR pipeline #83
• Prototypes & full design rationale: the "Libreria realtà aumentata con jsfeatNext" discussion.
• Prior art / reference implementation: WebARKitLib (OpenCV → WASM tracker) and webarkit-testing (build glue, JS controller, examples).