Jiwenc nv/robot viz by jiwenc-nv · Pull Request #995 · NVIDIA/IsaacTeleop

jiwenc-nv · 2026-08-21T17:36:49Z

Description

The MuJoCo-backed digital twin was buried in examples/mujoco_xr — compiled as part of that example, reachable only by installing it, and gated on the developer having a matching mujoco wheel. It is a Televiz capability, not an example. The backend moves to src/viz/robot_twin and its Python surface to isaacteleop.viz.robot, both shipping in the wheel; the app that is left behind is renamed examples/robot_viz.

deps/third_party now fetches MuJoCo and builds it under a private name, reached through dlopen/dlsym so the extension carries no undefined mj* for a foreign libmujoco to answer. Mujoco.cmake beside it states that contract. Users may pip install mujoco at any version, or none.

Two calls for a reviewer: the wheel now redistributes MuJoCo binaries, and every BUILD_VIZ build compiles it with no opt-out (~40 s, needs libegl-dev).

No tests here — they are #999, one commit on top of this branch. This alone drops six test files that exist on main, so the two should land together.

Type of change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Documentation update

isaacteleop_examples.mujoco_xr is gone; the docs that referenced it are updated.

Testing

ctest on Linux aarch64 (Jetson AGX Orin), Release, CUDA, Python 3.12: 316 passed, 0 failed. #999 on the same machine is 330 / 0.

readelf on the built artifacts confirms the isolation: one export, no libmujoco or EGL NEEDED, private SONAME, SYMBOLIC set.

Checklist

I have read and understood the contribution guidelines
I have run the linter and formatter with SKIP=check-copyright-year pre-commit run --all-files
I have made corresponding changes to the documentation
I have added tests that prove my fix/feature works (or explained why not)
I have signed off all my commits (git commit -s) per the DCO

coderabbitai · 2026-08-21T17:37:02Z

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

_{Comment @coderabbitai help to get the list of available commands.}

    "LocomotionRootCmdRetargeter",
    "LocomotionRootCmdRetargeterConfig",
    # Manipulator retargeters
+    "ControllerPoseSource",


    "LocomotionRootCmdRetargeterConfig",
    # Manipulator retargeters
+    "ControllerPoseSource",
+    "EngageAlignmentGate",


    # Manipulator retargeters
+    "ControllerPoseSource",
+    "EngageAlignmentGate",
+    "EngageGateConfig",


+    "ControllerPoseSource",
+    "EngageAlignmentGate",
+    "EngageGateConfig",
+    "GateVerdict",


+                # handles, so destroying the session first would pull them out from
+                # under it. `destroy()` is what releases this.
+                self._destroy.wait()
+        except BaseException as error:  # noqa: BLE001 -- reported to the owner


+    "JointMap",
+    "RobotTwin",
+    "RobotTwinPublisher",
+    "SceneTwin",


The MuJoCo-backed digital twin was buried in examples/mujoco_xr, compiled as part of the example and reachable only by installing that example. It is a Televiz capability, not an example, so the backend moves to src/viz/robot_twin and the Python surface to isaacteleop.viz.robot, both shipping in the wheel. What is left of the example is the app itself, renamed examples/robot_viz. The twin no longer depends on a `mujoco` wheel. cmake/Mujoco.cmake fetches MuJoCo and builds it under a private name, reached by dlopen/dlsym so the extension carries no undefined mj* for a foreign libmujoco to answer; the example's version-pin cross-check goes with it. Also here: the engage gate and controller pose source under isaacteleop.retargeters, and the twin runner in teleop_session_manager. Tests are stripped from this branch and land in a follow-up PR. Signed-off-by: Jiwen Cai <jiwenc@nvidia.com>

MuJoCo was fetched from src/viz/CMakeLists.txt, the one dependency not declared where every other one is. Its FetchContent_Declare now sits beside glfw and glm under the same BUILD_VIZ gate, and the isolation contract it exists for moves next to it as deps/third_party/Mujoco.cmake. Declaration order relative to every other dependency is unchanged. ISAACTELEOP_MUJOCO_VERSION becomes a cache variable because src/viz/robot_twin_tests is now a sibling of where it is set. Two mechanisms did not survive being asked to justify themselves. `-Wl,--exclude-libs,ALL` is redundant against the version script, a whitelist and so strictly stronger: on robot_twin_py the version script alone leaves 1 export and --exclude-libs alone leaves 11. The BUILD_SHARED_LIBS guard cannot fire -- upstream's add_library is unconditionally SHARED off Emscripten -- and the failure it converted was already loud. include_guard(GLOBAL) goes with the single include site. The OUTPUT_NAME comment had the interposition backwards. The wheel's extension modules carry DT_NEEDED on libmujoco.so.3.x, so it is ours answering theirs: loaded first under an unrenamed SONAME, our copy is the only libmujoco the process maps and the user's `import mujoco` silently runs on it. Signed-off-by: Jiwen Cai <jiwenc@nvidia.com>

The follow-up half of #995, which carries the twin with every test stripped out. All 20 files land here: src/viz/robot_twin_tests, examples/robot_viz/tests, the four src/viz/python_tests/test_robot_*.py, test_engage_gate.py and test_robot_twin_session.py, plus the CMake that registers them and the example README's Tests section. test_symbol_isolation.py runs against the shipped _robot_twin rather than a test-only extension carrying a second private MuJoCo. Review asked why the tests reached MuJoCo directly, and they should not have: that probe's two unique assertions exercised its own reimplemented longjmp handler rather than mj_guard.cpp, and its ELF assertions were the weaker copy of ones the shipped module already gets -- --version-script has nothing to hide in a module linking no static archive, and the shipped one links cudart_static. Two replacements do cover shipped code: that install_mujoco_handlers() wrote the twin's own mju_user_error and not the wheel's, and that an unguarded mju_error reaches mj_guard.cpp's abort rather than MuJoCo's exit(1). The example's tests are keyed on TARGET robot_twin_py, which says the actual dependency instead of restating where the twin gets built. Signed-off-by: Jiwen Cai <jiwenc@nvidia.com>

jiwenc-nv · 2026-08-23T21:12:46Z

+# rather than a knob -- gl_context.cpp makes its OpenGL context through EGL's device
+# platform, which is how it renders with no display, and Windows has no equivalent yet.
+if(CMAKE_SYSTEM_NAME STREQUAL "Linux")
+    # MuJoCo itself. Included here rather than at the root so the decision and the


why don't we add it even deeper? such as inside robot_twin.

Also, why would robot_twin_tests needs mujoco direclty? I think it should just test the interfaces of robot_twin, not its mujoco internals.

jiwenc-nv · 2026-08-23T21:13:44Z

-!/examples/mujoco_xr/python/isaacteleop_examples/mujoco_xr/assets/leader/leader_gripper.xml
-/examples/mujoco_xr/python/isaacteleop_examples/mujoco_xr/assets/follower/*
-!/examples/mujoco_xr/python/isaacteleop_examples/mujoco_xr/assets/follower/follower_arm.xml
+/examples/robot_viz/python/isaacteleop_examples/robot_viz/assets/leader/*


where does those xml come from? mujoco? if so, can we just download them?

jiwenc-nv · 2026-08-23T21:15:17Z

+    endif()
+    # Pure Python, but its tests import the robot twin, which ships with Televiz on
+    # Linux (src/viz/CMakeLists.txt says why not on Windows).
+    if(BUILD_VIZ AND CMAKE_SYSTEM_NAME STREQUAL "Linux")


this check against CMAKE_SYSTEM_NAME STREQUAL "Linux" is YAGNI. Fix or prove me wrong.

jiwenc-nv · 2026-08-23T22:27:22Z

 endif()
+
+# ==============================================================================
+# MuJoCo (robot twin physics + renderer)


do we really use any physics from MuJoCo at all? The use case of MuJoCo here in this PR should be visualization only. Please double check whether we are doing physics at all. That's YAGNI, if so, please remove.

jiwenc-nv · 2026-08-23T22:27:52Z

+        mujoco
+        GIT_REPOSITORY https://github.com/google-deepmind/mujoco.git
+        GIT_TAG        ${ISAACTELEOP_MUJOCO_VERSION}
+        # The ref is a tag, so it is fetchable without history. Measured here: 140 MB of


too verbose. no need to comment GIT_SHALLOW remove.

jiwenc-nv · 2026-08-23T22:30:15Z

+        GIT_SHALLOW    TRUE
+    )
+
+    # Plain variables, not the `CACHE ... FORCE` the entries above use: MuJoCo sets


check: should those go into mujoco.cmake? or it is required to configure before FetchContent_MakeAvailable(mujoco)

jiwenc-nv · 2026-08-23T22:31:43Z

   ``pip install isaacteleop`` already ships the compiled ``isaacteleop.viz`` module. See
   `Other Build options`_ for the full option table.

+   On **Linux**, ``BUILD_VIZ=ON`` also builds the robot twin's scene backend


YAGNI. Please remove.

jiwenc-nv · 2026-08-23T22:32:04Z

   * - **Televiz visualization**
     - ``BUILD_VIZ``
-     - Auto: ``ON`` when Vulkan and the CUDA Toolkit are detected, else ``OFF``. Force with ``-DBUILD_VIZ=ON`` / ``-DBUILD_VIZ=OFF``. (Most users don't need this — ``pip install isaacteleop`` already ships the compiled ``isaacteleop.viz`` module.)
+     - Auto: ``ON`` when Vulkan and the CUDA Toolkit are detected, else ``OFF``. Force with ``-DBUILD_VIZ=ON`` / ``-DBUILD_VIZ=OFF``. On Linux it also builds the robot twin's scene backend, which needs ``libegl-dev`` and fetches MuJoCo on the first configure. (Most users don't need this — ``pip install isaacteleop`` already ships the compiled ``isaacteleop.viz`` module.)


YAGNI. Please remove.

github-code-quality Bot found potential problems Aug 21, 2026

View reviewed changes

jiwenc-nv force-pushed the jiwenc-nv/robot-viz branch 2 times, most recently from 7e638fd to f5de61f Compare August 21, 2026 23:51

github-code-quality Bot found potential problems Aug 21, 2026

View reviewed changes

Comment thread src/python/isaacteleop/viz/robot/__init__.py

"JointMap",

"RobotTwin",

"RobotTwinPublisher",

"SceneTwin",

jiwenc-nv mentioned this pull request Aug 23, 2026

Robot twin: tests #999

Draft

6 tasks

jiwenc-nv added 2 commits August 23, 2026 22:07

jiwenc-nv force-pushed the jiwenc-nv/robot-viz branch from 124b2f0 to bce10b7 Compare August 23, 2026 22:10

jiwenc-nv commented Aug 23, 2026

View reviewed changes

Navigation Menu

Uh oh!

Jiwenc nv/robot viz - #995

Jiwenc nv/robot viz#995
jiwenc-nv wants to merge 2 commits into
mainfrom
jiwenc-nv/robot-viz

jiwenc-nv commented Aug 21, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Aug 21, 2026 •

edited

Loading

Draft PR not reviewed

Uh oh!

Uh oh!

jiwenc-nv Aug 23, 2026

Uh oh!

jiwenc-nv Aug 23, 2026

Uh oh!

jiwenc-nv Aug 23, 2026

Uh oh!

jiwenc-nv Aug 23, 2026

Uh oh!

jiwenc-nv Aug 23, 2026

Uh oh!

jiwenc-nv Aug 23, 2026

Uh oh!

jiwenc-nv Aug 23, 2026

Uh oh!

jiwenc-nv Aug 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

jiwenc-nv commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Testing

Checklist

Uh oh!

coderabbitai Bot commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Draft PR not reviewed

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

jiwenc-nv commented Aug 21, 2026 •

edited

Loading

coderabbitai Bot commented Aug 21, 2026 •

edited

Loading