feat(teleop): render SIMPLE scenes natively in Unity instead of streaming video - #2
Open
Murilovisk0 wants to merge 31 commits into
Open
feat(teleop): render SIMPLE scenes natively in Unity instead of streaming video#2Murilovisk0 wants to merge 31 commits into
Murilovisk0 wants to merge 31 commits into
Conversation
Groundwork for rendering SIMPLE scenes natively in a headset instead of streaming a flat video feed of MuJoCo's own render. Isaac already acts as a pure renderer driven by MuJoCo state -- IsaacSimSimulator.sync_states() forces qpos and object poses in and zeroes velocities so it never simulates -- and this occupies the same slot with Unity. Three pieces: - coordinates: the MuJoCo (right-handed, Z-up) to Unity (left-handed, Y-up) mapping, in one place. Doing it here rather than in C# keeps a single implementation; two would drift. - scene_export: walks a compiled MjModel into scene.json plus one OBJ per visual mesh. Reads the compiled model, not the source MJCF, because compilation recenters mesh vertices on their center of mass and bakes the compensating offset into geom_pos/geom_quat -- applying the mesh transform on top of that is how a robot arrives in pieces. - protocol: a 20-byte header plus 28 bytes per body carrying world poses. Sending resolved poses rather than joint angles means Unity needs no kinematic model: no joint axes to map, no rest rotations to calibrate. Robot links, free objects and articulated parts share one path, and the renderer cannot drift out of sync with the physics. Geom selection is per body: visual geoms where a body has them, collision geoms where it has none. The fallback matters -- _build_object adds manipulable objects as convex collision meshes only, so a visual-only filter would silently drop every object in the scene. scripts/test_unity_export.py verifies the chain against a real G1 model. The load-bearing check reassembles each geom's world pose in Unity space from the streamed body pose and the exported local pose, and compares it with MuJoCo's own: max error 4.7e-08 m over 36 geoms. Needs only mujoco and numpy, so it runs without Isaac Sim. Known gap: objects rendered from collision hulls look faceted. Wiring in each asset's original visual mesh is the next refinement. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Transport for the state packets, reusing the WebRTC connection the Unity client already establishes. The channel is opened with ordered=False and maxRetransmits=0. Pose state is disposable -- every frame supersedes the last -- so a reliable ordered channel optimises for the wrong thing: one lost packet stalls the stream behind retransmissions and the operator sees the scene freeze, then jump. Unreliable delivery drops the straggler and lands the next frame on time. Unity is the offerer and opens its own reliable `tracker` channel for poses and controller state. The state channel runs the other way and is opened here by the answering peer, which WebRTC permits once the SCTP association exists. That keeps the reliability settings in one language instead of two, and they do survive the trip: the loopback test asserts the receiving peer sees ordered=false and maxRetransmits=0 rather than the silently-reset defaults. UnityStateChannel attaches to any existing RTCPeerConnection so it can be dropped into a signaling server that already exists; UnityStateServer is a self-contained one for the SIMPLE side. publish() is safe to call from the synchronous sim loop -- the send is marshalled onto the loop thread -- and returns False rather than raising when the peer is gone or the link is congested, since a dropped frame is a normal outcome here, not an error. scripts/test_unity_webrtc.py runs a real negotiation against a stand-in Unity client and streams 60 frames of G1 poses through it. Trickle ICE gets its own check: the loopback client ships candidates inside the SDP, so _parse_candidate would otherwise never run despite being on the live path for a real client. aiortc and websockets go in a `unity` extra rather than the base deps. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
UnityRenderBridge ties the export and the transport to a running simulation, and `simple teleop --unity` turns it on. The bridge is deliberately not an agent. Rendering in Unity is orthogonal to what drives the robot -- a VR operator, a keyboard, a scripted policy, a checkpoint under evaluation -- so it only reads state and never actuates. Folding it into PicoSonicAgent would have tied the view to one particular way of controlling the robot for no reason. Episode boundaries are the load-bearing part. SIMPLE resamples objects per episode and update_layout() compiles a fresh MjModel when it does, at which point Unity's loaded geometry is stale: streaming new poses into it would drive the wrong objects, body 7 being a mug one episode and a drill the next. The bridge detects this by object identity -- holding a reference to the model it exported, so a freed model cannot be mistaken for the current one by address reuse -- re-exports, and rolls the scene_id. A client that has not reloaded sees the mismatch in the packet header instead of animating nonsense. Re-export writes to disk, so the CLI calls resync() on the reset boundary rather than paying for it mid-episode. tick() throttles itself to --unity-publish-hz and is cheap to call every step whatever the sim rate. Also adds set_scene_id() to the transport. The bridge was reaching into the server's private attributes to roll the id, which happened to work and would have broken quietly the first time the transport changed shape. Two fixes found by writing the test: - The bridge was built before the first env.reset() in the CLI, which would have raised AttributeError: mjModel does not exist until update_layout() runs, and it is reset() that calls it. Construction now happens after, and _export() raises an explanatory error rather than letting the next caller rediscover this. - publish_hz=0 as "publish on every tick" was implicit in the arithmetic and undocumented. scripts/test_unity_bridge.py stubs the transport, since the WebRTC path has its own test, and drives a scene swap end to end: 31 bodies to 32, new scene_id, adopted by the server, reflected in scene.json on disk and in the packets that follow. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds --unity-ice-host, plus the candidate filtering the existing python_webrtc.py already found it needed. On a multi-homed host -- a physical NIC, a Tailscale tun, often a docker bridge -- aiortc advertises a host candidate for every interface, and the headset can spend its connection attempt on one that is unreachable from its side of the tailnet. Pinning the address to advertise removes the guesswork. Link-local candidates are dropped in both directions: they are per-interface and meaningless across a link, so they only add ICE pairs that are guaranteed to fail while the connection waits for them to time out. Also warns, once, when a state packet outgrows a 1280-byte MTU. That is Tailscale's tun MTU, and after IP, UDP, DTLS and SCTP headers it leaves about 1195 bytes -- 42 bodies. The G1 alone is 31, so a scene with a dozen manipulable objects crosses the line. It matters more here than it normally would: with no retransmits, SCTP delivers a fragmented message only if every fragment arrives, so splitting a packet in three roughly triples the drop rate the unreliable channel was chosen to keep low. Worth knowing before diagnosing it as jitter. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The flag was only on cli/teleop.py, but teleop_decoupled_wbc.py is the entry point actually in use, so --unity did nothing for the workflow it was built for. Same wiring as the other CLI: build the bridge after the first reset, since update_layout() compiles mjModel and reset() is what calls it; resync on both reset paths, including the auto-reset between recorded episodes; tick inside a telemetry timer so Unity publishing shows up alongside the other per-step costs when the loop overruns. Recording is untouched. The bridge only reads state, so it neither perturbs the sim nor changes what lands in the dataset -- the images that go to disk still come from Mujoco, and Unity is only what the operator sees. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
SIMPLE configures no logging, so the effective level is WARNING and every logger.info in this module went nowhere: signaling up, client connected, offer received, answer sent, channel opened. All of it invisible. That is survivable until a headset fails to connect, which is precisely when those lines are the only thing that says where the handshake stopped. Debugging it from the Unity side alone means guessing whether the peer ever reached the server at all. A handler is attached to the simple.teleop.unity logger only -- never the root, and never when the application has already configured one -- and can be turned off with verbose=False. Also logs each handshake step, so a stalled connection points at the stage it died in rather than just staying quiet. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The signaling handler took a single argument, which is right for websockets >= 14 but wrong for 12 and 13, where the server calls handler(websocket, path). This package allows >= 12, so on an older install every connection raised TypeError before the handshake and the client saw only close code 1011. That failure was close to undiagnosable: websockets logs the cause on its own logger, which nothing here configures, so the traceback reached neither side. A Unity client just never received the state channel, with the scene loading normally over HTTP the whole time -- pointing suspicion at the channel rather than at the handshake that never completed. Two changes, both about finding this class of failure rather than the signature itself: - a wrapper around the session logs any exception, including the ones thrown during setup, which the existing try/except did not cover. - the websockets version is logged at startup, since the behaviour that matters here depends on it and nothing else reports it. Found by connecting to a running server as a stand-in client. The loopback test could not have caught it: it runs against whatever websockets is installed, and on 16.x this code path is correct. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
aiortc sits in the optional 'unity' extra and was imported inside the connection handler, which deferred a missing install to the worst moment available: the server bound the port, logged itself ready, and then failed on the first client with close code 1011 and no cause on either side. From the operator's seat the scene loaded fine over HTTP and the state channel simply never appeared -- pointing suspicion at the channel, the headset, the network, at everything except a package that was never installed. Checking at construction reports it once, at startup, naming the install command. Lazy import still holds: nothing here is imported unless --unity is passed. Found by connecting to a live server and reading the traceback the previous commit made visible. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Unity rejected every answer with "Invalid SDP line", naming no line. The cause is in the round trip, not in any single line. aiortc ends its SDP with CRLF; WebRTCSignalingUnity splits on line breaks with StringSplitOptions.None, which turns that trailing CRLF into an empty final element, rejoins with CRLF, and appends one more of its own. The result carries a blank line, and libwebrtc refuses it. python_webrtc.py never hit this because clean_sdp_for_unity returns "\r\n".join(lines) with no trailing newline. That detail is easy to read as formatting; it is load-bearing, and porting the module without it is what broke the handshake. Same shape is reproduced here, along with the attribute filter. Current aiortc emits neither a=extmap-allow-mixed nor a=ice-options -- verified against a generated answer -- but the filter costs nothing and the working implementation had it. rewrite_host_candidates now returns how many it changed, so the log says "pinned 2 host candidate(s)" instead of claiming success when the SDP contained none to rewrite. The test reproduces Unity's split-and-rejoin and asserts both directions: that a raw SDP does grow a blank line, and that a cleaned one does not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The state channel opens, both ends agree on ordered=False and maxRetransmits=0, Python publishes -- and the Unity receiver counts nothing at all, not even a malformed packet. Two very different faults produce that same silence, and nothing on either side distinguishes them. So the channel now sends two frames on open: one of 20 bytes, one padded past a 1280-byte MTU. Both declare zero bodies. The small one decodes as a valid empty frame and lands in `applied`; the padded one fails the length check and lands in `droppedMalformed`. Neither moves geometry, and between them the receiver's counters say which fault it is: applied 1, malformed 1 both sizes arrive; size is not the problem applied 1, malformed 0 only the small one; fragmentation is applied 0, malformed 0 the channel delivers nothing Worth noting that an aiortc client on the same link does receive the full-size 1448-byte packets -- verified by connecting to the running server -- so whatever this is, it is specific to the peer, not to the wire. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous commit added two probe messages on channel open and left this test counting every packet as a pose frame, so it failed with "62 de 60 enviados" -- and I pushed it that way. The probes are now separated by size and asserted on rather than merely tolerated. They are the diagnostic that tells a size problem from a dead channel on a real link, which makes their own delivery worth a check: a probe that never arrives diagnoses nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Nothing above one MTU was reaching Unity. The channel opened, both ends agreed on the settings, Python published, and the receiver counted nothing -- not even a malformed packet. The size probe settled it: the 20-byte frame landed, the 1395-byte one did not. maxRetransmits=0 was the cause. That setting was chosen on the argument that a dropped pose frame costs nothing, since the next one supersedes it. The argument holds, but only while a message fits inside one MTU. Above that SCTP fragments, and an unreliable fragmented message survives only if every fragment does. Against a libwebrtc peer across a 1280-byte Tailscale link, none did. An aiortc client on the same link received the same 1448-byte packets without loss, so this is the peer's reassembly, not the wire. Unordered was the half worth keeping. Head-of-line blocking comes from ordering, not from retransmission: later frames still overtake a message being retransmitted, and the receiver drops the straggler as stale when it lands. So the property that motivated the original choice survives, and fragmented messages now arrive. max_retransmits=0 remains available, and is the better setting once a packet fits one MTU -- 42 bodies at the current 28 bytes each. Getting back under that line by chunking or quantising is the real fix for large scenes; this restores a working stream first. The fragmentation warning now depends on the mode: a hard warning when the channel cannot retransmit, and a note about latency when it can. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…vers Nothing arrived at all under the previous setting. Every receiver counter sat at zero -- not applied, not stale, not malformed, not even the new "ignored, no scene loaded". OnStateMessage was never called once. Three configurations, measured against the real client rather than reasoned about: ordered=False, maxRetransmits=0 only messages under one MTU arrive ordered=False, maxRetransmits=None nothing arrives ordered=True, maxRetransmits=None the default now An aiortc client on the same link received every packet under all three, including the 1448-byte frames, so the constraint is Unity's WebRTC and not the network. The tracker channel Unity opens for itself is ordered and reliable and has worked throughout; matching it is the configuration with evidence behind it. This gives up what the original design was built around. Ordering means a lost packet stalls the ones behind it until SCTP retransmits, which is the freeze-then-jump the unordered channel existed to avoid. Over a direct link that is rare, the receiver already discards stale frames, and a stream that stalls occasionally beats one that never arrives. --unity-unordered puts the old behaviour one flag away, for retesting when the peer's WebRTC is upgraded. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both channels open and neither delivers anything to its far side, but opening a channel is DCEP control traffic -- it proves the SCTP association carries control messages, and says nothing about data. on_tracker was never wired up, so Unity's own channel has been running untested this whole time while the state channel took the blame. Counting it splits the question: if tracker messages arrive, data does cross and something about the state channel specifically is wrong; if they do not, no data crosses in either direction and the state channel was never the thing at fault. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Data crosses Unity -> Python (500 tracker messages) and nothing crosses the other way, on the same association. That asymmetry points at stream assignment rather than at the link. RFC 8832 splits the id space by DTLS role: the client takes even ids, the server odd. aiortc answers with a=setup:active and so is the client here. If both peers end up on the same id, data sent from this side lands on a stream the far side treats as its own outgoing channel, and is dropped with nothing logged anywhere. The id is now logged on open, on both ends, which either shows the collision or rules it out. Also fixes the host-candidate matcher, which tested for " typ host " with a trailing space. aiortc ends those lines at "typ host", so the test never matched and --unity-ice-host reported "pinned 0" against an answer full of host candidates. The check now looks at the token after "typ". The test fixture gained a candidate in the unpadded form, which is the one that was going unnoticed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
TeleopPolicy consumes a BaseStreamer. PicoStreamer backs it with the XRoboToolkit SDK and VuerStreamer with a WebXR browser session, and both take the headset's XR session for themselves -- which leaves none for a Unity app, so the operator cannot be inside the 3D scene while driving the robot. That is the whole point of rendering the scene natively, and either existing streamer gives it back. This backs the same interface with the tracker channel the Unity client already sends. One app renders and steers. Split in two so most of it is testable here: UnityStreamerCore holds the arithmetic and knows nothing of decoupled_wbc, and make_unity_streamer wraps it in the BaseStreamer the policy wants, importing that lazily. The constants are copied from PicoStreamer and VuerStreamer rather than chosen -- dead zone, velocity limits, height range, the 50 Hz assumption -- so Unity does not feel different from the other two for no reason. Degenerate poses hold the previous value instead of replacing it. A controller that sleeps or leaves the tracking volume reports zeros, and passing those through sends the arm lunging at the origin. Unity gains grip on the wire. The finger encoding needs trigger and squeeze together to pick which finger closes, and TrackerSender was sending only the trigger, so two of the three grasps were unreachable. Not verified: whether headset_relative_wrist produces the basis WristsPreProcessor expects. TeleVuer's own conversion lives in a package that is not vendored here, so this reproduces its description rather than its code, and a wrong basis still passes every test -- it shows up as arms that track smoothly and point somewhere else. WristsPreProcessor also mirrors the right wrist for any device outside its allowlist, so "unity" has to be added there next to "pico" and "vuer". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Operators reported the wrists feeling wrong after the move from TeleVuer to Unity, and described it as the wrist sitting mid-controller in one and at the base in the other. That is a real difference: OpenXR runtimes disagree about where on the controller body the reported pose sits, and the gap is a few centimetres. It is a pivot error, not an offset error, which is why it resisted tuning. The existing correction in xr_teleoperate shifts the target in the robot's world frame -- a constant vector regardless of hand orientation. A grip-origin gap is fixed in the *controller's* frame and rotates with the hand, so a world-frame constant cannot cancel it: turn the wrist and the residual swings around. The robot's hand orbits instead of rotating on the spot. apply_grip_offset shifts along the controller's own axes, before the frame change. The test measures both halves of the claim: with the offset at the pivot, spinning the controller in place translates the target by 0; without it, the same motion drags it around a 10 cm circle. Default is zero, so nothing changes silently -- a constant baked in here would be indistinguishable from a frame bug. To calibrate: hold the controller still and rotate it in place; whichever way the robot's wrist drifts is the axis to correct. Rotation is left alone, which matches what xr_teleoperate found empirically: roll on the controller already gives clean roll on the robot. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The pieces that are actually Unity-specific in wiring it up. attach_unity_streamer uses the hook PicoDecoupledAgent and VuerDecoupledAgent both rely on: TeleopStreamer leaves body_streamer as None for a device name it does not recognise, so the policy is built with "unity" and the streamer assigned over the top. That name also avoids DummyStreamer, which drags in ROS 2. UnityButtonPoller keeps dropping the elastic band and resetting the environment out of the streamer. Those are simulator concerns -- the teleop policy has no business knowing an elastic band exists -- and both agents keep them at agent level for the same reason. Read-and-clear, so one press cannot reset twice, and edge-triggered, since at 50 Hz a level-triggered reset fires fifty times while the hands are still closing. Not written: a full UnityDecoupledAgent. Most of VuerDecoupledAgent is WBC assembly with nothing Unity-specific in it -- observation building, the stabilise path, the policy step -- and copying those 250 lines here would duplicate an unmerged branch with no way to run any of it on this machine. Once metaquest_teleoperation lands, the agent is a subclass that swaps the streamer, the button bindings, and drops the stereo push entirely, since Unity renders the scene rather than receiving a video of it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Brings in the WebXR teleoperation work so the Unity input path has something to attach to. The merge is one-way: this branch takes their commits, theirs is untouched. What arrives that this branch needs: - decoupled_wbc now points at the AKCIT-RL fork on integration/metaquest, which is where TeleopPolicy and BaseStreamer live. make_unity_streamer imports BaseStreamer from there, so nothing in the Unity input path runs without it. - VuerDecoupledAgent, the working example of injecting a streamer into the WBC pipeline, and the base a UnityDecoupledAgent can subclass instead of duplicating 250 lines of observation building and policy stepping. - The televuer submodule and the pyproject wiring that goes with it. Both overlapping files merged cleanly and were checked rather than trusted: the CLI kept their PicoDecoupledAgent -> VuerDecoupledAgent swap alongside the --unity flags and the bridge, and pyproject kept the unity extra next to their sonic additions. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Closes the loop. --unity streams the scene to the Unity client; --unity-teleop makes that same client the source of the operator's poses, so one application renders and steers over one connection. Subclasses VuerDecoupledAgent rather than repeating it. Most of that agent is WBC assembly with nothing headset-specific in it -- observation building, the stabilise path, the policy step, the band descent -- and three things actually differ: - Input comes from the tracker channel through a UnityTrackerSource instead of a WebXR browser session. - Drop and reset are re-read from Unity's controller payload. - There is no rendering to do. VuerDecoupledAgent resizes a stereo pair and pushes it every step; the Unity client draws the scene from geometry, which is the entire point of this path. That drops the per-step cv2.resize, the video buffer, and the coupling between the operator's viewpoint and the simulator's frame rate. The device name stays "vuer" deliberately. WristsPreProcessor mirrors the right wrist for any device outside its allowlist, and the name selects a wrist convention rather than hardware: Unity reports OpenXR grip poses, the same convention TeleVuer delivers. Renaming it without extending that allowlist would reflect the right arm alone, which reads as broken IK rather than a wrong frame. --unity-teleop without --unity is refused at startup. The tracker channel is opened by the bridge, so the combination would leave the agent holding its rest pose forever with nothing on screen to explain why. Untested here: decoupled_wbc is a submodule pointer on this machine, so none of this has been executed. The four suites that do not need it still pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The drop binding fell back to holding both A buttons, on a comment claiming the thumbstick click was not in TrackerSender's payload. It is, and has been: leftStickClick and rightStickClick have shipped in PosePacket since before any of this. The parser simply never read them. The fallback was worse than a missing feature. Right A raises the base, so every height adjustment was one left-A away from dropping the robot, and neither binding matched what the guide and VuerDecoupledAgent document. Drop is now the right thumbstick click, the same binding VuerDecoupledAgent uses, and the test asserts right A alone does not trigger it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
_on_episode_reset returned immediately without --record, which tied two unrelated things together. Recording decides whether frames reach the disk. Everything in that function is about making the robot usable: releasing the elastic band, resetting the WBC pipeline, and engaging the lower-body RL policy. Without it the robot is half configured. No balance policy, so it collapses the moment the viewer opens, and the band then recovers it into a hang. Teleoperation appears dead whatever the operator presses, because get_action returns the elastic-band command and never reaches the teleop one -- the buttons work, the result simply has nowhere to go. Found while an operator pressed the activation button and nothing happened. Recording itself is untouched: the exporter is still built only under --record, and every write is still behind `if exporter is not None`. Only the path that was already broken changes. The teleop policy still starts deactivated on purpose, so the arms cannot snap to wherever the controllers happen to be at connection time. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Pushing the stick forward drove the robot backwards. The negation was copied from VuerStreamer, whose own comment gives the reason it needs one: WebXR's gamepad spec reports y positive when the stick is pulled towards the operator. Unity's primary2DAxis is positive away from them, so the same negation reverses an axis that was already correct. The three axis signs are now named constants with that difference written down, rather than three unexplained minus signs to re-derive. Strafe and yaw agree between the runtimes and keep theirs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Unity's wrists arrived head-relative and stopped there. The IK solves about an origin near the waist, so every target sat 45 cm low and 15 cm behind -- down by the robot's thighs while the operator's hands were at chest height. TeleVuerWrapper applies this pair of offsets, and the xr_teleoperate bridge that drives a real G1 reproduces it. They describe the G1's build rather than any headset, which is why both paths carry the same numbers; only the Unity path had dropped them. The failure does not look like a missing translation. An unreachable target makes the IK return the least-bad pose it can find, so the arms track the hands loosely and point somewhere else -- which reads as an inverted axis. The existing frame tests could not catch this: they check that the basis change is orthogonal and that the head subtraction is exact, and both stay true with the offset missing. Added a reachability check instead -- an operator standing with hands at chest height must produce a target the arm can reach -- which fails at z = -0.40 m without the offset. Also corrected the note in UnityDecoupledAgent claiming "vuer" is in a WristsPreProcessor allowlist. VuerDecoupledAgent picks that string precisely because TeleopStreamer does not recognise it, which is the opposite. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both hands raised lifted one arm and dropped the other. That asymmetry cannot come from this side -- wrist_poses() runs the same closure over both hands -- so it had to come from the one component that treats them differently, and reading it changed the picture entirely. WristsPreProcessor is differential, not absolute. It stores the wrist pose at calibration and thereafter applies inv(wrist_at_calib) @ wrist_now to the robot's hand in that hand's own frame. A hand translation therefore reaches the robot as R_hand_frame @ R_wrist.T @ delta: routed through the operator's wrist orientation, and correct only if that orientation matches the robot's hand frame. Its calibrate() asserts that match for exactly two names, "pico" and "vuer", and applies a per-arm correction for every other -- hand_rotation_correction on the left, the same composed with a half turn about Z on the right. Unity reports XRNode device poses rather than the WebXR grip poses TeleVuer delivers, so the assertion does not hold and the correction is the one we want. The G1's two hand frames are mirrored, which is why a single wrong convention shows up as opposite errors on the two arms. Threads the name through TELEOP_DEVICE so the Vuer agent keeps "vuer" and this one selects "unity". Both are still unrecognised by TeleopStreamer, so body_streamer stays None and DummyStreamer still does not pull in ROS 2; FingersPreProcessor.calibrate ignores the name outright. --unity-wrist-correction/--no-unity-wrist-correction picks the branch at runtime. Settling this needs a headset, and an edit-rebuild cycle per comparison is a poor trade for one argument. Also corrects 867334c, which claimed the missing waist offset left the target unreachable and explained the posture. It cannot: a constant translation appears on both sides of inv(wrist_at_calib) @ wrist_now and cancels exactly, so that commit was a no-op. The offset is kept for parity with TeleVuerWrapper and the xr_teleoperate bridge, and both it and its test now say so instead of claiming to fix anything. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Left arm tracking and right arm fully inverted: raise it and the robot lowers. The stock per-device correction did not fix it, and on the evidence no constant could have. WristsPreProcessor maps E = E0 @ inv(A) @ (inv(W0) @ W) @ A, so a hand translation d reaches the robot as R_e @ A.T @ R_w.T @ d, routed through the operator's wrist orientation and applied in the robot's hand frame. A is the rotation reconciling the two, and the preprocessor picks it from the device name: identity for "pico" and "vuer", a hardcoded pair for anything else. Both are assertions about what a particular headset reports. Unity sends XRNode device poses, not WebXR grip poses, so neither holds. A does not need guessing. Setting R_e @ A.T @ R_w.T = I gives A = R_w.T @ R_e, and both rotations are already stored by calibrate() -- so this reads them back and overwrites the guess. Rotation falls out of the same substitution: a hand rotation dR arrives as dR @ R_e, turning the robot's hand by dR in the world frame. This is why the failure was asymmetric. R_e differs between the arms -- the G1's hand frames are mirror images -- so the solved A differs too, and each arm gets what it needs. One constant cannot serve both, which is what one arm tracking while the other inverted was telling us. Wrapping calibrate() rather than solving once at startup, because A depends on where the operator's hands were when they pressed activate. The test reproduces the preprocessor's arithmetic against a stand-in with mirrored hand frames and drives the whole path -- calibrate, move, read where the robot's hand went. It asserts both stock branches break an arm before asserting the solved one fixes both, so it fails if the bug is reintroduced and would have failed on the code that shipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Answering the question directly: no offset was being applied. UnityTrackerSource defaulted to (0,0,0) and the CLI never passed anything else, so apply_grip_offset returned early on every frame. The machinery was written and then left unused, deliberately -- a silent constant would have been indistinguishable from the frame bug we were still chasing at the time. That bug is fixed now, so the knob can be turned. --unity-grip-offset "x,y,z" in metres, in the controller's own axes. Parsed with the other argument checks so a typo fails before the environment is built rather than once the headset is on. Rejects anything over 0.5 m, since the plausible mistake is typing centimetres. The test pins down why this knob and not the other one. Rotating the controller about its own origin, the hand moves 2.8 cm with a 6 cm controller-local offset and exactly 0.0 with a world-frame constant -- the same cancellation that made WAIST_FROM_HEAD a no-op. A grip offset right-multiplies the pose, so it conjugates the delta instead of appearing on both sides of it. It also pins the calibration signal: with no offset, turning the controller in place moves the robot's hand by exactly zero. Any drift the operator sees is therefore the pivot error itself, which is what makes it measurable by hand. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Adds an optional Unity frontend that renders a SIMPLE scene as real 3D geometry in the headset, instead of streaming a flat video feed of MuJoCo's render.
Enabled with
simple teleop --unity. Off by default, and nothing on the existing paths changes when it is off.Why
The current VR teleop path sends a side-by-side stereo video of MuJoCo's own render to the headset. That couples the operator's viewpoint to the full round trip through the simulator and an H.264 encoder — easily 100–200 ms. The vestibular system notices, and that is the classic recipe for VR sickness.
Streaming geometry instead decouples the two. Head movement re-renders locally at headset refresh (~11 ms). The simulator's latency does not disappear, but it now affects only the contents of the scene, not the point of view.
Bandwidth is a secondary benefit: 888 bytes/frame for a 31-body G1, about 53 KB/s at 60 Hz, against roughly 500 KB/s for the stereo video path.
Approach
SIMPLE already had the right shape for this.
IsaacSimSimulator.sync_states()pulls qpos and object poses out of MuJoCo, forces them into Isaac and zeroes the velocities so Isaac never simulates. Isaac is not a simulator there — it is a slave renderer. Unity takes the same slot.teleop/unity/coordinates.pyteleop/unity/scene_export.pyMjModel→scene.json+ one OBJ per visual meshteleop/unity/protocol.pyteleop/unity/webrtc_state.pyteleop/unity/bridge.pyThe Unity client lives in a separate repo.
Decisions worth reviewing
World poses, not joint angles. MuJoCo has already run forward kinematics, so sending resolved poses means Unity needs no kinematic model: no joint axes to map, no rest rotations to calibrate, no sign conventions to guess. Robot links, free objects and articulated parts share one path, and the renderer cannot drift out of sync with the physics. Costs 28 bytes/body instead of 4 bytes/joint.
The compiled model, not the source MJCF. Compilation recenters mesh vertices on their center of mass and bakes the compensating offset into
geom_pos/geom_quat. On the G1,mesh_pos[0]and the pelvisgeom_posare the same[0, 0, -0.076]for exactly this reason, andmesh_quatdeparts from identity by over a radian on some links. Reading the compiled model keeps those consistent; re-parsing the XML would mean reproducing MuJoCo's compiler by hand.Unreliable channel (
ordered=False,maxRetransmits=0). Pose state is disposable — every frame supersedes the last. A reliable ordered channel optimises for the wrong thing: one lost packet stalls the stream behind retransmissions and the scene freezes, then jumps.The bridge is not an agent. Rendering is orthogonal to what drives the robot — VR operator, keyboard, scripted policy, checkpoint under evaluation. It only reads state and never actuates.
Conversion lives in Python only. One implementation. Two would drift, and the failure mode is a subtly mirrored robot.
Two traps worth knowing about
Objects would have silently vanished. The first version selected visual geoms by
contype == 0. That is right for the G1, which separates visual and collision geometry, butMujocoSimulator._build_objectadds manipulable objects as convex collision meshes and nothing else — so the filter would have dropped every object in every scene. Selection is now per body: visual geoms where a body has them, collision geoms where it has none, taggedfrom_collision. Those render as the solver's convex decomposition — correctly placed, visibly faceted. Swapping in each asset's original visual mesh is a follow-up.Episode boundaries. SIMPLE resamples objects per episode and
update_layout()compiles a freshMjModel. Unity's loaded geometry is then stale, and streaming new poses into it would drive the wrong objects — body 7 being a mug one episode and a drill the next. The bridge detects this by object identity (holding a reference to the exported model, so a freed one cannot be mistaken for the current by address reuse), re-exports, and rolls thescene_idcarried in every packet header. A client that has not reloaded sees the mismatch instead of animating nonsense.Verification
Three suites, needing only
mujocoandnumpy(plus theunityextra for the WebRTC one) — they run without Isaac Sim:mju_quat2Matordered=false,maxRetransmits=0scene_id, adopted and reflected in packetsNot verified
None of this has run against SIMPLE itself. It was developed on Windows, where
isaacsim,gear_sonicandunitree_sdk2pywill not install, so everything was validated against a standalone Unitree G1 MJCF.cli/teleop.pyhas only been syntax-checked.Two things the first real run will settle:
MjSpecthat_setup_sceneassembles — built programmatically withattach()of sub-specs — exports as cleanly as a standalonescene.xml. Prefixed body names are the likely surprise.Notes for a reviewer
aiortcandwebsocketsare in a newunityextra, not the base dependencies.cli/teleop.py(import ordering, unused unpacked variables). Left alone rather than inflating this diff with unrelated changes.--unity-ice-host <tailscale-ip>: on a multi-homed host aiortc advertises a candidate per interface and the headset can pick one it cannot reach. This mirrors the--ice-hostflagpython_webrtc.pyalready needed.