examples.md

Examples

Run GUI examples from make shell-gui; run headless data/training jobs from make shell. Root numbered examples teach the shared API; IOAI product-specific collection and evaluation live under examples/policy_baseline/.

Example	Purpose
examples/01_collect.py	Collect one task with a motion-planner agent.
examples/02_mimic.py	Expand a dataset with IsaacLab Mimic.
examples/03_train.py	Train a selected registered policy backend.
examples/04_eval.py	Evaluate a checkpoint through PolicyAgent.
examples/05_custom_agent.py	Implement a custom BaseAgent.
examples/06_collect_component_task.py	Collect PickToShelf/SortToShelf component task data.
examples/07_compound_task.py	Run a coherent full task with TaskFlowAgent.
examples/policy_baseline/01_collect_ioai_pick.py	Collect one IOAI Pick product into its own dataset.
examples/policy_baseline/02_collect_ioai_place.py	Collect standalone Place data from a saved Nav→Place Scenario.
examples/policy_baseline/03_collect_ioai_sim_scene.py	Run the complete IOAI pipeline and save its Place-start Scenario.
examples/generate_ioai_table_layout.py	Select and save one of the 12 supported IOAI table layouts.
examples/policy_baseline/04_eval_ioai_policy_pipeline.py	Evaluate the YAML-routed multi-product policy pipeline.
examples/vision_baseline/	Traditional YOLO-seg + FoundationPose baseline (01-07). See YOLO Segmentation Workflow and FoundationPose Pick Baseline.

Basic Pipeline

python examples/01_collect.py
python examples/02_mimic.py --task GalbotG1-PickCube-v0
python examples/03_train.py --task GalbotG1-PickCube-v0 --export-inference
python examples/04_eval.py --task GalbotG1-PickCube-v0 \
  --checkpoint <run>/inference/model_inference.pth --headless

--export-inference keeps the complete training checkpoint and writes a separate FP32 bundle containing only the deployed policy weights, the robomimic configuration, and action-normalization metadata. Use --inference-output-dir to choose the bundle directory and --overwrite-inference to replace an existing complete inference bundle.

Select the training architecture explicitly with --backend:

# Feed-forward behavior cloning.
python examples/03_train.py \
  --backend robomimic_bc \
  --task GalbotG1-PickCube-v0 \
  --dataset-path data/pick_cube_demos_mimic.hdf5 \
  --output-dir outputs/pick_cube_bc \
  --epochs 50 --export-inference

# Recurrent behavior cloning.
python examples/03_train.py \
  --backend robomimic_bc_rnn \
  --sequence-length 10 \
  --task GalbotG1-PickCube-v0 \
  --dataset-path data/pick_cube_demos_mimic.hdf5 \
  --output-dir outputs/pick_cube_bc_rnn \
  --epochs 50 --export-inference

# Transformer behavior cloning.
python examples/03_train.py \
  --backend robomimic_bc_transformer \
  --sequence-length 10 \
  --task GalbotG1-PickCube-v0 \
  --dataset-path data/pick_cube_demos_mimic.hdf5 \
  --output-dir outputs/pick_cube_bc_transformer \
  --epochs 50 --export-inference

robomimic_diffusion remains the default. --sequence-length is accepted only for RNN and Transformer BC; invalid model/argument combinations fail before training. Evaluate with the same backend used to create the checkpoint, for example examples/04_eval.py --backend robomimic_bc_rnn .... All commands run inside make shell, which preserves the repository's pinned Isaac Sim Python, PyTorch, CUDA, and robomimic versions.

The public register_policy_backend(name, factory) seam supports project-local non-robomimic adapters implementing the Policy contract. It is explicit and process-local: no package scanning or fallback model selection occurs.

Overhead Review Video

Review recording is a process-wide environment capability, not an example CLI. Set IOAILAB_REVIEW_VIDEO_PATH when running any script that uses make_env(...) or foundation_pose_ioai_sim_scene.make_perception_env(...) (the FoundationPose vision baselines: 05, 06, 07), since the latter goes through the same ioailab.envs._factory seam:

IOAILAB_REVIEW_VIDEO_PATH=artifacts/gum_pick_review.mp4 \
python examples/04_eval.py \
  --task GalbotG1-IOAISimScene-Pick-v0 \
  --checkpoint <run>/inference/model_inference.pth \
  --product gum

IOAILAB_REVIEW_VIDEO_PATH=artifacts/ioai_pipeline_review.mp4 \
python examples/policy_baseline/04_eval_ioai_policy_pipeline.py \
  --table-layout outputs/ioai_table_layout.yaml \
  --policy-manifest outputs/ioai_policy_baseline.yaml

IOAILAB_REVIEW_VIDEO_PATH=artifacts/vision_pick_all_review.mp4 \
python examples/vision_baseline/07_fp_eval_pick_all.py \
  --yolo-model playground/Checkpoints/g1_ioaisimscene_pick_v0_front_head_rgb_camera/weights/best.pt \
  --headless

The environment switch attaches a floating camera to the moving G1 base, above and slightly behind its center, looking forward and down over the robot and its workspace. Its forward-shifted mount keeps the head out of the central workspace view and follows base navigation. It records one frame per control step and writes H.264 MP4 through the imageio/FFmpeg versions already pinned in the development image. Parallel environment rows are tiled into one frame, then rotated clockwise into a 540x960 portrait video so the robot-to-shelf workspace axis is vertical.

The camera is absent unless the environment variable is set. It is a review-only scene sensor: it is not added to policy observations, checkpoints, or HDF5 recorder terms. A non-MP4 path or explicitly disabling cameras while requesting video is an error; there is no alternate codec or observation fallback.

01_collect.py shows the motion-planner path by default. It also contains commented blocks for TeleopAgent and final-scenario export. For GP001 teleop, use GalbotG1-PickCube-Teleop-v0 with TeleopAgent.from_device("gp001", task=task_id); rejected demos can be removed with dataset.drop() after an env.collect(...) candidate during done plus keep/drop/exit review.

Motion-planning collection is expert data generation. Expert tasks may have empty reward and curriculum managers because the planner, action stepping, and task termination define the episode boundary. Do not add dummy reward or curriculum terms only to produce manager summaries.

IOAI Product Pick Data

Use examples/policy_baseline/01_collect_ioai_pick.py to collect one product-specific Pick dataset. The product is required and remains fixed across every parallel environment row and reset. The default output is data/ioai_sim_scene_pick/<product>.hdf5.

python examples/policy_baseline/01_collect_ioai_pick.py \
  --product beefnoodle \
  --episodes 100 \
  --num-envs 4 \
  --max-steps 1000 \
  --headless

Add --no-rgb to omit obs/front_head_rgb and collect a low-dimensional dataset. Keep RGB and no-RGB demonstrations in separate files because an existing HDF5 path is appended rather than replaced:

python examples/policy_baseline/01_collect_ioai_pick.py \
  --product water \
  --dataset-path data/ioai_sim_scene_pick/water_no_rgb.hdf5 \
  --no-rgb \
  --episodes 100 \
  --num-envs 4 \
  --max-steps 1000 \
  --headless

03_train.py detects the observation fields in the HDF5 file and trains the no-RGB dataset from robot_joint_pos automatically.

To save the successful Pick terminal state for Place calibration, run one episode in one environment and provide a YAML output path:

python examples/policy_baseline/01_collect_ioai_pick.py --product coffee --episodes 1 \
  --num-envs 1 --save-end-scenario .tmp/ioai_pick_calibration/coffee.yaml

Scenario export rejects parallel rows or multiple episodes so the saved state has one unambiguous successful terminal grasp.

The exact product IDs are water, cocacola, beefnoodle, coffee, pringles, gum, cocoa, orangejuice, pepsichips, and applejuice. The tray follows the ordered five-slot layout A1, A2, B1, cocoa, and B2. A1 fixes gum, A2 fixes cocacola, B1 samples coffee, beefnoodle, pringles, or water, cocoa uses its own fixed center slot, and B2 samples pepsichips, orangejuice, or applejuice.

Product-specific collection reproduces the stage order A1, A2, B1, cocoa, then B2 by emptying every earlier slot. The cocoa slot remains alongside B2 during cocoa collection and is empty for B2 collection. Cocoa uses a compact +/-5 mm X and +/-2 mm Y randomization window; A2 and B2 are shifted outward so every sampled A2-cocoa-B2 arrangement retains at least 4 mm pairwise clearance. Invalid product IDs and conflicting occupancy options fail without selecting another product or arm. Cocoa stays inside the Pick task: after the right arm picks it, the same episode transfers it to the right-side black tray_2, releases it, and returns the arm to carry. It never enters the IOAI Place task.

The final B2 Pick sees the already delivered Cocoa in the black tray, so its RGB training and evaluation must use the explicit post_cocoa context instead of the standalone empty-tray context:

python examples/policy_baseline/01_collect_ioai_pick.py \
  --product pepsichips \
  --pipeline-context post_cocoa \
  --dataset-path data/ioai_sim_scene_pick/pepsichips_post_cocoa.hdf5 \
  --episodes 128 --num-envs 16 --max-steps 1000 --headless

python examples/03_train.py \
  --task GalbotG1-IOAISimScene-Pick-v0 \
  --dataset-path data/ioai_sim_scene_pick/pepsichips_post_cocoa.hdf5 \
  --output-dir outputs/ioai_sim_scene_pick/pepsichips_post_cocoa \
  --epochs 50 --num-data-workers 8 --export-inference

python examples/04_eval.py \
  --task GalbotG1-IOAISimScene-Pick-v0 \
  --checkpoint outputs/ioai_sim_scene_pick/pepsichips_post_cocoa/<run>/inference/model_inference.pth \
  --product pepsichips --pipeline-context post_cocoa \
  --episodes 10 --num-envs 1 --max-steps 1000 --headless

post_cocoa is intentionally valid only for pepsichips; it changes scene occupancy, not the policy action or observation schema.

IOAI Pick→Nav→Place Pipeline And Place Scenario

Use examples/policy_baseline/03_collect_ioai_sim_scene.py to run the one-shot coherent task. The required product selects one of the nine non-cocoa products. The environment resets once, Pick randomizes and grasps the product, Nav drives to the Place pose and adjusts the legs, and Place continues from that physical state. At the exact Nav→Place transition, the example saves a Scenario for standalone Place data generation.

python examples/policy_baseline/03_collect_ioai_sim_scene.py \
  --product coffee \
  --save-place-scenario data/ioai_sim_scene_place/scenarios/coffee.yaml \
  --episodes 1 \
  --num-envs 1 \
  --max-steps 3000

To capture 16 real Place starts reached by a trained Pick policy instead of the default cuRobo Pick planner, use collection output with one environment:

python examples/policy_baseline/03_collect_ioai_sim_scene.py \
  --product cocacola \
  --pick-checkpoint outputs/ioai_sim_scene_pick/cocacola/<run>/inference/model_inference.pth \
  --save-place-scenario-dir data/ioai_sim_scene_place/scenarios/cocacola \
  --episodes 16 \
  --num-envs 1 \
  --max-steps 3000

By default, this runs the complete pipeline without enabling IsaacLab's HDF5 Recorder. Add --dataset-path data/ioai_sim_scene/coffee.hdf5 only when the full-pipeline action/observation trajectory is also needed. That HDF5 is not the Place initialization state. --save-place-scenario stores the robot base, all articulation joints, product poses, and gripper state at Nav→Place; if omitted, it defaults to data/ioai_sim_scene_place/scenarios/<product>.yaml. --save-place-scenario-dir instead stores every Nav→Place transition as 000.yaml, 001.yaml, and so on. In this mode, --episodes is the required Scenario count rather than the attempt count. Failed attempts continue until that count is reached; --max-scenario-attempts defaults to ten times the target and bounds collection explicitly. Collection output refuses to overwrite a directory that already contains Scenario YAML files, and verifies that the requested number of episodes was captured. The Scenario capture implementation lives in the IOAI task package; the example only selects the output mode.

Use that Scenario directory to generate no-RGB standalone Place demonstrations without any product/gripper pose reconstruction:

python examples/policy_baseline/02_collect_ioai_place.py \
  --product cocacola \
  --init-scenario data/ioai_sim_scene_place/scenarios/cocacola \
  --dataset-path data/ioai_sim_scene_place/cocacola_no_rgb.hdf5 \
  --episodes 128 \
  --num-envs 1 \
  --max-steps 1000 \
  --no-rgb

Every standalone reset independently samples one Scenario from the file or directory, then perturbs only the selected product in the active gripper local frame: x/y each use +/-2 mm, while z uses -3 mm to +1 mm to reduce upward placement for tapered products such as gum and coffee. It preserves product orientation and all gripper joints, clears the selected product velocity, and leaves every non-selected product unchanged. --no-rgb omits obs/front_head_rgb; the resulting HDF5 keeps actions, robot_joint_pos, and simulator state, and 03_train.py automatically configures robomimic Diffusion Policy with only low-dimensional observations. The option does not disable camera rendering in the live task.

The flow uses TaskFlowAgent.from_env(env) with the existing cuRobo Pick and Place planners plus the task-local Nav sequence by default. --pick-checkpoint overrides only the Pick phase with PolicyAgent; Nav and Place keep their task defaults. Neither path resets, teleports, nor reconstructs the product between phases. cocoa remains entirely inside its independent Pick black-tray transfer and is rejected by both full-flow Place and standalone Place.

All IOAI Pick, Nav, Place, and coherent scenes use assets/hdris/3065ba15-86e3-49f1-92e9-32c79cf2a77e.exr as a visible lat-long DomeLight background. The repository-local path resolves inside Docker without host-specific absolute paths; the external asset bundle must include this file.

Standalone Nav is registered as GalbotG1-IOAISimScene-Nav-v0. It requires both task_options={"init_scenario": ..., "pick_product": ...} and completes only after base position, base yaw, and the runtime shelf-layer leg posture are within tolerance.

Evaluate standalone Pick products with examples/04_eval.py --product <id>. The command prints each product's successful episode count and success rate before shutting down Isaac Sim, so the final summary remains visible in the terminal. Standalone Place evaluation restores the matching Nav-to-Place Scenario. The default is data/ioai_sim_scene_place/scenarios/<product>.yaml:

python examples/04_eval.py \
  --task GalbotG1-IOAISimScene-Place-v0 \
  --checkpoint outputs/ioai_sim_scene_place/water/<run>/inference/model_inference.pth \
  --product water \
  --episodes 10 \
  --num-envs 1 \
  --max-steps 1000

Use --init-scenario path/to/water.yaml to override the default. With --product all, the explicit path must be a template containing {product}, for example --init-scenario 'scenarios/{product}.yaml'. Coherent planner validation uses examples/policy_baseline/03_collect_ioai_sim_scene.py; continuous multi-product policy evaluation uses the layout generator and evaluator below.

IOAI Multi-product Policy Baseline

First choose one of the 12 supported B1/B2 combinations before Isaac Sim starts:

python examples/generate_ioai_table_layout.py \
  --output outputs/ioai_table_layout.yaml \
  --seed 0

The resulting YAML fixes the product identity in A1/A2/B1/cocoa/B2 for every episode in the evaluation run. Reset still applies the task's normal small within-slot position randomization; only B1/B2 identity sampling is replaced.

examples/policy_baseline/04_eval_ioai_policy_pipeline.py then evaluates the fixed order A1 gum -> A2 cocacola -> B1 -> cocoa -> B2 in one physical episode. Non-cocoa products use TaskFlowAgent.from_env(...): the YAML-selected Pick and Place policies override those two phase_agents, while Nav remains the task default. Cocoa runs only the task's Pick phase to the right black tray. An outer sequence returns the robot to the table after every product.

Policy routing is explicit; the example never scans for a latest run. Paths are resolved relative to the manifest file:

schema: ioailabIoaiPolicyManifest-v0
products:
  gum:
    pick_checkpoint: ioai_sim_scene_pick/gum/<run>/inference/model_inference.pth
    place_checkpoint: ioai_sim_scene_place/gum/<run>/inference/model_inference.pth
    place_start_scenario: ../data/ioai_sim_scene_place/scenarios/gum.yaml
  cocoa:
    pick_checkpoint: ioai_sim_scene_pick/cocoa/<run>/inference/model_inference.pth

Every non-cocoa bundle in the selected work-order prefix must contain both checkpoints and either one Nav-to-Place Scenario YAML or a directory of captured Scenarios. Cocoa accepts only pick_checkpoint. Before the environment is created, preflight verifies every Scenario in the source, the exact products named by the layout YAML, checkpoint sidecars, task IDs, bilateral 16-D action shape, Scenario metadata, and unique checkpoint bindings.

python examples/policy_baseline/04_eval_ioai_policy_pipeline.py \
  --table-layout outputs/ioai_table_layout.yaml \
  --policy-manifest outputs/ioai_policy_baseline.yaml \
  --episodes 12 \
  --max-products 5 \
  --num-envs 1 \
  --max-steps 15000 \
  --policy-cache-size 2 \
  --seed 0

--max-products 1..5 selects only the fixed work-order prefix. Version 1 requires one environment so one layout maps to one synchronous policy route. Policies load on demand; --policy-cache-size 2 retains at most two backend models, resets inference history at every activation, and releases the least-recently-used model on eviction.

Each product attempt has independent Pick/Nav/Place step budgets (default 1200 each) and a 20-step drop confirmation window. A timed-out phase or sustained product drop records a product-local failure, runs the normal Return sequence, and continues with the next item instead of waiting for the episode-wide --max-steps. Override these bounds with --pick-max-steps, --nav-max-steps, --place-max-steps, --drop-confirm-steps, and --delivery-stable-steps. Cocoa hands off to the normal Return sequence only after it has been held, released, and remains upright inside the strict black-tray geometry for 10 consecutive control steps; Return, rather than the Pick policy, owns posture recovery.

The evaluator reports per-episode success/failure and, on a product failure, which product and phase failed; it continues through the rest of the work order instead of stopping at the first failed product.

At Nav entry, the pipeline selects the captured Scenario whose held-arm and gripper joints are nearest to the live Pick terminal. That same Scenario drives the physical base/leg/arm alignment and the Place wheel observation reference. After reaching the shelf approach, Nav continuously controls the base, legs, and both arms together until the complete Place-start state satisfies the strict Scenario thresholds. Scenario state is never restored or teleported. Product activation updates selected-product, Pick target, and Place target runtime state without moving the robot, grippers, or products. Phase switching is owned by TaskFlowAgent; the example does not contain a second manual Pick/Nav/Place runner.

Component Tasks

Use examples/06_collect_component_task.py for standalone PickToShelf and SortToShelf phases. Select one COMPONENT_PRESET at the top of the file, then run the script.

PickToShelf presets target GalbotG1-PickToShelf-Pick-v0, GalbotG1-PickToShelf-Nav-v0, and GalbotG1-PickToShelf-Place-v0.

python examples/06_collect_component_task.py \
  --save-end-scenario data/pick_to_shelf/scenarios/nav_start.yaml

python examples/06_collect_component_task.py \
  --init-scenario data/pick_to_shelf/scenarios/nav_start.yaml \
  --save-end-scenario data/pick_to_shelf/scenarios/place_start.yaml

python examples/06_collect_component_task.py \
  --sorting-object red_cube \
  --save-end-scenario data/sort_to_shelf/scenarios/place_start_red_cube.yaml

GalbotG1-SortToShelf-Nav-v0 uses the task-local nav_sequence_agent: drive the base first, then set the place-start posture.

Coherent Tasks

Use examples/07_compound_task.py for full task flows. The default path uses task-owned phase agents; the file also shows how to override phase agents with planner or policy agents.

python examples/07_compound_task.py --task GalbotG1-PickToShelf-v0 --headless

python examples/07_compound_task.py --task GalbotG1-SortToShelf-v0 \
  --sorting-object red_cube --headless

python examples/07_compound_task.py --task GalbotG1-PickToShelf-v0 \
  --mode collect --dataset-path data/pick_to_shelf/full_expert.hdf5 --headless

Any BaseAgent can drive the same agent.act(env) -> env.step(action) loop: CuroboPlannerAgent, TeleopAgent, PolicyAgent, and TaskFlowAgent all return full IsaacLab action tensors.