A few simple-eval episodes seem to start from a broken scene (object fallen / tipped over / stuck in the table)

[npuhandsome]

Hi, and first of all thank you for open-sourcing SIMPLE together with the evaluation assets. It has been a great platform to work with, so we wanted to report something we ran into, in case it is useful to you or to other users.

Setup

• SIMPLE @ b49c1ae (Isaac backend, IsaacSim 4.5), robot g1_sonic
• Eval assets from USC-PSI-Lab/psi-data/simple-eval, 10 episodes per task and dr-level
• Standard eval CLI from the README (simple.cli.eval_decoupled_wbc --data-dir data/evals/simple-eval//dr-level- ...)

What we observed
In a small number of episodes the scene already looks unrecoverable in the very first recorded frame, before the policy has moved. We went through the first frames of all human-teleoperation tasks x 3 dr-levels x 10 episodes; these are the ones we found (first frames attached below):

1. G1WholebodyOpenTrashCanTeleop-v0, dr-level-2, episodes 0, 2, 8: the trash can lies on its side on the floor. Episodes 4, 6: the trash can is tilted and partly inside the table. That is 5 of the 10 episodes at this level.
2. G1WholebodyLocomotionPickBetweenTablesTeleop-v0, dr-level-2, episodes 7 and 9: the cracker box is on the floor next to the table. Episode 6: the box lies on its side on the table.
3. G1WholebodyXMoveBendPickTeleop-v0, dr-level-1, episode 2, and G1WholebodyBendPickAndPlaceTeleop-v0, dr-level-0, episode 2: the cracker box is on the floor in front of the table and the table is empty.
4. G1WholebodyHandoverTeleop-v0, dr-level-0 episodes 1 and 4, dr-level-1 episode 1: instead of lying flat between the two hands, the box starts standing on its end or resting against the gripper, and tips over onto the hand within the first second.

These initial states are deterministic: across the different policies we evaluated, the first frames are pixel-identical, and every policy fails every episode in items 1-3. For these cells the success rate therefore seems to be bounded by the scene rather than by the policy, which makes cross-paper comparison a bit tricky, especially at dr-level-2.

A few questions, if you have a moment

1. Is this expected behaviour for these episodes, or does it look like something went wrong when they were generated or settled?
2. Would you consider regenerating the affected episodes, so that numbers reported on simple-eval stay comparable across papers?
3. Could you share how these episodes were treated in the numbers reported in the SIMPLE paper?

Thanks again for the great work and for maintaining the benchmark.

[songlin]

@npuhandsome Hi, thanks for reporting, we will fix it soon @Nizhenhao-3