[Bug] Can't open viewer while running GPU parallelized rendering

[SumeetBatra]

Hi!

I created a custom task on top of PickCube that loads a 3D scene from the Matterport3D dataset to add visual distractions during model evaluation. The function to load the scene looks like this:

def load_Matterport(builder):
    paths = sorted(list(glob.glob(os.path.join("./data/matterport3d/*.glb"))))
    path = random.choice(paths)
    pose = Pose(q=[0, 0, 0, 1])  # y-axis up for Matterport scenes
    # Add offset to workspace
    offset = np.array([0, 0, 0.8])
    builder.initial_pose = Pose(-offset)
    builder.add_visual_from_file(path, pose)
    actor = builder.build_static(name='background')
    # actor.set_pose(Pose(-offset))
    return actor

and is called inside a custom subclass of TableSceneBuilder like this:

class RandomBackgroundTableSceneBuilder(TableSceneBuilder):
    def build(self):
      ...
      load_Matterport(builder)
      builder.add_visual_from_file(
          filename=table_model_file, scale=[scale] * 3, pose=table_pose
      )
      ...

which then gets used inside the _load_scene function.

However, I run into the following error when I try to render a rollout from this env with render_mode=human:

RuntimeError: Modifying a scene (add/remove object/camera) is not allowed after creating the batched render system.

This is easily avoidable by setting the sim_backed to CPU, however if I train the model with sim_backend=gpu (which I am to leverage parallel env training), then evaluating the model with sim_backend=cpu results in significantly worse performance. Not sure if this is a known consequence of using different rendering backends or a bug, but just thought I'd bring it up here in case it's a solvable problem.

Thanks!

[StoneT2000]

I think there may be some issue at the moment with using the viewer in addition to running GPU parallelized rendering. Someone raised the same issue on discord (which I am not sure how to link to on github). I'll try and get around to investigating this, this issue has been around for months now.

A temporary workaround is to train on the GPU backend, evaluate on the GPU backend and save trajectories with the RecordEpisode wrapper. Now if you want to use the viewer (done with render_mode=human), you can replay collected trajectories on the GPU in the CPU environment using environment states by doing a for loop like

state_dicts = load_state_dicts_from_trajectory.h5
env = gym.make(env_id, num_envs=1, render_mode="human", **other_env_kwargs)
for i in range(100):
  env.set_state_dict(state_dicts[i])
  env.render_human() # or env.render() if render_mode=="human"

This is in fact how all of the nice videos/photos are generated usually on our documentation since high quality ray-tracing only works with the single CPU sim at the moment. Let me know if you need any help on the specifics of how to replay trajectories with environment state data, it might be a little bit under-documented at the moment and the code above is pseudocode-ish.