Is rendering serialized across envs when using RecordEpisode

[wangerforcs]

Hi, I am currently using RecordEpisode with num_envs=128 to generate a large batch of demonstration data. However, after profiling the execution, I noticed that the rendering process for the 128 cameras appears to be serialized on the GPU.
The area highlighted in the red box corresponds to the rendering pass of a single camera.

Is this serial rendering behavior expected in the current SAPIEN/ManiSkill rendering pipeline? Are there specific parameters (e.g., in gym.make or RecordEpisode) that I need to configure to enable real parallel rendering for the cameras?

[StoneT2000]

This is real parallel rendering. Are you not getting speed ups?

[wangerforcs]

Thank you for replying. I find it is not as fast as I think with low GPU util.

I'm using the diffusion policy from the baselines and inference through make_eval_envs. I want to save many successful rollout videos, so I set save_video=True and sim_backend="physx_cuda".
As I understand it, the rendering call chain in ManiSkill is: render_rgb_array -> get_human_render_camera_images -> camera.capture -> take_picture
When RecordEpisode is used with GPU rendering and num_envs > 1, the code eventually calls camera_group.take_picture, which corresponds to BatchedCamera::takePicture in SAPIEN. Internally, it renders once per camera:

for (auto &cam : mCameras) { 
    cam->getInternalRenderer().render(cam->getInternalCamera(), {}, {}, {}, {});
}

From GPU metrics profiled by nsys, I see exactly num_envs rendering cycles — the region marked in red — so it looks like rendering for different environments is serialized on the GPU. I'm not sure why this happens. Is this expected behavior?

[StoneT2000]

These are just queuing up the rendering calls. What's the rendering FPS you are getting?

@fbxiang may have a better answer.

[wangerforcs]

When rendering videos at 512×512 resolution and num_envs=128, I’m observing a total throughput of around 210 FPS. The profiler reports the following:

env.step: 211.882 steps/s, 1.655 parallel steps/s, 200 steps in 120.822s
    CPU mem: 3033.281 MB, GPU mem: 1976.926 MB