Note

This is a Hugging Face dataset. For large datasets, ensure huggingface_hub>=1.1.3 to avoid rate limits. Learn more in the Hugging Face integration docs.

Hugging Face

RoboLab-EgoX → FiftyOne (Native Multimodal MCAP)#

preview

The complete DAVIAN-Robotics/RoboLab-EgoX corpus, policy rollouts recorded on NVIDIA’s RoboLab manipulation benchmark, converted to native multimodal MCAP episodes.

Each take carries three synchronized camera views with a matching 16-bit depth stream, per-camera intrinsics, whole-arm telemetry, end-effector pose, and the task instruction. Takes keep the benchmark’s own success label, so failed rollouts sit alongside successful ones.

Installation#

pip install fiftyone

Usage#

import fiftyone as fo
import fiftyone.utils.huggingface as fouh

dataset = fouh.load_from_hub(
    "Voxel51/RoboLab-EgoX",
    name="RoboLab-EgoX",
    persistent=True,
)
fo.launch_app(dataset)

Successful rollouts only:

view = dataset.match({"success": True})

What you get#

  • 4,000 .mcap episodes of 81 frames each at 15 fps, 6.0 hours total

  • All 28 RoboLab tasks across all 99 background scenes

  • 632 successful and 3,368 failed rollouts, labelled, with a median of 126 failures per task

  • Cameras as foxglove.CompressedVideo: /ego-camera from the wrist, /exo-left-camera and /exo-right-camera from the exterior views, all 640x360 H.264 passed through without re-encoding

  • Depth as 16-bit PNG on /ego-depth, /exo-left-depth and /exo-right-depth, at 320x180

  • /ego-calibration, /exo-left-calibration, /exo-right-calibration as foxglove.CameraCalibration

  • /joint-positions (13 joints) and /actions (8), each with a timeline plot

  • /end-effector-pose as foxglove.PoseInFrame

  • /instruction carrying the task’s language instruction

  • Per-episode fields: take_name, task, background, instruction, success, source_episode, num_frames, duration

Notes on the conversion#

Depth marks no-return pixels with the uint16 ceiling, 65535, rather than zero. That is over 60% of a typical exterior frame and about 12% of a wrist frame, so any viewer that maps the full 16-bit range will render the real depth as flat black. Mask 65535 before scaling. The values are carried through unchanged.

Depth is half the camera resolution, 320x180 against 640x360. The published CameraCalibration describes the 640x360 colour frame, so projecting a depth frame to 3D means halving fx, fy, cx and cy first.

In 153 of the 4,000 takes the source mp4s carry a trailing empty packet and hold 80 real frames rather than 81, so all three camera streams are one frame shorter than depth and telemetry. num_frames follows the robot-state clock and reports 81.

Eight of the 28 tasks have no successful rollout anywhere in the source, so those contribute failures only.

License & attribution#

The RoboLab benchmark, its tasks and its scene assets are released by NVIDIA under Apache 2.0; see NVLabs/RoboLab. The rollouts converted here were recorded and published by DAVIAN-Robotics. This subset is distributed under Apache 2.0 and ships the license text as LICENSE.

Changes from the source: conversion from mp4 and HDF5 to MCAP, and PNG encoding of the depth stacks. Camera streams are the source H.264 packets passed through unchanged.

Citation#

@article{yang2026robolab,
  title={RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies},
  author={Yang, Xuning and Dagli, Rishit and Zook, Alex and Hadfield, Hugo and Goyal, Ankit and Birchfield, Stan and Ramos, Fabio and Tremblay, Jonathan},
  journal={arXiv preprint arXiv:2604.09860},
  year={2026}
}