#### NOTE
This is a **Hugging Face dataset**. For large datasets, ensure `huggingface_hub>=1.1.3` to avoid rate limits. Learn more in the Hugging Face integration docs.
# RoboLab-EgoX → FiftyOne (Native Multimodal MCAP)
The complete
[DAVIAN-Robotics/RoboLab-EgoX](https://huggingface.co/datasets/DAVIAN-Robotics/RoboLab-EgoX)
corpus, policy rollouts recorded on NVIDIA’s
[RoboLab](https://github.com/NVLabs/RoboLab) manipulation benchmark,
converted to native multimodal MCAP episodes.
Each take carries three synchronized camera views with a matching 16-bit
depth stream, per-camera intrinsics, whole-arm telemetry, end-effector pose,
and the task instruction. Takes keep the benchmark’s own success label, so
failed rollouts sit alongside successful ones.
## Installation
```bash
pip install fiftyone
```
## Usage
```python
import fiftyone as fo
import fiftyone.utils.huggingface as fouh
dataset = fouh.load_from_hub(
"Voxel51/RoboLab-EgoX",
name="RoboLab-EgoX",
persistent=True,
)
fo.launch_app(dataset)
```
Successful rollouts only:
```python
view = dataset.match({"success": True})
```
## What you get
- 4,000 `.mcap` episodes of 81 frames each at 15 fps, 6.0 hours total
- All **28 RoboLab tasks** across all **99 background scenes**
- **632 successful and 3,368 failed** rollouts, labelled, with a median
of 126 failures per task
- Cameras as `foxglove.CompressedVideo`: `/ego-camera` from the wrist,
`/exo-left-camera` and `/exo-right-camera` from the exterior views, all
640x360 Annex-B H.264 without B-frames
- Depth as 16-bit PNG on `/ego-depth`, `/exo-left-depth` and
`/exo-right-depth`, at 320x180
- `/ego-calibration`, `/exo-left-calibration`, `/exo-right-calibration` as
`foxglove.CameraCalibration`
- `/joint-positions` (13 joints) and `/actions` (8), each with a timeline plot
- `/end-effector-pose` as `foxglove.PoseInFrame`
- `/instruction` carrying the task’s language instruction
- Per-episode fields: `take_name`, `task`, `background`, `instruction`,
`success`, `source_episode`, `num_frames`, `duration`
## Notes on the conversion
Depth marks no-return pixels with the uint16 ceiling, `65535`, rather than
zero. That is over 60% of a typical exterior frame and about 12% of a wrist
frame, so any viewer that maps the full 16-bit range will render the real
depth as flat black. Mask `65535` before scaling. The values are carried
through unchanged.
Depth is half the camera resolution, 320x180 against 640x360. The
published `CameraCalibration` describes the 640x360 colour frame, so
projecting a depth frame to 3D means halving `fx`, `fy`, `cx` and `cy`
first.
In 153 of the 4,000 takes the source mp4s carry a trailing empty packet and
hold 80 real frames rather than 81, so all three camera streams are one
frame shorter than depth and telemetry. `num_frames` follows the
robot-state clock and reports 81.
Eight of the 28 tasks have no successful rollout anywhere in the source, so
those contribute failures only.
## License & attribution
The RoboLab benchmark, its tasks and its scene assets are released by NVIDIA
under [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0); see
[NVLabs/RoboLab](https://github.com/NVLabs/RoboLab). The rollouts converted
here were recorded and published by
[DAVIAN-Robotics](https://huggingface.co/DAVIAN-Robotics). This subset is
distributed under Apache 2.0 and ships the license text as `LICENSE`.
Changes from the source: conversion from mp4 and HDF5 to MCAP, PNG encoding
of the depth stacks, and re-encoding of the camera streams to Annex-B H.264
without B-frames.
## Citation
```bibtex
@article{yang2026robolab,
title={RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies},
author={Yang, Xuning and Dagli, Rishit and Zook, Alex and Hadfield, Hugo and Goyal, Ankit and Birchfield, Stan and Ramos, Fabio and Tremblay, Jonathan},
journal={arXiv preprint arXiv:2604.09860},
year={2026}
}
```