#### NOTE
This is a **Hugging Face dataset**. For large datasets, ensure `huggingface_hub>=1.1.3` to avoid rate limits. Learn more in the Hugging Face integration docs.
This is a [FiftyOne](https://github.com/voxel51/fiftyone) dataset with 2,400 samples.
# Installation
If you haven’t already, install FiftyOne:
```bash
pip install -U fiftyone
```
# Usage
```python
import fiftyone as fo
from fiftyone.utils.huggingface import load_from_hub
# Load the dataset
# Note: other available arguments include 'max_samples', etc
dataset = load_from_hub("Voxel51/robocasa-MG_100")
# Launch the App
session = fo.launch_app(dataset)
```
# Dataset Card for RoboCasa MG_100 (FiftyOne)
The full **RoboCasa MG_100** dataset — 2,400 machine-generated kitchen-manipulation
episodes covering 24 atomic robot-skill tasks (100 episodes per task) — imported into
FiftyOne from [DAVIAN-Robotics/robocasa-MG_100](https://huggingface.co/datasets/DAVIAN-Robotics/robocasa-MG_100)
and re-exported as a self-contained LeRobotDataset v3.0. `MG_100` corresponds to the
`Generated-100` data condition in the [RoboCasa paper](https://arxiv.org/abs/2406.02523):
a 1/30 random subsample of the full MimicGen-generated dataset, yielding 100
autonomously-generated demonstrations per atomic task (2,400 = 24 tasks × 100 demos).
## Dataset Details
### Dataset Description
- **Curated by:** RoboCasa authors (Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang,
Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, Yuke Zhu — UT Austin & NVIDIA
Research); converted to LeRobot v3.0 format and shared on the Hub by DAVIAN Robotics
- **Funded by:** National Science Foundation (FRR-2145283, EFRI-2318065); Office of
Naval Research (N00014-22-1-2204)
- **Shared by:** [DAVIAN-Robotics](https://huggingface.co/DAVIAN-Robotics) (original
LeRobot conversion); this FiftyOne import shared by the FiftyOne community
- **Language(s):** English (task instructions)
- **License:** Apache 2.0
### Dataset Sources
- **Repository:** [DAVIAN-Robotics/robocasa-MG_100](https://huggingface.co/datasets/DAVIAN-Robotics/robocasa-MG_100)
(part of the [RoboCasa-LeRobot-v3.0](https://huggingface.co/collections/DAVIAN-Robotics)
collection) · [RoboCasa code](https://robocasa.ai) · [MimicGen code](https://github.com/NVlabs/mimicgen)
- **Paper:** [RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots (arXiv:2406.02523)](https://arxiv.org/abs/2406.02523)
- **Demo:** [robocasa.ai](https://robocasa.ai)
## Uses
### Direct Use
Exploring and visualizing simulated kitchen-manipulation episodes in the FiftyOne App —
inspecting synchronized triple-camera RGB video alongside proprioceptive state/action
trajectories, filtering or grouping by task/skill family, and prototyping imitation
learning data pipelines (e.g. behavioral cloning, as in the source paper’s
BC-Transformer experiments) at a manageable scale before moving to the larger
`Generated-3000` (72,000-episode) release.
### Out-of-Scope Use
This dataset contains only 24 of RoboCasa’s 25 atomic tasks (kitchen navigation is
excluded upstream — see *Parsing decisions*) and none of the 75 composite/long-horizon
tasks from the paper. It is entirely synthetic (MimicGen-generated) data rendered with
the lightweight MuJoCo renderer and AI-generated textures for domain randomization; it
is not suitable on its own for evaluating sim-to-real transfer, human-teleoperation data
quality, or composite/multi-stage task performance — see the paper’s `Human-50` and
composite-task splits for those.
## Dataset Structure
This is a **multimodal** FiftyOne dataset (`dataset.media_type == "multimodal"`) with
**2,400 samples**, one sample per episode. Each sample’s media (3 video streams) is not
copied into per-sample files; instead it is resolved through a `media_reference` that
points into the exported LeRobotDataset v3.0 source (`data/`, `videos/`, `meta/` in this
repo) at import time — this is how FiftyOne represents LeRobot episodes natively.
### Fields
| Field | FiftyOne type | Description |
|-----------------------------------|----------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `id` | `ObjectIdField` | FiftyOne sample id |
| `media_reference` | `MediaReferenceField` | Pointer into the LeRobot source’s `data/`, `videos/*`, and `meta/` files for this episode (chunk/file indexes, frame range, per-video timestamp ranges) — resolved on demand, not duplicated per sample |
| `tags` | `ListField(StringField)` | FiftyOne sample tags (empty by default) |
| `metadata` | `Metadata` (`size_bytes`, `mime_type`) | Standard FiftyOne sample metadata |
| `created_at` / `last_modified_at` | `DateTimeField` | FiftyOne bookkeeping timestamps |
| `episode_index` | `IntField` | Episode index (`0`–`2399`) |
| `task` | `StringField` | Task instruction for the episode (verbatim from source `meta/tasks.parquet`) |
| `tasks` | `ListField(StringField)` | Full task list for the episode (LeRobot `tasks` array; length 1 for every episode here) |
| `length` | `IntField` | Number of frames in the episode (verbatim from source); ranges 89–838 |
| `duration` | `FloatField` | Episode duration in seconds (`length / fps`); ranges 4.45–41.9 s |
| `robot_type` | `StringField` | `robomimic` (verbatim from source `meta/info.json`) |
| `fps` | `FloatField` | Recording frame rate, 20.0 (verbatim from source) |
The per-frame numeric features (`observation.state` (16,), `action` (12,),
`next.done` (1,)) and the 3 per-frame video streams are **not** flattened into sample
fields — they remain in the LeRobot `data/*.parquet` and `videos/*/*.mp4` files
referenced by `media_reference`, and are surfaced by the FiftyOne App’s State & Action
and Streams viewer tabs rather than as queryable sample-level fields.
| Per-frame feature | Shape | Description |
|---------------------------------------------------|-----------------------|-------------------------------------------------------------------------------------------------|
| `action` | `[12]` | Robot end-effector delta-pose command + gripper (`action_0`–`action_11`) |
| `observation.state` | `[16]` | Base-to-eef position (3) + quaternion (4), gripper qpos (2), base position (3) + quaternion (4) |
| `observation.images.robot0_agentview_left_image` | `[128, 128, 3]` video | Left workspace camera, AV1/yuv420p @ 20 fps |
| `observation.images.robot0_agentview_right_image` | `[128, 128, 3]` video | Right workspace camera, AV1/yuv420p @ 20 fps |
| `observation.images.robot0_eye_in_hand_image` | `[128, 128, 3]` video | Wrist-mounted eye-in-hand camera, AV1/yuv420p @ 20 fps |
| `next.done` | `[1]` bool | Episode-termination flag |
### Label types and why
There are no traditional detection/classification/segmentation labels. `task` is stored
as a plain `StringField` rather than `fo.Classification` because task instructions here
are free-form natural-language strings disambiguating object identity and location
(e.g. *“pick the banana from the counter and place it in the sink”*) — 291 unique
strings across 24 underlying skill families, not a fixed closed-set taxonomy. Episode
counts per task are highly imbalanced by design: fixed-object tasks (e.g. *“turn on the
sink faucet”*, *“press the start button on the microwave”*) have exactly 100 episodes
each, while pick-and-place tasks are split across many object-category variants (as few
as 1 episode for some object/location combinations) because MimicGen samples object
identity per generation attempt.
### `dataset.info` contents
```python
{
"lerobot": {
"format": "LeRobotDataset",
"format_major": 3,
"episode_count": 2400,
"imported_episode_count": 2400,
"skipped_episodes": [],
}
}
```
### Parsing decisions
- **Full dataset, not a subset:** the source repo ships all 2,400 episodes in a single
`data/chunk-000/file-000.parquet` shard plus 21 video shard files (~3.7 GB total), so
every episode was downloaded and imported — no episode subsetting was needed or
performed.
- **`tasks.parquet` repair:** the source `meta/tasks.parquet` stored the task string as
the pandas index (`__index_level_0__` column) rather than a `task` column. This was
rewritten locally to have literal `task_index`/`task` columns before import; no row
values were changed.
- **Re-export, not a thin reference to the original repo:** this repo is a
self-contained LeRobotDataset v3.0 export (via FiftyOne’s `LeRobotDatasetExporter`),
not a pointer back to `DAVIAN-Robotics/robocasa-MG_100`. Since all 2,400 source
episodes are imported, episode indices are unchanged (`0`–`2399`); per-episode and
global statistics (`meta/stats.json`, per-episode `stats/*` columns) were recomputed
from the exported rows during export.
- **Excluded upstream (kitchen navigation):** the source paper’s atomic-task set
includes a `NavigateKitchen` task (25th atomic skill), but the `Generated-100`/
`Generated-3000` MimicGen datasets exclude it because MimicGen cannot generate mobile
navigation trajectories (paper, Sec. V-A, footnote 2) — this is a property of the
source data, not something dropped during FiftyOne import.
- **Video codec:** all three camera streams use AV1 in `yuv420p` at 128×128 — AV1
decodes fine in current Chromium-based browsers and in FiftyOne’s App.
- **No held-out or unlabeled split:** all 2,400 episodes are in a single `train` split
(per source `meta/info.json` `"splits": {"train": "0:2400"}`) with every episode
language-annotated; nothing was withheld.
## Dataset Creation
### Curation Rationale
RoboCasa was built to study whether large-scale simulated data can substitute for
costly human teleoperation in training generalist household-manipulation policies. The
`Generated-100` condition specifically exists to measure how policy performance scales
with the *quantity* of MimicGen-generated data at a fixed 100-demos-per-task budget,
positioned between the smaller `Generated-100`-vs-`Human-50` comparison and the full
72,000-episode `Generated-3000` release (paper, Sec. V-A). This FiftyOne import performs
no additional re-curation of the episode set.
### Source Data
#### Data Collection and Processing
- **Simulator:** RoboCasa, built on [robosuite](https://github.com/ARISE-Initiative/robosuite)
(MuJoCo-based), with 120 procedurally-styled kitchen scenes (10 floor plans × 12
styles) and AI-generated wall/floor/counter/cabinet textures (via MidJourney) used as
domain randomization during rendering.
- **Base human demonstrations:** four human operators collected 50 teleoperated
demonstrations per atomic task using a 3D SpaceMouse, each in a randomly sampled
kitchen scene (1,250 demonstrations across the 25 atomic tasks).
- **Machine-generated expansion:** [MimicGen](https://arxiv.org/abs/2310.17596) decomposes
each human demonstration into object-centric manipulation segments, then re-targets
and stitches those segments to synthesize new demonstrations in novel scene/object
configurations, keeping only rollouts that succeed (rejection sampling). This produced
72,000 trajectories across 24 atomic tasks (`Generated-3000` = 3,000 per task); this
repo, `MG_100`, is a random 1/30 subsample at 100 demos per task (2,400 episodes
total). `NavigateKitchen` is excluded because MimicGen cannot generate mobile
navigation trajectories.
- **Robot / control:** a Franka Panda arm on an Omron mobile base (Omni-Frankie
configuration), Operational Space Control at 20 Hz, matching this dataset’s `fps`.
- **`observation.state` (16,) layout:** `robot0_base_to_eef_pos` (3) +
`robot0_base_to_eef_quat` (4) + `robot0_gripper_qpos` (2) + `robot0_base_pos` (3) +
`robot0_base_quat` (4). **`action` (12,)** is a 12-dimensional delta end-effector /
gripper / base command. Full per-dimension names are in the source
`meta/info.json` (`features.*.names`).
- **Cameras:** two static 128×128 workspace views (`robot0_agentview_left_image`,
`robot0_agentview_right_image`) plus one 128×128 wrist-mounted `robot0_eye_in_hand_image`,
all rendered with the lightweight MuJoCo renderer at 20 fps for this dataset release
(the paper notes an Omniverse-rendered option is planned for a future release).
#### Who are the source data producers?
Collected and generated by the RoboCasa authors (UT Austin Robot Perception and
Learning Lab and NVIDIA Research; full author list above) via a combination of
in-person human teleoperation (base demonstrations) and automated MimicGen trajectory
synthesis (this dataset’s episodes).
### Annotations
#### Annotation process
Each of the 24 atomic tasks has a fixed natural-language template describing the skill
and the specific fixture/object/burner/side involved (e.g. per-object-category variants
for pick-and-place, per-burner variants for stove knobs); MimicGen-generated episodes
inherit the instruction associated with their task/object configuration. In this
dataset, this is surfaced as the sample-level `task` field.
#### Who are the annotators?
Task instruction templates were authored by the RoboCasa authors as part of task design
(paper, Sec. IV-A, Fig. 10); no separate crowd-annotation process was used for episode
labels.
#### Personal and Sensitive Information
None — this is a synthetic simulation dataset (MuJoCo-rendered kitchen scenes, robot
joint/pose state, and task instruction strings); no human subjects data.
## Citation
**BibTeX:**
```bibtex
@article{nasiriany2024robocasa,
title={RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots},
author={Nasiriany, Soroush and Maddukuri, Abhiram and Zhang, Lance and Parikh, Adeet and Lo, Aaron and Joshi, Abhishek and Mandlekar, Ajay and Zhu, Yuke},
journal={arXiv preprint arXiv:2406.02523},
year={2024}
}
```
**APA:**
Nasiriany, S., Maddukuri, A., Zhang, L., Parikh, A., Lo, A., Joshi, A., Mandlekar, A., & Zhu, Y. (2024). *RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots*. arXiv:2406.02523.
## More Information
This is the full [DAVIAN-Robotics/robocasa-MG_100](https://huggingface.co/datasets/DAVIAN-Robotics/robocasa-MG_100)
dataset (2,400 episodes) re-exported as a self-contained LeRobotDataset v3.0 for
FiftyOne. See [robocasa.ai](https://robocasa.ai) for the simulator, task suite, and the
larger `Generated-3000` (72,000-episode) and human-teleoperated (`Human-50`) releases in
the [RoboCasa-LeRobot-v3.0](https://huggingface.co/collections/DAVIAN-Robotics)
collection.
## Dataset Card Authors
[Harpreet Sahota](https://huggingface.co/harpreetsahota)
## Dataset Card Contact
[Harpreet Sahota](https://huggingface.co/harpreetsahota)