Note

This is a Hugging Face dataset. For large datasets, ensure huggingface_hub>=1.1.3 to avoid rate limits. Learn more in the Hugging Face integration docs.

This is a FiftyOne dataset with 2,400 samples.

Installation#

If you haven’t already, install FiftyOne:

pip install -U fiftyone

Usage#

import fiftyone as fo
from fiftyone.utils.huggingface import load_from_hub

# Load the dataset
# Note: other available arguments include 'max_samples', etc
dataset = load_from_hub("Voxel51/robocasa-MG_100")

# Launch the App
session = fo.launch_app(dataset)

Dataset Card for RoboCasa MG_100 (FiftyOne)#

RoboCasa MG_100 preview

The full RoboCasa MG_100 dataset — 2,400 machine-generated kitchen-manipulation episodes covering 24 atomic robot-skill tasks (100 episodes per task) — imported into FiftyOne from DAVIAN-Robotics/robocasa-MG_100 and re-exported as a self-contained LeRobotDataset v3.0. MG_100 corresponds to the Generated-100 data condition in the RoboCasa paper: a 1/30 random subsample of the full MimicGen-generated dataset, yielding 100 autonomously-generated demonstrations per atomic task (2,400 = 24 tasks × 100 demos).

Dataset Details#

Dataset Description#

Curated by: RoboCasa authors (Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, Yuke Zhu — UT Austin & NVIDIA Research); converted to LeRobot v3.0 format and shared on the Hub by DAVIAN Robotics
Funded by: National Science Foundation (FRR-2145283, EFRI-2318065); Office of Naval Research (N00014-22-1-2204)
Shared by: DAVIAN-Robotics (original LeRobot conversion); this FiftyOne import shared by the FiftyOne community
Language(s): English (task instructions)
License: Apache 2.0

Dataset Sources#

Repository: DAVIAN-Robotics/robocasa-MG_100 (part of the RoboCasa-LeRobot-v3.0 collection) · RoboCasa code · MimicGen code
Paper: RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots (arXiv:2406.02523)
Demo: robocasa.ai

Uses#

Direct Use#

Exploring and visualizing simulated kitchen-manipulation episodes in the FiftyOne App — inspecting synchronized triple-camera RGB video alongside proprioceptive state/action trajectories, filtering or grouping by task/skill family, and prototyping imitation learning data pipelines (e.g. behavioral cloning, as in the source paper’s BC-Transformer experiments) at a manageable scale before moving to the larger Generated-3000 (72,000-episode) release.

Out-of-Scope Use#

This dataset contains only 24 of RoboCasa’s 25 atomic tasks (kitchen navigation is excluded upstream — see Parsing decisions) and none of the 75 composite/long-horizon tasks from the paper. It is entirely synthetic (MimicGen-generated) data rendered with the lightweight MuJoCo renderer and AI-generated textures for domain randomization; it is not suitable on its own for evaluating sim-to-real transfer, human-teleoperation data quality, or composite/multi-stage task performance — see the paper’s Human-50 and composite-task splits for those.

Dataset Structure#

This is a multimodal FiftyOne dataset (dataset.media_type == "multimodal") with 2,400 samples, one sample per episode. Each sample’s media (3 video streams) is not copied into per-sample files; instead it is resolved through a media_reference that points into the exported LeRobotDataset v3.0 source (data/, videos/, meta/ in this repo) at import time — this is how FiftyOne represents LeRobot episodes natively.

Fields#

Field	FiftyOne type	Description
`id`	`ObjectIdField`	FiftyOne sample id
`media_reference`	`MediaReferenceField`	Pointer into the LeRobot source’s `data/`, `videos/*`, and `meta/` files for this episode (chunk/file indexes, frame range, per-video timestamp ranges) — resolved on demand, not duplicated per sample
`tags`	`ListField(StringField)`	FiftyOne sample tags (empty by default)
`metadata`	`Metadata` (`size_bytes`, `mime_type`)	Standard FiftyOne sample metadata
`created_at` / `last_modified_at`	`DateTimeField`	FiftyOne bookkeeping timestamps
`episode_index`	`IntField`	Episode index (`0`–`2399`)
`task`	`StringField`	Task instruction for the episode (verbatim from source `meta/tasks.parquet`)
`tasks`	`ListField(StringField)`	Full task list for the episode (LeRobot `tasks` array; length 1 for every episode here)
`length`	`IntField`	Number of frames in the episode (verbatim from source); ranges 89–838
`duration`	`FloatField`	Episode duration in seconds (`length / fps`); ranges 4.45–41.9 s
`robot_type`	`StringField`	`robomimic` (verbatim from source `meta/info.json`)
`fps`	`FloatField`	Recording frame rate, 20.0 (verbatim from source)

The per-frame numeric features (observation.state (16,), action (12,), next.done (1,)) and the 3 per-frame video streams are not flattened into sample fields — they remain in the LeRobot data/*.parquet and videos/*/*.mp4 files referenced by media_reference, and are surfaced by the FiftyOne App’s State & Action and Streams viewer tabs rather than as queryable sample-level fields.

Per-frame feature	Shape	Description
`action`	`[12]`	Robot end-effector delta-pose command + gripper (`action_0`–`action_11`)
`observation.state`	`[16]`	Base-to-eef position (3) + quaternion (4), gripper qpos (2), base position (3) + quaternion (4)
`observation.images.robot0_agentview_left_image`	`[128, 128, 3]` video	Left workspace camera, AV1/yuv420p @ 20 fps
`observation.images.robot0_agentview_right_image`	`[128, 128, 3]` video	Right workspace camera, AV1/yuv420p @ 20 fps
`observation.images.robot0_eye_in_hand_image`	`[128, 128, 3]` video	Wrist-mounted eye-in-hand camera, AV1/yuv420p @ 20 fps
`next.done`	`[1]` bool	Episode-termination flag

Label types and why#

There are no traditional detection/classification/segmentation labels. task is stored as a plain StringField rather than fo.Classification because task instructions here are free-form natural-language strings disambiguating object identity and location (e.g. “pick the banana from the counter and place it in the sink”) — 291 unique strings across 24 underlying skill families, not a fixed closed-set taxonomy. Episode counts per task are highly imbalanced by design: fixed-object tasks (e.g. “turn on the sink faucet”, “press the start button on the microwave”) have exactly 100 episodes each, while pick-and-place tasks are split across many object-category variants (as few as 1 episode for some object/location combinations) because MimicGen samples object identity per generation attempt.

`dataset.info` contents#

{
    "lerobot": {
        "format": "LeRobotDataset",
        "format_major": 3,
        "episode_count": 2400,
        "imported_episode_count": 2400,
        "skipped_episodes": [],
    }
}

Parsing decisions#

Full dataset, not a subset: the source repo ships all 2,400 episodes in a single data/chunk-000/file-000.parquet shard plus 21 video shard files (~3.7 GB total), so every episode was downloaded and imported — no episode subsetting was needed or performed.
tasks.parquet repair: the source meta/tasks.parquet stored the task string as the pandas index (__index_level_0__ column) rather than a task column. This was rewritten locally to have literal task_index/task columns before import; no row values were changed.
Re-export, not a thin reference to the original repo: this repo is a self-contained LeRobotDataset v3.0 export (via FiftyOne’s LeRobotDatasetExporter), not a pointer back to DAVIAN-Robotics/robocasa-MG_100. Since all 2,400 source episodes are imported, episode indices are unchanged (0–2399); per-episode and global statistics (meta/stats.json, per-episode stats/* columns) were recomputed from the exported rows during export.
Excluded upstream (kitchen navigation): the source paper’s atomic-task set includes a NavigateKitchen task (25th atomic skill), but the Generated-100/ Generated-3000 MimicGen datasets exclude it because MimicGen cannot generate mobile navigation trajectories (paper, Sec. V-A, footnote 2) — this is a property of the source data, not something dropped during FiftyOne import.
Video codec: all three camera streams use AV1 in yuv420p at 128×128 — AV1 decodes fine in current Chromium-based browsers and in FiftyOne’s App.
No held-out or unlabeled split: all 2,400 episodes are in a single train split (per source meta/info.json "splits": {"train": "0:2400"}) with every episode language-annotated; nothing was withheld.

Dataset Creation#

Curation Rationale#

RoboCasa was built to study whether large-scale simulated data can substitute for costly human teleoperation in training generalist household-manipulation policies. The Generated-100 condition specifically exists to measure how policy performance scales with the quantity of MimicGen-generated data at a fixed 100-demos-per-task budget, positioned between the smaller Generated-100-vs-Human-50 comparison and the full 72,000-episode Generated-3000 release (paper, Sec. V-A). This FiftyOne import performs no additional re-curation of the episode set.

Source Data#

Data Collection and Processing#

Simulator: RoboCasa, built on robosuite (MuJoCo-based), with 120 procedurally-styled kitchen scenes (10 floor plans × 12 styles) and AI-generated wall/floor/counter/cabinet textures (via MidJourney) used as domain randomization during rendering.
Base human demonstrations: four human operators collected 50 teleoperated demonstrations per atomic task using a 3D SpaceMouse, each in a randomly sampled kitchen scene (1,250 demonstrations across the 25 atomic tasks).
Machine-generated expansion: MimicGen decomposes each human demonstration into object-centric manipulation segments, then re-targets and stitches those segments to synthesize new demonstrations in novel scene/object configurations, keeping only rollouts that succeed (rejection sampling). This produced 72,000 trajectories across 24 atomic tasks (Generated-3000 = 3,000 per task); this repo, MG_100, is a random 1/30 subsample at 100 demos per task (2,400 episodes total). NavigateKitchen is excluded because MimicGen cannot generate mobile navigation trajectories.
Robot / control: a Franka Panda arm on an Omron mobile base (Omni-Frankie configuration), Operational Space Control at 20 Hz, matching this dataset’s fps.
observation.state (16,) layout: robot0_base_to_eef_pos (3) + robot0_base_to_eef_quat (4) + robot0_gripper_qpos (2) + robot0_base_pos (3) + robot0_base_quat (4). action (12,) is a 12-dimensional delta end-effector / gripper / base command. Full per-dimension names are in the source meta/info.json (features.*.names).
Cameras: two static 128×128 workspace views (robot0_agentview_left_image, robot0_agentview_right_image) plus one 128×128 wrist-mounted robot0_eye_in_hand_image, all rendered with the lightweight MuJoCo renderer at 20 fps for this dataset release (the paper notes an Omniverse-rendered option is planned for a future release).

Who are the source data producers?#

Collected and generated by the RoboCasa authors (UT Austin Robot Perception and Learning Lab and NVIDIA Research; full author list above) via a combination of in-person human teleoperation (base demonstrations) and automated MimicGen trajectory synthesis (this dataset’s episodes).

Annotations#

Annotation process#

Each of the 24 atomic tasks has a fixed natural-language template describing the skill and the specific fixture/object/burner/side involved (e.g. per-object-category variants for pick-and-place, per-burner variants for stove knobs); MimicGen-generated episodes inherit the instruction associated with their task/object configuration. In this dataset, this is surfaced as the sample-level task field.

Who are the annotators?#

Task instruction templates were authored by the RoboCasa authors as part of task design (paper, Sec. IV-A, Fig. 10); no separate crowd-annotation process was used for episode labels.

Personal and Sensitive Information#

None — this is a synthetic simulation dataset (MuJoCo-rendered kitchen scenes, robot joint/pose state, and task instruction strings); no human subjects data.

Citation#

BibTeX:

@article{nasiriany2024robocasa,
  title={RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots},
  author={Nasiriany, Soroush and Maddukuri, Abhiram and Zhang, Lance and Parikh, Adeet and Lo, Aaron and Joshi, Abhishek and Mandlekar, Ajay and Zhu, Yuke},
  journal={arXiv preprint arXiv:2406.02523},
  year={2024}
}

APA:

Nasiriany, S., Maddukuri, A., Zhang, L., Parikh, A., Lo, A., Joshi, A., Mandlekar, A., & Zhu, Y. (2024). RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots. arXiv:2406.02523.

More Information#

This is the full DAVIAN-Robotics/robocasa-MG_100 dataset (2,400 episodes) re-exported as a self-contained LeRobotDataset v3.0 for FiftyOne. See robocasa.ai for the simulator, task suite, and the larger Generated-3000 (72,000-episode) and human-teleoperated (Human-50) releases in the RoboCasa-LeRobot-v3.0 collection.

Dataset Card Authors#

Harpreet Sahota

Dataset Card Contact#