#### NOTE This is a **Hugging Face dataset**. For large datasets, ensure `huggingface_hub>=1.1.3` to avoid rate limits. Learn more in the Hugging Face integration docs. ![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Dataset-yellow) # Dataset Card for EgoDex Test Set (FiftyOne Multimodal) ![preview](https://huggingface.co/datasets/Voxel51/egodex-test-multimodal/resolve/main/egodex.gif) EgoDex is the largest and most diverse dataset of egocentric dexterous manipulation to date, published at ICLR 2026. Collected with Apple Vision Pro and ARKit, each episode pairs 1920×1080 30 Hz egocentric video with dense 3D skeletal pose annotations (SE(3) transforms for 68 body and finger joints), GPT-4 language descriptions, and camera extrinsics. This FiftyOne dataset contains the **test set** (3,243 episodes across 111 tabletop manipulation tasks), converted to MCAP format for synchronized multimodal exploration in the FiftyOne App. Each MCAP episode streams H.264 video, a 3D animated skeleton, a 2D skeleton overlay projected onto the video frame, and per-episode metadata—all on a shared 30 Hz timeline. ## Installation ```bash pip install -U fiftyone ``` ## Usage ```python import fiftyone as fo from fiftyone.utils.huggingface import load_from_hub dataset = load_from_hub("Voxel51/egodex-test-multimodal") session = fo.launch_app(dataset) ``` ## Dataset Details ### Dataset Description - **Curated by:** Ryan Hoque, Peide Huang, David J. Yoon, Mouli Sivapurapu, Jian Zhang (Apple) - **Funded by:** Apple - **Shared by:** Apple; FiftyOne conversion by Harpreet Sahota - **Language(s):** English - **License:** CC-BY-NC-ND 4.0 ### Dataset Sources - **Repository:** https://github.com/apple/ml-egodex - **Paper:** https://arxiv.org/abs/2505.11709 - **Demo:** https://huggingface.co/spaces/harpreetsahota/egodex-fiftyone ## Uses ### Direct Use - **Dexterous manipulation research**: Pretraining and evaluating imitation learning policies for hand trajectory prediction. The paper’s benchmark evaluates best-of-K Euclidean distance for 48-dimensional action chunks (wrist positions + 6D orientations + fingertip positions) over 1–3 second horizons. - **Egocentric video understanding**: Action recognition, human-object interaction detection, affordance learning, and object contact point modeling. - **Video generation and world modeling**: Language-conditioned egocentric video synthesis and world model training from the paired video, pose, and language modalities. - **Robotics pretraining**: Co-training with small-scale robot datasets; learning manipulation priors from human-object interaction trajectories for downstream fine-tuning. - **FiftyOne multimodal exploration**: Browse, filter, and compare episodes in the App across tasks, task types, duration, and language descriptions. The 3D panel renders the animated skeleton; the Image panel shows the 2D skeleton overlay directly on the video. ### Out-of-Scope Use This dataset contains only the test set (3,243 episodes, 7 hours). Training models from scratch requires the full training set (725 hours, available via the source repository). The dataset covers tabletop manipulation in constrained studio environments; use cases requiring scene diversity or outdoor settings are out of scope. Commercial use is excluded by the CC-BY-NC-ND license. ## Dataset Structure ### Overview The FiftyOne dataset is **flat** with `media_type="multimodal"`. Each sample corresponds to one manipulation episode stored as a single `.mcap` file. There are 3,243 samples covering 111 distinct tasks. Episodes are organized by task (`task` field). No explicit train/val/test splits exist within this dataset — the entire dataset is the held-out 1% test partition from the paper’s 99/1 train/test split. ### Sample Fields | Field | FiftyOne type | Description | |-------------------|--------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| | `filepath` | `StringField` | Absolute path to the `.mcap` file for this episode | | `task` | `StringField` | Canonical task name using underscores (e.g., `basic_pick_place`) | | `task_type` | `StringField` | Task category: `reset`, `reset-free`, or `reversible` | | `episode_index` | `IntField` | Numeric index of the episode within its task folder | | `language` | `Classification` | GPT-4-generated natural language description of the manipulation. For reversible tasks, the applicable direction is pre-selected using `which_llm_description`. | | `language2` | `StringField` | Inverse-direction description for reversible tasks; empty string for non-reversible episodes | | `objects` | `ListField(StringField)` | Manipulated objects identified by GPT-4 (e.g., `["stapler"]`) | | `verbs` | `ListField(StringField)` | Action verbs identified by GPT-4 (e.g., `["pick", "place"]`) | | `environment` | `StringField` | Structured string with table material, position, background, and hand used | | `duration_s` | `FloatField` | Episode duration in seconds (range: ~1.7 s to 30+ s) | | `frame_count` | `IntField` | Number of 30 Hz frames; equals `round(duration_s * 30)` | | `message_count` | `IntField` | Total MCAP messages across all channels (≈ `frame_count × 4 + 2`) | | `has_video` | `BooleanField` | True for all episodes (`foxglove.CompressedVideo` present) | | `has_skeleton` | `BooleanField` | True for all episodes (`foxglove.SceneUpdate` skeleton present) | | `has_calibration` | `BooleanField` | True for all episodes (`foxglove.CameraCalibration` present) | | `hdf5_path` | `StringField` | Path to the source HDF5 file; used by training pipelines that need raw SE(3) data | ### MCAP Channels (per episode) Each `.mcap` file contains six synchronized channels at 30 Hz (except calibration and metadata, which are logged once): | Topic | Schema | Rate | Description | |-----------------------|------------------------------|------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `/camera/image_raw` | `foxglove.CompressedVideo` | 30 Hz | H.264 encoded egocentric video, 1920×1080. Linked to calibration via `mcap.calibration_topic` metadata. | | `/camera/calibration` | `foxglove.CameraCalibration` | Once at t₀ | Constant pinhole intrinsics: fx=fy=736.6339, cx=960, cy=540. Distortion model `plumb_bob` with D=[0,0,0,0,0] (ARKit output is undistorted). | | `/skeleton` | `foxglove.SceneUpdate` | 30 Hz | 3D animated skeleton as a `LineList` of bone segments in the ARKit world frame. Colors follow the devkit convention: index=tomato, middle=pale turquoise, ring=green-yellow, little=light blue, thumb=violet, arm/spine=grey. | | `/skeleton_2d` | `foxglove.ImageAnnotations` | 30 Hz | 2D skeleton projected onto the image plane using `inv(cam_ext) @ joint_tf` then pinhole projection with K. `PointsAnnotation` with `LineList` type; same per-finger color scheme as `/skeleton`. | | `/tf` | `foxglove.FrameTransform` | 30 Hz | Dynamic camera extrinsics (parent=`world`, child=`camera`) extracted from the ARKit `transforms/camera` field. All 68 body joint positions are encoded in `/skeleton` and `/skeleton_2d` rather than as individual TF frames. | | `/metadata` | `egodex_metadata` (JSON) | Once at t₀ | Task name, task type, language description, objects, verbs, environment string, session name, annotator version, and collector code. | ### 3D Coordinate Frame All transforms originate from ARKit’s stationary origin frame set at the start of each recording session. The world frame is **not globally consistent across episodes** — each session initializes its own origin. For training, poses should be converted to the camera frame using `inv(cam_ext) @ joint_tf` before computing action representations, as done in the paper’s benchmark. ### Source Annotations (in raw HDF5, accessible via `hdf5_path`) The raw HDF5 files (not stored in this FiftyOne dataset) contain: - **`transforms/`** (N×4×4 float32): SE(3) pose in ARKit world frame for all 68 joints plus the camera. Joints: 20 upper body (hip, spine1–7, neck1–4, left/right shoulder/arm/forearm/hand), 24 left finger joints, 24 right finger joints. - **`confidences/`** (N float32): ARKit tracking confidence 0–1. Wrist confidence = whole-hand detection; finger confidence = relative to wrist. Absent in ~6% of episodes. - **`camera/intrinsic`** (3×3 float32): Constant across all files. - **HDF5 attributes**: `session_name`, `environment`, `extra` (collector ID), `annotator_version`, `llm_description`, `llm_description2`, `llm_type`, `which_llm_description`, `llm_objects`, `llm_verbs`. ### Parsing Decisions - **Reversible-task language**: For reversible episodes, `which_llm_description` (‘1’ or ‘2’) determines which of `llm_description`/`llm_description2` applies to that episode. The applicable description is stored in `language`; the inverse is stored in `language2`. - **Session name normalization**: Raw `session_name` attributes contained several malformed variants (double `.mov` extension, all-dash separators, 5-digit years, trailing characters). A regex extractor handles all observed variants; ~29 of 3,243 episodes were affected. - **Video transcoding**: Source MP4 uses MPEG-4 Part 2 codec. Each episode is transcoded to H.264 (libx264, `-bf 0`, `-g 30`, level 5.0) before MCAP authoring to satisfy `foxglove.CompressedVideo` requirements. - **Timestamps**: HDF5 files contain no per-frame timestamps. Frame timestamps are synthesized as `session_start_ns + frame_index × (1e9/30)`, where `session_start_ns` is parsed from `session_name`. ## Dataset Creation ### Curation Rationale Imitation learning for robot manipulation is bottlenecked by data scarcity. Unlike text or images, there is no Internet-scale corpus of dexterous manipulation data. EgoDex addresses this by exploiting the passive scalability of egocentric human video: data is a byproduct of performing everyday tasks while wearing an Apple Vision Pro, rather than requiring deliberate robot operation. The dataset prioritizes behavioral diversity — 194 tasks covering a wide range of manipulation behaviors beyond pick-and-place — over scene diversity. ARKit’s on-device SLAM and multi-camera calibration provide precise per-joint pose annotations without post-hoc estimation, which is critical for fine-grained dexterous manipulation data. ### Source Data #### Data Collection and Processing All data is collected with Apple Vision Pro running visionOS 2, using ARKit for production-grade pose tracking. Data is recorded in sessions of approximately 10–15 minutes, each containing many individual episodes demarcated by pause/resume of the recording app. Raw video is compressed using modern video codecs (the raw data would otherwise exceed 500 TB). Language metadata is collected per-session by data collectors and then processed through GPT-4 to produce structured, consistent natural language descriptions. #### Who are the source data producers? Data collectors at Apple performed demonstrations of tabletop manipulation tasks while wearing Apple Vision Pro. No further information about individual collectors is publicly available. ### Annotations #### Annotation process 3D skeletal pose annotations are produced automatically at recording time by ARKit running on-device, using multiple calibrated cameras and on-device SLAM. This is a production-grade system without manual annotation. Language annotations are produced in two stages: (1) data collectors write brief free-text task descriptions; (2) GPT-4 synthesizes collector notes (task name, environment details, object descriptions) into a single polished natural language description per episode. #### Who are the annotators? Pose annotations are machine-generated by ARKit. Language annotations are generated by GPT-4 from collector-written metadata; the collectors are Apple employees. #### Personal and Sensitive Information The dataset does not contain faces, voices, or other biometric identifiers beyond hand and arm motion. The Apple Vision Pro captures only tabletop scenes from a first-person viewpoint; backgrounds are controlled studio environments. No personally identifiable information is included in the released data. ## Citation **BibTeX:** ```bibtex @inproceedings{egodex, title = {EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video}, author = {Ryan Hoque and Peide Huang and David J. Yoon and Mouli Sivapurapu and Jian Zhang}, booktitle = {International Conference on Learning Representations (ICLR)}, year = {2026}, url = {https://arxiv.org/abs/2505.11709} } ``` **APA:** Hoque, R., Huang, P., Yoon, D. J., Sivapurapu, M., & Zhang, J. (2026). EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video. *International Conference on Learning Representations (ICLR)*. https://arxiv.org/abs/2505.11709 ## More Information - Full dataset (training + test + additional data): https://github.com/apple/ml-egodex - FiftyOne documentation: https://docs.voxel51.com - MCAP format specification: https://mcap.dev ## Dataset Card Authors [Harpreet Sahota](https://huggingface.co/harpreetsahota)