Note

This is a Hugging Face dataset. For large datasets, ensure huggingface_hub>=1.1.3 to avoid rate limits. Learn more in the Hugging Face integration docs.

Hugging Face

Dataset Card for EgoDex Test Set (FiftyOne Multimodal)#

preview

EgoDex is the largest and most diverse dataset of egocentric dexterous manipulation to date, published at ICLR 2026. Collected with Apple Vision Pro and ARKit, each episode pairs 1920×1080 30 Hz egocentric video with dense 3D skeletal pose annotations (SE(3) transforms for 68 body and finger joints), GPT-4 language descriptions, and camera extrinsics.

This FiftyOne dataset contains the test set (3,243 episodes across 111 tabletop manipulation tasks), converted to MCAP format for synchronized multimodal exploration in the FiftyOne App. Each MCAP episode streams H.264 video, a 3D animated skeleton, a 2D skeleton overlay projected onto the video frame, and per-episode metadata—all on a shared 30 Hz timeline.

Installation#

pip install -U fiftyone

Usage#

import fiftyone as fo
from fiftyone.utils.huggingface import load_from_hub

dataset = load_from_hub("Voxel51/egodex-test-multimodal")
session = fo.launch_app(dataset)

Dataset Details#

Dataset Description#

  • Curated by: Ryan Hoque, Peide Huang, David J. Yoon, Mouli Sivapurapu, Jian Zhang (Apple)

  • Funded by: Apple

  • Shared by: Apple; FiftyOne conversion by Harpreet Sahota

  • Language(s): English

  • License: CC-BY-NC-ND 4.0

Dataset Sources#

  • Repository: https://github.com/apple/ml-egodex

  • Paper: https://arxiv.org/abs/2505.11709

  • Demo: https://huggingface.co/spaces/harpreetsahota/egodex-fiftyone

Uses#

Direct Use#

  • Dexterous manipulation research: Pretraining and evaluating imitation learning policies for hand trajectory prediction. The paper’s benchmark evaluates best-of-K Euclidean distance for 48-dimensional action chunks (wrist positions + 6D orientations + fingertip positions) over 1–3 second horizons.

  • Egocentric video understanding: Action recognition, human-object interaction detection, affordance learning, and object contact point modeling.

  • Video generation and world modeling: Language-conditioned egocentric video synthesis and world model training from the paired video, pose, and language modalities.

  • Robotics pretraining: Co-training with small-scale robot datasets; learning manipulation priors from human-object interaction trajectories for downstream fine-tuning.

  • FiftyOne multimodal exploration: Browse, filter, and compare episodes in the App across tasks, task types, duration, and language descriptions. The 3D panel renders the animated skeleton; the Image panel shows the 2D skeleton overlay directly on the video.

Out-of-Scope Use#

This dataset contains only the test set (3,243 episodes, 7 hours). Training models from scratch requires the full training set (725 hours, available via the source repository). The dataset covers tabletop manipulation in constrained studio environments; use cases requiring scene diversity or outdoor settings are out of scope. Commercial use is excluded by the CC-BY-NC-ND license.

Dataset Structure#

Overview#

The FiftyOne dataset is flat with media_type="multimodal". Each sample corresponds to one manipulation episode stored as a single .mcap file. There are 3,243 samples covering 111 distinct tasks.

Episodes are organized by task (task field). No explicit train/val/test splits exist within this dataset — the entire dataset is the held-out 1% test partition from the paper’s 99/1 train/test split.

Sample Fields#

Field

FiftyOne type

Description

filepath

StringField

Absolute path to the .mcap file for this episode

task

StringField

Canonical task name using underscores (e.g., basic_pick_place)

task_type

StringField

Task category: reset, reset-free, or reversible

episode_index

IntField

Numeric index of the episode within its task folder

language

Classification

GPT-4-generated natural language description of the manipulation. For reversible tasks, the applicable direction is pre-selected using which_llm_description.

language2

StringField

Inverse-direction description for reversible tasks; empty string for non-reversible episodes

objects

ListField(StringField)

Manipulated objects identified by GPT-4 (e.g., ["stapler"])

verbs

ListField(StringField)

Action verbs identified by GPT-4 (e.g., ["pick", "place"])

environment

StringField

Structured string with table material, position, background, and hand used

duration_s

FloatField

Episode duration in seconds (range: ~1.7 s to 30+ s)

frame_count

IntField

Number of 30 Hz frames; equals round(duration_s * 30)

message_count

IntField

Total MCAP messages across all channels (≈ frame_count × 4 + 2)

has_video

BooleanField

True for all episodes (foxglove.CompressedVideo present)

has_skeleton

BooleanField

True for all episodes (foxglove.SceneUpdate skeleton present)

has_calibration

BooleanField

True for all episodes (foxglove.CameraCalibration present)

hdf5_path

StringField

Path to the source HDF5 file; used by training pipelines that need raw SE(3) data

MCAP Channels (per episode)#

Each .mcap file contains six synchronized channels at 30 Hz (except calibration and metadata, which are logged once):

Topic

Schema

Rate

Description

/camera/image_raw

foxglove.CompressedVideo

30 Hz

H.264 encoded egocentric video, 1920×1080. Linked to calibration via mcap.calibration_topic metadata.

/camera/calibration

foxglove.CameraCalibration

Once at t₀

Constant pinhole intrinsics: fx=fy=736.6339, cx=960, cy=540. Distortion model plumb_bob with D=[0,0,0,0,0] (ARKit output is undistorted).

/skeleton

foxglove.SceneUpdate

30 Hz

3D animated skeleton as a LineList of bone segments in the ARKit world frame. Colors follow the devkit convention: index=tomato, middle=pale turquoise, ring=green-yellow, little=light blue, thumb=violet, arm/spine=grey.

/skeleton_2d

foxglove.ImageAnnotations

30 Hz

2D skeleton projected onto the image plane using inv(cam_ext) @ joint_tf then pinhole projection with K. PointsAnnotation with LineList type; same per-finger color scheme as /skeleton.

/tf

foxglove.FrameTransform

30 Hz

Dynamic camera extrinsics (parent=world, child=camera) extracted from the ARKit transforms/camera field. All 68 body joint positions are encoded in /skeleton and /skeleton_2d rather than as individual TF frames.

/metadata

egodex_metadata (JSON)

Once at t₀

Task name, task type, language description, objects, verbs, environment string, session name, annotator version, and collector code.

3D Coordinate Frame#

All transforms originate from ARKit’s stationary origin frame set at the start of each recording session. The world frame is not globally consistent across episodes — each session initializes its own origin. For training, poses should be converted to the camera frame using inv(cam_ext) @ joint_tf before computing action representations, as done in the paper’s benchmark.

Source Annotations (in raw HDF5, accessible via hdf5_path)#

The raw HDF5 files (not stored in this FiftyOne dataset) contain:

  • transforms/<joint_name> (N×4×4 float32): SE(3) pose in ARKit world frame for all 68 joints plus the camera. Joints: 20 upper body (hip, spine1–7, neck1–4, left/right shoulder/arm/forearm/hand), 24 left finger joints, 24 right finger joints.

  • confidences/<joint_name> (N float32): ARKit tracking confidence 0–1. Wrist confidence = whole-hand detection; finger confidence = relative to wrist. Absent in ~6% of episodes.

  • camera/intrinsic (3×3 float32): Constant across all files.

  • HDF5 attributes: session_name, environment, extra (collector ID), annotator_version, llm_description, llm_description2, llm_type, which_llm_description, llm_objects, llm_verbs.

Parsing Decisions#

  • Reversible-task language: For reversible episodes, which_llm_description (‘1’ or ‘2’) determines which of llm_description/llm_description2 applies to that episode. The applicable description is stored in language; the inverse is stored in language2.

  • Session name normalization: Raw session_name attributes contained several malformed variants (double .mov extension, all-dash separators, 5-digit years, trailing characters). A regex extractor handles all observed variants; ~29 of 3,243 episodes were affected.

  • Video transcoding: Source MP4 uses MPEG-4 Part 2 codec. Each episode is transcoded to H.264 (libx264, -bf 0, -g 30, level 5.0) before MCAP authoring to satisfy foxglove.CompressedVideo requirements.

  • Timestamps: HDF5 files contain no per-frame timestamps. Frame timestamps are synthesized as session_start_ns + frame_index × (1e9/30), where session_start_ns is parsed from session_name.

Dataset Creation#

Curation Rationale#

Imitation learning for robot manipulation is bottlenecked by data scarcity. Unlike text or images, there is no Internet-scale corpus of dexterous manipulation data. EgoDex addresses this by exploiting the passive scalability of egocentric human video: data is a byproduct of performing everyday tasks while wearing an Apple Vision Pro, rather than requiring deliberate robot operation. The dataset prioritizes behavioral diversity — 194 tasks covering a wide range of manipulation behaviors beyond pick-and-place — over scene diversity. ARKit’s on-device SLAM and multi-camera calibration provide precise per-joint pose annotations without post-hoc estimation, which is critical for fine-grained dexterous manipulation data.

Source Data#

Data Collection and Processing#

All data is collected with Apple Vision Pro running visionOS 2, using ARKit for production-grade pose tracking. Data is recorded in sessions of approximately 10–15 minutes, each containing many individual episodes demarcated by pause/resume of the recording app. Raw video is compressed using modern video codecs (the raw data would otherwise exceed 500 TB). Language metadata is collected per-session by data collectors and then processed through GPT-4 to produce structured, consistent natural language descriptions.

Who are the source data producers?#

Data collectors at Apple performed demonstrations of tabletop manipulation tasks while wearing Apple Vision Pro. No further information about individual collectors is publicly available.

Annotations#

Annotation process#

3D skeletal pose annotations are produced automatically at recording time by ARKit running on-device, using multiple calibrated cameras and on-device SLAM. This is a production-grade system without manual annotation. Language annotations are produced in two stages: (1) data collectors write brief free-text task descriptions; (2) GPT-4 synthesizes collector notes (task name, environment details, object descriptions) into a single polished natural language description per episode.

Who are the annotators?#

Pose annotations are machine-generated by ARKit. Language annotations are generated by GPT-4 from collector-written metadata; the collectors are Apple employees.

Personal and Sensitive Information#

The dataset does not contain faces, voices, or other biometric identifiers beyond hand and arm motion. The Apple Vision Pro captures only tabletop scenes from a first-person viewpoint; backgrounds are controlled studio environments. No personally identifiable information is included in the released data.

Citation#

BibTeX:

@inproceedings{egodex,
  title     = {EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video},
  author    = {Ryan Hoque and Peide Huang and David J. Yoon and Mouli Sivapurapu and Jian Zhang},
  booktitle = {International Conference on Learning Representations (ICLR)},
  year      = {2026},
  url       = {https://arxiv.org/abs/2505.11709}
}

APA:

Hoque, R., Huang, P., Yoon, D. J., Sivapurapu, M., & Zhang, J. (2026). EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2505.11709

More Information#

  • Full dataset (training + test + additional data): https://github.com/apple/ml-egodex

  • FiftyOne documentation: https://docs.voxel51.com

  • MCAP format specification: https://mcap.dev

Dataset Card Authors#

Harpreet Sahota