#### NOTE This is a **Hugging Face dataset**. For large datasets, ensure `huggingface_hub>=1.1.3` to avoid rate limits. Learn more in the Hugging Face integration docs. ![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Dataset-yellow) This is a subset of the HIW-500 dataset for use in FiftyOne. # Installation If you haven’t already, install FiftyOne: ```bash pip install -U fiftyone ``` # Usage ```python import fiftyone as fo import fiftyone.utils.huggingface as fouh dataset = fouh.load_from_hub("Voxel51/HIW-500-LeRobot-48ep") session = fo.launch_app(dataset) ``` # Dataset Card for HIW-500 (48-episode FiftyOne subset) ![HIW-500 preview](https://huggingface.co/datasets/Voxel51/HIW-500-LeRobot-48ep/resolve/main/hiw-500.gif) ## Dataset Details ### Dataset Description This is a 48-episode subset of [BitRobot/HIW-500-LeRobot](https://huggingface.co/datasets/BitRobot/HIW-500-LeRobot), the LeRobot v3.0 conversion of **HIW-500: Humanoids In-the-Wild Dataset**. HIW-500 captures human whole-body teleoperation of a Unitree G1 humanoid performing household tasks in real homes in Southeast Asia, where layout, object state, lighting, clutter, and operator style vary from episode to episode. The full dataset targets research on mobile manipulation, bimanual interaction, long-horizon household skills, and imitation learning from in-the-wild demonstrations. The full source dataset totals 500+ hours of demonstrations, 23,743 episodes, ~10 TB of data (2.15 TB in the HF parquet/video re-encoding), 12 real homes, 11 household task categories, 161 fine-grained subtask labels, and 148K+ subtask annotations. This subset samples 48 episodes chosen to cover all 11 task categories, converted to a self-contained LeRobot v3.0 export via the FiftyOne LeRobot exporter. - **Curated by:** Harpreet Sahota (subset curation and FiftyOne conversion); original data collected by BitRobot, Unitree, and Hugging Face - **Funded by:** BitRobot Foundation, Unitree, Hugging Face - **Shared by:** BitRobot Foundation (original), this repo (FiftyOne subset) - **Language(s):** English (language annotations, not imported into this subset — see [Parsing decisions](#parsing-decisions)) - **License:** cc-by-4.0 ### Dataset Sources - **Repository:** [BitRobot/HIW-500-LeRobot](https://huggingface.co/datasets/BitRobot/HIW-500-LeRobot) (LeRobot v3.0 format); [BitRobot/HIW-500](https://huggingface.co/datasets/BitRobot/HIW-500) (raw ROS bag / MCAP) - **Project page:** [HIW-500: Humanoids In-the-Wild Dataset](https://bitrobot-foundation.github.io/humanoids-in-the-wild-500-hours/) ## Uses ### Direct Use Exploring and visualizing whole-body humanoid teleoperation episodes in FiftyOne — browsing head/wrist camera streams alongside joint state and action traces, filtering by task, and inspecting episode length/duration distributions. Suitable for prototyping imitation-learning data pipelines, sanity-checking a slice of HIW-500 before downloading the full 2+ TB dataset, or building qualitative demos of mobile manipulation and bimanual household tasks. ### Out-of-Scope Use Not suitable for training production imitation-learning policies (48 episodes is a small, non-uniform sample of an 11-task, 23,743-episode dataset — see class counts in [Fields](#fields)). Not suitable for any use requiring the fine-grained subtask language annotations (`language_persistent`, `language_events`) or camera intrinsics/extrinsics, since neither is present in this subset (see [Parsing decisions](#parsing-decisions)). ## Dataset Structure This subset has 48 samples, one per episode, media type `multimodal`. Each sample references three synchronized video streams (head + two wrist cameras) plus per-frame robot state/action arrays stored in the referenced Parquet shard; these are not separate FiftyOne fields but are exposed through the sample’s `media_reference` in the FiftyOne App’s Streams / State & Action tabs. ### Fields | Field | Type | Description | |-------------------|--------------------------|--------------------------------------------------------------------------------------------------------------------| | `media_reference` | `MediaReferenceField` | Points to the episode’s video streams and Parquet data (frame-level state/action, not expanded into sample fields) | | `episode_index` | `IntField` | Episode index in this subset’s re-exported numbering (`0..47`) | | `task` | `StringField` | Primary task label for the episode (free text, e.g. `"restocking fridge"`) | | `tasks` | `ListField(StringField)` | All coarse task labels associated with the episode (usually length 1) | | `length` | `IntField` | Number of frames in the episode | | `duration` | `FloatField` | Episode duration in seconds (`length / fps`) | | `robot_type` | `StringField` | Robot platform, always `"unitree_g1"` in this subset | | `fps` | `FloatField` | Frame rate, `30.0` | Task distribution across the 48 episodes: | Task | Episodes | |----------------------------------------|------------| | setting the table | 15 | | clothes washing | 6 | | hang keys on a hook | 5 | | move the pillow to the sofa from floor | 4 | | picking trash to rubbish bin | 4 | | sweep floor | 4 | | kitchen organization | 3 | | restocking fridge | 2 | | building children table | 2 | | clean up the room | 2 | | hang hanger | 1 | Per-frame features referenced by `media_reference` (from source `meta/info.json`), not expanded into FiftyOne sample fields: | Feature | Dtype | Shape | Notes | |--------------------------------------------------------------------|-----------------|------------------|----------------------------------------------------------------------------------------------| | `observation.images.head` | video | `[480, 1280, 3]` | AV1, yuv420p, 30 fps | | `observation.images.left_wrist` | video | `[480, 640, 3]` | AV1, yuv420p, 30 fps | | `observation.images.right_wrist` | video | `[480, 640, 3]` | AV1, yuv420p, 30 fps | | `observation.state` | float32 | `[29]` | 29-DoF joint positions (hips, knees, ankles, waist, shoulders, elbows, wrists) | | `observation.state.wbc` | float32 | `[23]` | Whole-body-control state: pivot velocity/pose, left/right end-effector pose, trigger/squeeze | | `action` | float32 | `[23]` | Action trace, same 23-dim layout as `observation.state.wbc` | | `timestamp`, `frame_index`, `episode_index`, `index`, `task_index` | float32 / int64 | `[1]` | Standard LeRobot bookkeeping columns | ### Label types and why `task` is a `StringField`, not a `Classification` — the source stores it as free text per episode (one coarse task label out of 11), and the FiftyOne LeRobot importer surfaces it as-is rather than coercing it to a label type. `tasks` preserves the full source list in case an episode carries more than one coarse label (none do in this subset). ### `dataset.info` contents `dataset.info["lerobot"]` carries the LeRobot import summary, including `skipped_episodes: []` (all 48 requested episodes imported successfully) and the source `codebase_version: "v3.0"`. ### Parsing decisions - **Episode selection:** the source repo packs many episodes per Parquet/video shard file. The cheapest single-episode “shard-0” set (all streams at `file_index == 0`) contained only 1 episode (episode 0, 17,265 frames), which would have skewed heavily toward one task. Instead, 6 shard groups were selected across the episode-index range to cover **all 11 task categories** with fewer, denser downloads: episodes `2611-2618` (hang hanger / hang keys on a hook / kitchen organization), `14266-14274` (kitchen organization / move pillow / picking trash), `12471-12481` (restocking fridge / setting the table / sweep floor), `20027-20038` (clothes washing / setting the table), `2197-2202` (clean up the room / clothes washing), and `23220-23221` (building children table) — 48 episodes total, 24 shard files, 3.78 GB downloaded (of the source’s ~2 TB). - **Re-export:** pushing to the Hub re-indexes episodes to `0..47`, remaps `task_index` to only the tasks present in this subset, and recomputes per-episode and global stats — these no longer match the source dataset’s global stats or original episode numbering. - **Excluded modalities:** `language_persistent` and `language_events` (both declared `dtype: "language"` in source `info.json`) are **not imported**. FiftyOne’s LeRobot importer only recognizes `video` and `image` dtypes; these fields carried the dataset’s 161 fine-grained subtask labels and 148K+ subtask annotations (per-turn dicts with `role`, `content`, `style`, `timestamp`), which are lost in this subset. Only the coarse `task`/`tasks` fields (11 categories) survive. - **Excluded metadata:** camera intrinsics and extrinsics (mentioned in the source README’s metadata section) are not exposed as FiftyOne fields and are not carried into this subset’s export. - **`tasks.parquet` repair:** the source `meta/tasks.parquet` stores the task string as the pandas index (column `__index_level_0__`) rather than a literal `task` column. FiftyOne’s importer tolerates this, but the Hub-push exporter requires literal `task_index`/`task` columns, so this file was rewritten with explicit columns before export (values unchanged). - **Codec:** all three video streams are AV1/yuv420p, which decodes fine in Chromium-based browsers (the FiftyOne App). ## Dataset Creation ### Curation Rationale Full HIW-500-LeRobot is ~2 TB, too large to download and explore casually. This subset gives a representative, browsable sample — one or more episodes per task category — for exploring the dataset’s structure, camera views, and state/action layout in FiftyOne before committing to the full download. ### Source Data #### Data Collection and Processing Per the source project page and README: data was collected via human whole-body teleoperation of Unitree G1 robots in real homes across Southeast Asia. Each episode records synchronized head and wrist camera streams (RGB, stereo IR on the wrists), 29-DoF joint state, whole-body-control end-effector state, and action traces, at 30 fps. Task and fine-grained subtask boundaries were annotated per episode; only the coarse `task` label survives in this subset (see [Parsing decisions](#parsing-decisions)). #### Who are the source data producers? Human operators performing whole-body teleoperation of Unitree G1 robots in participating homes, organized by BitRobot, Unitree, and Hugging Face. ### Annotations #### Personal and Sensitive Information Episodes were recorded in real homes and may contain identifiable personal environments (room layouts, personal belongings). No information is provided by the source about anonymization or consent procedures for the physical spaces recorded. ## Citation **BibTeX:** ```bibtex @misc{hiw500_2026, title={HIW-500: Humanoids In-the-Wild Dataset for Robot Learning}, author={BitRobot and Unitree and Hugging Face}, year={2026}, howpublished={\url{https://bitrobot-foundation.github.io/humanoids-in-the-wild-500-hours/}} } ``` **APA:** BitRobot, Unitree, and Hugging Face. (2026). *HIW-500: Humanoids In-the-Wild Dataset for Robot Learning*. https://bitrobot-foundation.github.io/humanoids-in-the-wild-500-hours/ ## More Information For the full dataset (23,743 episodes, ~2 TB in LeRobot format), see [BitRobot/HIW-500-LeRobot](https://huggingface.co/datasets/BitRobot/HIW-500-LeRobot). For raw ROS bag / MCAP recordings, see [BitRobot/HIW-500](https://huggingface.co/datasets/BitRobot/HIW-500). For commercial licensing or custom data collection, contact BitRobot via the [project page](https://bitrobot-foundation.github.io/humanoids-in-the-wild-500-hours/). ## Dataset Card Authors [Harpreet Sahota](https://huggingface.co/harpreetsahota) ## Dataset Card Contact [Harpreet Sahota](https://huggingface.co/harpreetsahota)