Note

This is a Hugging Face dataset. For large datasets, ensure huggingface_hub>=1.1.3 to avoid rate limits. Learn more in the Hugging Face integration docs.

This is a subset of the HIW-500 dataset for use in FiftyOne.

Installation#

If you haven’t already, install FiftyOne:

pip install -U fiftyone

Usage#

import fiftyone as fo
import fiftyone.utils.huggingface as fouh

dataset = fouh.load_from_hub("Voxel51/HIW-500-LeRobot-48ep")
session = fo.launch_app(dataset)

Dataset Card for HIW-500 (48-episode FiftyOne subset)#

HIW-500 preview

Dataset Details#

Dataset Description#

This is a 48-episode subset of BitRobot/HIW-500-LeRobot, the LeRobot v3.0 conversion of HIW-500: Humanoids In-the-Wild Dataset. HIW-500 captures human whole-body teleoperation of a Unitree G1 humanoid performing household tasks in real homes in Southeast Asia, where layout, object state, lighting, clutter, and operator style vary from episode to episode. The full dataset targets research on mobile manipulation, bimanual interaction, long-horizon household skills, and imitation learning from in-the-wild demonstrations.

The full source dataset totals 500+ hours of demonstrations, 23,743 episodes, ~10 TB of data (2.15 TB in the HF parquet/video re-encoding), 12 real homes, 11 household task categories, 161 fine-grained subtask labels, and 148K+ subtask annotations. This subset samples 48 episodes chosen to cover all 11 task categories, converted to a self-contained LeRobot v3.0 export via the FiftyOne LeRobot exporter.

Curated by: Harpreet Sahota (subset curation and FiftyOne conversion); original data collected by BitRobot, Unitree, and Hugging Face
Funded by: BitRobot Foundation, Unitree, Hugging Face
Shared by: BitRobot Foundation (original), this repo (FiftyOne subset)
Language(s): English (language annotations, not imported into this subset — see Parsing decisions)
License: cc-by-4.0

Dataset Sources#

Repository: BitRobot/HIW-500-LeRobot (LeRobot v3.0 format); BitRobot/HIW-500 (raw ROS bag / MCAP)
Project page: HIW-500: Humanoids In-the-Wild Dataset

Uses#

Direct Use#

Exploring and visualizing whole-body humanoid teleoperation episodes in FiftyOne — browsing head/wrist camera streams alongside joint state and action traces, filtering by task, and inspecting episode length/duration distributions. Suitable for prototyping imitation-learning data pipelines, sanity-checking a slice of HIW-500 before downloading the full 2+ TB dataset, or building qualitative demos of mobile manipulation and bimanual household tasks.

Out-of-Scope Use#

Not suitable for training production imitation-learning policies (48 episodes is a small, non-uniform sample of an 11-task, 23,743-episode dataset — see class counts in Fields). Not suitable for any use requiring the fine-grained subtask language annotations (language_persistent, language_events) or camera intrinsics/extrinsics, since neither is present in this subset (see Parsing decisions).

Dataset Structure#

This subset has 48 samples, one per episode, media type multimodal. Each sample references three synchronized video streams (head + two wrist cameras) plus per-frame robot state/action arrays stored in the referenced Parquet shard; these are not separate FiftyOne fields but are exposed through the sample’s media_reference in the FiftyOne App’s Streams / State & Action tabs.

Fields#

Field	Type	Description
`media_reference`	`MediaReferenceField`	Points to the episode’s video streams and Parquet data (frame-level state/action, not expanded into sample fields)
`episode_index`	`IntField`	Episode index in this subset’s re-exported numbering (`0..47`)
`task`	`StringField`	Primary task label for the episode (free text, e.g. `"restocking fridge"`)
`tasks`	`ListField(StringField)`	All coarse task labels associated with the episode (usually length 1)
`length`	`IntField`	Number of frames in the episode
`duration`	`FloatField`	Episode duration in seconds (`length / fps`)
`robot_type`	`StringField`	Robot platform, always `"unitree_g1"` in this subset
`fps`	`FloatField`	Frame rate, `30.0`

Task distribution across the 48 episodes:

Task	Episodes
setting the table	15
clothes washing	6
hang keys on a hook	5
move the pillow to the sofa from floor	4
picking trash to rubbish bin	4
sweep floor	4
kitchen organization	3
restocking fridge	2
building children table	2
clean up the room	2
hang hanger	1

Per-frame features referenced by media_reference (from source meta/info.json), not expanded into FiftyOne sample fields:

Feature	Dtype	Shape	Notes
`observation.images.head`	video	`[480, 1280, 3]`	AV1, yuv420p, 30 fps
`observation.images.left_wrist`	video	`[480, 640, 3]`	AV1, yuv420p, 30 fps
`observation.images.right_wrist`	video	`[480, 640, 3]`	AV1, yuv420p, 30 fps
`observation.state`	float32	`[29]`	29-DoF joint positions (hips, knees, ankles, waist, shoulders, elbows, wrists)
`observation.state.wbc`	float32	`[23]`	Whole-body-control state: pivot velocity/pose, left/right end-effector pose, trigger/squeeze
`action`	float32	`[23]`	Action trace, same 23-dim layout as `observation.state.wbc`
`timestamp`, `frame_index`, `episode_index`, `index`, `task_index`	float32 / int64	`[1]`	Standard LeRobot bookkeeping columns

Label types and why#

task is a StringField, not a Classification — the source stores it as free text per episode (one coarse task label out of 11), and the FiftyOne LeRobot importer surfaces it as-is rather than coercing it to a label type. tasks preserves the full source list in case an episode carries more than one coarse label (none do in this subset).

`dataset.info` contents#

dataset.info["lerobot"] carries the LeRobot import summary, including skipped_episodes: [] (all 48 requested episodes imported successfully) and the source codebase_version: "v3.0".

Parsing decisions#

Episode selection: the source repo packs many episodes per Parquet/video shard file. The cheapest single-episode “shard-0” set (all streams at file_index == 0) contained only 1 episode (episode 0, 17,265 frames), which would have skewed heavily toward one task. Instead, 6 shard groups were selected across the episode-index range to cover all 11 task categories with fewer, denser downloads: episodes 2611-2618 (hang hanger / hang keys on a hook / kitchen organization), 14266-14274 (kitchen organization / move pillow / picking trash), 12471-12481 (restocking fridge / setting the table / sweep floor), 20027-20038 (clothes washing / setting the table), 2197-2202 (clean up the room / clothes washing), and 23220-23221 (building children table) — 48 episodes total, 24 shard files, 3.78 GB downloaded (of the source’s ~2 TB).
Re-export: pushing to the Hub re-indexes episodes to 0..47, remaps task_index to only the tasks present in this subset, and recomputes per-episode and global stats — these no longer match the source dataset’s global stats or original episode numbering.
Excluded modalities: language_persistent and language_events (both declared dtype: "language" in source info.json) are not imported. FiftyOne’s LeRobot importer only recognizes video and image dtypes; these fields carried the dataset’s 161 fine-grained subtask labels and 148K+ subtask annotations (per-turn dicts with role, content, style, timestamp), which are lost in this subset. Only the coarse task/tasks fields (11 categories) survive.
Excluded metadata: camera intrinsics and extrinsics (mentioned in the source README’s metadata section) are not exposed as FiftyOne fields and are not carried into this subset’s export.
tasks.parquet repair: the source meta/tasks.parquet stores the task string as the pandas index (column __index_level_0__) rather than a literal task column. FiftyOne’s importer tolerates this, but the Hub-push exporter requires literal task_index/task columns, so this file was rewritten with explicit columns before export (values unchanged).
Codec: all three video streams are AV1/yuv420p, which decodes fine in Chromium-based browsers (the FiftyOne App).

Dataset Creation#

Curation Rationale#

Full HIW-500-LeRobot is ~2 TB, too large to download and explore casually. This subset gives a representative, browsable sample — one or more episodes per task category — for exploring the dataset’s structure, camera views, and state/action layout in FiftyOne before committing to the full download.

Source Data#

Data Collection and Processing#

Per the source project page and README: data was collected via human whole-body teleoperation of Unitree G1 robots in real homes across Southeast Asia. Each episode records synchronized head and wrist camera streams (RGB, stereo IR on the wrists), 29-DoF joint state, whole-body-control end-effector state, and action traces, at 30 fps. Task and fine-grained subtask boundaries were annotated per episode; only the coarse task label survives in this subset (see Parsing decisions).

Who are the source data producers?#

Human operators performing whole-body teleoperation of Unitree G1 robots in participating homes, organized by BitRobot, Unitree, and Hugging Face.

Annotations#

Personal and Sensitive Information#

Episodes were recorded in real homes and may contain identifiable personal environments (room layouts, personal belongings). No information is provided by the source about anonymization or consent procedures for the physical spaces recorded.

Citation#

BibTeX:

@misc{hiw500_2026,
  title={HIW-500: Humanoids In-the-Wild Dataset for Robot Learning},
  author={BitRobot and Unitree and Hugging Face},
  year={2026},
  howpublished={\url{https://bitrobot-foundation.github.io/humanoids-in-the-wild-500-hours/}}
}

APA:

BitRobot, Unitree, and Hugging Face. (2026). HIW-500: Humanoids In-the-Wild Dataset for Robot Learning. https://bitrobot-foundation.github.io/humanoids-in-the-wild-500-hours/

More Information#

For the full dataset (23,743 episodes, ~2 TB in LeRobot format), see BitRobot/HIW-500-LeRobot. For raw ROS bag / MCAP recordings, see BitRobot/HIW-500. For commercial licensing or custom data collection, contact BitRobot via the project page.

Dataset Card Authors#

Harpreet Sahota

Dataset Card Contact#