Note

This is a Hugging Face dataset. For large datasets, ensure huggingface_hub>=1.1.3 to avoid rate limits. Learn more in the Hugging Face integration docs.

Dataset Card for Hoi! Dataset (in FiftyOne format)#

preview

This dataset card covers the FiftyOne packaging of the Hoi! dataset (Engelbracht et al., CVPR 2026). The FiftyOne version contains 88 episodes from the full 3048-sequence release, selected to cover the available embodiment types and scene variety in the local extract.

Installation#

pip install -U fiftyone

Usage#

import fiftyone as fo
import fiftyone.utils.huggingface as fouh

dataset = fouh.load_from_hub(
    "Voxel51/hoi-dataset-fiftyone",
    name="hoi-dataset-fiftyone",
    persistent=True,
)


session = fo.launch_app(dataset)

Dataset Details#

Dataset Description#

Hoi! captures humans interacting with everyday articulated objects — drawers, cabinet doors, fridge doors, dishwashers — in real furnished rooms. Each interaction is recorded simultaneously under multiple embodiments and from multiple viewpoints, with all streams time-aligned and spatially registered to a shared Leica world frame.

The Hoi! Gripper is the instrumented embodiment: a handheld 2-finger parallel gripper carrying a Bota SensONE 6-DoF force-torque sensor, two GelSight Digit tactile sensors, a Dynamixel XM430-W350-T motor, a ZED Mini stereo camera, and Project Aria glasses. The full dataset (not this FiftyOne extract) contains 3048 sequences across 381 articulated objects in 38 environments.

Curated by: Tim Engelbracht, René Zurbrügg, Matteo Wohlrapp, Martin Büchner, Abhinav Valada, Marc Pollefeys, Hermann Blum, Zuria Bauer
Funded by: ETH AI Center, ETH Foundation Project 2025-FS-352, SNSF Advanced Grant 216260, Lamarr Institute for Machine Learning and Artificial Intelligence, Robotics Institute Germany, Google, Meta
Shared by: ETH Zurich, Technical University of Munich, University of Freiburg, Microsoft, University of Bonn
Language(s): N/A (no language annotations)
License: CC BY 4.0

Dataset Sources#

Repository: github.com/timengelbracht/hoi-dataset-tools
Paper: Engelbracht et al., Hoi! — A Multimodal Dataset for Force-Grounded, Cross-View Articulated Manipulation, CVPR 2026. arXiv:2512.04884
Demo: timengelbracht.github.io/Hoi-Dataset-Website

Uses#

Direct Use#

Force estimation from vision: Each gripper episode provides per-frame force-torque readings (6-axis, 100 Hz) paired with RGB and depth observations, supporting training and evaluation of models that predict interaction forces from visual input.
Tactile force estimation: GelSight Digit images (20 Hz) are paired with FT sensor ground truth, supporting tactile representation learning and contact-force estimation.
Articulated object estimation: Posed egocentric and exocentric RGB, 3D Leica point clouds, and manual articulation annotations (prismatic/revolute type, axis) support in-the-wild articulation estimation research.
Cross-embodiment transfer: The same objects are operated under three embodiments (gripper, hand, UMI) with spatial registration, enabling study of how manipulation skills transfer across morphologies.
Multimodal policy learning: Paired visual and force data, with ground-truth trajectories in a common world frame, support learning force-aware manipulation policies.

Out-of-Scope Use#

The dataset was recorded in a furniture store exhibition area, a university lab, and private apartments. It does not cover outdoor environments, non-articulated objects, or fine-grained dexterous manipulation. The FiftyOne extract (88 episodes) covers 7 scenes; models trained on it will not generalize to the full 38-environment release.

Dataset Structure#

Topology#

Flat multimodal dataset. Each of the 88 samples is one .mcap file representing one interaction episode (a single open or close of an articulated object). media_type = "multimodal". No groups, no splits.

Episode inventory#

Scene	gripper	hand	umi
bathroom_1	8	—	—
bedroom_1	16	—	—
fridge_1	8	—	—
kitchen_7	10	—	—
office_1	14	—	—
oven_2	10	—	—
wardrobe_2	8	6	8
Total	74	6	8

Sample fields#

Field	FiftyOne type	Description
`filepath`	`StringField`	Absolute path to the `.mcap` file
`scene`	`StringField`	Recording location (e.g. `bathroom_1`, `wardrobe_2`)
`effector`	`StringField`	Manipulation embodiment: `gripper`, `hand`, or `umi`
`window_number`	`IntField`	0-based window index within the scene/effector combination
`state`	`StringField`	Interaction state: `open` or `close`
`duration_s`	`FloatField`	Episode duration in seconds (4.7–18.0 s)
`message_count`	`IntField`	Total MCAP messages across all channels
`channel_count`	`IntField`	Number of live channels (channels with at least one message)
`topics`	`ListField(StringField)`	Sorted list of topic strings present in the file
`schemas`	`ListField(StringField)`	Sorted list of schema names present in the file
`has_image`	`BooleanField`	True if any `foxglove.CompressedImage` channel is present
`has_pointcloud`	`BooleanField`	True if any `foxglove.PointCloud` channel is present
`has_force`	`BooleanField`	True if `hoi.ForceWrench` is present (wardrobe_2 gripper only)
`has_hand`	`BooleanField`	True if `hoi.HandTracking` is present
`has_gaze`	`BooleanField`	True if `hoi.EyeGaze` is present
`has_scene3d`	`BooleanField`	True if `foxglove.SceneUpdate` is present
`has_pose`	`BooleanField`	True if `foxglove.FrameTransform` is present

Capability flags are derived from schema names, not topic names.

MCAP channels per embodiment#

All embodiments share:

Topic	Schema	FiftyOne viewer tile
`/aria_human/image`	`foxglove.CompressedImage`	Image — Aria glasses worn by the human operator, JPEG ~30 fps
`/aria_human/pose`	`foxglove.FrameTransform`	3D — dynamic `world → human_path` transform at ~1 kHz
`/aria_human/trajectory`	`foxglove.SceneUpdate`	3D — human body path as a polyline in world frame
`/aria_human/hand_landmarks_3d`	`foxglove.SceneUpdate`	3D — 21-sphere hand skeleton in world frame (absent in some episodes)
`/aria_human/hand_tracking`	`hoi.HandTracking`	Message + Plot — JSON: 2D landmarks, wrist world coords, palm normals
`/aria_human/eye_gaze`	`hoi.EyeGaze`	Message + Plot — JSON: `left_yaw_rads`, `right_yaw_rads`, `pitch_rads`, `depth_m`
`/leica/room_scan`	`foxglove.PointCloud`	3D — RGBA colored Leica room scan in world frame (1 message, static)

Gripper-only additions (74 episodes):

Topic	Schema	Notes
`/aria_gripper/image`	`foxglove.CompressedImage`	Gripper-mounted Aria RGB, JPEG ~30 fps
`/aria_gripper/pose`	`foxglove.FrameTransform`	`world → gripper_path` at ~1 kHz
`/aria_gripper/trajectory`	`foxglove.SceneUpdate`	Gripper path polyline in world frame
`/aria_gripper/hand_tracking`	`hoi.HandTracking`	Same schema as human
`/aria_gripper/eye_gaze`	`hoi.EyeGaze`	Same schema as human

wardrobe_2 gripper only (8 episodes, richest):

Topic	Schema	Notes
`/digit/left/image`	`foxglove.CompressedImage`	DIGIT tactile sensor left finger, ~20 fps
`/digit/right/image`	`foxglove.CompressedImage`	DIGIT tactile sensor right finger, ~20 fps
`/ft_sensor/wrench`	`hoi.ForceWrench`	JSON: `fx/fy/fz`, `tx/ty/tz` raw and filtered at ~100 Hz
`/iphone_1/image`	`foxglove.CompressedImage`	iPhone 1 RGB, JPEG ~30 fps
`/iphone_1/pointcloud`	`foxglove.PointCloud`	iPhone depth-to-point-cloud in world frame, ~30 fps
`/leica/pointcloud`	`foxglove.PointCloud`	Leica room scan (same data as `/leica/room_scan`, alternate topic)

UMI-only addition (8 episodes):

Topic	Schema	Notes
`/umi_gripper/trajectory`	`foxglove.SceneUpdate`	UMI gripper path polyline in world frame

Custom JSON schemas#

hoi.ForceWrench, hoi.HandTracking, and hoi.EyeGaze are JSON schemas with no built-in decoder in FiftyOne 1.21.0. They appear in the Message tile as raw JSON. Their numeric fields (fx, fy, fz, left_yaw_rads, depth_m, etc.) are chartable in the Plot tile.

Saved views#

The dataset ships with 8 saved views:

View	Count	Description
All Modalities	8	wardrobe_2 gripper — every channel including force and tactile
Gripper	74	All gripper episodes across 7 scenes
Hand	6	Human hand only, wardrobe_2
UMI	8	UMI gripper + trajectory, wardrobe_2
Open interactions	43	Episodes where the object is opened
Close interactions	45	Episodes where the object is closed
Has 3D point cloud	88	All episodes (all have a Leica room scan)
Has eye gaze + hand tracking	80	Episodes where Plot tile fields are available

3D viewer setup#

Set the 3D tile reference frame to world. All geometry — point clouds, trajectory polylines, hand skeleton spheres, and FrameTransform edges — uses frame_id = "world". FrameTransform messages are dynamic (high-rate pose updates tracking the human and gripper), not static rig extrinsics. No CameraCalibration messages exist, so there are no frustum overlays.

Timestamps in the MCAP files are nanoseconds elapsed from a session-relative origin, not Unix epoch. The timeline shows elapsed seconds; duration values are correct.

Parsing decisions#

One sample per episode. The interaction splitting JSON (interaction_splitting_info_{id}_confirmed.json) defines window boundaries per scene. Each MCAP file corresponds to one confirmed window.
Capability flags from schema names. Topic-name matching produces false positives (e.g. /os_node/imu_packets would match an IMU substring). Flags are set from the schema registry instead.
wardrobe_2 has no confirmed JSON. Windows for wardrobe_2 were not recoverable from the confirmed annotation file, so no window_start_ns or window_end_ns fields are attached for those 22 episodes.

Viewer limitations of this extract#

No image projection of 3D geometry. The FiftyOne MCAP viewer can overlay 3D landmarks onto a 2D camera image when (1) a CameraCalibration message exists, (2) the image channel declares its calibration topic via channel metadata, and (3) the image topic’s last path segment contains image_raw, image_rect, or image_rectified. None of these conditions are met: no CameraCalibration messages exist in any of the 88 files, the calibration linkage metadata was never written, and the image topics use /image without a suffix. As a result, the 3D hand skeleton renders in the 3D tile only — it is not projected onto the Aria or iPhone camera images. The calibration files exist in the raw Bonndata release but were not packaged into the MCAP files.
iPhone depth absent outside wardrobe_2 gripper. The iPhone 13 Pro has a LiDAR scanner, so every session with an iPhone captured depth. However, the depth-to-point-cloud channel (/iphone_1/pointcloud) was only written into the wardrobe_2 gripper MCAP files. The 52 other episodes that include iPhone RGB images have no corresponding depth channel. The raw depth data is available in Bonndata archives (camera_depth.zip per iPhone module) but was not packaged for those locations.
Force-torque and DIGIT tactile absent from 66 of 74 gripper episodes. All gripper recording sessions used the Hoi! Gripper, which always carries the force-torque sensor and DIGIT tactile sensors. The /ft_sensor/wrench, /digit/left/image, and /digit/right/image channels were only written into the wardrobe_2 gripper MCAP files. The other six locations (bathroom_1, bedroom_1, fridge_1, kitchen_7, office_1, oven_2) have gripper RGB and Leica point cloud but no force or tactile channels. The raw force and tactile data is available in Bonndata archives for those locations.

Dataset Creation#

Curation Rationale#

Existing articulation and manipulation datasets either provide scale without force sensing, or provide force sensing without multi-view or multi-embodiment coverage. The dataset was designed to pair “what is seen” with “what is done and what is felt” across human and robotic embodiments operating the same objects, to support research that links visual perception to physical interaction forces.

Source Data#

Data Collection and Processing#

Seven human demonstrators operated articulated household objects under four conditions: human hand only, human hand with wrist camera, UMI gripper, and Hoi! Gripper. Recording sessions covered 3–11 articulated parts per session. A dynamic QR code encoding the current Unix timestamp was displayed to all cameras to enable temporal alignment. Each scene was scanned with a Leica RTC360 before and after interaction, yielding 2–5 point clouds per location that served as ground truth geometry and the shared spatial reference frame.

Time alignment was performed in post-processing using the QR-code timestamps, achieving ~10–25 ms accuracy per stream. Spatial alignment used hloc visual localization against the Leica scan database, estimating a single rigid transformation per device trajectory into the common world frame. Trajectory accuracy against Qualisys motion-capture ground truth: RMSE position ≤ 6 mm, RMSE rotation ≤ 16 mrad.

The Hoi! Gripper’s 6-DoF force-torque sensor (Bota SensONE) operates at 100 Hz with accuracy < 2% across all axes and a noise-free resolution of 70–100 mN / 0.6–2.1 mNm at 100 Hz. Gravity compensation was applied by estimating sensor biases during no-contact windows and subtracting the gravitational component using the known end-effector mass and center of mass. A Butterworth filter of order 4 was applied to the compensated signals.

The dataset was recorded primarily in the exhibition area of a furniture store, with additional sequences in a university lab and private apartments.

Who are the source data producers?#

Seven human demonstrators collected all interaction data. No details about their demographics are provided in the paper.

Annotations#

Annotation process#

Interaction boundaries were identified automatically using the QR-code timing signal and verified manually with a lightweight annotation tool. Articulation type (prismatic or revolute) and axis were annotated using the ArtiPoint annotation tool. 3D object masks were generated by prompting SAM v2 on panoramic images and lifting the predicted mask to 3D using the point cloud.

Who are the annotators?#

[More Information Needed]

Personal and Sensitive Information#

The dataset was collected in public and semi-public spaces. Face anonymization was applied to the released RGB streams using EgoBlur. The MCAP files in this FiftyOne extract contain the anonymized images.

Citation#

BibTeX:

@InProceedings{Engelbracht_2026_CVPR,
    author    = {Engelbracht, Tim and Zurbrügg, René and Wohlrapp, Matteo and
                 Büchner, Martin and Valada, Abhinav and Pollefeys, Marc and
                 Blum, Hermann and Bauer, Zuria},
    title     = {Hoi! - A Multimodal Dataset for Force-Grounded, Cross-View
                 Articulated Manipulation},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and
                 Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2026},
    pages     = {8880--8890}
}

APA:

Engelbracht, T., Zurbrügg, R., Wohlrapp, M., Büchner, M., Valada, A., Pollefeys, M., Blum, H., & Bauer, Z. (2026). Hoi! — A Multimodal Dataset for Force-Grounded, Cross-View Articulated Manipulation. CVPR 2026, 8880–8890.

More Information#

The full Hoi! dataset (3048 sequences, 38 scenes, 381 articulated objects) is available at bonndata.uni-bonn.de. Raw streams per scene — including ZED Mini stereo video, Aria depth, iPhone RGB-D, and full calibration files — are available as zip archives from Bonndata; they are not included in this FiftyOne extract.

The dev kit and data processing pipeline are at github.com/timengelbracht/hoi-dataset-tools.

Dataset Card Authors#

Harpreet Sahota