#### NOTE
This is a **Hugging Face dataset**. For large datasets, ensure `huggingface_hub>=1.1.3` to avoid rate limits. Learn more in the <a href="https://docs.voxel51.com/integrations/huggingface.html#loading-datasets-from-the-hub" target="_blank">Hugging Face integration docs</a>.

<a href="https://huggingface.co/datasets/Voxel51/hoi-dataset-fiftyone" target="_blank">![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Dataset-yellow)</a>

# Dataset Card for Hoi! Dataset (in FiftyOne format)

![preview](https://huggingface.co/datasets/Voxel51/hoi-dataset-fiftyone/resolve/main/hoi_dataset.gif)

This dataset card covers the FiftyOne packaging of the Hoi! dataset
(Engelbracht et al., CVPR 2026). The FiftyOne version contains 88 episodes
from the full 3048-sequence release, selected to cover the available embodiment
types and scene variety in the local extract.

## Installation

```bash
pip install -U fiftyone
```

## Usage

```python
import fiftyone as fo
import fiftyone.utils.huggingface as fouh

dataset = fouh.load_from_hub(
    "Voxel51/hoi-dataset-fiftyone",
    name="hoi-dataset-fiftyone",
    persistent=True,
)


session = fo.launch_app(dataset)
```

## Dataset Details

### Dataset Description

Hoi! captures humans interacting with everyday articulated objects — drawers,
cabinet doors, fridge doors, dishwashers — in real furnished rooms. Each
interaction is recorded simultaneously under multiple embodiments and from
multiple viewpoints, with all streams time-aligned and spatially registered to
a shared Leica world frame.

The Hoi! Gripper is the instrumented embodiment: a handheld 2-finger parallel
gripper carrying a Bota SensONE 6-DoF force-torque sensor, two GelSight Digit
tactile sensors, a Dynamixel XM430-W350-T motor, a ZED Mini stereo camera, and
Project Aria glasses. The full dataset (not this FiftyOne extract) contains
3048 sequences across 381 articulated objects in 38 environments.

- **Curated by:** Tim Engelbracht, René Zurbrügg, Matteo Wohlrapp, Martin
  Büchner, Abhinav Valada, Marc Pollefeys, Hermann Blum, Zuria Bauer
- **Funded by:** ETH AI Center, ETH Foundation Project 2025-FS-352, SNSF
  Advanced Grant 216260, Lamarr Institute for Machine Learning and Artificial
  Intelligence, Robotics Institute Germany, Google, Meta
- **Shared by:** ETH Zurich, Technical University of Munich, University of
  Freiburg, Microsoft, University of Bonn
- **Language(s):** N/A (no language annotations)
- **License:** CC BY 4.0

### Dataset Sources

- **Repository:** [github.com/timengelbracht/hoi-dataset-tools](https://github.com/timengelbracht/hoi-dataset-tools)
- **Paper:** Engelbracht et al., *Hoi! — A Multimodal Dataset for Force-Grounded, Cross-View Articulated Manipulation*, CVPR 2026. [arXiv:2512.04884](https://arxiv.org/abs/2512.04884)
- **Demo:** [timengelbracht.github.io/Hoi-Dataset-Website](https://timengelbracht.github.io/Hoi-Dataset-Website/)

## Uses

### Direct Use

- **Force estimation from vision:** Each gripper episode provides per-frame
  force-torque readings (6-axis, 100 Hz) paired with RGB and depth observations,
  supporting training and evaluation of models that predict interaction forces
  from visual input.
- **Tactile force estimation:** GelSight Digit images (20 Hz) are paired with
  FT sensor ground truth, supporting tactile representation learning and
  contact-force estimation.
- **Articulated object estimation:** Posed egocentric and exocentric RGB, 3D
  Leica point clouds, and manual articulation annotations (prismatic/revolute
  type, axis) support in-the-wild articulation estimation research.
- **Cross-embodiment transfer:** The same objects are operated under three
  embodiments (gripper, hand, UMI) with spatial registration, enabling study
  of how manipulation skills transfer across morphologies.
- **Multimodal policy learning:** Paired visual and force data, with ground-truth
  trajectories in a common world frame, support learning force-aware manipulation
  policies.

### Out-of-Scope Use

The dataset was recorded in a furniture store exhibition area, a university lab,
and private apartments. It does not cover outdoor environments, non-articulated
objects, or fine-grained dexterous manipulation. The FiftyOne extract (88
episodes) covers 7 scenes; models trained on it will not generalize to the
full 38-environment release.

## Dataset Structure

### Topology

Flat multimodal dataset. Each of the 88 samples is one `.mcap` file
representing one interaction episode (a single open or close of an articulated
object). `media_type = "multimodal"`. No groups, no splits.

### Episode inventory

| Scene      | gripper   | hand   | umi   |
|------------|-----------|--------|-------|
| bathroom_1 | 8         | —      | —     |
| bedroom_1  | 16        | —      | —     |
| fridge_1   | 8         | —      | —     |
| kitchen_7  | 10        | —      | —     |
| office_1   | 14        | —      | —     |
| oven_2     | 10        | —      | —     |
| wardrobe_2 | 8         | 6      | 8     |
| **Total**  | **74**    | **6**  | **8** |

### Sample fields

| Field            | FiftyOne type            | Description                                                    |
|------------------|--------------------------|----------------------------------------------------------------|
| `filepath`       | `StringField`            | Absolute path to the `.mcap` file                              |
| `scene`          | `StringField`            | Recording location (e.g. `bathroom_1`, `wardrobe_2`)           |
| `effector`       | `StringField`            | Manipulation embodiment: `gripper`, `hand`, or `umi`           |
| `window_number`  | `IntField`               | 0-based window index within the scene/effector combination     |
| `state`          | `StringField`            | Interaction state: `open` or `close`                           |
| `duration_s`     | `FloatField`             | Episode duration in seconds (4.7–18.0 s)                       |
| `message_count`  | `IntField`               | Total MCAP messages across all channels                        |
| `channel_count`  | `IntField`               | Number of live channels (channels with at least one message)   |
| `topics`         | `ListField(StringField)` | Sorted list of topic strings present in the file               |
| `schemas`        | `ListField(StringField)` | Sorted list of schema names present in the file                |
| `has_image`      | `BooleanField`           | True if any `foxglove.CompressedImage` channel is present      |
| `has_pointcloud` | `BooleanField`           | True if any `foxglove.PointCloud` channel is present           |
| `has_force`      | `BooleanField`           | True if `hoi.ForceWrench` is present (wardrobe_2 gripper only) |
| `has_hand`       | `BooleanField`           | True if `hoi.HandTracking` is present                          |
| `has_gaze`       | `BooleanField`           | True if `hoi.EyeGaze` is present                               |
| `has_scene3d`    | `BooleanField`           | True if `foxglove.SceneUpdate` is present                      |
| `has_pose`       | `BooleanField`           | True if `foxglove.FrameTransform` is present                   |

Capability flags are derived from schema names, not topic names.

### MCAP channels per embodiment

All embodiments share:

| Topic                           | Schema                     | FiftyOne viewer tile                                                              |
|---------------------------------|----------------------------|-----------------------------------------------------------------------------------|
| `/aria_human/image`             | `foxglove.CompressedImage` | Image — Aria glasses worn by the human operator, JPEG ~30 fps                     |
| `/aria_human/pose`              | `foxglove.FrameTransform`  | 3D — dynamic `world → human_path` transform at ~1 kHz                             |
| `/aria_human/trajectory`        | `foxglove.SceneUpdate`     | 3D — human body path as a polyline in world frame                                 |
| `/aria_human/hand_landmarks_3d` | `foxglove.SceneUpdate`     | 3D — 21-sphere hand skeleton in world frame (absent in some episodes)             |
| `/aria_human/hand_tracking`     | `hoi.HandTracking`         | Message + Plot — JSON: 2D landmarks, wrist world coords, palm normals             |
| `/aria_human/eye_gaze`          | `hoi.EyeGaze`              | Message + Plot — JSON: `left_yaw_rads`, `right_yaw_rads`, `pitch_rads`, `depth_m` |
| `/leica/room_scan`              | `foxglove.PointCloud`      | 3D — RGBA colored Leica room scan in world frame (1 message, static)              |

Gripper-only additions (74 episodes):

| Topic                         | Schema                     | Notes                                  |
|-------------------------------|----------------------------|----------------------------------------|
| `/aria_gripper/image`         | `foxglove.CompressedImage` | Gripper-mounted Aria RGB, JPEG ~30 fps |
| `/aria_gripper/pose`          | `foxglove.FrameTransform`  | `world → gripper_path` at ~1 kHz       |
| `/aria_gripper/trajectory`    | `foxglove.SceneUpdate`     | Gripper path polyline in world frame   |
| `/aria_gripper/hand_tracking` | `hoi.HandTracking`         | Same schema as human                   |
| `/aria_gripper/eye_gaze`      | `hoi.EyeGaze`              | Same schema as human                   |

wardrobe_2 gripper only (8 episodes, richest):

| Topic                  | Schema                     | Notes                                                              |
|------------------------|----------------------------|--------------------------------------------------------------------|
| `/digit/left/image`    | `foxglove.CompressedImage` | DIGIT tactile sensor left finger, ~20 fps                          |
| `/digit/right/image`   | `foxglove.CompressedImage` | DIGIT tactile sensor right finger, ~20 fps                         |
| `/ft_sensor/wrench`    | `hoi.ForceWrench`          | JSON: `fx/fy/fz`, `tx/ty/tz` raw and filtered at ~100 Hz           |
| `/iphone_1/image`      | `foxglove.CompressedImage` | iPhone 1 RGB, JPEG ~30 fps                                         |
| `/iphone_1/pointcloud` | `foxglove.PointCloud`      | iPhone depth-to-point-cloud in world frame, ~30 fps                |
| `/leica/pointcloud`    | `foxglove.PointCloud`      | Leica room scan (same data as `/leica/room_scan`, alternate topic) |

UMI-only addition (8 episodes):

| Topic                     | Schema                 | Notes                                    |
|---------------------------|------------------------|------------------------------------------|
| `/umi_gripper/trajectory` | `foxglove.SceneUpdate` | UMI gripper path polyline in world frame |

### Custom JSON schemas

`hoi.ForceWrench`, `hoi.HandTracking`, and `hoi.EyeGaze` are JSON schemas
with no built-in decoder in FiftyOne 1.21.0. They appear in the Message tile
as raw JSON. Their numeric fields (`fx`, `fy`, `fz`, `left_yaw_rads`,
`depth_m`, etc.) are chartable in the Plot tile.

### Saved views

The dataset ships with 8 saved views:

| View                         |   Count | Description                                                    |
|------------------------------|---------|----------------------------------------------------------------|
| All Modalities               |       8 | wardrobe_2 gripper — every channel including force and tactile |
| Gripper                      |      74 | All gripper episodes across 7 scenes                           |
| Hand                         |       6 | Human hand only, wardrobe_2                                    |
| UMI                          |       8 | UMI gripper + trajectory, wardrobe_2                           |
| Open interactions            |      43 | Episodes where the object is opened                            |
| Close interactions           |      45 | Episodes where the object is closed                            |
| Has 3D point cloud           |      88 | All episodes (all have a Leica room scan)                      |
| Has eye gaze + hand tracking |      80 | Episodes where Plot tile fields are available                  |

### 3D viewer setup

Set the 3D tile reference frame to `world`. All geometry — point clouds,
trajectory polylines, hand skeleton spheres, and FrameTransform edges — uses
`frame_id = "world"`. FrameTransform messages are dynamic (high-rate pose
updates tracking the human and gripper), not static rig extrinsics. No
CameraCalibration messages exist, so there are no frustum overlays.

Timestamps in the MCAP files are nanoseconds elapsed from a session-relative
origin, not Unix epoch. The timeline shows elapsed seconds; duration values
are correct.

### Parsing decisions

- **One sample per episode.** The interaction splitting JSON
  (`interaction_splitting_info_{id}_confirmed.json`) defines window boundaries
  per scene. Each MCAP file corresponds to one confirmed window.
- **Capability flags from schema names.** Topic-name matching produces false
  positives (e.g. `/os_node/imu_packets` would match an IMU substring). Flags
  are set from the schema registry instead.
- **wardrobe_2 has no confirmed JSON.** Windows for wardrobe_2 were not
  recoverable from the confirmed annotation file, so no `window_start_ns` or
  `window_end_ns` fields are attached for those 22 episodes.

### Viewer limitations of this extract

- **No image projection of 3D geometry.** The FiftyOne MCAP viewer can overlay
  3D landmarks onto a 2D camera image when (1) a `CameraCalibration` message
  exists, (2) the image channel declares its calibration topic via channel
  metadata, and (3) the image topic’s last path segment contains `image_raw`,
  `image_rect`, or `image_rectified`. None of these conditions are met: no
  `CameraCalibration` messages exist in any of the 88 files, the calibration
  linkage metadata was never written, and the image topics use `/image` without
  a suffix. As a result, the 3D hand skeleton renders in the 3D tile only — it
  is not projected onto the Aria or iPhone camera images. The calibration files
  exist in the raw Bonndata release but were not packaged into the MCAP files.
- **iPhone depth absent outside wardrobe_2 gripper.** The iPhone 13 Pro has a
  LiDAR scanner, so every session with an iPhone captured depth. However, the
  depth-to-point-cloud channel (`/iphone_1/pointcloud`) was only written into
  the wardrobe_2 gripper MCAP files. The 52 other episodes that include iPhone
  RGB images have no corresponding depth channel. The raw depth data is
  available in Bonndata archives (`camera_depth.zip` per iPhone module) but was
  not packaged for those locations.
- **Force-torque and DIGIT tactile absent from 66 of 74 gripper episodes.** All
  gripper recording sessions used the Hoi! Gripper, which always carries the
  force-torque sensor and DIGIT tactile sensors. The `/ft_sensor/wrench`,
  `/digit/left/image`, and `/digit/right/image` channels were only written into
  the wardrobe_2 gripper MCAP files. The other six locations (bathroom_1,
  bedroom_1, fridge_1, kitchen_7, office_1, oven_2) have gripper RGB and Leica
  point cloud but no force or tactile channels. The raw force and tactile data
  is available in Bonndata archives for those locations.

## Dataset Creation

### Curation Rationale

Existing articulation and manipulation datasets either provide scale without
force sensing, or provide force sensing without multi-view or multi-embodiment
coverage. The dataset was designed to pair “what is seen” with “what is done
and what is felt” across human and robotic embodiments operating the same
objects, to support research that links visual perception to physical
interaction forces.

### Source Data

#### Data Collection and Processing

Seven human demonstrators operated articulated household objects under four
conditions: human hand only, human hand with wrist camera, UMI gripper, and
Hoi! Gripper. Recording sessions covered 3–11 articulated parts per session.
A dynamic QR code encoding the current Unix timestamp was displayed to all
cameras to enable temporal alignment. Each scene was scanned with a Leica
RTC360 before and after interaction, yielding 2–5 point clouds per location
that served as ground truth geometry and the shared spatial reference frame.

Time alignment was performed in post-processing using the QR-code timestamps,
achieving ~10–25 ms accuracy per stream. Spatial alignment used hloc visual
localization against the Leica scan database, estimating a single rigid
transformation per device trajectory into the common world frame. Trajectory
accuracy against Qualisys motion-capture ground truth: RMSE position ≤ 6 mm,
RMSE rotation ≤ 16 mrad.

The Hoi! Gripper’s 6-DoF force-torque sensor (Bota SensONE) operates at
100 Hz with accuracy < 2% across all axes and a noise-free resolution of
70–100 mN / 0.6–2.1 mNm at 100 Hz. Gravity compensation was applied by
estimating sensor biases during no-contact windows and subtracting the
gravitational component using the known end-effector mass and center of mass.
A Butterworth filter of order 4 was applied to the compensated signals.

The dataset was recorded primarily in the exhibition area of a furniture store,
with additional sequences in a university lab and private apartments.

#### Who are the source data producers?

Seven human demonstrators collected all interaction data. No details about
their demographics are provided in the paper.

### Annotations

#### Annotation process

Interaction boundaries were identified automatically using the QR-code timing
signal and verified manually with a lightweight annotation tool. Articulation
type (prismatic or revolute) and axis were annotated using the ArtiPoint
annotation tool. 3D object masks were generated by prompting SAM v2 on
panoramic images and lifting the predicted mask to 3D using the point cloud.

#### Who are the annotators?

[More Information Needed]

#### Personal and Sensitive Information

The dataset was collected in public and semi-public spaces. Face anonymization
was applied to the released RGB streams using EgoBlur. The MCAP files in this
FiftyOne extract contain the anonymized images.

## Citation

**BibTeX:**

```bibtex
@InProceedings{Engelbracht_2026_CVPR,
    author    = {Engelbracht, Tim and Zurbrügg, René and Wohlrapp, Matteo and
                 Büchner, Martin and Valada, Abhinav and Pollefeys, Marc and
                 Blum, Hermann and Bauer, Zuria},
    title     = {Hoi! - A Multimodal Dataset for Force-Grounded, Cross-View
                 Articulated Manipulation},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and
                 Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2026},
    pages     = {8880--8890}
}
```

**APA:**

Engelbracht, T., Zurbrügg, R., Wohlrapp, M., Büchner, M., Valada, A.,
Pollefeys, M., Blum, H., & Bauer, Z. (2026). Hoi! — A Multimodal Dataset for
Force-Grounded, Cross-View Articulated Manipulation. *CVPR 2026*, 8880–8890.

## More Information

The full Hoi! dataset (3048 sequences, 38 scenes, 381 articulated objects) is
available at [bonndata.uni-bonn.de](https://bonndata.uni-bonn.de). Raw streams
per scene — including ZED Mini stereo video, Aria depth, iPhone RGB-D, and
full calibration files — are available as zip archives from Bonndata; they are
not included in this FiftyOne extract.

The dev kit and data processing pipeline are at
[github.com/timengelbracht/hoi-dataset-tools](https://github.com/timengelbracht/hoi-dataset-tools).

## Dataset Card Authors

Harpreet Sahota