#### NOTE
This is a **Hugging Face dataset**. For large datasets, ensure `huggingface_hub>=1.1.3` to avoid rate limits. Learn more in the Hugging Face integration docs.
# Dataset Card for HapTile
This is a [FiftyOne](https://github.com/voxel51/fiftyone) dataset with 1699 samples. Each sample is one teleoperated manipulation episode, stored as a native multimodal MCAP episode.
The source is [HapTile](https://huggingface.co/datasets/HapTile2026/HapTile), a haptic-informed vision-tactile-language-action dataset. A UR5e arm with a Robotiq 2F-85 gripper works through contact-rich tabletop tasks under teleoperation. Each gripper finger carries a vision-based tactile sensor that films a gel pad printed with a marker grid, so the contact shows as the markers moving. Two RGB-D cameras watch the scene, one facing the table and one on the wrist, and the haptic feedback the operator felt through the teleoperation rig is recorded alongside the robot state.
## Installation
If you haven’t already, install FiftyOne:
```bash
pip install -U fiftyone
```
## Usage
```python
import fiftyone as fo
import fiftyone.utils.huggingface as fouh
# Load the dataset
# Note: other available arguments include 'max_samples', etc
dataset = fouh.load_from_hub(
"Voxel51/HapTile",
name="HapTile",
persistent=True,
)
# Launch the App
session = fo.launch_app(dataset)
```
## Dataset Details
### Dataset Description
1,699 episodes across 38 tasks, totalling 664,045 frames and 12.36 hours of recording. In 1,592 episodes the operator felt haptic feedback at some point, at the levels `firm`, `mild` and `soft`.
- **Curated by:** The HapTile authors: Amirhosein Alian, Yongqiang Zhao, Shiyi Gu, Xuyang Zhang, Zhuo Chen, Christopher E. Mower, Haitham Bou-Ammar and Shan Luo (source release)
- **Funded by:** [More Information Needed]
- **Shared by:** Voxel51 (FiftyOne conversion)
- **Language(s):** English (task instructions)
- **License:** CC BY 4.0
### Dataset Sources
- **Repository:** [HapTile2026/HapTile](https://huggingface.co/datasets/HapTile2026/HapTile) (source release); [Voxel51/HapTile](https://huggingface.co/datasets/Voxel51/HapTile) (this conversion)
- **Paper:** [HapTile: A Haptic-Informed Vision-Tactile-Language-Action Dataset for Contact-Rich Imitation Learning](https://arxiv.org/abs/2606.04825) (arXiv:2606.04825, 2026)
- **Demo:** [More Information Needed]
## Uses
### Direct Use
Imitation learning for contact-rich manipulation from vision, tactile, haptic and language signals, using the two RGB-D cameras, the two fingertip tactile streams, the haptic feedback levels, the robot state and the per-episode task instruction.
### Out-of-Scope Use
Metric depth estimation: the depth streams are 8-bit renderings rather than metric depth. Other out-of-scope uses are [More Information Needed].
## Dataset Structure
**Topology**
An ungrouped FiftyOne dataset with `media_type="multimodal"`. One sample is one episode, and its `filepath` is a `.fo.mcap` file under `data//`. There are 1,699 samples across 38 tasks, no sample tags, and an empty `dataset.info`. The dataset has no FiftyOne label fields; every signal lives inside the MCAP file as a channel and is shown by the App’s multimodal viewer. The `configs` entry in the frontmatter maps `data/**` to a single `train` split.
**Sample fields**
| Field | FiftyOne type | Description |
|------------------------------------------------------------------------|--------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------|
| `id`, `filepath`, `tags`, `metadata`, `created_at`, `last_modified_at` | built-in | Standard FiftyOne sample fields |
| `task` | `StringField` | Task name, lower-cased, e.g. `fold_tshirt` (38 values) |
| `session` | `StringField` | Session stamp of the episode, e.g. `0505_164308` (unique per episode) |
| `episode_index` | `IntField` | Index of the episode within its task |
| `recorded` | `StringField` | Wall-clock time of the episode’s first frame, ISO 8601 without a time zone |
| `instruction` | `StringField` | Task instruction text for the episode (81 distinct strings) |
| `num_frames` | `IntField` | Frames in the episode, 12 to 1,838 |
| `duration` | `FloatField` | Episode length in seconds, 0.8 to 122.8 |
| `fps` | `FloatField` | Average frame rate of the episode, 11.6 to 18.7 |
| `end_effector_path_m` | `FloatField` | Length of the end-effector path in metres |
| `gripper_min`, `gripper_max` | `FloatField` | Smallest and largest gripper position in the episode |
| `peak_marker_motion_left`, `peak_marker_motion_right` | `FloatField` | Peak marker motion measured on the left and right fingertip gel pads |
| `haptic_feedback` | `ListField(StringField)` | Haptic feedback levels the operator felt during the episode (`firm`, `mild`, `soft`); empty when none, or when the episode has no haptic channel |
| `haptic_feedback_frames` | `IntField` | Number of frames carrying haptic feedback; empty for episodes with no haptic channel |
Across episodes, `firm` appears in 1,481, `mild` in 414 and `soft` in 575. Of the 107 episodes with an empty `haptic_feedback`, the 47 `move_mobile_box` episodes have no haptic channel at all (their `haptic_feedback_frames` is empty) and the other 60 have `haptic_feedback_frames` of 0.
**Episode contents**
Each MCAP episode contains these channels:
| Channel | Schema or content |
|-----------------------------------------------------|--------------------------------------------------------------------------------------------|
| `/front-camera`, `/wrist-camera` | The two RGB views, `foxglove.CompressedVideo` |
| `/tactile-left`, `/tactile-right` | The two fingertip gel pads, `foxglove.CompressedVideo` |
| `/front-depth`, `/wrist-depth` | Depth renderings at 640x480, `foxglove.CompressedVideo` |
| `/tactile.plot` | The marker motion each fingertip measures, with the release’s summary of the two alongside |
| `/haptic-feedback` | The feedback level the operator felt, written at each change |
| `/end-effector-pose` | End-effector pose in the UR5e base frame, `foxglove.PoseInFrame` |
| `/end-effector.plot`, `/end-effector-velocity.plot` | `x`, `y`, `z` and the rotation vector; linear and angular speed |
| `/joints.plot`, `/joint-velocities.plot` | The six arm joints |
| `/command.plot` | The commanded joint targets and gripper |
| `/gripper.plot` | The gripper position |
| `/instruction` | The task text |
The colour and tactile streams are 320x240 except in `move_mobile_box`, whose 47 episodes record them at 640x480.
**Tasks**
Episodes are grouped by task. The Instruction column shows the most common instruction for the task; 19 tasks carry several instruction variants (81 distinct strings in all), which differ in the target object, colour or location.
| Task | Episodes | Frames | Minutes | Instruction |
|------------------------|------------|----------|-----------|---------------------------------------------------------------------------------------------|
| `insert_peg` | 50 | 40,656 | 45.2 | insert the peg in the red hole |
| `move_mobile_box` | 61 | 29,702 | 29.2 | move the mobile box to the right side of the workspace |
| `move_disposable_cup` | 60 | 26,898 | 32.4 | move the disposable cup to the bottom right side of the workspace |
| `put_apple` | 63 | 24,319 | 27.1 | move the apple onto the red plate |
| `turn_cleanser_bottle` | 30 | 23,114 | 25.7 | pour liquid from the cleanser bottle into the yellow bowl |
| `wipe_whiteboard` | 50 | 22,860 | 25.5 | use the sponge to wipe the whiteboard and remove the marker drawing |
| `put_spray_bottle` | 50 | 22,674 | 25.2 | put the transparent spray bottle on top of the drawer |
| `move_plush_toy` | 70 | 22,636 | 27 | move the plush toy to the bottom left side of the workspace |
| `move_cable` | 55 | 22,063 | 26.5 | move the cable to the bottom left side of the workspace |
| `remove_laundry_pod` | 60 | 21,006 | 23.4 | remove a laundry pod from the box and place it on the green Tshirt in the laundry basket |
| `stack_glass_cups` | 30 | 20,638 | 23 | add the glass cup on the table to the stack of glass cups on the left side of the dish rack |
| `move_water_bottle` | 49 | 20,165 | 22.5 | move the water bottle to the bottom right side of the workspace |
| `put_orange` | 47 | 19,647 | 21.9 | put the orange in the purple bowl |
| `turn_can` | 30 | 19,466 | 21.7 | turn the can upright and place it onto the white tray |
| `put_baseball` | 29 | 17,698 | 19.7 | put the baseball in the purple bowl |
| `fold_tshirt` | 30 | 17,641 | 19.6 | fold the Tshirt from its bottom |
| `stack_disposable_cup` | 51 | 17,118 | 19.1 | pick up the top cup from the stack of disposable cups and place it on the table |
| `remove_sugar_bag` | 61 | 17,064 | 19 | remove one sugar bag and place it on the table |
| `turn_water_bottle` | 39 | 16,842 | 18.8 | turn the water bottle upright and place it onto the white tray |
| `put_sponge` | 31 | 16,825 | 18.7 | put the sponge in the red plate |
| `move_rubiks_cube` | 30 | 15,202 | 16.9 | move the rubik’s cube to the bottom right side of the workspace |
| `remove_cloth` | 30 | 15,018 | 16.7 | remove the green Tshirt from the laundry basket and place it on the table |
| `move_can` | 30 | 14,732 | 16.4 | move the can to the top right side of the workspace |
| `put_spoon` | 49 | 14,391 | 16 | put the spoon in the glass cup |
| `put_spatula` | 49 | 14,360 | 16 | put the spatula on top of the pan |
| `put_stack_glass_cups` | 27 | 14,161 | 15.8 | put the stack of glass cups onto the red plate |
| `put_banana` | 42 | 13,831 | 15.4 | put the banana onto the red plate |
| `put_fork` | 50 | 13,823 | 15.4 | put the fork in the right side of the plate |
| `remove_screwdriver` | 50 | 13,601 | 15.1 | remove the red screwdriver from the table and place it on the left side of the workspace |
| `press_coffee_machine` | 50 | 13,562 | 15.1 | turn on the coffee machine by pressing the button |
| `remove_tissue` | 47 | 12,839 | 14.3 | remove a paper tissue from the box and place it on the table |
| `move_spoon` | 49 | 12,309 | 13.7 | move the spoon to the left side of the plate |
| `put_lego` | 60 | 11,525 | 12.8 | put the black lego piece on the box |
| `stack_bowls` | 50 | 11,486 | 12.8 | add the green bowl to the top of the bowl stack |
| `put_golf_ball` | 50 | 9,350 | 10.4 | put the golf ball into the blue bowl |
| `remove_sticky_note` | 30 | 9,057 | 10.1 | remove the green sticky note and place it on the table |
| `put_strawberry` | 30 | 7,960 | 8.9 | put the strawberry onto the red plate |
| `move_toy_car` | 30 | 7,806 | 8.7 | move the toy car into the transparent box |
**Parsing decisions**
- **Source layout:** the source ships one zip per task holding a folder per episode, each with a `trajectory.h5` that embeds the six camera streams as MPEG-4 Part 2 videos.
- **Video encoding:** every stream is re-encoded to Annex-B H.264 without B-frames, one access unit per frame, frame for frame against the source’s own count.
- **Clock:** the source stamps every frame with the wall-clock time it was recorded, without a time zone. Episodes are placed on that clock relative to their first frame, so the frame rate varies as the recording did, and the start time is kept in `recorded`.
- **Depth:** the depth streams arrive as 8-bit renderings rather than metric depth and are carried as video like the colour streams.
- **Haptic channel:** the release records the haptic channel in three ways. Most episodes carry the feedback level with the summed marker motion of both fingertips and its normalized form, which `/tactile.plot` carries as `sum` and `normalized`. Some carry the larger of the two fingertips instead, carried as `max` and `normalized`. The 47 episodes of `move_mobile_box` carry no haptic channel at all, so they have no `/haptic-feedback` stream and their `haptic_feedback` fields are empty.
- **Stream lengths:** where the source’s streams disagree in length, the difference is trailing: a final row with an empty timestamp, or a state array or video one frame longer than the rest. Every stream is cut to the shortest, which drops the longer streams’ extra trailing frame in 40 episodes. One folder with an empty trajectory is left out.
- **Constant arrays:** the source’s `activated` flag is set on every frame of every episode and its 30-channel `touch` array is zero throughout, so neither is carried.
- **Pose:** the pose is published as a position and a rotation vector and is carried as one on `/end-effector.plot`, alongside the quaternion `/end-effector-pose` needs.
- **Gripper:** the gripper entry of the source’s joint array duplicates `gripper_position` and is carried once, on `/gripper.plot`.
- **Left out:** task names are lower-cased. The handful of still frames some episode folders carry beside the trajectory are not reproduced, and neither are the source’s per-episode frame-rate reports, whose content the timestamps already hold.
## Dataset Creation
### Curation Rationale
[More Information Needed]
### Source Data
#### Data Collection and Processing
Recorded under teleoperation on a UR5e arm with a Robotiq 2F-85 gripper working through contact-rich tabletop tasks, with a vision-based tactile sensor on each gripper finger, a front RGB-D camera and a wrist RGB-D camera. The FiftyOne conversion reads the source’s per-task zips and writes one MCAP episode per episode folder; see Parsing decisions above for the changes made.
#### Who are the source data producers?
The authors of the HapTile release, listed under Dataset Description. Details on the teleoperators are [More Information Needed].
### Annotations
#### Annotation process
The dataset has no FiftyOne label fields and the conversion adds no annotations. Each episode carries the task instruction text and the haptic feedback levels as recorded by the source release; how the instruction texts were written is [More Information Needed].
#### Who are the annotators?
[More Information Needed]
#### Personal and Sensitive Information
[More Information Needed]
## Citation
**BibTeX:**
```bibtex
@article{alian2026haptile,
title = {HapTile: A Haptic-Informed Vision-Tactile-Language-Action
Dataset for Contact-Rich Imitation Learning},
author = {Alian, Amirhosein and Zhao, Yongqiang and Gu, Shiyi and
Zhang, Xuyang and Chen, Zhuo and Mower, Christopher E. and
Bou-Ammar, Haitham and Luo, Shan},
journal = {arXiv preprint arXiv:2606.04825},
year = {2026}
}
```
**APA:**
Alian, A., Zhao, Y., Gu, S., Zhang, X., Chen, Z., Mower, C. E., Bou-Ammar, H., & Luo, S. (2026). HapTile: A haptic-informed vision-tactile-language-action dataset for contact-rich imitation learning. *arXiv preprint arXiv:2606.04825*.
## More Information
The source release is distributed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), and this conversion is distributed under the same license.
Changes from the source: conversion to the FiftyOne MCAP flavor, re-encoding of the six camera streams from MPEG-4 Part 2 to H.264, the recording clock rebased to each episode’s first frame, every stream cut to the shortest where the source’s lengths disagree, the pose carried as a quaternion beside the source’s rotation vector, task names lower-cased, the empty trajectory and the constant `activated` and `touch` arrays left out, and the robot state, tactile, haptic and instruction streams encoded as message streams.
## Dataset Card Authors
[More Information Needed]
## Dataset Card Contact
[More Information Needed]