Note

This is a Hugging Face dataset. For large datasets, ensure huggingface_hub>=1.1.3 to avoid rate limits. Learn more in the Hugging Face integration docs.

Dataset Card for HapTile#

preview

This is a FiftyOne dataset with 1699 samples. Each sample is one teleoperated manipulation episode, stored as a native multimodal MCAP episode.

The source is HapTile, a haptic-informed vision-tactile-language-action dataset. A UR5e arm with a Robotiq 2F-85 gripper works through contact-rich tabletop tasks under teleoperation. Each gripper finger carries a vision-based tactile sensor that films a gel pad printed with a marker grid, so the contact shows as the markers moving. Two RGB-D cameras watch the scene, one facing the table and one on the wrist, and the haptic feedback the operator felt through the teleoperation rig is recorded alongside the robot state.

Installation#

If you haven’t already, install FiftyOne:

pip install -U fiftyone

Usage#

import fiftyone as fo
import fiftyone.utils.huggingface as fouh

# Load the dataset
# Note: other available arguments include 'max_samples', etc
dataset = fouh.load_from_hub(
    "Voxel51/HapTile",
    name="HapTile",
    persistent=True,
)

# Launch the App
session = fo.launch_app(dataset)

Dataset Details#

Dataset Description#

1,699 episodes across 38 tasks, totalling 664,045 frames and 12.36 hours of recording. In 1,592 episodes the operator felt haptic feedback at some point, at the levels firm, mild and soft.

Curated by: The HapTile authors: Amirhosein Alian, Yongqiang Zhao, Shiyi Gu, Xuyang Zhang, Zhuo Chen, Christopher E. Mower, Haitham Bou-Ammar and Shan Luo (source release)
Funded by: [More Information Needed]
Shared by: Voxel51 (FiftyOne conversion)
Language(s): English (task instructions)
License: CC BY 4.0

Dataset Sources#

Repository: HapTile2026/HapTile (source release); Voxel51/HapTile (this conversion)
Paper: HapTile: A Haptic-Informed Vision-Tactile-Language-Action Dataset for Contact-Rich Imitation Learning (arXiv:2606.04825, 2026)
Demo: [More Information Needed]

Uses#

Direct Use#

Imitation learning for contact-rich manipulation from vision, tactile, haptic and language signals, using the two RGB-D cameras, the two fingertip tactile streams, the haptic feedback levels, the robot state and the per-episode task instruction.

Out-of-Scope Use#

Metric depth estimation: the depth streams are 8-bit renderings rather than metric depth. Other out-of-scope uses are [More Information Needed].

Dataset Structure#

Topology

An ungrouped FiftyOne dataset with media_type="multimodal". One sample is one episode, and its filepath is a .fo.mcap file under data/<task>/. There are 1,699 samples across 38 tasks, no sample tags, and an empty dataset.info. The dataset has no FiftyOne label fields; every signal lives inside the MCAP file as a channel and is shown by the App’s multimodal viewer. The configs entry in the frontmatter maps data/** to a single train split.

Sample fields

Field	FiftyOne type	Description
`id`, `filepath`, `tags`, `metadata`, `created_at`, `last_modified_at`	built-in	Standard FiftyOne sample fields
`task`	`StringField`	Task name, lower-cased, e.g. `fold_tshirt` (38 values)
`session`	`StringField`	Session stamp of the episode, e.g. `0505_164308` (unique per episode)
`episode_index`	`IntField`	Index of the episode within its task
`recorded`	`StringField`	Wall-clock time of the episode’s first frame, ISO 8601 without a time zone
`instruction`	`StringField`	Task instruction text for the episode (81 distinct strings)
`num_frames`	`IntField`	Frames in the episode, 12 to 1,838
`duration`	`FloatField`	Episode length in seconds, 0.8 to 122.8
`fps`	`FloatField`	Average frame rate of the episode, 11.6 to 18.7
`end_effector_path_m`	`FloatField`	Length of the end-effector path in metres
`gripper_min`, `gripper_max`	`FloatField`	Smallest and largest gripper position in the episode
`peak_marker_motion_left`, `peak_marker_motion_right`	`FloatField`	Peak marker motion measured on the left and right fingertip gel pads
`haptic_feedback`	`ListField(StringField)`	Haptic feedback levels the operator felt during the episode (`firm`, `mild`, `soft`); empty when none, or when the episode has no haptic channel
`haptic_feedback_frames`	`IntField`	Number of frames carrying haptic feedback; empty for episodes with no haptic channel

Across episodes, firm appears in 1,481, mild in 414 and soft in 575. Of the 107 episodes with an empty haptic_feedback, the 47 move_mobile_box episodes have no haptic channel at all (their haptic_feedback_frames is empty) and the other 60 have haptic_feedback_frames of 0.

Episode contents

Each MCAP episode contains these channels:

Channel	Schema or content
`/front-camera`, `/wrist-camera`	The two RGB views, `foxglove.CompressedVideo`
`/tactile-left`, `/tactile-right`	The two fingertip gel pads, `foxglove.CompressedVideo`
`/front-depth`, `/wrist-depth`	Depth renderings at 640x480, `foxglove.CompressedVideo`
`/tactile.plot`	The marker motion each fingertip measures, with the release’s summary of the two alongside
`/haptic-feedback`	The feedback level the operator felt, written at each change
`/end-effector-pose`	End-effector pose in the UR5e base frame, `foxglove.PoseInFrame`
`/end-effector.plot`, `/end-effector-velocity.plot`	`x`, `y`, `z` and the rotation vector; linear and angular speed
`/joints.plot`, `/joint-velocities.plot`	The six arm joints
`/command.plot`	The commanded joint targets and gripper
`/gripper.plot`	The gripper position
`/instruction`	The task text

The colour and tactile streams are 320x240 except in move_mobile_box, whose 47 episodes record them at 640x480.

Tasks

Episodes are grouped by task. The Instruction column shows the most common instruction for the task; 19 tasks carry several instruction variants (81 distinct strings in all), which differ in the target object, colour or location.

Task	Episodes	Frames	Minutes	Instruction
`insert_peg`	50	40,656	45.2	insert the peg in the red hole
`move_mobile_box`	61	29,702	29.2	move the mobile box to the right side of the workspace
`move_disposable_cup`	60	26,898	32.4	move the disposable cup to the bottom right side of the workspace
`put_apple`	63	24,319	27.1	move the apple onto the red plate
`turn_cleanser_bottle`	30	23,114	25.7	pour liquid from the cleanser bottle into the yellow bowl
`wipe_whiteboard`	50	22,860	25.5	use the sponge to wipe the whiteboard and remove the marker drawing
`put_spray_bottle`	50	22,674	25.2	put the transparent spray bottle on top of the drawer
`move_plush_toy`	70	22,636	27.0	move the plush toy to the bottom left side of the workspace
`move_cable`	55	22,063	26.5	move the cable to the bottom left side of the workspace
`remove_laundry_pod`	60	21,006	23.4	remove a laundry pod from the box and place it on the green Tshirt in the laundry basket
`stack_glass_cups`	30	20,638	23.0	add the glass cup on the table to the stack of glass cups on the left side of the dish rack
`move_water_bottle`	49	20,165	22.5	move the water bottle to the bottom right side of the workspace
`put_orange`	47	19,647	21.9	put the orange in the purple bowl
`turn_can`	30	19,466	21.7	turn the can upright and place it onto the white tray
`put_baseball`	29	17,698	19.7	put the baseball in the purple bowl
`fold_tshirt`	30	17,641	19.6	fold the Tshirt from its bottom
`stack_disposable_cup`	51	17,118	19.1	pick up the top cup from the stack of disposable cups and place it on the table
`remove_sugar_bag`	61	17,064	19.0	remove one sugar bag and place it on the table
`turn_water_bottle`	39	16,842	18.8	turn the water bottle upright and place it onto the white tray
`put_sponge`	31	16,825	18.7	put the sponge in the red plate
`move_rubiks_cube`	30	15,202	16.9	move the rubik’s cube to the bottom right side of the workspace
`remove_cloth`	30	15,018	16.7	remove the green Tshirt from the laundry basket and place it on the table
`move_can`	30	14,732	16.4	move the can to the top right side of the workspace
`put_spoon`	49	14,391	16.0	put the spoon in the glass cup
`put_spatula`	49	14,360	16.0	put the spatula on top of the pan
`put_stack_glass_cups`	27	14,161	15.8	put the stack of glass cups onto the red plate
`put_banana`	42	13,831	15.4	put the banana onto the red plate
`put_fork`	50	13,823	15.4	put the fork in the right side of the plate
`remove_screwdriver`	50	13,601	15.1	remove the red screwdriver from the table and place it on the left side of the workspace
`press_coffee_machine`	50	13,562	15.1	turn on the coffee machine by pressing the button
`remove_tissue`	47	12,839	14.3	remove a paper tissue from the box and place it on the table
`move_spoon`	49	12,309	13.7	move the spoon to the left side of the plate
`put_lego`	60	11,525	12.8	put the black lego piece on the box
`stack_bowls`	50	11,486	12.8	add the green bowl to the top of the bowl stack
`put_golf_ball`	50	9,350	10.4	put the golf ball into the blue bowl
`remove_sticky_note`	30	9,057	10.1	remove the green sticky note and place it on the table
`put_strawberry`	30	7,960	8.9	put the strawberry onto the red plate
`move_toy_car`	30	7,806	8.7	move the toy car into the transparent box

Parsing decisions

Source layout: the source ships one zip per task holding a folder per episode, each with a trajectory.h5 that embeds the six camera streams as MPEG-4 Part 2 videos.
Video encoding: every stream is re-encoded to Annex-B H.264 without B-frames, one access unit per frame, frame for frame against the source’s own count.
Clock: the source stamps every frame with the wall-clock time it was recorded, without a time zone. Episodes are placed on that clock relative to their first frame, so the frame rate varies as the recording did, and the start time is kept in recorded.
Depth: the depth streams arrive as 8-bit renderings rather than metric depth and are carried as video like the colour streams.
Haptic channel: the release records the haptic channel in three ways. Most episodes carry the feedback level with the summed marker motion of both fingertips and its normalized form, which /tactile.plot carries as sum and normalized. Some carry the larger of the two fingertips instead, carried as max and normalized. The 47 episodes of move_mobile_box carry no haptic channel at all, so they have no /haptic-feedback stream and their haptic_feedback fields are empty.
Stream lengths: where the source’s streams disagree in length, the difference is trailing: a final row with an empty timestamp, or a state array or video one frame longer than the rest. Every stream is cut to the shortest, which drops the longer streams’ extra trailing frame in 40 episodes. One folder with an empty trajectory is left out.
Constant arrays: the source’s activated flag is set on every frame of every episode and its 30-channel touch array is zero throughout, so neither is carried.
Pose: the pose is published as a position and a rotation vector and is carried as one on /end-effector.plot, alongside the quaternion /end-effector-pose needs.
Gripper: the gripper entry of the source’s joint array duplicates gripper_position and is carried once, on /gripper.plot.
Left out: task names are lower-cased. The handful of still frames some episode folders carry beside the trajectory are not reproduced, and neither are the source’s per-episode frame-rate reports, whose content the timestamps already hold.

Dataset Creation#

Curation Rationale#

[More Information Needed]

Source Data#

Data Collection and Processing#

Recorded under teleoperation on a UR5e arm with a Robotiq 2F-85 gripper working through contact-rich tabletop tasks, with a vision-based tactile sensor on each gripper finger, a front RGB-D camera and a wrist RGB-D camera. The FiftyOne conversion reads the source’s per-task zips and writes one MCAP episode per episode folder; see Parsing decisions above for the changes made.

Who are the source data producers?#

The authors of the HapTile release, listed under Dataset Description. Details on the teleoperators are [More Information Needed].

Annotations#

Annotation process#

The dataset has no FiftyOne label fields and the conversion adds no annotations. Each episode carries the task instruction text and the haptic feedback levels as recorded by the source release; how the instruction texts were written is [More Information Needed].

Who are the annotators?#

[More Information Needed]

Personal and Sensitive Information#

[More Information Needed]

Citation#

BibTeX:

@article{alian2026haptile,
  title   = {HapTile: A Haptic-Informed Vision-Tactile-Language-Action
             Dataset for Contact-Rich Imitation Learning},
  author  = {Alian, Amirhosein and Zhao, Yongqiang and Gu, Shiyi and
             Zhang, Xuyang and Chen, Zhuo and Mower, Christopher E. and
             Bou-Ammar, Haitham and Luo, Shan},
  journal = {arXiv preprint arXiv:2606.04825},
  year    = {2026}
}

APA:

Alian, A., Zhao, Y., Gu, S., Zhang, X., Chen, Z., Mower, C. E., Bou-Ammar, H., & Luo, S. (2026). HapTile: A haptic-informed vision-tactile-language-action dataset for contact-rich imitation learning. arXiv preprint arXiv:2606.04825.

More Information#

The source release is distributed under CC BY 4.0, and this conversion is distributed under the same license.

Changes from the source: conversion to the FiftyOne MCAP flavor, re-encoding of the six camera streams from MPEG-4 Part 2 to H.264, the recording clock rebased to each episode’s first frame, every stream cut to the shortest where the source’s lengths disagree, the pose carried as a quaternion beside the source’s rotation vector, task names lower-cased, the empty trajectory and the constant activated and touch arrays left out, and the robot state, tactile, haptic and instruction streams encoded as message streams.

Dataset Card Authors#

[More Information Needed]

Dataset Card Contact#

[More Information Needed]