Note

This is a Hugging Face dataset. For large datasets, ensure huggingface_hub>=1.1.3 to avoid rate limits. Learn more in the Hugging Face integration docs.

Hugging Face

Dataset Card for FreeTacMan (FiftyOne)#

FiftyOne

image/png

FreeTacMan (FiftyOne) is a grouped FiftyOne video dataset built from the OpenDriveLab/FreeTacMan visuo-tactile manipulation dataset. Each group is a single demonstration trajectory, with one video per camera/sensor view and the full tool-center-point (TCP) and gripper trajectory stored as per-frame fields.

Installation#

If you haven’t already, install FiftyOne:

pip install -U fiftyone

Usage#

import fiftyone as fo
from fiftyone.utils.huggingface import load_from_hub

# Load the dataset from the Hugging Face Hub
dataset = load_from_hub("Voxel51/FreeTacMan")

# Launch the App
session = fo.launch_app(dataset)

Dataset Details#

Dataset Description#

FreeTacMan is a large-scale, high-precision visuo-tactile manipulation dataset collected with a robot-free, human-operated data collection system. It targets contact-rich manipulation, pairing synchronized RGB video from a wrist-mounted fisheye camera with two GelSight-style optical tactile sensors, plus dense 6-DoF end-effector trajectories. The source dataset contains over 3,000k visuo-tactile image pairs and more than 10k trajectories across 50 task categories.

This FiftyOne version reorganizes the source data into a grouped video dataset so the tactile and visual streams for a demonstration can be viewed side by side in the FiftyOne App, with the robot proprioception rendered as per-frame numeric fields.

  • Curated by: OpenDriveLab

  • Funded by: [More Information Needed]

  • Shared by: Longyan Wu, Checheng Yu, Jieji Ren, Li Chen, Yufei Jiang, Ran Huang, Guoying Gu, Hongyang Li

  • Language(s): N/A (video, tactile, and trajectory data; no text)

  • License: MIT (source dataset license)

Dataset Sources#

Uses#

Direct Use#

This dataset is intended for research on visuo-tactile robot learning, including:

  • Imitation learning of contact-rich manipulation policies (e.g. Action Chunking Transformer-style models) that fuse wrist-camera RGB with optical tactile sensor observations and 6-DoF end-effector trajectories.

  • Tactile representation / contrastive pretraining of a tactile encoder aligned with a visual encoder, as described in the source paper.

  • Benchmarking tactile-vs-vision-only policies on contact-rich tasks (e.g. slip detection, force-sensitive grasping, precise insertion).

Out-of-Scope Use#

[More Information Needed]. The source paper notes the tactile sensor is an optical (camera-based) gel-deformation sensor rather than a calibrated force/torque sensor, so the dataset should not be treated as ground-truth contact force measurements. The paper’s own experiments were validated on a PIPER 6-DoF arm; direct transfer of learned policies to other embodiments has not been established by the source paper.

Dataset Structure#

Dataset name: FreeTacMan

Media type: group

Default group slice: tactile_sensor_1

Summary#

Property

Value

Groups (trajectories)

6,228

Video samples (total)

18,084

Tasks

45

Group slices

tactile_sensor_1, tactile_sensor_2, fisheye_camera

Groups and slices#

Each group is one demonstration trajectory. The linked slices are the synchronized streams recorded during that demonstration:

Slice

Media type

Samples

Description

tactile_sensor_1 (default)

video

6,228

First optical tactile sensor stream

tactile_sensor_2

video

6,228

Second optical tactile sensor stream

fisheye_camera

video

5,628

Wrist-mounted fisheye RGB camera stream

fisheye_camera is missing from ~600 groups because a small subset of tasks were recorded with only the two tactile-sensor streams and no wrist camera. Switch slices in the FiftyOne App to compare the tactile imprints against the RGB view for the same demonstration. Videos are transcoded to H.264 / yuv420p for in-App playback; the source videos are MPEG-4 Part 2.

Sample-level fields#

Field

Type

Description

trajectory_id

string

Unique id, e.g. Hold_3 ({task}_{demo_idx})

task

fo.Classification

Task name, e.g. PourWater

demo_idx

int

Demonstration index within the task

camera

string

Original source camera id for this sample (camera1/camera2/camera3; camera1 maps to tactile_sensor_1, camera2 to tactile_sensor_2, camera3 to fisheye_camera)

num_timesteps

int

Number of trajectory timesteps for this demo

Frame-level fields#

The 6-DoF end-effector trajectory is replicated on every stream in the group, so proprioception can be read off any slice frame-by-frame. Per the source paper, these values come from an OptiTrack motion-capture system tracking the in-situ gripper interface at 300 Hz, downsampled and synchronized to the 30 Hz camera frame rate:

Field

Type

Description

timestamp

float

Unix timestamp of the frame

tcp_pos_x / tcp_pos_y / tcp_pos_z

float

TCP (tool center point) position

tcp_euler_x / tcp_euler_y / tcp_euler_z

float

TCP orientation (Euler angles)

quat_w / quat_x / quat_y / quat_z

float

TCP orientation (quaternion)

gripper_distance

float

Gripper opening distance

Tasks#

This dataset spans 45 contact-rich manipulation tasks, including ArrangeFruit, CakePiping, CutBanana, PourWater, ScrewInTheBulb, SqueezeToothpaste, ThreadNeedle, UsbPlug, WipeBoard, Write, and more.

from fiftyone import ViewField as F

# All demonstrations of a single task
pour = dataset.match(F("task.label") == "PourWater")

# Just the tactile view for every demonstration
tactile = dataset.select_group_slices("tactile_sensor_1")

Dataset Creation#

Curation Rationale#

Existing demonstration-collection setups for contact-rich manipulation are either expensive/complex real-robot teleoperation rigs (motion-capture, VR/AR, primary-replica arms) or handheld interfaces whose multi-link trigger-based grippers introduce mechanical backlash that blurs tactile cues. FreeTacMan was built to eliminate that backlash: a wearable, in-situ visuo-tactile sensor sits directly at the operator’s fingertip, giving zero-mechanical-attenuation tactile feedback while remaining robot-free and cross-embodiment. The dataset was curated by collecting many contact-rich manipulation demonstrations with this system to support tactile-conditioned imitation learning research.

Source Data#

Data Collection and Processing#

Each demonstration frame pairs a wrist-mounted fisheye-camera RGB image (180° field of view, 640×480 @ 30 FPS) with two camera-based optical tactile sensor images (640×480 @ 30 FPS, one per fingertip), plus the 6-DoF end -effector pose and gripper width. End-effector pose is derived from an OptiTrack motion-capture system tracking five retroreflective markers on the in-situ gripper interface at 300 Hz (mean tracking error 0.118 mm), with marker coordinates transformed into the robot base frame and downsampled to synchronize with the 30 Hz camera streams.

For this FiftyOne release, the source per-task directories of {task}_{demo_idx}_camera{1,2,3}.mp4 videos and {task}_{demo_idx}_traj.csv trajectory files were parsed into a grouped video dataset: each trajectory CSV’s 12 columns were mapped to per-frame fields, source MPEG-4 Part 2 videos were transcoded to H.264/yuv420p MP4 for in-App playback, and each demonstration’s camera streams were joined into a fo.Group() with slices renamed to tactile_sensor_1, tactile_sensor_2, and fisheye_camera.

Who are the source data producers?#

Demonstrations were collected by human operators wearing/holding the FreeTacMan wearable interface. The source paper’s user study describes 12 volunteer operators collecting demonstrations across 8 tasks to evaluate the data-collection system itself; [More Information Needed] on the exact number and identity of operators who produced the full 45-task, 6,228-trajectory dataset released publicly, as this is a larger-scale release beyond the paper’s user study.

Annotations#

Annotation process#

There is no manual human annotation. The task label is assigned from the source directory/file naming convention (one folder per task). All frame -level trajectory fields (tcp_pos_*, tcp_euler_*, quat_*, gripper_distance, timestamp) are captured automatically by the OptiTrack motion-capture system described above, not hand-labeled.

Who are the annotators?#

Not applicable — trajectory fields are sensor/motion-capture output rather than human annotations. [More Information Needed] on task-label assignment beyond the directory-naming convention.

Personal and Sensitive Information#

[More Information Needed]. The source paper does not discuss personal or sensitive information; recordings are of a wrist-mounted camera and fingertip-mounted tactile sensors during tabletop manipulation tasks.

Citation#

If you use the source FreeTacMan dataset, cite:

BibTeX:

@article{wu2025freetacman,
  title   = {FreeTacMan: Robot-free Visuo-Tactile Data Collection System for Contact-rich Manipulation},
  author  = {Wu, Longyan and Yu, Checheng and Ren, Jieji and Chen, Li and Jiang, Yufei and Huang, Ran and Gu, Guoying and Li, Hongyang},
  journal = {IEEE International Conference on Robotics and Automation (ICRA)},
  year    = {2026}
}

APA:

Wu, L., Yu, C., Ren, J., Chen, L., Jiang, Y., Huang, R., Gu, G., & Li, H. (2026). FreeTacMan: Robot-free visuo-tactile data collection system for contact-rich manipulation. IEEE International Conference on Robotics and Automation (ICRA).

More Information#

Dataset Card Authors#

Harpreet Sahota (FiftyOne conversion and card)