Note

This is a Hugging Face dataset. For large datasets, ensure huggingface_hub>=1.1.3 to avoid rate limits. Learn more in the Hugging Face integration docs.

Hugging Face

DreamTac → FiftyOne (Native Multimodal MCAP)#

preview

DreamTac from Peking University, converted to native multimodal MCAP episodes.

A Franka Emika Panda works through contact-rich tabletop tasks while four cameras record on one 20 fps clock: a third-person view, a wrist view, and two Xense Photon vision-based tactile sensors mounted on the gripper fingertips. The fingertips are the point of the release. Each is a gel pad printed with a marker grid, and the grid deforms where the object presses, so the moment of contact is visible rather than inferred from a force reading.

Installation#

pip install fiftyone

Usage#

import fiftyone as fo
import fiftyone.utils.huggingface as fouh

dataset = fouh.load_from_hub(
    "Voxel51/DreamTac",
    name="DreamTac",
    persistent=True,
)
fo.launch_app(dataset)

The trajectories that moved the furthest:

view = dataset.sort_by("end_effector_path_m", reverse=True)
fo.launch_app(dataset, view=view)

What you get#

703 trajectories across 12 tasks, 506,078 frames, 7.03 hours. Every frame carries all four views, so the set holds 2,024,312 camera frames in total.

Each episode contains:

  • /camera, the third-person view at 224x224, as foxglove.CompressedVideo

  • /wrist-camera, the same from the wrist

  • /tactile-left and /tactile-right, the two Xense Photon fingertips

  • /end-effector-pose, position and orientation in the Franka base frame, as foxglove.PoseInFrame

  • /end-effector.plot, the same as x, y, z and the rotation vector

  • /gripper.plot, the gripper opening

  • /instruction, the task text, written at each change so a task with sub-steps shows them on the timeline

Task

Episodes

Frames

Minutes

Mean path

Instruction

pick_bread

99

105,603

88.0

1.83 m

Pick up the baguette.

clean_whiteboard

104

86,843

72.4

1.11 m

Clean the whiteboard with an eraser.

pick_place

100

73,308

61.1

0.94 m

Pick up the object and place it at the target location.

pick_usb

102

68,670

57.2

0.75 m

Pick up the USB plug.

stack_bowl

100

47,539

39.6

0.71 m

Stack the bowl.

insert_screw

48

31,046

25.9

0.56 m

Insert the nut onto the screw.

push_hongzhong_pos2

34

28,559

23.8

0.63 m

Push the red dragon Mahjong tile to the target position.

push_hongzhong_pos3

25

25,628

21.4

0.82 m

Push the red dragon Mahjong tile to the target position.

push_hongzhong_pos1

33

18,356

15.3

0.49 m

Push the red dragon Mahjong tile to the target position.

cut_banana

38

10,512

8.8

0.20 m

Cut the banana with a knife.

pick_hongzhong

18

9,413

7.8

0.61 m

Pick up the red dragon Mahjong tile.

insert_usb

2

601

0.5

0.23 m

Insert the USB plug into the port.

Episodes carry the fields task, episode_index, instruction, num_frames, duration, fps, end_effector_path_m, end_effector_z_min_m, end_effector_z_max_m, gripper_min, gripper_max and tactile_sensor.

The three push_hongzhong tasks are the same push from three different start positions, and insert_usb holds two trajectories against pick_usb’s 102, so the set is not balanced across tasks.

Notes on the conversion#

The source ships one Zarr v2 group per task in the Open-X-Tactile layout, with every trajectory laid end to end and meta/episode_ends marking the boundaries. The per-frame clock restarts at each boundary, so episodes are cut there and each one starts at zero.

The four image streams arrive as raw uint8 arrays and are encoded to Annex-B H.264 without B-frames, one access unit per message.

The end-effector rotation is published as a rotation vector in radians and is carried as one, in /end-effector.plot, alongside the quaternion /end-effector-pose needs.

Arm joint angles are not in the source and are not carried. The gripper is a single opening scalar.

The release excludes 21 trajectories from a no_tactile batch and 4 that failed source validation, leaving the 703 here. Episode and frame counts match the release’s own source_qc_summary.json on all 12 tasks.

License & attribution#

The source release is distributed under CC BY 4.0, and this conversion is distributed under the same license.

@article{lou2026dreamtac,
  title   = {Dream-Tac: A Unified Tactile World Action Model for
             Contact-Rich Robot Manipulation},
  author  = {Lou, Yunfan and Ye, Yifan and Fu, Yankai and Cen, Jun and
             Chi, Xiaowei and Lyu, Yaoxu},
  journal = {arXiv preprint arXiv:2606.08737},
  year    = {2026}
}

Changes from the source: conversion to the FiftyOne MCAP flavor, encoding of the four image streams to H.264, splitting of the concatenated task groups at their episode boundaries, and encoding of the pose, gripper and instruction streams as message streams.