#### NOTE
This is a **Hugging Face dataset**. For large datasets, ensure `huggingface_hub>=1.1.3` to avoid rate limits. Learn more in the Hugging Face integration docs.
# DreamTac → FiftyOne (Native Multimodal MCAP)
[DreamTac](https://huggingface.co/datasets/cloudfan/DreamTac) from Peking
University, converted to native multimodal MCAP episodes.
A Franka Emika Panda works through contact-rich tabletop tasks while four
cameras record on one 20 fps clock: a third-person view, a wrist view, and two
Xense Photon vision-based tactile sensors mounted on the gripper fingertips.
The fingertips are the point of the release. Each is a gel pad printed with a
marker grid, and the grid deforms where the object presses, so the moment of
contact is visible rather than inferred from a force reading.
## Installation
```bash
pip install fiftyone
```
## Usage
```python
import fiftyone as fo
import fiftyone.utils.huggingface as fouh
dataset = fouh.load_from_hub(
"Voxel51/DreamTac",
name="DreamTac",
persistent=True,
)
fo.launch_app(dataset)
```
The trajectories that moved the furthest:
```python
view = dataset.sort_by("end_effector_path_m", reverse=True)
fo.launch_app(dataset, view=view)
```
## What you get
703 trajectories across 12 tasks, 506,078 frames, 7.03 hours. Every frame
carries all four views, so the set holds 2,024,312 camera frames in total.
Each episode contains:
- `/camera`, the third-person view at 224x224, as `foxglove.CompressedVideo`
- `/wrist-camera`, the same from the wrist
- `/tactile-left` and `/tactile-right`, the two Xense Photon fingertips
- `/end-effector-pose`, position and orientation in the Franka base frame, as
`foxglove.PoseInFrame`
- `/end-effector.plot`, the same as `x`, `y`, `z` and the rotation vector
- `/gripper.plot`, the gripper opening
- `/instruction`, the task text, written at each change so a task with
sub-steps shows them on the timeline
| Task | Episodes | Frames | Minutes | Mean path | Instruction |
|-----------------------|------------|----------|-----------|-------------|----------------------------------------------------------|
| `pick_bread` | 99 | 105,603 | 88 | 1.83 m | Pick up the baguette. |
| `clean_whiteboard` | 104 | 86,843 | 72.4 | 1.11 m | Clean the whiteboard with an eraser. |
| `pick_place` | 100 | 73,308 | 61.1 | 0.94 m | Pick up the object and place it at the target location. |
| `pick_usb` | 102 | 68,670 | 57.2 | 0.75 m | Pick up the USB plug. |
| `stack_bowl` | 100 | 47,539 | 39.6 | 0.71 m | Stack the bowl. |
| `insert_screw` | 48 | 31,046 | 25.9 | 0.56 m | Insert the nut onto the screw. |
| `push_hongzhong_pos2` | 34 | 28,559 | 23.8 | 0.63 m | Push the red dragon Mahjong tile to the target position. |
| `push_hongzhong_pos3` | 25 | 25,628 | 21.4 | 0.82 m | Push the red dragon Mahjong tile to the target position. |
| `push_hongzhong_pos1` | 33 | 18,356 | 15.3 | 0.49 m | Push the red dragon Mahjong tile to the target position. |
| `cut_banana` | 38 | 10,512 | 8.8 | 0.20 m | Cut the banana with a knife. |
| `pick_hongzhong` | 18 | 9,413 | 7.8 | 0.61 m | Pick up the red dragon Mahjong tile. |
| `insert_usb` | 2 | 601 | 0.5 | 0.23 m | Insert the USB plug into the port. |
Episodes carry the fields `task`, `episode_index`, `instruction`,
`num_frames`, `duration`, `fps`, `end_effector_path_m`,
`end_effector_z_min_m`, `end_effector_z_max_m`, `gripper_min`, `gripper_max`
and `tactile_sensor`.
The three `push_hongzhong` tasks are the same push from three different start
positions, and `insert_usb` holds two trajectories against `pick_usb`’s 102,
so the set is not balanced across tasks.
## Notes on the conversion
The source ships one Zarr v2 group per task in the Open-X-Tactile layout, with
every trajectory laid end to end and `meta/episode_ends` marking the
boundaries. The per-frame clock restarts at each boundary, so episodes are cut
there and each one starts at zero.
The four image streams arrive as raw `uint8` arrays and are encoded to Annex-B
H.264 without B-frames, one access unit per message.
The end-effector rotation is published as a rotation vector in radians and is
carried as one, in `/end-effector.plot`, alongside the quaternion
`/end-effector-pose` needs.
Arm joint angles are not in the source and are not carried. The gripper is a
single opening scalar.
The release excludes 21 trajectories from a `no_tactile` batch and 4 that
failed source validation, leaving the 703 here. Episode and frame counts match
the release’s own `source_qc_summary.json` on all 12 tasks.
## License & attribution
The source release is distributed under
[CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), and this
conversion is distributed under the same license.
```bibtex
@article{lou2026dreamtac,
title = {Dream-Tac: A Unified Tactile World Action Model for
Contact-Rich Robot Manipulation},
author = {Lou, Yunfan and Ye, Yifan and Fu, Yankai and Cen, Jun and
Chi, Xiaowei and Lyu, Yaoxu},
journal = {arXiv preprint arXiv:2606.08737},
year = {2026}
}
```
Changes from the source: conversion to the FiftyOne MCAP flavor, encoding of
the four image streams to H.264, splitting of the concatenated task groups at
their episode boundaries, and encoding of the pose, gripper and instruction
streams as message streams.