Note

This is a Hugging Face dataset. For large datasets, ensure huggingface_hub>=1.1.3 to avoid rate limits. Learn more in the Hugging Face integration docs.

This is a FiftyOne dataset with 23 samples.

Installation#

If you haven’t already, install FiftyOne:

pip install -U fiftyone

Usage#

import fiftyone as fo
from fiftyone.utils.huggingface import load_from_hub

# Load the dataset
# Note: other available arguments include 'max_samples', etc
dataset = load_from_hub("Voxel51/trex-dataset-23ep")

# Launch the App
session = fo.launch_app(dataset)

Dataset Card for T-Rex Dataset (23-episode FiftyOne subset)#

T-Rex Dataset preview

A 23-episode subset of the T-Rex Dataset, a large-scale, tactile-reactive bimanual manipulation dataset collected via teleoperation on a Dexmate Vega-1 robot with two Sharpa Wave dexterous hands. The full dataset (5,464 episodes, 1.5 TB) is published at zekaiwang/trex_dataset; this repo holds episodes 0–22 re-packaged as a self-contained LeRobotDataset v3.0 export and loaded into FiftyOne for exploration.

Dataset Details#

Dataset Description#

Curated by: Dantong Niu, Zhuoyang Liu, Zekai Wang, Boning Shao, Zhao-Heng Yin, Anirudh Pai, Yuvan Sharma, Stefano Saravalle, Ruijie Zheng, Jing Wang, Ryan Punamiya, Mengda Xu, Yuqi Xie, Yunfan Jiang, Letian Fu, Konstantinos Kallidromitis, Matteo Gioia, Junyi Zhang, Jiaxin Ge, Haiwen Feng, Fabio Galasso, Wei Zhan, David M. Chan, Yutong Bai, Roei Herzig, Jiahui Lei, Fei-Fei Li, Ken Goldberg, Jitendra Malik, Pieter Abbeel, Yuke Zhu, Danfei Xu, Jim Fan, Trevor Darrell
Shared by: zekaiwang (original dataset); this FiftyOne subset shared by the FiftyOne community
Language(s): English (task captions)
License: MIT (© 2026 The Regents of the University of California)

Dataset Sources#

Repository: zekaiwang/trex_dataset · T-Rex code
Paper: T-Rex: Tactile-Reactive Dexterous Manipulation (arXiv:2606.17055)
Demo: Project Page

Uses#

Direct Use#

Exploring and visualizing tactile-reactive bimanual manipulation episodes in the FiftyOne App — inspecting synchronized RGB + tactile video streams alongside joint state/action trajectories and per-fingertip force readings, filtering by task/motor primitive/object, and prototyping data loaders before working with the full 5,464-episode dataset.

Out-of-Scope Use#

This 23-episode subset is not a statistically representative sample of the full dataset (it is simply the first LeRobot data/video shard) and should not be used to draw conclusions about task, object, or motor-primitive distributions across the full T-Rex Dataset — load the full zekaiwang/trex_dataset repo for that.

Dataset Structure#

This is a multimodal FiftyOne dataset (dataset.media_type == "multimodal") with 23 samples, one sample per episode. Each sample’s media (23 video streams) is not copied into per-sample files; instead it is resolved through a media_reference that points into the exported LeRobotDataset v3.0 source (data/, videos/, meta/ in this repo) at import time — this is how FiftyOne represents LeRobot episodes natively.

Fields#

Field	FiftyOne type	Description
`id`	`ObjectIdField`	FiftyOne sample id
`media_reference`	`MediaReferenceField`	Pointer into the LeRobot source’s `data/`, `videos/*`, and `meta/` files for this episode (chunk/file indexes, frame range, per-video timestamp ranges) — resolved on demand, not duplicated per sample
`tags`	`ListField(StringField)`	FiftyOne sample tags (empty by default)
`metadata`	`Metadata` (`size_bytes`, `mime_type`)	Standard FiftyOne sample metadata
`created_at` / `last_modified_at`	`DateTimeField`	FiftyOne bookkeeping timestamps
`episode_index`	`IntField`	Episode index within this subset (`0`–`22`, remapped on export from the source dataset’s original indices)
`task`	`StringField`	Primary task caption for the episode (verbatim from source `caption`)
`tasks`	`ListField(StringField)`	Full task list for the episode (LeRobot `tasks` array; length 1 for every episode here)
`length`	`IntField`	Number of frames in the episode (verbatim from source)
`duration`	`FloatField`	Episode duration in seconds (`length / fps`)
`robot_type`	`StringField`	`dexmate_vega1_and_sharpa_wave` (verbatim from source `meta/info.json`)
`fps`	`FloatField`	Recording frame rate, 30.0 (verbatim from source)

The per-frame numeric features (observation.state (58,), action (58,), observation.tactile_force (60,)) and the 23 per-frame video streams are not flattened into sample fields — they remain in the LeRobot data/*.parquet and videos/*/*.mp4 files referenced by media_reference, and are surfaced by the FiftyOne App’s State & Action, Streams, and Statistics viewer tabs rather than as queryable sample-level fields.

Label types and why#

There are no traditional detection/classification/segmentation labels in this dataset. task is stored as a plain StringField rather than fo.Classification because task captions here are free-form natural language instructions (5,370 unique across the full dataset, 22 unique in this subset), not a fixed closed-set taxonomy — a Classification label with logits/confidence semantics does not fit a free-text instruction.

`dataset.info` contents#

{
    "lerobot": {
        "format": "LeRobotDataset",
        "format_major": 3,
        "episode_count": 5464,          # total episodes in the full source dataset
        "imported_episode_count": 23,   # episodes actually imported into this subset
        "skipped_episodes": [],
    }
}

Parsing decisions#

Which episodes, and why: episodes 0–22 were selected because they are the largest contiguous, zero-gap block of episodes whose data/chunk-000/file-000.parquet shard and every one of the 23 videos/<key>/chunk-000/file-000.mp4 shards are shared — i.e. the smallest set of source files that had to be downloaded to get a complete, non-truncated set of episodes, given a limited local disk budget. It is not a curated or stratified sample.
Re-export, not a thin reference to the original repo: this repo is a self-contained LeRobotDataset v3.0 export (via FiftyOne’s LeRobotDatasetExporter), not a pointer back to zekaiwang/trex_dataset. Episode indices, frame index, and video/data chunk-file coordinates were rewritten during export so the 23 episodes are contiguous (0–22) and self-consistent; task indices were remapped to only the tasks actually present in this subset. Per-episode and global statistics (meta/stats.json, per-episode stats/* columns) were recomputed from the exported rows, not carried over from the source’s global stats.
Tactile video codec (read before decoding outside FiftyOne): the 20 tactile streams (observation.images.tactile_{left,right}_{raw,deform}_{finger}) are encoded losslessly (libx264 -qp 0) because their pixel values are physically meaningful raw sensor/deformation readings. They are grayscale, full-range yuvj420p, which forces the H.264 High 4:4:4 Predictive profile — most browsers cannot decode this profile in a plain <video>/WebCodecs pipeline (no thumbnails in the HF preview or generic players), so use FiftyOne, ffmpeg, PyAV, or torchcodec to view them locally. The 3 RGB streams (head_left, left_wrist, right_wrist) use standard limited-range yuv420p/BT.709 and preview normally everywhere.
No held-out or unlabeled split: all 23 episodes are in a single train split with every episode language-annotated; nothing was withheld.

Dataset Creation#

Curation Rationale#

The full T-Rex Dataset was collected to study tactile-reactive dexterous manipulation — pairing bimanual arm/hand joint trajectories with synchronized per-fingertip tactile sensing so that policies can condition on touch, not just vision and proprioception. This subset exists purely as a lightweight, disk-budget-friendly slice for exploration and tooling in FiftyOne; it was not re-curated for content.

Source Data#

Data Collection and Processing#

Robot: Dexmate Vega-1 dual-arm mobile robot (7 actuated joints per arm) with two Sharpa Wave dexterous hands (5 fingertip tactile sensors each). During collection the wheels, torso, and head joints are fixed; only the 14 arm joints and two hands are actuated.
Cameras: a head-mounted ZED X Mini stereo camera (left monocular RGB stream recorded) plus two wide-view ZED X One S monocular RGB wrist cameras, all at 640×360, 30 fps.
Tactile sensing: each hand’s 5 fingertip sensors contribute a raw sensor image (tactile_*_raw_*, 320×240) and an estimated deformation map (tactile_*_deform_*, 240×240), both stored as lossless grayscale video, plus an estimated 6-axis net wrench per fingertip in observation.tactile_force (60,) = (left, right) × (thumb…pinky) × (Fx, Fy, Fz, Mx, My, Mz).
Teleoperation: Manus gloves capture fingertip positions retargeted to the Sharpa Wave hands via the manufacturer’s differential-inverse-kinematics package (Pinocchio + CasADi). Two VIVE trackers provide SE(3) wrist poses converted to arm joint commands via differential inverse kinematics (Pink), low-pass filtered, and tracked by the manufacturer’s low-level cascade PID controller. A 30 Hz high-level thread records observations and joint-space targets (action) while a 300 Hz low-level thread runs control; the dataset’s 30 fps matches the high-level loop.
observation.state / action layout (58,): [L_arm 7 | L_hand 22 | R_arm 7 | R_hand 22] joint positions (observation.state) and target joint positions (action). Full per-dimension joint names are in the source meta/info.json (features[*].names).

Who are the source data producers?#

Collected by the T-Rex authors (UC Berkeley and collaborating institutions; see author list above) via in-person teleoperation with the hardware/software stack described above.

Annotations#

Annotation process#

Each episode is labeled with a human-verified natural-language caption (task instruction), a motor_primitive category (one of 22, e.g. reach, lift_and_place), a canonical object name, and — only for lift_and_place episodes — a canonical target/receptacle name (null otherwise). In this subset, caption is surfaced as the sample-level task field.

Personal and Sensitive Information#

None identified — the dataset contains robot joint/tactile sensor data and task captions describing tabletop manipulation of everyday objects; no human subjects data.

Citation#

BibTeX:

@misc{trex2026,
  title={T-Rex: Tactile-Reactive Dexterous Manipulation},
  author={Dantong Niu and Zhuoyang Liu and Zekai Wang and Boning Shao and Zhao-Heng Yin and Anirudh Pai and Yuvan Sharma and Stefano Saravalle and Ruijie Zheng and Jing Wang and Ryan Punamiya and Mengda Xu and Yuqi Xie and Yunfan Jiang and Letian Fu and Konstantinos Kallidromitis and Matteo Gioia and Junyi Zhang and Jiaxin Ge and Haiwen Feng and Fabio Galasso and Wei Zhan and David M. Chan and Yutong Bai and Roei Herzig and Jiahui Lei and Fei-Fei Li and Ken Goldberg and Jitendra Malik and Pieter Abbeel and Yuke Zhu and Danfei Xu and Jim Fan and Trevor Darrell},
  year={2026},
  eprint={2606.17055},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  url={https://arxiv.org/abs/2606.17055},
}

APA:

Niu, D., Liu, Z., Wang, Z., Shao, B., Yin, Z.-H., Pai, A., Sharma, Y., Saravalle, S., Zheng, R., Wang, J., Punamiya, R., Xu, M., Xie, Y., Jiang, Y., Fu, L., Kallidromitis, K., Gioia, M., Zhang, J., Ge, J., Feng, H., Galasso, F., Zhan, W., Chan, D. M., Bai, Y., Herzig, R., Lei, J., Li, F.-F., Goldberg, K., Malik, J., Abbeel, P., Zhu, Y., Xu, D., Fan, J., & Darrell, T. (2026). T-Rex: Tactile-Reactive Dexterous Manipulation. arXiv:2606.17055.

More Information#

This is a 23-episode subset of the full zekaiwang/trex_dataset (5,464 episodes, ~50 hours, 1.5 TB), produced for local exploration under a limited disk budget. See the source repo for the full dataset, the T-Rex Quick Start tools, and the Colab notebook.

Dataset Card Authors#

Harpreet Sahota

Dataset Card Contact#