#### NOTE
This is a **Hugging Face dataset**. For large datasets, ensure `huggingface_hub>=1.1.3` to avoid rate limits. Learn more in the <a href="https://docs.voxel51.com/integrations/huggingface.html#loading-datasets-from-the-hub" target="_blank">Hugging Face integration docs</a>.

<a href="https://huggingface.co/datasets/Voxel51/trex-dataset-23ep" target="_blank">![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Dataset-yellow)</a>

This is a [FiftyOne](https://github.com/voxel51/fiftyone) dataset with 23 samples.

# Installation

If you haven’t already, install FiftyOne:

```bash
pip install -U fiftyone
```

# Usage

```python
import fiftyone as fo
from fiftyone.utils.huggingface import load_from_hub

# Load the dataset
# Note: other available arguments include 'max_samples', etc
dataset = load_from_hub("Voxel51/trex-dataset-23ep")

# Launch the App
session = fo.launch_app(dataset)
```

# Dataset Card for T-Rex Dataset (23-episode FiftyOne subset)

![T-Rex Dataset preview](https://huggingface.co/datasets/Voxel51/trex-dataset-23ep/resolve/main/trex-dataset.gif)

A 23-episode subset of the **T-Rex Dataset**, a large-scale, tactile-reactive bimanual
manipulation dataset collected via teleoperation on a Dexmate Vega-1 robot with two
Sharpa Wave dexterous hands. The full dataset (5,464 episodes, 1.5 TB) is published at
[zekaiwang/trex_dataset](https://huggingface.co/datasets/zekaiwang/trex_dataset); this
repo holds episodes `0`–`22` re-packaged as a self-contained LeRobotDataset v3.0 export
and loaded into FiftyOne for exploration.

## Dataset Details

### Dataset Description

- **Curated by:** Dantong Niu, Zhuoyang Liu, Zekai Wang, Boning Shao, Zhao-Heng Yin, Anirudh Pai, Yuvan Sharma, Stefano Saravalle, Ruijie Zheng, Jing Wang, Ryan Punamiya, Mengda Xu, Yuqi Xie, Yunfan Jiang, Letian Fu, Konstantinos Kallidromitis, Matteo Gioia, Junyi Zhang, Jiaxin Ge, Haiwen Feng, Fabio Galasso, Wei Zhan, David M. Chan, Yutong Bai, Roei Herzig, Jiahui Lei, Fei-Fei Li, Ken Goldberg, Jitendra Malik, Pieter Abbeel, Yuke Zhu, Danfei Xu, Jim Fan, Trevor Darrell
- **Shared by:** zekaiwang (original dataset); this FiftyOne subset shared by the FiftyOne community
- **Language(s):** English (task captions)
- **License:** MIT (© 2026 The Regents of the University of California)

### Dataset Sources

- **Repository:** [zekaiwang/trex_dataset](https://huggingface.co/datasets/zekaiwang/trex_dataset) · [T-Rex code](https://github.com/ZhuoyangLiu2005/T-Rex)
- **Paper:** [T-Rex: Tactile-Reactive Dexterous Manipulation (arXiv:2606.17055)](https://arxiv.org/abs/2606.17055)
- **Demo:** [Project Page](https://tactile-rex.github.io/)

## Uses

### Direct Use

Exploring and visualizing tactile-reactive bimanual manipulation episodes in the FiftyOne
App — inspecting synchronized RGB + tactile video streams alongside joint state/action
trajectories and per-fingertip force readings, filtering by task/motor primitive/object,
and prototyping data loaders before working with the full 5,464-episode dataset.

### Out-of-Scope Use

This 23-episode subset is not a statistically representative sample of the full dataset
(it is simply the first LeRobot data/video shard) and should not be used to draw
conclusions about task, object, or motor-primitive distributions across the full T-Rex
Dataset — load the full `zekaiwang/trex_dataset` repo for that.

## Dataset Structure

This is a **multimodal** FiftyOne dataset (`dataset.media_type == "multimodal"`) with
**23 samples**, one sample per episode. Each sample’s media (23 video streams) is not
copied into per-sample files; instead it is resolved through a `media_reference` that
points into the exported LeRobotDataset v3.0 source (`data/`, `videos/`, `meta/` in this
repo) at import time — this is how FiftyOne represents LeRobot episodes natively.

### Fields

| Field                             | FiftyOne type                          | Description                                                                                                                                                                                             |
|-----------------------------------|----------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `id`                              | `ObjectIdField`                        | FiftyOne sample id                                                                                                                                                                                      |
| `media_reference`                 | `MediaReferenceField`                  | Pointer into the LeRobot source’s `data/`, `videos/*`, and `meta/` files for this episode (chunk/file indexes, frame range, per-video timestamp ranges) — resolved on demand, not duplicated per sample |
| `tags`                            | `ListField(StringField)`               | FiftyOne sample tags (empty by default)                                                                                                                                                                 |
| `metadata`                        | `Metadata` (`size_bytes`, `mime_type`) | Standard FiftyOne sample metadata                                                                                                                                                                       |
| `created_at` / `last_modified_at` | `DateTimeField`                        | FiftyOne bookkeeping timestamps                                                                                                                                                                         |
| `episode_index`                   | `IntField`                             | Episode index within this subset (`0`–`22`, remapped on export from the source dataset’s original indices)                                                                                              |
| `task`                            | `StringField`                          | Primary task caption for the episode (verbatim from source `caption`)                                                                                                                                   |
| `tasks`                           | `ListField(StringField)`               | Full task list for the episode (LeRobot `tasks` array; length 1 for every episode here)                                                                                                                 |
| `length`                          | `IntField`                             | Number of frames in the episode (verbatim from source)                                                                                                                                                  |
| `duration`                        | `FloatField`                           | Episode duration in seconds (`length / fps`)                                                                                                                                                            |
| `robot_type`                      | `StringField`                          | `dexmate_vega1_and_sharpa_wave` (verbatim from source `meta/info.json`)                                                                                                                                 |
| `fps`                             | `FloatField`                           | Recording frame rate, 30.0 (verbatim from source)                                                                                                                                                       |

The per-frame numeric features (`observation.state` (58,), `action` (58,),
`observation.tactile_force` (60,)) and the 23 per-frame video streams are **not**
flattened into sample fields — they remain in the LeRobot `data/*.parquet` and
`videos/*/*.mp4` files referenced by `media_reference`, and are surfaced by the
FiftyOne App’s State & Action, Streams, and Statistics viewer tabs rather than as
queryable sample-level fields.

### Label types and why

There are no traditional detection/classification/segmentation labels in this dataset.
`task` is stored as a plain `StringField` rather than `fo.Classification` because task
captions here are free-form natural language instructions (5,370 unique across the full
dataset, 22 unique in this subset), not a fixed closed-set taxonomy — a `Classification`
label with `logits`/`confidence` semantics does not fit a free-text instruction.

### `dataset.info` contents

```python
{
    "lerobot": {
        "format": "LeRobotDataset",
        "format_major": 3,
        "episode_count": 5464,          # total episodes in the full source dataset
        "imported_episode_count": 23,   # episodes actually imported into this subset
        "skipped_episodes": [],
    }
}
```

### Parsing decisions

- **Which episodes, and why:** episodes `0`–`22` were selected because they are the
  largest contiguous, zero-gap block of episodes whose `data/chunk-000/file-000.parquet`
  shard **and** every one of the 23 `videos/<key>/chunk-000/file-000.mp4` shards are
  shared — i.e. the smallest set of source files that had to be downloaded to get a
  complete, non-truncated set of episodes, given a limited local disk budget. It is not
  a curated or stratified sample.
- **Re-export, not a thin reference to the original repo:** this repo is a
  self-contained LeRobotDataset v3.0 export (via FiftyOne’s `LeRobotDatasetExporter`),
  not a pointer back to `zekaiwang/trex_dataset`. Episode indices, frame `index`, and
  video/data chunk-file coordinates were rewritten during export so the 23 episodes are
  contiguous (`0`–`22`) and self-consistent; task indices were remapped to only the
  tasks actually present in this subset. Per-episode and global statistics
  (`meta/stats.json`, per-episode `stats/*` columns) were recomputed from the exported
  rows, not carried over from the source’s global stats.
- **Tactile video codec (read before decoding outside FiftyOne):** the 20 tactile
  streams (`observation.images.tactile_{left,right}_{raw,deform}_{finger}`) are encoded
  losslessly (`libx264 -qp 0`) because their pixel values are physically meaningful raw
  sensor/deformation readings. They are grayscale, full-range `yuvj420p`, which forces
  the H.264 **High 4:4:4 Predictive** profile — most browsers cannot decode this profile
  in a plain `<video>`/WebCodecs pipeline (no thumbnails in the HF preview or generic
  players), so use FiftyOne, ffmpeg, PyAV, or torchcodec to view them locally. The 3
  RGB streams (`head_left`, `left_wrist`, `right_wrist`) use standard limited-range
  `yuv420p`/BT.709 and preview normally everywhere.
- **No held-out or unlabeled split:** all 23 episodes are in a single `train` split with
  every episode language-annotated; nothing was withheld.

## Dataset Creation

### Curation Rationale

The full T-Rex Dataset was collected to study tactile-reactive dexterous manipulation —
pairing bimanual arm/hand joint trajectories with synchronized per-fingertip tactile
sensing so that policies can condition on touch, not just vision and proprioception. This
subset exists purely as a lightweight, disk-budget-friendly slice for exploration and
tooling in FiftyOne; it was not re-curated for content.

### Source Data

#### Data Collection and Processing

- **Robot:** Dexmate Vega-1 dual-arm mobile robot (7 actuated joints per arm) with two
  Sharpa Wave dexterous hands (5 fingertip tactile sensors each). During collection the
  wheels, torso, and head joints are fixed; only the 14 arm joints and two hands are
  actuated.
- **Cameras:** a head-mounted ZED X Mini stereo camera (left monocular RGB stream
  recorded) plus two wide-view ZED X One S monocular RGB wrist cameras, all at 640×360,
  30 fps.
- **Tactile sensing:** each hand’s 5 fingertip sensors contribute a raw sensor image
  (`tactile_*_raw_*`, 320×240) and an estimated deformation map (`tactile_*_deform_*`,
  240×240), both stored as lossless grayscale video, plus an estimated 6-axis net wrench
  per fingertip in `observation.tactile_force` (60,) = (left, right) × (thumb…pinky) ×
  (Fx, Fy, Fz, Mx, My, Mz).
- **Teleoperation:** Manus gloves capture fingertip positions retargeted to the Sharpa
  Wave hands via the manufacturer’s differential-inverse-kinematics package (Pinocchio +
  CasADi). Two VIVE trackers provide SE(3) wrist poses converted to arm joint commands
  via differential inverse kinematics (Pink), low-pass filtered, and tracked by the
  manufacturer’s low-level cascade PID controller. A 30 Hz high-level thread records
  observations and joint-space targets (`action`) while a 300 Hz low-level thread runs
  control; the dataset’s 30 fps matches the high-level loop.
- **`observation.state` / `action` layout (58,):** `[L_arm 7 | L_hand 22 | R_arm 7 | R_hand 22]` joint positions (`observation.state`) and target joint positions
  (`action`). Full per-dimension joint names are in the source `meta/info.json`
  (`features[*].names`).

#### Who are the source data producers?

Collected by the T-Rex authors (UC Berkeley and collaborating institutions; see author
list above) via in-person teleoperation with the hardware/software stack described above.

### Annotations

#### Annotation process

Each episode is labeled with a human-verified natural-language `caption`
(task instruction), a `motor_primitive` category (one of 22, e.g. `reach`,
`lift_and_place`), a canonical `object` name, and — only for `lift_and_place` episodes —
a canonical `target`/receptacle name (null otherwise). In this subset, `caption` is
surfaced as the sample-level `task` field.

#### Personal and Sensitive Information

None identified — the dataset contains robot joint/tactile sensor data and task
captions describing tabletop manipulation of everyday objects; no human subjects data.

## Citation

**BibTeX:**

```bibtex
@misc{trex2026,
  title={T-Rex: Tactile-Reactive Dexterous Manipulation},
  author={Dantong Niu and Zhuoyang Liu and Zekai Wang and Boning Shao and Zhao-Heng Yin and Anirudh Pai and Yuvan Sharma and Stefano Saravalle and Ruijie Zheng and Jing Wang and Ryan Punamiya and Mengda Xu and Yuqi Xie and Yunfan Jiang and Letian Fu and Konstantinos Kallidromitis and Matteo Gioia and Junyi Zhang and Jiaxin Ge and Haiwen Feng and Fabio Galasso and Wei Zhan and David M. Chan and Yutong Bai and Roei Herzig and Jiahui Lei and Fei-Fei Li and Ken Goldberg and Jitendra Malik and Pieter Abbeel and Yuke Zhu and Danfei Xu and Jim Fan and Trevor Darrell},
  year={2026},
  eprint={2606.17055},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  url={https://arxiv.org/abs/2606.17055},
}
```

**APA:**

Niu, D., Liu, Z., Wang, Z., Shao, B., Yin, Z.-H., Pai, A., Sharma, Y., Saravalle, S., Zheng, R., Wang, J., Punamiya, R., Xu, M., Xie, Y., Jiang, Y., Fu, L., Kallidromitis, K., Gioia, M., Zhang, J., Ge, J., Feng, H., Galasso, F., Zhan, W., Chan, D. M., Bai, Y., Herzig, R., Lei, J., Li, F.-F., Goldberg, K., Malik, J., Abbeel, P., Zhu, Y., Xu, D., Fan, J., & Darrell, T. (2026). *T-Rex: Tactile-Reactive Dexterous Manipulation*. arXiv:2606.17055.

## More Information

This is a 23-episode subset of the full [zekaiwang/trex_dataset](https://huggingface.co/datasets/zekaiwang/trex_dataset)
(5,464 episodes, ~50 hours, 1.5 TB), produced for local exploration under a limited
disk budget. See the source repo for the full dataset, the
[T-Rex Quick Start](https://github.com/ZhuoyangLiu2005/T-Rex/tree/main/dataset_quickstart)
tools, and the [Colab notebook](https://colab.research.google.com/github/ZhuoyangLiu2005/T-Rex/blob/main/dataset_quickstart/quickstart.ipynb).

## Dataset Card Authors

[Harpreet Sahota](https://huggingface.co/harpreetsahota)

## Dataset Card Contact

[Harpreet Sahota](https://huggingface.co/harpreetsahota)