#### NOTE
This is a **Hugging Face dataset**. For large datasets, ensure `huggingface_hub>=1.1.3` to avoid rate limits. Learn more in the Hugging Face integration docs.
# ABC-130k → FiftyOne (Native Multimodal MCAP)
Use [FiftyOne](https://github.com/voxel51/fiftyone) to explore **[ABC-130k](https://abc.bot/)**,
the largest open bimanual robot-teleoperation dataset (134,806 episodes / 195 tasks / 3,553 hours).
Each episode renders **natively** from its `.mcap`: synchronized multi-camera video with robot
arm/end-effector telemetry charted against the video timeline.
- **This dataset:** [`Voxel51/ABC-130k`](https://huggingface.co/datasets/Voxel51/ABC-130k) — load directly with `load_from_hub`
- **Loader (FiftyOne):** [`github.com/Burhan-Q/fiftyone-abc130k`](https://github.com/Burhan-Q/fiftyone-abc130k)
- **Full dataset:** [`XDOF/ABC-130k`](https://huggingface.co/datasets/XDOF/ABC-130k)
- **Original project:** [abc.bot](https://abc.bot/) · [paper](https://arxiv.org/abs/2606.27375) · [training code](https://github.com/amazon-far/abc)
> This is an **unofficial**, **modified** subset of ABC-130k repackaged for FiftyOne (Apache-2.0, same as
> the source). The `.fo.mcap` episodes are **not verbatim** — see License & attribution.
> All credit for the dataset belongs to the original authors (see [Citation](#citation)).
## Installation
```bash
pip install "fiftyone[multimodal]>=1.19.0" huggingface_hub
```
> [!IMPORTANT]
> Native multimodal MCAP support needs **`fiftyone[multimodal]>=1.19.0`** (the `multimodal` extra *and*
> version ≥ 1.19.0), and **`VFF_MULTIMODAL=1` must be set before `fiftyone` is imported**.
## Usage
Want a ready-made validation subset? Load the public
**[`Voxel51/ABC-130k`](https://huggingface.co/datasets/Voxel51/ABC-130k)** dataset. It’s a diversity-first
subset of the val split (~40 episodes across ~40 tasks, all-H.264, with telemetry plot sidecars) and loads
straight into FiftyOne. Load it with:
```python
# NOTE: set VFF_MULTIMODAL=1 environment variable first!
import fiftyone as fo
import fiftyone.utils.huggingface as fouh
dataset = fouh.load_from_hub("Voxel51/ABC-130k", name="ABC-130k", persistent=True)
fo.launch_app(dataset)
```
Loading the full ABC-130k dataset
Use the loader repo [`github.com/Burhan-Q/fiftyone-abc130k`](https://github.com/Burhan-Q/fiftyone-abc130k) (`abc130k.py` + `abc130k_reencode.py`) to enumerate, download, and ingest any
split or custom subset from the gated **`XDOF/ABC-130k`** source into a FiftyOne dataset. Follow the
same setup instructions outlined above, plus ensure to install **`ffmpeg`** (with libx264) and make
it available on your `PATH`.
Command line:
```bash
# Diversity-first val subset: one small (H.264) episode from each of 40 tasks (~4 GB)
python abc130k.py --name ABC-130k --split val --max-tasks 40 --episodes-per-task 1
# The entire val split, H.264 only (large; H.265 episodes are skipped)
python abc130k.py --name ABC-130k-full --split val
# Preview the selection without downloading
python abc130k.py --name ABC-130k --split val --max-tasks 40 --dry-run
```
**Flags** (`--name` is required; all others are optional):
| Flag | Value | Description |
|-----------------------|---------------------------------|----------------------------------------------------------------------------|
| `--name` | string (**required**) | FiftyOne dataset name to create or populate |
| `--split` | `val` | `train` (default `val`) | Which split to load |
| `--max-tasks` | int | Keep at most N distinct tasks, spread across the split for diversity |
| `--episodes-per-task` | int | Keep at most K episodes per task |
| `--max-size-mb` | float | Skip episodes larger than this before downloading (cheap H.265 pre-filter) |
| `--budget-gb` | float | Stop selecting once the cumulative download would exceed this cap |
| `--largest-first` | flag (no value) | Prefer larger episodes per task (default prefers smaller / H.264) |
| `--fps-cap` | float (default `45`) | Re-encode episodes whose camera fps exceeds this to 30 fps; `0` disables |
| `--no-plot` | flag (no value) | Skip the JSON `.plot` telemetry sidecar channels |
| `--include-h265` | flag (no value) | Also load H.265 episodes (they will not render) |
| `--overwrite` | flag (no value) | Delete an existing dataset of the same name first |
| `--not-persistent` | flag (no value) | Do not persist the dataset across sessions |
| `--data-dir` | path (default `_data/abc-130k`) | Local directory to download episodes into |
| `--token` | string | HF token (falls back to the `HF_TOKEN` env var) |
| `--dry-run` | flag (no value) | Print the selection without downloading anything |
Python:
```python
import abc130k # sets VFF_MULTIMODAL before importing fiftyone
import fiftyone as fo
dataset = abc130k.download_and_load("ABC-130k", split="val", max_tasks=40, episodes_per_task=1)
session = fo.launch_app(dataset)
```
### Sensor-data plotting
> [!IMPORTANT]
> The loader points each sample at an auto-generated **`episode.fo.mcap`** (the raw episode plus JSON
> `.plot` telemetry channels) — without it, the arm/EE telemetry plots nothing. It ≈ doubles
> on-disk size but is self-contained, so the original `episode.mcap` files can be deleted to reclaim space.
Every numeric telemetry field (`position`, `velocity`, `torque`, gripper opening, end-effector `pose`)
is stored as a `repeated double` **array**, which FiftyOne’s scalar-only MCAP **Plot** panel skips. The
loader mirrors each into flat scalar **`.plot`** JSON channels at the source timestamps (on by
default; `--no-plot` disables):
- array fields → per-index scalars: `position_0…5`, `velocity_0…6`, `torque_0…6`, gripper `position_0`
- 4×4 end-effector `pose` → **`pose_x/y/z`** (translation, m) + **`pose_roll/pitch/yaw`** (orientation, rad)
To view: open a sample modal → **Add tile → Plot** → expand e.g. `/left-arm-state.plot` → check
`pose_x/y/z` (or joint `position_*`). The series track the playhead alongside the camera tiles.
### Rendering notes (H.264 & frame rate)
- **The viewer decodes H.264 only.** In ABC-130k, RealSense stations are all-H.264 (render); ZED-X
stations are all-H.265 (show “No preview frames”). `h264_only=True` (default) downloads, detects, and
skips H.265 episodes; add `--max-size-mb` to avoid downloading them, or `--include-h265` to keep them.
- **High-fps episodes are auto-converted to 30 fps.** ~60 fps episodes stall the viewer (“Waiting for
H.264 keyframe”); those above `--fps-cap` (default 45) are re-encoded to 30 fps (GOP-30, no B-frames).
`source_fps` / `reencoded_to_30fps` record what happened.
### Dataset facts
- **val split:** 189 tasks / 1,671 episodes / ~309 GB; layout
`data///episode_/episode.mcap` (one `.mcap` = one sample).
- The public HF **tree API** returns file paths + LFS sizes **without a token**, so a size-bounded
subset can be planned before authenticating (how `--dry-run` works).