Note

This is a Hugging Face dataset. For large datasets, ensure huggingface_hub>=1.1.3 to avoid rate limits. Learn more in the Hugging Face integration docs.

Hugging Face

ABC-130k → FiftyOne (Native Multimodal MCAP)#

Use FiftyOne to explore ABC-130k, the largest open bimanual robot-teleoperation dataset (134,806 episodes / 195 tasks / 3,553 hours). Each episode renders natively from its .mcap: synchronized multi-camera video with robot arm/end-effector telemetry charted against the video timeline.

This is an unofficial, modified subset of ABC-130k repackaged for FiftyOne (Apache-2.0, same as the source). The .fo.mcap episodes are not verbatim — see License & attribution. All credit for the dataset belongs to the original authors (see Citation).

Installation#

pip install "fiftyone[multimodal]>=1.19.0" huggingface_hub

[!IMPORTANT] Native multimodal MCAP support needs fiftyone[multimodal]>=1.19.0 (the multimodal extra and version ≥ 1.19.0), and VFF_MULTIMODAL=1 must be set before fiftyone is imported.

Usage#

Want a ready-made validation subset? Load the public Voxel51/ABC-130k dataset. It’s a diversity-first subset of the val split (~40 episodes across ~40 tasks, all-H.264, with telemetry plot sidecars) and loads straight into FiftyOne. Load it with:

# NOTE: set VFF_MULTIMODAL=1 environment variable first!

import fiftyone as fo
import fiftyone.utils.huggingface as fouh

dataset = fouh.load_from_hub("Voxel51/ABC-130k", name="ABC-130k", persistent=True)
fo.launch_app(dataset)
Loading the full ABC-130k dataset

Use the loader repo github.com/Burhan-Q/fiftyone-abc130k (abc130k.py + abc130k_reencode.py) to enumerate, download, and ingest any split or custom subset from the gated XDOF/ABC-130k source into a FiftyOne dataset. Follow the same setup instructions outlined above, plus ensure to install ffmpeg (with libx264) and make it available on your PATH.

Command line:

# Diversity-first val subset: one small (H.264) episode from each of 40 tasks (~4 GB)
python abc130k.py --name ABC-130k --split val --max-tasks 40 --episodes-per-task 1

# The entire val split, H.264 only (large; H.265 episodes are skipped)
python abc130k.py --name ABC-130k-full --split val

# Preview the selection without downloading
python abc130k.py --name ABC-130k --split val --max-tasks 40 --dry-run

Flags (--name is required; all others are optional):

Flag

Value

Description

--name

string (required)

FiftyOne dataset name to create or populate

--split

val | train (default val)

Which split to load

--max-tasks

int

Keep at most N distinct tasks, spread across the split for diversity

--episodes-per-task

int

Keep at most K episodes per task

--max-size-mb

float

Skip episodes larger than this before downloading (cheap H.265 pre-filter)

--budget-gb

float

Stop selecting once the cumulative download would exceed this cap

--largest-first

flag (no value)

Prefer larger episodes per task (default prefers smaller / H.264)

--fps-cap

float (default 45)

Re-encode episodes whose camera fps exceeds this to 30 fps; 0 disables

--no-plot

flag (no value)

Skip the JSON <topic>.plot telemetry sidecar channels

--include-h265

flag (no value)

Also load H.265 episodes (they will not render)

--overwrite

flag (no value)

Delete an existing dataset of the same name first

--not-persistent

flag (no value)

Do not persist the dataset across sessions

--data-dir

path (default _data/abc-130k)

Local directory to download episodes into

--token

string

HF token (falls back to the HF_TOKEN env var)

--dry-run

flag (no value)

Print the selection without downloading anything

Python:

import abc130k  # sets VFF_MULTIMODAL before importing fiftyone
import fiftyone as fo

dataset = abc130k.download_and_load("ABC-130k", split="val", max_tasks=40, episodes_per_task=1)
session = fo.launch_app(dataset)

Sensor-data plotting#

[!IMPORTANT] The loader points each sample at an auto-generated episode.fo.mcap (the raw episode plus JSON <topic>.plot telemetry channels) — without it, the arm/EE telemetry plots nothing. It ≈ doubles on-disk size but is self-contained, so the original episode.mcap files can be deleted to reclaim space.

Every numeric telemetry field (position, velocity, torque, gripper opening, end-effector pose) is stored as a repeated double array, which FiftyOne’s scalar-only MCAP Plot panel skips. The loader mirrors each into flat scalar <topic>.plot JSON channels at the source timestamps (on by default; --no-plot disables):

  • array fields → per-index scalars: position_0…5, velocity_0…6, torque_0…6, gripper position_0

  • 4×4 end-effector pose → pose_x/y/z (translation, m) + pose_roll/pitch/yaw (orientation, rad)

To view: open a sample modal → Add tile → Plot → expand e.g. /left-arm-state.plot → check pose_x/y/z (or joint position_*). The series track the playhead alongside the camera tiles.

Rendering notes (H.264 & frame rate)#

  • The viewer decodes H.264 only. In ABC-130k, RealSense stations are all-H.264 (render); ZED-X stations are all-H.265 (show “No preview frames”). h264_only=True (default) downloads, detects, and skips H.265 episodes; add --max-size-mb to avoid downloading them, or --include-h265 to keep them.

  • High-fps episodes are auto-converted to 30 fps. ~60 fps episodes stall the viewer (“Waiting for H.264 keyframe”); those above --fps-cap (default 45) are re-encoded to 30 fps (GOP-30, no B-frames). source_fps / reencoded_to_30fps record what happened.

Dataset facts#

  • val split: 189 tasks / 1,671 episodes / ~309 GB; layout data/<split>/<task>/episode_<uuid>/episode.mcap (one .mcap = one sample).

  • The public HF tree API returns file paths + LFS sizes without a token, so a size-bounded subset can be planned before authenticating (how --dry-run works).

What you get#

One FiftyOne sample per episode, media_type = "multimodal" — Foxglove CompressedVideo cameras (1 top + 2 wrist) plus RobotState/GripperState protobuf telemetry per arm, for both -state and -action. No depth / LiDAR / point clouds. Per-sample fields:

Field

Type

Description

task / episode_id

StringField

Task name / source episode UUID

split

StringField

val or train (also a sample tag)

size_mb

FloatField

Source episode size

station

StringField

mono (RealSense/H.264) or stereo (ZED-X/H.265)

topics / camera_topics

ListField

All MCAP topics / camera streams

plot_topics

ListField

The JSON <topic>.plot telemetry channels

camera_codecs

DictField

{topic: format} (e.g. h264)

duration_s / n_messages

FloatField / IntField

Episode duration / message count

source_fps / reencoded_to_30fps

FloatField / BooleanField

Original fps / whether re-encoded to 30 fps

License & attribution#

ABC-130k is released under the Apache License 2.0 by its original authors (abc.bot · amazon-far/abc). This redistribution keeps the same license; see the source Hugging Face dataset page for the original terms.

Modified subset. The .fo.mcap files here are not verbatim ABC-130k episodes: camera streams in the ~60 fps episodes (2 of 40) are re-encoded to 30 fps, and every episode carries added JSON <topic>.plot telemetry channels for the FiftyOne App’s Plot panel. All other robot/camera data matches the source. The loader code that produced them lives at github.com/Burhan-Q/fiftyone-abc130k.

Citation#

Please use the canonical citation from abc.bot / the paper:

@misc{abc2026,
  title         = {Scalable Behavior Cloning with Open Data, Training, and Evaluation},
  author        = {Arthur Allshire and Himanshu Gaurav Singh and Ritvik Singh and Adam Rashid and Hongsuk Choi and David McAllister and Justin Yu and Yiyuan Chen and Huang Huang and Pieter Abbeel and Xi Chen and Rocky Duan and Phillip Isola and Jitendra Malik and Fred Shentu and Guanya Shi and Philipp Wu and Angjoo Kanazawa},
  year          = {2026},
  eprint        = {2606.27375},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  doi           = {10.48550/arXiv.2606.27375},
  url           = {https://arxiv.org/abs/2606.27375},
}