Note

This is a Hugging Face dataset. For large datasets, ensure huggingface_hub>=1.1.3 to avoid rate limits. Learn more in the Hugging Face integration docs.

This is a FiftyOne dataset with 302 samples.

Installation#

If you haven’t already, install FiftyOne:

pip install -U fiftyone

Usage#

import fiftyone as fo
from fiftyone.utils.huggingface import load_from_hub

# Load the dataset
# Note: other available arguments include 'max_samples', etc
dataset = load_from_hub("Voxel51/fmb-multi-302ep")

# Launch the App
session = fo.launch_app(dataset)

Dataset Card for FMB multi-object (302-episode FiftyOne subset)#

fmb_multi preview

A 302-episode subset of fmb_multi, an unofficial community LeRobotDataset v3 port of the FMB (Functional Manipulation Benchmark) multi-object manipulation demonstrations — contact-rich assembly-board tasks recorded with a Franka Panda arm. The full fmb_multi repo (1,804 episodes across 3 assembly boards) is published at robot-lev/fmb_multi; this repo holds episodes 0–301 re-packaged as a self-contained LeRobotDataset v3.0 export and loaded into FiftyOne for exploration.

Dataset Details#

Dataset Description#

Curated by: Jianlan Luo, Charles Xu, Fangchen Liu, Liam Tan, Zipeng Lin, Jeffrey Wu, Pieter Abbeel, Sergey Levine (original FMB dataset); LeRobot v3 port by the community (lvjonok/fmb-lerobot-port)
Shared by: robot-lev (LeRobot v3 port); this FiftyOne subset shared by the FiftyOne community
Language(s): English (skill primitive labels)
License: CC BY 4.0

Dataset Sources#

Repository: robot-lev/fmb_multi · original release: charlesxu0124/functional-manipulation-benchmark · single-object counterpart: robot-lev/fmb · port code: lvjonok/fmb-lerobot-port
Paper: FMB: a Functional Manipulation Benchmark for Generalizable Robotic Learning (arXiv:2401.08553)
Demo: Project page

Uses#

Direct Use#

Exploring and visualizing contact-rich robotic assembly episodes in the FiftyOne App — inspecting 4 synchronized camera views (2 side, 2 wrist) alongside joint/end-effector state, end-effector force/torque, jacobian, and cartesian actions, filtered by assembly board or skill primitive; prototyping data loaders before working with the full 1,804-episode fmb_multi repo or the original FMB release (which additionally includes depth).

Out-of-Scope Use#

This 302-episode subset is not a statistically representative sample of the full dataset (it is simply the first contiguous block of episodes whose data and all 4 video streams share the first storage shard) and should not be used to draw conclusions about the overall distribution of boards, objects, or trajectories across the full fmb_multi dataset — though it does happen to include a roughly even mix of all 3 boards (100/100/102 episodes). This subset contains no depth data (see Parsing decisions) — not suitable for depth-conditioned policy work.

Dataset Structure#

This is a multimodal FiftyOne dataset (dataset.media_type == "multimodal") with 302 samples, one sample per episode. Each sample’s media (4 video streams) is not copied into per-sample files; instead it is resolved through a media_reference that points into the exported LeRobotDataset v3.0 source (data/, videos/, meta/ in this repo) at import time — this is how FiftyOne represents LeRobot episodes natively.

Fields#

Field	FiftyOne type	Description
`id`	`ObjectIdField`	FiftyOne sample id
`media_reference`	`MediaReferenceField`	Pointer into the LeRobot source’s `data/`, `videos/*`, and `meta/` files for this episode (chunk/file indexes, frame range, per-video timestamp ranges) — resolved on demand, not duplicated per sample
`tags`	`ListField(StringField)`	FiftyOne sample tags (empty by default)
`metadata`	`Metadata` (`size_bytes`, `mime_type`)	Standard FiftyOne sample metadata
`created_at` / `last_modified_at`	`DateTimeField`	FiftyOne bookkeeping timestamps
`episode_index`	`IntField`	Episode index within this subset (`0`–`301`, contiguous with the source dataset’s original indices since this block happened to start at 0)
`task`	`StringField`	The skill primitive active at the episode’s first frame, one of `insert`/`go_to_board`/`grasp`/`place_on_fixture`/`move_up`/`regrasp` — not a full description of the episode; see “Parsing decisions”
`tasks`	`ListField(StringField)`	The full, order-independent set of skill primitives that occur somewhere in the episode (typically all 6, since a full assembly trajectory passes through each primitive)
`length`	`IntField`	Number of frames in the episode (verbatim from source)
`duration`	`FloatField`	Episode duration in seconds (`length / fps`)
`robot_type`	`StringField`	`franka` (verbatim from source `meta/info.json`)
`fps`	`FloatField`	Recording frame rate, 10.0 — nominal, not measured (see Parsing decisions)

The per-frame numeric features and the 4 per-frame video streams are not flattened into sample fields — they remain in the LeRobot data/*.parquet and videos/*/*.mp4 files referenced by media_reference, and are surfaced by the FiftyOne App’s State & Action, Streams, and Statistics viewer tabs rather than as queryable sample-level fields. Per source meta/info.json, these per-frame features are:

feature	shape	description
`observation.state`	`(28,)`	concatenation of `joint_position` (7) + joint velocity (7) + `ee_pose` (7) + EE velocity (6) + `gripper` (1)
`observation.state.joint_position`	`(7,)`	joint positions
`observation.state.ee_pose`	`(7,)`	end-effector pose: xyz + quaternion, base frame
`observation.state.gripper`	`(1,)`	gripper state, `0`=open, `1`=closed
`observation.force`	`(3,)`	end-effector force, end-effector frame
`observation.torque`	`(3,)`	end-effector torque, end-effector frame
`observation.jacobian`	`(42,)`	robot Jacobian (6×7), flattened
`action`	`(7,)`	commanded cartesian action: xyz, rpy, gripper
`observation.images.{side_1, side_2, wrist_1, wrist_2}`	`(256, 256, 3)` video, AV1	2 side cameras + 2 wrist cameras, RGB (converted from source BGR)

Label types and why#

There are no traditional detection/classification/segmentation labels. task and tasks are plain string fields (not fo.Classification) because the underlying labels are a small fixed set of skill-primitive names (6 total: insert, go_to_board, grasp, place_on_fixture, move_up, regrasp) that describe temporal phases within an episode rather than a single per-episode category — collapsing them to one Classification per episode would misrepresent that every full trajectory passes through most or all of these phases in sequence.

`dataset.info` contents#

{
    "lerobot": {
        "format": "LeRobotDataset",
        "format_major": 3,
        "episode_count": 1804,           # total episodes in the full source dataset
        "imported_episode_count": 302,   # episodes actually imported into this subset
        "skipped_episodes": [],
    }
}

Parsing decisions#

Which episodes, and why: episodes 0–301 were selected because they are the largest contiguous, zero-gap block of episodes whose data/chunk-000/file-000.parquet shard and every one of the 4 videos/<key>/chunk-000/file-000.mp4 shards are shared — i.e. the smallest set of source files that had to be downloaded (~770 MB) to get a complete, non-truncated set of episodes, given a limited local disk budget. It happens to include a near-even split across all 3 assembly boards (100/100/102 episodes for board_1/board_2/board_3 respectively, per the source meta/fmb_episodes.json), but this was incidental, not a stratified selection.
task is a single frame’s label, not an episode summary. The source’s per-episode tasks list (in meta/tasks.parquet/episode metadata) enumerates every skill primitive active during the trajectory (go_to_board → grasp → insert → move_up → place_on_fixture → regrasp, though not always all 6 or in that exact order); LeRobot’s standard task field takes just one value (in this port, the primitive tagged at frame 0’s task_index). Use tasks, not task, if you need the full set of primitives an episode covers.
board/object_id/trajectory_id are not FiftyOne sample fields. They live in the source’s meta/fmb_episodes.json (not carried into this LeRobot v3 export or into FiftyOne’s field schema): each entry has stem, n_frames, board (board_1/board_2/board_3), subset ("multi_object" for all episodes here), object_id, trajectory_id, primitives (the per-episode primitive list), and global_index. If you need per-episode board/object metadata, join back to the original source repo’s meta/fmb_episodes.json by episode_index.
Depth dropped, by the upstream port (not by this FiftyOne subset): the source FMB release includes 4 depth maps per frame; fmb_multi’s LeRobot v3 port is RGB + F/T + proprioception + action only.
BGR → RGB, done by the upstream port: FMB’s original .npy demonstrations store images in BGR; this port converts them to RGB before video encoding.
fps: 10 is nominal, not measured. Per the source README, the original FMB .npy demonstrations carry no per-frame timestamps; frames map 1:1 onto a synthetic 10 fps grid rather than reflecting an actual recording rate.
Action is the FMB commanded action as-is — no next-pose reconstruction or relabeling was applied by the port.
Re-export, not a thin reference to the original repo: this repo is a self-contained LeRobotDataset v3.0 export (via FiftyOne’s LeRobotDatasetExporter), not a pointer back to robot-lev/fmb_multi. Task indices were remapped to only the tasks actually present in this subset (all 6 of the source’s 6). Per-episode and global statistics were recomputed from the exported rows, not carried over from the source’s global stats.

Dataset Creation#

Curation Rationale#

FMB was designed as a benchmark for generalizable robotic manipulation learning, specifically targeting contact-rich, multi-stage assembly tasks (insertion onto fixtures across multiple board/object variants) that stress both perception and force-aware control — harder to generalize than simple pick-place. This LeRobot v3 port converts each demonstration .npy into one LeRobot episode; this FiftyOne subset exists purely as a lightweight, disk-budget-friendly slice for exploration and tooling, not a re-curation of scenario content.

Source Data#

Data Collection and Processing#

Robot: Franka Panda arm.
Cameras: 4 fixed viewpoints — side_1, side_2 (external) and wrist_1, wrist_2 (wrist-mounted) — each at 256×256 RGB, AV1-encoded in this port (originally BGR in the source .npy files).
Sensing: joint position/velocity, end-effector pose/velocity, gripper state, end-effector force/torque (EE frame), and the flattened 6×7 robot Jacobian.
Task structure: each demonstration is a full assembly trajectory decomposed into discrete skill primitives (approach the board, grasp the object, insert it, move up, place it on a fixture, regrasp), spanning 3 distinct assembly boards (board_1/board_2/board_3) in the multi-object subset.
Original FMB collection (per the paper): demonstrations collected across multiple objects/boards for the Functional Manipulation Benchmark, designed to test generalization of manipulation policies across object/task variation.

Who are the source data producers?#

Collected by the FMB authors (UC Berkeley) as part of the Functional Manipulation Benchmark project.

Annotations#

Annotation process#

Skill-primitive segmentation (primitives/task_index) comes from the original FMB demonstration collection/labeling process, not a separate annotation pass added by this port. The LeRobot v3 conversion (lvjonok/fmb-lerobot-port) maps each frame’s original primitive label to a task_index and derives the per-episode tasks set from it.

Personal and Sensitive Information#

None identified — the dataset contains robot proprioception/force-torque data and RGB video of tabletop assembly-board manipulation; no human subjects data.

Citation#

BibTeX:

@article{luo2024fmb,
  title   = {FMB: a Functional Manipulation Benchmark for Generalizable Robotic Learning},
  author  = {Luo, Jianlan and Xu, Charles and Liu, Fangchen and Tan, Liam and Lin, Zipeng and Wu, Jeffrey and Abbeel, Pieter and Levine, Sergey},
  journal = {arXiv preprint arXiv:2401.08553},
  year    = {2024}
}

APA:

Luo, J., Xu, C., Liu, F., Tan, L., Lin, Z., Wu, J., Abbeel, P., & Levine, S. (2024). FMB: a Functional Manipulation Benchmark for Generalizable Robotic Learning. arXiv:2401.08553.

More Information#

This is a 302-episode subset of the community LeRobot v3 port robot-lev/fmb_multi (1,804 episodes), itself a derivative of the original FMB release, produced for local exploration under a limited disk budget. See the FMB project page for the full dataset (including depth) and the single-object counterpart robot-lev/fmb.

Dataset Card Authors#

Harpreet Sahota

Dataset Card Contact#