#### NOTE
This is a **Hugging Face dataset**. For large datasets, ensure `huggingface_hub>=1.1.3` to avoid rate limits. Learn more in the Hugging Face integration docs.
This is a [FiftyOne](https://github.com/voxel51/fiftyone) dataset with 302 samples.
# Installation
If you haven’t already, install FiftyOne:
```bash
pip install -U fiftyone
```
# Usage
```python
import fiftyone as fo
from fiftyone.utils.huggingface import load_from_hub
# Load the dataset
# Note: other available arguments include 'max_samples', etc
dataset = load_from_hub("Voxel51/fmb-multi-302ep")
# Launch the App
session = fo.launch_app(dataset)
```
# Dataset Card for FMB multi-object (302-episode FiftyOne subset)
A 302-episode subset of **fmb_multi**, an unofficial community LeRobotDataset v3 port of
the **FMB (Functional Manipulation Benchmark)** multi-object manipulation demonstrations —
contact-rich assembly-board tasks recorded with a Franka Panda arm. The full `fmb_multi`
repo (1,804 episodes across 3 assembly boards) is published at
[robot-lev/fmb_multi](https://huggingface.co/datasets/robot-lev/fmb_multi); this repo
holds episodes `0`–`301` re-packaged as a self-contained LeRobotDataset v3.0 export and
loaded into FiftyOne for exploration.
## Dataset Details
### Dataset Description
- **Curated by:** Jianlan Luo, Charles Xu, Fangchen Liu, Liam Tan, Zipeng Lin, Jeffrey Wu, Pieter Abbeel, Sergey Levine (original FMB dataset); LeRobot v3 port by the community (`lvjonok/fmb-lerobot-port`)
- **Shared by:** robot-lev (LeRobot v3 port); this FiftyOne subset shared by the FiftyOne community
- **Language(s):** English (skill primitive labels)
- **License:** CC BY 4.0
### Dataset Sources
- **Repository:** [robot-lev/fmb_multi](https://huggingface.co/datasets/robot-lev/fmb_multi) · original release: [charlesxu0124/functional-manipulation-benchmark](https://huggingface.co/datasets/charlesxu0124/functional-manipulation-benchmark) · single-object counterpart: [robot-lev/fmb](https://huggingface.co/datasets/robot-lev/fmb) · port code: [lvjonok/fmb-lerobot-port](https://github.com/lvjonok/fmb-lerobot-port)
- **Paper:** [FMB: a Functional Manipulation Benchmark for Generalizable Robotic Learning (arXiv:2401.08553)](https://arxiv.org/abs/2401.08553)
- **Demo:** [Project page](https://functional-manipulation-benchmark.github.io)
## Uses
### Direct Use
Exploring and visualizing contact-rich robotic assembly episodes in the FiftyOne App —
inspecting 4 synchronized camera views (2 side, 2 wrist) alongside joint/end-effector
state, end-effector force/torque, jacobian, and cartesian actions, filtered by assembly
board or skill primitive; prototyping data loaders before working with the full
1,804-episode `fmb_multi` repo or the original FMB release (which additionally includes
depth).
### Out-of-Scope Use
This 302-episode subset is not a statistically representative sample of the full dataset
(it is simply the first contiguous block of episodes whose data and all 4 video streams
share the first storage shard) and should not be used to draw conclusions about the
overall distribution of boards, objects, or trajectories across the full `fmb_multi`
dataset — though it does happen to include a roughly even mix of all 3 boards (100/100/102
episodes). This subset contains no depth data (see Parsing decisions) — not suitable for
depth-conditioned policy work.
## Dataset Structure
This is a **multimodal** FiftyOne dataset (`dataset.media_type == "multimodal"`) with
**302 samples**, one sample per episode. Each sample’s media (4 video streams) is not
copied into per-sample files; instead it is resolved through a `media_reference` that
points into the exported LeRobotDataset v3.0 source (`data/`, `videos/`, `meta/` in this
repo) at import time — this is how FiftyOne represents LeRobot episodes natively.
### Fields
| Field | FiftyOne type | Description |
|-----------------------------------|----------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `id` | `ObjectIdField` | FiftyOne sample id |
| `media_reference` | `MediaReferenceField` | Pointer into the LeRobot source’s `data/`, `videos/*`, and `meta/` files for this episode (chunk/file indexes, frame range, per-video timestamp ranges) — resolved on demand, not duplicated per sample |
| `tags` | `ListField(StringField)` | FiftyOne sample tags (empty by default) |
| `metadata` | `Metadata` (`size_bytes`, `mime_type`) | Standard FiftyOne sample metadata |
| `created_at` / `last_modified_at` | `DateTimeField` | FiftyOne bookkeeping timestamps |
| `episode_index` | `IntField` | Episode index within this subset (`0`–`301`, contiguous with the source dataset’s original indices since this block happened to start at 0) |
| `task` | `StringField` | The skill primitive active at the episode’s first frame, one of `insert`/`go_to_board`/`grasp`/`place_on_fixture`/`move_up`/`regrasp` — **not** a full description of the episode; see “Parsing decisions” |
| `tasks` | `ListField(StringField)` | The full, order-independent set of skill primitives that occur somewhere in the episode (typically all 6, since a full assembly trajectory passes through each primitive) |
| `length` | `IntField` | Number of frames in the episode (verbatim from source) |
| `duration` | `FloatField` | Episode duration in seconds (`length / fps`) |
| `robot_type` | `StringField` | `franka` (verbatim from source `meta/info.json`) |
| `fps` | `FloatField` | Recording frame rate, 10.0 — **nominal**, not measured (see Parsing decisions) |
The per-frame numeric features and the 4 per-frame video streams are **not** flattened
into sample fields — they remain in the LeRobot `data/*.parquet` and `videos/*/*.mp4`
files referenced by `media_reference`, and are surfaced by the FiftyOne App’s State &
Action, Streams, and Statistics viewer tabs rather than as queryable sample-level fields.
Per source `meta/info.json`, these per-frame features are:
| feature | shape | description |
|---------------------------------------------------------|----------------------------|--------------------------------------------------------------------------------------------------------------|
| `observation.state` | `(28,)` | concatenation of `joint_position` (7) + joint velocity (7) + `ee_pose` (7) + EE velocity (6) + `gripper` (1) |
| `observation.state.joint_position` | `(7,)` | joint positions |
| `observation.state.ee_pose` | `(7,)` | end-effector pose: xyz + quaternion, base frame |
| `observation.state.gripper` | `(1,)` | gripper state, `0`=open, `1`=closed |
| `observation.force` | `(3,)` | end-effector force, **end-effector frame** |
| `observation.torque` | `(3,)` | end-effector torque, **end-effector frame** |
| `observation.jacobian` | `(42,)` | robot Jacobian (6×7), flattened |
| `action` | `(7,)` | commanded cartesian action: xyz, rpy, gripper |
| `observation.images.{side_1, side_2, wrist_1, wrist_2}` | `(256, 256, 3)` video, AV1 | 2 side cameras + 2 wrist cameras, RGB (converted from source BGR) |
### Label types and why
There are no traditional detection/classification/segmentation labels. `task` and
`tasks` are plain string fields (not `fo.Classification`) because the underlying labels
are a small fixed set of skill-primitive names (6 total: `insert`, `go_to_board`,
`grasp`, `place_on_fixture`, `move_up`, `regrasp`) that describe **temporal phases within
an episode** rather than a single per-episode category — collapsing them to one
`Classification` per episode would misrepresent that every full trajectory passes
through most or all of these phases in sequence.
### `dataset.info` contents
```python
{
"lerobot": {
"format": "LeRobotDataset",
"format_major": 3,
"episode_count": 1804, # total episodes in the full source dataset
"imported_episode_count": 302, # episodes actually imported into this subset
"skipped_episodes": [],
}
}
```
### Parsing decisions
- **Which episodes, and why:** episodes `0`–`301` were selected because they are the
largest contiguous, zero-gap block of episodes whose `data/chunk-000/file-000.parquet`
shard **and** every one of the 4 `videos//chunk-000/file-000.mp4` shards are
shared — i.e. the smallest set of source files that had to be downloaded (~770 MB) to
get a complete, non-truncated set of episodes, given a limited local disk budget. It
happens to include a near-even split across all 3 assembly boards (100/100/102
episodes for board_1/board_2/board_3 respectively, per the source
`meta/fmb_episodes.json`), but this was incidental, not a stratified selection.
- **`task` is a single frame’s label, not an episode summary.** The source’s per-episode
`tasks` list (in `meta/tasks.parquet`/episode metadata) enumerates every skill
primitive active during the trajectory (`go_to_board → grasp → insert → move_up → place_on_fixture → regrasp`, though not always all 6 or in that exact order); LeRobot’s
standard `task` field takes just one value (in this port, the primitive tagged at
frame 0’s `task_index`). Use `tasks`, not `task`, if you need the full set of
primitives an episode covers.
- **`board`/`object_id`/`trajectory_id` are not FiftyOne sample fields.** They live in
the source’s `meta/fmb_episodes.json` (not carried into this LeRobot v3 export or into
FiftyOne’s field schema): each entry has `stem`, `n_frames`, `board`
(`board_1`/`board_2`/`board_3`), `subset` (`"multi_object"` for all episodes here),
`object_id`, `trajectory_id`, `primitives` (the per-episode primitive list), and
`global_index`. If you need per-episode board/object metadata, join back to the
original source repo’s `meta/fmb_episodes.json` by `episode_index`.
- **Depth dropped, by the upstream port** (not by this FiftyOne subset): the source FMB
release includes 4 depth maps per frame; `fmb_multi`’s LeRobot v3 port is RGB + F/T +
proprioception + action only.
- **BGR → RGB**, done by the upstream port: FMB’s original `.npy` demonstrations store
images in BGR; this port converts them to RGB before video encoding.
- **`fps: 10` is nominal, not measured.** Per the source README, the original FMB `.npy`
demonstrations carry no per-frame timestamps; frames map 1:1 onto a synthetic 10 fps
grid rather than reflecting an actual recording rate.
- **Action is the FMB commanded action as-is** — no next-pose reconstruction or
relabeling was applied by the port.
- **Re-export, not a thin reference to the original repo:** this repo is a
self-contained LeRobotDataset v3.0 export (via FiftyOne’s `LeRobotDatasetExporter`),
not a pointer back to `robot-lev/fmb_multi`. Task indices were remapped to only the
tasks actually present in this subset (all 6 of the source’s 6). Per-episode and
global statistics were recomputed from the exported rows, not carried over from the
source’s global stats.
## Dataset Creation
### Curation Rationale
FMB was designed as a benchmark for generalizable robotic manipulation learning,
specifically targeting contact-rich, multi-stage assembly tasks (insertion onto fixtures
across multiple board/object variants) that stress both perception and force-aware
control — harder to generalize than simple pick-place. This LeRobot v3 port converts
each demonstration `.npy` into one LeRobot episode; this FiftyOne subset exists purely as
a lightweight, disk-budget-friendly slice for exploration and tooling, not a re-curation
of scenario content.
### Source Data
#### Data Collection and Processing
- **Robot:** Franka Panda arm.
- **Cameras:** 4 fixed viewpoints — `side_1`, `side_2` (external) and `wrist_1`,
`wrist_2` (wrist-mounted) — each at 256×256 RGB, AV1-encoded in this port (originally
BGR in the source `.npy` files).
- **Sensing:** joint position/velocity, end-effector pose/velocity, gripper state,
end-effector force/torque (EE frame), and the flattened 6×7 robot Jacobian.
- **Task structure:** each demonstration is a full assembly trajectory decomposed into
discrete skill primitives (approach the board, grasp the object, insert it, move up,
place it on a fixture, regrasp), spanning 3 distinct assembly boards
(`board_1`/`board_2`/`board_3`) in the multi-object subset.
- **Original FMB collection (per the paper):** demonstrations collected across multiple
objects/boards for the Functional Manipulation Benchmark, designed to test
generalization of manipulation policies across object/task variation.
#### Who are the source data producers?
Collected by the FMB authors (UC Berkeley) as part of the Functional Manipulation
Benchmark project.
### Annotations
#### Annotation process
Skill-primitive segmentation (`primitives`/`task_index`) comes from the original FMB
demonstration collection/labeling process, not a separate annotation pass added by this
port. The LeRobot v3 conversion (`lvjonok/fmb-lerobot-port`) maps each frame’s original
primitive label to a `task_index` and derives the per-episode `tasks` set from it.
#### Personal and Sensitive Information
None identified — the dataset contains robot proprioception/force-torque data and RGB
video of tabletop assembly-board manipulation; no human subjects data.
## Citation
**BibTeX:**
```bibtex
@article{luo2024fmb,
title = {FMB: a Functional Manipulation Benchmark for Generalizable Robotic Learning},
author = {Luo, Jianlan and Xu, Charles and Liu, Fangchen and Tan, Liam and Lin, Zipeng and Wu, Jeffrey and Abbeel, Pieter and Levine, Sergey},
journal = {arXiv preprint arXiv:2401.08553},
year = {2024}
}
```
**APA:**
Luo, J., Xu, C., Liu, F., Tan, L., Lin, Z., Wu, J., Abbeel, P., & Levine, S. (2024). *FMB: a Functional Manipulation Benchmark for Generalizable Robotic Learning*. arXiv:2401.08553.
## More Information
This is a 302-episode subset of the community LeRobot v3 port
[robot-lev/fmb_multi](https://huggingface.co/datasets/robot-lev/fmb_multi) (1,804
episodes), itself a derivative of the original
[FMB release](https://huggingface.co/datasets/charlesxu0124/functional-manipulation-benchmark),
produced for local exploration under a limited disk budget. See the
[FMB project page](https://functional-manipulation-benchmark.github.io) for the full
dataset (including depth) and the single-object counterpart
[robot-lev/fmb](https://huggingface.co/datasets/robot-lev/fmb).
## Dataset Card Authors
[Harpreet Sahota](https://huggingface.co/harpreetsahota)
## Dataset Card Contact
[Harpreet Sahota](https://huggingface.co/harpreetsahota)