#### NOTE
This is a **Hugging Face dataset**. For large datasets, ensure `huggingface_hub>=1.1.3` to avoid rate limits. Learn more in the Hugging Face integration docs.
# Egocentric RGB-D + EMG/Force → FiftyOne (Native Multimodal MCAP)
The
[Lo6yu/egocentric_dataset](https://huggingface.co/datasets/Lo6yu/egocentric_dataset)
daily activity release, converted to native multimodal MCAP episodes.
Someone wearing a depth camera and an EMG band on each wrist works through
eight household tasks: sweeping and mopping, tidying a bedroom, a dining
room and a living room, loading a washing machine, washing hands at a sink,
wiping hands on paper towel, and applying hand cream. Alongside the video
the release carries the muscle signal from both forearms, a hand skeleton
per frame, and an estimate of how hard each finger is pressing.
Every episode is cut into subtasks a reviewer has described in English, so
the 122 segments read as instructions rather than indices: *pick up dustpan
and broom*, *fit the trash bag into the basin*, *restock paper rolls*.
## Installation
```bash
pip install fiftyone
```
## Usage
```python
import fiftyone as fo
import fiftyone.utils.huggingface as fouh
dataset = fouh.load_from_hub(
"Voxel51/Egocentric-EMG-Force",
name="Egocentric-EMG-Force",
persistent=True,
)
fo.launch_app(dataset)
```
The tasks with the firmest contact:
```python
view = dataset.match({"peak_finger_force": {"$gt": 40}})
```
Episodes that involve folding:
```python
view = dataset.match({"phases": {"$elemMatch": {"$regex": "fold"}}})
```
## What you get
Eight episodes and 38.2 minutes of recording. Each episode carries:
- `/camera`, the first-person view at 640x480, as
`foxglove.CompressedVideo`
- `/depth`, the matching Z16 depth frame, as 16-bit PNG
- `/emg-left.plot` and `/emg-right.plot`, eight channels per wrist at
roughly 550 Hz
- `/wrist-imu-left.plot` and `/wrist-imu-right.plot`, accelerometer, gyro
and orientation from each band
- `/finger-force-{left,right}` and `/finger-force-{left,right}.plot`,
newtons for thumb, index, middle, ring and pinky
- `/finger-contact-{left,right}.plot`, the per-finger contact confidence
- `/finger-flexion-{left,right}.plot`, mean joint flexion per finger
- `/hand-keypoints`, the 21-point skeleton for each visible hand, as
`foxglove.SceneUpdate`
- `/subtask` and `/subtask.plot`, each segment’s English label at the frame
it begins
- `/instruction`, the task the episode is carrying out
Across the whole set that comes to 67,185 camera and depth frames,
**2,539,493 EMG samples**, 60,254 hand skeletons, 133,721 per-finger force
readings and 122 described subtask segments over 121 distinct phases.
Episodes carry the fields `package_id`, `task`, `task_description`,
`scene_description`, `subtasks`, `phases`, `objects`, `num_camera_frames`,
`num_depth_frames`, `num_emg_samples`, `num_hand_keypoint_frames`,
`num_force_frames`, `num_subtasks`, `has_emg`, `peak_finger_force`,
`duration`, `fps`, `rgb_width` and `rgb_height`.
## Notes on the conversion
Video is re-encoded to Annex-B H.264 without B-frames. The source stores it
as lossless FFV1 in MKV, which the viewer does not decode.
Depth is carried as 16-bit PNG at its published scale. The source stores raw
Z16 frames in a zstd stream with no per-frame header.
The wrist EMG leads the camera by 80 ms. The release records that offset and
publishes a corrected clock, which is the one used here, so the muscle signal
lines up with the frame it belongs to.
Contact force is an estimate derived from hand pose and depth, not a reading
from an instrumented glove. It saturates at 45 N; across the set only a
handful of samples reach that ceiling.
Hand joints are published in the camera frame as the release computes them,
and each skeleton is stamped from the camera frame it was measured on. A
frame occasionally reports the same hand more than once, and the first
detection of each side is the one published.
`hand-cream-application` names its contact columns
`dynamic_finger_force_N_*` where the other seven write
`finger_contact_force_N_*`. Both are read.
## License & attribution
The source dataset is released under
[CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/), and this
conversion is distributed under the same license. Use is limited to
non-commercial purposes.
Changes from the source: conversion to the FiftyOne MCAP flavor, re-encoding
of the video to H.264, decoding of the Z16 depth stream to PNG, and encoding
of the sensor tables and semantic labels as message streams.