Note

This is a Hugging Face dataset. For large datasets, ensure huggingface_hub>=1.1.3 to avoid rate limits. Learn more in the Hugging Face integration docs.

Dataset Card for M3ED sample#

preview

This is a FiftyOne dataset with 3 samples. Each sample is one sequence from M3ED, the multi-robot, multi-sensor, multi-environment event dataset from the GRASP Laboratory at the University of Pennsylvania, stored as a native multimodal MCAP episode.

A car, a quadrotor and a Boston Dynamics Spot carry the same sensor head: a stereo pair of Prophesee EVK4 HD event cameras at 1280x720, a stereo pair of grayscale cameras and a color camera at 1280x800, an inertial unit and an Ouster OS1-64 LiDAR, through cities, forests and buildings by day and night. The release ships ground-truth poses and depth for the left event camera beside each sequence. This sample carries one sequence from each platform: the car, car_urban_day_horse; the quadrotor, falcon_outdoor_night_high_beams; Spot, spot_indoor_stairwell.

Installation#

If you haven’t already, install FiftyOne:

pip install -U fiftyone

Usage#

import fiftyone as fo
import fiftyone.utils.huggingface as fouh

# Load the dataset
# Note: other available arguments include 'max_samples', etc
dataset = fouh.load_from_hub(
    "Voxel51/M3ED-Sample",
    name="M3ED-Sample",
    persistent=True,
)

# Launch the App
session = fo.launch_app(dataset)

Dataset Details#

Dataset Description#

3 sequences, one per platform, totalling 192 seconds of recording, 3,631,437,863 events, 4,802 image triplets and 1,921 LiDAR scans.

Curated by: GRASP Laboratory, University of Pennsylvania (source release)
Funded by: [More Information Needed]
Shared by: Voxel51 (FiftyOne conversion)
Language(s): Not applicable (sensor data)
License: CC-BY-SA-4.0

Dataset Sources#

Repository: m3ed.io (source release); Voxel51/M3ED-Sample (this conversion)
Paper: M3ED: Multi-Robot, Multi-Sensor, Multi-Environment Event Dataset (CVPR Workshops 2023)
Demo: [More Information Needed]

Uses#

Direct Use#

Work with event cameras alongside grayscale and color cameras, LiDAR and an inertial unit across a car, a quadrotor and a legged robot, using the ground-truth poses and depth for the left event camera that ship with each sequence.

Out-of-Scope Use#

[More Information Needed]

Dataset Structure#

Topology

An ungrouped FiftyOne dataset with media_type="multimodal". One sample is one sequence, and its filepath is a .fo.mcap file. There are 3 samples (car_urban_day_horse, falcon_outdoor_night_high_beams, spot_indoor_stairwell), no sample tags, and an empty dataset.info. The dataset has no FiftyOne label fields; every signal lives inside the MCAP file as a channel and is shown by the App’s multimodal viewer.

Sample fields

Field	FiftyOne type	Description
`id`, `filepath`, `tags`, `metadata`, `created_at`, `last_modified_at`	built-in	Standard FiftyOne sample fields
`sequence`	`StringField`	Sequence name, e.g. `car_urban_day_horse`
`platform`	`StringField`	Robot that carried the sensor head: `car`, `falcon` or `spot`
`environment`	`StringField`	Where and when the sequence was recorded, e.g. `urban_day`, `outdoor_night`, `indoor`
`recorded`	`StringField`	Recording date
`duration`	`FloatField`	Sequence length in seconds
`num_events_left`, `num_events_right`	`IntField`	Events produced by the left and the right event camera
`peak_event_rate_mev_s`	`FloatField`	The busier event camera’s busiest 1/30 s window, in millions of events per second
`num_images`	`IntField`	Image triplets (grayscale left, grayscale right and color)
`num_lidar_scans`	`IntField`	LiDAR scans
`num_lidar_points`	`IntField`	LiDAR points across all scans
`num_ground_truth_poses`	`IntField`	Poses on the ground-truth trajectory of the left event camera
`ground_truth_path_m`	`FloatField`	Length of the ground-truth trajectory in metres
`valid_depth_fraction`	`FloatField`	Share of depth ground-truth pixels holding a value
`mean_image_brightness`	`FloatField`	Mean pixel value of the color camera

Episode contents

Each MCAP episode contains these channels:

Channel	Schema or content
`/events-left`, `/events-right`	Every event each event camera produced, in windows of 1/30 s, as `foxglove.PointCloud` with `x` and `y` the raw pixel, `z` the time since the window opened in milliseconds and `polarity` 1 for a brightness increase and 0 for a decrease; each message is stamped at its window’s close
`/event-frames-left`, `/event-frames-right`	A render of each window, ON events white and OFF events black on gray, `foxglove.CompressedVideo`
`/images-left`, `/images-right`, `/images-rgb`	The grayscale pair and the color camera at 1280x800, `foxglove.CompressedVideo`
`/lidar-points`	The Ouster scans, `foxglove.PointCloud` with `x`, `y`, `z`, `reflectivity` and `signal`
`/imu.plot`	The sensor head’s inertial unit
`/ground-truth`, `/ground-truth.plot`	The left event camera’s pose relative to its first ground-truth pose, `foxglove.PoseInFrame`, with its position as a plot
`/depth-ground-truth`	The depth ground truth for the left event camera, `foxglove.CompressedImage` (16-bit PNG, millimetres, 0 where there is none)
`<camera>-calibration`	A calibration topic beside each camera stream, `foxglove.CameraCalibration`
`/tf`	Each sensor’s pose in the left event camera’s frame, `foxglove.FrameTransform`
`/sequence`	Names the sequence and its platform

Sequences

Sequence	Platform	Environment	Duration	Events	Peak event rate	LiDAR scans	Path
`car_urban_day_horse`	car	urban day	28.7 s	846,851,622	45.9 M/s	286	47 m
`falcon_outdoor_night_high_beams`	falcon	outdoor night	64.9 s	901,446,929	47.4 M/s	648	46 m
`spot_indoor_stairwell`	spot	indoor	98.9 s	1,883,139,312	63.8 M/s	987	30 m

Parsing decisions

Clock: the release’s timestamps run in microseconds from each recording’s start, which its stats file gives in Unix time; every stream is placed on that clock.
Events: every event in each event camera’s data is carried, at the raw pixel, with each camera’s distortion on its calibration topic. Both cameras’ windows run from the first event of either, so an event’s time is its window’s stamp less 1/30 s plus its z in milliseconds.
Video encoding: the cameras are re-encoded to Annex-B H.264 without B-frames, one access unit per frame.
Color order: the release stores the color camera’s pixels in blue-green-red order, and they are carried in red-green-blue order, keeping the green cast they have in the release.
LiDAR: the LiDAR packets the release stores are decoded to points with the Ouster SDK, keeping the returns with a range.
Ground-truth poses: the release gives the ground-truth pose as the transform from the first pose’s frame into the current one, so /ground-truth carries its inverse, the camera’s pose in the first pose’s frame.
Depth: the depth ground truth, which the release stores as 32-bit floats in metres, is carried as 16-bit PNG in millimetres, rounded.
Extrinsics: the release’s T_to_prophesee_left for each sensor maps that sensor’s points into the left event camera, which is the sensor’s pose in that frame, and is carried as /tf.
Left out: a few inertial samples the release marks as untimed, the LiDAR’s own inertial unit and the semantic labels are not reproduced.

Dataset Creation#

Curation Rationale#

[More Information Needed]

Source Data#

Data Collection and Processing#

Recorded with a sensor head carried by a car, a quadrotor and a Boston Dynamics Spot, through cities, forests and buildings by day and night. The FiftyOne conversion reads the release’s HDF5 files and writes one MCAP episode per sequence; see Parsing decisions above for the changes made.

Who are the source data producers?#

The GRASP Laboratory at the University of Pennsylvania.

Annotations#

Annotation process#

The conversion adds no annotations. The release ships ground-truth poses and depth for the left event camera with each sequence; how they were produced is [More Information Needed].

Who are the annotators?#

[More Information Needed]

Personal and Sensitive Information#

[More Information Needed]

Citation#

Cite:

BibTeX:

@InProceedings{Chaney_2023_CVPR,
  author = {Chaney, Kenneth and Cladera, Fernando and Wang, Ziyun and Bisulco, Anthony and Hsieh, M. Ani and Korpela, Christopher and Kumar, Vijay and Taylor, Camillo J. and Daniilidis, Kostas},
  title = {M3ED: Multi-Robot, Multi-Sensor, Multi-Environment Event Dataset},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
  month = {June},
  year = {2023},
  pages = {4015-4022}
}

APA:

Chaney, K., Cladera, F., Wang, Z., Bisulco, A., Hsieh, M. A., Korpela, C., Kumar, V., Taylor, C. J., & Daniilidis, K. (2023). M3ED: Multi-robot, multi-sensor, multi-environment event dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (pp. 4015-4022).

More Information#

M3ED is distributed under the Creative Commons Attribution-ShareAlike 4.0 International license (CC-BY-SA-4.0), and this conversion is distributed under the same license.

Changes from the source: 3 of the release’s sequences, converted from HDF5 to the FiftyOne MCAP flavor, each event camera’s events cut into 1/30 s windows carried as point clouds with a grayscale render of each window, H.264 encoding of the cameras with the color camera’s channels in red-green-blue order, the LiDAR packets decoded to points, the ground-truth poses inverted, the depth ground truth carried as 16-bit millimetres, and untimed inertial samples, the LiDAR’s inertial unit and the semantic labels left out.

Dataset Card Authors#

[More Information Needed]

Dataset Card Contact#

[More Information Needed]