Note

This is a Hugging Face dataset. For large datasets, ensure huggingface_hub>=1.1.3 to avoid rate limits. Learn more in the Hugging Face integration docs.

Dataset Card for DSEC sample (FiftyOne multimodal MCAP)#

preview

This is a FiftyOne dataset with 6 samples. Each sample is one driving sequence, stored as a native multimodal MCAP episode.

The source is DSEC, the stereo event camera dataset for driving from the Robotics and Perception Group at the University of Zurich. A car carries a stereo pair of Prophesee event cameras at 640x480 and a stereo pair of global-shutter color cameras at 1440x1080 through Zurich, Thun and Interlaken. Disparity derived from LiDAR scans is the ground truth for both pairs, and some sequences add optical flow ground truth for the event cameras. The release holds 53 sequences; this sample carries 6 of the training sequences, which come with their ground truth, chosen across the three places.

Installation#

If you haven’t already, install FiftyOne:

pip install -U fiftyone

Usage#

import fiftyone as fo
import fiftyone.utils.huggingface as fouh

# Load the dataset
# Note: other available arguments include 'max_samples', etc
dataset = fouh.load_from_hub(
    "Voxel51/DSEC-Sample",
    name="DSEC-Sample",
    persistent=True,
)

# Launch the App
session = fo.launch_app(dataset)

Dataset Details#

Dataset Description#

6 sequences, 4 in Zurich, 1 in Thun and 1 in Interlaken, totalling 105 seconds of driving, 2,233,378,133 events and 2,112 image pairs.

Curated by: Robotics and Perception Group, University of Zurich (source release)
Funded by: [More Information Needed]
Shared by: Voxel51 (FiftyOne conversion)
Language(s): Not applicable (sensor data)
License: CC-BY-SA-4.0

Dataset Sources#

Repository: DSEC (source release); Voxel51/DSEC-Sample (this conversion)
Paper: DSEC: A Stereo Event Camera Dataset for Driving Scenarios (IEEE Robotics and Automation Letters, 2021)
Demo: [More Information Needed]

Uses#

Direct Use#

Stereo disparity estimation for both the event cameras and the color cameras, and optical flow estimation from event cameras on the sequences that carry flow, evaluated against the ground truth shipped in each episode.

Out-of-Scope Use#

[More Information Needed]

Dataset Structure#

Topology

An ungrouped FiftyOne dataset with media_type="multimodal". One sample is one sequence, and its filepath is a .fo.mcap file. There are 6 samples (interlaken_00_c, thun_00_a, zurich_city_02_a, zurich_city_04_b, zurich_city_09_b, zurich_city_11_a), no sample tags, and an empty dataset.info. The dataset has no FiftyOne label fields; every signal, including the ground truth, lives inside the MCAP file as a channel and is shown by the App’s multimodal viewer.

Sample fields

Field	FiftyOne type	Description
`id`, `filepath`, `tags`, `metadata`, `created_at`, `last_modified_at`	built-in	Standard FiftyOne sample fields
`sequence`	`StringField`	Sequence name, e.g. `interlaken_00_c`
`place`	`StringField`	Where the sequence was recorded: Zurich, Thun or Interlaken
`duration`	`FloatField`	Sequence length in seconds
`num_events_left`, `num_events_right`	`IntField`	Events produced by the left and the right event camera
`peak_event_rate_mev_s`	`FloatField`	The busier camera’s busiest 1/30 s window, in millions of events per second
`num_images`	`IntField`	Color image pairs
`num_disparity_maps`	`IntField`	Disparity ground truth maps
`num_optical_flow_maps`	`IntField`	Optical flow ground truth maps; 0 on sequences without flow
`mean_image_brightness`	`FloatField`	Mean pixel value of the left color images

Episode contents

Each MCAP episode contains these channels:

Channel	Schema or content
`/events-left`, `/events-right`	Every event each event camera produced, in windows of 1/30 s, as `foxglove.PointCloud`: `x` and `y` are the raw pixel, `z` is the time since the window opened in milliseconds and `polarity` is 1 for a brightness increase and 0 for a decrease. Each message is stamped at its window’s close
`/event-frames-left`, `/event-frames-right`	A render of each window, ON events white and OFF events black on gray, `foxglove.CompressedVideo`
`/images-left`, `/images-right`	The rectified color images at 20 Hz, `foxglove.CompressedVideo`
`/disparity-event`, `/disparity-image`	Disparity ground truth for each stereo pair at 10 Hz, `foxglove.CompressedImage` (16-bit PNG, 256 per pixel of disparity, 0 where there is none)
`/optical-flow-forward`, `/optical-flow-backward`	The event cameras’ optical flow ground truth, `foxglove.CompressedImage` (16-bit PNG); only on the sequences that have them
`<camera>-calibration`	A calibration topic beside each camera stream, `foxglove.CameraCalibration`
`/tf`	The four cameras and their rectified frames, `foxglove.FrameTransform`
`/sequence`	Names the sequence and its place

Sequences

Sequence	Place	Duration	Events	Peak event rate	Image pairs	Disparity maps	Flow maps	Mean image brightness
`interlaken_00_c`	Interlaken	26.8 s	884,252,854	28.8 M/s	537	269		92.5
`thun_00_a`	Thun	11.9 s	261,111,191	20.5 M/s	239	120	41	79.6
`zurich_city_02_a`	Zurich	11.7 s	312,356,446	22.3 M/s	235	118	64	82.8
`zurich_city_04_b`	Zurich	13.4 s	247,359,435	20.6 M/s	269	135		91.9
`zurich_city_09_b`	Zurich	18.3 s	298,543,673	22.6 M/s	367	184		26.5
`zurich_city_11_a`	Zurich	23.2 s	229,754,534	12.4 M/s	465	233	231	78.5

Parsing decisions

Why MCAP: the dataset is one multimodal sample per sequence, so the event streams, color images, ground truth and calibration play back on one shared clock instead of being split into per-frame samples.
Events: every event in each event camera’s file is carried, on the timestamps the release records in microseconds on the rig’s own clock. Both cameras’ windows run from the first event of either, so the two cameras share their window stamps, and an event’s time is its window’s stamp less 1/30 s plus its z in milliseconds. The events are carried at the raw, distorted pixel, with each event camera’s distortion on its calibration topic.
Color images: the release’s rectified images, re-encoded to Annex-B H.264 without B-frames, one access unit per frame, on the release’s image timestamps.
Disparity and optical flow: the maps are carried as the release ships them. /disparity-event lies in the rectified frame of the left event camera and /disparity-image in that of the left color camera. Each optical flow map is stamped at the start of the interval it covers. The optical flow PNGs hold three 16-bit channels: horizontal and vertical flow as (value - 2^15) / 128 pixels, and a validity flag.
Transforms: /tf is the release’s calibration. The release maps points from each camera into the next and from each camera into its rectified frame; each transform is carried as the inverse, the child’s pose in its parent.
Left out: the raw LiDAR and the semantic labels the release offers are not carried.

Dataset Creation#

Curation Rationale#

[More Information Needed]

Source Data#

Data Collection and Processing#

Recorded with a car carrying a stereo pair of Prophesee event cameras and a stereo pair of global-shutter color cameras through Zurich, Thun and Interlaken. The FiftyOne conversion writes one MCAP episode per sequence; see Parsing decisions above for the changes made.

Who are the source data producers?#

The Robotics and Perception Group at the University of Zurich.

Annotations#

Annotation process#

The dataset has no human annotations in FiftyOne; the ground truth is carried inside each episode. Disparity is derived from LiDAR scans, and some sequences add optical flow ground truth for the event cameras. Further details of how the release produced it are [More Information Needed].

Who are the annotators?#

[More Information Needed]

Personal and Sensitive Information#

[More Information Needed]

Citation#

Work using the dataset should cite:

BibTeX:

@Article{Gehrig21ral,
  author = {Mathias Gehrig and Willem Aarents and Daniel Gehrig and Davide Scaramuzza},
  title = {DSEC: A Stereo Event Camera Dataset for Driving Scenarios},
  journal = {IEEE Robotics and Automation Letters},
  year = {2021},
  doi = {10.1109/LRA.2021.3068942}
}

@InProceedings{Gehrig3dv2021,
  author = {Mathias Gehrig and Mario Millh\"ausler and Daniel Gehrig and Davide Scaramuzza},
  title = {E-RAFT: Dense Optical Flow from Event Cameras},
  booktitle = {International Conference on 3D Vision (3DV)},
  year = {2021}
}

APA:

Gehrig, M., Aarents, W., Gehrig, D., & Scaramuzza, D. (2021). DSEC: A stereo event camera dataset for driving scenarios. IEEE Robotics and Automation Letters. https://doi.org/10.1109/LRA.2021.3068942

Gehrig, M., Millhäusler, M., Gehrig, D., & Scaramuzza, D. (2021). E-RAFT: Dense optical flow from event cameras. In International Conference on 3D Vision (3DV).

More Information#

DSEC is distributed under the Creative Commons Attribution-ShareAlike 4.0 International license (CC-BY-SA-4.0), and this conversion is distributed under the same license.

Changes from the source: 6 of the release’s training sequences, converted to the FiftyOne MCAP flavor, each event camera’s events cut into 1/30 s windows carried as point clouds with a grayscale render of each window, H.264 encoding of the rectified color images, the release’s calibration carried as camera intrinsics and inverted transforms, and the raw LiDAR and semantic labels left out.

Dataset Card Authors#

[More Information Needed]

Dataset Card Contact#

[More Information Needed]