Note

This is a Hugging Face dataset. For large datasets, ensure huggingface_hub>=1.1.3 to avoid rate limits. Learn more in the Hugging Face integration docs.

Hugging Face

Dataset Card for aiMotive Multimodal Dataset#

preview

The aiMotive Multimodal Dataset is a 176-scene autonomous driving dataset with synchronized and calibrated LiDAR, camera, and radar sensors providing 360-degree field-of-view coverage with sensor redundancy. Scenes were captured in highway, urban, and suburban environments across three countries during daytime, night, and rain. The dataset contains 26,583 annotated frames with 3D bounding boxes for 14 object classes (425k+ instances), ~24% of which are beyond 75 m from the ego vehicle, enabling long-range perception research. Parsed into FiftyOne multimodal format as 176 MCAP episodes with synchronized camera images, LiDAR point clouds, radar targets, GPS/IMU data, static sensor transforms, and per-frame 3D box annotations rendered as colored scene entities and projected 2D overlays.

Installation#

pip install -U fiftyone

Usage#

import fiftyone as fo
from fiftyone.utils.huggingface import load_from_hub

# Load the dataset
dataset = load_from_hub("Voxel51/aimotive-multimodal")

# Launch the App
session = fo.launch_app(dataset)

Dataset Details#

Dataset Description#

  • Curated by: aiMotive (Vehicle Testing & Engineering, Calibration, aiNotate, and Object Detection teams)

  • Funded by: aiMotive

  • Shared by: aiMotive

  • Language(s): N/A

  • License: CC BY-NC-SA 4.0

Dataset Sources#

  • Repository: https://github.com/aimotive/aimotive_dataset

  • Paper: https://arxiv.org/abs/2211.09445

  • Demo: [More Information Needed]

Uses#

Direct Use#

The dataset supports 3D object detection, multi-object tracking, and motion prediction research in autonomous driving. Its redundant 360-degree sensor layout (LiDAR + cameras + radars) and extended annotation range (up to 200 m) make it suited for training and benchmarking long-range perception models and multimodal sensor fusion methods. The presence of consistent track IDs across frames enables end-to-end tracking model development. GPS/IMU ego-motion data enables odometry and lead-car prediction tasks.

Out-of-Scope Use#

The dataset must not be used for any military or harmful application. Commercial use is not permitted under the CC BY-NC-SA 4.0 license.

Dataset Structure#

Overview#

The FiftyOne dataset is a multimodal dataset containing 176 samples, one per 15-second episode. Each sample’s filepath points to an MCAP file that encodes all sensor streams for that episode in a single time-indexed container. The dataset includes a train split (151 episodes, 21,402 frames) and a val split (25 episodes, 5,181 frames).

In the FiftyOne App, each sample opens the multimodal viewer with four synchronized tiles:

  • Image tile — per-camera JPEG streams with 3D-to-2D projected bounding box overlays

  • 3D tile — LiDAR point cloud with colored 3D bounding box cubes (set reference frame to lidar)

  • Map tile — GPS trajectory derived from GNSS+INS

  • Message tile — raw JSON payloads for radar targets and annotation objects

Sample Fields#

Field

FiftyOne type

Description

filepath

StringField

Absolute path to the episode MCAP file

split

StringField

"train" or "val"

condition

StringField

Recording environment: "highway", "night", "rain", or "urban"

episode_name

StringField

Raw episode directory name, e.g. "20210401-073402-00.18.00-00.18.15@Jarvis"

session_id

StringField

Session timestamp prefix, e.g. "20210401-073402"

platform

StringField

Recording vehicle ID: "Jarvis", "Yoda", "Cali", or "Sogun"

duration_s

FloatField

Episode duration in seconds (≈ 15 s)

message_count

IntField

Total number of MCAP messages in the episode

channel_count

IntField

Number of distinct MCAP channels in the episode

topics

ListField(StringField)

List of MCAP channel topic strings

schemas

ListField(StringField)

List of Foxglove schema names used in the episode

has_image

BooleanField

True for all 176 samples

has_pointcloud

BooleanField

True for all 176 samples

has_gps

BooleanField

True for all 176 samples

has_3d_boxes

BooleanField

True for all 176 samples

object_classes

ListField(StringField)

Sorted list of annotation class names present in this episode

total_annotated_objects

IntField

Total annotated object count summed across all frames

avg_objects_per_frame

FloatField

Mean annotated objects per frame for this episode

num_annotated_frames

IntField

Number of annotation frames in this episode

objects_by_class

DictField

Per-class object counts, e.g. {"CAR": 842, "PEDESTRIAN": 17}

MCAP Channels per Sample#

Each MCAP file contains the following channels:

Topic

Foxglove schema

Viewer tile

/camera/{front,back,left,right}/image_raw

foxglove.CompressedImage

Image

/camera/{front,back,left,right}/calibration

foxglove.CameraCalibration

Image

/camera/{front,back,left,right}/annotations

foxglove.ImageAnnotations

Image (projected 2D box edges)

/lidar/top

foxglove.PointCloud

3D

/annotations/scene

foxglove.SceneUpdate

3D (colored bounding box cubes)

/annotations/3d_boxes

aimotive.Annotations3D

Message (raw JSON per frame)

/radar/{front,back}

aimotive.RadarTargets

Message (raw JSON per frame)

/gps

foxglove.LocationFix

Map

/tf_static

foxglove.FrameTransform

3D (sensor-to-body static transforms)

Annotation Classes#

14 object classes: BICYCLE, BUS, CAR, MOTORCYCLE, OTHER-OBJECT, OTHER-RIDEABLE, PEDESTRIAN, PICKUP, RIDER, SHOPPING-CART, TRAILER, TRAIN, TRUCK, VAN.

3D Bounding Box Representation#

Each annotated object carries: a 3D center point in the body coordinate frame (BoundingBox3D Origin X/Y/Z), full extents in meters (BoundingBox3D Extent X/Y/Z), orientation as a quaternion (BoundingBox3D Orientation Quat W/X/Y/Z), object class (ObjectType), and a persistent track ID (ObjectId). In the MCAP, these are written twice: as raw JSON on /annotations/3d_boxes (all fields verbatim) and as transformed CubePrimitive scene entities on /annotations/scene (origins and orientations re-expressed in the lidar sensor frame so they co-register with the LiDAR point cloud with no TF traversal).

Camera Sensor Details#

The front and back cameras use the OpenCV pinhole model (5 distortion coefficients, plumb_bob). The left and right cameras use the equidistant fisheye model (4 distortion coefficients, equidistant). Back-camera resolution is 1920×1216; all other cameras are 1280×704. The 2D projected box overlays on each camera stream are computed by transforming 3D box corners from the body frame into each camera frame using the calibrated extrinsic matrix, then projecting with the camera-specific intrinsic model.

Platform Distribution#

Platform

Episodes

Yoda

116

Jarvis

45

Cali

12

Sogun

3

Parsing Decisions#

  • Timestamp alignment: Yoda-platform episodes derive Unix timestamps from egomotion2.json (time field, Unix seconds). Jarvis-platform episodes anchor device-boot nanosecond timestamps via the wall-clock start time in session.yml.

  • LiDAR transform: The LiDAR calibration entry provides pos_meter and yaw_pitch_roll_deg rather than a full 4×4 matrix; the extrinsic is reconstructed using ZYX Euler convention (vehicle standard).

  • Scene entity frame: 3D bounding box SceneUpdate entities are expressed in the lidar sensor frame (transformed from body frame at parse time) so that the reference frame lidar shows both geometry streams without requiring TF traversal.

  • LAZ filtering: LiDAR revolution files smaller than 226 bytes (empty/padding) and points within 0.5 m of the origin are discarded.

Dataset Creation#

Curation Rationale#

Existing autonomous driving datasets at the time of release lacked simultaneous sensor redundancy (coverage by at least two modalities) and extended annotation range. Most benchmarks annotate objects only within 80–100 m and do not include radar sensors, which limits robustness in adverse weather and the development of highway-capable long-range perception systems. The aiMotive dataset was created to address both gaps: a redundant 360-degree sensor layout (LiDAR + cameras + radars) combined with annotations extending to 200 m.

Source Data#

Data Collection and Processing#

Data was collected across three countries on two continents (California, USA; Austria; Hungary) using four instrumented vehicles (three Toyota Camry and one Toyota Prius). The recording phase spanned approximately one year to capture diverse seasons, weather conditions (sun, cloud, rain, glare), times of day (daytime, night), and geographic environments (highway, suburban, urban). Each scene is 15 seconds long. The dataset was anonymized using DashcamCleaner (face and license plate blurring). No additional preprocessing was applied to the raw sensor data beyond anonymization.

Who are the source data producers?#

The data was collected and annotated by full-time aiMotive employees across the Vehicle Testing & Engineering, Calibration, aiNotate, and Object Detection teams. The data collection process satisfied requirements set by an Institutional Review Board approval and aiMotive’s Data Protection Policy. No students, crowdworkers, or contractors were involved.

Annotations#

Annotation process#

Training annotations were generated automatically using a non-causal LiDAR-based tracker that accumulates point clouds across the full 15-second scene to fit 3D cuboids. The resulting annotations were manually quality-checked against thresholds of <10% position and size divergence and <5° orientation error. Scenes failing quality checks were discarded. Validation annotations were produced by human annotators using both LiDAR and camera data to fit cuboids, with the same orientation precision requirement. All annotated objects carry a persistent track ID consistent across all frames of a scene.

Who are the annotators?#

Training data was annotated automatically and quality-checked by aiMotive employees. Validation data was labeled by hired human annotators and supervised through a two-phase review process, with final approval from a quality-checking supervisor.

Personal and Sensitive Information#

Faces and license plates in camera images were blurred using DashcamCleaner prior to release. The raw unblurred data was not retained, in compliance with GDPR requirements.

Citation#

BibTeX:

@misc{matuszka2022aimotivedatasetmultimodaldataset,
      title={aiMotive Dataset: A Multimodal Dataset for Robust Autonomous Driving with Long-Range Perception},
      author={Tamás Matuszka and Iván Barton and Ádám Butykai and Péter Hajas and Dávid Kiss and Domonkos Kovács and Sándor Kunsági-Máté and Péter Lengyel and Gábor Németh and Levente Pető and Dezső Ribli and Dávid Szeghy and Szabolcs Vajna and Bálint Varga},
      year={2022},
      eprint={2211.09445},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2211.09445},
}

APA:

Matuszka, T., Barton, I., Butykai, Á., Hajas, P., Kiss, D., Kovács, D., Kunsági-Máté, S., Lengyel, P., Németh, G., Pető, L., Ribli, D., Szeghy, D., Vajna, S., & Varga, B. (2022). aiMotive Dataset: A Multimodal Dataset for Robust Autonomous Driving with Long-Range Perception. arXiv:2211.09445.

More Information#

The dataset DOI is 10.34740/kaggle/ds/2738461. The original dataset and baseline models (VoxelNet, BEVDepth, BEVFusion variants) are available at https://github.com/aimotive/aimotive_dataset. Contact: tamas.matuszka@aimotive.com.

Dataset Card Authors#

Harpreet Sahota

Dataset Card Contact#

[More Information Needed]