#### NOTE
This is a **Hugging Face dataset**. For large datasets, ensure `huggingface_hub>=1.1.3` to avoid rate limits. Learn more in the <a href="https://docs.voxel51.com/integrations/huggingface.html#loading-datasets-from-the-hub" target="_blank">Hugging Face integration docs</a>.

<a href="https://huggingface.co/datasets/Voxel51/aimotive-multimodal" target="_blank">![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Dataset-yellow)</a>

# Dataset Card for aiMotive Multimodal Dataset

![preview](https://huggingface.co/datasets/Voxel51/aimotive-multimodal/resolve/main/aimotive.gif)

The aiMotive Multimodal Dataset is a 176-scene autonomous driving dataset
with synchronized and calibrated LiDAR, camera, and radar sensors providing
360-degree field-of-view coverage with sensor redundancy. Scenes were
captured in highway, urban, and suburban environments across three countries
during daytime, night, and rain. The dataset contains 26,583 annotated
frames with 3D bounding boxes for 14 object classes (425k+ instances),
~24% of which are beyond 75 m from the ego vehicle, enabling long-range
perception research. Parsed into FiftyOne multimodal format as 176 MCAP
episodes with synchronized camera images, LiDAR point clouds, radar targets,
GPS/IMU data, static sensor transforms, and per-frame 3D box annotations
rendered as colored scene entities and projected 2D overlays.

## Installation

```bash
pip install -U fiftyone
```

## Usage

```python
import fiftyone as fo
from fiftyone.utils.huggingface import load_from_hub

# Load the dataset
dataset = load_from_hub("Voxel51/aimotive-multimodal")

# Launch the App
session = fo.launch_app(dataset)
```

## Dataset Details

### Dataset Description

- **Curated by:** aiMotive (Vehicle Testing & Engineering, Calibration, aiNotate, and Object Detection teams)
- **Funded by:** aiMotive
- **Shared by:** aiMotive
- **Language(s):** N/A
- **License:** [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/)

### Dataset Sources

- **Repository:** https://github.com/aimotive/aimotive_dataset
- **Paper:** https://arxiv.org/abs/2211.09445
- **Demo:** [More Information Needed]

## Uses

### Direct Use

The dataset supports 3D object detection, multi-object tracking, and motion prediction research in autonomous driving. Its redundant 360-degree sensor layout (LiDAR + cameras + radars) and extended annotation range (up to 200 m) make it suited for training and benchmarking long-range perception models and multimodal sensor fusion methods. The presence of consistent track IDs across frames enables end-to-end tracking model development. GPS/IMU ego-motion data enables odometry and lead-car prediction tasks.

### Out-of-Scope Use

The dataset must not be used for any military or harmful application. Commercial use is not permitted under the CC BY-NC-SA 4.0 license.

## Dataset Structure

### Overview

The FiftyOne dataset is a **multimodal** dataset containing 176 samples, one per 15-second episode. Each sample’s `filepath` points to an MCAP file that encodes all sensor streams for that episode in a single time-indexed container. The dataset includes a `train` split (151 episodes, 21,402 frames) and a `val` split (25 episodes, 5,181 frames).

In the FiftyOne App, each sample opens the multimodal viewer with four synchronized tiles:

- **Image tile** — per-camera JPEG streams with 3D-to-2D projected bounding box overlays
- **3D tile** — LiDAR point cloud with colored 3D bounding box cubes (set reference frame to `lidar`)
- **Map tile** — GPS trajectory derived from GNSS+INS
- **Message tile** — raw JSON payloads for radar targets and annotation objects

### Sample Fields

| Field                     | FiftyOne type            | Description                                                                   |
|---------------------------|--------------------------|-------------------------------------------------------------------------------|
| `filepath`                | `StringField`            | Absolute path to the episode MCAP file                                        |
| `split`                   | `StringField`            | `"train"` or `"val"`                                                          |
| `condition`               | `StringField`            | Recording environment: `"highway"`, `"night"`, `"rain"`, or `"urban"`         |
| `episode_name`            | `StringField`            | Raw episode directory name, e.g. `"20210401-073402-00.18.00-00.18.15@Jarvis"` |
| `session_id`              | `StringField`            | Session timestamp prefix, e.g. `"20210401-073402"`                            |
| `platform`                | `StringField`            | Recording vehicle ID: `"Jarvis"`, `"Yoda"`, `"Cali"`, or `"Sogun"`            |
| `duration_s`              | `FloatField`             | Episode duration in seconds (≈ 15 s)                                          |
| `message_count`           | `IntField`               | Total number of MCAP messages in the episode                                  |
| `channel_count`           | `IntField`               | Number of distinct MCAP channels in the episode                               |
| `topics`                  | `ListField(StringField)` | List of MCAP channel topic strings                                            |
| `schemas`                 | `ListField(StringField)` | List of Foxglove schema names used in the episode                             |
| `has_image`               | `BooleanField`           | `True` for all 176 samples                                                    |
| `has_pointcloud`          | `BooleanField`           | `True` for all 176 samples                                                    |
| `has_gps`                 | `BooleanField`           | `True` for all 176 samples                                                    |
| `has_3d_boxes`            | `BooleanField`           | `True` for all 176 samples                                                    |
| `object_classes`          | `ListField(StringField)` | Sorted list of annotation class names present in this episode                 |
| `total_annotated_objects` | `IntField`               | Total annotated object count summed across all frames                         |
| `avg_objects_per_frame`   | `FloatField`             | Mean annotated objects per frame for this episode                             |
| `num_annotated_frames`    | `IntField`               | Number of annotation frames in this episode                                   |
| `objects_by_class`        | `DictField`              | Per-class object counts, e.g. `{"CAR": 842, "PEDESTRIAN": 17}`                |

### MCAP Channels per Sample

Each MCAP file contains the following channels:

| Topic                                         | Foxglove schema              | Viewer tile                           |
|-----------------------------------------------|------------------------------|---------------------------------------|
| `/camera/{front,back,left,right}/image_raw`   | `foxglove.CompressedImage`   | Image                                 |
| `/camera/{front,back,left,right}/calibration` | `foxglove.CameraCalibration` | Image                                 |
| `/camera/{front,back,left,right}/annotations` | `foxglove.ImageAnnotations`  | Image (projected 2D box edges)        |
| `/lidar/top`                                  | `foxglove.PointCloud`        | 3D                                    |
| `/annotations/scene`                          | `foxglove.SceneUpdate`       | 3D (colored bounding box cubes)       |
| `/annotations/3d_boxes`                       | `aimotive.Annotations3D`     | Message (raw JSON per frame)          |
| `/radar/{front,back}`                         | `aimotive.RadarTargets`      | Message (raw JSON per frame)          |
| `/gps`                                        | `foxglove.LocationFix`       | Map                                   |
| `/tf_static`                                  | `foxglove.FrameTransform`    | 3D (sensor-to-body static transforms) |

### Annotation Classes

14 object classes: `BICYCLE`, `BUS`, `CAR`, `MOTORCYCLE`, `OTHER-OBJECT`, `OTHER-RIDEABLE`, `PEDESTRIAN`, `PICKUP`, `RIDER`, `SHOPPING-CART`, `TRAILER`, `TRAIN`, `TRUCK`, `VAN`.

### 3D Bounding Box Representation

Each annotated object carries: a 3D center point in the body coordinate frame (`BoundingBox3D Origin X/Y/Z`), full extents in meters (`BoundingBox3D Extent X/Y/Z`), orientation as a quaternion (`BoundingBox3D Orientation Quat W/X/Y/Z`), object class (`ObjectType`), and a persistent track ID (`ObjectId`). In the MCAP, these are written twice: as raw JSON on `/annotations/3d_boxes` (all fields verbatim) and as transformed `CubePrimitive` scene entities on `/annotations/scene` (origins and orientations re-expressed in the `lidar` sensor frame so they co-register with the LiDAR point cloud with no TF traversal).

### Camera Sensor Details

The front and back cameras use the OpenCV pinhole model (5 distortion coefficients, `plumb_bob`). The left and right cameras use the equidistant fisheye model (4 distortion coefficients, `equidistant`). Back-camera resolution is 1920×1216; all other cameras are 1280×704. The 2D projected box overlays on each camera stream are computed by transforming 3D box corners from the body frame into each camera frame using the calibrated extrinsic matrix, then projecting with the camera-specific intrinsic model.

### Platform Distribution

| Platform   |   Episodes |
|------------|------------|
| Yoda       |        116 |
| Jarvis     |         45 |
| Cali       |         12 |
| Sogun      |          3 |

### Parsing Decisions

- **Timestamp alignment:** Yoda-platform episodes derive Unix timestamps from `egomotion2.json` (`time` field, Unix seconds). Jarvis-platform episodes anchor device-boot nanosecond timestamps via the wall-clock start time in `session.yml`.
- **LiDAR transform:** The LiDAR calibration entry provides `pos_meter` and `yaw_pitch_roll_deg` rather than a full 4×4 matrix; the extrinsic is reconstructed using ZYX Euler convention (vehicle standard).
- **Scene entity frame:** 3D bounding box `SceneUpdate` entities are expressed in the `lidar` sensor frame (transformed from body frame at parse time) so that the reference frame `lidar` shows both geometry streams without requiring TF traversal.
- **LAZ filtering:** LiDAR revolution files smaller than 226 bytes (empty/padding) and points within 0.5 m of the origin are discarded.

## Dataset Creation

### Curation Rationale

Existing autonomous driving datasets at the time of release lacked simultaneous sensor redundancy (coverage by at least two modalities) and extended annotation range. Most benchmarks annotate objects only within 80–100 m and do not include radar sensors, which limits robustness in adverse weather and the development of highway-capable long-range perception systems. The aiMotive dataset was created to address both gaps: a redundant 360-degree sensor layout (LiDAR + cameras + radars) combined with annotations extending to 200 m.

### Source Data

#### Data Collection and Processing

Data was collected across three countries on two continents (California, USA; Austria; Hungary) using four instrumented vehicles (three Toyota Camry and one Toyota Prius). The recording phase spanned approximately one year to capture diverse seasons, weather conditions (sun, cloud, rain, glare), times of day (daytime, night), and geographic environments (highway, suburban, urban). Each scene is 15 seconds long. The dataset was anonymized using DashcamCleaner (face and license plate blurring). No additional preprocessing was applied to the raw sensor data beyond anonymization.

#### Who are the source data producers?

The data was collected and annotated by full-time aiMotive employees across the Vehicle Testing & Engineering, Calibration, aiNotate, and Object Detection teams. The data collection process satisfied requirements set by an Institutional Review Board approval and aiMotive’s Data Protection Policy. No students, crowdworkers, or contractors were involved.

### Annotations

#### Annotation process

Training annotations were generated automatically using a non-causal LiDAR-based tracker that accumulates point clouds across the full 15-second scene to fit 3D cuboids. The resulting annotations were manually quality-checked against thresholds of <10% position and size divergence and <5° orientation error. Scenes failing quality checks were discarded. Validation annotations were produced by human annotators using both LiDAR and camera data to fit cuboids, with the same orientation precision requirement. All annotated objects carry a persistent track ID consistent across all frames of a scene.

#### Who are the annotators?

Training data was annotated automatically and quality-checked by aiMotive employees. Validation data was labeled by hired human annotators and supervised through a two-phase review process, with final approval from a quality-checking supervisor.

#### Personal and Sensitive Information

Faces and license plates in camera images were blurred using DashcamCleaner prior to release. The raw unblurred data was not retained, in compliance with GDPR requirements.

## Citation

**BibTeX:**

```bibtex
@misc{matuszka2022aimotivedatasetmultimodaldataset,
      title={aiMotive Dataset: A Multimodal Dataset for Robust Autonomous Driving with Long-Range Perception},
      author={Tamás Matuszka and Iván Barton and Ádám Butykai and Péter Hajas and Dávid Kiss and Domonkos Kovács and Sándor Kunsági-Máté and Péter Lengyel and Gábor Németh and Levente Pető and Dezső Ribli and Dávid Szeghy and Szabolcs Vajna and Bálint Varga},
      year={2022},
      eprint={2211.09445},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2211.09445},
}
```

**APA:**

Matuszka, T., Barton, I., Butykai, Á., Hajas, P., Kiss, D., Kovács, D., Kunsági-Máté, S., Lengyel, P., Németh, G., Pető, L., Ribli, D., Szeghy, D., Vajna, S., & Varga, B. (2022). *aiMotive Dataset: A Multimodal Dataset for Robust Autonomous Driving with Long-Range Perception*. arXiv:2211.09445.

## More Information

The dataset DOI is 10.34740/kaggle/ds/2738461. The original dataset and baseline models (VoxelNet, BEVDepth, BEVFusion variants) are available at https://github.com/aimotive/aimotive_dataset. Contact: tamas.matuszka@aimotive.com.

## Dataset Card Authors

Harpreet Sahota

## Dataset Card Contact

[More Information Needed]