#### NOTE This is a **Hugging Face dataset**. For large datasets, ensure `huggingface_hub>=1.1.3` to avoid rate limits. Learn more in the Hugging Face integration docs. ![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Dataset-yellow) # Dataset Card for CMHT Autonomous Driving Multimodal (MCAP) ![image/png](https://huggingface.co/datasets/Voxel51/cmht-autonomous-driving/resolve/main/cmht.gif) This is a [FiftyOne](https://github.com/voxel51/fiftyone) dataset with 4 samples. ## Installation If you haven’t already, install FiftyOne: ```bash pip install -U fiftyone ``` ## Usage ```python import fiftyone as fo from fiftyone.utils.huggingface import load_from_hub # Load the dataset # Note: other available arguments include 'max_samples', etc dataset = load_from_hub("Voxel51/cmht-autonomous-driving") # Launch the App session = fo.launch_app(dataset) ``` ## Dataset Details ### Dataset Description The **CMHT Autonomous Dataset** is a multi-sensor autonomous-driving dataset recorded around Hamilton, Ontario by the Centre for Mechatronics and Hybrid Technologies (CMHT) at McMaster University. A single vehicle rig combines a Velodyne HDL-32E LiDAR, a Retina-4fn mmWave radar, a Logitech Brio monocular RGB camera, a FLIR A65 thermal (IR) camera, and GPS/IMU (built into the LiDAR unit), recorded live with ROS2 (galactic) into `ros2 bag` recordings. The dataset consists of over 9,000 labeled frames captured at 10-20 Hz across four drives spanning dusk/clear and night/rain conditions in downtown Hamilton, with every LiDAR-detected object labeled with its 3D position, size, rotation, classification, and object ID. This repository repackages the four raw ROS2 bag recordings as time-synchronized [MCAP](https://mcap.dev/) episodes for FiftyOne’s native [multimodal dataset support](https://docs.voxel51.com/user_guide/multimodal.html), with the original per-frame 3D tracklet labels (published separately by the authors as a “frame-by-frame extracted” release) embedded directly back into each episode’s MCAP timeline as ROS2 `vision_msgs` detection topics, synced to the exact sensor message each label was originally annotated from. Each sample is one continuous drive, viewable in FiftyOne’s tiled multimodal viewer with synchronized camera, thermal camera, LiDAR point cloud, radar, GPS, IMU, and 3D/2D object-detection playback. - **Curated by:** Howard Zhang, Ash Liu, Saied Habibi, Martin v. Mohrenschildt, and Ryan Ahmed (Centre for Mechatronics and Hybrid Technologies, McMaster University) — original data collection, sensor rig, and 3D tracklet labeling. This MCAP/FiftyOne multimodal repackaging (ROS2-bag-to-MCAP conversion, data-quality fixes, and re-embedding of the separately-published labels into the MCAP timeline) was prepared independently by Harpreet Sahota. - **Funded by:** None — per the paper’s Acknowledgements: “This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.” - **Shared by:** Harpreet Sahota (this repackaging); the original CMHT Autonomous Dataset is shared by McMaster University via the Federated Research Data Repository (FRDR) and a MacDrive file share. - **Language(s):** N/A (sensor data — camera, thermal camera, LiDAR, radar, GPS/IMU; no text). - **License:** [CC0 1.0](https://creativecommons.org/publicdomain/zero/1.0/), per the FRDR record. (The accompanying Data in Brief paper text itself is published under CC BY 4.0 — that license applies to the paper, not the dataset files.) ### Dataset Sources - **Repository:** [FRDR record](https://doi.org/10.20383/103.01024) (direct file host: [MacDrive Seafile share](https://macdrive.mcmaster.ca/d/2d54f23bc41f48bd9a2d/)) - **Paper:** Zhang, H., Liu, A., Habibi, S., Mohrenschildt, M. v., & Ahmed, R. (2025). CMHT autonomous dataset: A multi-sensor dataset including radar and IR for autonomous driving. *Data in Brief, 60*, 111552. [https://doi.org/10.1016/j.dib.2025.111552](https://doi.org/10.1016/j.dib.2025.111552) - **Demo:** [More Information Needed] ## Uses ### Direct Use - Multi-sensor fusion research combining LiDAR, radar, monocular camera, and thermal (IR) camera, including comparing sensor efficacy across weather/lighting conditions (dusk/clear vs. night/rain). - Training or evaluating 3D and 2D object detection/tracking models against the embedded ground-truth `Car`/`Truck`/`Van`/`Pedestrian`/`Bus`/`LongVehicle` tracklets (object IDs are preserved for tracking-across-time use cases). - Exercising and demoing FiftyOne’s multimodal MCAP support: synchronized playback of camera, thermal, 3D point cloud (LiDAR + radar), GPS/map, and embedded 2D/3D detection tiles across real autonomous-driving recordings. - Studying sensor-fusion performance in adverse conditions specifically, since 3 of the 4 drives were recorded at night in rain. ### Out-of-Scope Use - Using this as a large-scale benchmark comparable to KITTI, nuScenes, or Waymo — this is 4 drives with roughly 9,000 labeled frames total, an order of magnitude smaller than those datasets (an explicitly stated limitation in the source paper). - Relying on LiDAR resolution comparable to newer 64+ channel sensors — the Velodyne HDL-32E is a 32-channel unit, lower resolution than many current autonomous-driving LiDARs (also a stated paper limitation). - Sensor-fusion use cases that require overlapping fields of view across all sensors everywhere in the scene — the rig’s sensors only share overlapping FOV at the front of the vehicle (a stated paper limitation). - Pedestrian-heavy benchmarking or training — the paper notes the dataset has comparatively low pedestrian representation, which “may cause detrimental effects in AI training/testing.” - Treating the radar/GPS/IMU streams in the 3 `night_rain_*` drives as frame-exact synchronized with the camera/LiDAR streams — see `is_preliminary_sync` in [Parsing decisions](#parsing-decisions). - Any use requiring the *original* raw radar stream for `night_rain_2` — the radar-labeled frame release did not include this drive (see [Parsing decisions](#parsing-decisions)). ## Dataset Structure This is a flat (ungrouped) FiftyOne dataset with `media_type: "multimodal"` and **4 samples**. Each sample is one **episode** — a single continuous drive/recording session — stored as one `.mcap` file; FiftyOne infers the multimodal media type automatically from the `.mcap` file extension. There are no separate image, point-cloud, or video samples: the drive is the sample unit, and every stream inside it (camera, thermal camera, LiDAR, radar, GPS, IMU, and the embedded object-detection topics) is decoded live by FiftyOne’s multimodal viewer. No dataset-level splits are provided by the source data; the 4 drives (`dusk_clear_0`, `night_rain_0`, `night_rain_1`, `night_rain_2`) are the natural grouping, captured in the `drive_name` field. The dataset itself carries no sample tags, and `dataset.info` is empty (no extra dataset-level metadata beyond the per-sample fields below; in particular, the camera/IR calibration matrices used to derive the 2D detections are not stored anywhere in the FiftyOne dataset — they were only used transiently at MCAP-conversion time). ### Fields | Field | FiftyOne type | Description | |-----------------------|--------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------| | `filepath` | `StringField` | Absolute path to the drive’s `.mcap` episode file — the sample’s multimodal media | | `drive_name` | `StringField` | Drive/recording-session identifier: `dusk_clear_0`, `night_rain_0`, `night_rain_1`, or `night_rain_2` | | `weather` | `StringField` | `clear` or `rain`, parsed from the drive name | | `lighting` | `StringField` | `dusk` or `night`, parsed from the drive name | | `is_preliminary_sync` | `BooleanField` | `True` for the 3 `night_rain_*` drives, `False` for `dusk_clear_0` — see [Parsing decisions](#parsing-decisions) | | `duration_s` | `FloatField` | Episode duration in seconds, computed from the MCAP’s message-time span | | `message_count` | `IntField` | Total MCAP message count across all channels in the episode | | `channel_count` | `IntField` | Total MCAP channel (topic) count in the episode (9 for every drive) | | `topics` | `ListField(StringField)` | Every ROS2 topic name present in the episode’s MCAP (not standardized across drives — see [Parsing decisions](#parsing-decisions)) | | `schemas` | `ListField(StringField)` | Every distinct ROS2 message schema name present in the episode’s MCAP (identical set across all 4 drives) | | `has_image` | `BooleanField` | Whether the episode has a camera stream FiftyOne’s Image tile can decode (`sensor_msgs/msg/Image`) — `True` for all 4 | | `has_pointcloud` | `BooleanField` | Whether it has a decodable point-cloud stream for the 3D tile (`sensor_msgs/msg/PointCloud2`, covers both LiDAR and radar topics) — `True` for all 4 | | `has_gps` | `BooleanField` | Whether it has a decodable GPS fix stream for the Map tile (`sensor_msgs/msg/NavSatFix`) — `True` for all 4 | | `has_imu` | `BooleanField` | Whether it has a decodable IMU stream for the Plot tile (`sensor_msgs/msg/Imu`) — `True` for all 4 | | `has_3d_labels` | `BooleanField` | Whether the episode has an embedded `/Labels_3D` topic (`vision_msgs/msg/Detection3DArray`) — `True` for all 4 | | `has_2d_labels` | `BooleanField` | Whether the episode has embedded `/Labels_2D_front` / `/Labels_2D_ir` topics (`vision_msgs/msg/Detection2DArray`) — `True` for all 4 | Standard FiftyOne bookkeeping fields (`id`, `tags`, `metadata`, `created_at`, `last_modified_at`) are also present but not source-specific. ### Label types and why **No FiftyOne label fields (`Detections`, `Detections3D`, etc.) are attached directly to the sample.** Because each sample is a multi-minute continuous recording rather than a single frame, there is no single fixed-length list a sample-level label field could hold. Instead, the 3D and 2D object annotations are embedded as additional ROS2 message topics inside the same MCAP timeline as the sensor data, decoded live by FiftyOne’s multimodal viewer alongside the camera/LiDAR/radar tiles, exactly like the sensor topics themselves: - **`/Labels_3D`** (`vision_msgs/msg/Detection3DArray`) — one message per labeled LiDAR frame. Each `Detection3D` carries the object’s class (`results[0].hypothesis.class_id`, one of `Car`/`Truck`/`Van`/ `Pedestrian`/`Bus`/`LongVehicle`), a persistent tracking `id` (the source’s `obj_id`), and a `bbox` (`vision_msgs/msg/BoundingBox3D`) giving the 3D position, size, and orientation in the LiDAR frame. The source label format’s rotation (Euler `roll`/`pitch`/`yaw` in radians, only `yaw` non-zero in every sample observed) is converted to a quaternion (`scipy.spatial.transform.Rotation`, `ZYX` intrinsic order) because `vision_msgs/msg/BoundingBox3D.center` is a `geometry_msgs/msg/Pose`. - **`/Labels_2D_front`** and **`/Labels_2D_ir`** (`vision_msgs/msg/Detection2DArray`) — the same objects, projected onto the monocular and thermal camera image planes respectively, as normalized pixel bounding boxes (`vision_msgs/msg/BoundingBox2D`). These 2D boxes are **derived, not part of the original annotation** — the source dataset only ships 3D LiDAR-frame tracklets; the 2D boxes here were computed by this repackaging using the authors’ own projection method (see [Data Collection and Processing](#data-collection-and-processing)). Both label topics are only published at timestamps that had a matching labeled frame in the source release — most LiDAR/camera frames in a drive are unlabeled, so `/Labels_3D`/`/Labels_2D_*` messages are sparser than the sensor topics they’re synced to (see [Parsing decisions](#parsing-decisions) for coverage numbers). The `has_3d_labels`/`has_2d_labels` sample fields exist so episodes can be filtered without opening every MCAP file first, e.g. `dataset.match(F("has_3d_labels") & F("is_preliminary_sync") == False)`. ### Schemas present across episodes All 4 episodes have the identical schema set: `sensor_msgs/msg/Image` (monocular + thermal camera, on separate topics), `sensor_msgs/msg/Imu`, `sensor_msgs/msg/NavSatFix`, `sensor_msgs/msg/PointCloud2` (LiDAR + radar, on separate topics), `vision_msgs/msg/Detection2DArray`, and `vision_msgs/msg/Detection3DArray`. Topic *names*, however, are not consistent across drives (e.g. `night_rain_0/1/2` use `/PCL` for LiDAR and `/Cam_Image` for the monocular camera, while `dusk_clear_0` uses `/Lidar` and `/Camera` instead) — the `has_*` fields and this dataset’s own conversion pipeline key off schema name, not topic string. ### Parsing decisions - **One sample = one episode.** Each sample corresponds to a single raw ROS2 bag recording (one drive), never split into per-frame samples — FiftyOne’s multimodal viewer handles playback and scrubbing within an episode. - **GPS coordinates were converted from raw NMEA `ddmm.mmmm` to decimal degrees** before being written into `NavSatFix` messages (e.g. raw `4315.387, -7951.7195` → `43.256, -79.862`, correct for Hamilton, ON). Left unconverted, the Map tile would place every episode in the wrong location. - **`header.stamp` was broken/unusable on the LiDAR and camera topics in every raw bag** (a placeholder `sec=0` with a non-wall-clock `nanosec` counter) and was replaced with the message’s own bag log/receive time. Radar, IMU, and GPS headers already carried correct epoch timestamps and were left as-is. - **`vision_msgs` message definitions were registered into the `rosbags` typestore** (`vision_msgs_types.py`), copied from the ROS2 galactic `vision_msgs` package, since `rosbags` does not ship them by default and they are required to write `Detection2DArray`/`Detection3DArray` messages into the MCAP. - **Label-to-sensor-message frame sync required content-based matching, not positional counting.** The source dataset publishes per-frame labels against a separately-published, frame-by-frame “extracted” release (not the raw bag), and that release numbers each sensor’s frames independently (e.g. `lidar/00002000.pcd`, `label/00002000.json`). An initial approach assumed “labeled frame N == the Nth bag message on that topic,” which only holds for the shortest drive (`night_rain_2`) — on the other 3 drives, the raw bag contains extra per-topic messages that were dropped when the extracted release was built, and the drop count grows through the drive (e.g. on `dusk_clear_0`, labeled frame 2000’s LiDAR scan is actually the *2137th* `/Lidar` message in the bag). The fix, implemented in `convert_bag_to_mcap.py`, walks the bag once per sensor role and matches each labeled frame to its true bag message by exact content comparison (pixel equality for images; near-exact point-coordinate equality for LiDAR, since the raw bag stores `float32` and the extracted PCDs store `float64`), using a small forward-sliding window to tolerate the rare local reordering also observed in the raw data (e.g. `night_rain_2`’s IR camera has its first two frames swapped relative to bag arrival order). All 4 episodes were verified to have **0 unmatched labels** across every sensor role on every drive after this fix. - **`is_preliminary_sync` marks a real synchronization caveat, not a processing choice.** Only the LiDAR triggers the monocular and IR cameras; radar, GPS, and IMU run independently and are only approximately time-aligned. Each `night_rain_*` drive’s own source `README.txt` states this explicitly (“the lidar and cameras are synchronized but are missing timestamp information… radar and gps/imu data… is not synchronized with the lidar and cameras”), and only `dusk_clear_0` ships without that warning and has near-perfectly matched per-modality frame counts across all sensors. - **`night_rain_2` has no radar data in this dataset.** Its frame-by-frame extracted release ships no `radar/` folder at all (radar exists only in that drive’s raw bag under the source’s own sync caveat above), so its MCAP’s `/Radar` topic reflects the raw bag’s radar stream directly rather than a curated/labeled subset like the other 3 drives. - **PCD double-precision (`x y z intensity`/`vr`, `SIZE 8`) files from the extracted release are used only as a content-matching fingerprint source** for the frame-sync fix above, not published into the MCAP directly — the `/Lidar`/`/PCL` and `/Radar` `PointCloud2` topics in the MCAP are the raw bag’s own (`float32`) point-cloud messages. - **No ROS `/tf`/`/tf_static` topic exists in any source bag** (every message header uses the placeholder `frame_id: "map"`), so sensor extrinsics are not resolvable via a transform tree — they only exist in the separate per-drive calibration JSON files, which were used at conversion time to compute `/Labels_2D_front`/`/Labels_2D_ir` and are not themselves stored in the resulting FiftyOne dataset. ## Dataset Creation ### Curation Rationale The source dataset’s stated rationale (per the paper) is that a sensor fusion dataset combining radar and IR alongside the more common LiDAR and camera — across a range of weather/lighting conditions — was, to the authors’ knowledge, first of its kind, and useful for developing and evaluating sensor fusion techniques that remain robust when individual sensors (LiDAR, camera) degrade in poor weather. This FiftyOne repackaging’s rationale is to make the full sensor + annotation stack explorable as a single, time-synchronized artifact using FiftyOne’s multimodal MCAP support, rather than requiring users to separately parse raw ROS2 bags and cross-reference a disjoint, independently-numbered frame-extraction release to see labels alongside sensor data. ### Source Data #### Data Collection and Processing Per the source paper: data was collected with a vehicle-mounted sensor platform consisting of a Velodyne HDL-32E LiDAR (10 Hz, 32 channels, 100 m ± 0.02 m range), a FLIR A65 thermal camera (30 Hz, 640×512 resolution), a Retina-4fn mmWave radar (20 Hz, 250 m range), and a Logitech Brio monocular camera (30/60 FPS, 13 MP), controlled by a central laptop running ROS2-galactic on Ubuntu Linux. The LiDAR triggers both cameras; radar and GPS/IMU run independently and are synchronized to the LiDAR by closest timestamp. Camera-LiDAR extrinsic/intrinsic calibration used a checkerboard method; IR calibration used a heated checkerboard with detachable black cells for contrast; radar-LiDAR alignment used a translation-only extrinsic (rotation was found negligible). Raw recordings were captured as ROS2 bags and separately post-processed into a frame-by-frame, per-sensor folder structure using ROS2’s `ApproximateTimeSynchronizer` (50 ns tolerance). For this repackaging, each drive’s raw ROS2 bag (SQLite3 storage) was converted to a single `.mcap` file using the [`rosbags`](https://ternaris.gitlab.io/rosbags/) Python library, with the data-quality fixes and label re-embedding described in [Parsing decisions](#parsing-decisions). The 2D detection boxes were computed with a reimplementation of the projection math in the authors’ own reference script (`example/CMHT_projection.py`): standard pinhole projection (`p_camera = R @ p_lidar + t`, then `K @ p_camera` and divide by depth), with `R`/`t` from the calibration file’s 4×4 extrinsic and `K` from its 3×3 intrinsic, both row-major. #### Who are the source data producers? The Centre for Mechatronics and Hybrid Technologies (CMHT), Department of Mechanical Engineering, and the Department of Computing and Software, both at McMaster University, Hamilton, Ontario, Canada — vehicle-mounted sensor platform driven around downtown Hamilton. ### Annotations #### Annotation process Per the source paper’s Specifications Table, the extracted frames were manually labeled using [SUSTechPoints](https://github.com/naurril/SUSTechPOINTS). Every LiDAR-detected object was labeled with a 3D bounding box (position, size, rotation) and a classification in `Car`, `Truck`, `Van`, `Pedestrian`, `Bus`, `LongVehicle`, plus a persistent object ID for tracking across frames. The 2D detection boxes in this repository (`/Labels_2D_front`, `/Labels_2D_ir`) were **not** manually annotated — they are derived by projecting the same 3D labels onto the camera image planes, as described in [Data Collection and Processing](#data-collection-and-processing). #### Who are the annotators? [More Information Needed] — the paper documents the annotation tool (SUSTechPoints) and process but does not name individual annotators. #### Personal and Sensitive Information The paper’s Ethics Statement states: “The authors have read the ethical requirements and confirm that this dataset does not involve any human subjects, animal experiments, or data from social media platforms.” Nonetheless, this is real-world street driving footage recorded in downtown Hamilton, Ontario — the monocular and thermal camera streams may incidentally depict identifiable pedestrians, vehicles, and license plates, and the GPS stream records the vehicle’s actual driving routes through public roads. No additional anonymization was applied by this repackaging. ## Citation **BibTeX:** ```bibtex @article{zhang2025cmht, title={CMHT autonomous dataset: A multi-sensor dataset including radar and IR for autonomous driving}, author={Zhang, Howard and Liu, Ash and Habibi, Saied and Mohrenschildt, Martin v. and Ahmed, Ryan}, journal={Data in Brief}, volume={60}, pages={111552}, year={2025}, publisher={Elsevier}, doi={10.1016/j.dib.2025.111552} } ``` **APA:** Zhang, H., Liu, A., Habibi, S., Mohrenschildt, M. v., & Ahmed, R. (2025). CMHT autonomous dataset: A multi-sensor dataset including radar and IR for autonomous driving. *Data in Brief, 60*, 111552. https://doi.org/10.1016/j.dib.2025.111552 ## More Information This repository is an independently-curated MCAP/FiftyOne repackaging of the official CMHT Autonomous Dataset. It is not an official CMHT/McMaster artifact. For the original release (raw ROS2 bags, frame-by-frame extracted sensor data, and label JSON files as published by the authors), see the [FRDR record](https://doi.org/10.20383/103.01024) and the [Data in Brief paper](https://doi.org/10.1016/j.dib.2025.111552). ## Dataset Card Authors Harpreet Sahota ([@harpreetsahota](https://huggingface.co/harpreetsahota)) — MCAP repackaging, label re-embedding, and this card. Original dataset authors are listed under [Dataset Description](#dataset-description). ## Dataset Card Contact Harpreet Sahota — https://huggingface.co/harpreetsahota