Note

This is a Hugging Face dataset. For large datasets, ensure huggingface_hub>=1.1.3 to avoid rate limits. Learn more in the Hugging Face integration docs.

Hugging Face

Dataset Card for CMHT Autonomous Driving Multimodal (MCAP)#

image/png

This is a FiftyOne dataset with 4 samples.

Installation#

If you haven’t already, install FiftyOne:

pip install -U fiftyone

Usage#

import fiftyone as fo
from fiftyone.utils.huggingface import load_from_hub

# Load the dataset
# Note: other available arguments include 'max_samples', etc
dataset = load_from_hub("Voxel51/cmht-autonomous-driving")

# Launch the App
session = fo.launch_app(dataset)

Dataset Details#

Dataset Description#

The CMHT Autonomous Dataset is a multi-sensor autonomous-driving dataset recorded around Hamilton, Ontario by the Centre for Mechatronics and Hybrid Technologies (CMHT) at McMaster University. A single vehicle rig combines a Velodyne HDL-32E LiDAR, a Retina-4fn mmWave radar, a Logitech Brio monocular RGB camera, a FLIR A65 thermal (IR) camera, and GPS/IMU (built into the LiDAR unit), recorded live with ROS2 (galactic) into ros2 bag recordings. The dataset consists of over 9,000 labeled frames captured at 10-20 Hz across four drives spanning dusk/clear and night/rain conditions in downtown Hamilton, with every LiDAR-detected object labeled with its 3D position, size, rotation, classification, and object ID.

This repository repackages the four raw ROS2 bag recordings as time-synchronized MCAP episodes for FiftyOne’s native multimodal dataset support, with the original per-frame 3D tracklet labels (published separately by the authors as a “frame-by-frame extracted” release) embedded directly back into each episode’s MCAP timeline as ROS2 vision_msgs detection topics, synced to the exact sensor message each label was originally annotated from. Each sample is one continuous drive, viewable in FiftyOne’s tiled multimodal viewer with synchronized camera, thermal camera, LiDAR point cloud, radar, GPS, IMU, and 3D/2D object-detection playback.

  • Curated by: Howard Zhang, Ash Liu, Saied Habibi, Martin v. Mohrenschildt, and Ryan Ahmed (Centre for Mechatronics and Hybrid Technologies, McMaster University) — original data collection, sensor rig, and 3D tracklet labeling. This MCAP/FiftyOne multimodal repackaging (ROS2-bag-to-MCAP conversion, data-quality fixes, and re-embedding of the separately-published labels into the MCAP timeline) was prepared independently by Harpreet Sahota.

  • Funded by: None — per the paper’s Acknowledgements: “This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.”

  • Shared by: Harpreet Sahota (this repackaging); the original CMHT Autonomous Dataset is shared by McMaster University via the Federated Research Data Repository (FRDR) and a MacDrive file share.

  • Language(s): N/A (sensor data — camera, thermal camera, LiDAR, radar, GPS/IMU; no text).

  • License: CC0 1.0, per the FRDR record. (The accompanying Data in Brief paper text itself is published under CC BY 4.0 — that license applies to the paper, not the dataset files.)

Dataset Sources#

Uses#

Direct Use#

  • Multi-sensor fusion research combining LiDAR, radar, monocular camera, and thermal (IR) camera, including comparing sensor efficacy across weather/lighting conditions (dusk/clear vs. night/rain).

  • Training or evaluating 3D and 2D object detection/tracking models against the embedded ground-truth Car/Truck/Van/Pedestrian/Bus/LongVehicle tracklets (object IDs are preserved for tracking-across-time use cases).

  • Exercising and demoing FiftyOne’s multimodal MCAP support: synchronized playback of camera, thermal, 3D point cloud (LiDAR + radar), GPS/map, and embedded 2D/3D detection tiles across real autonomous-driving recordings.

  • Studying sensor-fusion performance in adverse conditions specifically, since 3 of the 4 drives were recorded at night in rain.

Out-of-Scope Use#

  • Using this as a large-scale benchmark comparable to KITTI, nuScenes, or Waymo — this is 4 drives with roughly 9,000 labeled frames total, an order of magnitude smaller than those datasets (an explicitly stated limitation in the source paper).

  • Relying on LiDAR resolution comparable to newer 64+ channel sensors — the Velodyne HDL-32E is a 32-channel unit, lower resolution than many current autonomous-driving LiDARs (also a stated paper limitation).

  • Sensor-fusion use cases that require overlapping fields of view across all sensors everywhere in the scene — the rig’s sensors only share overlapping FOV at the front of the vehicle (a stated paper limitation).

  • Pedestrian-heavy benchmarking or training — the paper notes the dataset has comparatively low pedestrian representation, which “may cause detrimental effects in AI training/testing.”

  • Treating the radar/GPS/IMU streams in the 3 night_rain_* drives as frame-exact synchronized with the camera/LiDAR streams — see is_preliminary_sync in Parsing decisions.

  • Any use requiring the original raw radar stream for night_rain_2 — the radar-labeled frame release did not include this drive (see Parsing decisions).

Dataset Structure#

This is a flat (ungrouped) FiftyOne dataset with media_type: "multimodal" and 4 samples. Each sample is one episode — a single continuous drive/recording session — stored as one .mcap file; FiftyOne infers the multimodal media type automatically from the .mcap file extension. There are no separate image, point-cloud, or video samples: the drive is the sample unit, and every stream inside it (camera, thermal camera, LiDAR, radar, GPS, IMU, and the embedded object-detection topics) is decoded live by FiftyOne’s multimodal viewer. No dataset-level splits are provided by the source data; the 4 drives (dusk_clear_0, night_rain_0, night_rain_1, night_rain_2) are the natural grouping, captured in the drive_name field. The dataset itself carries no sample tags, and dataset.info is empty (no extra dataset-level metadata beyond the per-sample fields below; in particular, the camera/IR calibration matrices used to derive the 2D detections are not stored anywhere in the FiftyOne dataset — they were only used transiently at MCAP-conversion time).

Fields#

Field

FiftyOne type

Description

filepath

StringField

Absolute path to the drive’s .mcap episode file — the sample’s multimodal media

drive_name

StringField

Drive/recording-session identifier: dusk_clear_0, night_rain_0, night_rain_1, or night_rain_2

weather

StringField

clear or rain, parsed from the drive name

lighting

StringField

dusk or night, parsed from the drive name

is_preliminary_sync

BooleanField

True for the 3 night_rain_* drives, False for dusk_clear_0 — see Parsing decisions

duration_s

FloatField

Episode duration in seconds, computed from the MCAP’s message-time span

message_count

IntField

Total MCAP message count across all channels in the episode

channel_count

IntField

Total MCAP channel (topic) count in the episode (9 for every drive)

topics

ListField(StringField)

Every ROS2 topic name present in the episode’s MCAP (not standardized across drives — see Parsing decisions)

schemas

ListField(StringField)

Every distinct ROS2 message schema name present in the episode’s MCAP (identical set across all 4 drives)

has_image

BooleanField

Whether the episode has a camera stream FiftyOne’s Image tile can decode (sensor_msgs/msg/Image) — True for all 4

has_pointcloud

BooleanField

Whether it has a decodable point-cloud stream for the 3D tile (sensor_msgs/msg/PointCloud2, covers both LiDAR and radar topics) — True for all 4

has_gps

BooleanField

Whether it has a decodable GPS fix stream for the Map tile (sensor_msgs/msg/NavSatFix) — True for all 4

has_imu

BooleanField

Whether it has a decodable IMU stream for the Plot tile (sensor_msgs/msg/Imu) — True for all 4

has_3d_labels

BooleanField

Whether the episode has an embedded /Labels_3D topic (vision_msgs/msg/Detection3DArray) — True for all 4

has_2d_labels

BooleanField

Whether the episode has embedded /Labels_2D_front / /Labels_2D_ir topics (vision_msgs/msg/Detection2DArray) — True for all 4

Standard FiftyOne bookkeeping fields (id, tags, metadata, created_at, last_modified_at) are also present but not source-specific.

Label types and why#

No FiftyOne label fields (Detections, Detections3D, etc.) are attached directly to the sample. Because each sample is a multi-minute continuous recording rather than a single frame, there is no single fixed-length list a sample-level label field could hold. Instead, the 3D and 2D object annotations are embedded as additional ROS2 message topics inside the same MCAP timeline as the sensor data, decoded live by FiftyOne’s multimodal viewer alongside the camera/LiDAR/radar tiles, exactly like the sensor topics themselves:

  • /Labels_3D (vision_msgs/msg/Detection3DArray) — one message per labeled LiDAR frame. Each Detection3D carries the object’s class (results[0].hypothesis.class_id, one of Car/Truck/Van/ Pedestrian/Bus/LongVehicle), a persistent tracking id (the source’s obj_id), and a bbox (vision_msgs/msg/BoundingBox3D) giving the 3D position, size, and orientation in the LiDAR frame. The source label format’s rotation (Euler roll/pitch/yaw in radians, only yaw non-zero in every sample observed) is converted to a quaternion (scipy.spatial.transform.Rotation, ZYX intrinsic order) because vision_msgs/msg/BoundingBox3D.center is a geometry_msgs/msg/Pose.

  • /Labels_2D_front and /Labels_2D_ir (vision_msgs/msg/Detection2DArray) — the same objects, projected onto the monocular and thermal camera image planes respectively, as normalized pixel bounding boxes (vision_msgs/msg/BoundingBox2D). These 2D boxes are derived, not part of the original annotation — the source dataset only ships 3D LiDAR-frame tracklets; the 2D boxes here were computed by this repackaging using the authors’ own projection method (see Data Collection and Processing).

Both label topics are only published at timestamps that had a matching labeled frame in the source release — most LiDAR/camera frames in a drive are unlabeled, so /Labels_3D//Labels_2D_* messages are sparser than the sensor topics they’re synced to (see Parsing decisions for coverage numbers). The has_3d_labels/has_2d_labels sample fields exist so episodes can be filtered without opening every MCAP file first, e.g. dataset.match(F("has_3d_labels") & F("is_preliminary_sync") == False).

Schemas present across episodes#

All 4 episodes have the identical schema set: sensor_msgs/msg/Image (monocular + thermal camera, on separate topics), sensor_msgs/msg/Imu, sensor_msgs/msg/NavSatFix, sensor_msgs/msg/PointCloud2 (LiDAR + radar, on separate topics), vision_msgs/msg/Detection2DArray, and vision_msgs/msg/Detection3DArray. Topic names, however, are not consistent across drives (e.g. night_rain_0/1/2 use /PCL for LiDAR and /Cam_Image for the monocular camera, while dusk_clear_0 uses /Lidar and /Camera instead) — the has_* fields and this dataset’s own conversion pipeline key off schema name, not topic string.

Parsing decisions#

  • One sample = one episode. Each sample corresponds to a single raw ROS2 bag recording (one drive), never split into per-frame samples — FiftyOne’s multimodal viewer handles playback and scrubbing within an episode.

  • GPS coordinates were converted from raw NMEA ddmm.mmmm to decimal degrees before being written into NavSatFix messages (e.g. raw 4315.387, -7951.7195 → 43.256, -79.862, correct for Hamilton, ON). Left unconverted, the Map tile would place every episode in the wrong location.

  • header.stamp was broken/unusable on the LiDAR and camera topics in every raw bag (a placeholder sec=0 with a non-wall-clock nanosec counter) and was replaced with the message’s own bag log/receive time. Radar, IMU, and GPS headers already carried correct epoch timestamps and were left as-is.

  • vision_msgs message definitions were registered into the rosbags typestore (vision_msgs_types.py), copied from the ROS2 galactic vision_msgs package, since rosbags does not ship them by default and they are required to write Detection2DArray/Detection3DArray messages into the MCAP.

  • Label-to-sensor-message frame sync required content-based matching, not positional counting. The source dataset publishes per-frame labels against a separately-published, frame-by-frame “extracted” release (not the raw bag), and that release numbers each sensor’s frames independently (e.g. lidar/00002000.pcd, label/00002000.json). An initial approach assumed “labeled frame N == the Nth bag message on that topic,” which only holds for the shortest drive (night_rain_2) — on the other 3 drives, the raw bag contains extra per-topic messages that were dropped when the extracted release was built, and the drop count grows through the drive (e.g. on dusk_clear_0, labeled frame 2000’s LiDAR scan is actually the 2137th /Lidar message in the bag). The fix, implemented in convert_bag_to_mcap.py, walks the bag once per sensor role and matches each labeled frame to its true bag message by exact content comparison (pixel equality for images; near-exact point-coordinate equality for LiDAR, since the raw bag stores float32 and the extracted PCDs store float64), using a small forward-sliding window to tolerate the rare local reordering also observed in the raw data (e.g. night_rain_2’s IR camera has its first two frames swapped relative to bag arrival order). All 4 episodes were verified to have 0 unmatched labels across every sensor role on every drive after this fix.

  • is_preliminary_sync marks a real synchronization caveat, not a processing choice. Only the LiDAR triggers the monocular and IR cameras; radar, GPS, and IMU run independently and are only approximately time-aligned. Each night_rain_* drive’s own source README.txt states this explicitly (“the lidar and cameras are synchronized but are missing timestamp information… radar and gps/imu data… is not synchronized with the lidar and cameras”), and only dusk_clear_0 ships without that warning and has near-perfectly matched per-modality frame counts across all sensors.

  • night_rain_2 has no radar data in this dataset. Its frame-by-frame extracted release ships no radar/ folder at all (radar exists only in that drive’s raw bag under the source’s own sync caveat above), so its MCAP’s /Radar topic reflects the raw bag’s radar stream directly rather than a curated/labeled subset like the other 3 drives.

  • PCD double-precision (x y z intensity/vr, SIZE 8) files from the extracted release are used only as a content-matching fingerprint source for the frame-sync fix above, not published into the MCAP directly — the /Lidar//PCL and /Radar PointCloud2 topics in the MCAP are the raw bag’s own (float32) point-cloud messages.

  • No ROS /tf//tf_static topic exists in any source bag (every message header uses the placeholder frame_id: "map"), so sensor extrinsics are not resolvable via a transform tree — they only exist in the separate per-drive calibration JSON files, which were used at conversion time to compute /Labels_2D_front//Labels_2D_ir and are not themselves stored in the resulting FiftyOne dataset.

Dataset Creation#

Curation Rationale#

The source dataset’s stated rationale (per the paper) is that a sensor fusion dataset combining radar and IR alongside the more common LiDAR and camera — across a range of weather/lighting conditions — was, to the authors’ knowledge, first of its kind, and useful for developing and evaluating sensor fusion techniques that remain robust when individual sensors (LiDAR, camera) degrade in poor weather.

This FiftyOne repackaging’s rationale is to make the full sensor + annotation stack explorable as a single, time-synchronized artifact using FiftyOne’s multimodal MCAP support, rather than requiring users to separately parse raw ROS2 bags and cross-reference a disjoint, independently-numbered frame-extraction release to see labels alongside sensor data.

Source Data#

Data Collection and Processing#

Per the source paper: data was collected with a vehicle-mounted sensor platform consisting of a Velodyne HDL-32E LiDAR (10 Hz, 32 channels, 100 m ± 0.02 m range), a FLIR A65 thermal camera (30 Hz, 640×512 resolution), a Retina-4fn mmWave radar (20 Hz, 250 m range), and a Logitech Brio monocular camera (30/60 FPS, 13 MP), controlled by a central laptop running ROS2-galactic on Ubuntu Linux. The LiDAR triggers both cameras; radar and GPS/IMU run independently and are synchronized to the LiDAR by closest timestamp. Camera-LiDAR extrinsic/intrinsic calibration used a checkerboard method; IR calibration used a heated checkerboard with detachable black cells for contrast; radar-LiDAR alignment used a translation-only extrinsic (rotation was found negligible). Raw recordings were captured as ROS2 bags and separately post-processed into a frame-by-frame, per-sensor folder structure using ROS2’s ApproximateTimeSynchronizer (50 ns tolerance).

For this repackaging, each drive’s raw ROS2 bag (SQLite3 storage) was converted to a single .mcap file using the rosbags Python library, with the data-quality fixes and label re-embedding described in Parsing decisions. The 2D detection boxes were computed with a reimplementation of the projection math in the authors’ own reference script (example/CMHT_projection.py): standard pinhole projection (p_camera = R @ p_lidar + t, then K @ p_camera and divide by depth), with R/t from the calibration file’s 4×4 extrinsic and K from its 3×3 intrinsic, both row-major.

Who are the source data producers?#

The Centre for Mechatronics and Hybrid Technologies (CMHT), Department of Mechanical Engineering, and the Department of Computing and Software, both at McMaster University, Hamilton, Ontario, Canada — vehicle-mounted sensor platform driven around downtown Hamilton.

Annotations#

Annotation process#

Per the source paper’s Specifications Table, the extracted frames were manually labeled using SUSTechPoints. Every LiDAR-detected object was labeled with a 3D bounding box (position, size, rotation) and a classification in Car, Truck, Van, Pedestrian, Bus, LongVehicle, plus a persistent object ID for tracking across frames. The 2D detection boxes in this repository (/Labels_2D_front, /Labels_2D_ir) were not manually annotated — they are derived by projecting the same 3D labels onto the camera image planes, as described in Data Collection and Processing.

Who are the annotators?#

[More Information Needed] — the paper documents the annotation tool (SUSTechPoints) and process but does not name individual annotators.

Personal and Sensitive Information#

The paper’s Ethics Statement states: “The authors have read the ethical requirements and confirm that this dataset does not involve any human subjects, animal experiments, or data from social media platforms.” Nonetheless, this is real-world street driving footage recorded in downtown Hamilton, Ontario — the monocular and thermal camera streams may incidentally depict identifiable pedestrians, vehicles, and license plates, and the GPS stream records the vehicle’s actual driving routes through public roads. No additional anonymization was applied by this repackaging.

Citation#

BibTeX:

@article{zhang2025cmht,
  title={CMHT autonomous dataset: A multi-sensor dataset including radar and IR for autonomous driving},
  author={Zhang, Howard and Liu, Ash and Habibi, Saied and Mohrenschildt, Martin v. and Ahmed, Ryan},
  journal={Data in Brief},
  volume={60},
  pages={111552},
  year={2025},
  publisher={Elsevier},
  doi={10.1016/j.dib.2025.111552}
}

APA:

Zhang, H., Liu, A., Habibi, S., Mohrenschildt, M. v., & Ahmed, R. (2025). CMHT autonomous dataset: A multi-sensor dataset including radar and IR for autonomous driving. Data in Brief, 60, 111552. https://doi.org/10.1016/j.dib.2025.111552

More Information#

This repository is an independently-curated MCAP/FiftyOne repackaging of the official CMHT Autonomous Dataset. It is not an official CMHT/McMaster artifact. For the original release (raw ROS2 bags, frame-by-frame extracted sensor data, and label JSON files as published by the authors), see the FRDR record and the Data in Brief paper.

Dataset Card Authors#

Harpreet Sahota (@harpreetsahota) — MCAP repackaging, label re-embedding, and this card. Original dataset authors are listed under Dataset Description.

Dataset Card Contact#

Harpreet Sahota — https://huggingface.co/harpreetsahota