#### NOTE
This is a **Hugging Face dataset**. For large datasets, ensure `huggingface_hub>=1.1.3` to avoid rate limits. Learn more in the Hugging Face integration docs.
# Dataset Card for OctoSense (FiftyOne Multimodal)
OctoSense is a time-synchronized, calibrated multi-sensor robot perception
dataset captured by an open-source eight-sensor rig on three platforms: a car,
a boat, and a Unitree Go2-W quadruped. This FiftyOne version packages eight
representative episodes as MCAP files — one sample per episode — loadable
directly in the FiftyOne App’s multimodal viewer with synchronized playback
across all sensor streams.
## Installation
```bash
pip install -U fiftyone
```
## Usage
```python
import fiftyone as fo
import fiftyone.utils.huggingface as fouh
dataset = fouh.load_from_hub("Voxel51/OctoSense")
session = fo.launch_app(dataset)
```
## Dataset Details
### Dataset Description
OctoSense provides 59 hours of time-synchronized driving data from eight
sensors: stereo RGB and event cameras, a thermal camera, an Ouster OS1-64
LiDAR, an IMU, RTK-corrected GPS, and vehicle CAN bus (car) or joint-angle
proprioception (quadruped). Sensors are hardware PPS-locked to a single clock
and post-processed with Kalman-smoothed affine synchronization. All sensors
record at native rates without dropped data: the RGB pair at 100 Hz, event
cameras at up to ~7 MEv/s, LiDAR at 10 Hz, IMU at 400 Hz (VectorNav) and
100 Hz (in-LiDAR), and GPS at 5 Hz. Car sequences additionally include
ego-motion optical flow, dense LiDAR-projected depth, pseudo-label semantic
segmentation, LiDAR-inertial odometry, and a fused GPS/LIO trajectory. The
dataset spans urban, suburban, and rural environments at sunrise through
nighttime, and includes deliberately degraded-sensor recordings.
- **Curated by:** Harpreet Sahota (FiftyOne multimodal packaging)
- **Original dataset by:** Anthony Bisulco, Jeremy Wang, Kostas Daniilidis, Randall Balestriero, Pratik Chaudhari (GRASP Laboratory, University of Pennsylvania; Brown University)
- **Funded by:** [More Information Needed]
- **Shared by:** Harpreet Sahota
- **Language(s):** Not applicable (sensor data)
- **License:** MIT
### Dataset Sources
- **Repository:** https://huggingface.co/datasets/anthonytec2/OctoSense
- **Paper:** https://arxiv.org/abs/2606.27317
- **Demo:** https://abisulco.com/octosense/
## Uses
### Direct Use
- Visual inspection and debugging of synchronized multi-sensor recordings in
the FiftyOne multimodal viewer (Image, 3D, Map, Plot, and Logs tiles on a
shared timeline).
- Filtering episodes by platform, split, sensor coverage, geographic bounding
box, speed, idle fraction, or ground-truth availability using the FiftyOne
SDK and App sidebar.
- Prototyping multi-modal perception pipelines against a known-good subset
before scaling to the full 382-sequence release.
- Evaluating sensor coverage and calibration quality across platforms before
importing additional sequences.
### Out-of-Scope Use
This FiftyOne package contains eight sequences (a curated subset). Training
production models requires the full OctoSense release
(`anthonytec2/OctoSense`). Radar range and bearing values use fitted scales
(1/16 m per count, −1 bearing sign) derived from LiDAR cross-validation;
they are not manufacturer-documented and should not be used as ground truth
for radar calibration.
## Dataset Structure
### Overview
**Media type:** `multimodal` — one `fo.Sample` per episode, `filepath`
pointing to a `.mcap` file. All sensor streams are encoded inside the MCAP
with time-aligned messages on a shared nanosecond clock. The FiftyOne App
renders them in synchronized Image, 3D, Map, Plot, and Logs tiles.
**Sample count:** 8 episodes (6 car, 1 boat, 1 Unitree quadruped)
**Total recording time:** ~46 minutes across all 8 episodes
### Sample Fields
| Field | FiftyOne type | Description |
|------------------|--------------------------|-------------------------------------------------------------------|
| `filepath` | `StringField` | Absolute path to the episode’s `.mcap` file |
| `bag_id` | `StringField` | ROS 2 bag identifier (e.g. `rosbag2_2026_01_09-11_32_05`) |
| `session` | `StringField` | Session label within the platform (e.g. `sess8`) |
| `platform` | `StringField` | Recording platform: `car`, `boat`, or `unitree` |
| `start_time` | `StringField` | Sequence wall-clock start time (ISO 8601) |
| `split` | `StringField` | `train` or `test` for car episodes; null for boat and unitree |
| `is_daytime` | `BooleanField` | Whether the sequence was recorded in daylight |
| `degraded` | `BooleanField` | Whether sensor degradation was logged for this sequence |
| `has_seg` | `BooleanField` | Whether pseudo-label semantic segmentation GT is available |
| `duration_s` | `FloatField` | Episode duration in seconds (from MCAP message timestamps) |
| `message_count` | `IntField` | Total MCAP messages across all channels |
| `channel_count` | `IntField` | Number of distinct MCAP channels |
| `topics` | `ListField(StringField)` | All MCAP topic strings in the episode |
| `schemas` | `ListField(StringField)` | All MCAP schema names (use for capability filtering) |
| `n_lidar_frames` | `IntField` | Number of LiDAR sweeps |
| `n_rgb_frames` | `IntField` | Number of RGB camera frames |
| `n_imu_samples` | `IntField` | Number of IMU samples |
| `n_gps_fixes` | `IntField` | GPS fixes logged (includes zero-fix rows on unitree) |
| `n_gps_valid` | `FloatField` | GPS fixes with valid lock |
| `n_events_left` | `IntField` | Total events from left event camera |
| `n_events_right` | `IntField` | Total events from right event camera |
| `gps_quality` | `StringField` | GPS fix quality string (e.g. `single_m`, `rtk`) |
| `gps_lat_min` | `FloatField` | Southern edge of episode geographic bounding box (degrees) |
| `gps_lat_max` | `FloatField` | Northern edge of episode geographic bounding box (degrees) |
| `gps_lon_min` | `FloatField` | Western edge of episode geographic bounding box (degrees) |
| `gps_lon_max` | `FloatField` | Eastern edge of episode geographic bounding box (degrees) |
| `mean_speed_mph` | `FloatField` | Mean driving speed in mph |
| `idle_fraction` | `FloatField` | Fraction of recording time at near-zero speed |
| `distance_m` | `FloatField` | Total distance driven in metres |
| `rgb_cal_id` | `StringField` | Camera calibration file identifier |
| `imu_cal_id` | `StringField` | IMU calibration file identifier |
| `lidar_cal_id` | `StringField` | LiDAR calibration file identifier |
| `sensor_dropout` | `StringField` | Dropout description if a sensor outage was logged; null otherwise |
### MCAP Stream Contents
Each `.mcap` file is a multi-channel recording. Topics and their schemas vary
by platform:
| Topic | Schema | Tile | Platforms |
|------------------------------|---------------------------------------------|--------|-------------|
| `/camera/left/image_raw` | `foxglove.CompressedVideo` (H.264, 10 fps) | Image | all |
| `/camera/right/image_raw` | `foxglove.CompressedVideo` (H.264, 10 fps) | Image | all |
| `/camera/infrared/image_raw` | `foxglove.CompressedVideo` (H.264, ~50 fps) | Image | all |
| `/camera/left_rect/depth` | `foxglove.RawImage` (32fc1 metres, 10 Hz) | Image | car |
| `/camera/left_rect/semantic` | `foxglove.CompressedImage` (PNG, 10 Hz) | Image | car daytime |
| `/camera/left_rect/flow` | `foxglove.CompressedVideo` (H.264, 10 fps) | Image | car |
| `/lidar/points` | `foxglove.PointCloud` (10 Hz) | 3D | all |
| `/gps` | `foxglove.LocationFix` | Map | car, boat |
| `/odom` | `foxglove.PoseInFrame` | 3D | car |
| `/imu/vectornav` | `imu_vectornav` (JSON) | Plot | all |
| `/imu/ouster` | `imu_ouster` (JSON) | Plot | all |
| `/car/*` | JSON scalars (speed, wheels, steer, CAN) | Plot | car |
| `/robot/*` | JSON scalars (low_state, sport, controller) | Plot | unitree |
| `/captions` | `foxglove.Log` | Logs | car |
| `/tf_static` | `foxglove.FrameTransforms` | — | all |
### Ground Truth Coverage
| Ground truth | Source | Coverage | Notes |
|-------------------------|---------------------------------------------|--------------------------------------------------------------|--------------------------------------------------------|
| Dense depth | LiDAR accumulation, `32fc1` metres | All 6 car episodes | ~12 % of pixels valid; 3–102 m range |
| Semantic segmentation | EoMT-Cityscapes-DINOv2-L-1024 pseudo-labels | Daytime car only (all 5 daytime car episodes in this subset) | 19 Cityscapes classes; `255` = ignore |
| Ego-motion optical flow | Derived at conversion from depth + poses | All 6 car episodes | Ego-motion only; moving objects are holes |
| Radar tracks | Mazda CAN `track_{1..6}` | Car | Range 1/16 m per count (fitted); bearing sign inverted |
### Filtering by Capability
Use the `schemas` list field to filter episodes rather than topic names (topic
substrings false-positive against non-decodable packet channels):
```python
import fiftyone as fo
from fiftyone import ViewField as F
dataset = fo.load_dataset("octosense")
# Car episodes with semantic segmentation
car_with_seg = dataset.match(F("has_seg") == True)
# Episodes with GPS
has_gps = dataset.match(F("schemas").contains("foxglove.LocationFix"))
# Filter by platform
cars = dataset.match(F("platform") == "car")
```
## Dataset Creation
### Curation Rationale
The eight episodes were selected to provide maximum coverage of platforms,
modalities, and conditions within a manageable subset:
- **Daytime car with all GT** (`car/sess8`, 53 s): shortest car sequence with
depth, semantic segmentation, and all CAN channels present.
- **Car sessions** (sess8 long, sess9, sess10 ×2, sess11): 163–597 s each,
from both train and test splits, spanning all four car sessions; selected
for near-zero sensor dropout, valid GPS lock, and minimal idle time.
- **Boat** (`boat/sess1`, 54 s): the only platform where the LiDAR ships raw
range images rather than XYZ, and where the boat IMU extrinsic differs.
- **Unitree** (`unitree/sess1`, 112 s): the only platform with `/robot` joint
state and no GPS or CAN data.
### Source Data
#### Data Collection and Processing
Data was collected from a single OctoSense rig mounted on each platform.
All sensors are hardware PPS-locked for temporal alignment and
post-processed with Kalman-smoothed affine synchronization. Source recordings
are ROS 2 bags converted to HDF5 and H.265 video files; the raw release is
hosted at `anthonytec2/OctoSense` on Hugging Face (~8.5 TB). This FiftyOne
package re-encodes video streams from HEVC to H.264 (libx264, no B-frames,
one access unit per MCAP message) for browser WebCodecs compatibility and
packages all channels into MCAP files using the Foxglove schema registry.
LiDAR points are stored at a 20-byte stride (x, y, z, intensity as float32;
r, g, b, a as uint8), with colour sampled from the nearest left-camera frame.
#### Who are the source data producers?
The OctoSense hardware platform and dataset were developed at the General,
Robotics, Automation, Sensing and Perception (GRASP) Laboratory, University
of Pennsylvania (Anthony Bisulco, Jeremy Wang, Kostas Daniilidis, Pratik
Chaudhari) and Brown University (Randall Balestriero). Data was collected
on Long Island and in Philadelphia, PA, USA.
### Annotations
#### Annotation process
All ground truth in OctoSense is machine-generated:
- **Depth** is accumulated from LiDAR sweeps projected onto the rectified left
camera frame.
- **Semantic segmentation** is produced by EoMT-Cityscapes-DINOv2-L-1024
applied to CLAHE-enhanced rectified left frames; it is a model’s output, not
human annotation.
- **Optical flow** is derived at MCAP-conversion time from depth and
LiDAR-inertial poses (`flow_gap = 2` frames, i.e. 0.2 s lookahead); it
captures ego-motion only — pixels covering dynamic objects (vehicles,
pedestrians) were excluded from the depth accumulation and appear as invalid.
- **Radar detection geometry** (range, bearing) uses scales fitted against
LiDAR-projected vehicle pixels in bird’s-eye view (1/16 m per `DIST_OBJ`
count, negative `ANG_OBJ` sign). These are not manufacturer-documented values.
#### Who are the annotators?
No human annotators. All labels are generated programmatically from sensor
data and pre-trained models as described above.
#### Personal and Sensitive Information
Recordings were made on public roads and a private waterway. No personally
identifiable information was intentionally captured. Faces and licence plates
visible in RGB frames are not blurred in the source data.
## Citation
**BibTeX:**
```bibtex
@article{bisulco2026octosense,
title={OctoSense: Self-Supervised Learning for Multimodal Robot Perception},
author={Bisulco, Anthony and Wang, Jeremy and Daniilidis, Kostas and Balestriero, Randall and Chaudhari, Pratik},
journal={arXiv preprint arXiv:2606.27317},
year={2026}
}
```
**APA:**
Bisulco, A., Wang, J., Daniilidis, K., Balestriero, R., & Chaudhari, P. (2026). OctoSense: Self-Supervised Learning for Multimodal Robot Perception. *arXiv preprint arXiv:2606.27317*.
## More Information
- Full dataset (8.5 TB, 382 sequences): https://huggingface.co/datasets/anthonytec2/OctoSense
- Devkit and conversion scripts: https://github.com/anthonytec2/OctoSense
- Project page and supplementary videos: https://abisulco.com/octosense/
- FiftyOne multimodal documentation: https://docs.voxel51.com/user_guide/app.html#multimodal-viewer
## Dataset Card Authors
Harpreet Sahota
## Dataset Card Contact
harpreetsahota