Note

This is a Hugging Face dataset. For large datasets, ensure huggingface_hub>=1.1.3 to avoid rate limits. Learn more in the Hugging Face integration docs.

Hugging Face

Dataset Card for OctoSense (FiftyOne Multimodal)#

image/png

OctoSense is a time-synchronized, calibrated multi-sensor robot perception dataset captured by an open-source eight-sensor rig on three platforms: a car, a boat, and a Unitree Go2-W quadruped. This FiftyOne version packages eight representative episodes as MCAP files — one sample per episode — loadable directly in the FiftyOne App’s multimodal viewer with synchronized playback across all sensor streams.

Installation#

pip install -U fiftyone

Usage#

import fiftyone as fo
import fiftyone.utils.huggingface as fouh

dataset = fouh.load_from_hub("Voxel51/OctoSense")
session = fo.launch_app(dataset)

Dataset Details#

Dataset Description#

OctoSense provides 59 hours of time-synchronized driving data from eight sensors: stereo RGB and event cameras, a thermal camera, an Ouster OS1-64 LiDAR, an IMU, RTK-corrected GPS, and vehicle CAN bus (car) or joint-angle proprioception (quadruped). Sensors are hardware PPS-locked to a single clock and post-processed with Kalman-smoothed affine synchronization. All sensors record at native rates without dropped data: the RGB pair at 100 Hz, event cameras at up to ~7 MEv/s, LiDAR at 10 Hz, IMU at 400 Hz (VectorNav) and 100 Hz (in-LiDAR), and GPS at 5 Hz. Car sequences additionally include ego-motion optical flow, dense LiDAR-projected depth, pseudo-label semantic segmentation, LiDAR-inertial odometry, and a fused GPS/LIO trajectory. The dataset spans urban, suburban, and rural environments at sunrise through nighttime, and includes deliberately degraded-sensor recordings.

  • Curated by: Harpreet Sahota (FiftyOne multimodal packaging)

  • Original dataset by: Anthony Bisulco, Jeremy Wang, Kostas Daniilidis, Randall Balestriero, Pratik Chaudhari (GRASP Laboratory, University of Pennsylvania; Brown University)

  • Funded by: [More Information Needed]

  • Shared by: Harpreet Sahota

  • Language(s): Not applicable (sensor data)

  • License: MIT

Dataset Sources#

  • Repository: https://huggingface.co/datasets/anthonytec2/OctoSense

  • Paper: https://arxiv.org/abs/2606.27317

  • Demo: https://abisulco.com/octosense/

Uses#

Direct Use#

  • Visual inspection and debugging of synchronized multi-sensor recordings in the FiftyOne multimodal viewer (Image, 3D, Map, Plot, and Logs tiles on a shared timeline).

  • Filtering episodes by platform, split, sensor coverage, geographic bounding box, speed, idle fraction, or ground-truth availability using the FiftyOne SDK and App sidebar.

  • Prototyping multi-modal perception pipelines against a known-good subset before scaling to the full 382-sequence release.

  • Evaluating sensor coverage and calibration quality across platforms before importing additional sequences.

Out-of-Scope Use#

This FiftyOne package contains eight sequences (a curated subset). Training production models requires the full OctoSense release (anthonytec2/OctoSense). Radar range and bearing values use fitted scales (1/16 m per count, −1 bearing sign) derived from LiDAR cross-validation; they are not manufacturer-documented and should not be used as ground truth for radar calibration.

Dataset Structure#

Overview#

Media type: multimodal — one fo.Sample per episode, filepath pointing to a .mcap file. All sensor streams are encoded inside the MCAP with time-aligned messages on a shared nanosecond clock. The FiftyOne App renders them in synchronized Image, 3D, Map, Plot, and Logs tiles.

Sample count: 8 episodes (6 car, 1 boat, 1 Unitree quadruped)

Total recording time: ~46 minutes across all 8 episodes

Sample Fields#

Field

FiftyOne type

Description

filepath

StringField

Absolute path to the episode’s .mcap file

bag_id

StringField

ROS 2 bag identifier (e.g. rosbag2_2026_01_09-11_32_05)

session

StringField

Session label within the platform (e.g. sess8)

platform

StringField

Recording platform: car, boat, or unitree

start_time

StringField

Sequence wall-clock start time (ISO 8601)

split

StringField

train or test for car episodes; null for boat and unitree

is_daytime

BooleanField

Whether the sequence was recorded in daylight

degraded

BooleanField

Whether sensor degradation was logged for this sequence

has_seg

BooleanField

Whether pseudo-label semantic segmentation GT is available

duration_s

FloatField

Episode duration in seconds (from MCAP message timestamps)

message_count

IntField

Total MCAP messages across all channels

channel_count

IntField

Number of distinct MCAP channels

topics

ListField(StringField)

All MCAP topic strings in the episode

schemas

ListField(StringField)

All MCAP schema names (use for capability filtering)

n_lidar_frames

IntField

Number of LiDAR sweeps

n_rgb_frames

IntField

Number of RGB camera frames

n_imu_samples

IntField

Number of IMU samples

n_gps_fixes

IntField

GPS fixes logged (includes zero-fix rows on unitree)

n_gps_valid

FloatField

GPS fixes with valid lock

n_events_left

IntField

Total events from left event camera

n_events_right

IntField

Total events from right event camera

gps_quality

StringField

GPS fix quality string (e.g. single_m, rtk)

gps_lat_min

FloatField

Southern edge of episode geographic bounding box (degrees)

gps_lat_max

FloatField

Northern edge of episode geographic bounding box (degrees)

gps_lon_min

FloatField

Western edge of episode geographic bounding box (degrees)

gps_lon_max

FloatField

Eastern edge of episode geographic bounding box (degrees)

mean_speed_mph

FloatField

Mean driving speed in mph

idle_fraction

FloatField

Fraction of recording time at near-zero speed

distance_m

FloatField

Total distance driven in metres

rgb_cal_id

StringField

Camera calibration file identifier

imu_cal_id

StringField

IMU calibration file identifier

lidar_cal_id

StringField

LiDAR calibration file identifier

sensor_dropout

StringField

Dropout description if a sensor outage was logged; null otherwise

MCAP Stream Contents#

Each .mcap file is a multi-channel recording. Topics and their schemas vary by platform:

Topic

Schema

Tile

Platforms

/camera/left/image_raw

foxglove.CompressedVideo (H.264, 10 fps)

Image

all

/camera/right/image_raw

foxglove.CompressedVideo (H.264, 10 fps)

Image

all

/camera/infrared/image_raw

foxglove.CompressedVideo (H.264, ~50 fps)

Image

all

/camera/left_rect/depth

foxglove.RawImage (32fc1 metres, 10 Hz)

Image

car

/camera/left_rect/semantic

foxglove.CompressedImage (PNG, 10 Hz)

Image

car daytime

/camera/left_rect/flow

foxglove.CompressedVideo (H.264, 10 fps)

Image

car

/lidar/points

foxglove.PointCloud (10 Hz)

3D

all

/gps

foxglove.LocationFix

Map

car, boat

/odom

foxglove.PoseInFrame

3D

car

/imu/vectornav

imu_vectornav (JSON)

Plot

all

/imu/ouster

imu_ouster (JSON)

Plot

all

/car/*

JSON scalars (speed, wheels, steer, CAN)

Plot

car

/robot/*

JSON scalars (low_state, sport, controller)

Plot

unitree

/captions

foxglove.Log

Logs

car

/tf_static

foxglove.FrameTransforms

—

all

Ground Truth Coverage#

Ground truth

Source

Coverage

Notes

Dense depth

LiDAR accumulation, 32fc1 metres

All 6 car episodes

~12 % of pixels valid; 3–102 m range

Semantic segmentation

EoMT-Cityscapes-DINOv2-L-1024 pseudo-labels

Daytime car only (all 5 daytime car episodes in this subset)

19 Cityscapes classes; 255 = ignore

Ego-motion optical flow

Derived at conversion from depth + poses

All 6 car episodes

Ego-motion only; moving objects are holes

Radar tracks

Mazda CAN track_{1..6}

Car

Range 1/16 m per count (fitted); bearing sign inverted

Filtering by Capability#

Use the schemas list field to filter episodes rather than topic names (topic substrings false-positive against non-decodable packet channels):

import fiftyone as fo
from fiftyone import ViewField as F

dataset = fo.load_dataset("octosense")

# Car episodes with semantic segmentation
car_with_seg = dataset.match(F("has_seg") == True)

# Episodes with GPS
has_gps = dataset.match(F("schemas").contains("foxglove.LocationFix"))

# Filter by platform
cars = dataset.match(F("platform") == "car")

Dataset Creation#

Curation Rationale#

The eight episodes were selected to provide maximum coverage of platforms, modalities, and conditions within a manageable subset:

  • Daytime car with all GT (car/sess8, 53 s): shortest car sequence with depth, semantic segmentation, and all CAN channels present.

  • Car sessions (sess8 long, sess9, sess10 ×2, sess11): 163–597 s each, from both train and test splits, spanning all four car sessions; selected for near-zero sensor dropout, valid GPS lock, and minimal idle time.

  • Boat (boat/sess1, 54 s): the only platform where the LiDAR ships raw range images rather than XYZ, and where the boat IMU extrinsic differs.

  • Unitree (unitree/sess1, 112 s): the only platform with /robot joint state and no GPS or CAN data.

Source Data#

Data Collection and Processing#

Data was collected from a single OctoSense rig mounted on each platform. All sensors are hardware PPS-locked for temporal alignment and post-processed with Kalman-smoothed affine synchronization. Source recordings are ROS 2 bags converted to HDF5 and H.265 video files; the raw release is hosted at anthonytec2/OctoSense on Hugging Face (~8.5 TB). This FiftyOne package re-encodes video streams from HEVC to H.264 (libx264, no B-frames, one access unit per MCAP message) for browser WebCodecs compatibility and packages all channels into MCAP files using the Foxglove schema registry. LiDAR points are stored at a 20-byte stride (x, y, z, intensity as float32; r, g, b, a as uint8), with colour sampled from the nearest left-camera frame.

Who are the source data producers?#

The OctoSense hardware platform and dataset were developed at the General, Robotics, Automation, Sensing and Perception (GRASP) Laboratory, University of Pennsylvania (Anthony Bisulco, Jeremy Wang, Kostas Daniilidis, Pratik Chaudhari) and Brown University (Randall Balestriero). Data was collected on Long Island and in Philadelphia, PA, USA.

Annotations#

Annotation process#

All ground truth in OctoSense is machine-generated:

  • Depth is accumulated from LiDAR sweeps projected onto the rectified left camera frame.

  • Semantic segmentation is produced by EoMT-Cityscapes-DINOv2-L-1024 applied to CLAHE-enhanced rectified left frames; it is a model’s output, not human annotation.

  • Optical flow is derived at MCAP-conversion time from depth and LiDAR-inertial poses (flow_gap = 2 frames, i.e. 0.2 s lookahead); it captures ego-motion only — pixels covering dynamic objects (vehicles, pedestrians) were excluded from the depth accumulation and appear as invalid.

  • Radar detection geometry (range, bearing) uses scales fitted against LiDAR-projected vehicle pixels in bird’s-eye view (1/16 m per DIST_OBJ count, negative ANG_OBJ sign). These are not manufacturer-documented values.

Who are the annotators?#

No human annotators. All labels are generated programmatically from sensor data and pre-trained models as described above.

Personal and Sensitive Information#

Recordings were made on public roads and a private waterway. No personally identifiable information was intentionally captured. Faces and licence plates visible in RGB frames are not blurred in the source data.

Citation#

BibTeX:

@article{bisulco2026octosense,
  title={OctoSense: Self-Supervised Learning for Multimodal Robot Perception},
  author={Bisulco, Anthony and Wang, Jeremy and Daniilidis, Kostas and Balestriero, Randall and Chaudhari, Pratik},
  journal={arXiv preprint arXiv:2606.27317},
  year={2026}
}

APA:

Bisulco, A., Wang, J., Daniilidis, K., Balestriero, R., & Chaudhari, P. (2026). OctoSense: Self-Supervised Learning for Multimodal Robot Perception. arXiv preprint arXiv:2606.27317.

More Information#

  • Full dataset (8.5 TB, 382 sequences): https://huggingface.co/datasets/anthonytec2/OctoSense

  • Devkit and conversion scripts: https://github.com/anthonytec2/OctoSense

  • Project page and supplementary videos: https://abisulco.com/octosense/

  • FiftyOne multimodal documentation: https://docs.voxel51.com/user_guide/app.html#multimodal-viewer

Dataset Card Authors#

Harpreet Sahota

Dataset Card Contact#

harpreetsahota