Run in Google Colab

View source on GitHub

Download notebook

# Step 1: Loading a Self Driving Car Dataset into FiftyOne ## Installation Before getting started,it is highly suggested that you create a venv or conda environment to manage dependencies. This is a difficult one to nail right so here is the steps to be able to run the getting started experience ## Exploring nuScenes in FiftyOne Due to the multi-sensor structure of self-driving car datasets, the dataset in FiftyOne will be a [Grouped Dataset](https://docs.voxel51.com/user_guide/groups.html?_gl=1*1uqzezz*_gcl_au*MTI3OTEyMjEwMy4xNzI2MTQ5NDk3) with some [Dynamic Group Views](https://docs.voxel51.com/user_guide/using_views.html?_gl=1*1s195kz*_gcl_au*MTI3OTEyMjEwMy4xNzI2MTQ5NDk3#view-groups) thrown in there as well. We will be using nuscenes for our examples. At a high level, we will group together our samples by their associated scene in nuScenes. At regular intervals of each keyframe or approximately every 0.5 seconds (2Hz), we incorporate data from every sensor type, including their respective detections. This amalgamation of data results in distinct groups, each representing the sensor perspective at a given keyframe. We do have each sensor input for every frame, but since only keyframes are annotated, we choose to only load those in. ## Ingesting nuScenes To get started with nuScenes in FiftyOne, first we need to set up our environment for nuScenes. It will require downloading the dataset or a snippet of it as well as downloading the nuScenes python sdk. Full steps on installing can be found [here](https://www.nuscenes.org/nuscenes?tutorial=nuscenes). We want two files, `v1.0-mini.tgz` (the mini version of the full v1.0 dataset) and `nuScenes-lidarseg-mini-v1.0.tar.bz2` The tgz files should be dropped into the data/ folder and unpacked with: Once your nuScenes is downloaded, we can kick things off. Let’s start by initializing both nuScenes as well as our FiftyOne dataset. We define our dataset as well as add a group to initialize the dataset to expect grouped data. ## Loading Sensor Data It is recommended that you make helper functions to load in each sensor modality. These functions should take in path or token that points to sensor data file and any associated labels or metadata. ### Camera Data Camera data is a bit more straightforward than the lidar data as there is no need to do any prep on the image or save it as a different format. We can go right ahead and grab the detections to add them. The only tricky part is these are no ordinary bounding boxes, they are 3D bounding boxes given in global coordinates! Luckily for us, nuScenes provides some easy ways to convert their bounding boxes to our pixel space relative to what camera the image came from. For camera data, we load all of our boxes for our sample, check to see which ones are in the frame of our camera data, and then add the cuboids to the sample. In order to add a cuboid or a 3D bounding box, we use [polylines](https://docs.voxel51.com/user_guide/using_datasets.html?_gl=1*1vonm1t*_gcl_au*MTI3OTEyMjEwMy4xNzI2MTQ5NDk3#cuboids) and the [from_cuboid()](https://docs.voxel51.com/api/fiftyone.core.labels.html?_gl=1*z39nk3*_gcl_au*MTI3OTEyMjEwMy4xNzI2MTQ5NDk3#fiftyone.core.labels.Polyline.from_cuboid) method. Let’s take a look at how it is done: ## Adding Samples to the Dataset Next we need to add our camera samples to make our first dataset! Let’s loop through all the scenes and add each camera angle, forming groups of 6 samples each.  ## Loading Point Cloud Data Loading a LIDAR sample from nuScenes is composed of two steps, generating the pointcloud and adding the detections. We must convert the binary point clouds to standard in order to ingest them. nuScenes also offers a LIDAR segmentation optional package that allows us to color each point cloud point a color corresponding to it’s class that we will be utilizing. We start with our lidar token, load in the color map and point cloud that corresponds to the token, and save them back to file with the new coloring and standard pcd point cloud file formatting. With our point cloud file now properly prepared for ingestion, we can move along to adding detections. To do so, we grab all the detections from the keyframe. We use nuScenes SDK’s built-in box methods to retrieve the location, rotation, and dimensions of the box. To match [FiftyOne’s 3D detection](https://docs.voxel51.com/user_guide/using_datasets.html#d-datasets) input, we take box.orientation.yaw_pitch_roll for rotation, box.wlh for width, length, and height, and box.center for its location. Note too that fo.Sample(filepath=filepath, group=group.element(sensor)) will automatically detect the pcd file and ingest the sample as a point cloud as well! After the method is run and detections are added, we have our LIDAR sample with detections! RADAR is an interesting case. Since we have already stored our 3D detections in the LIDAR sample and RADAR is laid on top of the LIDAR in the 3D visualizer, we don’t need to copy our detections for each point cloud. This simplifies loading RADAR to just: Let’s run it back again except this time we are taking cameras, LIDAR, and RADAR samples!  Awesome work! You now have a full-fledged self driving dataset in FiftyOne! In our next step, we will look at some advanced techniques for curation and how to enhance self driving datasets with [FiftyOne Brain](https://docs.voxel51.com/brain.html) and [FiftyOne Zoo](https://docs.voxel51.com/dataset_zoo/index.html).