Run in Google Colab

View source on GitHub

Download notebook

# Loading Depth Data In this tutorial, we will explore how to load and visualize depth estimation datasets in FiftyOne. We will work with two popular depth datasets that use different storage formats: DIODE (with NumPy arrays) and NYU Depth V2 (with image files). ## Installation Some packages are required to load and process the depth data: ## Representing Depth Data in FiftyOne FiftyOne’s [Heatmap](https://docs.voxel51.com/api/fiftyone.core.labels.html#fiftyone.core.labels.Heatmap) class is ideal for representing depth data: ```python fo.Heatmap( map=None, # 2D numpy array containing the data map_path=None, # OR path to the heatmap image on disk range=None # Optional [min, max] range for proper visualization ) ``` There are essentially two ways you can load depth data: 1. Parsing a 2D numpy array 2. Pointing to a depth map on disk The optional `range` parameter specifies the min/max values of the heatmap. By default: - Floating point arrays use [0, 1] - Integer arrays use [0, 255] - Image files use their native data type range This tutorial will show you how to accomplish loading depth data for both scenarios using two datasets: - DIODE dataset - NYU Depth Dataset V2 ## DIODE Dataset DIODE (Dense Indoor and Outdoor DEpth) is a dataset of high-resolution color images with accurate, dense, far-range depth measurements. The DIODE dataset was created by researchers from TTI-Chicago, University of Chicago, and Beihang University, and is released under the MIT license. It was last updated on March 31, 2020. It’s the first public dataset to include RGBD images of both indoor and outdoor scenes captured with a single sensor suite. - [Paper on arXiv](https://arxiv.org/abs/1908.00463) - [Project page](https://diode-dataset.org/) ### File Naming Conventions and Formats The dataset consists of RGB images, depth maps, and depth validity masks. Their formats are as follows: - RGB images (`*.png`): RGB images with a resolution of 1024 × 768. - Depth maps (`*_depth.npy`): Depth ground truth with the same resolution as the images. - Depth masks (`*_depth_mask.npy`): Binary depth validity masks where 1 indicates valid sensor returns and 0 otherwise. The relationship between depth maps and depth validity masks is quite important for working with depth data: - **Depth Maps** contain the actual distance measurements from the camera to surfaces in the scene. Each pixel value represents how far away that point is (usually in meters). However, depth sensors often have limitations. - **Depth Validity Masks** indicate which pixels in the depth map have reliable measurements: - A value of 1 means the depth value is valid and can be trusted - A value of 0 means the depth sensor couldn’t get a reliable reading at that pixel These invalid readings typically occur because: 1. Some surfaces are too reflective, transparent, or absorptive 2. Areas may be too far away or outside the sensor’s range 3. Occlusions where one object blocks the sensor’s view of another 4. Motion blur during capture Without the validity mask, you’d be treating unreliable depth values as real measurements, which would introduce significant errors in any algorithms or visualizations using the depth data. ### Downloading the DIODE Dataset We will download and extract the validation split of the DIODE dataset. This contains the RGB images, depth maps, and validity masks we’ll need: ### Downloading DIODE Metadata Next, download the metadata associated with this dataset and parse it to a Python dictionary: This is a helper function to iterate and parse the file paths: ### Loading DIODE Depth Data into FiftyOne This code converts the DIODE depth dataset into a FiftyOne dataset for visualization and exploration. #### Depth Maps as Heatmaps The function processes depth data in several important steps: 1. **Loading the raw depth**: The depth maps are loaded from NumPy files, containing metric distance values in meters. 2. **Applying the mask**: Not all pixels have valid depth measurements. The function applies the depth mask to zero out invalid measurements, ensuring we only visualize reliable data. 3. **Computing visualization range**: To create meaningful visualizations, the function calculates an appropriate min/max range based on the actual depth values present in each image. It uses the minimum value and the 99th percentile (capped at 300 meters) to avoid outliers skewing the visualization. This is informed by the [source code](https://github.com/diode-dataset/diode-devkit/blob/8b1765b7d801a5f5e2877c434ffe164e62ce8c90/diode.py#L60) for the DIODE Dev Kit. 4. **Creating the Heatmap**: The masked depth map is stored as a FiftyOne Heatmap with the calculated range, allowing for intuitive color-coded visualization when viewing the dataset. #### Depth Masks The depth mask indicates which depth measurements are valid: - A mask value of 1 means the depth measurement is valid and trustworthy - A mask value of 0 indicates an invalid measurement (typically due to reflective surfaces, sensor limitations, or occlusions) By storing both the masked depth map and the mask itself as separate fields, you can easily visualize which areas have valid depth readings and which don’t. This is particularly important for depth estimation tasks where you need to know which ground truth values you can rely on for training or evaluation. #### Dataset Structure The resulting FiftyOne dataset contains: - RGB images as the primary media - Depth maps as heatmaps with appropriate visualization ranges - Binary depth masks indicating valid measurements - Metadata fields including scene type, split, scene ID, and scan ID This structure makes it easy to filter, sort, and visualize the dataset based on different criteria, such as scene type or depth range. ## Loading NYU Depth V2 Dataset into FiftyOne The [NYU Depth V2 dataset](https://cs.nyu.edu/~fergus/datasets/nyu_depth_v2.html) is another popular depth estimation dataset with RGB images paired with depth maps. Unlike the DIODE dataset where we loaded depth maps from NumPy arrays, the NYU dataset stores depth maps as PNG images. When working with the NYU dataset, the main difference is how we access and load the depth information: ### Depth Maps as Image Files In the NYU dataset, depth maps are stored as PNG image files rather than NumPy arrays. These PNG files typically store depth values as 16-bit grayscale images to preserve precision. Unlike the DIODE example where we passed the depth array directly, we’ll now use the `map_path` parameter of the `Heatmap` class to reference the depth map files. When using `map_path`, FiftyOne will: 1. Load the depth map image file when needed 2. Handle the conversion from image to array internally 3. Apply the provided range for proper visualization ### Determining the Depth Range For PNG depth maps, you need to know how the depth values are encoded: - Some datasets store raw depth in millimeters or meters - Others normalize depth values to the 0-65535 range (for 16-bit PNGs) - The range may also be specified in the dataset documentation You’ll need to specify the appropriate range based on the dataset’s depth encoding to ensure proper visualization. In this example I will load with default values for `range`, which in this case will be `[0, 255]` since the map values are integers. ## Example Implementation Approach To create a FiftyOne dataset from your dataframe: 1. Iterate through each row in the dataframe 2. Create a sample with the RGB image path 3. Add the depth map as a Heatmap using the `map_path` parameter 4. Add any additional metadata (scene type, room, etc.) 5. Add the sample to your FiftyOne dataset This approach allows you to work with image-based depth maps just as effectively as with the array-based approach used for DIODE. Note, we will download a version of this dataset from [Kaggle](https://www.kaggle.com/datasets/sohaibanwaar1203/image-depth-estimation/data). Note: If the download fails, please rerun the dataset download cell. It’s important to ensure the dataset is fully and correctly downloaded in your environment. We’ll parse the training datset. First, load the file `nyu2_train.csv` into a dataframe. This contains paired RGB and depth paths: - `image_path`: Points to RGB images - `depth_path`: Points to depth maps as PNG files ### NYU Depth V2 Dataset in FiftyOne: This code creates a structured, browsable dataset in FiftyOne from the NYU Depth V2 dataset, which is a benchmark dataset for indoor depth estimation. The function takes a DataFrame containing paths to RGB images and their corresponding depth maps, and builds a FiftyOne dataset that allows for interactive visualization and analysis. ### 1. Dataset Organization The code creates a [persistent FiftyOne dataset](https://docs.voxel51.com/user_guide/using_datasets.html#dataset-persistence), meaning it will be saved to disk and can be reloaded in future sessions. It organizes the NYU Depth V2 data with meaningful metadata extracted from the file structure: - **Room Types**: Automatically extracted from directory names (e.g., “living_room”) - **Scene IDs**: Identifies specific room instances (e.g., “living_room_0038_out”) - **Frame Numbers**: Numeric identifiers for individual frames within a scene ### 2. Depth Map Handling The depth maps are integrated as FiftyOne Heatmap objects, which enables specialized visualization. The code uses the [map_path](https://docs.voxel51.com/api/fiftyone.core.labels.html#fiftyone.core.labels.Heatmap) parameter to reference the depth files directly. ### 3. Data Validation and Processing The code includes several validation steps: - Verifying required columns in the input DataFrame - Converting relative paths to absolute paths - Checking that files exist before processing - Extracting structured metadata from filenames and paths ### 4. Interactive Visualization Once created, this dataset can be explored in the FiftyOne App, where you can: - Browse through RGB-depth pairs - Filter by room type, scene, or frame number - Visualize depth maps with different colormaps - Sort and group samples based on metadata  You may have noticed that each of these datasets are sequences of frames, thus they can be parsed as videos. However, converting frame sequences to MP4 videos is inefficient because: 1. The conversion process is time-consuming 2. High-resolution videos consume excessive storage space 3. Machine learning tasks typically process individual frames anyway, making video conversion unnecessary Instead, you can use [group_by()](https://docs.voxel51.com/user_guide/using_views.html#sorting-and-grouping) to create a view that groups the data by scene, ordered by frame number/timestamp. When you load a [dynamic](https://docs.voxel51.com/user_guide/using_datasets.html#dataset-persistence) grouped view in the App, you’ll have the same experience as video datasets: • You can hover over tiles in the grid to animate scenes’ frame data • When you click on a tile, you’ll have familiar video player controls in the modal to navigate the scene