Run in Google Colab

View source on GitHub

Download notebook

# Step 2: Adding Instance Segmentation to a FiftyOne Dataset We will explore how to enrich your dataset by adding **instance segmentation predictions**. In this notebook, we’ll cover: - Using the FiftyOne Model Zoo to apply instance segmentation - Integrating predictions from a custom model (e.g., a model deployed via Intel Geti) --- ## Using a Instance Segmentation Dataset For education purposes, use this link in Drive for downloading an upgraded dataset with 100+ annotated unique images. Download the dataset with this [Link](https://cdn.voxel51.com/dataset/colombian_coffee-dataset_1600.zip) and unzip in your work folder. Let’s kick things off by loading the **colombian_coffee-dataset_1600**: (This is a new dataset, different from the one used in the last notebook.) --- ## Loading predictions using SAM2 With FiftyOne, you have tons of pretrained models at your disposal to use via the [FiftyOne Model Zoo](https://docs.voxel51.com/model_zoo/index.html) or using one of our [integrations](https://docs.voxel51.com/integrations/index.html) such as [HuggingFace](https://docs.voxel51.com/integrations/huggingface.html)! To get started using them, first load the model in and pass it into the apply_model function. Install SAM2 following the instuctions from this [Repo](https://github.com/facebookresearch/sam2). You can also jump to the next step of this tutorials to understand how SAM2 works with FiftyOne If you encounter any issues, please refer to the main SAM2 repository to verify the installation process [Repo](https://github.com/facebookresearch/sam2). Now apply Segment Anything [SAM2](https://voxel51.com/blog/sam-2-is-now-available-in-fiftyone/) from the FiftyOne Model Zoo. As you can see, some images in the dataset include ground truth annotations, but not all of them. With SAM2, we will apply segmentation across the entire dataset. (This could take around 1.5 hours) Alternatively, you can apply SAM only to the images that already have ground truth segmentations.  This will execute SAM only for images in the segmentation category. ## Loading predictions using a custom model (Intel Geti Example) Let’s now simulate the pipeline with a custom instance segmentation model. If you want to run the inference using the same example, please refer tho this example for your reference. Assuming you’ve already set up inference with a model (e.g., via OpenVINO + Intel Geti SDK): ### Preparing the models for inference The Intel Geti SDK will be used to run inference with Intel Geti Models. The deployment folder of the best model must be downloaded and unzipped in the same folder as the project. Download and unzip the [model](https://cdn.voxel51.com/model/geti_sdk-deployment_90.zip) ### Generating instance segmentation masks from polygons and bounding boxes This function extracts instance segmentation masks from polygon annotations, combining **detection (bounding boxes)** and **segmentation (masks)** in the same instance using `fo.Detection`. 1. **Load Image** – Reads and converts the image to RGB. 2. **Process Annotations** – Extracts polygon points, computes bounding boxes, and normalizes coordinates. 3. **Generate Masks** – Creates, crops, and resizes binary masks for each annotation. 4. **Save & Return** – Stores masks as temp files and returns `fo.Detection` objects, ensuring the bounding box and mask belong to the same instance. This enables accurate visualization and analysis in FiftyOne, preserving both object localization and shape details. Useful for visualizing or processing segmentation data in FiftyOne. For education purposes check what is happening in the first or last sample. Then you can apply this to the whole dataset 📝 Tip: Replace `prediction.objects` with your real output structure and masks. ### Run the prediction in the whole dataset This loop processes each sample in the dataset by loading the image, running inference using Geti SDK, and generating instance segmentation masks. The function extracts detections with both bounding boxes and masks, ensuring they belong to the same instance. These predictions are then stored in the sample under `"predictions_model"` using `fo.Detections`. Finally, the dataset is reloaded to reflect the updates. ## Compare Predictions in FiftyOne App Toggle between `ground_truth_segmentations`, `sam2_predictions`, and `predictions_geti_sdk` in the App to explore and compare different segmentations side-by-side!