Run in Google Colab

View source on GitHub

Download notebook

# Step 3: Using SAM 2 **Segment Anything 2 (SAM 2)** is a powerful segmentation model released in July 2024 that pushes the boundaries of image and video segmentation. It brings new capabilities to computer vision applications, including the ability to generate precise masks and track objects across frames in videos using just simple prompts. In this notebook, you’ll learn how to: - Understand the key innovations in SAM 2 - Apply SAM 2 to image datasets using bounding boxes, keypoints, or no prompts at all - Leverage SAM 2’s video segmentation and mask tracking capabilities with a single-frame prompt ## What is SAM 2? SAM 2 is the next generation of the Segment Anything Model, originally introduced by Meta in 2023. While SAM was designed for zero-shot segmentation on still images, SAM 2 adds robust video segmentation and tracking capabilities. With just a bounding box or a set of keypoints on a single frame, SAM 2 can segment and track objects across entire video sequences. ## Using SAM 2 for Images SAM 2 integrates directly with the FiftyOne Model Zoo, allowing you to apply segmentation to image datasets with minimal code. Whether you’re working with ground truth bounding boxes, keypoints, or want to explore automatic mask generation, FiftyOne makes the process seamless.  ## Using a custom segmentation dataset We will use a segmenation dataset with coffee beans, this is a FiftyOne Dataset. `pjramg/my_colombian_coffe_FO` ### Prompting with ground truth information in the 100 unique samples in the dataset  ### Apply SAM2 just the 100 unique samples SAM 2 can also segment entire images without needing any bounding boxes or keypoints. This zero-input mode is useful for generating segmentation masks for general visual analysis or bootstrapping annotation workflows.  In case you run out of memory, you can free up GPU space by clearing the cache with: ## Bonus with SAM2 ### Prompting with Keypoints Keypoint prompts are a great alternative to bounding boxes when working with articulated objects like people. Here, we filter images to include only people, generate keypoints using a keypoint model, and then use those keypoints to prompt SAM 2 for segmentation.  ## Using SAM 2 for Video SAM 2 brings game-changing capabilities to video understanding. It can track segmentations across frames from a single bounding box or keypoint prompt provided on the first frame. With this, you can propagate high-quality segmentation masks through entire sequences automatically.  ## Available SAM 2 Models in FiftyOne **Image Models:** - `segment-anything-2-hiera-tiny-image-torch` - `segment-anything-2-hiera-small-image-torch` - `segment-anything-2-hiera-base-plus-image-torch` - `segment-anything-2-hiera-large-image-torch` **Video Models:** - `segment-anything-2-hiera-tiny-video-torch` - `segment-anything-2-hiera-small-video-torch` - `segment-anything-2-hiera-base-plus-video-torch` - `segment-anything-2-hiera-large-video-torch`