<a href="https://github.com/voxel51/fiftyone-labs/tree/main/plugins/click_segmentation" target="_blank">![GitHub Repo](https://img.shields.io/badge/GitHub-Repository-black?logo=github)</a>

# Image segmentation via point prompts

Image segmentation via point prompts is a `fiftyone-labs` features that lets users interactively add keypoints to an image and use these keypoints for generating segmentation masks via a promptable segmentation model. There are two main components in this feaure:

1. The `click_segmentation` panel is an interactive tool for generating point prompts via clicks on the `Sample` modal. These point prompts can be input to a promptable segmentation model (such as Segment Anything Model) to create segmentation masks.
2. The `segment_with_prompts` operator is a standalone operator that can be used to run a promptable segmentation model with either box or keypoint prompts saved in a sample field. The operator may be applied to the entire dataset or a subset of the dataset.

## Usage

### Click To Segment Panel

```python
import fiftyone.zoo as foz
import fiftyone as fo

dataset = foz.load_zoo_dataset("quickstart")
# For a smoother experience, ensure that samples have metadata.
if dataset.count("metadata") != len(dataset):
    dataset.compute_metadata()

session = fo.launch_app(dataset)
```

*Select a sample and open Click To Segment panel*

<img src="https://raw.githubusercontent.com/voxel51/fiftyone-labs/main/plugins/click_segmentation/assets/open_panel.png" alt="Open panel" width="800">

*Enable capture mode by checking the box*

<img src="https://raw.githubusercontent.com/voxel51/fiftyone-labs/main/plugins/click_segmentation/assets/click_capture_enabled.png" alt="Click-To-Segment panel" width="800">

The panel has two modes of operation:

- [Multiple Click Capture](#multiple-click-capture): Users can save multiple sets of clicks and manually trigger segmentation.
- [Single Click Capture](#single-click-capture): Users can click on the sample modal and segmentation will auto-trigger.

#### Multiple Click Capture

*In the Sample modal, add a set of clicks for airplane and save as keypoints*

<img src="https://raw.githubusercontent.com/voxel51/fiftyone-labs/main/plugins/click_segmentation/assets/clicks_airplane.png" alt="Keypoints for airplane" width="800">

*Add another set of clicks for sky and save as keypoints*

<img src="https://raw.githubusercontent.com/voxel51/fiftyone-labs/main/plugins/click_segmentation/assets/clicks_sky.png" alt="Keypoints for sky" width="800">

*Choose a segmentation model from FiftyOne model zoo and click on Segment with Keypoints button*

<img src="https://raw.githubusercontent.com/voxel51/fiftyone-labs/main/plugins/click_segmentation/assets/seg_success_notification.png" alt="Segmentation successful" width="800">

*Segmentation masks will be added to the Sample*

<img src="https://raw.githubusercontent.com/voxel51/fiftyone-labs/main/plugins/click_segmentation/assets/output_multi_click_segmentation.png" alt="Output segmentations for multiple clicks" width="800">

#### Single Click Capture

*In the Click To Segment panel, enable single click segmentation by checking the box*

<img src="https://raw.githubusercontent.com/voxel51/fiftyone-labs/main/plugins/click_segmentation/assets/single_click_mode.png" alt="Single Click Segmentation" width="800">

*Add a click on the Sample modal to trigger auto-segmentation*

<img src="https://raw.githubusercontent.com/voxel51/fiftyone-labs/main/plugins/click_segmentation/assets/output_single_click_segmentation.png" alt="Keypoints for sky" width="800">

### Segment With Prompt Operator

*Launch the Segment With Prompts operator from the dropdown operators menu*

<img src="https://raw.githubusercontent.com/voxel51/fiftyone-labs/main/plugins/click_segmentation/assets/open_seg_with_prompts_op.png" alt="Segment-With-Prompts operator" width="800">

*Add inputs and trigger execution*

<img src="https://raw.githubusercontent.com/voxel51/fiftyone-labs/main/plugins/click_segmentation/assets/seg_with_prompts_op.png" alt="Segment-With-Prompts operator" width="800">

**Delegated Operation** To run `Segment With Prompts` operator with delegated operation, see the instructions [here](https://docs.voxel51.com/plugins/using_plugins.html#delegated-operations). You can also enable delegated operation via *Click To Segment* panel by checking the `Use delegated operation` checkbox next to *Segment With Keypoints* button.
