segment-anything-3-video-torch#

Open-vocabulary video segmentation that finds, segments, and tracks all objects matching a text concept across video frames.

Details

  • Model name: segment-anything-3-video-torch

  • Model source: facebookresearch/sam3

  • Model author: Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, et al.

  • Model license: SAM License

  • Model size: 3.45 GB

  • Exposes embeddings? no

  • Tags: segment-anything, torch, zero-shot, video, transformer, official

Requirements

  • Packages: torch, torchvision, sam3

  • CPU support

    • yes

  • GPU support

    • yes

Example usage

 1import fiftyone as fo
 2import fiftyone.zoo as foz
 3from fiftyone import ViewField as F
 4
 5dataset = foz.load_zoo_dataset("quickstart-video", max_samples=2)
 6
 7
 8# Concept mode: find and segment object with text prompts on selected frame, and track objects across frames
 9model = foz.load_zoo_model(
10    "segment-anything-3-video-torch",
11    classes=["person"],
12    operation_mode="concept",
13    propagation_direction="forward", # also supports backward and both
14    text_frame_idx=1,
15)
16dataset.apply_model(model, label_field="segmentations_concept")
17
18# Exemplar concept mode: text prompt + exemplar boxes on frame 1, propagate forward
19model = foz.load_zoo_model(
20    "segment-anything-3-video-torch",
21    classes=["person"],
22    operation_mode="concept",
23    propagation_direction="both",
24    prompt_frame_indices=[1],
25)
26dataset.apply_model(
27    model,
28    label_field="segmentations_concept_with_exemplar",
29    prompt_field="frames.person_detections",  # exemplar Detections on frame 1
30)
31
32# Visual mode: segment inside boxes and propagate to all frames
33model = foz.load_zoo_model("segment-anything-3-video-torch")
34dataset.apply_model(
35    model,
36    label_field="segmentations",
37    prompt_field="frames.detections",  # can contain Detections or Keypoints
38    prompt_frame_indices=[1],
39)
40
41session = fo.launch_app(dataset)