zero-shot-classification-transformer-torch

Finds any object you name in images without requiring training on those specific items.

Details

• Model name: zero-shot-classification-transformer-torch

• Model source: https://huggingface.co/docs/transformers/tasks/zero_shot_image_classification

• Model author: Thomas Wolf, et al.

• Model license: Apache 2.0

• Exposes embeddings? yes

• Tags: classification, logits, embeddings, torch, transformers, zero-shot, official

Requirements

• Packages: torch, torchvision, transformers

• CPU support

  • yes

• GPU support

  • yes

Example usage

import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset(
    "coco-2017",
    split="validation",
    dataset_name=fo.get_default_dataset_name(),
    max_samples=50,
    shuffle=True,
)

classes = ["person", "dog", "cat", "bird", "car", "tree", "chair"]

model = foz.load_zoo_model(
    "zero-shot-classification-transformer-torch",
    classes=classes,
)

dataset.apply_model(model, label_field="predictions")

session = fo.launch_app(dataset)

# some models make require additional arguments
# check the Hugging Face docs to see if any are needed

# for example, AltCLIP requires `padding=True` in its processor
model = foz.load_zoo_model(
    "zero-shot-classification-transformer-torch",
    classes=classes,
    name_or_path="BAAI/AltCLIP",
    transformers_processor_kwargs={
        "padding": True,
    }
)

dataset.apply_model(model, label_field="predictions")

session = fo.launch_app(dataset)