Model Zoo API Reference

You can interact with the Model Zoo either via the Python library or the CLI.

PythonCLI

The Model Zoo is accessible via the fiftyone.zoo package.

Listing zoo models

PythonCLI

You can list the available zoo models via list_zoo_models():

import fiftyone.zoo as foz

available_models = foz.list_zoo_models()

print(available_models)

['alexnet-imagenet-torch',
'deeplabv3-cityscapes-tf',
'deeplabv3-mnv2-cityscapes-tf',
...
'wide-resnet50-2-imagenet-torch',
'yolo-v2-coco-tf1'
]

To view the zoo models that you have downloaded, you can use list_downloaded_zoo_models():

import fiftyone as fo
import fiftyone.zoo as foz

downloaded_models = foz.list_downloaded_zoo_models()
fo.pprint(downloaded_models)

{
    'alexnet-imagenet-torch': (
        '/Users/Brian/fiftyone/__models__/alexnet-owt-4df8aa71.pth',
        <fiftyone.zoo.models.ZooModel object at 0x122d2fa58>,
    ),
    'densenet121-imagenet-torch': (
        '/Users/Brian/fiftyone/__models__/densenet121-a639ec97.pth',
        <fiftyone.zoo.models.ZooModel object at 0x122d608d0>,
    ),
    ...
}

Getting information about zoo models

PythonCLI

Each zoo model is represented by a ZooModel subclass, which contains information about the model, its package requirements and CPU/GPU support, and more. You can access this object for a given model via the get_zoo_model() method.

For example, let’s print some information about a Faster R-CNN PyTorch model:

import fiftyone.zoo as foz

zoo_model = foz.get_zoo_model("faster-rcnn-resnet50-fpn-coco-torch")

print("***** Model description *****")
print(zoo_model.description)

print("\n***** License *****")
print(zoo_model.license)

print("\n***** Tags *****")
print(zoo_model.tags)

print("\n***** Requirements *****")
print(zoo_model.requirements)

***** Model description *****
Faster R-CNN model with ResNet-50 FPN backbone trained on COCO. Source: https://pytorch.org/docs/stable/torchvision/models.html

***** License *****
BSD 3-Clause

***** Tags *****
['detection', 'coco', 'torch']

***** Requirements *****
{
    "packages": [
        "torch",
        "torchvision"
    ],
    "cpu": {
        "support": true
    },
    "gpu": {
        "support": true
    }
}

When a zoo model is downloaded, you can use find_zoo_model() to locate the downloaded model on disk:

For example, let’s get the path on disk to the Faster R-CNN model referenced above (assuming it is downloaded):

import fiftyone.zoo as foz

model_path = foz.find_zoo_model("faster-rcnn-resnet50-fpn-coco-torch")

Downloading zoo models

PythonCLI

You can download zoo models from the web via download_zoo_model().

For example, let’s download a Faster R-CNN PyTorch model:

import fiftyone.zoo as foz

model_path = foz.download_zoo_model("faster-rcnn-resnet50-fpn-coco-torch")

Downloading model from 'https://download.pytorch.org/models/fasterrcnn_resnet50_fpn_coco-258fb6c6.pth'...
 100% |██████████████████████████████████|    1.2Gb/1.2Gb [4.7s elapsed, 0s remaining, 294.7Mb/s]

Installing zoo model requirements

PythonCLI

Some models in the FiftyOne Model Zoo may require packages that are not installed by default when FiftyOne is installed.

You can check to see if your current environment satisfies the requirements for a particular zoo model via ensure_zoo_model_requirements():

import fiftyone.zoo as foz

# Raises an error if the requirements are not satisfied
foz.ensure_zoo_model_requirements("faster-rcnn-resnet50-fpn-coco-torch")

You can also use install_zoo_model_requirements() to install any necessary packages for a particular model:

import fiftyone.zoo as foz

foz.install_zoo_model_requirements("faster-rcnn-resnet50-fpn-coco-torch")

Loading zoo models

You can load a zoo model via load_zoo_model().

By default, the model will be automatically downloaded from the web the first time you access it if it is not already downloaded:

import fiftyone.zoo as foz

# The model will be downloaded from the web the first time you access it
model = foz.load_zoo_model("faster-rcnn-resnet50-fpn-coco-torch")

You can also provide additional arguments to load_zoo_model() to customize the import behavior:

# Load the zoo model and install any necessary requirements in order to
# use it (logging warnings if any issues arise)
model = foz.load_zoo_model(
    "faster-rcnn-resnet50-fpn-coco-torch",
    install_requirements=True,
    error_level=1,
)

Note

By default, FiftyOne will attempt to ensure that any requirements such as Python packages or CUDA versions are satisfied before loading the model, and an error will be raised if a requirement is not satisfied.

You can customize this behavior via the error_level argument to load_zoo_model(), or you can permanently adjust this behavior by setting the requirement_error_level parameter of your FiftyOne config.

An error_level of 0 will raise an error if a requirement is not satisfied, 1 will log a warning if the requirement is not satisfied, and 2 will ignore unsatisfied requirements.

If you are using a conda environment, it is recommended you use an error_level of 1 or 2, since FiftyOne uses pip to check for requirements.

Applying zoo models

PythonCLI

You can run inference on a dataset (or a subset of it specified by a DatasetView) with a zoo model by loading it and then calling apply_model():

For example, the snippet below loads the faster-rcnn-resnet50-fpn-coco-torch model from the Model Zoo and applies it to 10 random images from the quickstart dataset from the Dataset Zoo:

import fiftyone.zoo as foz

# Load zoo model
model = foz.load_zoo_model("faster-rcnn-resnet50-fpn-coco-torch")

# Load zoo dataset
dataset = foz.load_zoo_dataset("quickstart")
samples = dataset.take(10)

# Run inference
samples.apply_model(model, label_field="faster_rcnn")

Generating embeddings with zoo models

PythonCLI

Many models in the Model Zoo expose embeddings for their predictions. You can determine if a model supports embeddings by loading it and checking the Model.has_embeddings attribute:

import fiftyone.zoo as foz

# Load zoo model
model = foz.load_zoo_model("inception-v3-imagenet-torch")

# Check if model exposes embeddings
model.has_embeddings  # True

For models that expose embeddings, you can generate embeddings for all samples in a dataset (or a subset of it specified by a DatasetView) by calling compute_embeddings():

import fiftyone.zoo as foz

# Load zoo model
model = foz.load_zoo_model("inception-v3-imagenet-torch")
model.has_embeddings  # True

# Load zoo dataset
dataset = foz.load_zoo_dataset("quickstart")
samples = dataset.take(10)

# Generate embeddings for each sample and return them in a
# `num_samples x dim` array
embeddings = samples.compute_embeddings(model)

# Generate embeddings for each sample and store them in a sample field
samples.compute_embeddings(model, embeddings_field="embeddings")

You can also use compute_patch_embeddings() to generate embeddings for image patches defined by another label field, e.g,. the detections generated by a detection model.

Controlling where zoo models are downloaded

By default, zoo models are downloaded into subdirectories of fiftyone.config.model_zoo_dir corresponding to their names.

You can customize this backend by modifying the model_zoo_dir setting of your FiftyOne config.

JSONEnvironmentCode

Directly edit your FiftyOne config at ~/.fiftyone/config.json:

# Print your current config
fiftyone config

# Locate your config (and edit the `model_zoo_dir` field)
fiftyone constants FIFTYONE_CONFIG_PATH

Deleting zoo models

PythonCLI

You can delete the local copy of a zoo model via delete_zoo_model():

import fiftyone.zoo as foz

foz.delete_zoo_model("faster-rcnn-resnet50-fpn-coco-torch")

Adding models to the zoo

We frequently add new models to the Model Zoo, which will automatically become accessible to you when you update your FiftyOne package.

Note

FiftyOne is open source! You are welcome to contribute models to the public model zoo by submitting a pull request to the GitHub repository.

You can also add your own models to your local model zoo, enabling you to work with these models via the fiftyone.zoo package and the CLI using the same syntax that you would with publicly available models.

To add model(s) to your local zoo, you simply write a JSON manifest file in the format below to tell FiftyOne about the model(s). For example, the manifest below adds a second copy of the yolo-v2-coco-tf1 model to the zoo under the alias yolo-v2-coco-tf1-high-conf that only returns predictions whose confidence is at least 0.5:

{
    "models": [
        {
            "base_name": "yolo-v2-coco-tf1-high-conf",
            "base_filename": "yolo-v2-coco-high-conf.weights",
            "version": null,
            "description": "A YOLOv2 model with confidence threshold set to 0.5",
            "manager": {
                "type": "fiftyone.core.models.ModelManager",
                "config": {
                    "google_drive_id": "1ajuPZws47SOw3xJc4Wvk1yuiB3qv8ycr"
                }
            },
            "default_deployment_config_dict": {
                "type": "fiftyone.utils.eta.ETAModel",
                "config": {
                    "type": "eta.detectors.YOLODetector",
                    "config": {
                        "config_dir": "{{eta}}/tensorflow/darkflow/cfg/",
                        "config_path": "{{eta}}/tensorflow/darkflow/cfg/yolo.cfg",
                        "confidence_thresh": 0.5
                    }
                }
            },
            "requirements": {
                "cpu": {
                    "support": true,
                    "packages": ["tensorflow<2"]
                },
                "gpu": {
                    "support": true,
                    "packages": ["tensorflow-gpu<2"]
                }
            },
            "tags": ["detection", "coco", "tf1"],
            "date_added": "2020-12-11 13:45:51"
        }
    ]
}

Note

Adjusting the hard-coded threshold of the above model is possible via JSON-only changes in this case because the underlying eta.detectors.YOLODetector class exposes this as a parameter.

In practice, there is no need to hard-code confidence thresholds in models, since the apply_model() method supports supplying an optional confidence threshold that is applied post-facto to the predictions generated by any model.

Models manifest JSON files should have a models key that contains a list of serialized ZooModel class definitions that describe how to download and load the model.

Finally, expose your new models(s) to FiftyOne by adding your manifest to the model_zoo_manifest_paths parameter of your FiftyOne config. One way to do this is to set the FIFTYONE_MODEL_ZOO_MANIFEST_PATHS environment variable:

export FIFTYONE_MODEL_ZOO_MANIFEST_PATHS=/path/to/custom/manifest.json

Now you can load and apply the yolo-v2-coco-tf1-high-conf model as you would any other zoo model:

import fiftyone as fo
import fiftyone.zoo as foz

# Load custom model
model = foz.load_zoo_model("yolo-v2-coco-tf1-high-conf")

# Apply model to a dataset
dataset = fo.load_dataset(...)
dataset.apply_model(model, label_field="predictions")