# Model Zoo API Reference You can interact with the Model Zoo either via the Python library or the CLI. Python CLI The Model Zoo is accessible via the [`fiftyone.zoo`](../api/fiftyone.zoo.md#module-fiftyone.zoo) package. The [fiftyone zoo models](../cli/index.md#cli-fiftyone-zoo-models) command provides convenient utilities for working with models in the FiftyOne Model Zoo. ## Listing zoo models Python CLI You can list the available zoo models via [`list_zoo_models()`](../api/fiftyone.zoo.models.md#fiftyone.zoo.models.list_zoo_models): ```python import fiftyone.zoo as foz available_models = foz.list_zoo_models() print(available_models) ``` ```text ['alexnet-imagenet-torch', 'deeplabv3-cityscapes-tf', 'deeplabv3-mnv2-cityscapes-tf', ... 'wide-resnet50-2-imagenet-torch', 'yolo-v2-coco-tf1' ] ``` To view the zoo models that you have downloaded, you can use [`list_downloaded_zoo_models()`](../api/fiftyone.zoo.models.md#fiftyone.zoo.models.list_downloaded_zoo_models): ```python import fiftyone as fo import fiftyone.zoo as foz downloaded_models = foz.list_downloaded_zoo_models() fo.pprint(downloaded_models) ``` ```text { 'alexnet-imagenet-torch': ( '/Users/Brian/fiftyone/__models__/alexnet-owt-4df8aa71.pth', , ), 'densenet121-imagenet-torch': ( '/Users/Brian/fiftyone/__models__/densenet121-a639ec97.pth', , ), ... } ``` You can access information about the available zoo models via the [fiftyone zoo models list](../cli/index.md#cli-fiftyone-zoo-models-list) command. For example, to list the available zoo models and whether you have downloaded them, you can execute: ```shell fiftyone zoo models list ``` Models that have been downloaded are indicated by a checkmark in the `downloaded` column, and their location on disk is indicated by the `model_path` column. ## Getting information about zoo models Python CLI Each zoo model is represented by a [`ZooModel`](../api/fiftyone.zoo.models.md#fiftyone.zoo.models.ZooModel) subclass, which contains information about the model, its package requirements and CPU/GPU support, and more. You can access this object for a given model via the [`get_zoo_model()`](../api/fiftyone.zoo.models.md#fiftyone.zoo.models.get_zoo_model) method. For example, let’s print some information about a Faster R-CNN PyTorch model: ```python import fiftyone.zoo as foz zoo_model = foz.get_zoo_model("faster-rcnn-resnet50-fpn-coco-torch") print("***** Model description *****") print(zoo_model.description) print("\n***** License *****") print(zoo_model.license) print("\n***** Tags *****") print(zoo_model.tags) print("\n***** Requirements *****") print(zoo_model.requirements) ``` ```text ***** Model description ***** Faster R-CNN model with ResNet-50 FPN backbone trained on COCO. Source: https://pytorch.org/docs/stable/torchvision/models.html ***** License ***** BSD 3-Clause ***** Tags ***** ['detection', 'coco', 'torch'] ***** Requirements ***** { "packages": [ "torch", "torchvision" ], "cpu": { "support": true }, "gpu": { "support": true } } ``` When a zoo model is downloaded, you can use [`find_zoo_model()`](../api/fiftyone.zoo.models.md#fiftyone.zoo.models.find_zoo_model) to locate the downloaded model on disk: For example, let’s get the path on disk to the Faster R-CNN model referenced above (assuming it is downloaded): ```python import fiftyone.zoo as foz model_path = foz.find_zoo_model("faster-rcnn-resnet50-fpn-coco-torch") ``` You can view detailed information about a model (either downloaded or not) via the [fiftyone zoo models info](../cli/index.md#cli-fiftyone-zoo-models-info) command. For example, you can view information about a Faster R-CNN PyTorch model: ```shell fiftyone zoo models info faster-rcnn-resnet50-fpn-coco-torch ``` ```text ***** Model description ***** { "base_name": "faster-rcnn-resnet50-fpn-coco-torch", "base_filename": "fasterrcnn_resnet50_fpn_coco-258fb6c6.pth", "author": "Shaoqing Ren, et al.", "version": null, "url": null, "source": "https://pytorch.org/vision/main/models.html", "license": "BSD 3-Clause", "description": "Faster R-CNN model from `Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks `_ with ResNet-50 FPN backbone trained on COCO", "size_bytes": 167502836, "manager": { "type": "fiftyone.core.models.ModelManager", "config": { "url": "https://download.pytorch.org/models/fasterrcnn_resnet50_fpn_coco-258fb6c6.pth" } }, "default_deployment_config_dict": { "type": "fiftyone.zoo.models.torch.TorchvisionImageModel", "config": { "entrypoint_fcn": "torchvision.models.detection.faster_rcnn.fasterrcnn_resnet50_fpn", "entrypoint_args": { "weights": "FasterRCNN_ResNet50_FPN_Weights.DEFAULT" }, "output_processor_cls": "fiftyone.utils.torch.DetectorOutputProcessor", "labels_path": "{{eta-resources}}/ms-coco-labels.txt", "confidence_thresh": 0.3 } }, "requirements": { "packages": [ "torch", "torchvision" ], "cpu": { "support": true }, "gpu": { "support": true } }, "tags": [ "detection", "coco", "torch", "faster-rcnn", "resnet" ], "date_added": "2020-12-11T13:45:51" } ***** Model location ***** /Users/Brian/fiftyone/__models__/fasterrcnn_resnet50_fpn_coco-258fb6c6.pth ``` ## Downloading zoo models Python CLI You can download zoo models from the web via [`download_zoo_model()`](../api/fiftyone.zoo.models.md#fiftyone.zoo.models.download_zoo_model). For example, let’s download a Faster R-CNN PyTorch model: ```python import fiftyone.zoo as foz model_path = foz.download_zoo_model("faster-rcnn-resnet50-fpn-coco-torch") ``` ```text Downloading model from 'https://download.pytorch.org/models/fasterrcnn_resnet50_fpn_coco-258fb6c6.pth'... 100% |██████████████████████████████████| 1.2Gb/1.2Gb [4.7s elapsed, 0s remaining, 294.7Mb/s] ``` You can download zoo models from the web via the [fiftyone zoo models download](../cli/index.md#cli-fiftyone-zoo-models-download) command. For example, you can download a Faster R-CNN PyTorch model as follows: ```shell fiftyone zoo models download faster-rcnn-resnet50-fpn-coco-torch ``` ```text Downloading model from 'https://download.pytorch.org/models/fasterrcnn_resnet50_fpn_coco-258fb6c6.pth'... 100% |██████████████████████████████████| 1.2Gb/1.2Gb [4.7s elapsed, 0s remaining, 294.7Mb/s] ``` ## Installing zoo model requirements Python CLI Some models in the FiftyOne Model Zoo may require packages that are not installed by default when FiftyOne is installed. You can check to see if your current environment satisfies the requirements for a particular zoo model via [`ensure_zoo_model_requirements()`](../api/fiftyone.zoo.models.md#fiftyone.zoo.models.ensure_zoo_model_requirements): ```python import fiftyone.zoo as foz # Raises an error if the requirements are not satisfied foz.ensure_zoo_model_requirements("faster-rcnn-resnet50-fpn-coco-torch") ``` You can also use [`install_zoo_model_requirements()`](../api/fiftyone.zoo.models.md#fiftyone.zoo.models.install_zoo_model_requirements) to install any necessary packages for a particular model: ```python import fiftyone.zoo as foz foz.install_zoo_model_requirements("faster-rcnn-resnet50-fpn-coco-torch") ``` Some models in the FiftyOne Model Zoo may require packages that are not installed by default when FiftyOne is installed. You can view the requirements for a zoo model via the [fiftyone zoo models requirements](../cli/index.md#cli-fiftyone-zoo-models-requirements) command: ```shell fiftyone zoo models requirements faster-rcnn-resnet50-fpn-coco-torch ``` ```text ***** Model requirements ***** { "packages": [ "torch", "torchvision" ], "cpu": { "support": true }, "gpu": { "support": true } } ***** Current machine ***** GPU: no ``` You can use the `--ensure` flag to check to see if your current environment satisfies the requirements for a particular zoo model: ```shell # Raises an error if the requirements are not satisfied fiftyone zoo models requirements --ensure faster-rcnn-resnet50-fpn-coco-torch ``` You can also use the `--install` flag to install any necessary packages for a particular zoo model: ```shell fiftyone zoo models requirements --install faster-rcnn-resnet50-fpn-coco-torch ``` ## Loading zoo models You can load a zoo model via [`load_zoo_model()`](../api/fiftyone.zoo.models.md#fiftyone.zoo.models.load_zoo_model). By default, the model will be automatically downloaded from the web the first time you access it if it is not already downloaded: ```python import fiftyone.zoo as foz # The model will be downloaded from the web the first time you access it model = foz.load_zoo_model("faster-rcnn-resnet50-fpn-coco-torch") ``` You can also provide additional arguments to [`load_zoo_model()`](../api/fiftyone.zoo.models.md#fiftyone.zoo.models.load_zoo_model) to customize the import behavior: ```python # Load the zoo model and install any necessary requirements in order to # use it (logging warnings if any issues arise) model = foz.load_zoo_model( "faster-rcnn-resnet50-fpn-coco-torch", install_requirements=True, error_level=1, ) ``` #### NOTE By default, FiftyOne will attempt to ensure that any requirements such as Python packages or CUDA versions are satisfied before loading the model, and an error will be raised if a requirement is not satisfied. You can customize this behavior via the `error_level` argument to [`load_zoo_model()`](../api/fiftyone.zoo.models.md#fiftyone.zoo.models.load_zoo_model), or you can permanently adjust this behavior by setting the `requirement_error_level` parameter of your [FiftyOne config](../user_guide/config.md#configuring-fiftyone). An `error_level` of `0` will raise an error if a requirement is not satisfied, `1` will log a warning if the requirement is not satisfied, and `2` will ignore unsatisfied requirements. If you are using a `conda` environment, it is recommended you use an `error_level` of `1` or `2`, since FiftyOne uses `pip` to check for requirements. ## Applying zoo models Python CLI You can run inference on a dataset (or a subset of it specified by a [`DatasetView`](../api/fiftyone.core.view.md#fiftyone.core.view.DatasetView)) with a zoo model by loading it and then calling [`apply_model()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.apply_model): For example, the snippet below loads the `faster-rcnn-resnet50-fpn-coco-torch` model from the Model Zoo and applies it to 10 random images from the `quickstart` dataset from the Dataset Zoo: ```python import fiftyone.zoo as foz # Load zoo model model = foz.load_zoo_model("faster-rcnn-resnet50-fpn-coco-torch") # Load zoo dataset dataset = foz.load_zoo_dataset("quickstart") samples = dataset.take(10) # Run inference samples.apply_model(model, label_field="faster_rcnn") ``` You can run inference on a dataset with a zoo model via the [fiftyone zoo models apply](../cli/index.md#cli-fiftyone-zoo-models-apply) command. For example, the snippet below loads the `quickstart` dataset from the Dataset Zoo and applies the `faster-rcnn-resnet50-fpn-coco-torch` model from the Model Zoo to it: ```shell # Load zoo dataset fiftyone zoo datasets load quickstart # Apply zoo model fiftyone zoo models apply \ faster-rcnn-resnet50-fpn-coco-torch \ # model quickstart \ # dataset faster_rcnn # label field ``` ## Generating embeddings with zoo models Python CLI Many models in the Model Zoo expose embeddings for their predictions. You can determine if a model supports embeddings by loading it and checking the [`Model.has_embeddings`](../api/fiftyone.core.models.md#fiftyone.core.models.Model.has_embeddings) attribute: ```python import fiftyone.zoo as foz # Load zoo model model = foz.load_zoo_model("inception-v3-imagenet-torch") # Check if model exposes embeddings model.has_embeddings # True ``` For models that expose embeddings, you can generate embeddings for all samples in a dataset (or a subset of it specified by a [`DatasetView`](../api/fiftyone.core.view.md#fiftyone.core.view.DatasetView)) by calling [`compute_embeddings()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.compute_embeddings): ```python import fiftyone.zoo as foz # Load zoo model model = foz.load_zoo_model("inception-v3-imagenet-torch") model.has_embeddings # True # Load zoo dataset dataset = foz.load_zoo_dataset("quickstart") samples = dataset.take(10) # Generate embeddings for each sample and return them in a # `num_samples x dim` array embeddings = samples.compute_embeddings(model) # Generate embeddings for each sample and store them in a sample field samples.compute_embeddings(model, embeddings_field="embeddings") ``` You can also use [`compute_patch_embeddings()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.compute_patch_embeddings) to generate embeddings for image patches defined by another label field, e.g,. the detections generated by a detection model. For models that expose embeddings, you can generate embeddings for all samples in a dataset via the [fiftyone zoo models embed](../cli/index.md#cli-fiftyone-zoo-models-embed) command. For example, the snippet below loads the `quickstart` dataset from the Dataset Zoo and generates embeddings for each sample using the `inception-v3-imagenet-torch` model from the Model Zoo: ```shell # Load zoo dataset fiftyone zoo datasets load quickstart # Generate embeddings via zoo model fiftyone zoo models embed \ inception-v3-imagenet-torch \ # model quickstart \ # dataset embeddings # embeddings field ``` ## Controlling where zoo models are downloaded By default, zoo models are downloaded into subdirectories of `fiftyone.config.model_zoo_dir` corresponding to their names. You can customize this backend by modifying the `model_zoo_dir` setting of your [FiftyOne config](../user_guide/config.md#configuring-fiftyone). JSON Environment Code Directly edit your FiftyOne config at `~/.fiftyone/config.json`: ```shell # Print your current config fiftyone config # Locate your config (and edit the `model_zoo_dir` field) fiftyone constants FIFTYONE_CONFIG_PATH ``` Set the `FIFTYONE_MODEL_ZOO_DIR` environment variable: ```shell # Customize where zoo models are downloaded export FIFTYONE_MODEL_ZOO_DIR=/your/custom/directory ``` Set the `model_zoo_dir` config setting from Python code: ```python import fiftyone as fo # Customize where zoo models are downloaded fo.config.model_zoo_dir = "/your/custom/directory" ``` ## Deleting zoo models Python CLI You can delete the local copy of a zoo model via [`delete_zoo_model()`](../api/fiftyone.zoo.models.md#fiftyone.zoo.models.delete_zoo_model): ```python import fiftyone.zoo as foz foz.delete_zoo_model("faster-rcnn-resnet50-fpn-coco-torch") ``` You can delete the local copy of a zoo model via the [fiftyone zoo models delete](../cli/index.md#cli-fiftyone-zoo-models-delete) command: ```shell fiftyone zoo models delete faster-rcnn-resnet50-fpn-coco-torch ``` ## Adding models to the zoo We frequently add new models to the Model Zoo, which will automatically become accessible to you when you update your FiftyOne package. #### NOTE FiftyOne is open source! You are welcome to contribute models to the public model zoo by submitting a pull request to [the GitHub repository](https://github.com/voxel51/fiftyone). You can also add your own models to your local model zoo, enabling you to work with these models via the [`fiftyone.zoo`](../api/fiftyone.zoo.md#module-fiftyone.zoo) package and the CLI using the same syntax that you would with publicly available models. To add model(s) to your local zoo, you simply write a JSON manifest file in the format below to tell FiftyOne about the model(s). For example, the manifest below adds a second copy of the `yolo-v2-coco-tf1` model to the zoo under the alias `yolo-v2-coco-tf1-high-conf` that only returns predictions whose confidence is at least 0.5: ```json { "models": [ { "base_name": "yolo-v2-coco-tf1-high-conf", "base_filename": "yolo-v2-coco-high-conf.weights", "version": null, "description": "A YOLOv2 model with confidence threshold set to 0.5", "manager": { "type": "fiftyone.core.models.ModelManager", "config": { "google_drive_id": "1ajuPZws47SOw3xJc4Wvk1yuiB3qv8ycr" } }, "default_deployment_config_dict": { "type": "fiftyone.utils.eta.ETAModel", "config": { "type": "eta.detectors.YOLODetector", "config": { "config_dir": "{{eta}}/tensorflow/darkflow/cfg/", "config_path": "{{eta}}/tensorflow/darkflow/cfg/yolo.cfg", "confidence_thresh": 0.5 } } }, "requirements": { "cpu": { "support": true, "packages": ["tensorflow<2"] }, "gpu": { "support": true, "packages": ["tensorflow-gpu<2"] } }, "tags": ["detection", "coco", "tf1"], "date_added": "2020-12-11 13:45:51" } ] } ``` #### NOTE Adjusting the hard-coded threshold of the above model is possible via JSON-only changes in this case because the underlying [eta.detectors.YOLODetector](https://github.com/voxel51/eta/blob/main/eta/detectors/yolo.py) class exposes this as a parameter. In practice, there is no need to hard-code confidence thresholds in models, since the [`apply_model()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.apply_model) method supports supplying an optional confidence threshold that is applied post-facto to the predictions generated by any model. Models manifest JSON files should have a `models` key that contains a list of serialized [`ZooModel class definitions`](../api/fiftyone.zoo.models.md#fiftyone.zoo.models.ZooModel) that describe how to download and load the model. Finally, expose your new models(s) to FiftyOne by adding your manifest to the `model_zoo_manifest_paths` parameter of your [FiftyOne config](../user_guide/config.md#configuring-fiftyone). One way to do this is to set the `FIFTYONE_MODEL_ZOO_MANIFEST_PATHS` environment variable: ```shell export FIFTYONE_MODEL_ZOO_MANIFEST_PATHS=/path/to/custom/manifest.json ``` Now you can load and apply the `yolo-v2-coco-tf1-high-conf` model as you would any other zoo model: ```python import fiftyone as fo import fiftyone.zoo as foz # Load custom model model = foz.load_zoo_model("yolo-v2-coco-tf1-high-conf") # Apply model to a dataset dataset = fo.load_dataset(...) dataset.apply_model(model, label_field="predictions") ```