PyTorch Hub Integration#
FiftyOne integrates natively with PyTorch Hub, so you can load any Hub model and run inference on your FiftyOne datasets with just a few lines of code!
Loading a model#
Image models#
You can use the builtin
load_torch_hub_image_model()
utility to load models from the PyTorch Hub:
1import fiftyone.utils.torch as fout
2
3model = fout.load_torch_hub_image_model(
4 "pytorch/vision",
5 "resnet18",
6 hub_kwargs=dict(weights="ResNet18_Weights.DEFAULT"),
7)
The function returns a
TorchImageModel
instance that
wraps the raw Torch model in FiftyOne’s
Model interface, which means that you can
directly pass the model to builtin methods like
apply_model()
,
compute_embeddings()
,
compute_patch_embeddings()
,
compute_visualization()
, and
compute_similarity()
.
1import fiftyone.zoo as foz
2
3dataset = foz.load_zoo_dataset("quickstart")
4
5dataset.limit(10).apply_model(model, label_field="resnet18")
6
7# Logits
8print(dataset.first().resnet18.shape) # (1000,)
Note
In the above example, the resnet18
field is populated with raw logits.
Refer to this page to see how to configure
output processors to automatically parse model outputs into FiftyOne
label types.
Utilities#
FiftyOne also provides lower-level utilities for direct access to information about PyTorch Hub models:
1import fiftyone.utils.torch as fout
2
3# Load a raw Hub model
4model = fout.load_torch_hub_raw_model(
5 "facebookresearch/dinov2",
6 "dinov2_vits14",
7)
8print(type(model))
9# <class 'dinov2.models.vision_transformer.DinoVisionTransformer'>
10
11# Locate the `requirements.txt` for the model on disk
12req_path = fout.find_torch_hub_requirements("facebookresearch/dinov2")
13print(req_path)
14# '~/.cache/torch/hub/facebookresearch_dinov2_main/requirements.txt'
15
16# Load the package requirements for the model
17requirements = fout.load_torch_hub_requirements("facebookresearch/dinov2")
18print(requirements)
19# ['torch==2.0.0', 'torchvision==0.15.0', ...]
Example: YOLOv5#
Here’s how to load Ultralytics YOLOv5 and use it to generate object detections:
1from PIL import Image
2import numpy as np
3
4import fiftyone as fo
5import fiftyone.zoo as foz
6import fiftyone.utils.torch as fout
7
8class YOLOv5OutputProcessor(fout.OutputProcessor):
9 """Transforms ``ultralytics/yolov5`` outputs to FiftyOne format."""
10
11 def __call__(self, result, frame_size, confidence_thresh=None):
12 batch = []
13 for df in result.pandas().xywhn:
14 if confidence_thresh is not None:
15 df = df[df["confidence"] >= confidence_thresh]
16
17 batch.append(self._to_detections(df))
18
19 return batch
20
21 def _to_detections(self, df):
22 return fo.Detections(
23 detections=[
24 fo.Detection(
25 label=row.name,
26 bounding_box=[
27 row.xcenter - 0.5 * row.width,
28 row.ycenter - 0.5 * row.height,
29 row.width,
30 row.height,
31 ],
32 confidence=row.confidence,
33 )
34 for row in df.itertuples()
35 ]
36 )
37
38dataset = foz.load_zoo_dataset("quickstart")
39
40model = fout.load_torch_hub_image_model(
41 "ultralytics/yolov5",
42 "yolov5s",
43 hub_kwargs=dict(pretrained=True),
44 output_processor=YOLOv5OutputProcessor(),
45 raw_inputs=True,
46)
47
48# Generate predictions for a single image
49img = np.asarray(Image.open(dataset.first().filepath))
50predictions = model.predict(img)
51print(predictions) # <Detections: {...}>
52
53# Generate predictions for all images in a collection
54dataset.limit(10).apply_model(model, label_field="yolov5")
55dataset.count("yolov5.detections") # 26
Note
Did you know? Ultralytics YOLOv5 is natively available in the FiftyOne Model Zoo. You should also check out the Ultralytics integration!
Example: DINOv2#
Here’s how to load DINOv2 and use it to compute embeddings:
1from PIL import Image
2import numpy as np
3
4import fiftyone as fo
5import fiftyone.zoo as foz
6import fiftyone.utils.torch as fout
7
8dataset = foz.load_zoo_dataset("quickstart")
9
10model = fout.load_torch_hub_image_model(
11 "facebookresearch/dinov2",
12 "dinov2_vits14",
13 image_patch_size=14,
14 embeddings_layer="head",
15)
16assert model.has_embeddings
17
18# Embed a single image
19img = np.asarray(Image.open(dataset.first().filepath))
20embedding = model.embed(img)
21print(embedding.shape) # (384,)
22
23# Embed all images in a collection
24embeddings = dataset.limit(10).compute_embeddings(model)
25print(embeddings.shape) # (10, 384)
Note
Did you know? DINOv2 is natively available in the FiftyOne Model Zoo!
Adding Hub models to your local zoo#
You can add PyTorch Hub models to your local model zoo
and then load and use them via the fiftyone.zoo
package and the CLI
using the same syntax that you would with the
publicly available models:
1import fiftyone as fo
2import fiftyone.zoo as foz
3
4dataset = fo.load_dataset("...")
5model = foz.load_zoo_model("your-custom-model")
6
7dataset.apply_model(model, ...)
8dataset.compute_embeddings(model, ...)
Example: DINOv2#
Here’s how to add DINOv2 to your local model zoo and then load it to compute embeddings.
Create a custom manifest file and add DINOv2 to it:
{
"models": [
{
"base_name": "dinov2-vits14",
"description": "DINOv2: Learning Robust Visual Features without Supervision. Model: ViT-S/14 distilled",
"source": "https://github.com/facebookresearch/dinov2",
"default_deployment_config_dict": {
"type": "fiftyone.utils.torch.TorchImageModel",
"config": {
"entrypoint_fcn": "fiftyone.utils.torch.load_torch_hub_raw_model",
"entrypoint_args": {
"repo_or_dir": "facebookresearch/dinov2",
"model": "dinov2_vits14"
},
"image_patch_size": 14,
"embeddings_layer": "head"
}
}
}
]
}
Expose your manifest to FiftyOne by setting this environment variable:
export FIFTYONE_MODEL_ZOO_MANIFEST_PATHS=/path/to/custom-manifest.json
Now you can load and use the model using
load_zoo_model()
:
1import numpy as np
2from PIL import Image
3
4import fiftyone as fo
5import fiftyone.zoo as foz
6
7dataset = foz.load_zoo_dataset("quickstart")
8
9model = foz.load_zoo_model("dinov2-vits14")
10assert model.has_embeddings
11
12# Embed a single image
13img = np.asarray(Image.open(dataset.first().filepath))
14embedding = model.embed(img)
15print(embedding.shape) # (384,)
16
17# Embed all images in a collection
18embeddings = dataset.limit(10).compute_embeddings(model)
19print(embeddings.shape) # (10, 384)