Run in Google Colab

View source on GitHub

Download notebook

# DINOv3 visual search ## 1. Install Required Libraries We start by installing [FiftyOne](https://docs.voxel51.com/) and the Hugging Face `transformers` library. This will allow us to load the [DINOv3 model from Hugging Face](https://huggingface.co/facebook/dinov3-vits16-pretrain-lvd1689m) and use FiftyOne’s dataset visualization and analysis features. **Note:** We install `transformers` directly from the development branch to ensure compatibility with the latest DINOv3 features. Since the DINOv3 functionality is **not yet available** in the stable `transformers` release, we install it from the development branch. See the [FiftyOne + Hugging Face integration guide](https://docs.voxel51.com/integrations/huggingface.html) for more details on using experimental model versions. ## 2. Log in to Hugging Face We authenticate with Hugging Face to retrieve the latest model weights. You must have access to the model you want to load. See [Hugging Face authentication docs](https://huggingface.co/docs/huggingface_hub/quick-start#login) for details. ### Load a quick start dataset To explore more dataset option visit the docs [Dataset Zoo](https://docs.voxel51.com/dataset_zoo/index.html) or load your [own dataset](https://docs.voxel51.com/api/fiftyone.core.odm.utils.html#fiftyone.core.odm.utils.load_dataset) You can load any of the model available in Hugging Face [https://huggingface.co/collections/facebook/dinov3-68924841bd6b561778e31009](https://huggingface.co/collections/facebook/dinov3-68924841bd6b561778e31009) Thanks to the integration of Hugging Face in Fiftyone we are able to perform multiple tasks with the model, explore more here [Integration HuggingFace](https://docs.voxel51.com/integrations/huggingface.html) ## Working with DINOv3 Embeddings in FiftyOne In this example, we focus on using **DINOv3 embeddings** for visual search and similarity-based exploration in FiftyOne. ### Workflow 1. **Compute embeddings**
We run each image through the DINOv3 model and extract either:
- The **class token embedding** (for global representation), or - The **patch token embeddings** (for more granular, region-level similarity). Learn more: [Computing embeddings in FiftyOne](https://docs.voxel51.com/api/fiftyone.core.models.html#fiftyone.core.models.compute_embeddings). 2. **Visualize embeddings**
We project the embeddings into 2D space using dimensionality reduction (e.g., t-SNE or UMAP) so we can see clusters of visually similar images.
- FiftyOne makes this interactive through its Embeddings Visualization in the App. 3. **Compute similarity search**
Using FiftyOne’s similarity search tools, we select a **query image** (in this example, the first image in the dataset, but you can choose any).
- The system finds and ranks the most visually similar images based on embedding distance. - Docs: [Similarity search in FiftyOne](https://docs.voxel51.com/api/fiftyone.brain.similarity.html). 4. **Sort by similarity**
We display the results sorted from most to least similar, making it easy to:
- Detect near-duplicates. - Explore visual clusters. - Identify outliers in the dataset.  ## Classification Tasks with DINOv3 In this example, we use the **DINOv3** model to perform an image classification task by integrating its embeddings with a **Logistic Regression (linear)** classifier. ### Workflow 1. **Extract embeddings**
We feed each image through the DINOv3 model and extract the **class token embedding**.
- The class token acts as a compact representation of the entire image. - More on embeddings: [FiftyOne embeddings guide](https://docs.voxel51.com/tutorials/image_embeddings.html). 2. **Train a linear classifier**
Using the extracted embeddings as input features and the ground truth labels from our dataset, we train a **Logistic Regression (linear)** classifier to predict image classes.
- Linear classifiers are effective for high-dimensional feature spaces like DINOv3 embeddings. 3. **Run inference**
We pass unseen images through the same pipeline to generate embeddings and predict their class labels using the trained SVM.
4. **Evaluate results in FiftyOne**
We visualize and analyze the model predictions in FiftyOne, using:
- [Classification evaluation](https://docs.voxel51.com/user_guide/evaluation.html#classification-evaluation) to compute metrics like accuracy, precision, and recall. - [Confusion matrix](https://docs.voxel51.com/user_guide/plots.html#confusion-matrices) to see where the model is making mistakes. Get id, path, embeddinds and classes to create a classifier Evaluate the results of the classification  ## PCA/CLS Foreground Segmentation with DINOv3 This step builds a **foreground mask** per image straight from DINOv3’s internal features, no training required. It’s useful to: - Quickly **highlight the main subject** (saliency-ish) to reduce background bias - Improve **visual search** by focusing on foreground regions - Speed up **data curation** (find images with weak/strong foreground, spot occlusions) - Generate **lightweight pseudo-labels** that you can review in the FiftyOne App ### What we compute (high level) - **ViT models (DINOv3 ViT)**: for each image patch, compute the **cosine similarity to the CLS token**. Patches more aligned with CLS are treated as foreground. - **ConvNeXt-style models**: compute the cosine similarity of each spatial feature to the **global average feature vector**. We then **normalize → optionally smooth → threshold** the similarity map: 1. Min–max scale to `[0, 1]` 2. Optional average pooling in patch space to denoise 3. Threshold to get a **binary mask** (foreground/background) Finally, we upsample to the original image size and write the results into the dataset: - A **binary segmentation** field (`fo.Segmentation`) - Optionally, a **soft heatmap** field (`fo.Heatmap`) with values in `[0, 1]` These overlays render natively in the **FiftyOne App**.