Run in Google Colab

View source on GitHub

Download notebook

# Custom Embeddings for Anomaly Detection In this notebook, we will explore how to generate **custom embeddings** for **anomaly detection** using the **Padim model** from Anomalib. Unlike general-purpose embeddings from models like CLIP or ResNet, anomaly detection requires **task-specific embeddings** that can distinguish between normal and abnormal samples.  ## Learning Objectives: - Understand the difference between standard embeddings and anomaly-specific embeddings. - Explore how to compute embeddings using **Padim from Anomalib**. - Integrate these embeddings into a FiftyOne dataset. - Leverage FiftyOne for visualization and analysis. ## Why Use Custom Embeddings for Anomaly Detection? Pre-trained models like **CLIP or ResNet** generate **general-purpose embeddings** that focus on visual similarity. However, detecting **abnormalities** requires learning **subtle deviations** from normal patterns, which these models cannot capture effectively. Instead, we use a dedicated anomaly detection model like **Padim from Anomalib**, which: - Learns representations specific to normal and anomalous samples. - Extracts feature maps from an encoder (e.g., ResNet). - Compares new samples against normal feature distributions. ### Further Reading: - [Anomalib Documentation](https://github.com/openvinotoolkit/anomalib) - [Understanding Memory-Based Anomaly Detection](https://arxiv.org/pdf/2011.08785) ## Load the MVTec Dataset as usual ## Extracting Custom Embeddings from Padim (Anomalib) Instead of using a general embedding model, we will: 1. **Load a Padim anomaly detection model** using Anomalib. 2. **Run inference on a dataset** to extract anomaly embeddings. 3. **Store the embeddings in FiftyOne** for further visualization. **Relevant Documentation:** - [Anomalib Models](https://anomalib.readthedocs.io/en/latest/markdown/guides/reference/models/image/index.html) - [Remotely-sourced Zoo Models](https://docs.voxel51.com/model_zoo/remote.html) ## Integrating Anomaly Embeddings into FiftyOne Once we obtain embeddings from Padim, we will add them to our FiftyOne dataset. This allows us to: - Perform **similarity searches** based on anomaly scores. - Compare normal vs. abnormal sample distributions. - Leverage **FiftyOne App** to inspect anomalies. ```python import fiftyone as fo dataset = fo.Dataset("object_from_mvtec_ad") # Add embeddings to each sample for sample in dataset: ... # Convert to CPU NumPy for storage embedding_1d = patch_embedding.squeeze(0).cpu().numpy() # shape (D,) # Store as a list in a new field sample["embedding"] = embedding_1d.tolist() sample.save() ... ``` **Relevant Documentation:** [Adding Custom Fields to FiftyOne Datasets](https://docs.voxel51.com/user_guide/using_datasets.html) ## Selecting object from MVTec AD Dataset ## Calculating Embeddings using Inference with Padim Model ## Visualizing Embeddings in FiftyOne  ### Next Steps: Try using different anomaly detection models from Anomalib and compare their embeddings with FiftyOne’s visualization tools! 🚀