Run in Google Colab

View source on GitHub

Download notebook

# Integrations, Plugins, and Model Evaluation In this notebook, we will explore **FiftyOne’s integration capabilities**, **plugin system**, and **model evaluation tools**. This is particularly useful when working with external frameworks, custom plugins, and evaluating AI models.  ## Learning Objectives: - Understand how FiftyOne integrates with external tools. - Learn about FiftyOne’s plugin system and how to configure it. - Use the **voxel51/evaluation** plugin to evaluate models. --- ## FiftyOne Integrations and Plugins FiftyOne provides a flexible architecture that enables **seamless integration** with external libraries, models, and cloud services. This is achieved through **integrations** and **plugins** that extend FiftyOne’s core functionalities. | Feature | Description | |------------------|--------------------------------------------------------------------------------------------------------| | **Integrations** | Connects FiftyOne with platforms like PyTorch, Ultralytics, CVAT, Lightning Flash, and Albumentations. | | | Enables easy dataset ingestion from sources like Hugging Face and Benchmark datasets | | **Plugins** | Allow users to **extend FiftyOne’s functionality**. | | | Can be used for custom dataset visualizations, model evaluations, and interactive tools. | 🔗 **Relevant Documentation:** - [FiftyOne Integrations](https://voxel51.com/docs/fiftyone/integrations/index.html) - [FiftyOne Plugins](https://voxel51.com/docs/fiftyone/plugins/index.html) ## Load the MVTec Dataset as usual # Download dataset from source We can download the file from Google Drive using `gdown` Let’s get started by importing the FiftyOne library, and the utils we need for a COCO format dataset, depending of the dataset format you should change that option. [Supported Formats](https://docs.voxel51.com/user_guide/dataset_creation/datasets.html#supported-formats) ## Set up the dataset into Anomalib (custom folder) and train the model I want to clarify that Anomalib natively supports MVTec and you can use Datamodule callback for using MVTec AD dataset. However for the education purposes of this workshop we will manage MVTec Bottle subset as a custom folder in Anomalib. Selecting `OBJECT=bottle` in the next cell and adding normal and abnormal folders to the tree directory. ## Configuring Plugins in FiftyOne To use plugins in FiftyOne, we need to enable and configure them properly. ### Minimum Configuration Steps: 1. Ensure FiftyOne is installed and up to date. 2. Download plugind using `fiftyone` CLI. ## Set up the dataset into Anomalib (custom folder) and train the model I want to clarify that Anomalib natively supports MVTec and you can use Datamodule callback for using MVTec AD dataset. However for the education purposes of this workshop we will manage MVTec Bottle subset as a custom folder in Anomalib. Selecting `OBJECT=bottle` in the next cell and adding normal and abnormal folders to the tree directory. ## Model Evaluation in FiftyOne for anomaly detection in a bottle factory FiftyOne provides a built-in model evaluation system that supports various evaluation types, such as: - **Classification Evaluation** (Accuracy, Precision, Recall, F1-score) - **Object Detection Evaluation** (IoU, mAP) - **Segmentation Evaluation** We will use the [voxel51/evaluation plugin](https://github.com/voxel51/fiftyone-plugins/blob/main/plugins/evaluation/README.md) to evaluate our model’s performance. ### Steps: 1. Load a dataset with ground truth labels and predictions. 2. Select an evaluation method. 3. Run the evaluation and analyze the results. **Relevant Documentation:** [Evaluating Models in FiftyOne](https://voxel51.com/docs/fiftyone/user_guide/evaluation.html) ## Load the data into FiftyOne ## Check if there are previous evaluations ## Running Inference with Anomalib’s Engine This code demonstrates how to set up and run inference using Anomalib’s `Engine` with a **PaDiM** model. ## Visualizing Anomaly Predictions in Anomalib This code processes and visualizes the anomaly detection results from Anomalib. It extracts prediction details, displays anomaly maps, and overlays them on the input image. ## Running Anomaly Detection Inference and Grouping Results in FiftyOne This function, `run_inference`, performs **anomaly detection** on a collection of images using a pre-trained model in **Anomalib**. The results, including anomaly scores, masks, and heatmaps, are stored in [FiftyOne’s grouped dataset](https://docs.voxel51.com/user_guide/groups.html?highlight=group) format for better visualization and analysis. ### **Function Overview** ```python def run_inference(sample_collection, key, engine, threshold=0.5, ckpt_path=None): ``` - **sample_collection**: The input FiftyOne dataset containing images for inference. - **key**: Identifier for Anomalib model. - **engine**: The inference engine (e.g., Anomalib’s Engine) with the model loaded. - **threshold**: Score threshold for classifying images as “normal” or “abnormal”. - **ckpt_path:** Path to the trained model’s checkpoint. ## Reloading and Visualizing the Grouped Dataset in FiftyOne After running inference, we reload the grouped dataset to ensure all predictions are updated and then visualize the results in the **FiftyOne App**.  ## Evaluating Anomaly Classification with the Model Evaluation Plugin in FiftyOne When using the **Model Evaluation Plugin** in FiftyOne, we can evaluate model predictions using either: 1. **The Model Evaluation Panel in the FiftyOne App** – Interactive UI-based evaluation. 2. **The FiftyOne SDK** – Programmatic evaluation via Python scripts. **Which one to use?** - If you prefer an **interactive visual analysis**, the **FiftyOne App** provides an intuitive panel for evaluating models. - If you need **automated and reproducible results**, the **FiftyOne SDK** allows for batch evaluation and detailed reporting. ## Visualizing Evaluation Results After running the evaluation, we can inspect the results using the FiftyOne App: In the App, you can: Filter samples by correct or incorrect predictions. View IoU scores, confusion matrices, and detection overlaps. Generate customized evaluation reports. Relevant Documentation: [Interactive Model Evaluation in FiftyOne](https://github.com/voxel51/fiftyone-plugins/blob/main/plugins/evaluation/README.md)  ### Next Steps: Try using other plugins or **create your own** to extend FiftyOne’s capabilities! 🚀