Run in Google Colab

View source on GitHub

Download notebook

# Clustering and Labeling with Embeddings In this notebook, we will explore how to use **embeddings** to cluster and assign labels to dataset samples. This technique is valuable when dealing with unlabeled datasets or when refining existing labels.  ## Learning Objectives: - Understand clustering techniques and their role in dataset labeling. - Compute embeddings and cluster dataset samples. - Use FiftyOne Plugin for clustering MVTec dataset - Use FiftyOne for visualization and dataset management. ## Why Use Clustering for Labeling? Labeling datasets manually is expensive and time-consuming. **Clustering** helps by automatically grouping similar samples based on embeddings. This method is useful for: - **Unsupervised Learning**: Automatically discovering patterns in unlabeled data. - **Dataset Cleanup**: Detecting mislabeled samples. - **Efficient Annotation**: Pre-labeling groups for human annotators. Common clustering techniques include: - **K-Means** (partition-based clustering) - **DBSCAN** (density-based clustering) - **Hierarchical Clustering** (tree-based grouping) **Relevant Documentation:** - [Clustering in Machine Learning](https://en.wikipedia.org/wiki/Cluster_analysis) - [Introduction to Unsupervised Learning](https://towardsdatascience.com/a-guide-to-clustering-algorithms-e28af85da0b7/#:~:text=Clustering%20is%20an%20unsupervised%20Machine,and%20even%20some%20use%20cases.) Select a GPU Runtime if possible, install the requirements, restart the session, and verify the device information. ## Requirement step: Remove Category Labels (Keep Only Defects) Since category labels in your dataset are stored as `category.label`, you can remove them like is showing in the following cell. This removes all `category.label` fields from the dataset while keeping `defect.label`. ## Generating Embeddings Before clustering, we must **generate embeddings** that capture meaningful feature representations. FiftyOne provides multiple ways to obtain embeddings: - **Using pre-trained models** from FiftyOne’s Model Zoo (e.g., CLIP, ResNet). - **Extracting embeddings from custom models** (e.g., OpenAI, Anomalib, or self-trained networks). **Relevant Documentation:** [Computing Embeddings in FiftyOne](https://voxel51.com/docs/fiftyone/user_guide/brain.html#computing-embeddings) ## Clustering Dataset Samples with a FiftyOne Plugin Once we have embeddings, we can **cluster dataset samples** based on similarity. the theory around that is not complex but it could take some time for running and testing what do you want to do. For Example for clustering samples by similarity you maybe need to use the following code: Example using **K-Means Clustering**: ```python from sklearn.cluster import KMeans num_clusters = 5 kmeans = KMeans(n_clusters=num_clusters, random_state=42) cluster_labels = kmeans.fit_predict(embeddings) ``` After clustering, we can assign labels to our dataset: ```python import fiftyone as fo # Assign clustering results to FiftyOne dataset for sample, label in zip(dataset, cluster_labels): sample["cluster"] = int(label) # Store as an integer field sample.save() ``` ## But, What if I tell you you don’t need to write a code for doing that FiftyOne provides an extensible plugin system that allows users to enhance their workflows with additional functionalities, such as dataset visualization, embeddings analysis, and automated clustering. A FiftyOne Plugin is a modular component that extends the capabilities of the FiftyOne UI or backend. These plugins can be used for various tasks, including visualization, dataset processing, and machine learning model integration. ### Clustering Plugin in FiftyOne One powerful plugin available in FiftyOne is the Clustering Plugin, which allows users to group similar samples together based on embeddings. This is useful for tasks like: - Understanding patterns in your dataset - Identifying redundant or mislabeled data - Grouping anomalies or outliers **Relevant Documentation:** - [Clustering Algorithms](https://en.wikipedia.org/wiki/Cluster_analysis) - [FiftyOne Dataset Fields](https://docs.voxel51.com/user_guide/using_datasets.html#fields) - [FiftyOne Clustering Plugin](https://github.com/jacobmarks/clustering-plugin) ## Install the Clustering Plugin You can see how FiftyOne App allows you to cluster your dataset using a variety of algorithms: [K-Means](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html#sklearn.cluster.KMeans), [Birch](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.Birch.html#sklearn.cluster.Birch), [Agglomerative](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.AgglomerativeClustering.html#sklearn.cluster.AgglomerativeClustering), [HDBSCAN](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.AgglomerativeClustering.html#sklearn.cluster.HDBSCAN). It also serves as a proof of concept for adding new “types” of runs to FiftyOne!!! You will also need to have `scikit-learn` installed:  Try experimenting with different clustering methods (e.g., HDBSCAN, Birch clustering) and evaluate their impact on labeling quality! 🚀