Run in Google Colab   View source on GitHub   Download notebook
# Understanding and Using Embeddings Welcome to this hands-on workshop where we will explore **embeddings** and their importance in Visual AI. Embeddings play a crucial role in **image search, clustering, anomaly detection, and representation learning**. In this notebook, we will learn how to generate, visualize, and explore embeddings using **FiftyOne**. ![using_embeddings](https://cdn.voxel51.com/getting_started_manufacturing/notebook2/using_embeddings.webp) ## Learning Objectives: - Understand what embeddings are and why they matter in Visual AI. - Learn how to compute and store embeddings in FiftyOne. - Use embeddings for similarity search and visualization. - Leverage FiftyOne’s interactive tools to explore embeddings. ## What Are Embeddings? Embeddings are **vector representations** of data (images, videos, text, etc.) that capture meaningful characteristics. For images, embeddings store compressed feature representations learned by deep learning models. These features enable tasks such as: - **Similarity Search**: Find images that are visually similar. - **Clustering**: Group images with shared characteristics. - **Anomaly Detection**: Identify outliers in datasets. - **Transfer Learning**: Use learned embeddings to improve other AI tasks. ### Further Reading: - [Introduction to Embeddings](https://www.tensorflow.org/text/guide/word_embeddings) - [Feature Representations in Deep Learning](https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html) ## Generating Embeddings in FiftyOne FiftyOne provides seamless integration for embedding computation. You can extract embeddings using pre-trained deep learning models (such as CLIP, ResNet, or custom models) and store them in FiftyOne datasets. ### How It Works: 1. Load a dataset in FiftyOne. 2. Extract embeddings from a model. 3. Store and visualize embeddings. **Relevant Documentation:** [Computing and Storing Embeddings](https://voxel51.com/docs/fiftyone/user_guide/brain.html#computing-embeddings)
Note: You must install the `umap-learn>=0.5` package in order to use UMAP-based visualization. This is recommended, as UMAP is awesome! If you do not wish to install UMAP, try `method='tsne'` instead Select a GPU Runtime if possible, install the requirements, restart the session, and verify the device information. # Download dataset from source We can download the file from Google Drive using `gdown` Let’s get started by importing the FiftyOne library, and the utils we need for a COCO format dataset, depending of the dataset format you should change that option. [Supported Formats](https://docs.voxel51.com/user_guide/dataset_creation/datasets.html#supported-formats) ## Exploring and Visualizing Embeddings Once embeddings are generated, we can visualize them using **dimensionality reduction techniques** like: - **t-SNE (t-Distributed Stochastic Neighbor Embedding)** - **UMAP (Uniform Manifold Approximation and Projection)** These methods reduce the high-dimensional feature space into 2D/3D representations for interactive visualization. **Relevant Documentation:** [Visualizing Embeddings in FiftyOne](https://docs.voxel51.com/brain.html#visualizing-embeddings), [Dimensionality Reduction](https://docs.voxel51.com/brain.html#visualizing-embeddings)
Note: Be patient, it will take about 5-10 minutes to compute the embeddings. ## Performing Similarity Search with Embeddings With embeddings, we can search for visually similar images by computing the nearest neighbors in the embedding space. FiftyOne provides built-in tools to perform **similarity search** efficiently. **Relevant Documentation:** [Performing Similarity Search](https://voxel51.com/docs/fiftyone/user_guide/brain.html#similarity-search) ![similarity](https://cdn.voxel51.com/getting_started_manufacturing/notebook2/similarity.webp) ### Next Steps: Try using different models for embedding extraction, explore clustering techniques, and test similarity search with your own datasets! 🚀