Run in Google Colab

View source on GitHub

Download notebook

# Google Gemini Vision in FiftyOne The rapid advancement of multimodal AI models has opened new possibilities for computer vision workflows. Google’s [Gemini Vision](https://ai.google.dev/gemini-api/docs/vision) models combine powerful visual understanding with natural language processing, enabling sophisticated image analysis, generation, and manipulation tasks.  The [Gemini Vision Plugin](https://docs.voxel51.com/plugins/plugins_ecosystem/gemini_vision_plugin.html) for FiftyOne brings these capabilities directly into your data-centric workflows, allowing you to leverage Gemini’s vision-language models for dataset analysis, augmentation, and quality improvement. In this tutorial, we’ll demonstrate how to use the Gemini Vision Plugin with FiftyOne to analyze a real-world autonomous driving dataset, identify dataset issues, and use Gemini’s generative capabilities to improve data quality. Specifically, this walkthrough covers: - Installing and configuring the Gemini Vision Plugin for FiftyOne - Loading the KITTI autonomous driving dataset - Analyzing dataset quality and identifying biases using FiftyOne Brain - Using Gemini Vision to query and understand images - Detecting missing classes and annotation gaps - Generating new training images with text-to-image - Editing existing images to address dataset gaps - Transferring styles between images - Analyzing video content with Gemini’s video understanding capabilities **So, what’s the takeaway?** By combining FiftyOne’s dataset analysis capabilities with Gemini Vision’s multimodal AI features, you can build a powerful workflow for understanding, improving, and augmenting your computer vision datasets. ## What is Google Gemini Vision? [Google Gemini](https://deepmind.google/technologies/gemini/) is a family of multimodal AI models developed by Google DeepMind. Gemini Vision extends these models’ capabilities to understand and generate visual content: - **Multimodal Understanding**: Process both images and text together for deep contextual understanding - **1M Token Context Window**: Analyze large amounts of visual and textual data in a single request (with Gemini 3.0) - **Image Generation**: Create new images from text descriptions - **Image Editing**: Modify existing images based on natural language instructions - **Video Understanding**: Analyze and query video content with temporal awareness - **Adjustable Reasoning**: Control the depth of analysis with configurable thinking levels The Gemini Vision Plugin makes these capabilities accessible directly within your FiftyOne workflows, enabling seamless integration of generative AI into your data preparation pipelines. ## Setup To get started, you need to install [FiftyOne](https://docs.voxel51.com/getting_started/install.html) and the Gemini Vision Plugin: ### Configure Gemini API Access To use the Gemini Vision Plugin, you’ll need a Google Cloud account with the Gemini API enabled. **Important**: The Gemini API requires billing to be enabled on your Google Cloud account. You can get started at [Google AI Studio](https://aistudio.google.com/app/apikey). Once you have your API key, set it as an environment variable: Now import FiftyOne and related modules: ## Exploring Gemini Vision Capabilities with FiftyOne In this first part of the tutorial, we explore multiple Gemini Vision capabilities integrated with **FiftyOne**, showing how multimodal models can help you *understand, enrich, and debug* your datasets. We’ll walk through several real-world use cases, starting from simple dataset understanding and moving toward more advanced tasks like OCR, spatial reasoning, and video understanding. ### 1. Asking Questions About Your Dataset (Visual Q&A) We start with the simplest and often most powerful use case: **asking questions about your dataset**. Using the FiftyOne quickstart, we load a dataset and begin asking natural language questions such as: - *What objects appear most frequently?* - *Are there any anomalies or unusual samples?* - *What is happening in this image or scene?* This is a great first step to quickly understand what’s inside your database, validate assumptions, and spot potential issues before training or evaluation. > This approach is especially useful when working with large or unfamiliar datasets.  ### 2. OCR: Extracting Text and Adding It to Your Dataset Next, we move into **OCR (Optical Character Recognition)**. Here, Gemini Vision is used to dynamically extract structured information from documents such as invoices, receipts, or forms. For example: - Invoice numbers - Dates - Total amounts - Vendor names We then map this extracted information back into the dataset using **bounding boxes**, so the detected text becomes a first-class citizen in FiftyOne—searchable, filterable, and visualizable. This enables workflows like: - Auditing documents - Validating annotations - Building downstream analytics pipelines In this example, we load a receipts dataset directly from Hugging Face (`Voxel51/consolidated_receipt_dataset`) to show how quickly you can enrich real-world document data with OCR annotations inside FiftyOne. You can access more than 100 datasets already available in FiftyOne format—explore the full dataset ecosystem here: [https://docs.voxel51.com/dataset_zoo/index.html](https://docs.voxel51.com/dataset_zoo/index.html)  ### 3. Spatial Understanding with Keypoints and References In this section, we explore **spatial understanding**. The idea here is not just to recognize *what* is in an image, but *where* things are and how they relate to each other. Gemini Vision can point to specific regions or objects, and we can visualize these results using **FiftyOne keypoints and spatial annotations**. Examples include: - Pointing to specific components in an image - Highlighting regions of interest - Visualizing relationships between objects For this example, we use the [ALOHA Pen Uncap dataset](https://docs.voxel51.com/dataset_zoo/datasets_hf/aloha_pen_uncap.html) from Hugging Face, which contains real-world robot manipulation demonstrations captured from an egocentric perspective. The dataset is designed for studying fine-grained manipulation, action understanding, and spatial reasoning in robotics:  ### 4. Video Understanding: Events, Timing, and Context Finally, we extend these ideas to **video understanding**. Instead of analyzing a single image, we process full videos to: - Understand what is happening over time - Detect key events - Extract timestamps and temporal segments - Answer questions about sequences and actions This is particularly useful for applications like surveillance, sports analysis, robotics, or long-form video datasets.  ### Why This Matters By combining **Gemini Vision’s multimodal reasoning** with **FiftyOne’s dataset management and visualization**, you can: - Explore datasets faster - Add rich semantic metadata - Debug data issues visually - Build more interpretable AI systems Next, let’s put this into practice. We’ll walk through a hands-on driving example, starting with dataset exploration and then using Gemini to analyze, enrich, and improve the quality of the dataset, step by step. ## Load the KITTI Dataset For this tutorial, we’ll use the [KITTI Dataset](https://www.cvlibs.net/datasets/kitti/), a large-scale diverse driving dataset containing 7,481 annotated images, and the test split contains 7,518 unlabeled images, across various weather conditions, times of day, and scenes. KITTI is perfect for demonstrating Gemini Vision’s capabilities because: - It contains diverse real-world scenarios - It has complex multi-object scenes - It’s used for autonomous driving research, where dataset quality is critical - It may contain annotation biases and gaps that we can identify and address Downloading the dataset for the first time can take around 30 minutes, and for this tutorial, we’ll use a subset of the training split. Let’s visualize the dataset in the FiftyOne App:  ## Analyzing Dataset Quality with FiftyOne Brain Before we start using Gemini Vision, let’s analyze our dataset to understand its characteristics and identify potential issues. FiftyOne Brain provides powerful capabilities for dataset analysis. ### Identifying Class Imbalance and Bias First, let’s examine the distribution of object classes in our dataset to identify any biases or underrepresented categories. You can also explore these insights with the [Dashboard plugin](https://docs.voxel51.com/plugins/plugins_ecosystem/dashboard.html), which lets you build custom dashboards to visualize key statistics about your dataset. ### Computing Dataset Uniqueness Next, let’s use FiftyOne Brain to identify unique and potentially redundant samples in our dataset:  ### Detecting Near-Duplicate Images Duplicate or near-duplicate images can inflate evaluation metrics and waste training time. Let’s find them: ### Visualizing Dataset Embeddings Let’s compute embeddings and visualize the dataset structure to identify clusters and potential gaps:  The embeddings plot in the FiftyOne App reveals clustering patterns in the data. Isolated samples or sparse regions may indicate underrepresented scenarios that need more data. ## Using Gemini Vision for Image Understanding Now let’s use the Gemini Vision Plugin to query and understand images in our dataset. The plugin provides several operators that can be accessed through the FiftyOne App or programmatically. ### Querying Images with Natural Language Let’s select a few samples and use Gemini to analyze them. First, we’ll select some samples with specific objects: | Now, you can use the Gemini Vision Plugin operators from the FiftyOne App: 1. Select one or more samples in the App 2. Press the backtick key (\`) to open the operator browser 3. Search for “query_gemini_vision” or “Query Gemini Vision” 4. Enter your query, for example: - “Describe the weather and lighting conditions in this image” - “What time of day does this appear to be?” - “Are there any pedestrians or cyclists visible?” - “Describe potential safety hazards in this driving scene” The plugin will use Gemini Vision to analyze the image and return a text response, which you can save to a custom field in your dataset. ### Identifying Missing Annotations One powerful use case for Gemini Vision is identifying objects that may be missing from annotations. Let’s use it to audit our annotations:  ### Analyzing Difficult or Ambiguous Cases Let’s identify samples with many objects that might be challenging to annotate:  ## Detecting Missing Classes and Coverage Gaps Based on our class distribution analysis, we may have identified underrepresented object classes. Let’s systematically find which classes are missing or underrepresented: ### Identifying Scenario Coverage Gaps Beyond object classes, we should also consider scenario diversity. Let’s use Gemini Vision to categorize our images by scenario characteristics: ## Addressing Dataset Gaps with Image Generation Now that we’ve identified missing classes and underrepresented scenarios, let’s use Gemini’s text-to-image generation capabilities to create synthetic training data. ### Generating Images for Missing Classes The Gemini Vision Plugin includes a text-to-image generation operator. You can use it from the FiftyOne App: 1. Open the operator browser (backtick key) 2. Search for “generate_image” or “Generate Image” 3. Enter prompts for missing or rare classes: **Example prompts for autonomous driving scenarios:**  ## Editing Images to Augment Dataset Diversity In addition to generating new images, Gemini Vision can edit existing images based on natural language instructions. This is useful for creating variations and augmenting dataset diversity. ### Using the Image Editing Operator To edit images with Gemini Vision: 1. Select a single sample in the FiftyOne App 2. Open the operator browser (backtick key) 3. Search for “edit_image” or “Edit Image” 4. Enter editing instructions **Example editing prompts:** The edited image will be saved with the original prompt preserved in metadata.  ## Transferring Styles Between Images Gemini Vision can combine multiple images to create new scenes or transfer styles. This is useful for: - Transferring weather conditions from one image to another - Combining objects from different scenes - Creating composite training examples ### Using Multi-Image Composition To use multi-image composition: 1. Select 2-3 samples in the FiftyOne App 2. Open the operator browser 3. Search for “compose_images” or “Multi-Image Composition” 4. Enter composition instructions **Example composition prompts:**  ## Video Understanding with Gemini Vision Gemini Vision also supports video understanding, allowing you to analyze temporal sequences and extract insights from video data. This is particularly relevant for autonomous driving where temporal context matters. ### Loading Video Data Let’s load a video dataset to demonstrate Gemini’s video understanding capabilities: ### Querying Video Content The Gemini Vision Plugin includes a video understanding operator with multiple modes: 1. **Describe**: Get a detailed description of the video content 2. **Segment**: Identify temporal segments with different characteristics 3. **Extract**: Extract specific information (objects, actions, events) 4. **Question**: Ask specific questions about the video content To use video understanding: 1. Select a video sample in the FiftyOne App 2. Open the operator browser 3. Search for “analyze_video” or “Video Understanding” 4. Select the mode and enter your query **Example video queries:**  Now it’s your turn, keep exploring Gemini Vision to pull more insights from the video. You can try things like analyzing temporal patterns, extracting temporal annotations, and more. ### Analyzing Temporal Patterns Video understanding allows you to identify temporal patterns that aren’t visible in individual frames: ### Extracting Temporal Annotations The responses from video understanding can be used to create temporal annotations in FiftyOne: ## Pipeline: Scale Image Editing with Gemini Scale your dataset by applying multiple edit prompts to images programmatically. The pipeline uses the `image_editing` operator and stores metadata (`generation_type`, `prompt`) on each generated sample for easy filtering and visualization.  You’ve successfully scaled your dataset using Gemini’s image editing capabilities. Each generated image is stored with metadata (`generation_type`, `prompt`, `source_file`) for easy filtering and traceability. ## Summary In this tutorial, we’ve demonstrated how the Gemini Vision Plugin extends FiftyOne’s capabilities with powerful multimodal AI features: **Dataset Analysis:** - Used FiftyOne Brain to identify class imbalances, duplicates, and coverage gaps - Leveraged Gemini Vision to audit annotations and identify missing labels - Classified images by scenario characteristics (weather, time, scene type) **Dataset Enhancement:** - Generated synthetic images for underrepresented classes and scenarios - Edited existing images to create weather and lighting variations - Transferred styles between images to augment dataset diversity **Video Understanding:** - Analyzed temporal patterns in driving videos - Extracted event timestamps and segmented videos by characteristics - Queried video content with natural language By combining FiftyOne’s data-centric workflows with Gemini Vision’s multimodal AI capabilities, you can build higher-quality, more diverse datasets that lead to more robust computer vision models. For more information: - [Gemini Vision Plugin Documentation](https://docs.voxel51.com/plugins/plugins_ecosystem/gemini_vision_plugin.html) - [FiftyOne Brain Documentation](https://docs.voxel51.com/user_guide/brain.html) - [Google Gemini API Documentation](https://ai.google.dev/gemini-api/docs)