Note

This is a community plugin, an external project maintained by its respective author. Community plugins are not part of FiftyOne core and may change independently. Please review each plugin’s documentation and license before use.

Text-to-Image Plugin

Updates

• 2024-04-23: Added support for Stable Diffusion 3 (Thanks Dan Gural)

• 2023-12-19: Added support for Kandinsky-2.2 and Playground V2 models

• 2023-11-30: Version 1.2.0

  • adds local model running via diffusers (>=0.24.0)

  • adds calling from the Python SDK!

  • :warning: BREAKING CHANGE: the plugin and operator URIs have been changed from ai_art_gallery to text_to_image. If you have any saved pipelines that use the plugin, you will need to update the URIs.

• 2023-11-08: Version 1.1.0 adds support for DALLE-3 Model — upgrade to openai>=1.1.0 to use

• 2023-10-30: Added support for Segmind Stable Diffusion (SSD-1B) Model

• 2023-10-23: Added support for Latent Consistency Model

• 2023-10-18: Added support for SDXL, operator icon, and download location selection

Plugin Overview

This plugin is a Python plugin that allows you to generate images from text prompts and add them directly into your dataset.

:warning: This plugin is only verified to work for local datasets. It may not work for remote datasets.

Supported Models

This version of the plugin supports the following models:

• DALL-E2

• DALL-E3

• Kandinsky-2.2

• Latent Consistency Model

• Playground V2

• SDXL

• SDXL-Lighting

• Segmind Stable Diffusion (SSD-1B)

• Stable Diffusion

• Stable Diffusion 3

• VQGAN-CLIP

It is straightforward to add support for other models!

Watch On Youtube

Installation

fiftyone plugins download https://github.com/jacobmarks/text-to-image

If you want to use Replicate models, you will need to pip install replicate and set the environment variable REPLICATE_API_TOKEN with your API token.

If you want to use DALL-E2 or DALL-E3, you will need to pip install openai and set the environment variable OPENAI_API_KEY with your API key.

To run the Latency Consistency model locally with Hugging Face’s diffusers library, you will need diffusers>=0.24.0. If you need to, you can install it with pip install diffusers>=0.24.0.

To run Stable Diffusion 3, you will need to set up a Stability.ai account to get access to key. Then set the environment variable STABILITY_API_KEY with your API token.

Refer to the main README for more information about managing downloaded plugins and developing plugins locally.

Operators

txt2img

• Generates an image from a text prompt and adds it to the dataset

Python SDK

You can also use the txt2img operators from the Python SDK!

If you’re using fiftyone<=0.23.6, due to the way Jupyter Notebooks interact with asyncio, this will not work in a Jupyter Notebook. You will need to run this code in a Python script or in a Python console.

import fiftyone as fo
import fiftyone.operators as foo
import fiftyone.zoo as foz

dataset = fo.load_dataset("quickstart")

## Access the operator via its URI (plugin name + operator name)
t2i = foo.get_operator("@jacobmarks/text_to_image/txt2img")

## Run the operator

prompt = "A dog sitting in a field"
t2i(dataset, prompt=prompt, model_name="latent-consistency", delegate=False)

## Pass in model-specific arguments
t2i(
    dataset,
    prompt=prompt,
    model_name="latent-consistency",
    delegate=False,
    width=768,
    height=768,
    num_inference_steps=8,
)