Note

This is a community plugin, an external project maintained by its respective author. Community plugins are not part of FiftyOne core and may change independently. Please review each plugin’s documentation and license before use.

GitHub Repo

FiftyOne Plugin for OpenAI#

A FiftyOne plugin for labeling images with OpenAI vision models. Send images from your dataset to models like gpt-5.4-nano, gpt-5.4-mini, or gpt-5.4 and get structured labels back — classifications, tags, detections, captions, VQA answers, or OCR text — directly in the FiftyOne App.

Uses the OpenAI Responses API with Pydantic structured output for reliable, schema-validated responses.

Installation#

CLI#

fiftyone plugins download https://github.com/Burhan-Q/fo-openai

Python#

import fiftyone.plugins as fop

fop.download("https://github.com/Burhan-Q/fo-openai")

Install dependencies#

pip install openai pydantic

Requirements#

Setup#

Set your OpenAI API key as an environment variable:

export OPENAI_API_KEY="sk-..."

Or use the FiftyOne-specific name:

export FIFTYONE_OPENAI_API_KEY="sk-..."

Both are declared in fiftyone.yml and resolved automatically via FiftyOne’s plugin secrets system.

Usage#

From the FiftyOne App#

  1. Open a dataset in the FiftyOne App

  2. Press ` to open the operator browser

  3. Search for “Run OpenAI Inference”

  4. Select a model and task, then execute

Programmatic#

import fiftyone as fo
import fiftyone.operators as foo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart", max_samples=10)

foo.execute_operator(
    "@Burhan-Q/fo-openai/run_openai_inference",
    params={
        "model": "gpt-5.4-nano",
        "task": "classify",
        "classes": ["dog", "cat", "person", "vehicle", "other"],
    },
    dataset_name=dataset.name,
)

Note: classes accepts either a list of strings or a comma-separated string (e.g. "dog, cat, person").

Supported Tasks#

Task

Description

Output Type

Caption

Generate a text description of the image

fo.Classification

Classify

Assign a single class label

fo.Classification

Tag

Assign multiple labels

fo.Classifications

Detect

Locate objects with bounding boxes

fo.Detections

VQA

Answer a question about the image

fo.Classification

OCR

Extract visible text

fo.Classification

Features#

  • Structured output — responses parsed via client.responses.parse() with Pydantic models; no regex or string matching

  • Cost preview — estimated per-sample and total cost displayed before running, with token breakdown table

  • Few-shot exemplars — optionally provide labeled samples as reference examples to improve output quality

  • Class labels from your dataset — pick an existing label field to reuse its classes, or type new classes with autocomplete

  • Detection coordinate formats — pixel (recommended), 0-1000 integers, or 0-1 floats

  • Batch processing — configurable batch size and concurrency

  • Delegated execution — run large jobs in the background

  • Persistent settings — all configuration persists across app restarts via ExecutionStore

  • Opt-in logging — log to stderr and/or file, with auto-incrementing filenames

  • JSON config export/import — save and reuse run configurations

Configuration#

The operator form has model and base URL always visible at the top, with 4 tabs below:

Section

Settings

Model (always visible)

Model ID (e.g. gpt-5.4-nano, gpt-5.4-mini), custom base URL

Task tab

Task type, class labels (autocomplete), custom prompts, detection coordinate/box format, output field

Exemplars tab

Enable few-shot examples, select source (saved view, sample IDs, tag, or field), label field

Logging tab

Enable logging, log level, log file path

Advanced tab

Temperature, max output tokens, top P, request timeout, batch size, concurrency, image workers, image detail, system prompt override

A cost summary table is rendered below the tabs (always visible regardless of active tab).

Settings persist between runs. Use the Reset to defaults config mode to clear them.

Few-Shot Exemplars#

Provide labeled samples as reference examples sent alongside every inference call. The model sees your exemplar images with their correct labels before processing each target image.

Exemplar sources:

  • Saved view — a named FiftyOne saved view

  • Sample IDs — specific sample IDs

  • Tag — all samples matching a tag

  • Field — samples where a field equals a value

See docs/operations/exemplars.md for details.

Development#

Clone the repository and symlink into your FiftyOne plugins directory:

git clone https://github.com/Burhan-Q/fo-openai.git
ln -s "$(pwd)/fo-openai" "$(fiftyone config plugins_dir)/@Burhan-Q/fo-openai"

Run tests:

pip install pytest
python -m pytest tests/

License#

Apache 2.0