Run in Google Colab   View source on GitHub   Download notebook
# Building Annotation Workflows and Ontologies with FiftyOne This tutorial builds a complete annotation loop in FiftyOne: - **Curate** — select what to label with [embeddings](https://docs.voxel51.com/brain.html#brain-embeddings-visualization), [similarity](https://docs.voxel51.com/brain.html#brain-image-similarity), [uniqueness](https://docs.voxel51.com/brain.html#brain-image-uniqueness), and zero-shot tagging - **Annotate** — define an [ontology](https://docs.voxel51.com/user_guide/annotation.html), route samples through a multi-stage [workflow](https://voxel51.com/annotation), then draft a first pass with an agent or [zoo models](https://docs.voxel51.com/model_zoo/models.html) - **Review** — rank labels by [estimated error](https://docs.voxel51.com/brain.html#brain-label-mistakes) and fix the worst - **Evaluate** — [score the model](https://docs.voxel51.com/user_guide/evaluation.html#evaluating-detections-coco) by scenario and feed failures back into the next curation pass

This tutorial uses FiftyOne Enterprise

Workflows, ontologies, and Agentic Labeling are FiftyOne Enterprise features, shown here through the App UI. The analysis steps — embeddings, similarity, uniqueness, mistakenness, evaluation, and zero-shot pre-labeling — also run in open-source FiftyOne.

![The FiftyOne annotation loop: curate, annotate, review, evaluate](https://cdn.voxel51.com/tutorial_fiftyone_annotation_workflows/data_engine_flywheel.webp) ## Setup Install FiftyOne and the extras used below: ### Connect to FiftyOne Enterprise Set these two variables before importing `fiftyone` (generate a key under **Settings → API keys**). Skip this cell to run open source against a local dataset. ### Load the example data Use the [quickstart](https://docs.voxel51.com/dataset_zoo/datasets/quickstart.html) dataset (200 COCO images with `ground_truth` and `predictions`) from the [FiftyOne Dataset Zoo](https://docs.voxel51.com/dataset_zoo/index.html) as the running example. The agentic-labeling section later pulls in a second dataset from the [Hugging Face Hub](https://docs.voxel51.com/integrations/huggingface.html). > Enterprise datasets reference cloud media (`gs://`, `s3://`) — point sample `filepath`s at storage your deployment can read. ## Stage 1 — Curate: select what to label Decide what’s worth labeling before annotation starts. ### Visualize in embedding space Compute a 2D [visualization](https://docs.voxel51.com/brain.html#brain-embeddings-visualization), then open the **Embeddings panel** in the App. ![The Embeddings panel: the quickstart dataset visualized in 2D with the img_viz brain key](https://cdn.voxel51.com/tutorial_fiftyone_annotation_workflows/data_engine_embeddings.webp) ### Score redundancy and coverage Rank by [uniqueness](https://docs.voxel51.com/brain.html#brain-image-uniqueness) and [representativeness](https://docs.voxel51.com/brain.html#brain-image-representativeness), and flag [near-duplicates](https://docs.voxel51.com/brain.html#brain-near-duplicates). ### Search by text and similarity Build a [similarity index](https://docs.voxel51.com/brain.html#brain-image-similarity), then query by [natural language](https://docs.voxel51.com/brain.html#brain-similarity-text) or by an example image. ### Auto-tag by metadata Tag each image as an indoor or outdoor scene with a zero-shot [CLIP model](https://docs.voxel51.com/model_zoo/models.html) — a field you can color the Embeddings panel by and slice evaluation on later. ### Build the subset to annotate Save the most-unique samples as a [saved view](https://docs.voxel51.com/user_guide/using_views.html#saving-views) to send to annotation. ## Stage 2 — Annotate: ontologies, workflows, and model-assisted labeling Define a shared ontology, route the curated view through a multi-stage workflow, and speed up the first pass with models — so annotators verify rather than draw from scratch. ### Define an ontology in code, apply it in the App An [ontology](https://docs.voxel51.com/user_guide/annotation.html) is your shared label schema — classes, attributes, and a taxonomy. Define it **once in code**, then **apply it to a field in the App** so every annotator gets the same classes and attributes. Conditional attributes (`when=...`) appear only when relevant. > The ontology SDK writes to the database directly, so run it with a direct database connection (not API mode). Now **apply it in the App**, click by click: 1. Open the dataset, then open the **field schema editor** (the schema/settings control). 2. Click the field you want to govern — here **ground_truth**. 3. In **Edit field schema**, turn the **Ontology** toggle **on**. 4. Open the **Chosen ontology** dropdown and select **coco-objects**. 5. Click **Save**. Every annotation task on that field now offers exactly the classes and attributes you defined in code. ### Manage annotation with multi-stage workflows *FiftyOne Enterprise.* The **Annotate** tab turns your curated saved view into a managed, multi-stage labeling effort — assign stages, track progress per task, and automatically loop rejected work back to annotators. Follow it click by click. **Step 1 — Create a workflow.** Open the **Annotate** tab. Click **+ New workflow** (top right). In the dialog, type a **Name** (for example, `Annotate Most Unique`), optionally add a description, and click **Create workflow**. ![The New workflow dialog: name the workflow, then click Create workflow](https://cdn.voxel51.com/tutorial_fiftyone_annotation_workflows/data_engine_wf_new.webp) **Step 2 — Pick a template.** On the next screen, under **Start from a template**, click a card to add its stages to the canvas: - **Human-in-the-loop** — Annotate → Review (one QA pass). - **Two-tier review** — Annotate → Review → Final review (a second sign-off). Click this one for the rest of the walkthrough. - Or click **+ Blank canvas** to add stages yourself. ![The template picker: choose Two-tier review, or start from a blank canvas](https://cdn.voxel51.com/tutorial_fiftyone_annotation_workflows/data_engine_wf_templates.webp) **Step 3 — Point it at your data and start.** On the canvas, click the **Input samples** node. In the **Configuration** panel on the right, select **Saved view**, then choose **to_annotate** from the dropdown (or pick **Whole dataset**). The dashed red **Rejected → Annotate** edges are already wired by the template, so rejected samples will loop back. Click **Start workflow** (top right). ![Configuring Input samples to the to_annotate saved view, then Start workflow](https://cdn.voxel51.com/tutorial_fiftyone_annotation_workflows/data_engine_wf_canvas.webp) **Step 4 — Open your task queue.** Starting the workflow generates tasks and returns you to the **Annotate** tab. Under **My tasks**, each stage assigned to you shows its progress (for example, `4 / 10 samples · 6 left`) and an action button — **Label** for annotate stages, **Review** for review stages. The workflow now shows as **Running**. ![The Annotate dashboard: My tasks with Label and Review actions; the workflow is Running](https://cdn.voxel51.com/tutorial_fiftyone_annotation_workflows/data_engine_wf_tasks.webp) **Step 5 — Label.** Click **Label** next to the Annotate task. FiftyOne opens a **locked** labeling view of only your assigned samples. Draw labels against the ontology, then mark each finished sample **Labeled** — it gets a green **LABELED** badge and the progress bar advances. Use **Resume labeling** to pick up where you left off; close the task when the count reads `0 left`. ![The locked labeling view: samples marked with green LABELED badges as you finish them](https://cdn.voxel51.com/tutorial_fiftyone_annotation_workflows/data_engine_wf_label.webp) **Step 6 — Review: approve or reject.** Back on the **Annotate** tab, click **Review** next to the **First review** task. For each submitted sample, click **Approve** (✓) to accept it or **Reject** (✗) to send it back. Approved samples advance to **Final review**; rejected samples follow the **Rejected** edge back to the Annotate stage. Click **Task complete** when the queue is empty. ![The review view: each sample marked APPROVED or REJECTED, then Task complete](https://cdn.voxel51.com/tutorial_fiftyone_annotation_workflows/data_engine_wf_review.webp) **Step 7 — Close out the loop.** Rejected samples reappear as new **Annotate** tasks (the remaining count grows back). Re-label them, send them through review again, and clear the **Final review** task. When every stage reads `0 left`, the workflow flips to **Complete** — 10 / 10 samples at 100%. ![The dashboard with every task at 0 left and the workflow marked Complete (10/10, 100%)](https://cdn.voxel51.com/tutorial_fiftyone_annotation_workflows/data_engine_wf_complete.webp) ### Agentic labeling *FiftyOne Enterprise.* **Agentic Labeling** is a new annotation feature: train a labeling *agent* from a plain-language prompt plus a few example crops, test it, then run it across the dataset — so you can capture a concept that isn’t in your schema yet. Here we detect construction workers who are **not** wearing a hard hat. Load the [Voxel51 hard-hat detection dataset](https://huggingface.co/datasets/Voxel51/hard-hat-detection) from the [Hugging Face Hub](https://docs.voxel51.com/integrations/huggingface.html). Its `ground_truth` field holds Helmet / Person / Head boxes, but there is no “worker without a hard hat” class — that is exactly what the agent will add. **Step 1 — Open the Agentic Labeler.** In the App, click the **+** beside the sample-grid tab and choose **Agentic Labeler**. The panel opens on the **Runs** / **Agents** tabs. Click **+ Train Agent**. ![The Agentic Labeler panel, showing the Runs and Agents tabs and the Train Agent button](https://cdn.voxel51.com/tutorial_fiftyone_annotation_workflows/data_engine_agentic_panel.webp) **Step 2 — Describe the labels.** In the **Train new agent** form, type the instruction in the **TEXT PROMPT** box — here, *Find site workers not wearing hard hats*. Under **SETTINGS → Task**, click the **Detection** card (bounding boxes). In the **Allowed classes** box, type `no_hardhat` and press Enter so the agent assigns exactly that one label. ![The Train new agent form: text prompt, the Detection task card, and the no_hardhat allowed class](https://cdn.voxel51.com/tutorial_fiftyone_annotation_workflows/data_engine_agentic_train.webp) **Step 3 — Add visual prompts.** Scroll to **VISUAL PROMPTS** and give the agent examples. Under **Positive**, click **Pick from grid** and select five crops that *do* show the concept (workers with no hard hat). Under **Negative**, select five crops that *do not* (workers wearing one). ![Selecting five positive and five negative example crops as visual prompts](https://cdn.voxel51.com/tutorial_fiftyone_annotation_workflows/data_engine_agentic_visual.webp) **Step 4 — Test, then save.** Click **Run test** to preview the prompt on five sample images. Set **Offload to field** to `predictions` (where the labels will land) and inspect the thumbnails — `no_hardhat` boxes where the agent fired, and *I did not find any bounding boxes* where it correctly abstained. When the previews look right, click **Save agent**. ![Test results on five samples, set to offload to the predictions field, with the Save agent button](https://cdn.voxel51.com/tutorial_fiftyone_annotation_workflows/data_engine_agentic_test.webp) **Step 5 — Run the agent.** Back on the **Runs** tab, click **New Run**, choose the saved agent and the samples to label, and start it. The run card reports progress and flips to **Completed** — here `no_hardhat_detector` labeled 60 samples with 60 labels. ![The Runs tab with the no_hardhat_detector run marked Completed: 60 samples, 60 labels](https://cdn.voxel51.com/tutorial_fiftyone_annotation_workflows/data_engine_agentic_run.webp) **Step 6 — Review the agent’s labels.** Open a sample to see the agent’s `predictions` (the blue `no_hardhat` boxes) alongside the original `ground_truth`. Toggle either field in the right sidebar, filter by confidence, and bulk-approve or correct — the same review loop you built above. ![A sample with no_hardhat predictions next to ground_truth, both fields listed in the sidebar](https://cdn.voxel51.com/tutorial_fiftyone_annotation_workflows/data_engine_agentic_results.webp) ### A scriptable first pass (open source) Draft labels from the SDK with [zoo models](https://docs.voxel51.com/model_zoo/models.html): **YOLO-World** for open-vocabulary detection, **SAM2** for masks. ## Stage 3 — Review: find label mistakes Rank labels by estimated error so you review the exceptions, not everything. ### Rank by mistakenness The Brain’s [mistakenness](https://docs.voxel51.com/brain.html#brain-label-mistakes) score (`compute_mistakenness()`) uses model predictions to flag how likely each label is to be wrong. > Mistakenness covers detections and classifications. For semantic segmentation, rank by per-sample mIoU from [evaluate_segmentations()](https://docs.voxel51.com/user_guide/evaluation.html#evaluating-segmentations). ![Filtering to high-mistakenness samples to surface likely label errors](https://cdn.voxel51.com/tutorial_fiftyone_annotation_workflows/data_engine_mistakenness.webp) ### Embedding-based review In the Embeddings panel, lasso outliers that sit far from their class cluster and send just those samples to the review queue. ### Compare against a reference model Where predictions and labels disagree, one is likely wrong. [Evaluate](https://docs.voxel51.com/user_guide/evaluation.html#evaluating-detections-coco) and sort by unmatched ground truth; the Enterprise **AI Insights** panel automates this. ### Fix mistakes Correct labels in the App, or tag suspects from the SDK and route them back through the workflow. ## Stage 4 — Train and evaluate: close the loop Train on the verified set, evaluate by scenario, and feed the weakest slice back into curation. ### Export and train Export training-ready labels in [YOLOv5 format](https://docs.voxel51.com/user_guide/export_datasets.html#yolov5). In Enterprise, run training as a **delegated operation** on GPU. ### Evaluate by scenario Compute [COCO mAP](https://docs.voxel51.com/user_guide/evaluation.html#evaluating-detections-coco) overall and per scene type to find the weakest slice. The Enterprise **Model Evaluation panel** does this interactively. ![The Model Evaluation panel: compare runs and slice performance by scenario](https://cdn.voxel51.com/tutorial_fiftyone_annotation_workflows/data_engine_model_eval.webp) ### Close the loop Mine more of the weakest scenario and save it as the next batch to annotate. ## Summary The loop, end to end: 1. **Curate** — embeddings, uniqueness, similarity, and zero-shot tagging picked the subset to label. 2. **Annotate** — an ontology fixed the schema; a multi-stage workflow moved samples through label → review → send-back; an agent and zoo models drafted the first pass. 3. **Review** — mistakenness, embedding outliers, and model comparison surfaced bad labels to fix. 4. **Evaluate** — scenario analysis named the weakest slice and fed it back into curation. ### Next steps - [FiftyOne Annotation](https://voxel51.com/annotation) - [FiftyOne Brain](https://docs.voxel51.com/brain.html) - [Dataset Zoo](https://docs.voxel51.com/dataset_zoo/index.html) - [Evaluating models](https://docs.voxel51.com/user_guide/evaluation.html)