Table of Contents
Contents

Dataset Views¶

FiftyOne provides methods that allow you to sort, slice, and search your Dataset using any information that you have added to the Dataset. Performing these actions returns a DatasetView into your Dataset that will show only the samples and labels therein that match your criteria.

Note

DatasetView does not hold its contents in-memory. Views simply store the rule(s) that are applied to extract the content of interest from the underlying Dataset when the view is iterated/aggregated on.

This means, for example, that the contents of a DatasetView may change as the underlying Dataset is modified.

Overview¶

A DatasetView is returned whenever any sorting, slicing, or searching operation is performed on a Dataset.

You can explicitly create a view that contains an entire dataset via Dataset.view():

1
2
3
4
5
6
7
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart")

view = dataset.view()

print(view)

Dataset:        quickstart
Media type:     image
Num samples:    200
Sample fields:
    id:               fiftyone.core.fields.ObjectIdField
    filepath:         fiftyone.core.fields.StringField
    tags:             fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)
    metadata:         fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.ImageMetadata)
    created_at:       fiftyone.core.fields.DateTimeField
    last_modified_at: fiftyone.core.fields.DateTimeField
    ground_truth:     fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)
    uniqueness:       fiftyone.core.fields.FloatField
    predictions:      fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)
View stages:
    ---

You can access specific information about a view in the natural ways:

1
2
3
4
5
len(view)
# 200

view.media_type
# "image"

Like datasets, you access the samples in a view by iterating over it:

1
2
for sample in view:
    # Do something with `sample`

Or, you can access individual samples in a view by their ID or filepath:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
sample = view.take(1).first()

print(type(sample))
# fiftyone.core.sample.SampleView

same_sample = view[sample.id]
also_same_sample = view[sample.filepath]

view[other_sample_id]
# KeyError: sample non-existent or not in view

Note

Accessing samples in a DatasetView returns SampleView objects, not Sample objects. The two classes are largely interchangeable, but SampleView provides some extra features. See filtering sample contents for more details.

Saving views¶

If you find yourself frequently using/recreating certain views, you can use save_view() to save them on your dataset under a name of your choice:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
import fiftyone as fo
import fiftyone.zoo as foz
from fiftyone import ViewField as F

dataset = foz.load_zoo_dataset("quickstart")
dataset.persistent = True

# Create a view
cats_view = (
    dataset
    .select_fields("ground_truth")
    .filter_labels("ground_truth", F("label") == "cat")
    .sort_by(F("ground_truth.detections").length(), reverse=True)
)

# Save the view
dataset.save_view("cats-view", cats_view)

Then you can conveniently use load_saved_view() to load the view in a future session:

1
2
3
4
5
6
7
import fiftyone as fo

dataset = fo.load_dataset("quickstart")

# Retrieve a saved view
cats_view = dataset.load_saved_view("cats-view")
print(cats_view)

Note

Did you know? You can also save, load, and edit saved views directly from the App!

Saved views have certain editable metadata such as a description that you can view via get_saved_view_info() and update via update_saved_view_info():

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
# Get a saved view's editable info
print(dataset.get_saved_view_info("cats-view"))

# Update the saved view's name and add a description
info = dict(
    name="still-cats-view",
    description="a view that only contains cats",
)
dataset.update_saved_view_info("cats-view", info)

# Verify that the info has been updated
print(dataset.get_saved_view_info("still-cats-view"))

You can also use list_saved_views(), has_saved_view(), and delete_saved_view() to manage your saved views.

Note

Saved views only store the rule(s) used to extract content from the underlying dataset, not the actual content itself. Saving views is cheap. Don’t worry about storage space!

Keep in mind, though, that the contents of a saved view may change as the underlying dataset is modified. For example, if a save view contains samples with a certain tag, the view’s contents will change as you add/remove this tag from samples.

View stages¶

Dataset views encapsulate a pipeline of logical operations that determine which samples appear in the view (and perhaps what subset of their contents).

Each view operation is captured by a ViewStage:

1
2
3
# List available view operations on a dataset
print(dataset.list_view_stages())
# ['exclude', 'exclude_fields', 'exists', ..., 'skip', 'sort_by', 'take']

These operations are conveniently exposed as methods on Dataset instances, in which case they create an initial DatasetView:

1
2
3
4
5
# Random set of 100 samples from the dataset
random_view = dataset.take(100)

len(random_view)
# 100

They are also exposed on DatasetView instances, in which case they return another DatasetView with the operation appended to its internal pipeline so that multiple operations can be chained together:

1
2
# Sort `random_view` by filepath
sorted_random_view = random_view.sort_by("filepath")

The sections below discuss some interesting view stages in more detail. You can also refer to the fiftyone.core.stages module documentation for examples of using each stage.

Slicing¶

You can extract a range of Sample instances from a Dataset using skip() and limit() or, equivalently, by using array slicing:

1
2
3
4
5
# Skip the first 2 samples and take the next 3
range_view1 = dataset.skip(2).limit(3)

# Equivalently, using array slicing
range_view2 = dataset[2:5]

Samples can be accessed from views in all the same ways as for datasets. This includes using first() and last() to retrieve the first and last samples in a view, respectively, or accessing a sample directly from a DatasetView by its ID or filepath:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
view = dataset[10:100]

sample10 = view.first()
sample100 = view.last()

also_sample10 = view[sample10.id]
print(also_sample10.filepath == sample10.filepath)
# True

also_sample100 = view[sample100.filepath]
print(sample100.id == also_sample100.id)
# True

Note that, unlike datasets, SampleView objects are not singletons, since there are an infinite number of possible views into a particular Sample:

1
2
print(sample10 is also_sample10)
# False

Note

Accessing a sample by its integer index in a DatasetView is not allowed. The best practice is to lookup individual samples by ID or filepath, or use array slicing to extract a range of samples, and iterate over samples in a view.

view[0]
# KeyError: Accessing samples by numeric index is not supported.
# Use sample IDs, filepaths, slices, boolean arrays, or a boolean ViewExpression instead

You can also use boolean array indexing to create a DatasetView into a dataset or view given an array-like of bools defining the samples you wish to extract:

1
2
3
4
5
6
7
8
import numpy as np

# A boolean array encoding the samples to extract
bool_array = np.array(dataset.values("uniqueness")) > 0.7

view = dataset[bool_array]
print(len(view))
# 17

The above syntax is equivalent to the following select() statement:

1
2
3
4
5
6
import itertools

ids = itertools.compress(dataset.values("id"), bool_array)
view = dataset.select(ids)
print(len(view))
# 17

Note that, whenever possible, the above operations are more elegantly implemented using match filters:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
from fiftyone import ViewField as F

# ViewExpression defining the samples to match
expr = F("uniqueness") > 0.7

# Use a match() expression to define the view
view = dataset.match(expr)
print(len(view))
# 17

# Equivalent: using boolean expression indexing is allowed too
view = dataset[expr]
print(len(view))
# 17

Sorting¶

You can use sort_by() to sort the samples in a Dataset or DatasetView by a field of interest. The samples in the returned DatasetView can be sorted in ascending or descending order:

1
2
view = dataset.sort_by("filepath")
view = dataset.sort_by("filepath", reverse=True)

You can also sort by expressions!

1
2
3
4
5
6
7
from fiftyone import ViewField as F

# Sort by number of detections in `Detections` field `ground_truth`
view = dataset.sort_by(F("ground_truth.detections").length(), reverse=True)

print(len(view.first().ground_truth.detections))  # 39
print(len(view.last().ground_truth.detections))  # 0

Shuffling¶

The samples in a Dataset or DatasetView can be randomly shuffled using shuffle():

1
2
# Randomly shuffle the order of the samples in the dataset
view1 = dataset.shuffle()

An optional seed can be provided to make the shuffle deterministic:

1
2
3
4
5
6
7
8
9
# Randomly shuffle the samples in the dataset with a fixed seed

view2 = dataset.shuffle(seed=51)
print(view2.first().id)
# 5f31bbfcd0d78c13abe159b1

also_view2 = dataset.shuffle(seed=51)
print(also_view2.first().id)
# 5f31bbfcd0d78c13abe159b1

Random sampling¶

You can extract a random subset of the samples in a Dataset or DatasetView using take():

1
2
# Take 5 random samples from the dataset
view1 = dataset.take(5)

An optional seed can be provided to make the sampling deterministic:

1
2
3
4
5
6
7
8
9
# Take 5 random samples from the dataset with a fixed seed

view2 = dataset.take(5, seed=51)
print(view2.first().id)
# 5f31bbfcd0d78c13abe159b1

also_view2 = dataset.take(5, seed=51)
print(also_view2.first().id)
# 5f31bbfcd0d78c13abe159b1

Filtering¶

The real power of DatasetView is the ability to write your own search queries based on your data.

Querying samples¶

You can query for a subset of the samples in a dataset via the match() method. The syntax is:

1
match_view = dataset.match(expression)

where expression defines the matching expression to use to decide whether to include a sample in the view.

FiftyOne provides powerful ViewField and ViewExpression classes that allow you to use native Python operators to define your match expression. Simply wrap the target field of your sample in a ViewField and then apply comparison, logic, arithmetic or array operations to it to create a ViewExpression. You can use dot notation to refer to fields or subfields of the embedded documents in your samples. Any resulting ViewExpression that returns a boolean is a valid expression!

The code below shows a few examples. See the API reference for ViewExpression for a full list of supported operations.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
from fiftyone import ViewField as F

# Populate metadata on all samples
dataset.compute_metadata()

# Samples whose image is less than 48 KB
small_images_view = dataset.match(F("metadata.size_bytes") < 48 * 1024)

# Samples that contain at least one prediction with confidence above 0.99
# or whose label ifs "cat" or "dog"
match = (F("confidence") > 0.99) | (F("label").is_in(("cat", "dog")))
matching_view = dataset.match(
    F("predictions.detections").filter(match).length() > 0
)

Common filters¶

Convenience functions for common queries are also available.

Use the match_tags() method to match samples that have the specified tag(s) in their tags field:

1
2
3
4
5
# The validation split of the dataset
val_view = dataset.match_tags("validation")

# Union of the validation and test splits
val_test_view = dataset.match_tags(("validation", "test"))

Use exists() to only include samples for which a given Field exists and is not None:

1
2
# The subset of samples where predictions have been computed
predictions_view = dataset.exists("predictions")

Use select() and exclude() to restrict attention to or exclude samples from a view by their IDs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# Get the IDs of two random samples
sample_ids = [
    dataset.take(1).first().id,
    dataset.take(1).first().id,
]

# Include only samples with the given IDs in the view
selected_view = dataset.select(sample_ids)

# Exclude samples with the given IDs from the view
excluded_view = dataset.exclude(sample_ids)

Filtering sample contents¶

Dataset views can also be used to filter the contents of samples in the view. That’s why DatasetView instances return SampleView objects rather than Sample objects.

SampleView instances represent the content of your samples in all of the usual ways, with some important caveats:

  • If you modify the contents of a SampleView and then save() it, any changes that you made to the contents of the SampleView will be reflected in the database.

  • Sample views can exclude fields and filter elements of a field (e.g., omit certain detections from an array of detections in the sample). This means that SampleView instances need not contain all of the information in a sample.

  • Sample views are not singletons and thus you must explicitly reload() them in order to refresh their contents if the underlying sample has been modified elsewhere. However, extracting a SampleView from a DatasetView always returns the updated version of the sample’s contents.

You can use the select_fields() and exclude_fields() stages to select or exclude fields from the returned SampleView:

1
2
3
4
5
6
7
8
9
for sample in dataset.select_fields("ground_truth"):
    print(sample.id)            # OKAY: `id` is always available
    print(sample.ground_truth)  # OKAY: `ground_truth` was selected
    print(sample.predictions)   # AttributeError: `predictions` was not selected

for sample in dataset.exclude_fields("predictions"):
    print(sample.id)            # OKAY: `id` is always available
    print(sample.ground_truth)  # OKAY: `ground_truth` was not excluded
    print(sample.predictions)   # AttributeError: `predictions` was excluded

The filter_labels() stage is a powerful stage that allows you to filter the contents of Detections, Classifications, Polylines, and Keypoints fields, respectively.

Here are some self-contained examples for each task:

Classifications

Detections

Polylines

Keypoints

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
import fiftyone as fo
import fiftyone.zoo as foz
from fiftyone import ViewField as F

dataset = foz.load_zoo_dataset("imagenet-sample")

# Only include samples whose ground truth `label` is "slug" or "conch"
slug_conch_view = dataset.filter_labels(
    "ground_truth", (F("label") == "slug") | (F("label") == "conch")
)

session = fo.launch_app(view=slug_conch_view)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
import fiftyone as fo
import fiftyone.zoo as foz
from fiftyone import ViewField as F

dataset = foz.load_zoo_dataset("quickstart")

# Bboxes are in [top-left-x, top-left-y, width, height] format
bbox_area = F("bounding_box")[2] * F("bounding_box")[3]

# Only includes predictions whose bounding boxes have an area of at
# least 50% of the image, and only include samples with at least
# one prediction after filtering
large_boxes_view = dataset.filter_labels("predictions", bbox_area >= 0.5)

session = fo.launch_app(view=large_boxes_view)

Note

See the BDD100K dataset in the Dataset Zoo for download instructions.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
import fiftyone as fo
import fiftyone.zoo as foz
from fiftyone import ViewField as F

# The path to the source files that you manually downloaded
source_dir = "/path/to/dir-with-bdd100k-files"

dataset = foz.load_zoo_dataset(
    "bdd100k", split="validation", source_dir=source_dir
)

# Only include polylines that are filled (polygons, not polylines),
# and only include samples with at least one polygon after filtering
polygons_view = dataset.filter_labels("gt_polylines", F("filled") == True)

session = fo.launch_app(view=polygons_view)

Note

This example uses a Keypoint R-CNN model from the Model Zoo.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
import fiftyone as fo
import fiftyone.zoo as foz
from fiftyone import ViewField as F

dataset = foz.load_zoo_dataset("quickstart")

# Load a keypoint model
model = foz.load_zoo_model("keypoint-rcnn-resnet50-fpn-coco-torch")

# Grab a few samples that have people in them
person_view  = dataset.match(
    F("ground_truth.detections").map(F("label") == "person").length() > 0
).take(4)

person_view.apply_model(model, label_field="rcnn")

# Only include keypoints in the `rcnn_keypoints` field of each
# sample that have at least 10 vertices, and only include samples
# with at least one keypoint instance after filtering
many_points_view = dataset.filter_labels(
    "rcnn_keypoints", F("points").length() >= 10,
)

session = fo.launch_app(view=many_points_view)

You can also use the filter_field() stage to filter the contents of arbitrarily-typed fields:

1
2
# Remove tags from samples that don't include the "validation" tag
clean_tags_view = dataset.filter_field("tags", F().contains("validation"))

Note

When you create a DatasetView that contains filtered detections or classifications, the other labels are not removed from the source dataset, even if you save() a SampleView after modifying the filtered detections. This is because each label is updated individually, and other labels in the field are left unchanged.

view = dataset.filter_labels("predictions", ...)

for sample in view:
    predictions = sample.predictions

    # Modify the detections in the view
    for detection in predictions.detections:
        detection["new_field"] = True

    # Other detections in the `predictions` field of the samples that
    # did not appear in the `view` are not deleted or modified
    sample.save()

If you do want to delete data from your samples, assign a new value to the field:

view = dataset.filter_labels("predictions", ...)

for sample in view:
    sample.predictions = fo.Detections(...)

    # Existing detections in the `predictions` field of the samples
    # are deleted
    sample.save()

Grouping¶

You can use group_by() to dynamically group the samples in a collection by a specified field:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
import fiftyone as fo
import fiftyone.zoo as foz
from fiftyone import ViewField as F

dataset = foz.load_zoo_dataset("cifar10", split="test")

# Take 100 samples and group by ground truth label
view = dataset.take(100, seed=51).group_by("ground_truth.label")

print(view.media_type)  # group
print(len(view))  # 10

Note

Views generated by group_by() have media type group.

By default, the samples in each group are unordered, but you can provide the optional order_by and reverse arguments to group_by() to specify an ordering for the samples in each group:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
# Create an image dataset that contains one sample per frame of the
# `quickstart-video` dataset
dataset2 = (
    foz.load_zoo_dataset("quickstart-video")
    .to_frames(sample_frames=True)
    .clone()
)

print(len(dataset2))  # 1279

# Group by video ID and order each group by frame number
view2 = dataset2.group_by("sample_id", order_by="frame_number")

print(len(view2))  # 10
print(view2.values("frame_number"))
# [1, 1, 1, ..., 1]

sample_id = dataset2.take(1).first().sample_id
video = view2.get_dynamic_group(sample_id)

print(video.values("frame_number"))
# [1, 2, 3, ..., 120]

You can also group by an arbitrary expressions:

1
2
3
4
5
6
7
8
dataset3 = foz.load_zoo_dataset("quickstart")

# Group samples by the number of ground truth objects they contain
expr = F("ground_truth.detections").length()
view3 = dataset3.group_by(expr)

print(len(view3))  # 26
print(len(dataset3.distinct(expr)))  # 26

When you iterate over a dynamic grouped view, you get one example from each group. Like any other view, you can chain additional view stages to further refine the view’s contents:

1
2
3
4
5
# Sort the groups by label
sorted_view = view.sort_by("ground_truth.label")

for sample in sorted_view:
    print(sample.ground_truth.label)

airplane
automobile
bird
cat
deer
dog
frog
horse
ship
truck

In particular, you can use flatten() to unravel the samples in a dynamic grouped view back into a flat view:

1
2
3
4
5
6
# Unwind the sorted groups back into a flat collection
flat_sorted_view = sorted_view.flatten()

print(len(flat_sorted_view))  # 1000
print(flat_sorted_view.values("ground_truth.label"))
# ['airplane', 'airplane', 'airplane', ..., 'truck']

Note

Did you know? When you load dynamic group views in the App, the grid view shows the first example from each group, and you can click on any sample to open the modal and view all samples in the group.

You can use get_dynamic_group() to retrieve a view containing the samples with a specific group value of interest:

1
2
group = view.get_dynamic_group("horse")
print(len(group))  # 11

You can also use iter_dynamic_groups() to iterate over all groups in a dynamic group view:

1
2
for group in sorted_view.iter_dynamic_groups():
    print("%s: %d" % (group.first().ground_truth.label, len(group)))

airplane: 11
automobile: 10
bird: 8
cat: 12
deer: 6
dog: 7
frog: 10
horse: 11
ship: 12
truck: 13

Concatenating views¶

You can use concat() to concatenate views into the same dataset:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
import fiftyone as fo
import fiftyone.zoo as foz
from fiftyone import ViewField as F

dataset = foz.load_zoo_dataset("quickstart")

view1 = dataset.match(F("uniqueness") < 0.2)
view2 = dataset.match(F("uniqueness") > 0.7)

view = view1.concat(view2)

print(len(view) == len(view1) + len(view2))  # True

or you can use the equivalent + syntax sugar:

1
2
3
view = view1 + view2

print(len(view) == len(view1) + len(view2))  # True

Concatenating generated views such as patches and frames is also allowed:

1
2
3
4
5
6
7
8
gt_patches = dataset.to_patches("ground_truth")

patches1 = gt_patches[:10]
patches2 = gt_patches[-10:]

patches = patches1 + patches2

print(len(patches) == len(patches1) + len(patches2))  # True

as long as each view is derived from the same root generated view:

1
2
3
4
patches1 = dataset[:10].to_patches("ground_truth")
patches2 = dataset[-10:].to_patches("ground_truth")

patches = patches1 + patches2  # ERROR: not allowed

If you concatenate views that use select_fields() or exclude_fields() to manipulate the schema of individual views, the concatenated view will respect the schema of the first view in the chain:

1
2
3
4
5
6
7
view1 = dataset[:10].select_fields()
view2 = dataset[-10:]

view = view1 + view2

# Fields are omitted from `view2` to match schema of `view1`
print(view.last())

1
2
3
4
5
6
7
view1 = dataset[:10]
view2 = dataset[-10:].select_fields()

view = view1 + view2

# Missing fields from `view2` appear as `None` to match schema of `view1`
print(view.last())

Note that concat() will not prevent you from creating concatenated views that contain multiple (possibly filtered) versions of the same Sample, which results in views that contains duplicate sample IDs:

1
2
3
4
5
sample_id = dataset.first().id
view = (dataset + dataset).shuffle()

selected_view = view.select(sample_id)
print(len(selected_view))  # two samples have the same ID

Warning

The FiftyOne App is not designed to display views with duplicate sample IDs.

Date-based views¶

If your dataset contains date fields, you can construct dataset views that query/filter based on this information by simply writing the appropriate ViewExpression, using date, datetime and timedelta objects to define the required logic.

For example, you can use the match() stage to filter a dataset by date as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
from datetime import datetime, timedelta

import fiftyone as fo
from fiftyone import ViewField as F

dataset = fo.Dataset()
dataset.add_samples(
    [
        fo.Sample(
            filepath="image1.png",
            capture_date=datetime(2021, 8, 24, 1, 0, 0),
        ),
        fo.Sample(
            filepath="image2.png",
            capture_date=datetime(2021, 8, 24, 2, 0, 0),
        ),
        fo.Sample(
            filepath="image3.png",
            capture_date=datetime(2021, 8, 24, 3, 0, 0),
        ),
    ]
)

query_date = datetime(2021, 8, 24, 2, 1, 0)
query_delta = timedelta(minutes=30)

# Samples with capture date after 2021-08-24 02:01:00
view = dataset.match(F("capture_date") > query_date)
print(view)

# Samples with capture date within 30 minutes of 2021-08-24 02:01:00
view = dataset.match(abs(F("capture_date") - query_date) < query_delta)
print(view)

Note

As the example above demonstrates, ViewExpression instances may contain date, datetime and timedelta objects. Internally, subtracting two dates returns the number of milliseconds between them. Using timedelta allows these units to be abstracted away from the user.

Object patches¶

If your dataset contains label list fields like Detections or Polylines, then you can use to_patches() to create views that contain one sample per object patch in a specified label field of your dataset.

For example, you can extract patches for all ground truth objects in a detection dataset:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
import fiftyone as fo
import fiftyone.zoo as foz
from fiftyone import ViewField as F

dataset = foz.load_zoo_dataset("quickstart")

session = fo.launch_app(dataset)

# Convert to ground truth patches
gt_patches = dataset.to_patches("ground_truth")
print(gt_patches)

# View patches in the App
session.view = gt_patches

Dataset:     quickstart
Media type:  image
Num patches: 1232
Patch fields:
    id:               fiftyone.core.fields.ObjectIdField
    sample_id:        fiftyone.core.fields.ObjectIdField
    filepath:         fiftyone.core.fields.StringField
    tags:             fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)
    metadata:         fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.ImageMetadata)
    created_at:       fiftyone.core.fields.DateTimeField
    last_modified_at: fiftyone.core.fields.DateTimeField
    ground_truth:     fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detection)
View stages:
    1. ToPatches(field='ground_truth', config=None)

Note

You can pass the optional other_fields parameter to to_patches() to specify additional read-only sample-level fields that each patch should include from their parent samples.

Or, you could chain view stages to create a view that contains patches for a filtered set of predictions:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
# Now extract patches for confident person predictions
person_patches = (
    dataset
    .filter_labels(
        "predictions",
        (F("label") == "person") & (F("confidence") > 0.9)
    )
    .to_patches("predictions")
)
print(person_patches)

# View patches in the App
session.view = person_patches

Note

Did you know? You can convert to object patches view directly from the App!

Object patches views are just like any other dataset view in the sense that:

  • You can append view stages via the App view bar or views API

  • Any modifications to label tags that you make via the App’s tagging menu or via API methods like tag_labels() and untag_labels() will be reflected on the source dataset

  • Any modifications to the patch labels that you make by iterating over the contents of the view or calling set_values() or set_label_values() will be reflected on the source dataset

  • Calling save(), keep(), or keep_fields() on a patches view (typically one that contains additional view stages that filter or modify its contents) will sync any edits or deletions to the patch labels with the source dataset

However, because object patches views only contain a subset of the contents of a Sample from the source dataset, there are some differences compared to non-patch views:

  • Tagging or untagging patches (as opposed to their labels) will not affect the tags of the underlying Sample

  • Any edits that you make to sample-level fields of object patches views other than the field that defines the patches themselves will not be reflected on the source dataset

Note

Did you know? You can export object patches as classification datasets!

Evaluation patches¶

If you have run evaluation on predictions from an object detection model, then you can use to_evaluation_patches() to transform the dataset (or a view into it) into a new view that contains one sample for each true positive, false positive, and false negative example.

True positive examples will result in samples with both their ground truth and predicted fields populated, while false positive/negative examples will only have one of their corresponding predicted/ground truth fields populated, respectively.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart")

# Evaluate `predictions` w.r.t. labels in `ground_truth` field
dataset.evaluate_detections(
    "predictions", gt_field="ground_truth", eval_key="eval"
)

session = fo.launch_app(dataset)

# Convert to evaluation patches
eval_patches = dataset.to_evaluation_patches("eval")
print(eval_patches)

print(eval_patches.count_values("type"))
# {'fn': 246, 'fp': 4131, 'tp': 986}

# View patches in the App
session.view = eval_patches

Dataset:     quickstart
Media type:  image
Num patches: 5363
Patch fields:
    id:               fiftyone.core.fields.ObjectIdField
    sample_id:        fiftyone.core.fields.ObjectIdField
    filepath:         fiftyone.core.fields.StringField
    tags:             fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)
    metadata:         fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.ImageMetadata)
    created_at:       fiftyone.core.fields.DateTimeField
    last_modified_at: fiftyone.core.fields.DateTimeField
    predictions:      fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)
    ground_truth:     fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)
    type:             fiftyone.core.fields.StringField
    iou:              fiftyone.core.fields.FloatField
    crowd:            fiftyone.core.fields.BooleanField
View stages:
    1. ToEvaluationPatches(eval_key='eval', config=None)

Note

You can pass the optional other_fields parameter to to_patches() to specify additional read-only sample-level fields that each patch should include from their parent samples.

Refer to the evaluation guide guide for more information about running evaluations and using evaluation patches views to analyze object detection models.

Note

Did you know? You can convert to evaluation patches view directly from the App!

Evaluation patches views are just like any other dataset view in the sense that:

  • You can append view stages via the App view bar or views API

  • Any modifications to ground truth or predicted label tags that you make via the App’s tagging menu or via API methods like tag_labels() and untag_labels() will be reflected on the source dataset

  • Any modifications to the predicted or ground truth Label elements in the patches view that you make by iterating over the contents of the view or calling set_values() or set_label_values() will be reflected on the source dataset

  • Calling save(), keep(), or keep_fields() on an evaluation patches view (typically one that contains additional view stages that filter or modify its contents) will sync any predicted or ground truth Label edits or deletions with the source dataset

However, because evaluation patches views only contain a subset of the contents of a Sample from the source dataset, there are some differences compared to non-patch views:

  • Tagging or untagging patches themselves (as opposed to their labels) will not affect the tags of the underlying Sample

  • Any edits that you make to sample-level fields of evaluation patches views other than the ground truth/predicted label fields will not be reflected on the source dataset

Video views¶

Most view stages naturally support video datasets. For example, stages that refer to fields can be applied to the frame-level fields of video samples by prepending "frames." to the relevant parameters:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
import fiftyone as fo
import fiftyone.zoo as foz
from fiftyone import ViewField as F

dataset = foz.load_zoo_dataset("quickstart-video")

# Create a view that only contains vehicles
view = dataset.filter_labels("frames.detections", F("label") == "vehicle")

# Compare the number of objects in the view and the source dataset
print(dataset.count("frames.detections.detections"))  # 11345
print(view.count("frames.detections.detections"))  # 7511

In addition, FiftyOne provides a variety of dedicated view stages for performing manipulations that are unique to video data.

Clip views¶

You can use to_clips() to create views into your video datasets that contain one sample per clip defined by a specific field or expression in a video collection.

For example, if you have temporal detection labels on your dataset, then you can create a clips view that contains one sample per temporal segment by simply passing the name of the temporal detection field to to_clips():

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
import fiftyone as fo

dataset = fo.Dataset()

sample1 = fo.Sample(
    filepath="video1.mp4",
    events=fo.TemporalDetections(
        detections=[
            fo.TemporalDetection(label="meeting", support=[1, 3]),
            fo.TemporalDetection(label="party", support=[2, 4]),
        ]
    ),
)

sample2 = fo.Sample(
    filepath="video2.mp4",
    metadata=fo.VideoMetadata(total_frame_count=5),
    events=fo.TemporalDetections(
        detections=[
            fo.TemporalDetection(label="party", support=[1, 3]),
            fo.TemporalDetection(label="meeting", support=[3, 5]),
        ]
    ),
)

dataset.add_samples([sample1, sample2])

# Create a clips view with one clip per event
view = dataset.to_clips("events")
print(view)

# Verify that one sample per clip was created
print(dataset.count("events.detections"))  # 4
print(len(view))  # 4

Dataset:    2021.09.03.09.44.57
Media type: video
Num clips:  4
Clip fields:
    id:               fiftyone.core.fields.ObjectIdField
    sample_id:        fiftyone.core.fields.ObjectIdField
    filepath:         fiftyone.core.fields.StringField
    support:          fiftyone.core.fields.FrameSupportField
    tags:             fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)
    metadata:         fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.VideoMetadata)
    created_at:       fiftyone.core.fields.DateTimeField
    last_modified_at: fiftyone.core.fields.DateTimeField
    events:           fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Classification)
Frame fields:
    id:               fiftyone.core.fields.ObjectIdField
    frame_number:     fiftyone.core.fields.FrameNumberField
    created_at:       fiftyone.core.fields.DateTimeField
    last_modified_at: fiftyone.core.fields.DateTimeField
View stages:
    1. ToClips(field_or_expr='events', config=None)

All clips views contain a top-level support field that contains the [first, last] frame range of the clip within filepath, which points to the source video.

Note that the events field, which had type TemporalDetections in the source dataset, now has type Classification in the clips view, since each classification has a one-to-one relationship with its clip.

Note

You can pass the optional other_fields parameter to to_clips() to specify additional read-only sample-level fields that each clip should include from their parent samples.

Note

If you edit the support or Classification of a sample in a clips view created from temporal detections, the changes will be applied to the corresponding TemporalDetection in the source dataset.

Continuing from the example above, if you would like to see clips only for specific temporal detection labels, you can achieve this by first filtering the labels:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
from fiftyone import ViewField as F

# Create a clips view with one clip per meeting
view = (
    dataset
    .filter_labels("events", F("label") == "meeting")
    .to_clips("events")
)

print(view.values("events.label"))
# ['meeting', 'meeting']

Clips views can also be created based on frame-level labels, which provides a powerful query language that you can use to find segments of a video dataset that contain specific frame content of interest.

In the simplest case, you can provide the name of a frame-level list field (e.g., Classifications or Detections) to to_clips(), which will create one clip per contiguous range of frames that contain at least one label in the specified field:

1
2
3
4
5
6
7
8
9
import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart-video")

# Create a view that contains one clip per contiguous range of frames that
# contains at least one detection
view = dataset.to_clips("frames.detections")
print(view)

The above view turns out to not be very interesting, since every frame in the quickstart-video dataset contains at least one object. So, instead, lets first filter the objects so that we can construct a clips view that contains one clip per contiguous range of frames that contains at least one person:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
from fiftyone import ViewField as F

# Create a view that contains one clip per contiguous range of frames that
# contains at least one person
view = (
    dataset
    .filter_labels("frames.detections", F("label") == "person")
    .to_clips("frames.detections")
)
print(view)

Dataset:    quickstart-video
Media type: video
Num clips:  8
Clip fields:
    id:               fiftyone.core.fields.ObjectIdField
    sample_id:        fiftyone.core.fields.ObjectIdField
    filepath:         fiftyone.core.fields.StringField
    support:          fiftyone.core.fields.FrameSupportField
    tags:             fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)
    metadata:         fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.VideoMetadata)
    created_at:       fiftyone.core.fields.DateTimeField
    last_modified_at: fiftyone.core.fields.DateTimeField
Frame fields:
    id:               fiftyone.core.fields.ObjectIdField
    frame_number:     fiftyone.core.fields.FrameNumberField
    created_at:       fiftyone.core.fields.DateTimeField
    last_modified_at: fiftyone.core.fields.DateTimeField
    detections:       fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)
View stages:
    1. FilterLabels(field='frames.detections', filter={'$eq': ['$$this.label', 'person']}, only_matches=True)
    2. ToClips(field_or_expr='frames.detections', config=None)

When you iterate over the frames of a sample in a clip view, you will only get the frames within the [first, last] support of each clip:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
sample = view.last()

print(sample.support)
# [116, 120]

frame_numbers = []
for frame_number, frame in sample.frames.items():
    frame_numbers.append(frame_number)

print(frame_numbers)
# [116, 117, 118, 119, 120]

Note

Clips views created via to_clips() always contain all frame-level labels from the underlying dataset for their respective frame supports, even if frame-level filtering was applied in previous view stages. In other words, filtering prior to the to_clips() stage only affects the frame supports.

You can, however, apply frame-level filtering to clips by appending filtering operations after the to_clips() stage in your view, just like any other view.

More generally, you can provide an arbitrary ViewExpression to to_clips() that defines a boolean expression to apply to each frame. In this case, the clips view will contain one clip per contiguous range of frames for which the expression evaluates to true:

1
2
3
4
5
6
7
8
# Create a view that contains one clip per contiguous range of frames that
# contains at least 10 vehicles
view = (
    dataset
    .filter_labels("frames.detections", F("label") == "vehicle")
    .to_clips(F("detections.detections").length() >= 10)
)
print(view)

See this section for more information about constructing frame expressions.

Note

You can pass optional tol and min_len parameters to to_clips() to configure a missing frame tolerance and minimum length for clips generated from frame-level fields or expressions.

Clip views are just like any other dataset view in the sense that:

  • You can append view stages via the App view bar or views API

  • Any modifications to label tags that you make via the App’s tagging menu or via API methods like tag_labels() and untag_labels() will be reflected on the source dataset

  • Any modifications to the frame-level labels in a clips view that you make by iterating over the contents of the view or calling set_values() or set_label_values() will be reflected on the source dataset

  • Calling save(), keep(), or keep_fields() on a clips view (typically one that contains additional view stages that filter or modify its contents) will sync any frame-level edits or deletions with the source dataset

However, because clip views represent only a subset of a Sample from the source dataset, there are some differences compared to non-clip views:

  • Tagging or untagging clips (as opposed to their labels) will not affect the tags of the underlying Sample

  • Any edits that you make to sample-level fields of clip views will not be reflected on the source dataset (except for edits to the support and Classification field populated when generating clip views based on TemporalDetection labels, as described above)

Trajectory views¶

You can use to_trajectories() to create views into your video datasets that contain one sample per each unique object trajectory defined by their (label, index) in a frame-level Detections or Polylines field.

Trajectory views are a special case of clip views where each clip has been filtered to contain only the identifying object, rather than than all objects with the trajectory’s frame support.

For example, if you have frame-level object detections with their index attributes populated, then you can create a trajectories view that contains one clip for each object of a specific type using filter_labels() and to_trajectories() as shown below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
import fiftyone as fo
import fiftyone.zoo as foz
from fiftyone import ViewField as F

dataset = foz.load_zoo_dataset("quickstart-video")

# Create a trajectories view for the vehicles in the dataset
trajectories = (
    dataset
    .filter_labels("frames.detections", F("label") == "vehicle")
    .to_trajectories("frames.detections")
)
print(trajectories)

session = fo.launch_app(view=trajectories)

Dataset:    quickstart-video
Media type: video
Num clips:  109
Clip fields:
    id:               fiftyone.core.fields.ObjectIdField
    sample_id:        fiftyone.core.fields.ObjectIdField
    filepath:         fiftyone.core.fields.StringField
    support:          fiftyone.core.fields.FrameSupportField
    tags:             fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)
    metadata:         fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.VideoMetadata)
    created_at:       fiftyone.core.fields.DateTimeField
    last_modified_at: fiftyone.core.fields.DateTimeField
    detections:       fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.odm.embedded_document.DynamicEmbeddedDocument)
Frame fields:
    id:               fiftyone.core.fields.ObjectIdField
    frame_number:     fiftyone.core.fields.FrameNumberField
    created_at:       fiftyone.core.fields.DateTimeField
    last_modified_at: fiftyone.core.fields.DateTimeField
    detections:       fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)
View stages:
    1. FilterLabels(field='frames.detections', filter={'$eq': ['$$this.label', 'vehicle']}, only_matches=True, trajectories=False)
    2. ToTrajectories(field='frames.detections', config=None)

Warning

Trajectory views can contain significantly more frames than their source collection, since the number of frames is now O(# boxes) rather than O(# video frames).

Frame views¶

You can use to_frames() to create image views into your video datasets that contain one sample per frame in the dataset.

Note

Did you know? Using to_frames() enables you to execute workflows such as model evaluation and Brain methods that only support image collections to the frames of your video datasets!

In the simplest case, you can create a view that contains a sample for every frame of the videos in a Dataset or DatasetView:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart-video")

session = fo.launch_app(dataset)

# Create a frames view for the entire dataset
frames = dataset.to_frames(sample_frames=True)
print(frames)

# Verify that one sample per frame was created
print(dataset.sum("metadata.total_frame_count"))  # 1279
print(len(frames))  # 1279

# View frames in the App
session.view = frames

Dataset:     quickstart-video
Media type:  image
Num samples: 1279
Sample fields:
    id:               fiftyone.core.fields.ObjectIdField
    sample_id:        fiftyone.core.fields.ObjectIdField
    filepath:         fiftyone.core.fields.StringField
    frame_number:     fiftyone.core.fields.FrameNumberField
    tags:             fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)
    metadata:         fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.ImageMetadata)
    created_at:       fiftyone.core.fields.DateTimeField
    last_modified_at: fiftyone.core.fields.DateTimeField
    detections:       fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)
View stages:
    1. ToFrames(config=None)

The above example passes the sample_frames=True option to to_frames(), which causes the necessary frames of the input video collection to be sampled into directories of per-frame images on disk when the view is created. For large video datasets, this may take some time and require substantial disk space. The paths to each frame image will also be stored in a filepath field of each Frame of the source collection.

Note that, when using the sample_frames=True option, frames that have previously been sampled will not be resampled, so creating frame views into the same dataset will become faster after the frames have been sampled.

Note

The recommended way to use to_frames() is to first populate the filepath field of each Frame of your dataset offline, either by running it once with the sample_frames=True option or by manually sampling the frames yourself and populating the filepath frame field.

Then you can work with frame views efficiently via the default syntax:

# Creates a view with one sample per frame whose `filepath` is set
frames = dataset.to_frames()

More generally, to_frames() exposes a variety of parameters that you can use to configure the behavior of the video-to-image conversion process. You can also combine to_frames() with view stages like match_frames() to achieve fine-grained control over the specific frames you want to study.

For example, the snippet below creates a frames view that only contains samples for frames with at least 10 objects, sampling at most one frame per second:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
from fiftyone import ViewField as F

#
# Create a frames view that only contains frames with at least 10
# objects, sampled at a maximum frame rate of 1fps
#

num_objects = F("detections.detections").length()
view = dataset.match_frames(num_objects > 10)

frames = view.to_frames(max_fps=1)
print(frames)

# Compare the number of frames in each step
print(dataset.count("frames"))  # 1279
print(view.count("frames"))  # 354
print(len(frames))  # 13

# View frames in the App
session.view = frames

Frame views inherit all frame-level labels from the source video dataset, including their frame number. Each frame sample is also given a sample_id field that records the ID of the parent video sample, and any tags of the parent video sample are also included.

Frame views are just like any other image collection view in the sense that:

  • You can append view stages via the App view bar or views API

  • Any modifications to label tags that you make via the App’s tagging menu or via API methods like tag_labels() and untag_labels() will be reflected on the source dataset

  • Any edits (including additions, modifications, and deletions) to the fields of the samples in a frames view that you make by iterating over the contents of the view or calling set_values() or set_label_values() will be reflected on the source dataset

  • Calling save(), keep(), or keep_fields() on a frames view (typically one that contains additional view stages that filter or modify its contents) will sync any changes to the frames of the underlying video dataset

The only way in which frames views differ from regular image collections is that changes to the tags or metadata fields of frame samples will not be propagated to the frames of the underlying video dataset.

Frame patches views¶

Since frame views into video datasets behave just like any other view, you can chain to_frames() and to_patches() to create frame patch views into your video datasets that contain one sample per object patch in the frames of the dataset!

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart-video")

session = fo.launch_app(dataset)

# Create a frames view
frames = dataset.to_frames(sample_frames=True)

# Create a frame patches view
frame_patches = frames.to_patches("detections")
print(frame_patches)

# Verify that one sample per object was created
print(dataset.count("frames.detections.detections"))  # 11345
print(len(frame_patches))  # 11345

# View frame patches in the App
session.view = frame_patches

Dataset:     quickstart-video
Media type:  image
Num patches: 11345
Patch fields:
    id:               fiftyone.core.fields.ObjectIdField
    sample_id:        fiftyone.core.fields.ObjectIdField
    frame_id:         fiftyone.core.fields.ObjectIdField
    filepath:         fiftyone.core.fields.StringField
    frame_number:     fiftyone.core.fields.FrameNumberField
    tags:             fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)
    metadata:         fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.ImageMetadata)
    created_at:       fiftyone.core.fields.DateTimeField
    last_modified_at: fiftyone.core.fields.DateTimeField
    detections:       fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detection)
View stages:
    1. ToFrames(config=None)
    2. ToPatches(field='detections', config=None)

Querying frames¶

You can query for a subset of the frames in a video dataset via match_frames(). The syntax is:

1
match_view = dataset.match_frames(expression)

where expression defines the matching expression to use to decide whether to include a frame in the view.

FiftyOne provides powerful ViewField and ViewExpression classes that allow you to use native Python operators to define your match expression. Simply wrap the target frame field in a ViewField and then apply comparison, logic, arithmetic or array operations to it to create a ViewExpression. You can use dot notation to refer to fields or subfields of the embedded documents in your frames. Any resulting ViewExpression that returns a boolean is a valid expression!

The snippet below demonstrates a possible workflow. See the API reference for ViewExpression for a full list of supported operations.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
import fiftyone as fo
import fiftyone.zoo as foz
from fiftyone import ViewField as F

dataset = foz.load_zoo_dataset("quickstart-video")

# Create a view that only contains frames with at least 10 objects
num_objects = F("detections.detections").length()
view = dataset.match_frames(num_objects > 10)

# Compare the number of frames in each collection
print(dataset.count("frames"))  # 1279
print(view.count("frames"))  # 354

You can also use select_frames() and exclude_frames() to restrict attention to or exclude frames from a view by their IDs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# Get the IDs of a couple frames
frame_ids = [
    dataset.first().frames.first().id,
    dataset.last().frames.last().id,
]

# Select only the specified frames
selected_view = dataset.select_frames(frame_ids)

# Exclude frames with the given IDs from the view
excluded_view = dataset.exclude_frames(frame_ids)

Similarity views¶

If your dataset is indexed by similarity, then you can use the sort_by_similarity() stage to programmatically query your data by similarity to image(s) or object patch(es) of interest.

Image similarity¶

The example below indexes a dataset by image similarity using compute_similarity() and then uses sort_by_similarity() to sort the dataset by similarity to a chosen image:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
import fiftyone as fo
import fiftyone.brain as fob
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart")

# Index the dataset by image similarity
fob.compute_similarity(dataset, brain_key="image_sim")

session = fo.launch_app(dataset)

# Select a random query image
query_id = dataset.take(1).first().id

# Sort the samples by similarity to the query image
view = dataset.sort_by_similarity(query_id, brain_key="image_sim")
print(view)

# View results in the App
session.view = view

Note

Refer to the Brain guide for more information about generating similarity indexes, and check out the App guide to see how to sort images by similarity via point-and-click in the App!

Object similarity¶

The example below indexes the objects in a Detections field of a dataset by similarity using compute_similarity() and then uses sort_by_similarity() to retrieve the 15 most similar objects to a chosen object:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
import fiftyone as fo
import fiftyone.brain as fob
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart")

# Index the dataset by `ground_truth` object similarity
fob.compute_similarity(
    dataset, patches_field="ground_truth", brain_key="gt_sim"
)

# Convert to ground truth patches view
patches = dataset.to_patches("ground_truth")

# View patches in the App
session = fo.launch_app(view=patches)

# Select a random query object
query_id = patches.take(1).first().id

# Retrieve the 15 most similar objects
similar_objects = patches.sort_by_similarity(query_id, k=15, brain_key="gt_sim")

# View results in the App
session.view = similar_objects

Note

Refer to the Brain guide for more information about generating similarity indexes, and check out the App guide to see how to sort objects by similarity via point-and-click in the App!

Text similarity¶

When you create a similarity index powered by the CLIP model, you can pass arbitrary natural language queries to sort_by_similarity() along with the brain_key of a compatible similarity index:

Image similarity

Object similarity

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
import fiftyone as fo
import fiftyone.brain as fob
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart")

# Index images by similarity
image_index = fob.compute_similarity(
    dataset,
    model="clip-vit-base32-torch",
    brain_key="img_sim",
)

session = fo.launch_app(dataset)

# Perform a text query
query = "kites high in the air"
view = dataset.sort_by_similarity(query, k=15, brain_key="img_sim")

session.view = view

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
import fiftyone as fo
import fiftyone.brain as fob
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart")

# Index ground truth objects by similarity
object_index = fob.compute_similarity(
    dataset,
    patches_field="ground_truth",
    model="clip-vit-base32-torch",
    brain_key="gt_sim",
)

session = fo.launch_app(dataset)

# Convert to patches view
patches = dataset.to_patches("ground_truth")

# Perform a text query
query = "cute puppies"
view = patches.sort_by_similarity(query, k=15, brain_key="gt_sim")

session.view = view

You can verify that a similarity index supports text queries by checking that it supports_prompts:

1
2
info = dataset.get_brain_info(brain_key)
print(info.config.supports_prompts)  # True

Note

Refer to the Brain guide for more information about generating similarity indexes, and check out the App guide to see how to sort objects by text similarity via point-and-click in the App!

Geolocation¶

If your samples have geolocation data, then you can use the geo_near() and geo_within() stages to filter your data based on their location.

For example, you can use geo_near() to sort your samples by proximity to a location:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
import fiftyone as fo
import fiftyone.zoo as foz

TIMES_SQUARE = [-73.9855, 40.7580]

dataset = foz.load_zoo_dataset("quickstart-geo")

# Sort the samples by their proximity to Times Square, and only include
# samples within 5km
view = dataset.geo_near(TIMES_SQUARE, max_distance=5000)

Or, you can use geo_within() to only include samples that lie within a longitude-latitude polygon of your choice:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
import fiftyone as fo
import fiftyone.zoo as foz

MANHATTAN = [
    [
        [-73.949701, 40.834487],
        [-73.896611, 40.815076],
        [-73.998083, 40.696534],
        [-74.031751, 40.715273],
        [-73.949701, 40.834487],
    ]
]

dataset = foz.load_zoo_dataset("quickstart-geo")

# Only contains samples in Manhattan
view = dataset.geo_within(MANHATTAN)

Tagging contents¶

You can use the tag_samples() and untag_samples() methods to add or remove sample tags from the samples in a view:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
import fiftyone as fo
import fiftyone.zoo as foz
from fiftyone import ViewField as F

dataset = foz.load_zoo_dataset("quickstart")
dataset.untag_samples("validation") # remove pre-existing tags

# Perform a random 90-10 test-train split
dataset.take(0.1 * len(dataset)).tag_samples("test")
dataset.match_tags("test", bool=False).tag_samples("train")

print(dataset.count_sample_tags())
# {'train': 180, 'test': 20}

You can also use the tag_labels() and untag_labels() methods to add or remove label tags from the labels in one or more fields of a view:

1
2
3
4
5
6
# Add a tag to all low confidence predictions
view = dataset.filter_labels("predictions", F("confidence") < 0.06)
view.tag_labels("low_confidence", label_fields="predictions")

print(dataset.count_label_tags())
# {'low_confidence': 447}

Editing fields¶

You can perform arbitrary edits to a DatasetView by iterating over its contents and editing the samples directly:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
import random

import fiftyone as fo
import fiftyone.zoo as foz
from fiftyone import ViewField as F

dataset = foz.load_zoo_dataset("quickstart")

view = dataset.limit(50)

# Populate a new field on each sample in the view
for sample in view:
    sample["random"] = random.random()
    sample.save()

print(dataset.count("random"))  # 50
print(dataset.bounds("random")) # (0.0005, 0.9928)

However, the above pattern can be inefficient for large views because each sample.save() call makes a new connection to the database.

The iter_samples() method provides an autosave=True option that causes all changes to samples emitted by the iterator to be automatically saved using an efficient batch update strategy:

1
2
3
# Automatically saves sample edits in efficient batches
for sample in view.select_fields().iter_samples(autosave=True):
    sample["random"] = random.random()

Note

As the above snippet shows, you should also optimize your iteration by selecting only the required fields.

You can configure the default batching strategy that is used via your FiftyOne config, or you can configure the batching strategy on a per-method call basis by passing the optional batch_size and batching_strategy arguments to iter_samples().

You can also use the save_context() method to perform batched edits using the pattern below:

1
2
3
4
5
# Use a context to save sample edits in efficient batches
with view.save_context() as context:
    for sample in view.select_fields():
        sample["random"] = random.random()
        context.save(sample)

The benefit of the above approach versus passing autosave=True to iter_samples() is that context.save() allows you to be explicit about which samples you are editing, which avoids unnecessary computations if your loop only edits certain samples.

Another strategy for performing efficient batch edits is to use set_values() to set a field (or embedded field) on each sample in the collection in a single batch operation:

1
2
3
4
5
6
7
8
9
# Delete the field we added earlier
dataset.delete_sample_field("random")

# Equivalent way to populate a new field on each sample in a view
values = [random.random() for _ in range(len(view))]
view.set_values("random", values)

print(dataset.count("random"))  # 50
print(dataset.bounds("random")) # (0.0272, 0.9921)

Note

When possible, using set_values() is often more efficient than performing the equivalent operation via an explicit iteration over the DatasetView because it avoids the need to read SampleView instances into memory and sequentially save them.

Naturally, you can edit nested sample fields of a DatasetView by iterating over the view and editing the necessary data:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
# Create a view that contains only low confidence predictions
view = dataset.filter_labels("predictions", F("confidence") < 0.06)

# Add a tag to all predictions in the view
for sample in view:
    for detection in sample["predictions"].detections:
        detection.tags.append("low_confidence")

    sample.save()

print(dataset.count_label_tags())
# {'low_confidence': 447}

However, an equivalent and often more efficient approach is to use values() to extract the slice of data you wish to modify and then use set_values() to save the updated data in a single batch operation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
# Remove the tags we added in the previous variation
dataset.untag_labels("low_confidence")

# Load all predicted detections
# This is a list of lists of `Detection` instances for each sample
detections = view.values("predictions.detections")

# Add a tag to all low confidence detections
for sample_detections in detections:
    for detection in sample_detections:
        detection.tags.append("low_confidence")

# Save the updated predictions
view.set_values("predictions.detections", detections)

print(dataset.count_label_tags())
# {'low_confidence': 447}

In the particular case of updating the attributes of a Label field of your dataset, the edits may be most naturally represented as a mapping between label IDs and corresponding attribute values to set on each label. In such cases, you can use set_label_values() to conveniently perform the updates:

1
2
3
4
5
6
7
8
9
# Grab the IDs of all labels in `view`
label_ids = view.values("predictions.detections.id", unwind=True)

# Populate an `is_low_conf` attribute on all of the labels
values = {_id: True for _id in label_ids}
dataset.set_label_values("predictions.detections.is_low_conf", values)

print(dataset.count_values("predictions.detections.is_low_conf"))
# {True: 447, None: 5173}

Transforming fields¶

In certain situations, you may wish to temporarily modify the values of sample fields in the context of a DatasetView without modifying the underlying dataset. FiftyOne provides the set_field() and map_labels() methods for this purpose.

For example, suppose you would like to rename a group of labels to a single category in order to run your evaluation routine. You can use map_labels() to do this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
import fiftyone as fo
import fiftyone.zoo as foz
from fiftyone import ViewField as F

dataset = foz.load_zoo_dataset("quickstart")

ANIMALS = [
    "bear", "bird", "cat", "cow", "dog", "elephant", "giraffe",
    "horse", "sheep", "zebra"
]

# Replace all animal detection's labels with "animal"
mapping = {k: "animal" for k in ANIMALS}
animals_view = dataset.map_labels("predictions", mapping)

counts = animals_view.count_values("predictions.detections.label")
print(counts["animal"])
# 529

Or, suppose you would like to lower bound all confidences of objects in the predictions field of a dataset. You can use set_field() to do this:

1
2
3
4
5
6
7
8
# Lower bound all confidences in the `predictions` field to 0.5
bounded_view = dataset.set_field(
    "predictions.detections.confidence",
    F("confidence").max(0.5),
)

print(bounded_view.bounds("predictions.detections.confidence"))
# (0.5, 0.9999035596847534)

Note

In order to populate a new field using set_field(), you must first declare the new field on the dataset via add_sample_field():

# Record the number of predictions in each sample in a new field
dataset.add_sample_field("num_predictions", fo.IntField)
view = dataset.set_field("num_predictions", F("predictions.detections").length())

view.save("num_predictions")  # save the new field's values on the dataset
print(dataset.bounds("num_predictions"))  # (1, 100)

The ViewExpression language is quite powerful, allowing you to define complex operations without needing to write an explicit Python loop to perform the desired manipulation.

For example, the snippet below visualizes the top-5 highest confidence predictions for each sample in the dataset:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
from fiftyone import ViewField as F

# Extracts the 5 highest confidence predictions for each sample
top5_preds = F("detections").sort("confidence", reverse=True)[:5]

top5_view = (
    dataset
    .set_field("predictions.detections", top5_preds)
    .select_fields("predictions")
)

session = fo.launch_app(view=top5_view)

If you want to permanently save transformed view fields to the underlying dataset, you can do so by calling save() on the view and optionally passing the name(s) of specific field(s) that you want to save:

1
2
# Saves `predictions` field's contents in the view permanently to dataset
top5_view.save("predictions")

Saving and cloning¶

Ordinarily, when you define a DatasetView that extracts a specific subset of a dataset and its fields, the underlying Dataset is not modified. However, you can use save() to save the contents of a view you’ve created to the underlying dataset:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
import fiftyone as fo
import fiftyone.zoo as foz
from fiftyone import ViewField as F

dataset = foz.load_zoo_dataset("quickstart")

# Capitalize some labels
rename_view = dataset.map_labels("predictions", {"cat": "CAT", "dog": "DOG"})
rename_view.save()

print(dataset.count_values("predictions.detections.label"))
# {'CAT': 35, 'DOG': 49, ...}

# Discard all predictions with confidence below 0.3
high_conf_view = dataset.filter_labels(
    "predictions", F("confidence") > 0.3, only_matches=False
)
high_conf_view.save()

print(dataset.bounds("predictions.detections.confidence"))
# (0.3001, 0.9999)

Note that calling save() on a DatasetView will only save modifications to samples that are in the view; all other samples are left unchanged.

You can use keep() to delete samples from the underlying dataset that do not appear in a view you created:

1
2
3
4
5
6
7
8
9
print(len(dataset))
# 200

# Discard all samples with no people
people_view = dataset.filter_labels("ground_truth", F("label") == "person")
people_view.keep()

print(len(dataset))
# 94

and you can use keep_fields() to delete any sample/frame fields from the underlying dataset that you have excluded from a view you created:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# Delete the `predictions` field
view = dataset.exclude_fields("predictions")
view.keep_fields()

print(dataset)

# Delete all non-default fields
view = dataset.select_fields()
view.keep_fields()

print(dataset)

Alternatively, you can use clone() to create a new Dataset that contains a copy of (only) the contents of a DatasetView:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
# Reload full quickstart dataset
dataset.delete()
dataset = foz.load_zoo_dataset("quickstart")

# Create a new dataset that contains only the high confidence predictions
high_conf_view = dataset.filter_labels("predictions", F("confidence") > 0.3)
high_conf_dataset = high_conf_view.clone()

print(high_conf_dataset.bounds("predictions.detections.confidence"))
# (0.3001, 0.9999)

You can also use clone_sample_field() to copy the contents of a view’s field into a new field of the underlying Dataset:

1
2
3
4
5
6
7
8
9
print(dataset.count("predictions.detections"))  # 5620

# Make view containing only high confidence predictions
view = dataset.filter_labels("predictions", F("confidence") > 0.5)
print(view.count("predictions.detections"))  # 1564

# Copy high confidence predictions to a new field
view.clone_sample_field("predictions", "high_conf_predictions")
print(dataset.count("high_conf_predictions.detections"))  # 1564

Tips & tricks¶

Chaining view stages¶

View stages can be chained together to perform complex operations:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
from fiftyone import ViewField as F

# Extract the first 5 samples with the "validation" tag, alphabetically by
# filepath, whose images are >= 48 KB
complex_view = (
    dataset
    .match_tags("validation")
    .exists("metadata")
    .match(F("metadata.size_bytes") >= 48 * 1024)  # >= 48 KB
    .sort_by("filepath")
    .limit(5)
)

Filtering detections by area¶

Need to filter your detections by bounding box area? Use this ViewExpression!

1
2
3
4
5
6
7
8
9
from fiftyone import ViewField as F

# Bboxes are in [top-left-x, top-left-y, width, height] format
bbox_area = F("bounding_box")[2] * F("bounding_box")[3]

# Only contains boxes whose area is between 5% and 50% of the image
medium_boxes_view = dataset.filter_labels(
    "predictions", (0.05 <= bbox_area) & (bbox_area < 0.5)
)

FiftyOne stores bounding box coordinates as relative values in [0, 1]. However, you can use the expression below to filter by absolute pixel area:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
from fiftyone import ViewField as F

dataset.compute_metadata()

# Computes the area of each bounding box in pixels
bbox_area = (
    F("$metadata.width") * F("bounding_box")[2] *
    F("$metadata.height") * F("bounding_box")[3]
)

# Only contains boxes whose area is between 32^2 and 96^2 pixels
medium_boxes_view = dataset.filter_labels(
    "predictions", (32 ** 2 < bbox_area) & (bbox_area < 96 ** 2)
)

Removing a batch of samples from a dataset¶

You can easily remove a batch of samples from a Dataset by constructing a DatasetView that contains the samples, and then deleting them from the dataset as follows:

1
2
3
4
# Choose 10 samples at random
unlucky_samples = dataset.take(10)

dataset.delete_samples(unlucky_samples)

Efficiently iterating samples¶

If you have a dataset with larger fields, such as Classifications or Detections, it can be expensive to load entire samples into memory. If, for a particular use case, you are only interested in a subset of fields, you can use Dataset.select_fields() to load only the fields of interest.

Let’s say you have a dataset that looks like this:

Name:        open-images-v4-test
Media type:  image
Num samples: 1000
Persistent:  True
Tags:        []
Sample fields:
    id:                       fiftyone.core.fields.ObjectIdField
    filepath:                 fiftyone.core.fields.StringField
    tags:                     fiftyone.core.fields.ListField(StringField)
    metadata:                 fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.ImageMetadata)
    created_at:               fiftyone.core.fields.DateTimeField
    last_modified_at:         fiftyone.core.fields.DateTimeField
    open_images_id:           fiftyone.core.fields.StringField
    groundtruth_image_labels: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Classifications)
    groundtruth_detections:   fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)
    faster_rcnn:              fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)
    mAP:                      fiftyone.core.fields.FloatField
    AP_per_class:             fiftyone.core.fields.DictField

and you want to get a list of open_images_id’s for all samples in the dataset. Loading other fields is unnecessary; in fact, using Dataset.select_fields() to load only the open_images_id field speeds up the operation below by ~200X!

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
import time

start = time.time()
oids = []
for sample in dataset:
    oids.append(sample.open_images_id)

print(time.time() - start)
# 38.212332010269165

start = time.time()
oids = []
for sample in dataset.select_fields("open_images_id"):
    oids.append(sample.open_images_id)

print(time.time() - start)
# 0.20824909210205078