Using FiftyOne Datasets¶
After a Dataset
has been loaded or created, FiftyOne provides powerful
functionality to inspect, search, and modify it from a Dataset
-wide down to
a Sample
level.
The following sections provide details of how to use various aspects of a
FiftyOne Dataset
.
Datasets¶
Instantiating a Dataset
object creates a new dataset.
1 2 3 4 5 | import fiftyone as fo dataset1 = fo.Dataset("my_first_dataset") dataset2 = fo.Dataset("my_second_dataset") dataset3 = fo.Dataset() # generates a default unique name |
Check to see what datasets exist at any time via list_datasets()
:
1 2 | print(fo.list_datasets()) # ['my_first_dataset', 'my_second_dataset', '2020.08.04.12.36.29'] |
Load a dataset using
load_dataset()
.
Dataset objects are singletons. Cool!
1 2 | _dataset2 = fo.load_dataset("my_second_dataset") _dataset2 is dataset2 # True |
If you try to load a dataset via Dataset(...)
or create a new dataset via
load_dataset()
you’re going to
have a bad time:
1 2 3 4 5 6 | _dataset2 = fo.Dataset("my_second_dataset") # Dataset 'my_second_dataset' already exists; use `fiftyone.load_dataset()` # to load an existing dataset dataset4 = fo.load_dataset("my_fourth_dataset") # DoesNotExistError: Dataset 'my_fourth_dataset' not found |
Dataset media type¶
The media type of a dataset is determined by the
media type of the Sample
objects that it contains.
The media_type
property of a
dataset is set based on the first sample added to it:
1 2 3 4 5 6 7 8 9 10 11 12 | import fiftyone as fo dataset = fo.Dataset() print(dataset.media_type) # None sample = fo.Sample(filepath="/path/to/image.png") dataset.add_sample(sample) print(dataset.media_type) # "image" |
Note that datasets are homogeneous; they must contain samples of the same media type (except for grouped datasets):
1 2 3 | sample = fo.Sample(filepath="/path/to/video.mp4") dataset.add_sample(sample) # MediaTypeError: Sample media type 'video' does not match dataset media type 'image' |
The following media types are available:
Media type |
Description |
---|---|
|
Datasets that contain images |
|
Datasets that contain videos |
|
Datasets that contain 3D scenes |
|
Datasets that contain point clouds |
|
Datasets that contain grouped data slices |
Dataset persistence¶
By default, datasets are non-persistent. Non-persistent datasets are deleted from the database each time the database is shut down. Note that FiftyOne does not store the raw data in datasets directly (only the labels), so your source files on disk are untouched.
To make a dataset persistent, set its
persistent
property to
True
:
1 2 | # Make the dataset persistent dataset1.persistent = True |
Without closing your current Python shell, open a new shell and run:
1 2 3 4 5 | import fiftyone as fo # Verify that both persistent and non-persistent datasets still exist print(fo.list_datasets()) # ['my_first_dataset', 'my_second_dataset', '2020.08.04.12.36.29'] |
All three datasets are still available, since the database connection has not been terminated.
However, if you exit all processes with fiftyone
imported, then open a new
shell and run the command again:
1 2 3 4 5 | import fiftyone as fo # Verify that non-persistent datasets have been deleted print(fo.list_datasets()) # ['my_first_dataset'] |
you’ll see that the my_second_dataset
and 2020.08.04.12.36.29
datasets have
been deleted because they were not persistent.
Dataset version¶
The version of the fiftyone
package for which a dataset is formatted is
stored in the version
property
of the dataset.
If you upgrade your fiftyone
package and then load a dataset that was created
with an older version of the package, it will be automatically migrated to the
new package version (if necessary) the first time you load it.
Dataset tags¶
All Dataset
instances have a
tags
property that you can use to
store an arbitrary list of string tags.
1 2 3 4 5 6 7 8 9 10 11 | import fiftyone as fo dataset = fo.Dataset() # Add some tags dataset.tags = ["test", "projectA"] # Edit the tags dataset.tags.pop() dataset.tags.append("projectB") dataset.save() # must save after edits |
Note
You must call
dataset.save()
after updating
the dataset’s tags
property
in-place to save the changes to the database.
Dataset stats¶
You can use the stats()
method on
a dataset to obtain information about the size of the dataset on disk,
including its metadata in the database and optionally the size of the physical
media on disk:
1 2 3 4 5 6 | import fiftyone as fo import fiftyone.zoo as foz dataset = foz.load_zoo_dataset("quickstart") fo.pprint(dataset.stats(include_media=True)) |
{
'samples_count': 200,
'samples_bytes': 1290762,
'samples_size': '1.2MB',
'media_bytes': 24412374,
'media_size': '23.3MB',
'total_bytes': 25703136,
'total_size': '24.5MB',
}
You can also invoke
stats()
on a
dataset view to retrieve stats for a specific subset of
the dataset:
1 2 3 | view = dataset[:10].select_fields("ground_truth") fo.pprint(view.stats(include_media=True)) |
{
'samples_count': 10,
'samples_bytes': 10141,
'samples_size': '9.9KB',
'media_bytes': 1726296,
'media_size': '1.6MB',
'total_bytes': 1736437,
'total_size': '1.7MB',
}
Storing info¶
All Dataset
instances have an
info
property, which contains a
dictionary that you can use to store any JSON-serializable information you wish
about your dataset.
Datasets can also store more specific types of ancillary information such as class lists and mask targets.
1 2 3 4 5 6 7 8 9 10 11 12 13 | import fiftyone as fo dataset = fo.Dataset() # Store a class list in the dataset's info dataset.info = { "dataset_source": "https://...", "author": "...", } # Edit existing info dataset.info["owner"] = "..." dataset.save() # must save after edits |
Note
You must call
dataset.save()
after updating
the dataset’s info
property
in-place to save the changes to the database.
Dataset App config¶
All Dataset
instances have an
app_config
property that
contains a DatasetAppConfig
that you can use to store dataset-specific
settings that customize how the dataset is visualized in the
FiftyOne App.
1 2 3 4 5 6 7 8 | import fiftyone as fo import fiftyone.zoo as foz dataset = foz.load_zoo_dataset("quickstart") session = fo.launch_app(dataset) # View the dataset's current App config print(dataset.app_config) |
Multiple media fields¶
You can declare multiple media fields on a dataset and configure which field is used by various components of the App by default:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | import fiftyone.utils.image as foui # Generate some thumbnail images foui.transform_images( dataset, size=(-1, 32), output_field="thumbnail_path", output_dir="/tmp/thumbnails", ) # Configure when to use each field dataset.app_config.media_fields = ["filepath", "thumbnail_path"] dataset.app_config.grid_media_field = "thumbnail_path" dataset.save() # must save after edits session.refresh() |
You can set media_fallback=True
if you want the App to fallback to the
filepath
field if an alternate media field is missing for a particular
sample in the grid and/or modal:
1 2 3 | # Fallback to `filepath` if an alternate media field is missing dataset.app_config.media_fallback = True dataset.save() |
Custom color scheme¶
You can store a custom color scheme on a dataset that should be used by default whenever the dataset is loaded in the App:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 | dataset.evaluate_detections( "predictions", gt_field="ground_truth", eval_key="eval" ) # Store a custom color scheme dataset.app_config.color_scheme = fo.ColorScheme( color_pool=["#ff0000", "#00ff00", "#0000ff", "pink", "yellowgreen"], color_by="value", fields=[ { "path": "ground_truth", "colorByAttribute": "eval", "valueColors": [ {"value": "fn", "color": "#0000ff"}, # false negatives: blue {"value": "tp", "color": "#00ff00"}, # true positives: green ] }, { "path": "predictions", "colorByAttribute": "eval", "valueColors": [ {"value": "fp", "color": "#ff0000"}, # false positives: red {"value": "tp", "color": "#00ff00"}, # true positives: green ] } ] ) dataset.save() # must save after edits # Setting `color_scheme` to None forces the dataset's default color scheme # to be loaded session.color_scheme = None |
Note
Refer to the ColorScheme
class for documentation of the available
customization options.
Note
Did you know? You can also configure color schemes directly in the App!
Sidebar groups¶
You can configure the organization and default expansion state of the sidebar’s field groups:
1 2 3 4 5 6 7 8 9 10 11 12 | # Get the default sidebar groups for the dataset sidebar_groups = fo.DatasetAppConfig.default_sidebar_groups(dataset) # Collapse the `metadata` section by default print(sidebar_groups[2].name) # metadata sidebar_groups[2].expanded = False # Modify the dataset's App config dataset.app_config.sidebar_groups = sidebar_groups dataset.save() # must save after edits session.refresh() |
Disable frame filtering¶
Filtering by frame-level fields of video datasets in the App’s grid view can be expensive when the dataset is large.
You can disable frame filtering for a video dataset as follows:
1 2 3 4 5 6 7 8 9 | import fiftyone as fo import fiftyone.zoo as foz dataset = foz.load_zoo_dataset("quickstart-video") dataset.app_config.disable_frame_filtering = True dataset.save() # must save after edits session = fo.launch_app(dataset) |
Note
Did you know? You can also globally disable frame filtering for all video datasets via your App config.
Resetting a dataset’s App config¶
You can conveniently reset any property of a dataset’s App config by setting it
to None
:
1 2 3 4 5 6 7 | # Reset the dataset's color scheme dataset.app_config.color_scheme = None dataset.save() # must save after edits print(dataset.app_config) session.refresh() |
or you can reset the entire App config by setting the
app_config
property to
None
:
1 2 3 4 5 | # Reset App config dataset.app_config = None print(dataset.app_config) session = fo.launch_app(dataset) |
Note
Check out this section for more information about customizing the behavior of the App.
Storing class lists¶
All Dataset
instances have
classes
and
default_classes
properties that you can use to store the lists of possible classes for your
annotations/models.
The classes
property is a
dictionary mapping field names to class lists for a single Label
field of the
dataset.
If all Label
fields in your dataset have the same semantics, you can store a
single class list in the store a single target dictionary in the
default_classes
property of your dataset.
You can also pass your class lists to methods such as
evaluate_classifications()
,
evaluate_detections()
,
and export()
that
require knowledge of the possible classes in a dataset or field(s).
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | import fiftyone as fo dataset = fo.Dataset() # Set default classes dataset.default_classes = ["cat", "dog"] # Edit the default classes dataset.default_classes.append("other") dataset.save() # must save after edits # Set classes for the `ground_truth` and `predictions` fields dataset.classes = { "ground_truth": ["cat", "dog"], "predictions": ["cat", "dog", "other"], } # Edit a field's classes dataset.classes["ground_truth"].append("other") dataset.save() # must save after edits |
Note
You must call
dataset.save()
after updating
the dataset’s classes
and
default_classes
properties in-place to save the changes to the database.
Storing mask targets¶
All Dataset
instances have
mask_targets
and
default_mask_targets
properties that you can use to store label strings for the pixel values of
Segmentation
field masks.
The mask_targets
property
is a dictionary mapping field names to target dicts, each of which is a
dictionary defining the mapping between pixel values (2D masks) or RGB hex
strings (3D masks) and label strings for the Segmentation
masks in the
specified field of the dataset.
If all Segmentation
fields in your dataset have the same semantics, you can
store a single target dictionary in the
default_mask_targets
property of your dataset.
When you load datasets with Segmentation
fields in the App that have
corresponding mask targets, the label strings will appear in the App’s tooltip
when you hover over pixels.
You can also pass your mask targets to methods such as
evaluate_segmentations()
and export()
that
require knowledge of the mask targets for a dataset or field(s).
If you are working with 2D segmentation masks, specify target keys as integers:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | import fiftyone as fo dataset = fo.Dataset() # Set default mask targets dataset.default_mask_targets = {1: "cat", 2: "dog"} # Edit the default mask targets dataset.default_mask_targets[255] = "other" dataset.save() # must save after edits # Set mask targets for the `ground_truth` and `predictions` fields dataset.mask_targets = { "ground_truth": {1: "cat", 2: "dog"}, "predictions": {1: "cat", 2: "dog", 255: "other"}, } # Edit an existing mask target dataset.mask_targets["ground_truth"][255] = "other" dataset.save() # must save after edits |
If you are working with RGB segmentation masks, specify target keys as RGB hex strings:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | import fiftyone as fo dataset = fo.Dataset() # Set default mask targets dataset.default_mask_targets = {"#499CEF": "cat", "#6D04FF": "dog"} # Edit the default mask targets dataset.default_mask_targets["#FF6D04"] = "person" dataset.save() # must save after edits # Set mask targets for the `ground_truth` and `predictions` fields dataset.mask_targets = { "ground_truth": {"#499CEF": "cat", "#6D04FF": "dog"}, "predictions": { "#499CEF": "cat", "#6D04FF": "dog", "#FF6D04": "person" }, } # Edit an existing mask target dataset.mask_targets["ground_truth"]["#FF6D04"] = "person" dataset.save() # must save after edits |
Note
You must call
dataset.save()
after updating
the dataset’s
mask_targets
and
default_mask_targets
properties in-place to save the changes to the database.
Storing keypoint skeletons¶
All Dataset
instances have
skeletons
and
default_skeleton
properties that you can use to store keypoint skeletons for Keypoint
field(s)
of a dataset.
The skeletons
property is a
dictionary mapping field names to KeypointSkeleton
instances, each of which
defines the keypoint label strings and edge connectivity for the Keypoint
instances in the specified field of the dataset.
If all Keypoint
fields in your dataset have the same semantics, you can store
a single KeypointSkeleton
in the
default_skeleton
property of your dataset.
When you load datasets with Keypoint
fields in the App that have
corresponding skeletons, the skeletons will automatically be rendered and label
strings will appear in the App’s tooltip when you hover over the keypoints.
Keypoint skeletons can be associated with Keypoint
or Keypoints
fields
whose points
attributes all
contain a fixed number of semantically ordered points.
The edges
argument
contains lists of integer indexes that define the connectivity of the points in
the skeleton, and the optional
labels
argument
defines the label strings for each node in the skeleton.
For example, the skeleton below is defined by edges between the following nodes:
left hand <-> left shoulder <-> right shoulder <-> right hand
left eye <-> right eye <-> mouth
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | import fiftyone as fo dataset = fo.Dataset() # Set keypoint skeleton for the `ground_truth` field dataset.skeletons = { "ground_truth": fo.KeypointSkeleton( labels=[ "left hand" "left shoulder", "right shoulder", "right hand", "left eye", "right eye", "mouth", ], edges=[[0, 1, 2, 3], [4, 5, 6]], ) } # Edit an existing skeleton dataset.skeletons["ground_truth"].labels[-1] = "lips" dataset.save() # must save after edits |
Note
When using keypoint skeletons, each Keypoint
instance’s
points
list must always
respect the indexing defined by the field’s KeypointSkeleton
.
If a particular keypoint is occluded or missing for an object, use
[float("nan"), float("nan")]
in its
points
list.
Note
You must call
dataset.save()
after updating
the dataset’s
skeletons
and
default_skeleton
properties in-place to save the changes to the database.
Deleting a dataset¶
Delete a dataset explicitly via
Dataset.delete()
. Once a dataset
is deleted, any existing reference in memory will be in a volatile state.
Dataset.name
and
Dataset.deleted
will still be valid
attributes, but calling any other attribute or method will raise a
DoesNotExistError
.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | dataset = fo.load_dataset("my_first_dataset") dataset.delete() print(fo.list_datasets()) # [] print(dataset.name) # my_first_dataset print(dataset.deleted) # True print(dataset.persistent) # DoesNotExistError: Dataset 'my_first_dataset' is deleted |
Samples¶
An individual Sample
is always initialized with a filepath
to the
corresponding data on disk.
1 2 3 4 5 | # An image sample sample = fo.Sample(filepath="/path/to/image.png") # A video sample another_sample = fo.Sample(filepath="/path/to/video.mp4") |
Note
Creating a new Sample
does not load the source data into memory. Source
data is read only as needed by the App.
Adding samples to a dataset¶
A Sample
can easily be added to an existing Dataset
:
1 2 | dataset = fo.Dataset("example_dataset") dataset.add_sample(sample) |
When a sample is added to a dataset, the relevant attributes of the Sample
are automatically updated:
1 2 3 4 5 | print(sample.in_dataset) # True print(sample.dataset_name) # example_dataset |
Every sample in a dataset is given a unique ID when it is added:
1 2 | print(sample.id) # 5ee0ebd72ceafe13e7741c42 |
Multiple samples can be efficiently added to a dataset in batches:
1 2 3 4 5 6 7 8 9 10 11 12 13 | print(len(dataset)) # 1 dataset.add_samples( [ fo.Sample(filepath="/path/to/image1.jpg"), fo.Sample(filepath="/path/to/image2.jpg"), fo.Sample(filepath="/path/to/image3.jpg"), ] ) print(len(dataset)) # 4 |
Accessing samples in a dataset¶
FiftyOne provides multiple ways to access a Sample
in a Dataset
.
You can iterate over the samples in a dataset:
1 2 | for sample in dataset: print(sample) |
Use first()
and
last()
to retrieve the first and
last samples in a dataset, respectively:
1 2 | first_sample = dataset.first() last_sample = dataset.last() |
Samples can be accessed directly from datasets by their IDs or their filepaths.
Sample
objects are singletons, so the same Sample
instance is returned
whenever accessing the sample from the Dataset
:
1 2 3 4 5 6 7 | same_sample = dataset[sample.id] print(same_sample is sample) # True also_same_sample = dataset[sample.filepath] print(also_same_sample is sample) # True |
You can use dataset views to perform more sophisticated operations on samples like searching, filtering, sorting, and slicing.
Note
Accessing a sample by its integer index in a Dataset
is not allowed. The
best practice is to lookup individual samples by ID or filepath, or use
array slicing to extract a range of samples, and iterate over samples in a
view.
dataset[0]
# KeyError: Accessing dataset samples by numeric index is not supported.
# Use sample IDs, filepaths, slices, boolean arrays, or a boolean ViewExpression instead
Deleting samples from a dataset¶
Samples can be removed from a Dataset
through their ID, either one at a time
or in batches via
delete_samples()
:
1 2 3 4 5 6 | dataset.delete_samples(sample_id) # equivalent to above del dataset[sample_id] dataset.delete_samples([sample_id1, sample_id2]) |
Samples can also be removed from a Dataset
by passing Sample
instance(s)
or DatasetView
instances:
1 2 3 4 5 6 7 | # Remove a random sample sample = dataset.take(1).first() dataset.delete_samples(sample) # Remove 10 random samples view = dataset.take(10) dataset.delete_samples(view) |
If a Sample
object in memory is deleted from a dataset, it will revert to
a Sample
that has not been added to a Dataset
:
1 2 3 4 5 6 7 8 | print(sample.in_dataset) # False print(sample.dataset_name) # None print(sample.id) # None |
Fields¶
A Field
is an attribute of a Sample
that stores information about the
sample.
Fields can be dynamically created, modified, and deleted from samples on a
per-sample basis. When a new Field
is assigned to a Sample
in a Dataset
,
it is automatically added to the dataset’s schema and thus accessible on all
other samples in the dataset.
If a field exists on a dataset but has not been set on a particular sample, its
value will be None
.
Default sample fields¶
By default, all Sample
instances have the following fields:
Field |
Type |
Default |
Description |
---|---|---|---|
|
string |
|
The ID of the sample in its parent dataset, which
is generated automatically when the sample is
added to a dataset, or |
|
string |
REQUIRED |
The path to the source data on disk. Must be provided at sample creation time |
|
string |
N/A |
The media type of the sample. Computed
automatically from the provided |
|
list |
|
A list of string tags for the sample |
|
|
Type-specific metadata about the source data |
|
|
datetime |
|
The datetime that the sample was added to its
parent dataset, which is generated automatically,
or |
|
datetime |
|
The datetime that the sample was last modified,
which is updated automatically, or |
Note
The created_at
and last_modified_at
fields are
read-only and are automatically populated/updated
when you add samples to datasets and modify them, respectively.
1 2 3 4 5 | import fiftyone as fo sample = fo.Sample(filepath="/path/to/image.png") print(sample) |
<Sample: {
'id': None,
'media_type': 'image',
'filepath': '/path/to/image.png',
'tags': [],
'metadata': None,
'created_at': None,
'last_modified_at': None,
}>
Accessing fields of a sample¶
The names of available fields can be checked on any individual Sample
:
1 2 | sample.field_names # ('id', 'filepath', 'tags', 'metadata', 'created_at', 'last_modified_at') |
The value of a Field
for a given Sample
can be accessed either by either
attribute or item access:
1 2 | sample.filepath sample["filepath"] # equivalent |
Field schemas¶
You can use
get_field_schema()
to
retrieve detailed information about the schema of the samples in a dataset:
1 2 3 4 | dataset = fo.Dataset("a_dataset") dataset.add_sample(sample) dataset.get_field_schema() |
OrderedDict([
('id', <fiftyone.core.fields.ObjectIdField at 0x7fbaa862b358>),
('filepath', <fiftyone.core.fields.StringField at 0x11c77ae10>),
('tags', <fiftyone.core.fields.ListField at 0x11c790828>),
('metadata', <fiftyone.core.fields.EmbeddedDocumentField at 0x11c7907b8>),
('created_at', <fiftyone.core.fields.DateTimeField at 0x7fea48361af0>),
('last_modified_at', <fiftyone.core.fields.DateTimeField at 0x7fea48361b20>)]),
])
You can also view helpful information about a dataset, including its schema, by printing it:
1 | print(dataset) |
Name: a_dataset
Media type: image
Num samples: 1
Persistent: False
Tags: []
Sample fields:
id: fiftyone.core.fields.ObjectIdField
filepath: fiftyone.core.fields.StringField
tags: fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)
metadata: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.ImageMetadata)
created_at: fiftyone.core.fields.DateTimeField
last_modified_at: fiftyone.core.fields.DateTimeField
Note
Did you know? You can store metadata such as descriptions on your dataset’s fields!
Adding fields to a sample¶
New fields can be added to a Sample
using item assignment:
1 2 | sample["integer_field"] = 51 sample.save() |
If the Sample
belongs to a Dataset
, the dataset’s schema will automatically
be updated to reflect the new field:
1 | print(dataset) |
Name: a_dataset
Media type: image
Num samples: 1
Persistent: False
Tags: []
Sample fields:
id: fiftyone.core.fields.ObjectIdField
filepath: fiftyone.core.fields.StringField
tags: fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)
metadata: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.ImageMetadata)
created_at: fiftyone.core.fields.DateTimeField
last_modified_at: fiftyone.core.fields.DateTimeField
integer_field: fiftyone.core.fields.IntField
A Field
can be any primitive type, such as bool
, int
, float
, str
,
date
, datetime
, list
, dict
, or more complex data structures
like label types:
1 2 | sample["animal"] = fo.Classification(label="alligator") sample.save() |
Whenever a new field is added to a sample in a dataset, the field is available
on every other sample in the dataset with the value None
.
Fields must have the same type (or None
) across all samples in the dataset.
Setting a field to an inappropriate type raises an error:
1 2 3 | sample2.integer_field = "a string" sample2.save() # ValidationError: a string could not be converted to int |
Note
You must call sample.save()
in
order to persist changes to the database when editing samples that are in
datasets.
Adding fields to a dataset¶
You can also use
add_sample_field()
to
declare new sample fields directly on datasets without explicitly populating
any values on its samples:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 | import fiftyone as fo sample = fo.Sample( filepath="image.jpg", ground_truth=fo.Classification(label="cat"), ) dataset = fo.Dataset() dataset.add_sample(sample) # Declare new primitive fields dataset.add_sample_field("scene_id", fo.StringField) dataset.add_sample_field("quality", fo.FloatField) # Declare untyped list fields dataset.add_sample_field("more_tags", fo.ListField) dataset.add_sample_field("info", fo.ListField) # Declare a typed list field dataset.add_sample_field("also_tags", fo.ListField, subfield=fo.StringField) # Declare a new Label field dataset.add_sample_field( "predictions", fo.EmbeddedDocumentField, embedded_doc_type=fo.Classification, ) print(dataset.get_field_schema()) |
{
'id': <fiftyone.core.fields.ObjectIdField object at 0x7f9280803910>,
'filepath': <fiftyone.core.fields.StringField object at 0x7f92d273e0d0>,
'tags': <fiftyone.core.fields.ListField object at 0x7f92d2654f70>,
'metadata': <fiftyone.core.fields.EmbeddedDocumentField object at 0x7f9280803d90>,
'created_at': <fiftyone.core.fields.DateTimeField object at 0x7fea48361af0>,
'last_modified_at': <fiftyone.core.fields.DateTimeField object at 0x7fea48361b20>,
'ground_truth': <fiftyone.core.fields.EmbeddedDocumentField object at 0x7f92d2605190>,
'scene_id': <fiftyone.core.fields.StringField object at 0x7f9280803490>,
'quality': <fiftyone.core.fields.FloatField object at 0x7f92d2605bb0>,
'more_tags': <fiftyone.core.fields.ListField object at 0x7f92d08e4550>,
'info': <fiftyone.core.fields.ListField object at 0x7f92d264f9a0>,
'also_tags': <fiftyone.core.fields.ListField object at 0x7f92d264ff70>,
'predictions': <fiftyone.core.fields.EmbeddedDocumentField object at 0x7f92d2605640>,
}
Whenever a new field is added to a dataset, the field is immediately available
on all samples in the dataset with the value None
:
1 | print(sample) |
<Sample: {
'id': '642d8848f291652133df8d3a',
'media_type': 'image',
'filepath': '/Users/Brian/dev/fiftyone/image.jpg',
'tags': [],
'metadata': None,
'created_at': datetime.datetime(2024, 7, 22, 5, 0, 25, 372399),
'last_modified_at': datetime.datetime(2024, 7, 22, 5, 0, 25, 372399),
'ground_truth': <Classification: {
'id': '642d8848f291652133df8d38',
'tags': [],
'label': 'cat',
'confidence': None,
'logits': None,
}>,
'scene_id': None,
'quality': None,
'more_tags': None,
'info': None,
'also_tags': None,
'predictions': None,
}>
You can also declare nested fields on existing embedded documents using dot notation:
1 2 | # Declare a new attribute on a `Classification` field dataset.add_sample_field("predictions.breed", fo.StringField) |
Note
See this section for more options for dynamically expanding the schema of nested lists and embedded documents.
You can use get_field()
to
retrieve a Field
instance by its name or embedded.field.name
. And, if the
field contains an embedded document, you can call
get_field_schema()
to recursively inspect its nested fields:
1 2 3 4 5 6 7 8 9 10 11 | field = dataset.get_field("predictions") print(field.document_type) # <class 'fiftyone.core.labels.Classification'> print(set(field.get_field_schema().keys())) # {'logits', 'confidence', 'breed', 'tags', 'label', 'id'} # Directly retrieve a nested field field = dataset.get_field("predictions.breed") print(type(field)) # <class 'fiftyone.core.fields.StringField'> |
If your dataset contains a ListField
with no value type declared, you can add
the type later by appending []
to the field path:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | field = dataset.get_field("more_tags") print(field.field) # None # Declare the subfield types of an existing untyped list field dataset.add_sample_field("more_tags[]", fo.StringField) field = dataset.get_field("more_tags") print(field.field) # StringField # List fields can also contain embedded documents dataset.add_sample_field( "info[]", fo.EmbeddedDocumentField, embedded_doc_type=fo.DynamicEmbeddedDocument, ) field = dataset.get_field("info") print(field.field) # EmbeddedDocumentField print(field.field.document_type) # DynamicEmbeddedDocument |
Note
Declaring the value type of list fields is required if you want to filter by the list’s values in the App.
Editing sample fields¶
You can make any edits you wish to the fields of an existing Sample
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | sample = fo.Sample( filepath="/path/to/image.jpg", ground_truth=fo.Detections( detections=[ fo.Detection(label="CAT", bounding_box=[0.1, 0.1, 0.4, 0.4]), fo.Detection(label="dog", bounding_box=[0.5, 0.5, 0.4, 0.4]), ] ) ) detections = sample.ground_truth.detections # Edit an existing detection detections[0].label = "cat" # Add a new detection new_detection = fo.Detection(label="animals", bounding_box=[0, 0, 1, 1]) detections.append(new_detection) print(sample) sample.save() # if the sample is in a dataset |
Note
You must call sample.save()
in
order to persist changes to the database when editing samples that are in
datasets.
A common workflow is to iterate over a dataset or view and edit each sample:
1 2 3 | for sample in dataset: sample["new_field"] = ... sample.save() |
The iter_samples()
method
is an equivalent way to iterate over a dataset that provides a
progress=True
option that prints a progress bar tracking the status of the
iteration:
1 2 3 4 | # Prints a progress bar tracking the status of the iteration for sample in dataset.iter_samples(progress=True): sample["new_field"] = ... sample.save() |
The iter_samples()
method
also provides an autosave=True
option that causes all changes to samples
emitted by the iterator to be automatically saved using efficient batch
updates:
1 2 3 | # Automatically saves sample edits in efficient batches for sample in dataset.iter_samples(autosave=True): sample["new_field"] = ... |
Using autosave=True
can significantly improve performance when editing
large datasets. See this section for more information
on batch update patterns.
Removing fields from a sample¶
A field can be deleted from a Sample
using del
:
1 | del sample["integer_field"] |
If the Sample
is not yet in a dataset, deleting a field will remove it from
the sample. If the Sample
is in a dataset, the field’s value will be None
.
Fields can also be deleted at the Dataset
level, in which case they are
removed from every Sample
in the dataset:
1 2 3 4 | dataset.delete_sample_field("integer_field") sample.integer_field # AttributeError: Sample has no field 'integer_field' |
Storing field metadata¶
You can store metadata such as descriptions and other info on the fields of your dataset.
One approach is to manually declare the field with
add_sample_field()
with the appropriate metadata provided:
1 2 3 4 5 6 7 8 9 | import fiftyone as fo dataset = fo.Dataset() dataset.add_sample_field( "int_field", fo.IntField, description="An integer field" ) field = dataset.get_field("int_field") print(field.description) # An integer field |
You can also use
get_field()
to
retrieve a field and update it’s metadata at any time:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | import fiftyone as fo import fiftyone.zoo as foz dataset = foz.load_zoo_dataset("quickstart") dataset.add_dynamic_sample_fields() field = dataset.get_field("ground_truth") field.description = "Ground truth annotations" field.info = {"url": "https://fiftyone.ai"} field.save() # must save after edits field = dataset.get_field("ground_truth.detections.area") field.description = "Area of the box, in pixels^2" field.info = {"url": "https://fiftyone.ai"} field.save() # must save after edits dataset.reload() field = dataset.get_field("ground_truth") print(field.description) # Ground truth annotations print(field.info) # {'url': 'https://fiftyone.ai'} field = dataset.get_field("ground_truth.detections.area") print(field.description) # Area of the box, in pixels^2 print(field.info) # {'url': 'https://fiftyone.ai'} |
Note
You must call
field.save()
after updating
a fields’s description
and info
attributes in-place to
save the changes to the database.
Note
Did you know? You can view field metadata directly in the App by hovering over fields or attributes in the sidebar!
Read-only fields¶
Certain default sample fields like created_at
and last_modified_at
are read-only and thus cannot be manually edited:
1 2 3 4 5 6 7 8 9 10 11 12 13 | from datetime import datetime import fiftyone as fo sample = fo.Sample(filepath="/path/to/image.jpg") dataset = fo.Dataset() dataset.add_sample(sample) sample.created_at = datetime.utcnow() # ValueError: Cannot edit read-only field 'created_at' sample.last_modified_at = datetime.utcnow() # ValueError: Cannot edit read-only field 'last_modified_at' |
You can also manually mark additional fields or embedded fields as read-only at any time:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 | import fiftyone as fo import fiftyone.zoo as foz dataset = foz.load_zoo_dataset("quickstart") # Declare a new read-only field dataset.add_sample_field("uuid", fo.StringField, read_only=True) # Mark 'filepath' as read-only field = dataset.get_field("filepath") field.read_only = True field.save() # must save after edits # Mark a nested field as read-only field = dataset.get_field("ground_truth.detections.label") field.read_only = True field.save() # must save after edits sample = dataset.first() sample.filepath = "no.jpg" # ValueError: Cannot edit read-only field 'filepath' sample.ground_truth.detections[0].label = "no" sample.save() # ValueError: Cannot edit read-only field 'ground_truth.detections.label' |
Note
You must call
field.save()
after updating
a fields’s read_only
attributes in-place to save the changes to the database.
Note that read-only fields do not interfere with the ability to add/delete samples from datasets:
1 2 3 4 | sample = fo.Sample(filepath="/path/to/image.jpg", uuid="1234") dataset.add_sample(sample) dataset.delete_samples(sample) |
Any fields that you’ve manually marked as read-only may be reverted to editable at any time:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | sample = dataset.first() # Revert 'filepath' to editable field = dataset.get_field("filepath") field.read_only = False field.save() # must save after edits # Revert nested field to editable field = dataset.get_field("ground_truth.detections.label") field.read_only = False field.save() # must save after edits sample.filepath = "yes.jpg" sample.ground_truth.detections[0].label = "yes" sample.save() |
Summary fields¶
Summary fields allow you to efficiently perform queries on large datasets where directly querying the underlying field is prohibitively slow due to the number of objects/frames in the field.
For example, suppose you’re working on a
video dataset with frame-level objects, and you’re
interested in finding videos that contain specific classes of interest, eg
person
, in at least one frame:
1 2 3 4 5 6 7 8 | import fiftyone as fo import fiftyone.zoo as foz from fiftyone import ViewField as F dataset = foz.load_zoo_dataset("quickstart-video") dataset.set_field("frames.detections.detections.confidence", F.rand()).save() session = fo.launch_app(dataset) |
One approach is to directly query the frame-level field (frames.detections
in this case) in the App’s sidebar. However, when the dataset is large, such
queries are inefficient, as they cannot unlock
query performance and thus require
full collection scans over all frames to retrieve the relevant samples.
A more efficient approach is to first use
create_summary_field()
to summarize the relevant input field path(s):
1 2 3 4 5 6 7 8 9 | # Generate a summary field for object labels field_name = dataset.create_summary_field("frames.detections.detections.label") # The name of the summary field that was created print(field_name) # 'frames_detections_label' # Generate a summary field for [min, max] confidences dataset.create_summary_field("frames.detections.detections.confidence") |
Summary fields can be generated for sample-level and frame-level fields, and the input fields can be either categorical or numeric:
When the input field is categorical (string or boolean), the summary field of each sample is populated with the list of unique values observed in the field (across all frames for video samples):
1 2 3 | sample = dataset.first() print(sample.frames_detections_label) # ['vehicle', 'road sign', 'person'] |
You can also pass include_counts=True
to include counts for each
unique value in the summary field:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | # Generate a summary field for object labels and counts dataset.create_summary_field( "frames.detections.detections.label", field_name="frames_detections_label2", include_counts=True, ) sample = dataset.first() print(sample.frames_detections_label2) """ [ <DynamicEmbeddedDocument: {'label': 'road sign', 'count': 198}>, <DynamicEmbeddedDocument: {'label': 'vehicle', 'count': 175}>, <DynamicEmbeddedDocument: {'label': 'person', 'count': 120}>, ] """ |
When the input field is numeric (int, float, date, or datetime), the
summary field of each sample is populated with the [min, max]
range
of the values observed in the field (across all frames for video
samples):
1 2 3 | sample = dataset.first() print(sample.frames_detections_confidence) # <DynamicEmbeddedDocument: {'min': 0.01, 'max': 0.99}> |
You can also pass the group_by
parameter to specify an attribute to
group by to generate per-attribute [min, max]
ranges:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | # Generate a summary field for per-label [min, max] confidences dataset.create_summary_field( "frames.detections.detections.confidence", field_name="frames_detections_confidence2", group_by="label", ) sample = dataset.first() print(sample.frames_detections_confidence2) """ [ <DynamicEmbeddedDocument: {'label': 'vehicle', 'min': 0.00, 'max': 0.98}>, <DynamicEmbeddedDocument: {'label': 'person', 'min': 0.02, 'max': 0.97}>, <DynamicEmbeddedDocument: {'label': 'road sign', 'min': 0.01, 'max': 0.99}>, ] """ |
As the above examples illustrate, summary fields allow you to encode various types of information at the sample-level that you can directly query to find samples that contain specific values.
Moreover, summary fields are indexed by default and the App can natively leverage these indexes to provide performant filtering:
Note
Summary fields are automatically added to a summaries
sidebar group in the App for
easy access and organization.
They are also read-only by default, as they are implicitly derived from the contents of their source field and are not intended to be directly modified.
You can use
list_summary_fields()
to list the names of the summary fields on your dataset:
1 2 | print(dataset.list_summary_fields()) # ['frames_detections_label', 'frames_detections_confidence', ...] |
Since a summary field is derived from the contents of another field, it must be
updated whenever there have been modifications to its source field. You can use
check_summary_fields()
to check for summary fields that may need to be updated:
1 2 3 4 5 6 7 8 9 10 11 | # Newly created summary fields don't needed updating print(dataset.check_summary_fields()) # [] # Modify the dataset label_upper = F("label").upper() dataset.set_field("frames.detections.detections.label", label_upper).save() # Summary fields now (may) need updating print(dataset.check_summary_fields()) # ['frames_detections_label', 'frames_detections_confidence', ...] |
Note
Note that inclusion in
check_summary_fields()
is only a heuristic, as any sample modifications may not have affected
the summary’s source field.
Use update_summary_field()
to regenerate a summary field based on the current values of its source field:
1 | dataset.update_summary_field("frames_detections_label") |
Finally, use
delete_summary_field()
or delete_summary_fields()
to delete existing summary field(s) that you no longer need:
1 | dataset.delete_summary_field("frames_detections_label") |
Media type¶
When a Sample
is created, its media type is inferred from the filepath
to
the source media and available via the media_type
attribute of the sample,
which is read-only.
Media type is inferred from the MIME type of the file on disk, as per the table below:
MIME type/extension |
|
Description |
---|---|---|
|
|
Image sample |
|
|
Video sample |
|
|
3D sample |
|
|
Point cloud sample |
other |
|
Generic sample |
Note
The filepath
of a sample can be changed after the sample is created, but
the new filepath must have the same media type. In other words,
media_type
is immutable.
Tags¶
All Sample
instances have a tags
field, which is a string list. By default,
this list is empty, but you can use it to store information like dataset splits
or application-specific issues like low quality images:
1 2 3 4 5 6 7 8 9 10 11 | dataset = fo.Dataset("tagged_dataset") dataset.add_samples( [ fo.Sample(filepath="/path/to/image1.png", tags=["train"]), fo.Sample(filepath="/path/to/image2.png", tags=["test", "low_quality"]), ] ) print(dataset.distinct("tags")) # ["test", "low_quality", "train"] |
Note
Did you know? You can add, edit, and filter by sample tags directly in the App.
The tags
field can be used like a standard Python list:
1 2 3 | sample = dataset.first() sample.tags.append("new_tag") sample.save() |
Note
You must call sample.save()
in
order to persist changes to the database when editing samples that are in
datasets.
Datasets and views provide helpful methods such as
count_sample_tags()
,
tag_samples()
,
untag_samples()
,
and
match_tags()
that you can use to perform batch queries and edits to sample tags:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | import fiftyone as fo import fiftyone.zoo as foz dataset = foz.load_zoo_dataset("quickstart").clone() print(dataset.count_sample_tags()) # {'validation': 200} # Tag samples in a view test_view = dataset.limit(100) test_view.untag_samples("validation") test_view.tag_samples("test") print(dataset.count_sample_tags()) # {'validation': 100, 'test': 100} # Create a view containing samples with a specific tag validation_view = dataset.match_tags("validation") print(len(validation_view)) # 100 |
Metadata¶
All Sample
instances have a metadata
field, which can optionally be
populated with a Metadata
instance that stores data type-specific metadata
about the raw data in the sample. The FiftyOne App and
the FiftyOne Brain will use this provided metadata in
some workflows when it is available.
For image samples, the ImageMetadata
class is used to store
information about images, including their
size_bytes
,
mime_type
,
width
,
height
, and
num_channels
.
You can populate the metadata
field of an existing dataset by calling
Dataset.compute_metadata()
:
1 2 3 4 5 6 7 8 | import fiftyone.zoo as foz dataset = foz.load_zoo_dataset("quickstart") # Populate metadata fields (if necessary) dataset.compute_metadata() print(dataset.first()) |
Alternatively, FiftyOne provides a
ImageMetadata.build_for()
factory method that you can use to compute the metadata for your images
when constructing Sample
instances:
1 2 3 4 5 6 7 | image_path = "/path/to/image.png" metadata = fo.ImageMetadata.build_for(image_path) sample = fo.Sample(filepath=image_path, metadata=metadata) print(sample) |
<Sample: {
'id': None,
'media_type': 'image',
'filepath': '/path/to/image.png',
'tags': [],
'metadata': <ImageMetadata: {
'size_bytes': 544559,
'mime_type': 'image/png',
'width': 698,
'height': 664,
'num_channels': 3,
}>,
'created_at': None,
'last_modified_at': None,
}>
For video samples, the VideoMetadata
class is used to store
information about videos, including their
size_bytes
,
mime_type
,
frame_width
,
frame_height
,
frame_rate
,
total_frame_count
,
duration
, and
encoding_str
.
You can populate the metadata
field of an existing dataset by calling
Dataset.compute_metadata()
:
1 2 3 4 5 6 7 8 | import fiftyone.zoo as foz dataset = foz.load_zoo_dataset("quickstart-video") # Populate metadata fields (if necessary) dataset.compute_metadata() print(dataset.first()) |
Alternatively, FiftyOne provides a
VideoMetadata.build_for()
factory method that you can use to compute the metadata for your videos
when constructing Sample
instances:
1 2 3 4 5 6 7 | video_path = "/path/to/video.mp4" metadata = fo.VideoMetadata.build_for(video_path) sample = fo.Sample(filepath=video_path, metadata=metadata) print(sample) |
<Sample: {
'id': None,
'media_type': 'video',
'filepath': '/Users/Brian/Desktop/people.mp4',
'tags': [],
'metadata': <VideoMetadata: {
'size_bytes': 2038250,
'mime_type': 'video/mp4',
'frame_width': 1920,
'frame_height': 1080,
'frame_rate': 29.97002997002997,
'total_frame_count': 68,
'duration': 2.268933,
'encoding_str': 'avc1',
}>,
'created_at': None,
'last_modified_at': None,
'frames': <Frames: 0>,
}>
Dates and datetimes¶
You can store date information in FiftyOne datasets by populating fields with
date
or datetime
values:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | from datetime import date, datetime import fiftyone as fo dataset = fo.Dataset() dataset.add_samples( [ fo.Sample( filepath="image1.png", acquisition_time=datetime(2021, 8, 24, 21, 18, 7), acquisition_date=date(2021, 8, 24), ), fo.Sample( filepath="image2.png", acquisition_time=datetime.utcnow(), acquisition_date=date.today(), ), ] ) print(dataset) print(dataset.head()) |
Note
Did you know? You can create dataset views with date-based queries!
Internally, FiftyOne stores all dates as UTC timestamps, but you can provide
any valid datetime
object when setting a DateTimeField
of a sample,
including timezone-aware datetimes, which are internally converted to UTC
format for safekeeping.
1 2 3 4 5 6 7 8 9 10 11 12 13 | # A datetime in your local timezone now = datetime.utcnow().astimezone() sample = fo.Sample(filepath="image.png", acquisition_time=now) dataset = fo.Dataset() dataset.add_sample(sample) # Samples are singletons, so we reload so `sample` will contain values as # loaded from the database dataset.reload() sample.acquisition_time.tzinfo # None |
By default, when you access a datetime field of a sample in a dataset, it is
retrieved as a naive datetime
instance expressed in UTC format.
However, if you prefer, you can
configure FiftyOne to load datetime fields as
timezone-aware datetime
instances in a timezone of your choice.
Warning
FiftyOne assumes that all datetime
instances with no explicit timezone
are stored in UTC format.
Therefore, never use datetime.datetime.now()
when populating a datetime
field of a FiftyOne dataset! Instead, use datetime.datetime.utcnow()
.
Labels¶
The Label
class hierarchy is used to store semantic information about ground
truth or predicted labels in a sample.
Although such information can be stored in custom sample fields
(e.g, in a DictField
), it is recommended that you store label information in
Label
instances so that the FiftyOne App and the
FiftyOne Brain can visualize and compute on your
labels.
Note
All Label
instances are dynamic! You can add custom fields to your
labels to store custom information:
# Provide some default fields
label = fo.Classification(label="cat", confidence=0.98)
# Add custom fields
label["int"] = 5
label["float"] = 51.0
label["list"] = [1, 2, 3]
label["bool"] = True
label["dict"] = {"key": ["list", "of", "values"]}
You can also declare dynamic attributes on your dataset’s schema, which allows you to enforce type constraints, filter by these custom attributes in the App, and more.
FiftyOne provides a dedicated Label
subclass for many common tasks. The
subsections below describe them.
Regression¶
The Regression
class represents a numeric regression value for an image. The
value itself is stored in the
value
attribute of the
Regression
object. This may be a ground truth value or a model prediction.
The optional
confidence
attribute can
be used to store a score associated with the model prediction and can be
visualized in the App or used, for example, when
evaluating regressions.
1 2 3 4 5 6 7 8 | import fiftyone as fo sample = fo.Sample(filepath="/path/to/image.png") sample["ground_truth"] = fo.Regression(value=51.0) sample["prediction"] = fo.Classification(value=42.0, confidence=0.9) print(sample) |
<Sample: {
'id': None,
'media_type': 'image',
'filepath': '/path/to/image.png',
'tags': [],
'metadata': None,
'created_at': None,
'last_modified_at': None,
'ground_truth': <Regression: {
'id': '616c4bef36297ec40a26d112',
'tags': [],
'value': 51.0,
'confidence': None,
}>,
'prediction': <Classification: {
'id': '616c4bef36297ec40a26d113',
'tags': [],
'label': None,
'confidence': 0.9,
'logits': None,
'value': 42.0,
}>,
}>
Classification¶
The Classification
class represents a classification label for an image. The
label itself is stored in the
label
attribute of the
Classification
object. This may be a ground truth label or a model
prediction.
The optional
confidence
and
logits
attributes may be
used to store metadata about the model prediction. These additional fields can
be visualized in the App or used by Brain methods, e.g., when
computing label mistakes.
1 2 3 4 5 6 7 8 | import fiftyone as fo sample = fo.Sample(filepath="/path/to/image.png") sample["ground_truth"] = fo.Classification(label="sunny") sample["prediction"] = fo.Classification(label="sunny", confidence=0.9) print(sample) |
<Sample: {
'id': None,
'media_type': 'image',
'filepath': '/path/to/image.png',
'tags': [],
'metadata': None,
'created_at': None,
'last_modified_at': None,
'ground_truth': <Classification: {
'id': '5f8708db2018186b6ef66821',
'label': 'sunny',
'confidence': None,
'logits': None,
}>,
'prediction': <Classification: {
'id': '5f8708db2018186b6ef66822',
'label': 'sunny',
'confidence': 0.9,
'logits': None,
}>,
}>
Note
Did you know? You can store class lists for your models on your datasets.
Multilabel classification¶
The Classifications
class represents a list of classification labels for an
image. The typical use case is to represent multilabel annotations/predictions
for an image, where multiple labels from a model may apply to a given image.
The labels are stored in a
classifications
attribute of the object, which contains a list of Classification
instances.
Metadata about individual labels can be stored in the Classification
instances as usual; additionally, you can optionally store logits for the
overarching model (if applicable) in the
logits
attribute of the
Classifications
object.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | import fiftyone as fo sample = fo.Sample(filepath="/path/to/image.png") sample["ground_truth"] = fo.Classifications( classifications=[ fo.Classification(label="animal"), fo.Classification(label="cat"), fo.Classification(label="tabby"), ] ) sample["prediction"] = fo.Classifications( classifications=[ fo.Classification(label="animal", confidence=0.99), fo.Classification(label="cat", confidence=0.98), fo.Classification(label="tabby", confidence=0.72), ] ) print(sample) |
<Sample: {
'id': None,
'media_type': 'image',
'filepath': '/path/to/image.png',
'tags': [],
'metadata': None,
'created_at': None,
'last_modified_at': None,
'ground_truth': <Classifications: {
'classifications': [
<Classification: {
'id': '5f8708f62018186b6ef66823',
'label': 'animal',
'confidence': None,
'logits': None,
}>,
<Classification: {
'id': '5f8708f62018186b6ef66824',
'label': 'cat',
'confidence': None,
'logits': None,
}>,
<Classification: {
'id': '5f8708f62018186b6ef66825',
'label': 'tabby',
'confidence': None,
'logits': None,
}>,
],
'logits': None,
}>,
'prediction': <Classifications: {
'classifications': [
<Classification: {
'id': '5f8708f62018186b6ef66826',
'label': 'animal',
'confidence': 0.99,
'logits': None,
}>,
<Classification: {
'id': '5f8708f62018186b6ef66827',
'label': 'cat',
'confidence': 0.98,
'logits': None,
}>,
<Classification: {
'id': '5f8708f62018186b6ef66828',
'label': 'tabby',
'confidence': 0.72,
'logits': None,
}>,
],
'logits': None,
}>,
}>
Note
Did you know? You can store class lists for your models on your datasets.
Object detection¶
The Detections
class represents a list of object detections in an image. The
detections are stored in the
detections
attribute of
the Detections
object.
Each individual object detection is represented by a Detection
object. The
string label of the object should be stored in the
label
attribute, and the
bounding box for the object should be stored in the
bounding_box
attribute.
Note
FiftyOne stores box coordinates as floats in [0, 1]
relative to the
dimensions of the image. Bounding boxes are represented by a length-4 list
in the format:
[<top-left-x>, <top-left-y>, <width>, <height>]
Note
Did you know? FiftyOne also supports 3D detections!
In the case of model predictions, an optional confidence score for each
detection can be stored in the
confidence
attribute.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | import fiftyone as fo sample = fo.Sample(filepath="/path/to/image.png") sample["ground_truth"] = fo.Detections( detections=[fo.Detection(label="cat", bounding_box=[0.5, 0.5, 0.4, 0.3])] ) sample["prediction"] = fo.Detections( detections=[ fo.Detection( label="cat", bounding_box=[0.480, 0.513, 0.397, 0.288], confidence=0.96, ), ] ) print(sample) |
<Sample: {
'id': None,
'media_type': 'image',
'filepath': '/path/to/image.png',
'tags': [],
'metadata': None,
'created_at': None,
'last_modified_at': None,
'ground_truth': <Detections: {
'detections': [
<Detection: {
'id': '5f8709172018186b6ef66829',
'attributes': {},
'label': 'cat',
'bounding_box': [0.5, 0.5, 0.4, 0.3],
'mask': None,
'confidence': None,
'index': None,
}>,
],
}>,
'prediction': <Detections: {
'detections': [
<Detection: {
'id': '5f8709172018186b6ef6682a',
'attributes': {},
'label': 'cat',
'bounding_box': [0.48, 0.513, 0.397, 0.288],
'mask': None,
'confidence': 0.96,
'index': None,
}>,
],
}>,
}>
Note
Did you know? You can store class lists for your models on your datasets.
Like all Label
types, you can also add custom attributes to your detections
by dynamically adding new fields to each Detection
instance:
1 2 3 4 5 6 7 8 9 10 | import fiftyone as fo detection = fo.Detection( label="cat", bounding_box=[0.5, 0.5, 0.4, 0.3], age=51, # custom attribute mood="salty", # custom attribute ) print(detection) |
<Detection: {
'id': '60f7458c467d81f41c200551',
'attributes': {},
'tags': [],
'label': 'cat',
'bounding_box': [0.5, 0.5, 0.4, 0.3],
'mask': None,
'confidence': None,
'index': None,
'age': 51,
'mood': 'salty',
}>
Note
Did you know? You can view custom attributes in the App tooltip by hovering over the objects.
Instance segmentations¶
Object detections stored in Detections
may also have instance segmentation
masks, which should be stored in the
mask
attribute of each
Detection
.
The mask must be a 2D numpy array containing either booleans or 0/1 integers
encoding the extent of the instance mask within the
bounding_box
of the
object. The array can be of any size; it is stretched as necessary to fill the
object’s bounding box when visualizing in the App.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | import numpy as np import fiftyone as fo # Example instance mask mask = (np.random.randn(32, 32) > 0) sample = fo.Sample(filepath="/path/to/image.png") sample["prediction"] = fo.Detections( detections=[ fo.Detection( label="cat", bounding_box=[0.480, 0.513, 0.397, 0.288], mask=mask, confidence=0.96, ), ] ) print(sample) |
<Sample: {
'id': None,
'media_type': 'image',
'filepath': '/path/to/image.png',
'tags': [],
'metadata': None,
'created_at': None,
'last_modified_at': None,
'prediction': <Detections: {
'detections': [
<Detection: {
'id': '5f8709282018186b6ef6682b',
'attributes': {},
'label': 'cat',
'bounding_box': [0.48, 0.513, 0.397, 0.288],
'mask': array([[False, True, False, ..., True, True, False],
[ True, False, True, ..., False, True, True],
[False, True, False, ..., False, True, False],
...,
[ True, True, False, ..., False, False, True],
[ True, True, True, ..., True, True, False],
[False, True, True, ..., False, True, True]]),
'confidence': 0.96,
'index': None,
}>,
],
}>,
}>
Like all Label
types, you can also add custom attributes to your detections
by dynamically adding new fields to each Detection
instance:
1 2 3 4 5 6 7 8 9 10 11 12 | import numpy as np import fiftyone as fo detection = fo.Detection( label="cat", bounding_box=[0.5, 0.5, 0.4, 0.3], mask=np.random.randn(32, 32) > 0, age=51, # custom attribute mood="salty", # custom attribute ) print(detection) |
<Detection: {
'id': '60f74568467d81f41c200550',
'attributes': {},
'tags': [],
'label': 'cat',
'bounding_box': [0.5, 0.5, 0.4, 0.3],
'mask': array([[False, False, True, ..., True, True, False],
[ True, True, False, ..., True, False, True],
[False, False, True, ..., False, False, False],
...,
[False, False, True, ..., True, True, False],
[ True, False, True, ..., True, False, True],
[False, True, False, ..., True, True, True]]),
'confidence': None,
'index': None,
'age': 51,
'mood': 'salty',
}>
Note
Did you know? You can view custom attributes in the App tooltip by hovering over the objects.
Polylines and polygons¶
The Polylines
class represents a list of
polylines or
polygons in an image. The polylines
are stored in the
polylines
attribute of the
Polylines
object.
Each individual polyline is represented by a Polyline
object, which
represents a set of one or more semantically related shapes in an image. The
points
attribute contains a
list of lists of (x, y)
coordinates defining the vertices of each shape
in the polyline. If the polyline represents a closed curve, you can set the
closed
attribute to True
to
indicate that a line segment should be drawn from the last vertex to the first
vertex of each shape in the polyline. If the shapes should be filled when
rendering them, you can set the
filled
attribute to True
.
Polylines can also have string labels, which are stored in their
label
attribute.
Note
FiftyOne stores vertex coordinates as floats in [0, 1]
relative to the
dimensions of the image.
Note
Did you know? FiftyOne also supports 3D polylines!
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | import fiftyone as fo sample = fo.Sample(filepath="/path/to/image.png") # A simple polyline polyline1 = fo.Polyline( points=[[(0.3, 0.3), (0.7, 0.3), (0.7, 0.3)]], closed=False, filled=False, ) # A closed, filled polygon with a label polyline2 = fo.Polyline( label="triangle", points=[[(0.1, 0.1), (0.3, 0.1), (0.3, 0.3)]], closed=True, filled=True, ) sample["polylines"] = fo.Polylines(polylines=[polyline1, polyline2]) print(sample) |
<Sample: {
'id': None,
'media_type': 'image',
'filepath': '/path/to/image.png',
'tags': [],
'metadata': None,
'created_at': None,
'last_modified_at': None,
'polylines': <Polylines: {
'polylines': [
<Polyline: {
'id': '5f87094e2018186b6ef6682e',
'attributes': {},
'label': None,
'points': [[(0.3, 0.3), (0.7, 0.3), (0.7, 0.3)]],
'index': None,
'closed': False,
'filled': False,
}>,
<Polyline: {
'id': '5f87094e2018186b6ef6682f',
'attributes': {},
'label': 'triangle',
'points': [[(0.1, 0.1), (0.3, 0.1), (0.3, 0.3)]],
'index': None,
'closed': True,
'filled': True,
}>,
],
}>,
}>
Like all Label
types, you can also add custom attributes to your polylines by
dynamically adding new fields to each Polyline
instance:
1 2 3 4 5 6 7 8 9 10 11 | import fiftyone as fo polyline = fo.Polyline( label="triangle", points=[[(0.1, 0.1), (0.3, 0.1), (0.3, 0.3)]], closed=True, filled=True, kind="right", # custom attribute ) print(polyline) |
<Polyline: {
'id': '60f746b4467d81f41c200555',
'attributes': {},
'tags': [],
'label': 'triangle',
'points': [[(0.1, 0.1), (0.3, 0.1), (0.3, 0.3)]],
'confidence': None,
'index': None,
'closed': True,
'filled': True,
'kind': 'right',
}>
Note
Did you know? You can view custom attributes in the App tooltip by hovering over the objects.
Cuboids¶
You can store and visualize cuboids in FiftyOne using the
Polyline.from_cuboid()
method.
The method accepts a list of 8 (x, y)
points describing the vertices of the
cuboid in the format depicted below:
7--------6
/| /|
/ | / |
3--------2 |
| 4-----|--5
| / | /
|/ |/
0--------1
Note
FiftyOne stores vertex coordinates as floats in [0, 1]
relative to the
dimensions of the image.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 | import cv2 import numpy as np import fiftyone as fo def random_cuboid(frame_size): width, height = frame_size x0, y0 = np.array([width, height]) * ([0, 0.2] + 0.8 * np.random.rand(2)) dx, dy = (min(0.8 * width - x0, y0 - 0.2 * height)) * np.random.rand(2) x1, y1 = x0 + dx, y0 - dy w, h = (min(width - x1, y1)) * np.random.rand(2) front = [(x0, y0), (x0 + w, y0), (x0 + w, y0 - h), (x0, y0 - h)] back = [(x1, y1), (x1 + w, y1), (x1 + w, y1 - h), (x1, y1 - h)] vertices = front + back return fo.Polyline.from_cuboid( vertices, frame_size=frame_size, label="cuboid" ) frame_size = (256, 128) filepath = "/tmp/image.png" size = (frame_size[1], frame_size[0], 3) cv2.imwrite(filepath, np.full(size, 255, dtype=np.uint8)) dataset = fo.Dataset("cuboids") dataset.add_samples( [ fo.Sample(filepath=filepath, cuboid=random_cuboid(frame_size)) for _ in range(51)] ) session = fo.launch_app(dataset) |
Like all Label
types, you can also add custom attributes to your cuboids by
dynamically adding new fields to each Polyline
instance:
1 2 3 4 5 6 | polyline = fo.Polyline.from_cuboid( vertics, frame_size=frame_size, label="vehicle", filled=True, type="sedan", # custom attribute ) |
Note
Did you know? You can view custom attributes in the App tooltip by hovering over the objects.
Rotated bounding boxes¶
You can store and visualize rotated bounding boxes in FiftyOne using the
Polyline.from_rotated_box()
method, which accepts rotated boxes described by their center coordinates,
width/height, and counter-clockwise rotation, in radians.
Note
FiftyOne stores vertex coordinates as floats in [0, 1]
relative to the
dimensions of the image.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | import cv2 import numpy as np import fiftyone as fo def random_rotated_box(frame_size): width, height = frame_size xc, yc = np.array([width, height]) * (0.2 + 0.6 * np.random.rand(2)) w, h = 1.5 * (min(xc, yc, width - xc, height - yc)) * np.random.rand(2) theta = 2 * np.pi * np.random.rand() return fo.Polyline.from_rotated_box( xc, yc, w, h, theta, frame_size=frame_size, label="box" ) frame_size = (256, 128) filepath = "/tmp/image.png" size = (frame_size[1], frame_size[0], 3) cv2.imwrite(filepath, np.full(size, 255, dtype=np.uint8)) dataset = fo.Dataset("rotated-boxes") dataset.add_samples( [ fo.Sample(filepath=filepath, box=random_rotated_box(frame_size)) for _ in range(51) ] ) session = fo.launch_app(dataset) |
Like all Label
types, you can also add custom attributes to your rotated
bounding boxes by dynamically adding new fields to each Polyline
instance:
1 2 3 4 5 | polyline = fo.Polyline.from_rotated_box( xc, yc, width, height, theta, frame_size=frame_size, label="cat", mood="surly", # custom attribute ) |
Note
Did you know? You can view custom attributes in the App tooltip by hovering over the objects.
Keypoints¶
The Keypoints
class represents a collection of keypoint groups in an image.
The keypoint groups are stored in the
keypoints
attribute of the
Keypoints
object. Each element of this list is a Keypoint
object whose
points
attribute contains a
list of (x, y)
coordinates defining a group of semantically related
keypoints in the image.
For example, if you are working with a person model that outputs 18 keypoints
(left eye
, right eye
, nose
, etc.) per person, then each Keypoint
instance would represent one person, and a Keypoints
instance would represent
the list of people in the image.
Note
FiftyOne stores keypoint coordinates as floats in [0, 1]
relative to the
dimensions of the image.
Each Keypoint
object can have a string label, which is stored in its
label
attribute, and it can
optionally have a list of per-point confidences in [0, 1]
in its
confidence
attribute:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | import fiftyone as fo sample = fo.Sample(filepath="/path/to/image.png") sample["keypoints"] = fo.Keypoints( keypoints=[ fo.Keypoint( label="square", points=[(0.3, 0.3), (0.7, 0.3), (0.7, 0.7), (0.3, 0.7)], confidence=[0.6, 0.7, 0.8, 0.9], ) ] ) print(sample) |
<Sample: {
'id': None,
'media_type': 'image',
'filepath': '/path/to/image.png',
'tags': [],
'metadata': None,
'created_at': None,
'last_modified_at': None,
'keypoints': <Keypoints: {
'keypoints': [
<Keypoint: {
'id': '5f8709702018186b6ef66831',
'attributes': {},
'label': 'square',
'points': [(0.3, 0.3), (0.7, 0.3), (0.7, 0.7), (0.3, 0.7)],
'confidence': [0.6, 0.7, 0.8, 0.9],
'index': None,
}>,
],
}>,
}>
Like all Label
types, you can also add custom attributes to your keypoints by
dynamically adding new fields to each Keypoint
instance. As a special case,
if you add a custom list attribute to a Keypoint
instance whose length
matches the number of points, these values will be interpreted as per-point
attributes and rendered as such in the App:
1 2 3 4 5 6 7 8 9 10 11 | import fiftyone as fo keypoint = fo.Keypoint( label="rectangle", kind="square", # custom object attribute points=[(0.3, 0.3), (0.7, 0.3), (0.7, 0.7), (0.3, 0.7)], confidence=[0.6, 0.7, 0.8, 0.9], occluded=[False, False, True, False], # custom per-point attributes ) print(keypoint) |
<Keypoint: {
'id': '60f74723467d81f41c200556',
'attributes': {},
'tags': [],
'label': 'rectangle',
'points': [(0.3, 0.3), (0.7, 0.3), (0.7, 0.7), (0.3, 0.7)],
'confidence': [0.6, 0.7, 0.8, 0.9],
'index': None,
'kind': 'square',
'occluded': [False, False, True, False],
}>
If your keypoints have semantic meanings, you can store keypoint skeletons on your dataset to encode the meanings.
If you are working with keypoint skeletons and a particular point is missing or not visible for an instance, use nan values for its coordinates:
1 2 3 4 5 6 7 8 9 | keypoint = fo.Keypoint( label="rectangle", points=[ (0.3, 0.3), (float("nan"), float("nan")), # use nan to encode missing points (0.7, 0.7), (0.3, 0.7), ], ) |
Note
Did you know? When you view datasets with keypoint skeletons in the App, label strings and edges will be drawn when you visualize the keypoint fields.
Semantic segmentation¶
The Segmentation
class represents a semantic segmentation mask for an image
with integer values encoding the semantic labels for each pixel in the image.
The mask can either be stored on disk and referenced via the
mask_path
attribute or
stored directly in the database via the
mask
attribute.
Note
It is recommended to store segmentations on disk and reference them via the
mask_path
attribute,
for efficiency.
Note that mask_path
must contain the absolute path to the mask on disk in order to use the
dataset from different current working directories in the future.
Segmentation masks can be stored in either of these formats:
2D 8-bit or 16-bit images or numpy arrays
3D 8-bit RGB images or numpy arrays
Segmentation masks can have any size; they are stretched as necessary to fit the image’s extent when visualizing in the App.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | import cv2 import numpy as np import fiftyone as fo # Example segmentation mask mask_path = "/tmp/segmentation.png" mask = np.random.randint(10, size=(128, 128), dtype=np.uint8) cv2.imwrite(mask_path, mask) sample = fo.Sample(filepath="/path/to/image.png") sample["segmentation1"] = fo.Segmentation(mask_path=mask_path) sample["segmentation2"] = fo.Segmentation(mask=mask) print(sample) |
<Sample: {
'id': None,
'media_type': 'image',
'filepath': '/path/to/image.png',
'tags': [],
'metadata': None,
'created_at': None,
'last_modified_at': None,
'segmentation1': <Segmentation: {
'id': '6371d72425de9907b93b2a6b',
'tags': [],
'mask': None,
'mask_path': '/tmp/segmentation.png',
}>,
'segmentation2': <Segmentation: {
'id': '6371d72425de9907b93b2a6c',
'tags': [],
'mask': array([[8, 5, 5, ..., 9, 8, 5],
[0, 7, 8, ..., 3, 4, 4],
[5, 0, 2, ..., 0, 3, 4],
...,
[4, 4, 4, ..., 3, 6, 6],
[0, 9, 8, ..., 8, 0, 8],
[0, 6, 8, ..., 2, 9, 1]], dtype=uint8),
'mask_path': None,
}>,
}>
When you load datasets with Segmentation
fields containing 2D masks in the
App, each pixel value is rendered as a different color (if possible) from the
App’s color pool. When you view RGB segmentation masks in the App, the mask
colors are always used.
Note
Did you know? You can store semantic labels for your segmentation fields on your dataset. Then, when you view the dataset in the App, label strings will appear in the App’s tooltip when you hover over pixels.
Note
The pixel value 0
and RGB value #000000
are reserved “background”
classes that are always rendered as invisible in the App.
If mask targets are provided, all observed values not present in the targets are also rendered as invisible in the App.
Heatmaps¶
The Heatmap
class represents a continuous-valued heatmap for an image.
The map can either be stored on disk and referenced via the
map_path
attribute or stored
directly in the database via the map
attribute. When using the
map_path
attribute, heatmaps
may be 8-bit or 16-bit grayscale images. When using the
map
attribute, heatmaps should be 2D
numpy arrays. By default, the map values are assumed to be in [0, 1]
for
floating point arrays and [0, 255]
for integer-valued arrays, but you can
specify a custom [min, max]
range for a map by setting its optional
range
attribute.
Heatmaps can have any size; they are stretched as necessary to fit the image’s extent when visualizing in the App.
Note
It is recommended to store heatmaps on disk and reference them via the
map_path
attribute, for
efficiency.
Note that map_path
must contain the absolute path to the map on disk in order to use the
dataset from different current working directories in the future.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | import cv2 import numpy as np import fiftyone as fo # Example heatmap map_path = "/tmp/heatmap.png" map = np.random.randint(256, size=(128, 128), dtype=np.uint8) cv2.imwrite(map_path, map) sample = fo.Sample(filepath="/path/to/image.png") sample["heatmap1"] = fo.Heatmap(map_path=map_path) sample["heatmap2"] = fo.Heatmap(map=map) print(sample) |
<Sample: {
'id': None,
'media_type': 'image',
'filepath': '/path/to/image.png',
'tags': [],
'metadata': None,
'created_at': None,
'last_modified_at': None,
'heatmap1': <Heatmap: {
'id': '6371d9e425de9907b93b2a6f',
'tags': [],
'map': None,
'map_path': '/tmp/heatmap.png',
'range': None,
}>,
'heatmap2': <Heatmap: {
'id': '6371d9e425de9907b93b2a70',
'tags': [],
'map': array([[179, 249, 119, ..., 94, 213, 68],
[190, 202, 209, ..., 162, 16, 39],
[252, 251, 181, ..., 221, 118, 231],
...,
[ 12, 91, 201, ..., 14, 95, 88],
[164, 118, 171, ..., 21, 170, 5],
[232, 156, 218, ..., 224, 97, 65]], dtype=uint8),
'map_path': None,
'range': None,
}>,
}>
When visualizing heatmaps in the App, when the App is
in color-by-field mode, heatmaps are rendered in their field’s color with
opacity proportional to the magnitude of the heatmap’s values. For example, for
a heatmap whose range
is
[-10, 10]
, pixels with the value +9 will be rendered with 90% opacity, and
pixels with the value -3 will be rendered with 30% opacity.
When the App is in color-by-value mode, heatmaps are rendered using the
colormap defined by the colorscale
of your
App config, which can be:
The string name of any colorscale recognized by Plotly
A manually-defined colorscale like the following:
[ [0.000, "rgb(165,0,38)"], [0.111, "rgb(215,48,39)"], [0.222, "rgb(244,109,67)"], [0.333, "rgb(253,174,97)"], [0.444, "rgb(254,224,144)"], [0.555, "rgb(224,243,248)"], [0.666, "rgb(171,217,233)"], [0.777, "rgb(116,173,209)"], [0.888, "rgb(69,117,180)"], [1.000, "rgb(49,54,149)"], ]
The example code below demonstrates the possibilities that heatmaps provide by overlaying random gaussian kernels with positive or negative sign on an image dataset and configuring the App’s colorscale in various ways on-the-fly:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | import os import numpy as np import fiftyone as fo import fiftyone.zoo as foz def random_kernel(metadata): h = metadata.height // 2 w = metadata.width // 2 sign = np.sign(np.random.randn()) x, y = np.meshgrid(np.linspace(-1, 1, w), np.linspace(-1, 1, h)) x0, y0 = np.random.random(2) - 0.5 kernel = sign * np.exp(-np.sqrt((x - x0) ** 2 + (y - y0) ** 2)) return fo.Heatmap(map=kernel, range=[-1, 1]) dataset = foz.load_zoo_dataset("quickstart").select_fields().clone() dataset.compute_metadata() for sample in dataset: heatmap = random_kernel(sample.metadata) # Convert to on-disk map_path = os.path.join("/tmp/heatmaps", os.path.basename(sample.filepath)) heatmap.export_map(map_path, update=True) sample["heatmap"] = heatmap sample.save() session = fo.launch_app(dataset) |
1 2 3 | # Select `Settings -> Color by value` in the App # Heatmaps will now be rendered using your default colorscale (printed below) print(session.config.colorscale) |
1 2 3 | # Switch to a different named colorscale session.config.colorscale = "RdBu" session.refresh() |
1 2 3 4 5 6 7 8 9 10 | # Switch to a custom colorscale session.config.colorscale = [ [0.00, "rgb(166,206,227)"], [0.25, "rgb(31,120,180)"], [0.45, "rgb(178,223,138)"], [0.65, "rgb(51,160,44)"], [0.85, "rgb(251,154,153)"], [1.00, "rgb(227,26,28)"], ] session.refresh() |
Temporal detection¶
The TemporalDetection
class represents an event occurring during a specified
range of frames in a video.
The label
attribute
stores the detection label, and the
support
attribute
stores the [first, last]
frame range of the detection in the video.
The optional
confidence
attribute can be used to store a model prediction score, and you can add
custom attributes as well, which can be visualized in the
App.
1 2 3 4 5 6 | import fiftyone as fo sample = fo.Sample(filepath="/path/to/video.mp4") sample["events"] = fo.TemporalDetection(label="meeting", support=[10, 20]) print(sample) |
<Sample: {
'id': None,
'media_type': 'video',
'filepath': '/path/to/video.mp4',
'tags': [],
'metadata': None,
'created_at': None,
'last_modified_at': None,
'events': <TemporalDetection: {
'id': '61321c8ea36cb17df655f44f',
'tags': [],
'label': 'meeting',
'support': [10, 20],
'confidence': None,
}>,
'frames': <Frames: 0>,
}>
If your temporal detection data is represented as timestamps in seconds, you
can use the
from_timestamps()
factory method to perform the necessary conversion to frames automatically
based on the sample’s video metadata:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | import fiftyone as fo import fiftyone.zoo as foz # Download a video to work with dataset = foz.load_zoo_dataset("quickstart-video", max_samples=1) filepath = dataset.first().filepath sample = fo.Sample(filepath=filepath) sample.compute_metadata() sample["events"] = fo.TemporalDetection.from_timestamps( [1, 2], label="meeting", sample=sample ) print(sample) |
<Sample: {
'id': None,
'media_type': 'video',
'filepath': '~/fiftyone/quickstart-video/data/Ulcb3AjxM5g_053-1.mp4',
'tags': [],
'metadata': <VideoMetadata: {
'size_bytes': 1758809,
'mime_type': 'video/mp4',
'frame_width': 1920,
'frame_height': 1080,
'frame_rate': 29.97002997002997,
'total_frame_count': 120,
'duration': 4.004,
'encoding_str': 'avc1',
}>,
'created_at': None,
'last_modified_at': None,
'events': <TemporalDetection: {
'id': '61321e498d5f587970b29183',
'tags': [],
'label': 'meeting',
'support': [31, 60],
'confidence': None,
}>,
'frames': <Frames: 0>,
}>
The TemporalDetections
class holds a list of temporal detections for a
sample:
1 2 3 4 5 6 7 8 9 10 11 | import fiftyone as fo sample = fo.Sample(filepath="/path/to/video.mp4") sample["events"] = fo.TemporalDetections( detections=[ fo.TemporalDetection(label="meeting", support=[10, 20]), fo.TemporalDetection(label="party", support=[30, 60]), ] ) print(sample) |
<Sample: {
'id': None,
'media_type': 'video',
'filepath': '/path/to/video.mp4',
'tags': [],
'metadata': None,
'created_at': None,
'last_modified_at': None,
'events': <TemporalDetections: {
'detections': [
<TemporalDetection: {
'id': '61321ed78d5f587970b29184',
'tags': [],
'label': 'meeting',
'support': [10, 20],
'confidence': None,
}>,
<TemporalDetection: {
'id': '61321ed78d5f587970b29185',
'tags': [],
'label': 'party',
'support': [30, 60],
'confidence': None,
}>,
],
}>,
'frames': <Frames: 0>,
}>
Note
Did you know? You can store class lists for your models on your datasets.
3D detections¶
The App’s 3D visualizer supports rendering 3D object
detections represented as Detection
instances with their label
, location
,
dimensions
, and rotation
attributes populated as shown below:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | import fiftyone as fo # Object label label = "vehicle" # Object center `[x, y, z]` in scene coordinates location = [0.47, 1.49, 69.44] # Object dimensions `[x, y, z]` in scene units dimensions = [2.85, 2.63, 12.34] # Object rotation `[x, y, z]` around its center, in `[-pi, pi]` rotation = [0, -1.56, 0] # A 3D object detection detection = fo.Detection( label=label, location=location, dimensions=dimensions, rotation=rotation, ) |
Note
Did you know? You can view custom attributes in the App tooltip by hovering over the objects.
3D polylines¶
The App’s 3D visualizer supports rendering 3D
polylines represented as Polyline
instances with their label
and points3d
attributes populated as shown below:
1 2 3 4 5 6 7 8 9 10 11 | import fiftyone as fo # Object label label = "lane" # A list of lists of `[x, y, z]` points in scene coordinates describing # the vertices of each shape in the polyline points3d = [[[-5, -99, -2], [-8, 99, -2]], [[4, -99, -2], [1, 99, -2]]] # A set of semantically related 3D polylines polyline = fo.Polyline(label=label, points3d=points3d) |
Note
Did you know? You can view custom attributes in the App tooltip by hovering over the objects.
Geolocation¶
The GeoLocation
class can store single pieces of location data in its
properties:
point
: a[longitude, latitude]
pointline
: a line of longitude and latitude coordinates stored in the following format:[[lon1, lat1], [lon2, lat2], ...]
polygon
: a polygon of longitude and latitude coordinates stored in the format below, where the first element describes the boundary of the polygon and any remaining entries describe holes:[ [[lon1, lat1], [lon2, lat2], ...], [[lon1, lat1], [lon2, lat2], ...], ... ]
Note
All geolocation coordinates are stored in [longitude, latitude]
format.
If you have multiple geometries of each type that you wish to store on a single
sample, then you can use the GeoLocations
class and its appropriate
properties to do so.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | import fiftyone as fo sample = fo.Sample(filepath="/path/to/image.png") sample["location"] = fo.GeoLocation( point=[-73.9855, 40.7580], polygon=[ [ [-73.949701, 40.834487], [-73.896611, 40.815076], [-73.998083, 40.696534], [-74.031751, 40.715273], [-73.949701, 40.834487], ] ], ) print(sample) |
<Sample: {
'id': None,
'media_type': 'image',
'filepath': '/path/to/image.png',
'tags': [],
'metadata': None,
'created_at': None,
'last_modified_at': None,
'location': <GeoLocation: {
'id': '60481f3936dc48428091e926',
'tags': [],
'point': [-73.9855, 40.758],
'line': None,
'polygon': [
[
[-73.949701, 40.834487],
[-73.896611, 40.815076],
[-73.998083, 40.696534],
[-74.031751, 40.715273],
[-73.949701, 40.834487],
],
],
}>,
}>
Note
Did you know? You can create location-based views that filter your data by their location!
All location data is stored in GeoJSON format in the database. You can easily retrieve the raw GeoJSON data for a slice of your dataset using the values() aggregation:
1 2 3 4 5 6 7 | import fiftyone as fo import fiftyone.zoo as foz dataset = foz.load_zoo_dataset("quickstart-geo") values = dataset.take(5).values("location.point", _raw=True) print(values) |
[{'type': 'Point', 'coordinates': [-73.9592175465766, 40.71052995514191]},
{'type': 'Point', 'coordinates': [-73.97748118760413, 40.74660360881843]},
{'type': 'Point', 'coordinates': [-73.9508690871987, 40.766631164626]},
{'type': 'Point', 'coordinates': [-73.96569416502996, 40.75449283200206]},
{'type': 'Point', 'coordinates': [-73.97397106211423, 40.67925541341504]}]
Label tags¶
All Label
instances have a tags
field, which is a string list. By default,
this list is empty, but you can use it to store application-specific
information like whether the label is incorrect:
1 2 3 4 5 6 | detection = fo.Detection(label="cat", bounding_box=[0, 0, 1, 1]) detection.tags.append("mistake") print(detection.tags) # ["mistake"] |
Note
Did you know? You can add, edit, and filter by label tags directly in the App.
Datasets and views provide helpful methods such as
count_label_tags()
,
tag_labels()
,
untag_labels()
,
match_labels()
,
and
select_labels()
that you can use to perform batch queries and edits to label tags:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | import fiftyone as fo import fiftyone.zoo as foz from fiftyone import ViewField as F dataset = foz.load_zoo_dataset("quickstart").clone() # Tag all low confidence prediction view = dataset.filter_labels("predictions", F("confidence") < 0.1) view.tag_labels("potential_mistake", label_fields="predictions") print(dataset.count_label_tags()) # {'potential_mistake': 1555} # Create a view containing only tagged labels view = dataset.select_labels(tags="potential_mistake", fields="predictions") print(len(view)) # 173 print(view.count("predictions.detections")) # 1555 # Create a view containing only samples with at least one tagged label view = dataset.match_labels(tags="potential_mistake", fields="predictions") print(len(view)) # 173 print(view.count("predictions.detections")) # 5151 dataset.untag_labels("potential_mistake", label_fields="predictions") print(dataset.count_label_tags()) # {} |
Label attributes¶
The Detection
, Polyline
, and Keypoint
label types have an optional
attributes
field that you
can use to store custom attributes on the object.
The attributes
field is a
dictionary mapping attribute names to Attribute
instances, which contain the
value
of the attribute and any
associated metadata.
Warning
The attributes
field
will be removed in an upcoming release.
Instead, add custom attributes directly to your
Label
objects:
detection = fo.Detection(label="cat", bounding_box=[0.1, 0.1, 0.8, 0.8])
detection["custom_attribute"] = 51
# Equivalent
detection = fo.Detection(
label="cat",
bounding_box=[0.1, 0.1, 0.8, 0.8],
custom_attribute=51,
)
There are Attribute
subclasses for various types of attributes you may want
to store. Use the appropriate subclass when possible so that FiftyOne knows the
schema of the attributes that you’re storing.
Attribute class |
Value type |
Description |
---|---|---|
|
A boolean attribute |
|
|
A categorical attribute |
|
|
A numeric attribute |
|
|
A list attribute |
|
arbitrary |
A generic attribute of any type |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 | import fiftyone as fo sample = fo.Sample(filepath="/path/to/image.png") sample["ground_truth"] = fo.Detections( detections=[ fo.Detection( label="cat", bounding_box=[0.5, 0.5, 0.4, 0.3], attributes={ "age": fo.NumericAttribute(value=51), "mood": fo.CategoricalAttribute(value="salty"), }, ), ] ) sample["prediction"] = fo.Detections( detections=[ fo.Detection( label="cat", bounding_box=[0.480, 0.513, 0.397, 0.288], confidence=0.96, attributes={ "age": fo.NumericAttribute(value=51), "mood": fo.CategoricalAttribute( value="surly", confidence=0.95 ), }, ), ] ) print(sample) |
<Sample: {
'id': None,
'media_type': 'image',
'filepath': '/path/to/image.png',
'tags': [],
'metadata': None,
'created_at': None,
'last_modified_at': None,
'ground_truth': <Detections: {
'detections': [
<Detection: {
'id': '60f738e7467d81f41c20054c',
'attributes': {
'age': <NumericAttribute: {'value': 51}>,
'mood': <CategoricalAttribute: {
'value': 'salty', 'confidence': None, 'logits': None
}>,
},
'tags': [],
'label': 'cat',
'bounding_box': [0.5, 0.5, 0.4, 0.3],
'mask': None,
'confidence': None,
'index': None,
}>,
],
}>,
'prediction': <Detections: {
'detections': [
<Detection: {
'id': '60f738e7467d81f41c20054d',
'attributes': {
'age': <NumericAttribute: {'value': 51}>,
'mood': <CategoricalAttribute: {
'value': 'surly', 'confidence': 0.95, 'logits': None
}>,
},
'tags': [],
'label': 'cat',
'bounding_box': [0.48, 0.513, 0.397, 0.288],
'mask': None,
'confidence': 0.96,
'index': None,
}>,
],
}>,
}>
Note
Did you know? You can view attribute values in the App tooltip by hovering over the objects.
Converting label types¶
FiftyOne provides a number of utility methods to convert between different representations of certain label types, such as converting between instance segmentations, semantic segmentations, and polylines.
Let’s load some instance segmentations from the COCO dataset to see this in action:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | import fiftyone as fo import fiftyone.zoo as foz dataset = foz.load_zoo_dataset( "coco-2017", split="validation", label_types=["segmentations"], classes=["cat", "dog"], label_field="instances", max_samples=25, only_matching=True, ) sample = dataset.first() detections = sample["instances"] |
For example, you can use
Detections.to_polylines()
to convert instance segmentations to polylines:
1 2 3 | # Convert `Detections` to `Polylines` polylines = detections.to_polylines(tolerance=2) print(polylines) |
Or you can use
Detections.to_segmentation()
to convert instance segmentations to semantic segmentation masks:
1 2 3 4 5 6 7 8 9 10 11 12 | metadata = fo.ImageMetadata.build_for(sample.filepath) # Convert `Detections` to `Segmentation` segmentation = detections.to_segmentation( frame_size=(metadata.width, metadata.height), mask_targets={1: "cat", 2: "dog"}, ) # Export the segmentation to disk segmentation.export_mask("/tmp/mask.png", update=True) print(segmentation) |
Methods such as
Segmentation.to_detections()
and Segmentation.to_polylines()
also exist to transform semantic segmentations back into individual shapes.
In addition, the fiftyone.utils.labels
module contains a variety of
utility methods for converting entire collections’ labels between common
formats:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 | import fiftyone.utils.labels as foul # Convert instance segmentations to semantic segmentations stored on disk foul.objects_to_segmentations( dataset, "instances", "segmentations", output_dir="/tmp/segmentations", mask_targets={1: "cat", 2: "dog"}, ) # Convert instance segmentations to polylines format foul.instances_to_polylines(dataset, "instances", "polylines", tolerance=2) # Convert semantic segmentations to instance segmentations foul.segmentations_to_detections( dataset, "segmentations", "instances2", mask_targets={1: "cat", 2: "dog"}, mask_types="thing", # give each connected region a separate instance ) print(dataset) |
Name: coco-2017-validation-25
Media type: image
Num samples: 25
Persistent: False
Tags: []
Sample fields:
id: fiftyone.core.fields.ObjectIdField
filepath: fiftyone.core.fields.StringField
tags: fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)
metadata: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.ImageMetadata)
created_at: fiftyone.core.fields.DateTimeField
last_modified_at: fiftyone.core.fields.DateTimeField
instances: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)
segmentations: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Segmentation)
polylines: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Polylines)
instances2: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)
Note that, if your goal is to export the labels to disk, FiftyOne can
automatically coerce the labels into the correct
format based on the type of the label_field
and the dataset_type
that you
specify for the export without explicitly storing the transformed labels as a
new field on your dataset:
1 2 3 4 5 6 7 8 | # Export the instance segmentations in the `instances` field as semantic # segmentation images on disk dataset.export( label_field="instances", dataset_type=fo.types.ImageSegmentationDirectory, labels_path="/tmp/masks", mask_targets={1: "cat", 2: "dog"}, ) |
Dynamic attributes¶
Any field(s) of your FiftyOne datasets that contain DynamicEmbeddedDocument
values can have arbitrary custom attributes added to their instances.
For example, all Label classes and Metadata classes are dynamic, so you can add custom attributes to them as follows:
1 2 3 4 5 6 7 8 9 | # Provide some default attributes label = fo.Classification(label="cat", confidence=0.98) # Add custom attributes label["int"] = 5 label["float"] = 51.0 label["list"] = [1, 2, 3] label["bool"] = True label["dict"] = {"key": ["list", "of", "values"]} |
By default, dynamic attributes are not included in a dataset’s schema, which means that these attributes may contain arbitrary heterogeneous values across the dataset’s samples.
However, FiftyOne provides methods that you can use to formally declare custom dynamic attributes, which allows you to enforce type constraints, filter by these custom attributes in the App, and more.
You can use
get_dynamic_field_schema()
to detect the names and type(s) of any undeclared dynamic embedded document
attributes on a dataset:
1 2 3 4 5 6 | import fiftyone as fo import fiftyone.zoo as foz dataset = foz.load_zoo_dataset("quickstart") print(dataset.get_dynamic_field_schema()) |
{
'ground_truth.detections.iscrowd': <fiftyone.core.fields.FloatField>,
'ground_truth.detections.area': <fiftyone.core.fields.FloatField>,
}
You can then use
add_sample_field()
to
declare a specific dynamic embedded document attribute:
1 | dataset.add_sample_field("ground_truth.detections.iscrowd", fo.FloatField) |
or you can use the
add_dynamic_sample_fields()
method to declare all dynamic embedded document attribute(s) that contain
values of a single type:
1 | dataset.add_dynamic_sample_fields() |
Note
Pass the add_mixed=True
option to
add_dynamic_sample_fields()
if you wish to declare all dynamic attributes that contain mixed values
using a generic Field
type.
You can provide the optional flat=True
option to
get_field_schema()
to
retrieve a flattened version of a dataset’s schema that includes all embedded
document attributes as top-level keys:
1 | print(dataset.get_field_schema(flat=True)) |
{
'id': <fiftyone.core.fields.ObjectIdField>,
'filepath': <fiftyone.core.fields.StringField>,
'tags': <fiftyone.core.fields.ListField>,
'metadata': <fiftyone.core.fields.EmbeddedDocumentField>,
'metadata.size_bytes': <fiftyone.core.fields.IntField>,
'metadata.mime_type': <fiftyone.core.fields.StringField>,
'metadata.width': <fiftyone.core.fields.IntField>,
'metadata.height': <fiftyone.core.fields.IntField>,
'metadata.num_channels': <fiftyone.core.fields.IntField>,
'created_at': <fiftyone.core.fields.DateTimeField object at 0x7fea584bc730>,
'last_modified_at': <fiftyone.core.fields.DateTimeField object at 0x7fea584bc280>,
'ground_truth': <fiftyone.core.fields.EmbeddedDocumentField>,
'ground_truth.detections': <fiftyone.core.fields.ListField>,
'ground_truth.detections.id': <fiftyone.core.fields.ObjectIdField>,
'ground_truth.detections.tags': <fiftyone.core.fields.ListField>,
...
'ground_truth.detections.iscrowd': <fiftyone.core.fields.FloatField>,
'ground_truth.detections.area': <fiftyone.core.fields.FloatField>,
...
}
By default, dynamic attributes are not declared on a dataset’s schema when samples are added to it:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 | import fiftyone as fo sample = fo.Sample( filepath="/path/to/image.jpg", ground_truth=fo.Detections( detections=[ fo.Detection( label="cat", bounding_box=[0.1, 0.1, 0.4, 0.4], mood="surly", ), fo.Detection( label="dog", bounding_box=[0.5, 0.5, 0.4, 0.4], mood="happy", ) ] ) ) dataset = fo.Dataset() dataset.add_sample(sample) schema = dataset.get_field_schema(flat=True) assert "ground_truth.detections.mood" not in schema |
However, methods such as
add_sample()
,
add_samples()
,
add_dir()
,
from_dir()
, and
merge_samples()
provide an optional dynamic=True
option that you can provide to automatically
declare any dynamic embedded document attributes encountered while importing
data:
1 2 3 4 5 6 | dataset = fo.Dataset() dataset.add_sample(sample, dynamic=True) schema = dataset.get_field_schema(flat=True) assert "ground_truth.detections.mood" in schema |
Note that, when declaring dynamic attributes on non-empty datasets, you must
ensure that the attribute’s type is consistent with any existing values in that
field, e.g., by first running
get_dynamic_field_schema()
to check the existing type(s). Methods like
add_sample_field()
and
add_samples(..., dynamic=True)
do not validate newly declared field’s types against existing field values:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 | import fiftyone as fo sample1 = fo.Sample( filepath="/path/to/image1.jpg", ground_truth=fo.Classification( label="cat", mood="surly", age="bad-value", ), ) sample2 = fo.Sample( filepath="/path/to/image2.jpg", ground_truth=fo.Classification( label="dog", mood="happy", age=5, ), ) dataset = fo.Dataset() dataset.add_sample(sample1) # Either of these are problematic dataset.add_sample(sample2, dynamic=True) dataset.add_sample_field("ground_truth.age", fo.IntField) sample1.reload() # ValidationError: bad-value could not be converted to int |
If you declare a dynamic attribute with a type that is not compatible with
existing values in that field, you will need to remove that field from the
dataset’s schema using
remove_dynamic_sample_field()
in order for the dataset to be usable again:
1 2 | # Removes dynamic field from dataset's schema without deleting the values dataset.remove_dynamic_sample_field("ground_truth.age") |
You can use
select_fields()
and
exclude_fields()
to create views that select/exclude specific dynamic
attributes from your dataset and its schema:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | dataset.add_sample_field("ground_truth.age", fo.Field) sample = dataset.first() assert "ground_truth.age" in dataset.get_field_schema(flat=True) assert sample.ground_truth.has_field("age") # Omits the `age` attribute from the `ground_truth` field view = dataset.exclude_fields("ground_truth.age") sample = view.first() assert "ground_truth.age" not in view.get_field_schema(flat=True) assert not sample.ground_truth.has_field("age") # Only include `mood` (and default) attributes of the `ground_truth` field view = dataset.select_fields("ground_truth.mood") sample = view.first() assert "ground_truth.age" not in view.get_field_schema(flat=True) assert not sample.ground_truth.has_field("age") |
Custom embedded documents¶
If you work with collections of related fields that you would like to organize
under a single top-level field, you can achieve this by defining and using
custom EmbeddedDocument
and DynamicEmbeddedDocument
classes to populate
your datasetes.
Using custom embedded document classes enables you to access your data using the same object-oriented interface enjoyed by FiftyOne’s builtin label types.
The EmbeddedDocument
class represents a fixed collection of fields with
predefined types and optional default values, while the
DynamicEmbeddedDocument
class supports predefined fields but also allows
users to populate arbitrary custom fields at runtime, like FiftyOne’s
builtin label types.
Defining custom documents on-the-fly¶
The simplest way to define custom embedded documents on your datasets is to
declare empty DynamicEmbeddedDocument
field(s) and then incrementally
populate new dynamic attributes as needed.
To illustrate, let’s start by defining an empty embedded document field:
1 2 3 4 5 6 7 8 9 10 | import fiftyone as fo dataset = fo.Dataset() # Define an empty embedded document field dataset.add_sample_field( "camera_info", fo.EmbeddedDocumentField, embedded_doc_type=fo.DynamicEmbeddedDocument, ) |
From here, there are a variety of ways to add new embedded attributes to the field.
You can explicitly declare new fields using
add_sample_field()
:
1 2 3 4 | # Declare a new `camera_id` attribute dataset.add_sample_field("camera_info.camera_id", fo.StringField) assert "camera_info.camera_id" in dataset.get_field_schema(flat=True) |
or you can implicitly declare new fields using
add_samples()
with the
dynamic=True
flag:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | # Includes a new `quality` attribute sample1 = fo.Sample( filepath="/path/to/image1.jpg", camera_info=fo.DynamicEmbeddedDocument( camera_id="123456789", quality=51.0, ), ) sample2 = fo.Sample( filepath="/path/to/image2.jpg", camera_info=fo.DynamicEmbeddedDocument(camera_id="123456789"), ) # Automatically declares new dynamic attributes as they are encountered dataset.add_samples([sample1, sample2], dynamic=True) assert "camera_info.quality" in dataset.get_field_schema(flat=True) |
or you can implicitly declare new fields using
set_values()
with the dynamic=True
flag:
1 2 3 4 | # Populate a new `description` attribute on each sample in the dataset dataset.set_values("camera_info.description", ["foo", "bar"], dynamic=True) assert "camera_info.description" in dataset.get_field_schema(flat=True) |
Defining custom documents in modules¶
You can also define custom embedded document classes in Python modules and
packages that you maintain, using the appropriate types from the
fiftyone.core.fields
module to declare your fields and their types,
defaults, etc.
The benefit of this approach over the on-the-fly definition from the previous
section is that you can provide extra metadata such as whether fields are
required
or should have default
values if they are not explicitly set
during creation.
Warning
In order to work with datasets containing custom embedded documents defined
using this approach, you must configure your module_path
in
all environments where you intend to work with the datasets that use
these classes, not just the environment where you create the dataset.
To avoid this requirement, consider defining custom documents on-the-fly instead.
For example, suppose you add the following embedded document classes to a
foo.bar
module:
1 2 3 4 5 6 7 8 9 10 11 12 | from datetime import datetime import fiftyone as fo class CameraInfo(fo.EmbeddedDocument): camera_id = fo.StringField(required=True) quality = fo.FloatField() description = fo.StringField() class LabelMetadata(fo.DynamicEmbeddedDocument): created_at = fo.DateTimeField(default=datetime.utcnow) model_name = fo.StringField() |
and then foo.bar
to FiftyOne’s module_path
config setting (see
this page for more ways to register this):
export FIFTYONE_MODULE_PATH=foo.bar
# Verify module path
fiftyone config
You’re now free to use your custom embedded document classes as you please, whether this be top-level sample fields or nested fields:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 | import fiftyone as fo import foo.bar as fb sample = fo.Sample( filepath="/path/to/image.png", camera_info=fb.CameraInfo( camera_id="123456789", quality=99.0, ), weather=fo.Classification( label="sunny", confidence=0.95, metadata=fb.LabelMetadata( model_name="resnet50", description="A dynamic field", ) ), ) dataset = fo.Dataset() dataset.add_sample(sample) dataset.name = "test" dataset.persistent = True |
As long as foo.bar
is on your module_path
, this dataset can be loaded in
future sessions and manipulated as usual:
1 2 3 4 | import fiftyone as fo dataset = fo.load_dataset("test") print(dataset.first()) |
<Sample: {
'id': '6217b696d181786cff360740',
'media_type': 'image',
'filepath': '/path/to/image.png',
'tags': [],
'metadata': None,
'created_at': datetime.datetime(2024, 7, 22, 5, 16, 10, 701907),
'last_modified_at': datetime.datetime(2024, 7, 22, 5, 16, 10, 701907),
'camera_info': <CameraInfo: {
'camera_id': '123456789',
'quality': 99.0,
'description': None,
}>,
'weather': <Classification: {
'id': '6217b696d181786cff36073e',
'tags': [],
'label': 'sunny',
'confidence': 0.95,
'logits': None,
'metadata': <LabelMetadata: {
'created_at': datetime.datetime(2022, 2, 24, 16, 47, 18, 10000),
'model_name': 'resnet50',
'description': 'A dynamic field',
}>,
}>,
}>
Image datasets¶
Any Sample
whose filepath
is a file with MIME type image/*
is recognized
as a image sample, and datasets composed of image samples have media type
image
:
1 2 3 4 5 6 7 8 9 | import fiftyone as fo sample = fo.Sample(filepath="/path/to/image.png") dataset = fo.Dataset() dataset.add_sample(sample) print(dataset.media_type) # image print(sample) |
<Sample: {
'id': '6655ca275e20e244f2c8fe31',
'media_type': 'image',
'filepath': '/path/to/image.png',
'tags': [],
'metadata': None,
'created_at': datetime.datetime(2024, 7, 22, 5, 15, 8, 122038),
'last_modified_at': datetime.datetime(2024, 7, 22, 5, 15, 8, 122038),
}>
Example image dataset¶
To get started exploring image datasets, try loading the quickstart dataset from the zoo:
1 2 3 4 5 6 7 8 9 10 11 | import fiftyone as fo import fiftyone.zoo as foz dataset = foz.load_zoo_dataset("quickstart") print(dataset.count("ground_truth.detections")) # 1232 print(dataset.count("predictions.detections")) # 5620 print(dataset.count_values("ground_truth.detections.label")) # {'dog': 15, 'airplane': 24, 'dining table': 15, 'hot dog': 5, ...} session = fo.launch_app(dataset) |
Video datasets¶
Any Sample
whose filepath
is a file with MIME type video/*
is recognized
as a video sample, and datasets composed of video samples have media type
video
:
1 2 3 4 5 6 7 8 9 | import fiftyone as fo sample = fo.Sample(filepath="/path/to/video.mp4") dataset = fo.Dataset() dataset.add_sample(sample) print(dataset.media_type) # video print(sample) |
<Sample: {
'id': '6403ccef0a3af5bc780b5a10',
'media_type': 'video',
'filepath': '/path/to/video.mp4',
'tags': [],
'metadata': None,
'created_at': datetime.datetime(2024, 7, 22, 5, 3, 17, 229263),
'last_modified_at': datetime.datetime(2024, 7, 22, 5, 3, 17, 229263),
'frames': <Frames: 0>,
}>
All video samples have a reserved frames
attribute in which you can store
frame-level labels and other custom annotations for the video. The frames
attribute is a dictionary whose keys are frame numbers and whose values are
Frame
instances that hold all of the Label
instances and other
primitive-type fields for the frame.
Note
FiftyOne uses 1-based indexing for video frame numbers.
You can add, modify, and delete labels of any type as well as primitive fields such as integers, strings, and booleans using the same dynamic attribute syntax that you use to interact with samples:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | frame = fo.Frame( quality=97.12, weather=fo.Classification(label="sunny"), objects=fo.Detections( detections=[ fo.Detection(label="cat", bounding_box=[0.1, 0.1, 0.2, 0.2]), fo.Detection(label="dog", bounding_box=[0.7, 0.7, 0.2, 0.2]), ] ) ) # Add labels to the first frame of the video sample.frames[1] = frame sample.save() |
Note
You must call sample.save()
in
order to persist changes to the database when editing video samples and/or
their frames that are in datasets.
<Sample: {
'id': '6403ccef0a3af5bc780b5a10',
'media_type': 'video',
'filepath': '/path/to/video.mp4',
'tags': [],
'metadata': None,
'created_at': datetime.datetime(2024, 7, 22, 5, 3, 17, 229263),
'last_modified_at': datetime.datetime(2024, 7, 22, 5, 3, 17, 229263),
'frames': <Frames: 1>, <-- `frames` now contains 1 frame of labels
}>
Note
The frames
attribute of video samples behaves like a defaultdict; a new
Frame
will be created if the frame number does not exist when you access
it.
You can iterate over the frames in a video sample using the expected syntax:
1 2 | for frame_number, frame in sample.frames.items(): print(frame) |
<Frame: {
'id': '6403cd972a54cee076f88bd2',
'frame_number': 1,
'created_at': datetime.datetime(2024, 7, 22, 5, 3, 40, 839000),
'last_modified_at': datetime.datetime(2024, 7, 22, 5, 3, 40, 839000),
'quality': 97.12,
'weather': <Classification: {
'id': '609078d54653b0094e9baa52',
'tags': [],
'label': 'sunny',
'confidence': None,
'logits': None,
}>,
'objects': <Detections: {
'detections': [
<Detection: {
'id': '609078d54653b0094e9baa53',
'attributes': {},
'tags': [],
'label': 'cat',
'bounding_box': [0.1, 0.1, 0.2, 0.2],
'mask': None,
'confidence': None,
'index': None,
}>,
<Detection: {
'id': '609078d54653b0094e9baa54',
'attributes': {},
'tags': [],
'label': 'dog',
'bounding_box': [0.7, 0.7, 0.2, 0.2],
'mask': None,
'confidence': None,
'index': None,
}>,
],
}>,
}>
Notice that the dataset’s summary indicates that the dataset has media type
video
and includes the schema of any frame fields you add:
1 | print(dataset) |
Name: 2021.05.03.18.30.20
Media type: video
Num samples: 1
Persistent: False
Tags: []
Sample fields:
id: fiftyone.core.fields.ObjectIdField
filepath: fiftyone.core.fields.StringField
tags: fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)
metadata: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.VideoMetadata)
created_at: fiftyone.core.fields.DateTimeField
last_modified_at: fiftyone.core.fields.DateTimeField
Frame fields:
id: fiftyone.core.fields.ObjectIdField
frame_number: fiftyone.core.fields.FrameNumberField
created_at: fiftyone.core.fields.DateTimeField
last_modified_at: fiftyone.core.fields.DateTimeField
quality: fiftyone.core.fields.FloatField
weather: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Classification)
objects: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)
You can retrieve detailed information about the schema of the frames of a
video Dataset
using
dataset.get_frame_field_schema()
.
The samples in video datasets can be accessed
like usual, and the sample’s frame
labels can be modified by updating the frames
attribute of a Sample
:
1 2 3 4 5 6 7 8 9 | sample = dataset.first() for frame_number, frame in sample.frames.items(): frame["frame_str"] = str(frame_number) del frame["weather"] del frame["objects"] sample.save() print(sample.frames[1]) |
<Frame: {
'id': '6403cd972a54cee076f88bd2',
'frame_number': 1,
'created_at': datetime.datetime(2024, 7, 22, 5, 3, 40, 839000),
'last_modified_at': datetime.datetime(2024, 7, 22, 5, 4, 49, 430051),
'quality': 97.12,
'weather': None,
'objects': None,
'frame_str': '1',
}>
Note
You must call sample.save()
in
order to persist changes to the database when editing video samples and/or
their frames that are in datasets.
See this page for more information about building labeled video samples.
Example video dataset¶
To get started exploring video datasets, try loading the quickstart-video dataset from the zoo:
1 2 3 4 5 6 7 8 9 10 11 | import fiftyone as fo import fiftyone.zoo as foz dataset = foz.load_zoo_dataset("quickstart-video") print(dataset.count("frames")) # 1279 print(dataset.count("frames.detections.detections")) # 11345 print(dataset.count_values("frames.detections.detections.label")) # {'vehicle': 7511, 'road sign': 2726, 'person': 1108} session = fo.launch_app(dataset) |
3D datasets¶
Any Sample
whose filepath
is a file with extension .fo3d
is
recognized as a 3D sample, and datasets composed of 3D
samples have media type 3d
.
An FO3D file encapsulates a 3D scene constructed using the
Scene
class, which provides methods
to add, remove, and manipulate 3D objects in the scene. A scene is
internally represented as a n-ary tree of 3D objects, where each
object is a node in the tree. A 3D object is either a
3D mesh, point cloud,
or a 3D shape geometry.
A scene may be explicitly initialized with additional attributes, such as
camera
,
lights
, and
background
. By default, a
scene is created with neutral lighting, and a perspective camera whose
up
is set to Y
axis in a right-handed coordinate system.
After a scene is constructed, it should be written to the disk using the
scene.write()
method, which
serializes the scene into an FO3D file.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 | import fiftyone as fo scene = fo.Scene() scene.camera = fo.PerspectiveCamera(up="Z") mesh = fo.GltfMesh("mesh", "mesh.glb") mesh.rotation = fo.Euler(90, 0, 0, degrees=True) sphere1 = fo.SphereGeometry("sphere1", radius=2.0) sphere1.position = [-1, 0, 0] sphere1.default_material.color = "red" sphere2 = fo.SphereGeometry("sphere2", radius=2.0) sphere2.position = [-1, 0, 0] sphere2.default_material.color = "blue" scene.add(mesh, sphere1, sphere2) scene.write("/path/to/scene.fo3d") sample = fo.Sample(filepath="/path/to/scene.fo3d") dataset = fo.Dataset() dataset.add_sample(sample) print(dataset.media_type) # 3d |
To modify an exising scene, load it via
Scene.from_fo3d()
, perform any
necessary updates, and then re-write it to disk:
1 2 3 4 5 6 7 8 9 | import fiftyone as fo scene = fo.Scene.from_fo3d("/path/to/scene.fo3d") for node in scene.traverse(): if isinstance(node, fo.SphereGeometry): node.visible = False scene.write("/path/to/scene.fo3d") |
3D meshes¶
A 3D mesh is a collection of vertices, edges, and faces that define the shape
of a 3D object. Whereas some mesh formats store only the geometry of the mesh,
others also store the material properties and textures of the mesh. If a
mesh file contains material properties and textures, FiftyOne will
automatically load and display them. You may also
assign default material for your meshes by setting the
default_material
attribute of the mesh. In the absence of any material information,
meshes are assigned a
MeshStandardMaterial
with reasonable defaults that can also be dynamically configured from the app.
Please refer to material_3d
for more
details.
FiftyOne currently supports
GLTF
,
OBJ
,
PLY
,
STL
, and
FBX 7.x+
mesh formats.
Note
We recommend the GLTF
format for
3D meshes where possible, as it is the most compact, efficient, and
web-friendly format for storing and transmitting 3D models.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | import fiftyone as fo scene = fo.Scene() mesh1 = fo.GltfMesh("mesh1", "mesh.glb") mesh1.rotation = fo.Euler(90, 0, 0, degrees=True) mesh2 = fo.ObjMesh("mesh2", "mesh.obj") mesh3 = fo.PlyMesh("mesh3", "mesh.ply") mesh4 = fo.StlMesh("mesh4", "mesh.stl") mesh5 = fo.FbxMesh("mesh5", "mesh.fbx") scene.add(mesh1, mesh2, mesh3, mesh4, mesh5) scene.write("/path/to/scene.fo3d") |
3D point clouds¶
FiftyOne supports the PCD point cloud format. A code snippet to create a PCD object that can be added to a FiftyOne 3D scene is shown below:
1 2 3 4 5 6 7 8 9 10 11 | import fiftyone as fo pcd = fo.PointCloud("my-pcd", "point-cloud.pcd") pcd.default_material.shading_mode = "custom" pcd.default_material.custom_color = "red" pcd.default_material.point_size = 2 scene = fo.Scene() scene.add(pcd) scene.write("/path/to/scene.fo3d") |
You can customize the appearance of a point cloud by setting the
default_material
attribute of the point cloud object, or dynamically from
the app. Please refer to the
PointCloudMaterial
class for more details.
Note
If your scene contains multiple point clouds, you can control which point
cloud is included in
orthographic projections by
initializing it with flag_for_projection=True
.
Here’s how a typical PCD file is structured:
1 2 3 4 5 6 7 8 9 10 11 | import numpy as np import open3d as o3d points = np.array([(x1, y1, z1), (x2, y2, z2), ...]) colors = np.array([(r1, g1, b1), (r2, g2, b2), ...]) pcd = o3d.geometry.PointCloud() pcd.points = o3d.utility.Vector3dVector(points) pcd.colors = o3d.utility.Vector3dVector(colors) o3d.io.write_point_cloud("/path/to/point-cloud.pcd", pcd) |
Note
When working with modalities such as LIDAR, intensity data is assumed to be
encoded in the r
channel of the rgb
field of the
PCD files.
When coloring by intensity in the App, the intensity values are automatically scaled to use the full dynamic range of the colorscale.
3D shapes¶
FiftyOne provides a set of primitive 3D shape geometries that can be added to a 3D scene. The following 3D shape geometries are supported:
Box:
BoxGeometry
Sphere:
SphereGeometry
Cylinder:
CylinderGeometry
Plane:
PlaneGeometry
Similar to meshes and point clouds, shapes can be manipulated by setting their
position, rotation, and scale. Their appearance can be customized either by
setting the default_material
attribute of the shape object, or dynamically
from the app.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | import fiftyone as fo scene = fo.Scene() box = fo.BoxGeometry("box", width=0.5, height=0.5, depth=0.5) box.position = [0, 0, 1] box.default_material.color = "red" sphere = fo.SphereGeometry("sphere", radius=2.0) sphere.position = [-1, 0, 0] sphere.default_material.color = "blue" cylinder = fo.CylinderGeometry("cylinder", radius_top=0.5, height=1) cylinder.position = [0, 1, 0] plane = fo.PlaneGeometry("plane", width=2, height=2) plane.rotation = fo.Euler(90, 0, 0, degrees=True) scene.add(box, sphere, cylinder, plane) scene.write("/path/to/scene.fo3d") |
3D annotations¶
3D samples may contain any type and number of custom fields, including 3D detections and 3D polylines, which are natively visualizable by the App’s 3D visualizer.
Because 3D annotations are stored in dedicated fields of datasets rather than being embedded in FO3D files, they can be queried and filtered via dataset views and in the App just like other primitive/label fields.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | import fiftyone as fo scene = fo.Scene() scene.add(fo.GltfMesh("mesh", "mesh.gltf")) scene.write("/path/to/scene.fo3d") detection = fo.Detection( label="vehicle", location=[0.47, 1.49, 69.44], dimensions=[2.85, 2.63, 12.34], rotation=[0, -1.56, 0], ) sample = fo.Sample( filepath="/path/to/scene.fo3d", ground_truth=fo.Detections(detections=[detection]), ) |
Orthographic projection images¶
In order to visualize 3D datasets in the App’s grid view, you can use
compute_orthographic_projection_images()
to generate orthographic projection images of each scene:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | import fiftyone as fo import fiftyone.utils.utils3d as fou3d import fiftyone.zoo as foz # Load an example 3D dataset dataset = foz.load_zoo_dataset("quickstart-3d") # This dataset already has orthographic projections populated, but let's # recompute them to demonstrate the idea fou3d.compute_orthographic_projection_images( dataset, (-1, 512), # (width, height) of each image; -1 means aspect-preserving bounds=((-50, -50, -50), (50, 50, 50)), projection_normal=(0, -1, 0), output_dir="/tmp/quickstart-3d-proj", shading_mode="height", ) session = fo.launch_app(dataset) |
Note that the method also supports grouped datasets that contain 3D slice(s):
1 2 3 4 5 6 7 8 9 10 11 12 | import fiftyone as fo import fiftyone.utils.utils3d as fou3d import fiftyone.zoo as foz # Load an example group dataset that contains a 3D slice dataset = foz.load_zoo_dataset("quickstart-groups") # Populate orthographic projections fou3d.compute_orthographic_projection_images(dataset, (-1, 512), "/tmp/proj") dataset.group_slice = "pcd" session = fo.launch_app(dataset) |
Note
Orthographic projection images currently only include point clouds, not meshes or 3D shapes.
If a scene contains multiple point clouds, you can
control which point cloud to project by initializing it with
flag_for_projection=True
.
The above method populates an OrthographicProjectionMetadata
field on each
sample that contains the path to its projection image and other necessary
information to properly
visualize it in the App.
Refer to the
compute_orthographic_projection_images()
documentation for available parameters to customize the projections.
Example 3D datasets¶
To get started exploring 3D datasets, try loading the quickstart-3d dataset from the zoo:
1 2 3 4 5 6 7 8 9 | import fiftyone as fo import fiftyone.zoo as foz dataset = foz.load_zoo_dataset("quickstart-3d") print(dataset.count_values("ground_truth.label")) # {'bottle': 5, 'stairs': 5, 'keyboard': 5, 'car': 5, ...} session = fo.launch_app(dataset) |
Also check out the quickstart-groups dataset, which contains a point cloud slice:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | import fiftyone as fo import fiftyone.utils.utils3d as fou3d import fiftyone.zoo as foz dataset = foz.load_zoo_dataset("quickstart-groups") # Populate orthographic projections fou3d.compute_orthographic_projection_images(dataset, (-1, 512), "/tmp/proj") print(dataset.count("ground_truth.detections")) # 1100 print(dataset.count_values("ground_truth.detections.label")) # {'Pedestrian': 133, 'Car': 774, ...} dataset.group_slice = "pcd" session = fo.launch_app(dataset) |
Point cloud datasets¶
Warning
The point-cloud
media type has been deprecated in favor of the
3D media type.
While we’ll keep supporting the point-cloud
media type for backward
compatibility, we recommend using the 3d
media type for new datasets.
Any Sample
whose filepath
is a
PCD file
with extension .pcd
is recognized as a point cloud sample, and datasets
composed of point cloud samples have media type point-cloud
:
1 2 3 4 5 6 7 8 9 | import fiftyone as fo sample = fo.Sample(filepath="/path/to/point-cloud.pcd") dataset = fo.Dataset() dataset.add_sample(sample) print(dataset.media_type) # point-cloud print(sample) |
<Sample: {
'id': '6403ce64c8957c42bc8f9e67',
'media_type': 'point-cloud',
'filepath': '/path/to/point-cloud.pcd',
'tags': [],
'metadata': None,
'created_at': datetime.datetime(2024, 7, 22, 5, 16, 10, 701907),
'last_modified_at': datetime.datetime(2024, 7, 22, 5, 16, 10, 701907),
}>
Point cloud samples may contain any type and number of custom fields, including 3D detections and 3D polylines, which are natively visualizable by the App’s 3D visualizer.
DatasetViews¶
Previous sections have demonstrated how to add and interact with Dataset
components like samples, fields, and labels. The true power of FiftyOne lies in
the ability to search, sort, filter, and explore the contents of a Dataset
.
Behind this power is the DatasetView
. Whenever an operation
like match()
or
sort_by()
is applied to a
dataset, a DatasetView
is returned. As the name implies, a DatasetView
is a view into the data in your Dataset
that was produced by a series of
operations that manipulated your data in different ways.
A DatasetView
is composed of SampleView
objects for a subset of the samples
in your dataset. For example, a view may contain only samples with a given tag,
or samples whose labels meet a certain criteria.
In turn, each SampleView
represents a view into the content of the underlying
Sample
in the dataset. For example, a SampleView
may represent the contents
of a sample with Detections
below a specified threshold filtered out.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | import fiftyone as fo import fiftyone.zoo as foz from fiftyone import ViewField as F dataset = foz.load_zoo_dataset("quickstart") dataset.compute_metadata() # Create a view containing the 5 samples from the validation split whose # images are >= 48 KB that have the most predictions with confidence > 0.9 complex_view = ( dataset .match_tags("validation") .match(F("metadata.size_bytes") >= 48 * 1024) # >= 48 KB .filter_labels("predictions", F("confidence") > 0.9) .sort_by(F("predictions.detections").length(), reverse=True) .limit(5) ) # Check to see how many predictions there are in each matching sample print(complex_view.values(F("predictions.detections").length())) # [29, 20, 17, 15, 15] |
Merging datasets¶
The Dataset
class provides a powerful
merge_samples()
method
that you can use to merge the contents of another Dataset
or DatasetView
into an existing dataset.
By default, samples with the same absolute filepath
are merged, and top-level
fields from the provided samples are merged in, overwriting any existing values
for those fields, with the exception of list fields (e.g.,
tags) and label list fields (e.g.,
Detections), in which case the elements of the lists
themselves are merged. In the case of label list fields, labels with the same
id
in both collections are updated rather than duplicated.
The merge_samples()
method can be configured in numerous ways, including:
Which field to use as a merge key, or an arbitrary function defining the merge key
Whether existing samples should be modified or skipped
Whether new samples should be added or omitted
Whether new fields can be added to the dataset schema
Whether list fields should be treated as ordinary fields and merged as a whole rather than merging their elements
Whether to merge only specific fields, or all but certain fields
Mapping input fields to different field names of this dataset
For example, the following snippet demonstrates merging a new field into an existing dataset:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | import fiftyone as fo import fiftyone.zoo as foz dataset1 = foz.load_zoo_dataset("quickstart") # Create a dataset containing only ground truth objects dataset2 = dataset1.select_fields("ground_truth").clone() # Create a view containing only the predictions predictions_view = dataset1.select_fields("predictions") # Merge the predictions dataset2.merge_samples(predictions_view) print(dataset1.count("ground_truth.detections")) # 1232 print(dataset2.count("ground_truth.detections")) # 1232 print(dataset1.count("predictions.detections")) # 5620 print(dataset2.count("predictions.detections")) # 5620 |
Note that the argument to
merge_samples()
can be a
DatasetView
, which means that you can perform possibly-complex
transformations to the source dataset to select the
desired content to merge.
Consider the following variation of the above snippet, which demonstrates a
workflow where Detections
from another dataset are merged into a dataset with
existing Detections
in the same field:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | from fiftyone import ViewField as F # Create a new dataset that only contains predictions with confidence >= 0.9 dataset3 = ( dataset1 .select_fields("predictions") .filter_labels("predictions", F("confidence") > 0.9) ).clone() # Create a view that contains only the remaining predictions low_conf_view = dataset1.filter_labels("predictions", F("confidence") < 0.9) # Merge the low confidence predictions back in dataset3.merge_samples(low_conf_view, fields="predictions") print(dataset1.count("predictions.detections")) # 5620 print(dataset3.count("predictions.detections")) # 5620 |
Finally, the example below demonstrates the use of a custom merge key to define which samples to merge:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | import os # Create a dataset with 100 samples of ground truth labels dataset4 = dataset1[50:150].select_fields("ground_truth").clone() # Create a view with 50 overlapping samples of predictions predictions_view = dataset1[:100].select_fields("predictions") # Merge predictions into dataset, using base filename as merge key and # never inserting new samples dataset4.merge_samples( predictions_view, key_fcn=lambda sample: os.path.basename(sample.filepath), insert_new=False, ) print(len(dataset4)) # 100 print(len(dataset4.exists("predictions"))) # 50 |
Note
Did you know? You can use
merge_dir()
to directly
merge the contents of a dataset on disk into an existing FiftyOne
dataset without first
loading it into a temporary dataset and
then using
merge_samples()
to
perform the merge.
Cloning datasets¶
You can use clone()
to create a
copy of a dataset:
1 2 3 4 5 6 7 8 9 10 | import fiftyone as fo import fiftyone.zoo as foz dataset = foz.load_zoo_dataset("quickstart") dataset2 = dataset.clone() dataset2.add_sample_field("new_field", fo.StringField) # The source dataset is unaffected assert "new_field" not in dataset.get_field_schema() |
Dataset clones contain deep copies of all samples and dataset-level information in the source dataset. The source media files, however, are not copied.
Note
Did you know? You can also clone specific subsets of your datasets.
Batch updates¶
You are always free to perform any necessary modifications to a Dataset
by
iterating over it via a Python loop and explicitly
performing the edits that you require.
However, the Dataset
class provides a number of methods that allow you to
efficiently perform various common batch actions to your entire dataset.
Cloning, renaming, clearing, and deleting fields¶
You can use the
clone_sample_field()
,
rename_sample_field()
,
clear_sample_field()
,
and
delete_sample_field()
methods to efficiently perform common actions on the sample fields of a
Dataset
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | import fiftyone as fo import fiftyone.zoo as foz from fiftyone import ViewField as F dataset = foz.load_zoo_dataset("quickstart") # Clone an existing field dataset.clone_sample_field("predictions", "also_predictions") print("also_predictions" in dataset.get_field_schema()) # True # Rename a field dataset.rename_sample_field("also_predictions", "still_predictions") print("still_predictions" in dataset.get_field_schema()) # True # Clear a field (sets all values to None) dataset.clear_sample_field("still_predictions") print(dataset.count_values("still_predictions")) # {None: 200} # Delete a field dataset.delete_sample_field("still_predictions") |
You can also use dot notation to manipulate the fields or subfields of embedded documents in your dataset:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | sample = dataset.first() # Clone an existing embedded field dataset.clone_sample_field( "predictions.detections.label", "predictions.detections.also_label", ) print(sample.predictions.detections[0]["also_label"]) # "bird" # Rename an embedded field dataset.rename_sample_field( "predictions.detections.also_label", "predictions.detections.still_label", ) print(sample.predictions.detections[0]["still_label"]) # "bird" # Clear an embedded field (sets all values to None) dataset.clear_sample_field("predictions.detections.still_label") print(sample.predictions.detections[0]["still_label"]) # None # Delete an embedded field dataset.delete_sample_field("predictions.detections.still_label") |
Efficient batch edits¶
You are always free to perform arbitrary edits to a Dataset
by iterating over
its contents and editing the samples directly:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | import random import fiftyone as fo import fiftyone.zoo as foz from fiftyone import ViewField as F dataset = foz.load_zoo_dataset("quickstart") # Populate a new field on each sample in the dataset for sample in dataset: sample["random"] = random.random() sample.save() print(dataset.count("random")) # 200 print(dataset.bounds("random")) # (0.0007, 0.9987) |
However, the above pattern can be inefficient for large datasets because each
sample.save()
call makes a new
connection to the database.
The iter_samples()
method
provides an autosave=True
option that causes all changes to samples
emitted by the iterator to be automatically saved using an efficient batch
update strategy:
1 2 3 | # Automatically saves sample edits in efficient batches for sample in dataset.select_fields().iter_samples(autosave=True): sample["random"] = random.random() |
Note
As the above snippet shows, you should also optimize your iteration by selecting only the required fields.
You can configure the default batching strategy that is used via your
FiftyOne config, or you can configure the
batching strategy on a per-method call basis by passing the optional
batch_size
and batching_strategy
arguments to
iter_samples()
.
You can also use the
save_context()
method to perform batched edits using the pattern below:
1 2 3 4 5 | # Use a context to save sample edits in efficient batches with dataset.save_context() as context: for sample in dataset.select_fields(): sample["random"] = random.random() context.save(sample) |
The benefit of the above approach versus passing autosave=True
to
iter_samples()
is that
context.save()
allows you
to be explicit about which samples you are editing, which avoids unnecessary
computations if your loop only edits certain samples.
Setting values¶
Another strategy for performing efficient batch edits is to use
set_values()
to
set a field (or embedded field) on each sample in the dataset in a single
batch operation:
1 2 3 4 5 6 7 8 9 | # Delete the field we added earlier dataset.delete_sample_field("random") # Equivalent way to populate the field on each sample in the dataset values = [random.random() for _ in range(len(dataset))] dataset.set_values("random", values) print(dataset.count("random")) # 50 print(dataset.bounds("random")) # (0.0041, 0.9973) |
Note
When possible, using
set_values()
is often more efficient than performing the equivalent operation via an
explicit iteration over the Dataset
because it avoids the need to read
Sample
instances into memory and sequentially save them.
Similarly, you can edit nested sample fields of a Dataset
by iterating over
the dataset and editing the necessary data:
1 2 3 4 5 6 7 8 9 10 | # Add a tag to all low confidence predictions in the dataset for sample in dataset: for detection in sample["predictions"].detections: if detection.confidence < 0.06: detection.tags.append("low_confidence") sample.save() print(dataset.count_label_tags()) # {'low_confidence': 447} |
However, an equivalent and often more efficient approach is to use
values()
to
extract the slice of data you wish to modify and then use
set_values()
to
save the updated data in a single batch operation:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | # Remove the tags we added in the previous variation dataset.untag_labels("low_confidence") # Load all predicted detections # This is a list of lists of `Detection` instances for each sample detections = dataset.values("predictions.detections") # Add a tag to all low confidence detections for sample_detections in detections: for detection in sample_detections: if detection.confidence < 0.06: detection.tags.append("low_confidence") # Save the updated predictions dataset.set_values("predictions.detections", detections) print(dataset.count_label_tags()) # {'low_confidence': 447} |
Setting label values¶
Often when working with Label
fields, the edits you want to make may be
naturally represented as a mapping between label IDs and corresponding
attribute values to set on each Label
instance. In such cases, you can use
set_label_values()
to conveniently perform the updates:
1 2 3 4 5 6 7 8 9 10 | # Grab some random label IDs view = dataset.take(5, seed=51) label_ids = view.values("predictions.detections.id", unwind=True) # Populate a `random` attribute on all labels values = {_id: True for _id in label_ids} dataset.set_label_values("predictions.detections.random", values) print(dataset.count_values("predictions.detections.random")) # {True: 111, None: 5509} |