# Using FiftyOne Datasets
After a [`Dataset`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset) has been loaded or created, FiftyOne provides powerful
functionality to inspect, search, and modify it from a [`Dataset`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset)-wide down to
a [`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample) level.
The following sections provide details of how to use various aspects of a
FiftyOne [`Dataset`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset).
## Datasets
Instantiating a [`Dataset`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset) object creates a new dataset.
```python
import fiftyone as fo
dataset1 = fo.Dataset("my_first_dataset")
dataset2 = fo.Dataset("my_second_dataset")
dataset3 = fo.Dataset() # generates a default unique name
```
Check to see what datasets exist at any time via [`list_datasets()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.list_datasets):
```python
print(fo.list_datasets())
# ['my_first_dataset', 'my_second_dataset', '2020.08.04.12.36.29']
```
Load a dataset using
[`load_dataset()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.load_dataset).
Dataset objects are singletons. Cool!
```python
_dataset2 = fo.load_dataset("my_second_dataset")
_dataset2 is dataset2 # True
```
If you try to load a dataset via `Dataset(...)` or create a new dataset via
[`load_dataset()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.load_dataset) you’re going to
have a bad time:
```python
_dataset2 = fo.Dataset("my_second_dataset")
# Dataset 'my_second_dataset' already exists; use `fiftyone.load_dataset()`
# to load an existing dataset
dataset4 = fo.load_dataset("my_fourth_dataset")
# DoesNotExistError: Dataset 'my_fourth_dataset' not found
```
### Dataset media type
Available in:
Open SourceEnterprise
The media type of a dataset is determined by the
[media type](#using-media-type) of the [`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample) objects that it contains.
The [`media_type`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.media_type) property of a
dataset is set based on the first sample added to it:
```python
import fiftyone as fo
dataset = fo.Dataset()
print(dataset.media_type)
# None
sample = fo.Sample(filepath="/path/to/image.png")
dataset.add_sample(sample)
print(dataset.media_type)
# "image"
```
Note that datasets are homogeneous; they must contain samples of the same media
type (except for [grouped datasets](groups.md#groups)):
```python
sample = fo.Sample(filepath="/path/to/video.mp4")
dataset.add_sample(sample)
# MediaTypeError: Sample media type 'video' does not match dataset media type 'image'
```
The following media types are available:
| Media type | Description |
|---------------|-----------------------------------------------------------------------------------------------------------------|
| `image` | Datasets that contain
[images](#image-datasets) |
| `video` | Datasets that contain
[videos](#video-datasets) |
| `3d` | Datasets that contain
[3D scenes](#d-datasets) |
| `point-cloud` | Datasets that contain
[point clouds](#point-cloud-datasets) |
| `group` | Datasets that contain
[grouped data slices](groups.md#groups) |
| `unknown` † | Fallback value for datasets that contain
samples that are not one of the natively
available media types |
| custom † | Datasets that contain samples with a custom media
type will inherit that type |
#### NOTE
† [FiftyOne Enterprise](../enterprise/index.md#fiftyone-enterprise) users must upgrade their
deployment to 2.8.0+ in order to use `unknown` or “custom” media types.
### Dataset persistence
By default, datasets are non-persistent. Non-persistent datasets are deleted
from the database each time the database is shut down. Note that FiftyOne does
not store the raw data in datasets directly (only the labels), so your source
files on disk are untouched.
To make a dataset persistent, set its
[`persistent`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.persistent) property to
`True`:
```python
# Make the dataset persistent
dataset1.persistent = True
```
Without closing your current Python shell, open a new shell and run:
```python
import fiftyone as fo
# Verify that both persistent and non-persistent datasets still exist
print(fo.list_datasets())
# ['my_first_dataset', 'my_second_dataset', '2020.08.04.12.36.29']
```
All three datasets are still available, since the database connection has not
been terminated.
However, if you exit all processes with `fiftyone` imported, then open a new
shell and run the command again:
```python
import fiftyone as fo
# Verify that non-persistent datasets have been deleted
print(fo.list_datasets())
# ['my_first_dataset']
```
you’ll see that the `my_second_dataset` and `2020.08.04.12.36.29` datasets have
been deleted because they were not persistent.
### Dataset version
The version of the `fiftyone` package for which a dataset is formatted is
stored in the [`version`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.version) property
of the dataset.
If you upgrade your `fiftyone` package and then load a dataset that was created
with an older version of the package, it will be automatically migrated to the
new package version (if necessary) the first time you load it.
### Dataset tags
All [`Dataset`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset) instances have a
[`tags`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.tags) property that you can use to
store an arbitrary list of string tags.
```python
import fiftyone as fo
dataset = fo.Dataset()
# Add some tags
dataset.tags = ["test", "projectA"]
# Edit the tags
dataset.tags.pop()
dataset.tags.append("projectB")
dataset.save() # must save after edits
```
#### NOTE
You must call
[`dataset.save()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.save) after updating
the dataset’s [`tags`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.tags) property
in-place to save the changes to the database.
### Dataset stats
You can use the [`stats()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.stats) method on
a dataset to obtain information about the size of the dataset on disk,
including its metadata in the database and optionally the size of the physical
media on disk:
```python
import fiftyone as fo
import fiftyone.zoo as foz
dataset = foz.load_zoo_dataset("quickstart")
fo.pprint(dataset.stats(include_media=True))
```
```text
{
'samples_count': 200,
'samples_bytes': 1290762,
'samples_size': '1.2MB',
'media_bytes': 24412374,
'media_size': '23.3MB',
'total_bytes': 25703136,
'total_size': '24.5MB',
}
```
You can also invoke
[`stats()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.stats) on a
[dataset view](using_views.md#using-views) to retrieve stats for a specific subset of
the dataset:
```python
view = dataset[:10].select_fields("ground_truth")
fo.pprint(view.stats(include_media=True))
```
```text
{
'samples_count': 10,
'samples_bytes': 10141,
'samples_size': '9.9KB',
'media_bytes': 1726296,
'media_size': '1.6MB',
'total_bytes': 1736437,
'total_size': '1.7MB',
}
```
### Storing info
All [`Dataset`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset) instances have an
[`info`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.info) property, which contains a
dictionary that you can use to store any JSON-serializable information you wish
about your dataset.
Datasets can also store more specific types of ancillary information such as
[class lists](#storing-classes) and
[mask targets](#storing-mask-targets).
```python
import fiftyone as fo
dataset = fo.Dataset()
# Store a class list in the dataset's info
dataset.info = {
"dataset_source": "https://...",
"author": "...",
}
# Edit existing info
dataset.info["owner"] = "..."
dataset.save() # must save after edits
```
#### NOTE
You must call
[`dataset.save()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.save) after updating
the dataset’s [`info`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.info) property
in-place to save the changes to the database.
### Dataset App config
Available in:
Open SourceEnterprise
All [`Dataset`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset) instances have an
[`app_config`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.app_config) property that
contains a [`DatasetAppConfig`](../api/fiftyone.core.odm.dataset.md#fiftyone.core.odm.dataset.DatasetAppConfig) that you can use to store dataset-specific
settings that customize how the dataset is visualized in the
[FiftyOne App](app.md#fiftyone-app).
```python
import fiftyone as fo
import fiftyone.zoo as foz
dataset = foz.load_zoo_dataset("quickstart")
session = fo.launch_app(dataset)
# View the dataset's current App config
print(dataset.app_config)
```
#### Multiple media fields
Available in:
Open SourceEnterprise
You can declare [multiple media fields](app.md#app-multiple-media-fields) on a
dataset and configure which field is used by various components of the App by
default:
```python
import fiftyone.utils.image as foui
# Generate some thumbnail images
foui.transform_images(
dataset,
size=(-1, 32),
output_field="thumbnail_path",
output_dir="/tmp/thumbnails",
)
# Configure when to use each field
dataset.app_config.media_fields = ["filepath", "thumbnail_path"]
dataset.app_config.grid_media_field = "thumbnail_path"
dataset.save() # must save after edits
session.refresh()
```
You can set `media_fallback=True` if you want the App to fallback to the
`filepath` field if an alternate media field is missing for a particular
sample in the grid and/or modal:
```python
# Fallback to `filepath` if an alternate media field is missing
dataset.app_config.media_fallback = True
dataset.save()
```
#### Custom color scheme
Available in:
Open SourceEnterprise
You can store a [custom color scheme](app.md#app-color-schemes) on a dataset
that should be used by default whenever the dataset is loaded in the App:
```python
dataset.evaluate_detections(
"predictions", gt_field="ground_truth", eval_key="eval"
)
# Store a custom color scheme
dataset.app_config.color_scheme = fo.ColorScheme(
color_pool=["#ff0000", "#00ff00", "#0000ff", "pink", "yellowgreen"],
color_by="value",
fields=[
{
"path": "ground_truth",
"colorByAttribute": "eval",
"valueColors": [
{"value": "fn", "color": "#0000ff"}, # false negatives: blue
{"value": "tp", "color": "#00ff00"}, # true positives: green
]
},
{
"path": "predictions",
"colorByAttribute": "eval",
"valueColors": [
{"value": "fp", "color": "#ff0000"}, # false positives: red
{"value": "tp", "color": "#00ff00"}, # true positives: green
]
}
]
)
dataset.save() # must save after edits
# Setting `color_scheme` to None forces the dataset's default color scheme
# to be loaded
session.color_scheme = None
```
#### NOTE
Refer to the [`ColorScheme`](../api/fiftyone.core.odm.dataset.md#fiftyone.core.odm.dataset.ColorScheme) class for documentation of the available
customization options.
#### NOTE
Did you know? You can also configure color schemes
[directly in the App](app.md#app-color-schemes)!
#### Sidebar groups
You can configure the organization and default expansion state of the
[sidebar’s field groups](app.md#app-sidebar-groups):
```python
# Get the default sidebar groups for the dataset
sidebar_groups = fo.DatasetAppConfig.default_sidebar_groups(dataset)
# Collapse the `metadata` section by default
print(sidebar_groups[2].name) # metadata
sidebar_groups[2].expanded = False
# Modify the dataset's App config
dataset.app_config.sidebar_groups = sidebar_groups
dataset.save() # must save after edits
session.refresh()
```
#### Active fields
You can configure the default state of the
[sidebar’s checkboxes](app.md#app-fields-sidebar):
```python
# By default all label fields excluding Heatmap and Segmentation are active
active_fields = fo.DatasetAppConfig.default_active_fields(dataset)
# Add filepath and id fields
active_fields.paths.extend(["id", "filepath"])
# Active fields can be inverted setting exclude to True
# active_fields.exclude = True
# Modify the dataset's App config
dataset.app_config.active_fields = active_fields
dataset.save() # must save after edits
session.refresh()
```
#### Disable frame filtering
Available in:
Open SourceEnterprise
Filtering by frame-level fields of video datasets in the App’s grid view can be
expensive when the dataset is large.
You can disable frame filtering for a video dataset as follows:
```python
import fiftyone as fo
import fiftyone.zoo as foz
dataset = foz.load_zoo_dataset("quickstart-video")
dataset.app_config.disable_frame_filtering = True
dataset.save() # must save after edits
session = fo.launch_app(dataset)
```
#### NOTE
Did you know? You can also globally disable frame filtering for all video
datasets via your [App config](config.md#configuring-fiftyone-app).
#### Resetting a dataset’s App config
You can conveniently reset any property of a dataset’s App config by setting it
to `None`:
```python
# Reset the dataset's color scheme
dataset.app_config.color_scheme = None
dataset.save() # must save after edits
print(dataset.app_config)
session.refresh()
```
or you can reset the entire App config by setting the
[`app_config`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.app_config) property to
`None`:
```python
# Reset App config
dataset.app_config = None
print(dataset.app_config)
session = fo.launch_app(dataset)
```
#### NOTE
Check out [this section](app.md#app-config) for more information about
customizing the behavior of the App.
### Storing class lists
All [`Dataset`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset) instances have
[`classes`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.classes) and
[`default_classes`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.default_classes)
properties that you can use to store the lists of possible classes for your
annotations/models.
The [`classes`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.classes) property is a
dictionary mapping field names to class lists for a single [`Label`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Label) field of the
dataset.
If all [`Label`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Label) fields in your dataset have the same semantics, you can store a
single class list in the store a single target dictionary in the
[`default_classes`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.default_classes)
property of your dataset.
You can also pass your class lists to methods such as
[`evaluate_classifications()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.evaluate_classifications),
[`evaluate_detections()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.evaluate_detections),
and [`export()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.export) that
require knowledge of the possible classes in a dataset or field(s).
```python
import fiftyone as fo
dataset = fo.Dataset()
# Set default classes
dataset.default_classes = ["cat", "dog"]
# Edit the default classes
dataset.default_classes.append("other")
dataset.save() # must save after edits
# Set classes for the `ground_truth` and `predictions` fields
dataset.classes = {
"ground_truth": ["cat", "dog"],
"predictions": ["cat", "dog", "other"],
}
# Edit a field's classes
dataset.classes["ground_truth"].append("other")
dataset.save() # must save after edits
```
#### NOTE
You must call
[`dataset.save()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.save) after updating
the dataset’s [`classes`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.classes) and
[`default_classes`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.default_classes)
properties in-place to save the changes to the database.
### Storing mask targets
All [`Dataset`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset) instances have
[`mask_targets`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.mask_targets) and
[`default_mask_targets`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.default_mask_targets)
properties that you can use to store label strings for the pixel values of
[`Segmentation`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Segmentation) field masks.
The [`mask_targets`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.mask_targets) property
is a dictionary mapping field names to target dicts, each of which is a
dictionary defining the mapping between pixel values (2D masks) or RGB hex
strings (3D masks) and label strings for the [`Segmentation`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Segmentation) masks in the
specified field of the dataset.
If all [`Segmentation`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Segmentation) fields in your dataset have the same semantics, you can
store a single target dictionary in the
[`default_mask_targets`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.default_mask_targets)
property of your dataset.
When you load datasets with [`Segmentation`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Segmentation) fields in the App that have
corresponding mask targets, the label strings will appear in the App’s tooltip
when you hover over pixels.
You can also pass your mask targets to methods such as
[`evaluate_segmentations()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.evaluate_segmentations)
and [`export()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.export) that
require knowledge of the mask targets for a dataset or field(s).
If you are working with 2D segmentation masks, specify target keys as integers:
```python
import fiftyone as fo
dataset = fo.Dataset()
# Set default mask targets
dataset.default_mask_targets = {1: "cat", 2: "dog"}
# Edit the default mask targets
dataset.default_mask_targets[255] = "other"
dataset.save() # must save after edits
# Set mask targets for the `ground_truth` and `predictions` fields
dataset.mask_targets = {
"ground_truth": {1: "cat", 2: "dog"},
"predictions": {1: "cat", 2: "dog", 255: "other"},
}
# Edit an existing mask target
dataset.mask_targets["ground_truth"][255] = "other"
dataset.save() # must save after edits
```
If you are working with RGB segmentation masks, specify target keys as RGB hex
strings:
```python
import fiftyone as fo
dataset = fo.Dataset()
# Set default mask targets
dataset.default_mask_targets = {"#499CEF": "cat", "#6D04FF": "dog"}
# Edit the default mask targets
dataset.default_mask_targets["#FF6D04"] = "person"
dataset.save() # must save after edits
# Set mask targets for the `ground_truth` and `predictions` fields
dataset.mask_targets = {
"ground_truth": {"#499CEF": "cat", "#6D04FF": "dog"},
"predictions": {
"#499CEF": "cat", "#6D04FF": "dog", "#FF6D04": "person"
},
}
# Edit an existing mask target
dataset.mask_targets["ground_truth"]["#FF6D04"] = "person"
dataset.save() # must save after edits
```
#### NOTE
You must call
[`dataset.save()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.save) after updating
the dataset’s
[`mask_targets`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.mask_targets) and
[`default_mask_targets`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.default_mask_targets)
properties in-place to save the changes to the database.
### Storing keypoint skeletons
Available in:
Open SourceEnterprise
All [`Dataset`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset) instances have
[`skeletons`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.skeletons) and
[`default_skeleton`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.default_skeleton)
properties that you can use to store keypoint skeletons for [`Keypoint`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Keypoint) field(s)
of a dataset.
The [`skeletons`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.skeletons) property is a
dictionary mapping field names to [`KeypointSkeleton`](../api/fiftyone.core.odm.dataset.md#fiftyone.core.odm.dataset.KeypointSkeleton) instances, each of which
defines the keypoint label strings and edge connectivity for the [`Keypoint`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Keypoint)
instances in the specified field of the dataset.
If all [`Keypoint`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Keypoint) fields in your dataset have the same semantics, you can store
a single [`KeypointSkeleton`](../api/fiftyone.core.odm.dataset.md#fiftyone.core.odm.dataset.KeypointSkeleton) in the
[`default_skeleton`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.default_skeleton)
property of your dataset.
When you load datasets with [`Keypoint`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Keypoint) fields in the App that have
corresponding skeletons, the skeletons will automatically be rendered and label
strings will appear in the App’s tooltip when you hover over the keypoints.
Keypoint skeletons can be associated with [`Keypoint`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Keypoint) or [`Keypoints`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Keypoints) fields
whose [`points`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Keypoint.points) attributes all
contain a fixed number of semantically ordered points.
The [`edges`](../api/fiftyone.core.odm.dataset.md#fiftyone.core.odm.dataset.KeypointSkeleton.edges) argument
contains lists of integer indexes that define the connectivity of the points in
the skeleton, and the optional
[`labels`](../api/fiftyone.core.odm.dataset.md#fiftyone.core.odm.dataset.KeypointSkeleton.labels) argument
defines the label strings for each node in the skeleton.
For example, the skeleton below is defined by edges between the following
nodes:
```text
left hand <-> left shoulder <-> right shoulder <-> right hand
left eye <-> right eye <-> mouth
```
```python
import fiftyone as fo
dataset = fo.Dataset()
# Set keypoint skeleton for the `ground_truth` field
dataset.skeletons = {
"ground_truth": fo.KeypointSkeleton(
labels=[
"left hand" "left shoulder", "right shoulder", "right hand",
"left eye", "right eye", "mouth",
],
edges=[[0, 1, 2, 3], [4, 5, 6]],
)
}
# Edit an existing skeleton
dataset.skeletons["ground_truth"].labels[-1] = "lips"
dataset.save() # must save after edits
```
#### NOTE
When using keypoint skeletons, each [`Keypoint`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Keypoint) instance’s
[`points`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Keypoint.points) list must always
respect the indexing defined by the field’s [`KeypointSkeleton`](../api/fiftyone.core.odm.dataset.md#fiftyone.core.odm.dataset.KeypointSkeleton).
If a particular keypoint is occluded or missing for an object, use
`[float("nan"), float("nan")]` in its
[`points`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Keypoint.points) list.
#### NOTE
You must call
[`dataset.save()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.save) after updating
the dataset’s
[`skeletons`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.skeletons) and
[`default_skeleton`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.default_skeleton)
properties in-place to save the changes to the database.
### Storing camera calibration
Available in:
Open SourceEnterprise
All [`Dataset`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset) instances have
[`camera_intrinsics`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.camera_intrinsics) and
[`static_transforms`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.static_transforms)
properties that you can use to store camera calibration parameters for
multi-sensor datasets.
The [`camera_intrinsics`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.camera_intrinsics)
property is a dictionary mapping sensor/camera names to [`CameraIntrinsics`](../api/fiftyone.core.camera.md#fiftyone.core.camera.CameraIntrinsics)
instances that define the internal parameters of each camera:
```python
import fiftyone as fo
import fiftyone.zoo as foz
dataset = foz.load_zoo_dataset("quickstart-groups")
# Store intrinsics for a stereo camera setup (1080p parameters)
dataset.camera_intrinsics = {
"left": fo.PinholeCameraIntrinsics(fx=1100, fy=1100, cx=960, cy=540),
"right": fo.PinholeCameraIntrinsics(fx=1100, fy=1100, cx=960, cy=540),
}
dataset.save()
```
The [`static_transforms`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.static_transforms)
property is a dictionary mapping frame pairs to [`StaticTransform`](../api/fiftyone.core.camera.md#fiftyone.core.camera.StaticTransform) instances
that define the 6-DOF rigid transformations between coordinate frames. Keys
should be formatted as `"source_frame::target_frame"` or simply
`"source_frame"` (which implies `"world"` as the target):
```python
import fiftyone as fo
# Store sensor mounting positions relative to the vehicle center ("ego")
dataset.static_transforms = {
# Left camera: 1.5m forward, 0.5m left, 1.2m up
"left::ego": fo.StaticTransform(
translation=[1.5, 0.5, 1.2],
quaternion=[0, 0, 0, 1],
source_frame="left",
target_frame="ego",
),
# Right camera: 1.5m forward, 0.5m right, 1.2m up
"right::ego": fo.StaticTransform(
translation=[1.5, -0.5, 1.2],
quaternion=[0, 0, 0, 1],
source_frame="right",
target_frame="ego",
),
# Lidar: centered, 2.0m up (roof-mounted)
"lidar::ego": fo.StaticTransform(
translation=[0.0, 0.0, 2.0],
quaternion=[0, 0, 0, 1],
source_frame="lidar",
target_frame="ego",
),
}
dataset.save()
```
#### NOTE
See [this section](#camera-intrinsics-extrinsics) for more information
about creating and using camera intrinsics and extrinsics.
#### NOTE
You must call
[`dataset.save()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.save) after updating
the dataset’s
[`camera_intrinsics`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.camera_intrinsics)
and
[`static_transforms`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.static_transforms)
properties in-place to save the changes to the database.
### Deleting a dataset
Delete a dataset explicitly via
[`Dataset.delete()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.delete). Once a dataset
is deleted, any existing reference in memory will be in a volatile state.
[`Dataset.name`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset) and
[`Dataset.deleted`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset) will still be valid
attributes, but calling any other attribute or method will raise a
`DoesNotExistError`.
```python
dataset = fo.load_dataset("my_first_dataset")
dataset.delete()
print(fo.list_datasets())
# []
print(dataset.name)
# my_first_dataset
print(dataset.deleted)
# True
print(dataset.persistent)
# DoesNotExistError: Dataset 'my_first_dataset' is deleted
```
## Samples
An individual [`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample) is always initialized with a `filepath` to the
corresponding data on disk.
```python
# An image sample
sample = fo.Sample(filepath="/path/to/image.png")
# A video sample
another_sample = fo.Sample(filepath="/path/to/video.mp4")
```
#### NOTE
Creating a new [`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample) does not load the source data into memory. Source
data is read only as needed by the App.
### Adding samples to a dataset
A [`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample) can easily be added to an existing [`Dataset`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset):
```python
dataset = fo.Dataset("example_dataset")
dataset.add_sample(sample)
```
When a sample is added to a dataset, the relevant attributes of the [`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample)
are automatically updated:
```python
print(sample.in_dataset)
# True
print(sample.dataset_name)
# example_dataset
```
Every sample in a dataset is given a unique ID when it is added:
```python
print(sample.id)
# 5ee0ebd72ceafe13e7741c42
```
Multiple samples can be efficiently added to a dataset in batches:
```python
print(len(dataset))
# 1
dataset.add_samples(
[
fo.Sample(filepath="/path/to/image1.jpg"),
fo.Sample(filepath="/path/to/image2.jpg"),
fo.Sample(filepath="/path/to/image3.jpg"),
]
)
print(len(dataset))
# 4
```
### Accessing samples in a dataset
FiftyOne provides multiple ways to access a [`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample) in a [`Dataset`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset).
You can iterate over the samples in a dataset:
```python
for sample in dataset:
print(sample)
```
Use [`first()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.first) and
[`last()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.last) to retrieve the first and
last samples in a dataset, respectively:
```python
first_sample = dataset.first()
last_sample = dataset.last()
```
Samples can be accessed directly from datasets by their IDs or their filepaths.
[`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample) objects are singletons, so the same [`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample) instance is returned
whenever accessing the sample from the [`Dataset`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset):
```python
same_sample = dataset[sample.id]
print(same_sample is sample)
# True
also_same_sample = dataset[sample.filepath]
print(also_same_sample is sample)
# True
```
You can use [dataset views](using_views.md#using-views) to perform more sophisticated
operations on samples like searching, filtering, sorting, and slicing.
#### NOTE
Accessing a sample by its integer index in a [`Dataset`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset) is not allowed. The
best practice is to lookup individual samples by ID or filepath, or use
array slicing to extract a range of samples, and iterate over samples in a
view.
```python
dataset[0]
# KeyError: Accessing dataset samples by numeric index is not supported.
# Use sample IDs, filepaths, slices, boolean arrays, or a boolean ViewExpression instead
```
### Deleting samples from a dataset
Samples can be removed from a [`Dataset`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset) through their ID, either one at a time
or in batches via
[`delete_samples()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.delete_samples):
```python
dataset.delete_samples(sample_id)
# equivalent to above
del dataset[sample_id]
dataset.delete_samples([sample_id1, sample_id2])
```
Samples can also be removed from a [`Dataset`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset) by passing [`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample) instance(s)
or [`DatasetView`](../api/fiftyone.core.view.md#fiftyone.core.view.DatasetView) instances:
```python
# Remove a random sample
sample = dataset.take(1).first()
dataset.delete_samples(sample)
# Remove 10 random samples
view = dataset.take(10)
dataset.delete_samples(view)
```
If a [`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample) object in memory is deleted from a dataset, it will revert to
a [`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample) that has not been added to a [`Dataset`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset):
```python
print(sample.in_dataset)
# False
print(sample.dataset_name)
# None
print(sample.id)
# None
```
The [`last_deletion_at`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.last_deletion_at)
property of a [`Dataset`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset) tracks the datetime that a sample was last deleted
from the dataset:
```python
print(dataset.last_deletion_at)
# datetime.datetime(2025, 5, 4, 21, 0, 52, 942511)
```
## Fields
A [`Field`](../api/fiftyone.core.fields.md#fiftyone.core.fields.Field) is an attribute of a [`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample) that stores information about the
sample.
Fields can be dynamically created, modified, and deleted from samples on a
per-sample basis. When a new [`Field`](../api/fiftyone.core.fields.md#fiftyone.core.fields.Field) is assigned to a [`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample) in a [`Dataset`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset),
it is automatically added to the dataset’s schema and thus accessible on all
other samples in the dataset.
If a field exists on a dataset but has not been set on a particular sample, its
value will be `None`.
### Default sample fields
By default, all [`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample) instances have the following fields:
| Field | Type | Default | Description |
|--------------------|--------------------------------------------------------------------------------|--------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `id` | string | `None` | The ID of the sample in its parent dataset, which
is generated automatically when the sample is
added to a dataset, or `None` if the sample does
not belong to a dataset |
| `filepath` | string | **REQUIRED** | The path to the source data on disk. Must be
provided at sample creation time |
| `media_type` | string | N/A | The media type of the sample. Computed
automatically from the provided `filepath` |
| `tags` | list | `[]` | A list of string tags for the sample |
| `metadata` | [`Metadata`](../api/fiftyone.core.metadata.md#fiftyone.core.metadata.Metadata) | `None` | Type-specific metadata about the source data |
| `created_at` | datetime | `None` | The datetime that the sample was added to its
parent dataset, which is generated automatically,
or `None` if the sample does not belong to a
dataset |
| `last_modified_at` | datetime | `None` | The datetime that the sample was last modified,
which is updated automatically, or `None` if the
sample does not belong to a dataset |
#### NOTE
The `created_at` and `last_modified_at` fields are
[read-only](#read-only-fields) and are automatically populated/updated
when you add samples to datasets and modify them, respectively.
```python
import fiftyone as fo
sample = fo.Sample(filepath="/path/to/image.png")
print(sample)
```
```text
```
### Accessing fields of a sample
The names of available fields can be checked on any individual [`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample):
```python
sample.field_names
# ('id', 'filepath', 'tags', 'metadata', 'created_at', 'last_modified_at')
```
The value of a [`Field`](../api/fiftyone.core.fields.md#fiftyone.core.fields.Field) for a given [`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample) can be accessed either by either
attribute or item access:
```python
sample.filepath
sample["filepath"] # equivalent
```
### Field schemas
You can use
[`get_field_schema()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.get_field_schema) to
retrieve detailed information about the schema of the samples in a dataset:
```python
dataset = fo.Dataset("a_dataset")
dataset.add_sample(sample)
dataset.get_field_schema()
```
```text
OrderedDict([
('id', ),
('filepath', ),
('tags', ),
('metadata', ),
('created_at', ),
('last_modified_at', )]),
])
```
You can also view helpful information about a dataset, including its schema, by
printing it:
```python
print(dataset)
```
```text
Name: a_dataset
Media type: image
Num samples: 1
Persistent: False
Tags: []
Sample fields:
id: fiftyone.core.fields.ObjectIdField
filepath: fiftyone.core.fields.StringField
tags: fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)
metadata: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.ImageMetadata)
created_at: fiftyone.core.fields.DateTimeField
last_modified_at: fiftyone.core.fields.DateTimeField
```
#### NOTE
Did you know? You can [store metadata](#storing-field-metadata) such
as descriptions on your dataset’s fields!
### Adding fields to a sample
New fields can be added to a [`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample) using item assignment:
```python
sample["integer_field"] = 51
sample.save()
```
If the [`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample) belongs to a [`Dataset`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset), the dataset’s schema will automatically
be updated to reflect the new field:
```python
print(dataset)
```
```text
Name: a_dataset
Media type: image
Num samples: 1
Persistent: False
Tags: []
Sample fields:
id: fiftyone.core.fields.ObjectIdField
filepath: fiftyone.core.fields.StringField
tags: fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)
metadata: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.ImageMetadata)
created_at: fiftyone.core.fields.DateTimeField
last_modified_at: fiftyone.core.fields.DateTimeField
integer_field: fiftyone.core.fields.IntField
```
A [`Field`](../api/fiftyone.core.fields.md#fiftyone.core.fields.Field) can be any primitive type, such as `bool`, `int`, `float`, `str`,
`date`, `datetime`, `list`, `dict`, or more complex data structures
[like label types](#using-labels):
```python
sample["animal"] = fo.Classification(label="alligator")
sample.save()
```
Whenever a new field is added to a sample in a dataset, the field is available
on every other sample in the dataset with the value `None`.
Fields must have the same type (or `None`) across all samples in the dataset.
Setting a field to an inappropriate type raises an error:
```python
sample2.integer_field = "a string"
sample2.save()
# ValidationError: a string could not be converted to int
```
#### NOTE
You must call [`sample.save()`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample.save) in
order to persist changes to the database when editing samples that are in
datasets.
### Adding fields to a dataset
You can also use
[`add_sample_field()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.add_sample_field) to
declare new sample fields directly on datasets without explicitly populating
any values on its samples:
```python
import fiftyone as fo
sample = fo.Sample(
filepath="image.jpg",
ground_truth=fo.Classification(label="cat"),
)
dataset = fo.Dataset()
dataset.add_sample(sample)
# Declare new primitive fields
dataset.add_sample_field("scene_id", fo.StringField)
dataset.add_sample_field("quality", fo.FloatField)
# Declare untyped list fields
dataset.add_sample_field("more_tags", fo.ListField)
dataset.add_sample_field("info", fo.ListField)
# Declare a typed list field
dataset.add_sample_field("also_tags", fo.ListField, subfield=fo.StringField)
# Declare a new Label field
dataset.add_sample_field(
"predictions",
fo.EmbeddedDocumentField,
embedded_doc_type=fo.Classification,
)
print(dataset.get_field_schema())
```
```text
{
'id': ,
'filepath': ,
'tags': ,
'metadata': ,
'created_at': ,
'last_modified_at': ,
'ground_truth': ,
'scene_id': ,
'quality': ,
'more_tags': ,
'info': ,
'also_tags': ,
'predictions': ,
}
```
Whenever a new field is added to a dataset, the field is immediately available
on all samples in the dataset with the value `None`:
```python
print(sample)
```
```text
,
'scene_id': None,
'quality': None,
'more_tags': None,
'info': None,
'also_tags': None,
'predictions': None,
}>
```
You can also declare nested fields on existing embedded documents using dot
notation:
```python
# Declare a new attribute on a `Classification` field
dataset.add_sample_field("predictions.breed", fo.StringField)
```
#### NOTE
See [this section](#dynamic-attributes) for more options for
dynamically expanding the schema of nested lists and embedded documents.
You can use [`get_field()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.get_field) to
retrieve a [`Field`](../api/fiftyone.core.fields.md#fiftyone.core.fields.Field) instance by its name or `embedded.field.name`. And, if the
field contains an embedded document, you can call
[`get_field_schema()`](../api/fiftyone.core.fields.md#fiftyone.core.fields.EmbeddedDocumentField.get_field_schema)
to recursively inspect its nested fields:
```python
field = dataset.get_field("predictions")
print(field.document_type)
#
print(set(field.get_field_schema().keys()))
# {'logits', 'confidence', 'breed', 'tags', 'label', 'id'}
# Directly retrieve a nested field
field = dataset.get_field("predictions.breed")
print(type(field))
#
```
If your dataset contains a [`ListField`](../api/fiftyone.core.fields.md#fiftyone.core.fields.ListField) with no value type declared, you can add
the type later by appending `[]` to the field path:
```python
field = dataset.get_field("more_tags")
print(field.field) # None
# Declare the subfield types of an existing untyped list field
dataset.add_sample_field("more_tags[]", fo.StringField)
field = dataset.get_field("more_tags")
print(field.field) # StringField
# List fields can also contain embedded documents
dataset.add_sample_field(
"info[]",
fo.EmbeddedDocumentField,
embedded_doc_type=fo.DynamicEmbeddedDocument,
)
field = dataset.get_field("info")
print(field.field) # EmbeddedDocumentField
print(field.field.document_type) # DynamicEmbeddedDocument
```
#### NOTE
Declaring the value type of list fields is required if you want to filter
by the list’s values [in the App](app.md#app-filtering).
### Editing sample fields
You can make any edits you wish to the fields of an existing [`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample):
```python
sample = fo.Sample(
filepath="/path/to/image.jpg",
ground_truth=fo.Detections(
detections=[
fo.Detection(label="CAT", bounding_box=[0.1, 0.1, 0.4, 0.4]),
fo.Detection(label="dog", bounding_box=[0.5, 0.5, 0.4, 0.4]),
]
)
)
detections = sample.ground_truth.detections
# Edit an existing detection
detections[0].label = "cat"
# Add a new detection
new_detection = fo.Detection(label="animals", bounding_box=[0, 0, 1, 1])
detections.append(new_detection)
print(sample)
sample.save() # if the sample is in a dataset
```
#### NOTE
You must call [`sample.save()`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample.save) in
order to persist changes to the database when editing samples that are in
datasets.
A common workflow is to iterate over a dataset
[or view](using_views.md#editing-view-fields) and edit each sample:
```python
for sample in dataset:
sample["new_field"] = ...
sample.save()
```
The [`iter_samples()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.iter_samples) method
is an equivalent way to iterate over a dataset that provides a
`progress=True` option that prints a progress bar tracking the status of the
iteration:
```python
# Prints a progress bar tracking the status of the iteration
for sample in dataset.iter_samples(progress=True):
sample["new_field"] = ...
sample.save()
```
The [`iter_samples()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.iter_samples) method
also provides an `autosave=True` option that causes all changes to samples
emitted by the iterator to be automatically saved using efficient batch
updates:
```python
# Automatically saves sample edits in efficient batches
for sample in dataset.iter_samples(autosave=True):
sample["new_field"] = ...
```
Using `autosave=True` can significantly improve performance when editing
large datasets. See [this section](#batch-updates) for more information
on batch update patterns.
### Removing fields from a sample
A field can be deleted from a [`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample) using `del`:
```python
del sample["integer_field"]
```
If the [`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample) is not yet in a dataset, deleting a field will remove it from
the sample. If the [`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample) is in a dataset, the field’s value will be `None`.
Fields can also be deleted at the [`Dataset`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset) level, in which case they are
removed from every [`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample) in the dataset:
```python
dataset.delete_sample_field("integer_field")
sample.integer_field
# AttributeError: Sample has no field 'integer_field'
```
### Storing field metadata
Available in:
Open SourceEnterprise
You can store metadata such as descriptions and other info on the fields of
your dataset.
One approach is to manually declare the field with
[`add_sample_field()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.add_sample_field)
with the appropriate metadata provided:
```python
import fiftyone as fo
dataset = fo.Dataset()
dataset.add_sample_field(
"int_field", fo.IntField, description="An integer field"
)
field = dataset.get_field("int_field")
print(field.description) # An integer field
```
You can also use
[`get_field()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.get_field) to
retrieve a field and update it’s metadata at any time:
```python
import fiftyone as fo
import fiftyone.zoo as foz
dataset = foz.load_zoo_dataset("quickstart")
dataset.add_dynamic_sample_fields()
field = dataset.get_field("ground_truth")
field.description = "Ground truth annotations"
field.info = {"url": "https://fiftyone.ai"}
field.save() # must save after edits
field = dataset.get_field("ground_truth.detections.area")
field.description = "Area of the box, in pixels^2"
field.info = {"url": "https://fiftyone.ai"}
field.save() # must save after edits
dataset.reload()
field = dataset.get_field("ground_truth")
print(field.description) # Ground truth annotations
print(field.info) # {'url': 'https://fiftyone.ai'}
field = dataset.get_field("ground_truth.detections.area")
print(field.description) # Area of the box, in pixels^2
print(field.info) # {'url': 'https://fiftyone.ai'}
```
#### NOTE
You must call
[`field.save()`](../api/fiftyone.core.fields.md#fiftyone.core.fields.Field.save) after updating
a fields’s [`description`](../api/fiftyone.core.fields.md#fiftyone.core.fields.Field.description)
and [`info`](../api/fiftyone.core.fields.md#fiftyone.core.fields.Field.info) attributes in-place to
save the changes to the database.
#### NOTE
Did you know? You can view field metadata directly in the App by hovering
over fields or attributes [in the sidebar](app.md#app-fields-sidebar)!
### Read-only fields
Certain [default sample fields](#default-sample-fields) like `created_at`
and `last_modified_at` are read-only and thus cannot be manually edited:
```python
from datetime import datetime
import fiftyone as fo
sample = fo.Sample(filepath="/path/to/image.jpg")
dataset = fo.Dataset()
dataset.add_sample(sample)
sample.created_at = datetime.utcnow()
# ValueError: Cannot edit read-only field 'created_at'
sample.last_modified_at = datetime.utcnow()
# ValueError: Cannot edit read-only field 'last_modified_at'
```
You can also manually mark additional fields or embedded fields as read-only
at any time:
```python
import fiftyone as fo
import fiftyone.zoo as foz
dataset = foz.load_zoo_dataset("quickstart")
# Declare a new read-only field
dataset.add_sample_field("uuid", fo.StringField, read_only=True)
# Mark 'filepath' as read-only
field = dataset.get_field("filepath")
field.read_only = True
field.save() # must save after edits
# Mark a nested field as read-only
field = dataset.get_field("ground_truth.detections.label")
field.read_only = True
field.save() # must save after edits
sample = dataset.first()
sample.filepath = "no.jpg"
# ValueError: Cannot edit read-only field 'filepath'
sample.ground_truth.detections[0].label = "no"
sample.save()
# ValueError: Cannot edit read-only field 'ground_truth.detections.label'
```
#### NOTE
You must call
[`field.save()`](../api/fiftyone.core.fields.md#fiftyone.core.fields.Field.save) after updating
a fields’s [`read_only`](../api/fiftyone.core.fields.md#fiftyone.core.fields.Field.read_only)
attributes in-place to save the changes to the database.
Note that read-only fields do not interfere with the ability to add/delete
samples from datasets:
```python
sample = fo.Sample(filepath="/path/to/image.jpg", uuid="1234")
dataset.add_sample(sample)
dataset.delete_samples(sample)
```
Any fields that you’ve manually marked as read-only may be reverted to
editable at any time:
```python
sample = dataset.first()
# Revert 'filepath' to editable
field = dataset.get_field("filepath")
field.read_only = False
field.save() # must save after edits
# Revert nested field to editable
field = dataset.get_field("ground_truth.detections.label")
field.read_only = False
field.save() # must save after edits
sample.filepath = "yes.jpg"
sample.ground_truth.detections[0].label = "yes"
sample.save()
```
### Summary fields
Available in:
Open SourceEnterprise
Summary fields allow you to efficiently perform queries on large datasets where
directly querying the underlying field is prohibitively slow due to the number
of objects/frames in the field.
For example, suppose you’re working on a
[video dataset](#video-datasets) with frame-level objects, and you’re
interested in finding videos that contain specific classes of interest, eg
`person`, in at least one frame:
```python
import fiftyone as fo
import fiftyone.zoo as foz
from fiftyone import ViewField as F
dataset = foz.load_zoo_dataset("quickstart-video")
dataset.set_field("frames.detections.detections.confidence", F.rand()).save()
session = fo.launch_app(dataset)
```
One approach is to directly query the frame-level field (`frames.detections`
in this case) in the App’s sidebar. However, when the dataset is large, such
queries are inefficient, as they cannot unlock
[query performance](app.md#app-optimizing-query-performance) and thus require
full collection scans over all frames to retrieve the relevant samples.
A more efficient approach is to first use
[`create_summary_field()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.create_summary_field)
to summarize the relevant input field path(s):
```python
# Generate a summary field for object labels
field_name = dataset.create_summary_field("frames.detections.detections.label")
# The name of the summary field that was created
print(field_name)
# 'frames_detections_label'
# Generate a summary field for [min, max] confidences
dataset.create_summary_field("frames.detections.detections.confidence")
```
Summary fields can be generated for sample-level and frame-level fields, and
the input fields can be either categorical or numeric:
Categorical fields
Numeric fields
When the input field is categorical (string or boolean), the summary
field of each sample is populated with the list of unique values
observed in the field (across all frames for video samples):
```python
sample = dataset.first()
print(sample.frames_detections_label)
# ['vehicle', 'road sign', 'person']
```
You can also pass `include_counts=True` to include counts for each
unique value in the summary field:
```python
# Generate a summary field for object labels and counts
dataset.create_summary_field(
"frames.detections.detections.label",
field_name="frames_detections_label2",
include_counts=True,
)
sample = dataset.first()
print(sample.frames_detections_label2)
"""
[
,
,
,
]
"""
```
When the input field is numeric (int, float, date, or datetime), the
summary field of each sample is populated with the `[min, max]` range
of the values observed in the field (across all frames for video
samples):
```python
sample = dataset.first()
print(sample.frames_detections_confidence)
#
```
You can also pass the `group_by` parameter to specify an attribute to
group by to generate per-attribute `[min, max]` ranges:
```python
# Generate a summary field for per-label [min, max] confidences
dataset.create_summary_field(
"frames.detections.detections.confidence",
field_name="frames_detections_confidence2",
group_by="label",
)
sample = dataset.first()
print(sample.frames_detections_confidence2)
"""
[
,
,
,
]
"""
```
As the above examples illustrate, summary fields allow you to encode various
types of information at the sample-level that you can directly query to find
samples that contain specific values.
Moreover, summary fields are [indexed](app.md#app-optimizing-query-performance)
by default and the App can natively leverage these indexes to provide
performant filtering:
#### NOTE
Summary fields are automatically added to a `summaries`
[sidebar group](#dataset-app-config-sidebar-groups) in the App for
easy access and organization.
They are also [read-only](#read-only-fields) by default, as they are
implicitly derived from the contents of their source field and are not
intended to be directly modified.
You can use
[`list_summary_fields()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.list_summary_fields)
to list the names of the summary fields on your dataset:
```python
print(dataset.list_summary_fields())
# ['frames_detections_label', 'frames_detections_confidence', ...]
```
Since a summary field is derived from the contents of another field, it must be
updated whenever there have been modifications to its source field. You can use
[`check_summary_fields()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.check_summary_fields)
to check for summary fields that *may* need to be updated:
```python
# Newly created summary fields don't needed updating
print(dataset.check_summary_fields())
# []
# Modify the dataset
label_upper = F("label").upper()
dataset.set_field("frames.detections.detections.label", label_upper).save()
# Summary fields now (may) need updating
print(dataset.check_summary_fields())
# ['frames_detections_label', 'frames_detections_confidence', ...]
```
#### NOTE
Note that inclusion in
[`check_summary_fields()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.check_summary_fields)
is only a heuristic, as any sample modifications *may not* have affected
the summary’s source field.
Use [`update_summary_field()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.update_summary_field)
to regenerate a summary field based on the current values of its source field:
```python
dataset.update_summary_field("frames_detections_label")
```
Finally, use
[`delete_summary_field()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.delete_summary_field)
or [`delete_summary_fields()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.delete_summary_fields)
to delete existing summary field(s) that you no longer need:
```python
dataset.delete_summary_field("frames_detections_label")
```
## Media type
When a [`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample) is created, its media type is inferred from the `filepath` to
the source media and available via the `media_type` attribute of the sample,
which is read-only.
Optionally, the `media_type` keyword argument can be provided to the [`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample)
constructor to provide an explicit media type.
If `media_type` is not provided explicitly, it is inferred from the
[MIME type](https://en.wikipedia.org/wiki/Media_type) of the file on disk,
as per the table below:
| MIME type/extension | `media_type` | Description |
|-----------------------|----------------|--------------------|
| `image/*` | `image` | Image sample |
| `video/*` | `video` | Video sample |
| `*.fo3d` | `3d` | 3D sample |
| `*.pcd` | `point-cloud` | Point cloud sample |
| other | `unknown` | Generic sample |
#### NOTE
The `filepath` of a sample can be changed after the sample is created, but
the new filepath must have the same media type. In other words,
`media_type` is immutable.
#### NOTE
When creating new 3D datasets from direct 3D asset files such as `.glb`,
`.pcd`, or `.ply`, pass `media_type="3d"` explicitly.
## Tags
All [`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample) instances have a `tags` field, which is a string list. By default,
this list is empty, but you can use it to store information like dataset splits
or application-specific issues like low quality images:
```python
dataset = fo.Dataset("tagged_dataset")
dataset.add_samples(
[
fo.Sample(filepath="/path/to/image1.png", tags=["train"]),
fo.Sample(filepath="/path/to/image2.png", tags=["test", "low_quality"]),
]
)
print(dataset.distinct("tags"))
# ["test", "low_quality", "train"]
```
#### NOTE
Did you know? You can add, edit, and filter by sample tags
[directly in the App](app.md#app-tagging).
The `tags` field can be used like a standard Python list:
```python
sample = dataset.first()
sample.tags.append("new_tag")
sample.save()
```
#### NOTE
You must call [`sample.save()`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample.save) in
order to persist changes to the database when editing samples that are in
datasets.
Datasets and views provide helpful methods such as
[`count_sample_tags()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.count_sample_tags),
[`tag_samples()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.tag_samples),
[`untag_samples()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.untag_samples),
and
[`match_tags()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.match_tags)
that you can use to perform batch queries and edits to sample tags:
```python
import fiftyone as fo
import fiftyone.zoo as foz
dataset = foz.load_zoo_dataset("quickstart").clone()
print(dataset.count_sample_tags()) # {'validation': 200}
# Tag samples in a view
test_view = dataset.limit(100)
test_view.untag_samples("validation")
test_view.tag_samples("test")
print(dataset.count_sample_tags()) # {'validation': 100, 'test': 100}
# Create a view containing samples with a specific tag
validation_view = dataset.match_tags("validation")
print(len(validation_view)) # 100
```
## Metadata
All [`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample) instances have a `metadata` field, which can optionally be
populated with a [`Metadata`](../api/fiftyone.core.metadata.md#fiftyone.core.metadata.Metadata) instance that stores data type-specific metadata
about the raw data in the sample. The [FiftyOne App](app.md#fiftyone-app) and
the [FiftyOne Brain](../brain/index.md#fiftyone-brain) will use this provided metadata in
some workflows when it is available.
Images
Videos
For image samples, the [`ImageMetadata`](../api/fiftyone.core.metadata.md#fiftyone.core.metadata.ImageMetadata) class is used to store
information about images, including their
[`size_bytes`](../api/fiftyone.core.metadata.md#fiftyone.core.metadata.ImageMetadata.size_bytes),
[`mime_type`](../api/fiftyone.core.metadata.md#fiftyone.core.metadata.ImageMetadata.mime_type),
[`width`](../api/fiftyone.core.metadata.md#fiftyone.core.metadata.ImageMetadata.width),
[`height`](../api/fiftyone.core.metadata.md#fiftyone.core.metadata.ImageMetadata.height), and
[`num_channels`](../api/fiftyone.core.metadata.md#fiftyone.core.metadata.ImageMetadata.num_channels).
You can populate the `metadata` field of an existing dataset by calling
[`Dataset.compute_metadata()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.compute_metadata):
```python
import fiftyone.zoo as foz
dataset = foz.load_zoo_dataset("quickstart")
# Populate metadata fields (if necessary)
dataset.compute_metadata()
print(dataset.first())
```
Alternatively, FiftyOne provides a
[`ImageMetadata.build_for()`](../api/fiftyone.core.metadata.md#fiftyone.core.metadata.ImageMetadata.build_for)
factory method that you can use to compute the metadata for your images
when constructing [`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample) instances:
```python
image_path = "/path/to/image.png"
metadata = fo.ImageMetadata.build_for(image_path)
sample = fo.Sample(filepath=image_path, metadata=metadata)
print(sample)
```
```text
,
'created_at': None,
'last_modified_at': None,
}>
```
For video samples, the [`VideoMetadata`](../api/fiftyone.core.metadata.md#fiftyone.core.metadata.VideoMetadata) class is used to store
information about videos, including their
[`size_bytes`](../api/fiftyone.core.metadata.md#fiftyone.core.metadata.VideoMetadata.size_bytes),
[`mime_type`](../api/fiftyone.core.metadata.md#fiftyone.core.metadata.VideoMetadata.mime_type),
[`frame_width`](../api/fiftyone.core.metadata.md#fiftyone.core.metadata.VideoMetadata.frame_width),
[`frame_height`](../api/fiftyone.core.metadata.md#fiftyone.core.metadata.VideoMetadata.frame_height),
[`frame_rate`](../api/fiftyone.core.metadata.md#fiftyone.core.metadata.VideoMetadata.frame_rate),
[`total_frame_count`](../api/fiftyone.core.metadata.md#fiftyone.core.metadata.VideoMetadata.total_frame_count),
[`duration`](../api/fiftyone.core.metadata.md#fiftyone.core.metadata.VideoMetadata.duration), and
[`encoding_str`](../api/fiftyone.core.metadata.md#fiftyone.core.metadata.VideoMetadata.encoding_str).
You can populate the `metadata` field of an existing dataset by calling
[`Dataset.compute_metadata()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.compute_metadata):
```python
import fiftyone.zoo as foz
dataset = foz.load_zoo_dataset("quickstart-video")
# Populate metadata fields (if necessary)
dataset.compute_metadata()
print(dataset.first())
```
Alternatively, FiftyOne provides a
[`VideoMetadata.build_for()`](../api/fiftyone.core.metadata.md#fiftyone.core.metadata.VideoMetadata.build_for)
factory method that you can use to compute the metadata for your videos
when constructing [`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample) instances:
```python
video_path = "/path/to/video.mp4"
metadata = fo.VideoMetadata.build_for(video_path)
sample = fo.Sample(filepath=video_path, metadata=metadata)
print(sample)
```
```text
,
'created_at': None,
'last_modified_at': None,
'frames': ,
}>
```
## Dates and datetimes
Available in:
Open SourceEnterprise
### Builtin datetime fields
Available in:
Open SourceEnterprise
Datasets and samples have various builtin datetime fields that are
automatically updated when certain events occur.
The
[`Dataset.last_loaded_at`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.last_loaded_at)
property tracks the datetime that the dataset was last loaded:
```python
import fiftyone as fo
import fiftyone.zoo as foz
dataset = foz.load_zoo_dataset("quickstart")
print(dataset.last_loaded_at)
# 2025-05-04 21:00:45.559520
```
The
[`Dataset.last_modified_at`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.last_modified_at)
property tracks the datetime that dataset-level metadata was last modified,
including:
- when properties such as
[`name`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.name),
[`persistent`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.persistent),
[`tags`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.tags),
[`description`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.description),
[`info`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.info), and
[`app_config`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.app_config) are edited
- when fields are added or deleted from the dataset’s schema
- when group slices are added or deleted from the dataset’s schema
- when saved views or workspaces are added, edited, or deleted
- when annotation, brain, evaluation, or custom runs are added, edited, or
deleted
```python
last_modified_at1 = dataset.last_modified_at
dataset.name = "still-quickstart"
last_modified_at2 = dataset.last_modified_at
assert last_modified_at2 > last_modified_at1
dataset.app_config.sidebar_groups = ...
dataset.save()
last_modified_at3 = dataset.last_modified_at
assert last_modified_at3 > last_modified_at2
dataset.add_sample_field("foo", fo.StringField)
last_modified_at4 = dataset.last_modified_at
assert last_modified_at4 > last_modified_at3
```
#### NOTE
The
[`Dataset.last_modified_at`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.last_modified_at)
property is **not** updated when samples are added, edited, or deleted from
a dataset.
Use the methods described below to ascertain this information.
All samples have a builtin `last_modified_at` field that automatically tracks
the datetime that each sample was last modified:
```python
sample = dataset.first()
last_modified_at1 = sample.last_modified_at
sample.foo = "bar"
sample.save()
last_modified_at2 = sample.last_modified_at
assert last_modified_at2 > last_modified_at1
```
The `last_modified_at` field is indexed by default, which means you can
efficiently check when a dataset’s samples were last modified via
[`max()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.max):
```python
last_modified_at1 = dataset.max("last_modified_at")
dataset.add_samples(...)
last_modified_at2 = dataset.max("last_modified_at")
assert last_modified_at2 > last_modified_at1
dataset.set_field("foo", "spam").save()
last_modified_at3 = dataset.max("last_modified_at")
assert last_modified_at3 > last_modified_at2
```
The
[`Dataset.last_deletion_at`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.last_deletion_at)
property tracks the datetime that a sample was last deleted
from the dataset:
```python
last_deletion_at1 = dataset.last_deletion_at
dataset.delete_samples(...)
last_deletion_a2 = dataset.last_deletion_at
assert last_deletion_a2 > last_deletion_at1
```
**Video datasets**
The frames of [video datasets](#video-datasets) also have a builtin
`last_modified_at` field that automatically tracks the datetime that each
frame was last modified:
```python
import fiftyone as fo
import fiftyone.zoo as foz
dataset = foz.load_zoo_dataset("quickstart-video")
sample = dataset.first()
frame = sample.frames.first()
last_modified_at1 = frame.last_modified_at
frame["foo"] = "bar"
frame.save()
last_modified_at2 = frame.last_modified_at
assert last_modified_at2 > last_modified_at1
```
The `last_modified_at` frame field is indexed by default, which means you can
efficiently check when a dataset’s frames were last modified via
[`max()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.max):
```python
last_modified_at1 = dataset.max("frames.last_modified_at")
dataset.add_samples(...)
last_modified_at2 = dataset.max("frames.last_modified_at")
assert last_modified_at2 > last_modified_at1
dataset.set_field("frames.foo", "spam").save()
last_modified_at3 = dataset.max("last_modified_at")
assert last_modified_at3 > last_modified_at2
```
When frames are deleted from a dataset, the `last_modified_at` field of the
parent samples are automatically updated:
```python
sample = dataset.first()
last_modified_at1 = sample.last_modified_at
del sample.frames[1]
sample.save()
last_modified_at2 = sample.last_modified_at
assert last_modified_at2 > last_modified_at1
```
### Custom datetime fields
You can store date information in FiftyOne datasets by populating fields with
`date` or `datetime` values:
```python
from datetime import date, datetime
import fiftyone as fo
dataset = fo.Dataset()
dataset.add_samples(
[
fo.Sample(
filepath="image1.png",
acquisition_time=datetime(2021, 8, 24, 21, 18, 7),
acquisition_date=date(2021, 8, 24),
),
fo.Sample(
filepath="image2.png",
acquisition_time=datetime.utcnow(),
acquisition_date=date.today(),
),
]
)
print(dataset)
print(dataset.head())
```
#### NOTE
Did you know? You can [create dataset views](using_views.md#date-views) with
date-based queries!
Internally, FiftyOne stores all dates as UTC timestamps, but you can provide
any valid `datetime` object when setting a [`DateTimeField`](../api/fiftyone.core.fields.md#fiftyone.core.fields.DateTimeField) of a sample,
including timezone-aware datetimes, which are internally converted to UTC
format for safekeeping.
```python
# A datetime in your local timezone
now = datetime.utcnow().astimezone()
sample = fo.Sample(filepath="image.png", acquisition_time=now)
dataset = fo.Dataset()
dataset.add_sample(sample)
# Samples are singletons, so we reload so `sample` will contain values as
# loaded from the database
dataset.reload()
sample.acquisition_time.tzinfo # None
```
By default, when you access a datetime field of a sample in a dataset, it is
retrieved as a naive `datetime` instance expressed in UTC format.
However, if you prefer, you can
[configure FiftyOne](config.md#configuring-timezone) to load datetime fields as
timezone-aware `datetime` instances in a timezone of your choice.
#### WARNING
FiftyOne assumes that all `datetime` instances with no explicit timezone
are stored in UTC format.
Therefore, never use `datetime.datetime.now()` when populating a datetime
field of a FiftyOne dataset! Instead, use `datetime.datetime.utcnow()`.
## Labels
The [`Label`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Label) class hierarchy is used to store semantic information about ground
truth or predicted labels in a sample.
Although such information can be stored in custom sample fields
(e.g, in a [`DictField`](../api/fiftyone.core.fields.md#fiftyone.core.fields.DictField)), it is recommended that you store label information in
[`Label`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Label) instances so that the [FiftyOne App](app.md#fiftyone-app) and the
[FiftyOne Brain](../brain/index.md#fiftyone-brain) can visualize and compute on your
labels.
#### NOTE
All [`Label`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Label) instances are dynamic! You can add custom fields to your
labels to store custom information:
```python
# Provide some default fields
label = fo.Classification(label="cat", confidence=0.98)
# Add custom fields
label["int"] = 5
label["float"] = 51.0
label["list"] = [1, 2, 3]
label["bool"] = True
label["dict"] = {"key": ["list", "of", "values"]}
```
You can also [declare dynamic attributes](#dynamic-attributes) on your
dataset’s schema, which allows you to enforce type constraints, filter by
these custom attributes [in the App](app.md#app-filtering), and more.
FiftyOne provides a dedicated [`Label`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Label) subclass for many common tasks. The
subsections below describe them.
### Regression
The [`Regression`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Regression) class represents a numeric regression value for an image. The
value itself is stored in the
[`value`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Regression.value) attribute of the
[`Regression`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Regression) object. This may be a ground truth value or a model prediction.
The optional
[`confidence`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Regression.confidence) attribute can
be used to store a score associated with the model prediction and can be
visualized in the App or used, for example, when
[evaluating regressions](evaluation/regressions.md#evaluating-regressions).
```python
import fiftyone as fo
sample = fo.Sample(filepath="/path/to/image.png")
sample["ground_truth"] = fo.Regression(value=51.0)
sample["prediction"] = fo.Classification(value=42.0, confidence=0.9)
print(sample)
```
```text
,
'prediction': ,
}>
```
### Classification
The [`Classification`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Classification) class represents a classification label for an image. The
label itself is stored in the
[`label`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Classification.label) attribute of the
[`Classification`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Classification) object. This may be a ground truth label or a model
prediction.
The optional
[`confidence`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Classification.confidence) and
[`logits`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Classification.logits) attributes may be
used to store metadata about the model prediction. These additional fields can
be visualized in the App or used by Brain methods, e.g., when
[computing label mistakes](../brain/index.md#brain-label-mistakes).
```python
import fiftyone as fo
sample = fo.Sample(filepath="/path/to/image.png")
sample["ground_truth"] = fo.Classification(label="sunny")
sample["prediction"] = fo.Classification(label="sunny", confidence=0.9)
print(sample)
```
```text
,
'prediction': ,
}>
```
#### NOTE
Did you know? You can [store class lists](#storing-classes) for your
models on your datasets.
### Multilabel classification
The [`Classifications`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Classifications) class represents a list of classification labels for an
image. The typical use case is to represent multilabel annotations/predictions
for an image, where multiple labels from a model may apply to a given image.
The labels are stored in a
[`classifications`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Classifications.classifications)
attribute of the object, which contains a list of [`Classification`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Classification) instances.
Metadata about individual labels can be stored in the [`Classification`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Classification)
instances as usual; additionally, you can optionally store logits for the
overarching model (if applicable) in the
[`logits`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Classifications.logits) attribute of the
[`Classifications`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Classifications) object.
```python
import fiftyone as fo
sample = fo.Sample(filepath="/path/to/image.png")
sample["ground_truth"] = fo.Classifications(
classifications=[
fo.Classification(label="animal"),
fo.Classification(label="cat"),
fo.Classification(label="tabby"),
]
)
sample["prediction"] = fo.Classifications(
classifications=[
fo.Classification(label="animal", confidence=0.99),
fo.Classification(label="cat", confidence=0.98),
fo.Classification(label="tabby", confidence=0.72),
]
)
print(sample)
```
```text
,
,
,
],
'logits': None,
}>,
'prediction': ,
,
,
],
'logits': None,
}>,
}>
```
#### NOTE
Did you know? You can [store class lists](#storing-classes) for your
models on your datasets.
### Object detection
The [`Detections`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Detections) class represents a list of object detections in an image. The
detections are stored in the
[`detections`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Detections.detections) attribute of
the [`Detections`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Detections) object.
Each individual object detection is represented by a [`Detection`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Detection) object. The
string label of the object should be stored in the
[`label`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Detection.label) attribute, and the
bounding box for the object should be stored in the
[`bounding_box`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Detection.bounding_box) attribute.
#### NOTE
FiftyOne stores box coordinates as floats in `[0, 1]` relative to the
dimensions of the image. Bounding boxes are represented by a length-4 list
in the format:
```text
[, , , ]
```
#### NOTE
Did you know? FiftyOne also supports [3D detections](#d-detections)!
In the case of model predictions, an optional confidence score for each
detection can be stored in the
[`confidence`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Detection.confidence) attribute.
```python
import fiftyone as fo
sample = fo.Sample(filepath="/path/to/image.png")
sample["ground_truth"] = fo.Detections(
detections=[fo.Detection(label="cat", bounding_box=[0.5, 0.5, 0.4, 0.3])]
)
sample["prediction"] = fo.Detections(
detections=[
fo.Detection(
label="cat",
bounding_box=[0.480, 0.513, 0.397, 0.288],
confidence=0.96,
),
]
)
print(sample)
```
```text
,
],
}>,
'prediction': ,
],
}>,
}>
```
#### NOTE
Did you know? You can [store class lists](#storing-classes) for your
models on your datasets.
Like all [`Label`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Label) types, you can also add custom attributes to your detections
by dynamically adding new fields to each [`Detection`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Detection) instance:
```python
import fiftyone as fo
detection = fo.Detection(
label="cat",
bounding_box=[0.5, 0.5, 0.4, 0.3],
age=51, # custom attribute
mood="salty", # custom attribute
)
print(detection)
```
```text
```
#### NOTE
Did you know? You can view custom attributes in the
[App tooltip](app.md#app-sample-view) by hovering over the objects.
### Instance segmentations
Object detections stored in [`Detections`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Detections) may also have instance segmentation
masks.
These masks can be stored in one of two ways: either directly in the database
via the [`mask`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Detection.mask) attribute, or on
disk referenced by the
[`mask_path`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Detection.mask_path) attribute.
Masks stored directly in the database must be 2D numpy arrays
containing either booleans or 0/1 integers that encode the extent of the
instance mask within the
[`bounding_box`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Detection.bounding_box) of the
object.
For masks stored on disk, the
[`mask_path`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Detection.mask_path) attribute should
contain the file path to the mask image. We recommend storing masks as
single-channel PNG images, where a pixel value of 0 indicates the
background (rendered as transparent in the App), and any other
value indicates the object.
Masks can be of any size; they are stretched as necessary to fill the
object’s bounding box when visualizing in the App.
```python
import numpy as np
from PIL import Image
import fiftyone as fo
# Example instance mask
mask = ((np.random.randn(32, 32) > 0) * 255).astype(np.uint8)
mask_path = "/path/to/mask.png"
Image.fromarray(mask).save(mask_path)
sample = fo.Sample(filepath="/path/to/image.png")
sample["prediction"] = fo.Detections(
detections=[
fo.Detection(
label="cat",
bounding_box=[0.480, 0.513, 0.397, 0.288],
mask_path=mask_path,
confidence=0.96,
),
]
)
print(sample)
```
```text
,
],
}>,
}>
```
Like all [`Label`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Label) types, you can also add custom attributes to your instance
segmentations by dynamically adding new fields to each [`Detection`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Detection) instance:
```python
import numpy as np
import fiftyone as fo
detection = fo.Detection(
label="cat",
bounding_box=[0.5, 0.5, 0.4, 0.3],
mask_path="/path/to/mask.png",
age=51, # custom attribute
mood="salty", # custom attribute
)
print(detection)
```
```text
```
#### NOTE
Did you know? You can view custom attributes in the
[App tooltip](app.md#app-sample-view) by hovering over the objects.
### Polylines and polygons
The [`Polylines`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Polylines) class represents a list of
[polylines](https://en.wikipedia.org/wiki/Polygonal_chain) or
[polygons](https://en.wikipedia.org/wiki/Polygon) in an image. The polylines
are stored in the
[`polylines`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Polylines.polylines) attribute of the
[`Polylines`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Polylines) object.
Each individual polyline is represented by a [`Polyline`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Polyline) object, which
represents a set of one or more semantically related shapes in an image. The
[`points`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Polyline.points) attribute contains a
list of lists of `(x, y)` coordinates defining the vertices of each shape
in the polyline. If the polyline represents a closed curve, you can set the
[`closed`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Polyline.closed) attribute to `True` to
indicate that a line segment should be drawn from the last vertex to the first
vertex of each shape in the polyline. If the shapes should be filled when
rendering them, you can set the
[`filled`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Polyline.filled) attribute to `True`.
Polylines can also have string labels, which are stored in their
[`label`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Polyline.label) attribute.
#### NOTE
FiftyOne stores vertex coordinates as floats in `[0, 1]` relative to the
dimensions of the image.
#### NOTE
Did you know? FiftyOne also supports [3D polylines](#d-polylines)!
```python
import fiftyone as fo
sample = fo.Sample(filepath="/path/to/image.png")
# A simple polyline
polyline1 = fo.Polyline(
points=[[(0.3, 0.3), (0.7, 0.3), (0.7, 0.3)]],
closed=False,
filled=False,
)
# A closed, filled polygon with a label
polyline2 = fo.Polyline(
label="triangle",
points=[[(0.1, 0.1), (0.3, 0.1), (0.3, 0.3)]],
closed=True,
filled=True,
)
sample["polylines"] = fo.Polylines(polylines=[polyline1, polyline2])
print(sample)
```
```text
,
,
],
}>,
}>
```
Like all [`Label`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Label) types, you can also add custom attributes to your polylines by
dynamically adding new fields to each [`Polyline`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Polyline) instance:
```python
import fiftyone as fo
polyline = fo.Polyline(
label="triangle",
points=[[(0.1, 0.1), (0.3, 0.1), (0.3, 0.3)]],
closed=True,
filled=True,
kind="right", # custom attribute
)
print(polyline)
```
```text
```
#### NOTE
Did you know? You can view custom attributes in the
[App tooltip](app.md#app-sample-view) by hovering over the objects.
### Cuboids
Available in:
Open SourceEnterprise
You can store and visualize cuboids in FiftyOne using the
[`Polyline.from_cuboid()`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Polyline.from_cuboid)
method.
The method accepts a list of 8 `(x, y)` points describing the vertices of the
cuboid in the format depicted below:
```text
7--------6
/| /|
/ | / |
3--------2 |
| 4-----|--5
| / | /
|/ |/
0--------1
```
#### NOTE
FiftyOne stores vertex coordinates as floats in `[0, 1]` relative to the
dimensions of the image.
```python
import cv2
import numpy as np
import fiftyone as fo
def random_cuboid(frame_size):
width, height = frame_size
x0, y0 = np.array([width, height]) * ([0, 0.2] + 0.8 * np.random.rand(2))
dx, dy = (min(0.8 * width - x0, y0 - 0.2 * height)) * np.random.rand(2)
x1, y1 = x0 + dx, y0 - dy
w, h = (min(width - x1, y1)) * np.random.rand(2)
front = [(x0, y0), (x0 + w, y0), (x0 + w, y0 - h), (x0, y0 - h)]
back = [(x1, y1), (x1 + w, y1), (x1 + w, y1 - h), (x1, y1 - h)]
vertices = front + back
return fo.Polyline.from_cuboid(
vertices, frame_size=frame_size, label="cuboid"
)
frame_size = (256, 128)
filepath = "/tmp/image.png"
size = (frame_size[1], frame_size[0], 3)
cv2.imwrite(filepath, np.full(size, 255, dtype=np.uint8))
dataset = fo.Dataset("cuboids")
dataset.add_samples(
[
fo.Sample(filepath=filepath, cuboid=random_cuboid(frame_size))
for _ in range(51)]
)
session = fo.launch_app(dataset)
```
Like all [`Label`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Label) types, you can also add custom attributes to your cuboids by
dynamically adding new fields to each [`Polyline`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Polyline) instance:
```python
polyline = fo.Polyline.from_cuboid(
vertics, frame_size=frame_size,
label="vehicle",
filled=True,
type="sedan", # custom attribute
)
```
#### NOTE
Did you know? You can view custom attributes in the
[App tooltip](app.md#app-sample-view) by hovering over the objects.
### Rotated bounding boxes
Available in:
Open SourceEnterprise
You can store and visualize rotated bounding boxes in FiftyOne using the
[`Polyline.from_rotated_box()`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Polyline.from_rotated_box)
method, which accepts rotated boxes described by their center coordinates,
width/height, and counter-clockwise rotation, in radians.
#### NOTE
FiftyOne stores vertex coordinates as floats in `[0, 1]` relative to the
dimensions of the image.
```python
import cv2
import numpy as np
import fiftyone as fo
def random_rotated_box(frame_size):
width, height = frame_size
xc, yc = np.array([width, height]) * (0.2 + 0.6 * np.random.rand(2))
w, h = 1.5 * (min(xc, yc, width - xc, height - yc)) * np.random.rand(2)
theta = 2 * np.pi * np.random.rand()
return fo.Polyline.from_rotated_box(
xc, yc, w, h, theta, frame_size=frame_size, label="box"
)
frame_size = (256, 128)
filepath = "/tmp/image.png"
size = (frame_size[1], frame_size[0], 3)
cv2.imwrite(filepath, np.full(size, 255, dtype=np.uint8))
dataset = fo.Dataset("rotated-boxes")
dataset.add_samples(
[
fo.Sample(filepath=filepath, box=random_rotated_box(frame_size))
for _ in range(51)
]
)
session = fo.launch_app(dataset)
```
Like all [`Label`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Label) types, you can also add custom attributes to your rotated
bounding boxes by dynamically adding new fields to each [`Polyline`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Polyline) instance:
```python
polyline = fo.Polyline.from_rotated_box(
xc, yc, width, height, theta, frame_size=frame_size,
label="cat",
mood="surly", # custom attribute
)
```
#### NOTE
Did you know? You can view custom attributes in the
[App tooltip](app.md#app-sample-view) by hovering over the objects.
### Keypoints
Available in:
Open SourceEnterprise
The [`Keypoints`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Keypoints) class represents a collection of keypoint groups in an image.
The keypoint groups are stored in the
[`keypoints`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Keypoints.keypoints) attribute of the
[`Keypoints`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Keypoints) object. Each element of this list is a [`Keypoint`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Keypoint) object whose
[`points`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Keypoint.points) attribute contains a
list of `(x, y)` coordinates defining a group of semantically related
keypoints in the image.
For example, if you are working with a person model that outputs 18 keypoints
(`left eye`, `right eye`, `nose`, etc.) per person, then each [`Keypoint`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Keypoint)
instance would represent one person, and a [`Keypoints`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Keypoints) instance would represent
the list of people in the image.
#### NOTE
FiftyOne stores keypoint coordinates as floats in `[0, 1]` relative to the
dimensions of the image.
Each [`Keypoint`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Keypoint) object can have a string label, which is stored in its
[`label`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Keypoint.label) attribute, and it can
optionally have a list of per-point confidences in `[0, 1]` in its
[`confidence`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Keypoint.confidence) attribute:
```python
import fiftyone as fo
sample = fo.Sample(filepath="/path/to/image.png")
sample["keypoints"] = fo.Keypoints(
keypoints=[
fo.Keypoint(
label="square",
points=[(0.3, 0.3), (0.7, 0.3), (0.7, 0.7), (0.3, 0.7)],
confidence=[0.6, 0.7, 0.8, 0.9],
)
]
)
print(sample)
```
```text
,
],
}>,
}>
```
Like all [`Label`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Label) types, you can also add custom attributes to your keypoints by
dynamically adding new fields to each [`Keypoint`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Keypoint) instance. As a special case,
if you add a custom list attribute to a [`Keypoint`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Keypoint) instance whose length
matches the number of points, these values will be interpreted as per-point
attributes and rendered as such in the App:
```python
import fiftyone as fo
keypoint = fo.Keypoint(
label="rectangle",
kind="square", # custom object attribute
points=[(0.3, 0.3), (0.7, 0.3), (0.7, 0.7), (0.3, 0.7)],
confidence=[0.6, 0.7, 0.8, 0.9],
occluded=[False, False, True, False], # custom per-point attributes
)
print(keypoint)
```
```text
```
If your keypoints have semantic meanings, you can
[store keypoint skeletons](#storing-keypoint-skeletons) on your dataset to
encode the meanings.
If you are working with keypoint skeletons and a particular point is missing or
not visible for an instance, use nan values for its coordinates:
```python
keypoint = fo.Keypoint(
label="rectangle",
points=[
(0.3, 0.3),
(float("nan"), float("nan")), # use nan to encode missing points
(0.7, 0.7),
(0.3, 0.7),
],
)
```
#### NOTE
Did you know? When you view datasets with
[keypoint skeletons](#storing-keypoint-skeletons) in the App, label
strings and edges will be drawn when you visualize the keypoint fields.
### Semantic segmentation
Available in:
Open SourceEnterprise
The [`Segmentation`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Segmentation) class represents a semantic segmentation mask for an image
with integer values encoding the semantic labels for each pixel in the image.
The mask can either be stored on disk and referenced via the
[`mask_path`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Segmentation.mask_path) attribute or
stored directly in the database via the
[`mask`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Segmentation.mask) attribute.
#### NOTE
It is recommended to store segmentations on disk and reference them via the
[`mask_path`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Segmentation.mask_path) attribute,
for efficiency.
Note that [`mask_path`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Segmentation.mask_path)
must contain the **absolute path** to the mask on disk in order to use the
dataset from different current working directories in the future.
Segmentation masks can be stored in either of these formats:
- 2D 8-bit or 16-bit images or numpy arrays
- 3D 8-bit RGB images or numpy arrays
Segmentation masks can have any size; they are stretched as necessary to fit
the image’s extent when visualizing in the App.
```python
import cv2
import numpy as np
import fiftyone as fo
# Example segmentation mask
mask_path = "/tmp/segmentation.png"
mask = np.random.randint(10, size=(128, 128), dtype=np.uint8)
cv2.imwrite(mask_path, mask)
sample = fo.Sample(filepath="/path/to/image.png")
sample["segmentation1"] = fo.Segmentation(mask_path=mask_path)
sample["segmentation2"] = fo.Segmentation(mask=mask)
print(sample)
```
```text
,
'segmentation2': ,
}>
```
When you load datasets with [`Segmentation`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Segmentation) fields containing 2D masks in the
App, each pixel value is rendered as a different color (if possible) from the
App’s color pool. When you view RGB segmentation masks in the App, the mask
colors are always used.
#### NOTE
Did you know? You can [store semantic labels](#storing-mask-targets)
for your segmentation fields on your dataset. Then, when you view the
dataset in the App, label strings will appear in the App’s tooltip when you
hover over pixels.
#### NOTE
The pixel value `0` and RGB value `#000000` are reserved “background”
classes that are always rendered as invisible in the App.
If [mask targets](#storing-mask-targets) are provided, all observed
values not present in the targets are also rendered as invisible in the
App.
### Heatmaps
Available in:
Open SourceEnterprise
The [`Heatmap`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Heatmap) class represents a continuous-valued heatmap for an image.
The map can either be stored on disk and referenced via the
[`map_path`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Heatmap.map_path) attribute or stored
directly in the database via the [`map`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Heatmap.map)
attribute. When using the
[`map_path`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Heatmap.map_path) attribute, heatmaps
may be 8-bit or 16-bit grayscale images. When using the
[`map`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Heatmap.map) attribute, heatmaps should be 2D
numpy arrays. By default, the map values are assumed to be in `[0, 1]` for
floating point arrays and `[0, 255]` for integer-valued arrays, but you can
specify a custom `[min, max]` range for a map by setting its optional
[`range`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Heatmap.range) attribute.
Heatmaps can have any size; they are stretched as necessary to fit the
image’s extent when visualizing in the App.
#### NOTE
It is recommended to store heatmaps on disk and reference them via the
[`map_path`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Heatmap.map_path) attribute, for
efficiency.
Note that [`map_path`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Heatmap.map_path)
must contain the **absolute path** to the map on disk in order to use the
dataset from different current working directories in the future.
```python
import cv2
import numpy as np
import fiftyone as fo
# Example heatmap
map_path = "/tmp/heatmap.png"
map = np.random.randint(256, size=(128, 128), dtype=np.uint8)
cv2.imwrite(map_path, map)
sample = fo.Sample(filepath="/path/to/image.png")
sample["heatmap1"] = fo.Heatmap(map_path=map_path)
sample["heatmap2"] = fo.Heatmap(map=map)
print(sample)
```
```text
,
'heatmap2': ,
}>
```
When visualizing heatmaps [in the App](app.md#fiftyone-app), when the App is
in color-by-field mode, heatmaps are rendered in their field’s color with
opacity proportional to the magnitude of the heatmap’s values. For example, for
a heatmap whose [`range`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Heatmap.range) is
`[-10, 10]`, pixels with the value +9 will be rendered with 90% opacity, and
pixels with the value -3 will be rendered with 30% opacity.
When the App is in color-by-value mode, heatmaps are rendered using the
colormap defined by the `colorscale` of your
[App config](config.md#configuring-fiftyone-app), which can be:
- The string name of any colorscale
[recognized by Plotly](https://plotly.com/python/colorscales)
- A manually-defined colorscale like the following:
```default
[
[0.000, "rgb(165,0,38)"],
[0.111, "rgb(215,48,39)"],
[0.222, "rgb(244,109,67)"],
[0.333, "rgb(253,174,97)"],
[0.444, "rgb(254,224,144)"],
[0.555, "rgb(224,243,248)"],
[0.666, "rgb(171,217,233)"],
[0.777, "rgb(116,173,209)"],
[0.888, "rgb(69,117,180)"],
[1.000, "rgb(49,54,149)"],
]
```
The example code below demonstrates the possibilities that heatmaps provide by
overlaying random gaussian kernels with positive or negative sign on an image
dataset and configuring the App’s colorscale in various ways on-the-fly:
```python
import os
import numpy as np
import fiftyone as fo
import fiftyone.zoo as foz
def random_kernel(metadata):
h = metadata.height // 2
w = metadata.width // 2
sign = np.sign(np.random.randn())
x, y = np.meshgrid(np.linspace(-1, 1, w), np.linspace(-1, 1, h))
x0, y0 = np.random.random(2) - 0.5
kernel = sign * np.exp(-np.sqrt((x - x0) ** 2 + (y - y0) ** 2))
return fo.Heatmap(map=kernel, range=[-1, 1])
dataset = foz.load_zoo_dataset("quickstart").select_fields().clone()
dataset.compute_metadata()
for sample in dataset:
heatmap = random_kernel(sample.metadata)
# Convert to on-disk
map_path = os.path.join("/tmp/heatmaps", os.path.basename(sample.filepath))
heatmap.export_map(map_path, update=True)
sample["heatmap"] = heatmap
sample.save()
session = fo.launch_app(dataset)
```
```python
# Select `Settings -> Color by value` in the App
# Heatmaps will now be rendered using your default colorscale (printed below)
print(session.config.colorscale)
```
```python
# Switch to a different named colorscale
session.config.colorscale = "RdBu"
session.refresh()
```
```python
# Switch to a custom colorscale
session.config.colorscale = [
[0.00, "rgb(166,206,227)"],
[0.25, "rgb(31,120,180)"],
[0.45, "rgb(178,223,138)"],
[0.65, "rgb(51,160,44)"],
[0.85, "rgb(251,154,153)"],
[1.00, "rgb(227,26,28)"],
]
session.refresh()
```
#### NOTE
Did you know? You customize your App config in various ways, from
environment variables to directly editing a [`Session`](../api/fiftyone.core.session.md#fiftyone.core.session.Session) object’s config.
See [this page](config.md#configuring-fiftyone-app) for more details.
### Temporal detection
The [`TemporalDetection`](../api/fiftyone.core.labels.md#fiftyone.core.labels.TemporalDetection) class represents an event occurring during a specified
range of frames in a video.
The [`label`](../api/fiftyone.core.labels.md#fiftyone.core.labels.TemporalDetection.label) attribute
stores the detection label, and the
[`support`](../api/fiftyone.core.labels.md#fiftyone.core.labels.TemporalDetection.support) attribute
stores the `[first, last]` frame range of the detection in the video.
The optional
[`confidence`](../api/fiftyone.core.labels.md#fiftyone.core.labels.TemporalDetection.confidence)
attribute can be used to store a model prediction score, and you can add
[custom attributes](#using-labels) as well, which can be visualized in the
App.
```python
import fiftyone as fo
sample = fo.Sample(filepath="/path/to/video.mp4")
sample["events"] = fo.TemporalDetection(label="meeting", support=[10, 20])
print(sample)
```
```text
,
'frames': ,
}>
```
If your temporal detection data is represented as timestamps in seconds, you
can use the
[`from_timestamps()`](../api/fiftyone.core.labels.md#fiftyone.core.labels.TemporalDetection.from_timestamps)
factory method to perform the necessary conversion to frames automatically
based on the sample’s [video metadata](#using-metadata):
```python
import fiftyone as fo
import fiftyone.zoo as foz
# Download a video to work with
dataset = foz.load_zoo_dataset("quickstart-video", max_samples=1)
filepath = dataset.first().filepath
sample = fo.Sample(filepath=filepath)
sample.compute_metadata()
sample["events"] = fo.TemporalDetection.from_timestamps(
[1, 2], label="meeting", sample=sample
)
print(sample)
```
```text
,
'created_at': None,
'last_modified_at': None,
'events': ,
'frames': ,
}>
```
The [`TemporalDetections`](../api/fiftyone.core.labels.md#fiftyone.core.labels.TemporalDetections) class holds a list of temporal detections for a
sample:
```python
import fiftyone as fo
sample = fo.Sample(filepath="/path/to/video.mp4")
sample["events"] = fo.TemporalDetections(
detections=[
fo.TemporalDetection(label="meeting", support=[10, 20]),
fo.TemporalDetection(label="party", support=[30, 60]),
]
)
print(sample)
```
```text
,
,
],
}>,
'frames': ,
}>
```
#### NOTE
Did you know? You can [store class lists](#storing-classes) for your
models on your datasets.
### 3D detections
The App’s [3D visualizer](app.md#app-3d-visualizer) supports rendering 3D object
detections represented as [`Detection`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Detection) instances with their `label`, `location`,
`dimensions`, and `rotation` attributes populated as shown below:
```python
import fiftyone as fo
# Object label
label = "vehicle"
# Object center `[x, y, z]` in scene coordinates
location = [0.47, 1.49, 69.44]
# Object dimensions `[x, y, z]` in scene units
dimensions = [2.85, 2.63, 12.34]
# Object rotation `[x, y, z]` around its center, in `[-pi, pi]`
rotation = [0, -1.56, 0]
# A 3D object detection
detection = fo.Detection(
label=label,
location=location,
dimensions=dimensions,
rotation=rotation,
)
```
#### NOTE
Did you know? You can view custom attributes in the
[App tooltip](app.md#app-sample-view) by hovering over the objects.
### 3D polylines
The App’s [3D visualizer](app.md#app-3d-visualizer) supports rendering 3D
polylines represented as [`Polyline`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Polyline) instances with their `label` and `points3d`
attributes populated as shown below:
```python
import fiftyone as fo
# Object label
label = "lane"
# A list of lists of `[x, y, z]` points in scene coordinates describing
# the vertices of each shape in the polyline
points3d = [[[-5, -99, -2], [-8, 99, -2]], [[4, -99, -2], [1, 99, -2]]]
# A set of semantically related 3D polylines
polyline = fo.Polyline(label=label, points3d=points3d)
```
#### NOTE
Did you know? You can view custom attributes in the
[App tooltip](app.md#app-sample-view) by hovering over the objects.
### Geolocation
The [`GeoLocation`](../api/fiftyone.core.labels.md#fiftyone.core.labels.GeoLocation) class can store single pieces of location data in its
properties:
- [`point`](../api/fiftyone.core.labels.md#fiftyone.core.labels.GeoLocation.point): a
`[longitude, latitude]` point
- [`line`](../api/fiftyone.core.labels.md#fiftyone.core.labels.GeoLocation.line): a line of longitude
and latitude coordinates stored in the following format:
```default
[[lon1, lat1], [lon2, lat2], ...]
```
- [`polygon`](../api/fiftyone.core.labels.md#fiftyone.core.labels.GeoLocation.polygon): a polygon of
longitude and latitude coordinates stored in the format below, where the
first element describes the boundary of the polygon and any remaining
entries describe holes:
```default
[
[[lon1, lat1], [lon2, lat2], ...],
[[lon1, lat1], [lon2, lat2], ...],
...
]
```
#### NOTE
All geolocation coordinates are stored in `[longitude, latitude]`
format.
If you have multiple geometries of each type that you wish to store on a single
sample, then you can use the [`GeoLocations`](../api/fiftyone.core.labels.md#fiftyone.core.labels.GeoLocations) class and its appropriate
properties to do so.
```python
import fiftyone as fo
sample = fo.Sample(filepath="/path/to/image.png")
sample["location"] = fo.GeoLocation(
point=[-73.9855, 40.7580],
polygon=[
[
[-73.949701, 40.834487],
[-73.896611, 40.815076],
[-73.998083, 40.696534],
[-74.031751, 40.715273],
[-73.949701, 40.834487],
]
],
)
print(sample)
```
```text
,
}>
```
#### NOTE
Did you know? You can create
[location-based views](using_views.md#geolocation-views) that filter your data by
their location!
All location data is stored in
[GeoJSON format](https://en.wikipedia.org/wiki/GeoJSON) in the database. You
can easily retrieve the raw GeoJSON data for a slice of your dataset using the
[values()](using_aggregations.md#aggregations-values) aggregation:
```python
import fiftyone as fo
import fiftyone.zoo as foz
dataset = foz.load_zoo_dataset("quickstart-geo")
values = dataset.take(5).values("location.point", _raw=True)
print(values)
```
```text
[{'type': 'Point', 'coordinates': [-73.9592175465766, 40.71052995514191]},
{'type': 'Point', 'coordinates': [-73.97748118760413, 40.74660360881843]},
{'type': 'Point', 'coordinates': [-73.9508690871987, 40.766631164626]},
{'type': 'Point', 'coordinates': [-73.96569416502996, 40.75449283200206]},
{'type': 'Point', 'coordinates': [-73.97397106211423, 40.67925541341504]}]
```
### Label tags
All [`Label`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Label) instances have a `tags` field, which is a string list. By default,
this list is empty, but you can use it to store application-specific
information like whether the label is incorrect:
```python
detection = fo.Detection(label="cat", bounding_box=[0, 0, 1, 1])
detection.tags.append("mistake")
print(detection.tags)
# ["mistake"]
```
#### NOTE
Did you know? You can add, edit, and filter by label tags
[directly in the App](app.md#app-tagging).
Datasets and views provide helpful methods such as
[`count_label_tags()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.count_label_tags),
[`tag_labels()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.tag_labels),
[`untag_labels()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.untag_labels),
[`match_labels()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.match_labels),
and
[`select_labels()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.select_labels)
that you can use to perform batch queries and edits to label tags:
```python
import fiftyone as fo
import fiftyone.zoo as foz
from fiftyone import ViewField as F
dataset = foz.load_zoo_dataset("quickstart").clone()
# Tag all low confidence prediction
view = dataset.filter_labels("predictions", F("confidence") < 0.1)
view.tag_labels("potential_mistake", label_fields="predictions")
print(dataset.count_label_tags()) # {'potential_mistake': 1555}
# Create a view containing only tagged labels
view = dataset.select_labels(tags="potential_mistake", fields="predictions")
print(len(view)) # 173
print(view.count("predictions.detections")) # 1555
# Create a view containing only samples with at least one tagged label
view = dataset.match_labels(tags="potential_mistake", fields="predictions")
print(len(view)) # 173
print(view.count("predictions.detections")) # 5151
dataset.untag_labels("potential_mistake", label_fields="predictions")
print(dataset.count_label_tags()) # {}
```
### Label attributes
The [`Detection`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Detection), [`Polyline`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Polyline), and [`Keypoint`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Keypoint) label types have an optional
[`attributes`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Detection.attributes) field that you
can use to store custom attributes on the object.
The [`attributes`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Detection.attributes) field is a
dictionary mapping attribute names to [`Attribute`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Attribute) instances, which contain the
[`value`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Attribute.value) of the attribute and any
associated metadata.
#### WARNING
The [`attributes`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Detection.attributes) field
will be removed in an upcoming release.
Instead, [add custom attributes directly](#using-labels) to your
[`Label`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Label) objects:
```python
detection = fo.Detection(label="cat", bounding_box=[0.1, 0.1, 0.8, 0.8])
detection["custom_attribute"] = 51
# Equivalent
detection = fo.Detection(
label="cat",
bounding_box=[0.1, 0.1, 0.8, 0.8],
custom_attribute=51,
)
```
There are [`Attribute`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Attribute) subclasses for various types of attributes you may want
to store. Use the appropriate subclass when possible so that FiftyOne knows the
schema of the attributes that you’re storing.
| Attribute class | Value type | Description |
|----------------------------------------------------------------------------------------------------|--------------|---------------------------------|
| [`BooleanAttribute`](../api/fiftyone.core.labels.md#fiftyone.core.labels.BooleanAttribute) | `bool` | A boolean attribute |
| [`CategoricalAttribute`](../api/fiftyone.core.labels.md#fiftyone.core.labels.CategoricalAttribute) | `string` | A categorical attribute |
| [`NumericAttribute`](../api/fiftyone.core.labels.md#fiftyone.core.labels.NumericAttribute) | `float` | A numeric attribute |
| [`ListAttribute`](../api/fiftyone.core.labels.md#fiftyone.core.labels.ListAttribute) | `list` | A list attribute |
| [`Attribute`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Attribute) | arbitrary | A generic attribute of any type |
```python
import fiftyone as fo
sample = fo.Sample(filepath="/path/to/image.png")
sample["ground_truth"] = fo.Detections(
detections=[
fo.Detection(
label="cat",
bounding_box=[0.5, 0.5, 0.4, 0.3],
attributes={
"age": fo.NumericAttribute(value=51),
"mood": fo.CategoricalAttribute(value="salty"),
},
),
]
)
sample["prediction"] = fo.Detections(
detections=[
fo.Detection(
label="cat",
bounding_box=[0.480, 0.513, 0.397, 0.288],
confidence=0.96,
attributes={
"age": fo.NumericAttribute(value=51),
"mood": fo.CategoricalAttribute(
value="surly", confidence=0.95
),
},
),
]
)
print(sample)
```
```text
,
'mood': ,
},
'tags': [],
'label': 'cat',
'bounding_box': [0.5, 0.5, 0.4, 0.3],
'mask': None,
'confidence': None,
'index': None,
}>,
],
}>,
'prediction': ,
'mood': ,
},
'tags': [],
'label': 'cat',
'bounding_box': [0.48, 0.513, 0.397, 0.288],
'mask': None,
'confidence': 0.96,
'index': None,
}>,
],
}>,
}>
```
#### NOTE
Did you know? You can view attribute values in the
[App tooltip](app.md#app-sample-view) by hovering over the objects.
### Converting label types
FiftyOne provides a number of utility methods to convert between different
representations of certain label types, such as converting between
[instance segmentations](#instance-segmentation),
[semantic segmentations](#semantic-segmentation),
and [polylines](#polylines).
Let’s load some instance segmentations from the COCO dataset to see this in
action:
```python
import fiftyone as fo
import fiftyone.zoo as foz
dataset = foz.load_zoo_dataset(
"coco-2017",
split="validation",
label_types=["segmentations"],
classes=["cat", "dog"],
label_field="instances",
max_samples=25,
only_matching=True,
)
sample = dataset.first()
detections = sample["instances"]
```
For example, you can use
[`Detections.to_polylines()`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Detections.to_polylines)
to convert instance segmentations to polylines:
```python
# Convert `Detections` to `Polylines`
polylines = detections.to_polylines(tolerance=2)
print(polylines)
```
Or you can use
[`Detections.to_segmentation()`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Detections.to_segmentation)
to convert instance segmentations to semantic segmentation masks:
```python
metadata = fo.ImageMetadata.build_for(sample.filepath)
# Convert `Detections` to `Segmentation`
segmentation = detections.to_segmentation(
frame_size=(metadata.width, metadata.height),
mask_targets={1: "cat", 2: "dog"},
)
# Export the segmentation to disk
segmentation.export_mask("/tmp/mask.png", update=True)
print(segmentation)
```
Methods such as
[`Segmentation.to_detections()`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Segmentation.to_detections)
and [`Segmentation.to_polylines()`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Segmentation.to_polylines)
also exist to transform semantic segmentations back into individual shapes.
In addition, the [`fiftyone.utils.labels`](../api/fiftyone.utils.labels.md#module-fiftyone.utils.labels) module contains a variety of
utility methods for converting entire collections’ labels between common
formats:
```python
import fiftyone.utils.labels as foul
# Convert instance segmentations to semantic segmentations stored on disk
foul.objects_to_segmentations(
dataset,
"instances",
"segmentations",
output_dir="/tmp/segmentations",
mask_targets={1: "cat", 2: "dog"},
)
# Convert instance segmentations to polylines format
foul.instances_to_polylines(dataset, "instances", "polylines", tolerance=2)
# Convert semantic segmentations to instance segmentations
foul.segmentations_to_detections(
dataset,
"segmentations",
"instances2",
mask_targets={1: "cat", 2: "dog"},
mask_types="thing", # give each connected region a separate instance
)
print(dataset)
```
```shell
Name: coco-2017-validation-25
Media type: image
Num samples: 25
Persistent: False
Tags: []
Sample fields:
id: fiftyone.core.fields.ObjectIdField
filepath: fiftyone.core.fields.StringField
tags: fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)
metadata: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.ImageMetadata)
created_at: fiftyone.core.fields.DateTimeField
last_modified_at: fiftyone.core.fields.DateTimeField
instances: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)
segmentations: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Segmentation)
polylines: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Polylines)
instances2: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)
```
Note that, if your goal is to export the labels to disk, FiftyOne can
[automatically coerce](export_datasets.md#export-label-coercion) the labels into the correct
format based on the type of the `label_field` and the `dataset_type` that you
specify for the export without explicitly storing the transformed labels as a
new field on your dataset:
```python
# Export the instance segmentations in the `instances` field as semantic
# segmentation images on disk
dataset.export(
label_field="instances",
dataset_type=fo.types.ImageSegmentationDirectory,
labels_path="/tmp/masks",
mask_targets={1: "cat", 2: "dog"},
)
```
### Projecting 3D detections to 2D
FiftyOne provides functionality through
[`detections_3d_to_cuboids_2d`](../api/fiftyone.utils.labels.md#fiftyone.utils.labels.detections_3d_to_cuboids_2d)
to project 3D object detections into 2D images as polylines. This functionality
is specific to grouped datasets with multiple modalities such as images and
point clouds.
The 3D detections (`in_field`) should be stored in a field of type
[Detection](#d-detections) with the necessary attributes such as
`location`, `dimensions`, and `rotation` populated. You need to provide
the following additional information for the projections:
- **Coordinate transformation data** used to convert 3D bounding boxes from the
point cloud coordinate system into the camera coordinate system.
- Represented as a dictionary mapping the key field (defaults to sample id) of
the spatial slice to a list of transformation tuples.
- Each transformation is a `(translation, rotation)` tuple, where:
- `translation`: a 3-element vector (`list[float]` or `np.ndarray`)
specifying the translation in 3D space
- `rotation`: a 3x3 rotation matrix (`list[float]` or `np.ndarray`)
specifying the orientation
- **[Optional] Forward flags** indicating whether to apply the corresponding
transformation in the forward direction (from point cloud to camera) or in the
inverse direction (from camera to point cloud).
- Represented as a dictionary mapping the key field (defaults to sample id) of
the spatial slice to a list of boolean flags. The length of the list must
match the number of transformations for the sample.
- If not provided, all transformations are assumed to be applied in the
forward direction.
- **Camera model** used to account for camera intrinsics and lens distortion
during 3D-to-2D projection.
- Specified via the `camera_model` argument, which accepts a projection
function with the signature:
```python
projection_function(points3d: np.ndarray, camera_params: dict) -> np.ndarray
```
- `points3d`: an `(N, 3)` NumPy array of 3D points to be projected
- `camera_params`: a dictionary of camera parameters required by the
projection function
FiftyOne provides a pinhole camera model via the
[`utils3d.pinhole_projector`](../api/fiftyone.utils.utils3d.md#fiftyone.utils.utils3d.pinhole_projector)
function, which is used by default if no camera model is specified.
#### NOTE
For the pinhole camera model, the following orientation is assumed:
- x axis points to the right in the image plane
- y axis points down in the image plane
- z axis points forward from the camera
To use the pinhole camera model, you must ensure that your 3D boxes are
oriented in the same way, likely by applying an additional rotation
transformation.
- **Camera parameter data** to project the 3D boxes onto the 2D image plane.
- Represented as a dictionary mapping key field of the camera slice to dicts
with the keys required by the camera model you are using.
- For example, for the default pinhole camera model, the dictionary should
contain the key `intrinsics`, which is a 3x3 `np.ndarray` camera
intrinsics matrix of the form:
```default
[[fx, 0, cx],
[ 0, fy, cy],
[ 0, 0, 1]]
```
The example code below demonstrates how to perform this projection using
FiftyOne:
```python
import fiftyone as fo
from fiftyone.utils.labels import detections_3d_to_cuboids_2d
import numpy as np
dataset = fo.load_dataset("your_grouped_dataset")
# Example transformation: Assume that for all samples, the same
# transformation is applied. A 90 degree rotation around the x axis to align
# the point cloud and camera coordinate systems with 0 translation
dataset.group_slice = "spatial_slice"
rotation = np.array([[1, 0, 0],
[0, 0, -1],
[0, 1, 0]])
translation = np.array([0, 0, 0])
# Example camera parameters: Assume that for all samples, the same camera
# intrinsics are used. Note that camera_params has to use camera slice fields
# as keys
intrinsics = np.array([[1000, 0, 512],
[0, 1000, 384],
[0, 0, 1]])
transformations = {}
camera_params = {}
for group in dataset.iter_groups():
lidar_sample = group["spatial_slice"]
transformations[lidar_sample.id] = [(translation, rotation)]
camera_sample = group["camera_slice"]
camera_params[camera_sample.id] = {
"intrinsics": intrinsics
}
# Assuming a grouped dataset with image and point cloud modalities
# - image slice named "camera_slice"
# - point cloud slice named "spatial_slice"
# - 3D detections stored in the "detections_3d" field
# The projected 2D cuboids will be stored in the "cuboids_2d" field
detections_3d_to_cuboids_2d(
dataset,
spatial_slice_name="spatial_slice",
camera_slice_name="camera_slice",
in_field="detections_3d",
out_field="cuboids_2d",
transformations=transformations,
camera_params=camera_params,
)
```
## Dynamic attributes
Available in:
Open SourceEnterprise
Any field(s) of your FiftyOne datasets that contain [`DynamicEmbeddedDocument`](../api/fiftyone.core.odm.embedded_document.md#fiftyone.core.odm.embedded_document.DynamicEmbeddedDocument)
values can have arbitrary custom attributes added to their instances.
For example, all [Label classes](#using-labels) and
[Metadata classes](#using-metadata) are dynamic, so you can add custom
attributes to them as follows:
```python
# Provide some default attributes
label = fo.Classification(label="cat", confidence=0.98)
# Add custom attributes
label["int"] = 5
label["float"] = 51.0
label["list"] = [1, 2, 3]
label["bool"] = True
label["dict"] = {"key": ["list", "of", "values"]}
```
By default, dynamic attributes are not included in a
[dataset’s schema](#field-schemas), which means that these attributes may
contain arbitrary heterogeneous values across the dataset’s samples.
However, FiftyOne provides methods that you can use to formally declare custom
dynamic attributes, which allows you to enforce type constraints, filter by
these custom attributes [in the App](app.md#app-filtering), and more.
You can use
[`get_dynamic_field_schema()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.get_dynamic_field_schema)
to detect the names and type(s) of any undeclared dynamic embedded document
attributes on a dataset:
```python
import fiftyone as fo
import fiftyone.zoo as foz
dataset = foz.load_zoo_dataset("quickstart")
print(dataset.get_dynamic_field_schema())
```
```text
{
'ground_truth.detections.iscrowd': ,
'ground_truth.detections.area': ,
}
```
You can then use
[`add_sample_field()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.add_sample_field) to
declare a specific dynamic embedded document attribute:
```python
dataset.add_sample_field("ground_truth.detections.iscrowd", fo.FloatField)
```
or you can use the
[`add_dynamic_sample_fields()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.add_dynamic_sample_fields)
method to declare all dynamic embedded document attribute(s) that contain
values of a single type:
```python
dataset.add_dynamic_sample_fields()
```
#### NOTE
Pass the `add_mixed=True` option to
[`add_dynamic_sample_fields()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.add_dynamic_sample_fields)
if you wish to declare all dynamic attributes that contain mixed values
using a generic [`Field`](../api/fiftyone.core.fields.md#fiftyone.core.fields.Field) type.
You can provide the optional `flat=True` option to
[`get_field_schema()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.get_field_schema) to
retrieve a flattened version of a dataset’s schema that includes all embedded
document attributes as top-level keys:
```python
print(dataset.get_field_schema(flat=True))
```
```text
{
'id': ,
'filepath': ,
'tags': ,
'metadata': ,
'metadata.size_bytes': ,
'metadata.mime_type': ,
'metadata.width': ,
'metadata.height': ,
'metadata.num_channels': ,
'created_at': ,
'last_modified_at': ,
'ground_truth': ,
'ground_truth.detections': ,
'ground_truth.detections.id': ,
'ground_truth.detections.tags': ,
...
'ground_truth.detections.iscrowd': ,
'ground_truth.detections.area': ,
...
}
```
By default, dynamic attributes are not declared on a dataset’s schema when
samples are added to it:
```python
import fiftyone as fo
sample = fo.Sample(
filepath="/path/to/image.jpg",
ground_truth=fo.Detections(
detections=[
fo.Detection(
label="cat",
bounding_box=[0.1, 0.1, 0.4, 0.4],
mood="surly",
),
fo.Detection(
label="dog",
bounding_box=[0.5, 0.5, 0.4, 0.4],
mood="happy",
)
]
)
)
dataset = fo.Dataset()
dataset.add_sample(sample)
schema = dataset.get_field_schema(flat=True)
assert "ground_truth.detections.mood" not in schema
```
However, methods such as
[`add_sample()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.add_sample),
[`add_samples()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.add_samples),
[`add_dir()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.add_dir),
[`from_dir()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.from_dir), and
[`merge_samples()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.merge_samples)
provide an optional `dynamic=True` option that you can provide to automatically
declare any dynamic embedded document attributes encountered while importing
data:
```python
dataset = fo.Dataset()
dataset.add_sample(sample, dynamic=True)
schema = dataset.get_field_schema(flat=True)
assert "ground_truth.detections.mood" in schema
```
Note that, when declaring dynamic attributes on non-empty datasets, you must
ensure that the attribute’s type is consistent with any existing values in that
field, e.g., by first running
[`get_dynamic_field_schema()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.get_dynamic_field_schema)
to check the existing type(s). Methods like
[`add_sample_field()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.add_sample_field)
and
[`add_samples(..., dynamic=True)`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.add_samples)
do not validate newly declared field’s types against existing field values:
```python
import fiftyone as fo
sample1 = fo.Sample(
filepath="/path/to/image1.jpg",
ground_truth=fo.Classification(
label="cat",
mood="surly",
age="bad-value",
),
)
sample2 = fo.Sample(
filepath="/path/to/image2.jpg",
ground_truth=fo.Classification(
label="dog",
mood="happy",
age=5,
),
)
dataset = fo.Dataset()
dataset.add_sample(sample1)
# Either of these are problematic
dataset.add_sample(sample2, dynamic=True)
dataset.add_sample_field("ground_truth.age", fo.IntField)
sample1.reload() # ValidationError: bad-value could not be converted to int
```
If you declare a dynamic attribute with a type that is not compatible with
existing values in that field, you will need to remove that field from the
dataset’s schema using
[`remove_dynamic_sample_field()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.remove_dynamic_sample_field)
in order for the dataset to be usable again:
```python
# Removes dynamic field from dataset's schema without deleting the values
dataset.remove_dynamic_sample_field("ground_truth.age")
```
You can use
[`select_fields()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.select_fields)
and
[`exclude_fields()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.exclude_fields)
to create [views](using_views.md#using-views) that select/exclude specific dynamic
attributes from your dataset and its schema:
```python
dataset.add_sample_field("ground_truth.age", fo.Field)
sample = dataset.first()
assert "ground_truth.age" in dataset.get_field_schema(flat=True)
assert sample.ground_truth.has_field("age")
# Omits the `age` attribute from the `ground_truth` field
view = dataset.exclude_fields("ground_truth.age")
sample = view.first()
assert "ground_truth.age" not in view.get_field_schema(flat=True)
assert not sample.ground_truth.has_field("age")
# Only include `mood` (and default) attributes of the `ground_truth` field
view = dataset.select_fields("ground_truth.mood")
sample = view.first()
assert "ground_truth.age" not in view.get_field_schema(flat=True)
assert not sample.ground_truth.has_field("age")
```
## Custom embedded documents
If you work with collections of related fields that you would like to organize
under a single top-level field, you can achieve this by defining and using
custom [`EmbeddedDocument`](../api/fiftyone.core.odm.embedded_document.md#fiftyone.core.odm.embedded_document.EmbeddedDocument) and [`DynamicEmbeddedDocument`](../api/fiftyone.core.odm.embedded_document.md#fiftyone.core.odm.embedded_document.DynamicEmbeddedDocument) classes to populate
your datasets.
Using custom embedded document classes enables you to access your data using
the same object-oriented interface enjoyed by FiftyOne’s
[builtin label types](#using-labels).
The [`EmbeddedDocument`](../api/fiftyone.core.odm.embedded_document.md#fiftyone.core.odm.embedded_document.EmbeddedDocument) class represents a fixed collection of fields with
predefined types and optional default values, while the
[`DynamicEmbeddedDocument`](../api/fiftyone.core.odm.embedded_document.md#fiftyone.core.odm.embedded_document.DynamicEmbeddedDocument) class supports predefined fields but also allows
users to populate arbitrary custom fields at runtime, like FiftyOne’s
[builtin label types](#using-labels).
### Defining custom documents on-the-fly
The simplest way to define custom embedded documents on your datasets is to
declare empty [`DynamicEmbeddedDocument`](../api/fiftyone.core.odm.embedded_document.md#fiftyone.core.odm.embedded_document.DynamicEmbeddedDocument) field(s) and then incrementally
populate new [dynamic attributes](#dynamic-attributes) as needed.
To illustrate, let’s start by defining an empty embedded document field:
```python
import fiftyone as fo
dataset = fo.Dataset()
# Define an empty embedded document field
dataset.add_sample_field(
"camera_info",
fo.EmbeddedDocumentField,
embedded_doc_type=fo.DynamicEmbeddedDocument,
)
```
From here, there are a variety of ways to add new embedded attributes to the
field.
You can explicitly declare new fields using
[`add_sample_field()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.add_sample_field):
```python
# Declare a new `camera_id` attribute
dataset.add_sample_field("camera_info.camera_id", fo.StringField)
assert "camera_info.camera_id" in dataset.get_field_schema(flat=True)
```
or you can implicitly declare new fields using
[`add_samples()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.add_samples) with the
`dynamic=True` flag:
```python
# Includes a new `quality` attribute
sample1 = fo.Sample(
filepath="/path/to/image1.jpg",
camera_info=fo.DynamicEmbeddedDocument(
camera_id="123456789",
quality=51.0,
),
)
sample2 = fo.Sample(
filepath="/path/to/image2.jpg",
camera_info=fo.DynamicEmbeddedDocument(camera_id="123456789"),
)
# Automatically declares new dynamic attributes as they are encountered
dataset.add_samples([sample1, sample2], dynamic=True)
assert "camera_info.quality" in dataset.get_field_schema(flat=True)
```
or you can implicitly declare new fields using
[`set_values()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.set_values)
with the `dynamic=True` flag:
```python
# Populate a new `description` attribute on each sample in the dataset
dataset.set_values("camera_info.description", ["foo", "bar"], dynamic=True)
assert "camera_info.description" in dataset.get_field_schema(flat=True)
```
### Defining custom documents in modules
You can also define custom embedded document classes in Python modules and
packages that you maintain, using the appropriate types from the
[`fiftyone.core.fields`](../api/fiftyone.core.fields.md#module-fiftyone.core.fields) module to declare your fields and their types,
defaults, etc.
The benefit of this approach over the on-the-fly definition from the previous
section is that you can provide extra metadata such as whether fields are
`required` or should have `default` values if they are not explicitly set
during creation.
#### WARNING
In order to work with datasets containing custom embedded documents defined
using this approach, you must configure your `module_path` in
*all environments* where you intend to work with the datasets that use
these classes, not just the environment where you create the dataset.
To avoid this requirement, consider defining custom documents
[on-the-fly](#defining-custom-documents-on-the-fly) instead.
For example, suppose you add the following embedded document classes to a
`foo.bar` module:
```python
from datetime import datetime
import fiftyone as fo
class CameraInfo(fo.EmbeddedDocument):
camera_id = fo.StringField(required=True)
quality = fo.FloatField()
description = fo.StringField()
class LabelMetadata(fo.DynamicEmbeddedDocument):
created_at = fo.DateTimeField(default=datetime.utcnow)
model_name = fo.StringField()
```
and then `foo.bar` to FiftyOne’s `module_path` config setting (see
[this page](config.md#configuring-fiftyone) for more ways to register this):
```shell
export FIFTYONE_MODULE_PATH=foo.bar
# Verify module path
fiftyone config
```
You’re now free to use your custom embedded document classes as you please,
whether this be top-level sample fields or nested fields:
```python
import fiftyone as fo
import foo.bar as fb
sample = fo.Sample(
filepath="/path/to/image.png",
camera_info=fb.CameraInfo(
camera_id="123456789",
quality=99.0,
),
weather=fo.Classification(
label="sunny",
confidence=0.95,
metadata=fb.LabelMetadata(
model_name="resnet50",
description="A dynamic field",
)
),
)
dataset = fo.Dataset()
dataset.add_sample(sample)
dataset.name = "test"
dataset.persistent = True
```
As long as `foo.bar` is on your `module_path`, this dataset can be loaded in
future sessions and manipulated as usual:
```python
import fiftyone as fo
dataset = fo.load_dataset("test")
print(dataset.first())
```
```text
,
'weather': ,
}>,
}>
```
## Image datasets
Any [`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample) whose `filepath` is a file with MIME type `image/*` is recognized
as a image sample, and datasets composed of image samples have media type
`image`:
```python
import fiftyone as fo
sample = fo.Sample(filepath="/path/to/image.png")
dataset = fo.Dataset()
dataset.add_sample(sample)
print(dataset.media_type) # image
print(sample)
```
```text
```
### Example image dataset
To get started exploring image datasets, try loading the
[quickstart dataset](../dataset_zoo/datasets/quickstart.md#dataset-zoo-quickstart) from the zoo:
```python
import fiftyone as fo
import fiftyone.zoo as foz
dataset = foz.load_zoo_dataset("quickstart")
print(dataset.count("ground_truth.detections")) # 1232
print(dataset.count("predictions.detections")) # 5620
print(dataset.count_values("ground_truth.detections.label"))
# {'dog': 15, 'airplane': 24, 'dining table': 15, 'hot dog': 5, ...}
session = fo.launch_app(dataset)
```
## Video datasets
Available in:
Open SourceEnterprise
Any [`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample) whose `filepath` is a file with MIME type `video/*` is recognized
as a video sample, and datasets composed of video samples have media type
`video`:
```python
import fiftyone as fo
sample = fo.Sample(filepath="/path/to/video.mp4")
dataset = fo.Dataset()
dataset.add_sample(sample)
print(dataset.media_type) # video
print(sample)
```
```text
,
}>
```
All video samples have a reserved `frames` attribute in which you can store
frame-level labels and other custom annotations for the video. The `frames`
attribute is a dictionary whose keys are frame numbers and whose values are
[`Frame`](../api/fiftyone.core.frame.md#fiftyone.core.frame.Frame) instances that hold all of the [`Label`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Label) instances and other
primitive-type fields for the frame.
#### NOTE
FiftyOne uses 1-based indexing for video frame numbers.
You can add, modify, and delete [labels of any type](#using-labels) as
well as primitive fields such as integers, strings, and booleans using the same
dynamic attribute syntax that you use to
[interact with samples](#adding-sample-fields):
```python
frame = fo.Frame(
quality=97.12,
weather=fo.Classification(label="sunny"),
objects=fo.Detections(
detections=[
fo.Detection(label="cat", bounding_box=[0.1, 0.1, 0.2, 0.2]),
fo.Detection(label="dog", bounding_box=[0.7, 0.7, 0.2, 0.2]),
]
)
)
# Add labels to the first frame of the video
sample.frames[1] = frame
sample.save()
```
#### NOTE
You must call [`sample.save()`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample.save) in
order to persist changes to the database when editing video samples and/or
their frames that are in datasets.
```text
, <-- `frames` now contains 1 frame of labels
}>
```
#### NOTE
The `frames` attribute of video samples behaves like a defaultdict; a new
[`Frame`](../api/fiftyone.core.frame.md#fiftyone.core.frame.Frame) will be created if the frame number does not exist when you access
it.
You can iterate over the frames in a video sample using the expected syntax:
```python
for frame_number, frame in sample.frames.items():
print(frame)
```
```text
,
'objects': ,
,
],
}>,
}>
```
Notice that the dataset’s summary indicates that the dataset has media type
`video` and includes the schema of any frame fields you add:
```python
print(dataset)
```
```text
Name: 2021.05.03.18.30.20
Media type: video
Num samples: 1
Persistent: False
Tags: []
Sample fields:
id: fiftyone.core.fields.ObjectIdField
filepath: fiftyone.core.fields.StringField
tags: fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)
metadata: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.VideoMetadata)
created_at: fiftyone.core.fields.DateTimeField
last_modified_at: fiftyone.core.fields.DateTimeField
Frame fields:
id: fiftyone.core.fields.ObjectIdField
frame_number: fiftyone.core.fields.FrameNumberField
created_at: fiftyone.core.fields.DateTimeField
last_modified_at: fiftyone.core.fields.DateTimeField
quality: fiftyone.core.fields.FloatField
weather: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Classification)
objects: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)
```
You can retrieve detailed information about the schema of the frames of a
video [`Dataset`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset) using
[`dataset.get_frame_field_schema()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.get_frame_field_schema).
The samples in video datasets can be accessed
[like usual](#accessing-samples-in-a-dataset), and the sample’s frame
labels can be modified by updating the `frames` attribute of a [`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample):
```python
sample = dataset.first()
for frame_number, frame in sample.frames.items():
frame["frame_str"] = str(frame_number)
del frame["weather"]
del frame["objects"]
sample.save()
print(sample.frames[1])
```
```text
```
#### NOTE
You must call [`sample.save()`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample.save) in
order to persist changes to the database when editing video samples and/or
their frames that are in datasets.
[See this page](import_datasets.md#loading-custom-datasets) for more information about
building labeled video samples.
### Example video dataset
To get started exploring video datasets, try loading the
[quickstart-video dataset](../dataset_zoo/datasets/quickstart_video.md#dataset-zoo-quickstart-video) from the zoo:
```python
import fiftyone as fo
import fiftyone.zoo as foz
dataset = foz.load_zoo_dataset("quickstart-video")
print(dataset.count("frames")) # 1279
print(dataset.count("frames.detections.detections")) # 11345
print(dataset.count_values("frames.detections.detections.label"))
# {'vehicle': 7511, 'road sign': 2726, 'person': 1108}
session = fo.launch_app(dataset)
```
### Linking labels across frames
Available in:
Open SourceEnterprise
When working with video datasets, you may want to represent the fact that
multiple frame-level labels correspond to the same logical object moving
through the video.
You can achieve this linking by assigning the same [`Instance`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Instance) to the
`instance` attribute of the relevant [`Detection`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Detection), [`Keypoint`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Keypoint), or [`Polyline`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Polyline)
objects across the frames of a [`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample):
```python
import fiftyone as fo
sample = fo.Sample(filepath="/path/to/video.mp4")
# Create instance representing a logical object
person_instance = fo.Instance()
# Add labels for the person in frame 1
sample.frames[1]["objects"] = fo.Detections(
detections=[
fo.Detection(
label="person",
bounding_box=[0.1, 0.1, 0.2, 0.2],
instance=person_instance, # link this detection
)
]
)
# Add labels for the same person in frame 2
sample.frames[2]["objects"] = fo.Detections(
detections=[
fo.Detection(
label="person",
bounding_box=[0.12, 0.11, 0.2, 0.2],
instance=person_instance, # link this detection
)
]
)
```
#### NOTE
Linking labels in this way enables helpful interactions in the FiftyOne
App. See [this section](app.md#app-linking-labels) for more details.
## 3D datasets
Available in:
Open SourceEnterprise
3D datasets have `media_type="3d"` and can be created from supported 3D asset
files directly, or from `.fo3d` scene files.
Direct assets are the simplest choice when a sample is a single
[mesh](#d-meshes), [point cloud](#d-point-clouds), or
[Gaussian splat reconstruction](#d-gaussian-splats):
```python
import fiftyone as fo
samples = [
fo.Sample(filepath="/path/to/model.glb", media_type="3d"),
fo.Sample(filepath="/path/to/point-cloud.pcd", media_type="3d"),
fo.Sample(filepath="/path/to/mesh.ply", media_type="3d"),
fo.Sample(filepath="/path/to/reconstruction.spz", media_type="3d"),
]
dataset = fo.Dataset()
dataset.add_samples(samples)
print(dataset.media_type) # 3d
```
Features such as
[camera intrinsics and extrinsics](#camera-intrinsics-extrinsics), camera
frustum rendering, and [3D annotation](annotation.md#creating-3d-polylines) are
available whether your sample points directly to a
supported 3D asset or to an `.fo3d` scene.
Wrap assets in `.fo3d` when you need advanced scene customization such as
lights, camera configuration, transformations, materials, shapes, or multiple
assets in one scene. An FO3D file encapsulates a 3D scene constructed using the
[`Scene`](../api/fiftyone.core.threed.md#fiftyone.core.threed.Scene) class, which provides methods
to add, remove, and manipulate 3D objects in the scene. A scene is
internally represented as a n-ary tree of 3D objects, where each
object is a node in the tree. A 3D object is either a
[3D mesh](#d-meshes), [point cloud](#d-point-clouds),
[Gaussian splat](#d-gaussian-splats), or
[3D shape geometry](#d-shapes).
A scene may be explicitly initialized with additional attributes, such as
[`camera`](../api/fiftyone.core.threed.camera.md#module-fiftyone.core.threed.camera),
[`lights`](../api/fiftyone.core.threed.lights.md#module-fiftyone.core.threed.lights), and
[`background`](../api/fiftyone.core.threed.md#fiftyone.core.threed.SceneBackground). By default, a
scene is created with neutral lighting, and a perspective camera whose
`up` is set to `Y` axis in a right-handed coordinate system.
After a scene is constructed, it should be written to the disk using the
[`scene.write()`](../api/fiftyone.core.threed.md#fiftyone.core.threed.Scene.write) method, which
serializes the scene into an FO3D file.
```python
import fiftyone as fo
scene = fo.Scene()
scene.camera = fo.PerspectiveCamera(up="Z")
mesh = fo.GltfMesh("mesh", "mesh.glb")
mesh.rotation = fo.Euler(90, 0, 0, degrees=True)
sphere1 = fo.SphereGeometry("sphere1", radius=2.0)
sphere1.position = [-1, 0, 0]
sphere1.default_material.color = "red"
sphere2 = fo.SphereGeometry("sphere2", radius=2.0)
sphere2.position = [1, 0, 0]
sphere2.default_material.color = "blue"
scene.add(mesh, sphere1, sphere2)
scene.write("/path/to/scene.fo3d")
sample = fo.Sample(filepath="/path/to/scene.fo3d")
dataset = fo.Dataset()
dataset.add_sample(sample)
print(dataset.media_type) # 3d
```
To modify an existing scene, load it via
[`Scene.from_fo3d()`](../api/fiftyone.core.threed.md#fiftyone.core.threed.Scene.from_fo3d), perform any
necessary updates, and then re-write it to disk:
```python
import fiftyone as fo
scene = fo.Scene.from_fo3d("/path/to/scene.fo3d")
for node in scene.traverse():
if isinstance(node, fo.SphereGeometry):
node.visible = False
scene.write("/path/to/scene.fo3d")
```
### 3D meshes
A 3D mesh is a collection of vertices, edges, and faces that define the shape
of a 3D object. Whereas some mesh formats store only the geometry of the mesh,
others also store the material properties and textures of the mesh. If a
mesh file contains material properties and textures, FiftyOne will
automatically load and display them. You may also
assign default material for your meshes by setting the
`default_material`
attribute of the mesh. In the absence of any material information,
meshes are assigned a
[`MeshStandardMaterial`](../api/fiftyone.core.threed.md#fiftyone.core.threed.MeshStandardMaterial)
with reasonable defaults that can also be dynamically configured from the app.
Please refer to [`material_3d`](../api/fiftyone.core.threed.material_3d.md#module-fiftyone.core.threed.material_3d) for more
details.
FiftyOne currently supports
[`GLTF`](../api/fiftyone.core.threed.md#fiftyone.core.threed.GltfMesh),
[`OBJ`](../api/fiftyone.core.threed.md#fiftyone.core.threed.ObjMesh),
[`PLY`](../api/fiftyone.core.threed.md#fiftyone.core.threed.PlyMesh),
[`STL`](../api/fiftyone.core.threed.md#fiftyone.core.threed.StlMesh), and
[`FBX 7.x+`](../api/fiftyone.core.threed.md#fiftyone.core.threed.FbxMesh) mesh formats.
#### NOTE
We recommend the [`GLTF`](../api/fiftyone.core.threed.md#fiftyone.core.threed.GltfMesh) format for
3D meshes where possible, as it is the most compact, efficient, and
web-friendly format for storing and transmitting 3D models.
```python
import fiftyone as fo
scene = fo.Scene()
mesh1 = fo.GltfMesh("mesh1", "mesh.glb")
mesh1.rotation = fo.Euler(90, 0, 0, degrees=True)
mesh2 = fo.ObjMesh("mesh2", "mesh.obj")
mesh3 = fo.PlyMesh("mesh3", "mesh.ply")
mesh4 = fo.StlMesh("mesh4", "mesh.stl")
mesh5 = fo.FbxMesh("mesh5", "mesh.fbx")
scene.add(mesh1, mesh2, mesh3, mesh4, mesh5)
scene.write("/path/to/scene.fo3d")
```
### 3D point clouds
FiftyOne supports the
[PCD](https://pointclouds.org/documentation/tutorials/pcd_file_format.html)
point cloud format. A code snippet to create a PCD object that can be added
to a FiftyOne 3D scene is shown below:
```python
import fiftyone as fo
pcd = fo.PointCloud("my-pcd", "point-cloud.pcd")
pcd.default_material.shading_mode = "custom"
pcd.default_material.custom_color = "red"
pcd.default_material.point_size = 2
scene = fo.Scene()
scene.add(pcd)
scene.write("/path/to/scene.fo3d")
```
You can customize the appearance of a point cloud by setting the
`default_material` attribute of the point cloud object, or dynamically from
the app. Please refer to the
[`PointCloudMaterial`](../api/fiftyone.core.threed.md#fiftyone.core.threed.PointCloudMaterial)
class for more details.
#### NOTE
If your scene contains multiple point clouds, you can control which point
cloud is included in
[orthographic projections](#orthographic-projection-images) by
initializing it with `flag_for_projection=True`.
Here’s how a typical PCD file is structured:
```python
import numpy as np
from pypcd4 import PointCloud
# XYZ coordinates
points = np.array([(x1, y1, z1), (x2, y2, z2), ...], dtype=np.float32)
# RGB values in [0, 255]
colors = np.array([(r1, g1, b1), (r2, g2, b2), ...], dtype=np.uint8)
rgb = PointCloud.encode_rgb(colors)
pcd = PointCloud.from_xyzrgb_points(np.column_stack((points, rgb)))
pcd.save("/path/to/point-cloud.pcd")
```
#### NOTE
When working with modalities such as LIDAR, intensity data is assumed to be
encoded in the `r` channel of the `rgb` field of the
[PCD files](https://pointclouds.org/documentation/tutorials/pcd_file_format.html).
When coloring by intensity [in the App](app.md#app-3d-visualizer), the
intensity values are automatically scaled to use the full dynamic range of
the colorscale.
### 3D Gaussian splats
FiftyOne supports Gaussian splat reconstructions in PLY, SPZ, SPLAT, KSPLAT,
SOG, and RAD formats. Splat files can be used directly as 3D samples or added
to FO3D scenes via
[`GaussianSplat`](../api/fiftyone.core.threed.md#fiftyone.core.threed.GaussianSplat):
```python
import fiftyone as fo
# Use a splat file directly as a 3D sample
sample = fo.Sample(
filepath="/path/to/reconstruction.spz",
media_type="3d",
)
# Or add one or more splats to an FO3D scene
splat = fo.GaussianSplat("reconstruction", "reconstruction.spz")
scene = fo.Scene()
scene.add(splat)
scene.write("/path/to/scene.fo3d")
```
You can tune each splat’s appearance and rendering from its settings panel in
the App’s [3D visualizer](app.md#app-3d-visualizer).
### 3D shapes
FiftyOne provides a set of primitive 3D shape geometries that can be added to
a 3D scene. The following 3D shape geometries are supported:
- Box: [`BoxGeometry`](../api/fiftyone.core.threed.md#fiftyone.core.threed.BoxGeometry)
- Sphere: [`SphereGeometry`](../api/fiftyone.core.threed.md#fiftyone.core.threed.SphereGeometry)
- Cylinder: [`CylinderGeometry`](../api/fiftyone.core.threed.md#fiftyone.core.threed.CylinderGeometry)
- Plane: [`PlaneGeometry`](../api/fiftyone.core.threed.md#fiftyone.core.threed.PlaneGeometry)
Similar to meshes and point clouds, shapes can be manipulated by setting their
position, rotation, and scale. Their appearance can be customized either by
setting the `default_material` attribute of the shape object, or dynamically
from the app.
```python
import fiftyone as fo
scene = fo.Scene()
box = fo.BoxGeometry("box", width=0.5, height=0.5, depth=0.5)
box.position = [0, 0, 1]
box.default_material.color = "red"
sphere = fo.SphereGeometry("sphere", radius=2.0)
sphere.position = [-1, 0, 0]
sphere.default_material.color = "blue"
cylinder = fo.CylinderGeometry("cylinder", radius_top=0.5, height=1)
cylinder.position = [0, 1, 0]
plane = fo.PlaneGeometry("plane", width=2, height=2)
plane.rotation = fo.Euler(90, 0, 0, degrees=True)
scene.add(box, sphere, cylinder, plane)
scene.write("/path/to/scene.fo3d")
```
### 3D annotations
3D samples may contain any type and number of custom fields, including
[3D detections](#d-detections) and [3D polylines](#d-polylines),
which are natively visualizable by the App’s
[3D visualizer](app.md#app-3d-visualizer).
Because 3D annotations are stored in dedicated fields of datasets rather than
being embedded in FO3D files, they can be queried and filtered via
[dataset views](using_views.md#view-filtering) and [in the App](app.md#app-filtering)
just like other primitive/label fields.
```python
import fiftyone as fo
scene = fo.Scene()
scene.add(fo.GltfMesh("mesh", "mesh.gltf"))
scene.write("/path/to/scene.fo3d")
detection = fo.Detection(
label="vehicle",
location=[0.47, 1.49, 69.44],
dimensions=[2.85, 2.63, 12.34],
rotation=[0, -1.56, 0],
)
sample = fo.Sample(
filepath="/path/to/scene.fo3d",
ground_truth=fo.Detections(detections=[detection]),
)
```
### Orthographic projection images
Available in:
Open SourceEnterprise
In order to visualize 3D datasets in the App’s grid view, you can use
[`compute_orthographic_projection_images()`](../api/fiftyone.utils.utils3d.md#fiftyone.utils.utils3d.compute_orthographic_projection_images)
to generate orthographic projection images of each scene:
```python
import fiftyone as fo
import fiftyone.utils.utils3d as fou3d
import fiftyone.zoo as foz
# Load an example 3D dataset
dataset = foz.load_zoo_dataset("quickstart-3d")
# This dataset already has orthographic projections populated, but let's
# recompute them to demonstrate the idea
fou3d.compute_orthographic_projection_images(
dataset,
(-1, 512), # (width, height) of each image; -1 means aspect-preserving
bounds=((-50, -50, -50), (50, 50, 50)),
projection_normal=(0, -1, 0),
output_dir="/tmp/quickstart-3d-proj",
shading_mode="height",
)
session = fo.launch_app(dataset)
```
Note that the method also supports [grouped datasets](groups.md#groups) that
contain 3D slice(s):
```python
import fiftyone as fo
import fiftyone.utils.utils3d as fou3d
import fiftyone.zoo as foz
# Load an example group dataset that contains a 3D slice
dataset = foz.load_zoo_dataset("quickstart-groups")
# Populate orthographic projections
fou3d.compute_orthographic_projection_images(dataset, (-1, 512), "/tmp/proj")
dataset.group_slice = "pcd"
session = fo.launch_app(dataset)
```
#### NOTE
Orthographic projection images currently only include point clouds, not
meshes or 3D shapes.
If a scene contains multiple [point clouds](#d-point-clouds), you can
control which point cloud to project by initializing it with
`flag_for_projection=True`.
The above method populates an [`OrthographicProjectionMetadata`](../api/fiftyone.utils.utils3d.md#fiftyone.utils.utils3d.OrthographicProjectionMetadata) field on each
sample that contains the path to its projection image and other necessary
information to properly
[visualize it in the App](app.md#app-3d-orthographic-projections).
Refer to the
[`compute_orthographic_projection_images()`](../api/fiftyone.utils.utils3d.md#fiftyone.utils.utils3d.compute_orthographic_projection_images)
documentation for available parameters to customize the projections.
#### NOTE
Did you know? You can use the [inbuilt projection](#projecting-3d-detections)
capabilities in FiftyOne to visualize 3D bounding boxes of 3D slices in grouped
datasets on 2D images.
### Example 3D datasets
To get started exploring 3D datasets, try loading the
[quickstart-3d dataset](../dataset_zoo/datasets/quickstart_3d.md#dataset-zoo-quickstart-3d) from the zoo:
```python
import fiftyone as fo
import fiftyone.zoo as foz
dataset = foz.load_zoo_dataset("quickstart-3d")
print(dataset.count_values("ground_truth.label"))
# {'bottle': 5, 'stairs': 5, 'keyboard': 5, 'car': 5, ...}
session = fo.launch_app(dataset)
```
Also check out the
[quickstart-groups dataset](../dataset_zoo/datasets/quickstart_groups.md#dataset-zoo-quickstart-groups), which
contains a point cloud slice:
```python
import fiftyone as fo
import fiftyone.utils.utils3d as fou3d
import fiftyone.zoo as foz
dataset = foz.load_zoo_dataset("quickstart-groups")
# Populate orthographic projections
fou3d.compute_orthographic_projection_images(dataset, (-1, 512), "/tmp/proj")
print(dataset.count("ground_truth.detections")) # 1100
print(dataset.count_values("ground_truth.detections.label"))
# {'Pedestrian': 133, 'Car': 774, ...}
dataset.group_slice = "pcd"
session = fo.launch_app(dataset)
```
## Point cloud datasets
#### WARNING
The `point-cloud` media type has been deprecated in favor of the
[3D media type](#d-datasets).
While we’ll keep supporting the `point-cloud` media type for backward
compatibility, we recommend using the `3d` media type for new datasets.
To use a PCD file as a 3D sample in a new dataset, create it with
`media_type="3d"` explicitly.
Any [`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample) whose `filepath` is a
[PCD file](https://pointclouds.org/documentation/tutorials/pcd_file_format.html)
with extension `.pcd` is recognized as a point cloud sample, and datasets
composed of point cloud samples have media type `point-cloud`:
```python
import fiftyone as fo
sample = fo.Sample(filepath="/path/to/point-cloud.pcd")
dataset = fo.Dataset()
dataset.add_sample(sample)
print(dataset.media_type) # point-cloud
print(sample)
```
```text
```
Point cloud samples may contain any type and number of custom fields, including
[3D detections](#d-detections) and [3D polylines](#d-polylines),
which are natively visualizable by the App’s
[3D visualizer](app.md#app-3d-visualizer).
## Camera intrinsics and extrinsics
Available in:
Open SourceEnterprise
FiftyOne provides support for storing and working with camera intrinsic and
extrinsic parameters, which are essential for 3D computer vision tasks such as
multi-sensor fusion, depth estimation, and 3D reconstruction.
#### NOTE
See [storing camera calibration](#storing-camera-calibration) for
information about storing intrinsics and extrinsics at the dataset level.
### Camera intrinsics
Camera intrinsics describe the internal parameters of a camera, including focal
length, principal point, and lens distortion. FiftyOne provides several
[`CameraIntrinsics`](../api/fiftyone.core.camera.md#fiftyone.core.camera.CameraIntrinsics) subclasses for different camera models:
- [`PinholeCameraIntrinsics`](../api/fiftyone.core.camera.md#fiftyone.core.camera.PinholeCameraIntrinsics): Simple pinhole camera model with no distortion
- [`OpenCVCameraIntrinsics`](../api/fiftyone.core.camera.md#fiftyone.core.camera.OpenCVCameraIntrinsics): Brown-Conrady distortion model with radial and
tangential coefficients (up to 8 parameters)
- [`OpenCVFisheyeCameraIntrinsics`](../api/fiftyone.core.camera.md#fiftyone.core.camera.OpenCVFisheyeCameraIntrinsics): Fisheye equidistant projection model
```python
import fiftyone as fo
# Create OpenCV camera intrinsics with distortion
intrinsics = fo.OpenCVCameraIntrinsics(
fx=1000.0, # focal length x (pixels)
fy=1000.0, # focal length y (pixels)
cx=960.0, # principal point x
cy=540.0, # principal point y
k1=-0.1, # radial distortion coefficient
k2=0.05, # radial distortion coefficient
p1=0.001, # tangential distortion coefficient
p2=-0.001, # tangential distortion coefficient
)
# Access the 3x3 intrinsic matrix
K = intrinsics.intrinsic_matrix
# Create intrinsics from a matrix
import numpy as np
K = np.array([
[1200.0, 0.0, 640.0],
[0.0, 1200.0, 480.0],
[0.0, 0.0, 1.0]
])
intrinsics = fo.PinholeCameraIntrinsics.from_matrix(K)
```
### Sensor extrinsics
The [`StaticTransform`](../api/fiftyone.core.camera.md#fiftyone.core.camera.StaticTransform) class represents a 6-DOF rigid transformation between
coordinate frames, defined by a translation vector and a rotation quaternion.
```python
import fiftyone as fo
# Create a camera-to-ego transformation
extrinsics = fo.StaticTransform(
translation=[1.5, 0.0, 1.2], # [tx, ty, tz]
quaternion=[0.0, 0.0, 0.0, 1.0], # [qx, qy, qz, qw] (scalar-last)
source_frame="camera_front",
target_frame="ego",
)
# Access the 4x4 transformation matrix
T = extrinsics.transform_matrix
# Get the 3x3 rotation matrix
R = extrinsics.rotation_matrix
# Compute the inverse transformation (ego -> camera_front)
inv_extrinsics = extrinsics.inverse()
# Compose transformations (A->B composed with B->C = A->C)
camera_to_world = camera_to_ego.compose(ego_to_world)
```
You can also create extrinsics from a transformation matrix:
```python
import numpy as np
import fiftyone as fo
# 4x4 homogeneous transformation matrix
T = np.eye(4)
T[:3, 3] = [1.0, 2.0, 3.0] # translation
extrinsics = fo.StaticTransform.from_matrix(
T,
source_frame="camera",
target_frame="world",
)
```
#### NOTE
For low-level transformation utilities such as quaternion math, coordinate
system conversions (OpenCV, OpenGL, ROS, etc.), and matrix operations, see
the [`fiftyone.utils.transforms`](../api/fiftyone.utils.transforms.md#module-fiftyone.utils.transforms) module.
### Resolving calibration
When working with samples, you can resolve camera calibration parameters using
the [`resolve_intrinsics()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.resolve_intrinsics)
and [`resolve_transformation()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.resolve_transformation)
methods. These methods implement a resolution chain that checks multiple
sources in order of precedence.
For intrinsics, the resolution order is:
1. Sample-level [`CameraIntrinsics`](../api/fiftyone.core.camera.md#fiftyone.core.camera.CameraIntrinsics) field (inline value)
2. Sample-level [`CameraIntrinsicsRef`](../api/fiftyone.core.camera.md#fiftyone.core.camera.CameraIntrinsicsRef) field (reference to dataset-level)
3. Group slice name lookup in `dataset.camera_intrinsics` (for grouped
datasets)
For extrinsics, use the `source_frame` and `target_frame` parameters to
specify the desired transformation. The `chain_via` parameter enables
composing transformations through intermediate frames (e.g., camera → ego →
world).
Resolution precedence for extrinsics is:
1. Direct `source_frame -> target_frame` match
2. If provided, explicit `chain_via` path only
3. If `chain_via` is omitted and `target_frame="world"`, automatic
chaining through up to two intermediate frames
Automatic chaining only applies when a unique forward path exists. If no valid
path exists, or more than one valid path exists, no automatic chain is chosen
and resolution returns `None`. When deterministic path selection matters,
provide `chain_via` explicitly.
#### NOTE
For [grouped datasets](groups.md#groups), the group slice name plays an
important role in resolution:
- **Intrinsics**: When resolving intrinsics, if no sample-level value is
found, the current `dataset.group_slice` name is used to look up the
intrinsics in `dataset.camera_intrinsics`
- **Extrinsics**: When `source_frame` is not specified, it is
automatically inferred from the group slice name
This means you can store calibration keyed by slice name (e.g., `"left"`,
`"right"`) and resolution will automatically find the correct parameters
for the current slice.
The following example demonstrates a typical autonomous vehicle workflow with
static sensor calibration stored at the dataset level and dynamic ego pose
stored at the sample level:
```python
import fiftyone as fo
import fiftyone.zoo as foz
dataset = foz.load_zoo_dataset("quickstart-groups")
# Store static intrinsics (1080p parameters)
dataset.camera_intrinsics = {
"left": fo.PinholeCameraIntrinsics(fx=1100, fy=1100, cx=960, cy=540),
"right": fo.PinholeCameraIntrinsics(fx=1100, fy=1100, cx=960, cy=540),
}
# Store static sensor mounting positions relative to vehicle center ("ego")
dataset.static_transforms = {
"left::ego": fo.StaticTransform(
translation=[1.5, 0.5, 1.2],
quaternion=[0, 0, 0, 1],
source_frame="left",
target_frame="ego",
),
"right::ego": fo.StaticTransform(
translation=[1.5, -0.5, 1.2],
quaternion=[0, 0, 0, 1],
source_frame="right",
target_frame="ego",
),
}
dataset.save()
# Focus on the left camera
dataset.group_slice = "left"
sample = dataset.first()
# Store dynamic ego pose at sample level (vehicle location in the world)
sample["ego_pose"] = fo.StaticTransform(
translation=[100.0, 50.0, 0.0],
quaternion=[0, 0, 0, 1],
source_frame="ego",
target_frame="world",
)
sample.save()
# Resolve intrinsics (infers "left" from dataset.group_slice)
intrinsics = dataset.resolve_intrinsics(sample)
# Resolve full transform chain: left camera -> ego -> world
extrinsics = dataset.resolve_transformation(
sample,
source_frame="left",
target_frame="world",
chain_via=["ego"],
)
```
### 3D projection
The [`CameraProjector`](../api/fiftyone.core.camera.md#fiftyone.core.camera.CameraProjector) class provides utilities for projecting points between 3D
world coordinates and 2D image coordinates.
```python
import fiftyone as fo
# Using intrinsics and extrinsics resolved from the previous example
projector = fo.CameraProjector(intrinsics, extrinsics)
# Project a world point onto the image
# A stop sign at world coordinates (110, 52, 2):
# 10m in front of car, 2m left, 2m high
world_point = [[110.0, 52.0, 2.0]]
pixel = projector.project(world_point)
print(pixel) # [[x, y]] image coordinates
```
You can also unproject 2D points back to 3D if you have depth information:
```python
import numpy as np
import fiftyone as fo
# Unproject pixel coordinates to 3D using known depth
pixels = np.array([[960.0, 540.0]]) # image center
depths = np.array([10.0]) # 10 meters away
points_3d = projector.unproject(pixels, depths)
```
## Generic datasets
Any [`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample) whose `filepath` does not infer a known media type will be
assigned a media type of `unknown`. Adding these samples to a [`Dataset`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset) will
result in a generic dataset with a media type of `unknown`.
```python
import fiftyone as fo
sample = fo.Sample(filepath="/path/to/file.json")
dataset = fo.Dataset()
dataset.add_sample(sample)
print(dataset.media_type) # unknown
print(sample)
```
```text
```
## Custom datasets
When a [`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample) is created, a custom value can be provided as the `media_type`
keyword argument. Adding the sample to a [`Dataset`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset) will result in a dataset
with `media_type` inherited from the sample. Custom media types can be used
to extend functionality for sample types that are not natively supported.
For App support in the grid or the modal, pair custom media types with
[custom sample renderers](../plugins/developing_plugins.md#custom-sample-renderers). This plugin feature
lets you provide domain-specific rendering for custom media types whose samples
may not be handled by FiftyOne’s built-in media renderers.
```python
import fiftyone as fo
sample = fo.Sample(filepath="/path/to/file.aac", media_type="audio")
dataset = fo.Dataset()
dataset.add_sample(sample)
print(dataset.media_type) # audio
print(sample)
```
```text
```
## DatasetViews
Previous sections have demonstrated how to add and interact with [`Dataset`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset)
components like samples, fields, and labels. The true power of FiftyOne lies in
the ability to search, sort, filter, and explore the contents of a [`Dataset`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset).
Behind this power is the [`DatasetView`](../api/fiftyone.core.view.md#fiftyone.core.view.DatasetView). Whenever an operation
like [`match()`](../api/fiftyone.core.view.md#fiftyone.core.view.DatasetView.match) or
[`sort_by()`](../api/fiftyone.core.view.md#fiftyone.core.view.DatasetView.sort_by) is applied to a
dataset, a [`DatasetView`](../api/fiftyone.core.view.md#fiftyone.core.view.DatasetView) is returned. As the name implies, a [`DatasetView`](../api/fiftyone.core.view.md#fiftyone.core.view.DatasetView)
is a *view* into the data in your [`Dataset`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset) that was produced by a series of
operations that manipulated your data in different ways.
A [`DatasetView`](../api/fiftyone.core.view.md#fiftyone.core.view.DatasetView) is composed of [`SampleView`](../api/fiftyone.core.sample.md#fiftyone.core.sample.SampleView) objects for a subset of the samples
in your dataset. For example, a view may contain only samples with a given tag,
or samples whose labels meet a certain criteria.
In turn, each [`SampleView`](../api/fiftyone.core.sample.md#fiftyone.core.sample.SampleView) represents a view into the content of the underlying
[`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample) in the dataset. For example, a [`SampleView`](../api/fiftyone.core.sample.md#fiftyone.core.sample.SampleView) may represent the contents
of a sample with [`Detections`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Detections) below a specified threshold filtered out.
```python
import fiftyone as fo
import fiftyone.zoo as foz
from fiftyone import ViewField as F
dataset = foz.load_zoo_dataset("quickstart")
dataset.compute_metadata()
# Create a view containing the 5 samples from the validation split whose
# images are >= 48 KB that have the most predictions with confidence > 0.9
complex_view = (
dataset
.match_tags("validation")
.match(F("metadata.size_bytes") >= 48 * 1024) # >= 48 KB
.filter_labels("predictions", F("confidence") > 0.9)
.sort_by(F("predictions.detections").length(), reverse=True)
.limit(5)
)
# Check to see how many predictions there are in each matching sample
print(complex_view.values(F("predictions.detections").length()))
# [29, 20, 17, 15, 15]
```
## Merging datasets
The [`Dataset`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset) class provides a powerful
[`merge_samples()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.merge_samples) method
that you can use to merge the contents of another [`Dataset`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset) or [`DatasetView`](../api/fiftyone.core.view.md#fiftyone.core.view.DatasetView)
into an existing dataset.
By default, samples with the same absolute `filepath` are merged, and top-level
fields from the provided samples are merged in, overwriting any existing values
for those fields, with the exception of list fields (e.g.,
[tags](#using-tags)) and label list fields (e.g.,
[Detections](#object-detection)), in which case the elements of the lists
themselves are merged. In the case of label list fields, labels with the same
`id` in both collections are updated rather than duplicated.
The [`merge_samples()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.merge_samples)
method can be configured in numerous ways, including:
- Which field to use as a merge key, or an arbitrary function defining the
merge key
- Whether existing samples should be modified or skipped
- Whether new samples should be added or omitted
- Whether new fields can be added to the dataset schema
- Whether list fields should be treated as ordinary fields and merged as a
whole rather than merging their elements
- Whether to merge only specific fields, or all but certain fields
- Mapping input fields to different field names of this dataset
For example, the following snippet demonstrates merging a new field into an
existing dataset:
```python
import fiftyone as fo
import fiftyone.zoo as foz
dataset1 = foz.load_zoo_dataset("quickstart")
# Create a dataset containing only ground truth objects
dataset2 = dataset1.select_fields("ground_truth").clone()
# Create a view containing only the predictions
predictions_view = dataset1.select_fields("predictions")
# Merge the predictions
dataset2.merge_samples(predictions_view)
print(dataset1.count("ground_truth.detections")) # 1232
print(dataset2.count("ground_truth.detections")) # 1232
print(dataset1.count("predictions.detections")) # 5620
print(dataset2.count("predictions.detections")) # 5620
```
Note that the argument to
[`merge_samples()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.merge_samples) can be a
[`DatasetView`](../api/fiftyone.core.view.md#fiftyone.core.view.DatasetView), which means that you can perform possibly-complex
[transformations](using_views.md#using-views) to the source dataset to select the
desired content to merge.
Consider the following variation of the above snippet, which demonstrates a
workflow where [`Detections`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Detections) from another dataset are merged into a dataset with
existing [`Detections`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Detections) in the same field:
```python
from fiftyone import ViewField as F
# Create a new dataset that only contains predictions with confidence >= 0.9
dataset3 = (
dataset1
.select_fields("predictions")
.filter_labels("predictions", F("confidence") > 0.9)
).clone()
# Create a view that contains only the remaining predictions
low_conf_view = dataset1.filter_labels("predictions", F("confidence") < 0.9)
# Merge the low confidence predictions back in
dataset3.merge_samples(low_conf_view, fields="predictions")
print(dataset1.count("predictions.detections")) # 5620
print(dataset3.count("predictions.detections")) # 5620
```
Finally, the example below demonstrates the use of a custom merge key to define
which samples to merge:
```python
import os
# Create a dataset with 100 samples of ground truth labels
dataset4 = dataset1[50:150].select_fields("ground_truth").clone()
# Create a view with 50 overlapping samples of predictions
predictions_view = dataset1[:100].select_fields("predictions")
# Merge predictions into dataset, using base filename as merge key and
# never inserting new samples
dataset4.merge_samples(
predictions_view,
key_fcn=lambda sample: os.path.basename(sample.filepath),
insert_new=False,
)
print(len(dataset4)) # 100
print(len(dataset4.exists("predictions"))) # 50
```
#### NOTE
Did you know? You can use
[`merge_dir()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.merge_dir) to directly
merge the contents of a dataset on disk into an existing FiftyOne
dataset without first [loading it](import_datasets.md#importing-datasets) into a
temporary dataset and then using
[`merge_samples()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.merge_samples) to
perform the merge.
## Cloning datasets
You can use [`clone()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.clone) to create a
copy of a dataset:
```python
import fiftyone as fo
import fiftyone.zoo as foz
dataset = foz.load_zoo_dataset("quickstart")
dataset2 = dataset.clone()
dataset2.add_sample_field("new_field", fo.StringField)
# The source dataset is unaffected
assert "new_field" not in dataset.get_field_schema()
```
Dataset clones contain deep copies of all samples and dataset-level metadata
such as runs, saved views, and workspaces from the source dataset. The source
*media files*, however, are not copied.
#### NOTE
Did you know? You can also
[clone specific subsets](using_views.md#saving-and-cloning-views) of your datasets.
By default, cloned datasets also retain all
[custom indexes](app.md#app-optimizing-query-performance) that you’ve created on
the source collection, but you can control this by passing the optional
`include_indexes` parameter to
[`clone()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.clone):
```python
dataset.create_index("ground_truth.detections.label")
# Do not retain custom indexes on the cloned dataset
dataset2 = dataset.clone(include_indexes=False)
# Only include specific custom indexes
dataset2 = dataset.clone(include_indexes=["ground_truth.detections.label"])
```
## Batch updates
You are always free to perform any necessary modifications to a [`Dataset`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset) by
iterating over it via a Python loop and explicitly
[performing the edits](#editing-sample-fields) that you require.
However, the [`Dataset`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset) class provides a number of methods that allow you to
efficiently perform various common batch actions to your entire dataset.
### Cloning, renaming, clearing, and deleting fields
You can use the
[`clone_sample_field()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.clone_sample_field),
[`rename_sample_field()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.rename_sample_field),
[`clear_sample_field()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.clear_sample_field),
and
[`delete_sample_field()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.delete_sample_field)
methods to efficiently perform common actions on the sample fields of a
[`Dataset`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset):
```python
import fiftyone as fo
import fiftyone.zoo as foz
from fiftyone import ViewField as F
dataset = foz.load_zoo_dataset("quickstart")
# Clone an existing field
dataset.clone_sample_field("predictions", "also_predictions")
print("also_predictions" in dataset.get_field_schema()) # True
# Rename a field
dataset.rename_sample_field("also_predictions", "still_predictions")
print("still_predictions" in dataset.get_field_schema()) # True
# Clear a field (sets all values to None)
dataset.clear_sample_field("still_predictions")
print(dataset.count_values("still_predictions")) # {None: 200}
# Delete a field
dataset.delete_sample_field("still_predictions")
```
You can also use
[dot notation](https://docs.mongodb.com/manual/core/document/#dot-notation)
to manipulate the fields or subfields of embedded documents in your dataset:
```python
sample = dataset.first()
# Clone an existing embedded field
dataset.clone_sample_field(
"predictions.detections.label",
"predictions.detections.also_label",
)
print(sample.predictions.detections[0]["also_label"]) # "bird"
# Rename an embedded field
dataset.rename_sample_field(
"predictions.detections.also_label",
"predictions.detections.still_label",
)
print(sample.predictions.detections[0]["still_label"]) # "bird"
# Clear an embedded field (sets all values to None)
dataset.clear_sample_field("predictions.detections.still_label")
print(sample.predictions.detections[0]["still_label"]) # None
# Delete an embedded field
dataset.delete_sample_field("predictions.detections.still_label")
```
### Save contexts
Available in:
Open SourceEnterprise
You are always free to perform arbitrary edits to a [`Dataset`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset) by iterating over
its contents and editing the samples directly:
```python
import random
import fiftyone as fo
import fiftyone.zoo as foz
from fiftyone import ViewField as F
dataset = foz.load_zoo_dataset("quickstart")
# Populate a new field on each sample in the dataset
for sample in dataset:
sample["random"] = random.random()
sample.save()
print(dataset.count("random")) # 200
print(dataset.bounds("random")) # (0.0007, 0.9987)
```
However, the above pattern can be inefficient for large datasets because each
[`sample.save()`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample.save) call makes a new
connection to the database.
The [`iter_samples()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.iter_samples) method
provides an `autosave=True` option that causes all changes to samples
emitted by the iterator to be automatically saved using an efficient batch
update strategy:
```python
# Automatically saves sample edits in efficient batches
for sample in dataset.select_fields().iter_samples(autosave=True):
sample["random"] = random.random()
```
#### NOTE
As the above snippet shows, you should also optimize your iteration by
[selecting only](using_views.md#efficient-iteration-views) the required fields.
You can configure the default batching strategy that is used via your
[FiftyOne config](config.md#configuring-fiftyone), or you can configure the
batching strategy on a per-method call basis by passing the optional
`batch_size` and `batching_strategy` arguments to
[`iter_samples()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.iter_samples).
You can also use the
[`save_context()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.save_context)
method to perform batched edits using the pattern below:
```python
# Use a context to save sample edits in efficient batches
with dataset.save_context() as context:
for sample in dataset.select_fields():
sample["random"] = random.random()
context.save(sample)
```
The benefit of the above approach versus passing `autosave=True` to
[`iter_samples()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.iter_samples) is that
[`context.save()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SaveContext.save) allows you
to be explicit about which samples you are editing, which avoids unnecessary
computations if your loop only edits certain samples.
### Updating samples
The
[`update_samples()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.update_samples)
method provides an efficient interface for applying a function to each sample
in a collection and saving the sample edits:
```python
import fiftyone as fo
import fiftyone.zoo as foz
dataset = foz.load_zoo_dataset("cifar10", split="train")
view = dataset.select_fields("ground_truth")
def update_fcn(sample):
sample.ground_truth.label = sample.ground_truth.label.upper()
view.update_samples(update_fcn)
print(dataset.count_values("ground_truth.label"))
# {'DEER': 5000, 'HORSE': 5000, 'AIRPLANE': 5000, ..., 'DOG': 5000}
```
#### NOTE
As the above snippet shows, you should optimize your iteration by
[selecting only](using_views.md#efficient-iteration-views) the required fields.
By default,
[`update_samples()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.update_samples)
leverages a multiprocessing pool to parallelize the work across a number of
workers, resulting in significant performance improvements over the equivalent
[`iter_samples(autosave=True)`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.iter_samples)
syntax:
```python
for sample in view.iter_samples(autosave=True, progress=True):
update_fcn(sample)
```
Keep the following points in mind while using
[`update_samples()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.update_samples):
- The samples are not processed in any particular order
- Your `update_fcn` should not modify global state or variables defined
outside of the function
You can configure the number of workers that
[`update_samples()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.update_samples)
uses in a variety of ways:
- Configure the default number of workers used by all
[`update_samples()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.update_samples)
calls by setting the `default_process_pool_workers` value in your
[FiftyOne config](config.md#configuring-fiftyone)
- Manually configure the number of workers for a particular
[`update_samples()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.update_samples)
call by passing the `num_workers` parameter
- If neither of the above settings are applied,
[`update_samples()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.update_samples)
will use
[`recommend_process_pool_workers()`](../api/fiftyone.core.utils.md#fiftyone.core.utils.recommend_process_pool_workers)
to choose a number of worker processes, unless the method is called in a
daemon process (subprocess), in which case no workers are used
#### NOTE
You can set `default_process_pool_workers<=1` or `num_workers<=1` to
disable the use of multiprocessing pools in
[`update_samples()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.update_samples).
By default,
[`update_samples()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.update_samples)
evenly distributes samples to all workers in a single batch per worker.
However, you can pass the `batch_size` parameter to customize the number of
samples sent to each worker at a time:
```python
view.update_samples(update_fcn, batch_size=50, num_workers=4)
```
You can also pass `progress="workers"` to
[`update_samples()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.update_samples)
to render progress bar(s) for each worker:
```python
view.update_samples(update_fcn, num_workers=16, progress="workers")
```
```text
Batch 01/16: 100%|█████████████████████████████████████████████████| 3125/3125 [899.01it/s]
Batch 02/16: 100%|█████████████████████████████████████████████████| 3125/3125 [894.90it/s]
Batch 03/16: 100%|█████████████████████████████████████████████████| 3125/3125 [900.14it/s]
Batch 04/16: 100%|█████████████████████████████████████████████████| 3125/3125 [895.61it/s]
Batch 05/16: 100%|█████████████████████████████████████████████████| 3125/3125 [903.09it/s]
Batch 06/16: 100%|█████████████████████████████████████████████████| 3125/3125 [895.33it/s]
Batch 07/16: 100%|█████████████████████████████████████████████████| 3125/3125 [893.26it/s]
Batch 08/16: 100%|█████████████████████████████████████████████████| 3125/3125 [889.17it/s]
Batch 09/16: 100%|█████████████████████████████████████████████████| 3125/3125 [888.16it/s]
Batch 10/16: 100%|█████████████████████████████████████████████████| 3125/3125 [893.69it/s]
Batch 11/16: 100%|█████████████████████████████████████████████████| 3125/3125 [896.80it/s]
Batch 12/16: 100%|█████████████████████████████████████████████████| 3125/3125 [903.28it/s]
Batch 13/16: 100%|█████████████████████████████████████████████████| 3125/3125 [893.63it/s]
Batch 14/16: 100%|█████████████████████████████████████████████████| 3125/3125 [891.26it/s]
Batch 15/16: 100%|█████████████████████████████████████████████████| 3125/3125 [905.06it/s]
Batch 16/16: 100%|█████████████████████████████████████████████████| 3125/3125 [911.72it/s]
```
### Map operations
The
[`map_samples()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.map_samples)
method provides a powerful and efficient interface for iterating over samples,
applying a function to each sample, and returning the results as a generator.
```python
from collections import Counter
import fiftyone as fo
import fiftyone.zoo as foz
dataset = foz.load_zoo_dataset("cifar10", split="train")
view = dataset.select_fields("ground_truth")
def map_fcn(sample):
return sample.ground_truth.label.upper()
counter = Counter()
for _, label in view.map_samples(map_fcn):
counter[label] += 1
print(dict(counter))
# {'DEER': 5000, 'HORSE': 5000, 'AIRPLANE': 5000, ..., 'DOG': 5000}
```
#### NOTE
As the above snippet shows, you should optimize your iteration by
[selecting only](using_views.md#efficient-iteration-views) the required fields.
By default,
[`map_samples()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.map_samples)
leverages a multiprocessing pool to parallelize the work across a number of
workers, resulting in significant performance improvements over the equivalent
[`iter_samples()`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset.iter_samples) syntax:
```python
counter = Counter()
for sample in view.iter_samples(progress=True):
label = map_fcn(sample)
counter[label] += 1
```
Keep the following points in mind while using
[`map_samples()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.map_samples):
- The samples are not processed in any particular order
- Your `map_fcn` should not modify global state or variables defined
outside of the function
- If your `map_fcn` modifies samples in-place, you must pass `save=True`
to save these edits
#### NOTE
Your `map_fcn` cannot return [`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample) objects directly. If you are
tempted to do this, then chances are good that you can express the
operation more efficiently and idiomatically via
[dataset views](using_views.md#using-views).
You can configure the number of workers that
[`map_samples()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.map_samples)
uses in a variety of ways:
- Configure the default number of workers used by all
[`map_samples()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.map_samples)
calls by setting the `default_process_pool_workers` value in your
[FiftyOne config](config.md#configuring-fiftyone)
- Manually configure the number of workers for a particular
[`map_samples()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.map_samples)
call by passing the `num_workers` parameter
- If neither of the above settings are applied,
[`map_samples()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.map_samples)
will use
[`recommend_process_pool_workers()`](../api/fiftyone.core.utils.md#fiftyone.core.utils.recommend_process_pool_workers)
to choose a number of worker processes, unless the method is called in a
daemon process (subprocess), in which case no workers are used
#### NOTE
You can set `default_process_pool_workers<=1` or `num_workers<=1` to
disable the use of multiprocessing pools in
[`map_samples()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.map_samples).
By default,
[`map_samples()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.map_samples)
evenly distributes samples to all workers in a single shard per worker.
However, you can pass the `batch_size` parameter to customize the number of
samples sent to each worker at a time:
```python
counter = Counter()
for _, label in view.map_samples(map_fcn, batch_size=50, num_workers=4):
counter[label] += 1
```
You can also pass `progress="workers"` to
[`map_samples()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.map_samples)
to render progress bar(s) for each worker:
```python
counter = Counter()
for _, label in view.map_samples(map_fcn, num_workers=16, progress="workers"):
counter[label] += 1
```
```text
Batch 01/16: 100%|█████████████████████████████████████████████████| 3125/3125 [899.01it/s]
Batch 02/16: 100%|█████████████████████████████████████████████████| 3125/3125 [894.90it/s]
Batch 03/16: 100%|█████████████████████████████████████████████████| 3125/3125 [900.14it/s]
Batch 04/16: 100%|█████████████████████████████████████████████████| 3125/3125 [895.61it/s]
Batch 05/16: 100%|█████████████████████████████████████████████████| 3125/3125 [903.09it/s]
Batch 06/16: 100%|█████████████████████████████████████████████████| 3125/3125 [895.33it/s]
Batch 07/16: 100%|█████████████████████████████████████████████████| 3125/3125 [893.26it/s]
Batch 08/16: 100%|█████████████████████████████████████████████████| 3125/3125 [889.17it/s]
Batch 09/16: 100%|█████████████████████████████████████████████████| 3125/3125 [888.16it/s]
Batch 10/16: 100%|█████████████████████████████████████████████████| 3125/3125 [893.69it/s]
Batch 11/16: 100%|█████████████████████████████████████████████████| 3125/3125 [896.80it/s]
Batch 12/16: 100%|█████████████████████████████████████████████████| 3125/3125 [903.28it/s]
Batch 13/16: 100%|█████████████████████████████████████████████████| 3125/3125 [893.63it/s]
Batch 14/16: 100%|█████████████████████████████████████████████████| 3125/3125 [891.26it/s]
Batch 15/16: 100%|█████████████████████████████████████████████████| 3125/3125 [905.06it/s]
Batch 16/16: 100%|█████████████████████████████████████████████████| 3125/3125 [911.72it/s]
```
### Setting values
Another strategy for performing efficient batch edits is to use
[`set_values()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.set_values) to
set a field (or embedded field) on each sample in the dataset in a single
batch operation:
```python
import random
import fiftyone as fo
import fiftyone.zoo as foz
from fiftyone import ViewField as F
dataset = foz.load_zoo_dataset("quickstart")
# Two ways to populate a `random` field on each sample in the dataset
# Dict syntax (recommended): provide a dict mapping sample IDs to values
values = {id: random.random() for id in dataset.values("id")}
dataset.set_values("random", values, key_field="id")
print(dataset.bounds("random"))
# (0.0028, 0.9925)
# List syntax: provide one value for each sample in the dataset
values = [random.random() for _ in range(len(dataset))]
dataset.set_values("random", values)
print(dataset.bounds("random"))
# (0.0055, 0.9996)
```
When applicable, using
[`set_values()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.set_values)
is more efficient than performing the equivalent operation via an explicit
iteration over the [`Dataset`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset) because it avoids the need to read [`Sample`](../api/fiftyone.core.sample.md#fiftyone.core.sample.Sample)
instances into memory and sequentially save them.
As demonstrated above, you can use
[`set_values()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.set_values)
in two ways:
- **Dict syntax (recommended):** provide values as a dict whose keys specify
the `key_field` values of the samples whose field you want to set to the
corresponding values
- **List syntax:** provide values as a list, one for each sample in the
collection on which you are invoking this method
#### NOTE
The most performant strategy for setting large numbers of field values is
to use the dict syntax with `key_field="id"` when setting sample fields
and `key_field="frames.id"` when setting frame fields. All other syntaxes
internally convert to these IDs before ultimately performing the updates.
You can also use
[`set_values()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.set_values) to
optimize more complex operations, such as editing attributes of specific
object detections in a nested list.
Consider the following loop, which adds a tag to all low confidence predictions
in a field:
```python
# Add a tag to all low confidence predictions in the dataset
for sample in dataset:
for detection in sample["predictions"].detections:
if detection.confidence < 0.06:
detection.tags.append("low_confidence")
sample.save()
print(dataset.count_label_tags())
# {'low_confidence': 447}
```
An equivalent but more efficient approach is to use
[`values()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.values) to
extract the slice of data you wish to modify and then use
[`set_values()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.set_values) to
save the updated data in a single batch operation:
```python
# Remove the tags we added in the previous variation
dataset.untag_labels("low_confidence")
# Load the tags for all low confidence detections
view = dataset.filter_labels("predictions", F("confidence") < 0.06)
tags = view.values("predictions.detections.tags")
# Add the 'low_confidence' tag to each detection's tags list
for sample_tags in tags:
for detection_tags in sample_tags:
detection_tags.append("low_confidence")
# Save the updated tags
view.set_values("predictions.detections.tags", tags)
print(dataset.count_label_tags())
# {'low_confidence': 447}
```
You can also use
[`set_values()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.set_values) to
perform batch updates to frame-level fields:
```python
import random
import fiftyone as fo
import fiftyone.zoo as foz
dataset = foz.load_zoo_dataset("quickstart-video")
# Dict syntax (recommended): provide a dict mapping frame IDs to values
frame_ids = dataset.values("frames.id", unwind=True)
values = {id: random.random() for id in frame_ids}
dataset.set_values("frames.random", values, key_field="frames.id")
print(dataset.bounds("frames.random"))
# (0.00013, 0.9993)
# List syntax: provide lists of lists of values, each list containing a
# value for each frame in that sample of the dataset
values = []
for sample in dataset:
values.append([random.random() for _ in sample.frames])
dataset.set_values("frames.random", values)
print(dataset.bounds("frames.random"))
# (0.00055, 0.9995)
```
### Setting label values
Often when working with [`Label`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Label) fields, the edits you want to make may be
naturally represented as a mapping between label IDs and corresponding
attribute values to set on each [`Label`](../api/fiftyone.core.labels.md#fiftyone.core.labels.Label) instance. In such cases, you can use
[`set_label_values()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.set_label_values)
to efficiently perform the updates:
```python
import fiftyone as fo
import fiftyone.zoo as foz
from fiftyone import ViewField as F
dataset = foz.load_zoo_dataset("quickstart")
# Grab some labels
view = dataset.limit(5).filter_labels("predictions", F("confidence") > 0.5)
# Two ways to populate a `random` attribute on each label
# List syntax (recommended): provide sample IDs and label IDs
values = []
for sid, lids in zip(*view.values(["id", "predictions.detections.id"])):
for lid in lids:
values.append({"sample_id": sid, "label_id": lid, "value": True})
dataset.set_label_values("predictions.detections.random", values)
print(dataset.count_values("predictions.detections.random"))
# {True: 25, None: 5595}
# Dict syntax: provide only label IDs
label_ids = view.values("predictions.detections.id", unwind=True)
values = {_id: True for _id in label_ids}
dataset.set_label_values("predictions.detections.random", values)
print(dataset.count_values("predictions.detections.random"))
# {True: 25, None: 5595}
```
As demonstrated above, you can use
[`set_label_values()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.set_label_values)
in two ways:
- **List syntax (recommended):** provide a list of dicts of the form
`{"sample_id": sample_id, "label_id": label_id, "value": value}`
specifying the sample IDs and label IDs of each label you want to edit
- **Dict syntax:** provide a dict mapping label IDs to values
#### NOTE
[`set_label_values()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.set_label_values)
is most efficient when you use the list syntax for values that includes
the sample/frame ID of each label that you are modifying.