fiftyone.core.odm.database

Database utilities.

Copyright 2017-2024, Voxel51, Inc.

voxel51.com

Classes:

DatabaseConfigDocument(conn[, version, type])

Backing document for the database config.

Functions:

get_db_config()	Retrieves the database config.
establish_db_conn(config)	Establishes the database connection.
aggregate(collection, pipelines)	Executes one or more aggregations on a collection.
ensure_connection()	Ensures database connection exists
get_db_client()	Returns a database client.
get_db_conn()	Returns a connection to the database.
get_async_db_client([use_global])	Returns an async database client.
get_async_db_conn([use_global])	Returns an async connection to the database.
drop_database()	Drops the database.
sync_database()	Syncs all pending database writes to disk.
list_collections()	Returns a list of all collection names in the database.
drop_collection(collection_name)	Drops specified collection from the database.
drop_orphan_collections([dry_run])	Drops all orphan collections from the database.
drop_orphan_saved_views([dry_run])	Drops all orphan saved views from the database.
drop_orphan_runs([dry_run])	Drops all orphan runs from the database.
stream_collection(collection_name)	Streams the contents of the collection to stdout.
get_collection_stats(collection_name)	Sets stats about the collection.
count_documents(coll, pipeline)
export_document(doc, json_path)	Exports the document to disk in JSON format.
export_collection(docs, json_dir_or_path[, …])	Exports the collection to disk in JSON format.
import_document(json_path)	Imports a document from JSON on disk.
import_collection(json_dir_or_path[, key])	Imports the collection from JSON on disk.
insert_documents(docs, coll[, ordered, …])	Inserts documents into a collection.
bulk_write(ops, coll[, ordered, progress])	Performs a batch of write operations on a collection.
list_datasets()	Returns the list of available FiftyOne datasets.
patch_saved_views(dataset_name[, dry_run])	Ensures that the saved view documents in the views collection for the given dataset exactly match the IDs in its dataset document.
patch_workspaces(dataset_name[, dry_run])	Ensures that the workspace documents in the workspaces collection for the given dataset exactly match the IDs in its dataset document.
patch_annotation_runs(dataset_name[, dry_run])	Ensures that the annotation runs in the runs collection for the given dataset exactly match the values in its dataset document.
patch_brain_runs(dataset_name[, dry_run])	Ensures that the brain method runs in the runs collection for the given dataset exactly match the values in its dataset document.
patch_evaluations(dataset_name[, dry_run])	Ensures that the evaluation runs in the runs collection for the given dataset exactly match the values in its dataset document.
patch_runs(dataset_name[, dry_run])	Ensures that the runs in the runs collection for the given dataset exactly match the values in its dataset document.
delete_dataset(name[, dry_run])	Deletes the dataset with the given name.
delete_saved_view(dataset_name, view_name[, …])	Deletes the saved view with the given name from the dataset with the given name.
delete_saved_views(dataset_name[, dry_run])	Deletes all saved views from the dataset with the given name.
delete_annotation_run(name, anno_key[, dry_run])	Deletes the annotation run with the given key from the dataset with the given name.
delete_annotation_runs(name[, dry_run])	Deletes all annotation runs from the dataset with the given name.
delete_brain_run(name, brain_key[, dry_run])	Deletes the brain method run with the given key from the dataset with the given name.
delete_brain_runs(name[, dry_run])	Deletes all brain method runs from the dataset with the given name.
delete_evaluation(name, eval_key[, dry_run])	Deletes the evaluation run with the given key from the dataset with the given name.
delete_evaluations(name[, dry_run])	Deletes all evaluations from the dataset with the given name.
delete_run(name, run_key[, dry_run])	Deletes the run with the given key from the dataset with the given name.
delete_runs(name[, dry_run])	Deletes all runs from the dataset with the given name.

class fiftyone.core.odm.database.DatabaseConfigDocument(conn, version=None, type=None, *args, **kwargs)

Bases: object

Backing document for the database config.

Attributes:

Methods:

save()

version: str

type: str

save()

fiftyone.core.odm.database.get_db_config()

Retrieves the database config.

Returns

a DatabaseConfigDocument

fiftyone.core.odm.database.establish_db_conn(config)

Establishes the database connection.

If fiftyone.config.database_uri is defined, then we connect to that URI. Otherwise, a fiftyone.core.service.DatabaseService is created.

Parameters

config – a fiftyone.core.config.FiftyOneConfig

Raises

• ConnectionError – if a connection to mongod could not be established

• FiftyOneConfigError – if fiftyone.config.database_uri is not defined and mongod could not be found

• ServiceExecutableNotFound – if fiftyone.core.service.DatabaseService startup was attempted, but mongod was not found in fiftyone.db.bin

• RuntimeError – if the mongod found does not meet FiftyOne’s requirements, or validation could not occur

fiftyone.core.odm.database.aggregate(collection, pipelines)

Executes one or more aggregations on a collection.

Multiple aggregations are executed using multiple threads, and their results are returned as lists rather than cursors.

Parameters

• collection – a pymongo.collection.Collection or motor.motor_asyncio.AsyncIOMotorCollection

• pipelines – a MongoDB aggregation pipeline or a list of pipelines

Returns

• If a single pipeline is provided, a pymongo.command_cursor.CommandCursor or motor.motor_asyncio.AsyncIOMotorCommandCursor is returned

• If multiple pipelines are provided, each cursor is extracted into a list and the list of lists is returned

fiftyone.core.odm.database.ensure_connection()

Ensures database connection exists

fiftyone.core.odm.database.get_db_client()

Returns a database client.

Returns

a pymongo.mongo_client.MongoClient

fiftyone.core.odm.database.get_db_conn()

Returns a connection to the database.

Returns

a pymongo.database.Database

fiftyone.core.odm.database.get_async_db_client(use_global=False)

Returns an async database client.

Parameters

use_global – whether to use the global client singleton

Returns

a motor.motor_asyncio.AsyncIOMotorClient

fiftyone.core.odm.database.get_async_db_conn(use_global=False)

Returns an async connection to the database.

Returns

a motor.motor_asyncio.AsyncIOMotorDatabase

fiftyone.core.odm.database.drop_database()

Drops the database.

fiftyone.core.odm.database.sync_database()

Syncs all pending database writes to disk.

fiftyone.core.odm.database.list_collections()

Returns a list of all collection names in the database.

Returns

a list of all collection names

fiftyone.core.odm.database.drop_collection(collection_name)

Drops specified collection from the database.

Parameters

collection_name – the collection name

fiftyone.core.odm.database.drop_orphan_collections(dry_run=False)

Drops all orphan collections from the database.

Orphan collections are collections that are not associated with any known dataset or other collections used by FiftyOne.

Parameters

dry_run (False) – whether to log the actions that would be taken but not perform them

fiftyone.core.odm.database.drop_orphan_saved_views(dry_run=False)

Drops all orphan saved views from the database.

Orphan saved views are saved view documents that are not associated with any known dataset or other collections used by FiftyOne.

Parameters

dry_run (False) – whether to log the actions that would be taken but not perform them

fiftyone.core.odm.database.drop_orphan_runs(dry_run=False)

Drops all orphan runs from the database.

Orphan runs are runs that are not associated with any known dataset or other collections used by FiftyOne.

Parameters

dry_run (False) – whether to log the actions that would be taken but not perform them

fiftyone.core.odm.database.stream_collection(collection_name)

Streams the contents of the collection to stdout.

Parameters

collection_name – the name of the collection

fiftyone.core.odm.database.get_collection_stats(collection_name)

Sets stats about the collection.

Parameters

collection_name – the name of the collection

Returns

a stats dict

fiftyone.core.odm.database.count_documents(coll, pipeline)

fiftyone.core.odm.database.export_document(doc, json_path)

Exports the document to disk in JSON format.

Parameters

• doc – a BSON document dict

• json_path – the path to write the JSON file

fiftyone.core.odm.database.export_collection(docs, json_dir_or_path, key='documents', patt='{idx:06d}-{id}.json', num_docs=None, progress=None)

Exports the collection to disk in JSON format.

Parameters

• docs – an iterable containing the documents to export

• json_dir_or_path – the path to write a single JSON file containing the entire collection, or a directory in which to write per-document JSON files

• key ("documents") – the field name under which to store the documents when json_path is a single JSON file

• ("{idx (patt) – 06d}-{id}.json”): a filename pattern to use when json_path is a directory. The pattern may contain idx to refer to the index of the document in docs or id to refer to the document’s ID

• num_docs (None) – the total number of documents. If omitted, this must be computable via len(docs)

• progress (None) – whether to render a progress bar (True/False), use the default value fiftyone.config.show_progress_bars (None), or a progress callback function to invoke instead

fiftyone.core.odm.database.import_document(json_path)

Imports a document from JSON on disk.

Parameters

json_path – the path to the document

Returns

a BSON document dict

fiftyone.core.odm.database.import_collection(json_dir_or_path, key='documents')

Imports the collection from JSON on disk.

Parameters

• json_dir_or_path – the path to a JSON file on disk, or a directory containing per-document JSON files

• key ("documents") – the field name under which the documents are stored when json_path is a single JSON file

Returns

a tuple of

• an iterable of BSON documents

• the number of documents

fiftyone.core.odm.database.insert_documents(docs, coll, ordered=False, progress=None, num_docs=None)

Inserts documents into a collection.

The _id field of the input documents will be populated if it is not already set.

Parameters

• docs – an iterable of BSON document dicts

• coll – a pymongo collection

• ordered (False) – whether the documents must be inserted in order

• progress (None) – whether to render a progress bar (True/False), use the default value fiftyone.config.show_progress_bars (None), or a progress callback function to invoke instead

• num_docs (None) – the total number of documents. Only used when progress=True. If omitted, this will be computed via len(docs), if possible

Returns

a list of IDs of the inserted documents

fiftyone.core.odm.database.bulk_write(ops, coll, ordered=False, progress=False)

Performs a batch of write operations on a collection.

Parameters

• ops – a list of pymongo operations

• coll – a pymongo collection

• ordered (False) – whether the operations must be performed in order

• progress (False) – whether to render a progress bar (True/False), use the default value fiftyone.config.show_progress_bars (None), or a progress callback function to invoke instead

fiftyone.core.odm.database.list_datasets()

Returns the list of available FiftyOne datasets.

This is a low-level implementation of dataset listing that does not call fiftyone.core.dataset.list_datasets(), which is helpful if a database may be corrupted.

Returns

a list of Dataset names

fiftyone.core.odm.database.patch_saved_views(dataset_name, dry_run=False)

Ensures that the saved view documents in the views collection for the given dataset exactly match the IDs in its dataset document.

Parameters

• dataset_name – the name of the dataset

• dry_run (False) – whether to log the actions that would be taken but not perform them

fiftyone.core.odm.database.patch_workspaces(dataset_name, dry_run=False)

Ensures that the workspace documents in the workspaces collection for the given dataset exactly match the IDs in its dataset document.

Parameters

• dataset_name – the name of the dataset

• dry_run (False) – whether to log the actions that would be taken but not perform them

fiftyone.core.odm.database.patch_annotation_runs(dataset_name, dry_run=False)

Ensures that the annotation runs in the runs collection for the given dataset exactly match the values in its dataset document.

Parameters

• dataset_name – the name of the dataset

• dry_run (False) – whether to log the actions that would be taken but not perform them

fiftyone.core.odm.database.patch_brain_runs(dataset_name, dry_run=False)

Ensures that the brain method runs in the runs collection for the given dataset exactly match the values in its dataset document.

Parameters

• dataset_name – the name of the dataset

• dry_run (False) – whether to log the actions that would be taken but not perform them

fiftyone.core.odm.database.patch_evaluations(dataset_name, dry_run=False)

Ensures that the evaluation runs in the runs collection for the given dataset exactly match the values in its dataset document.

Parameters

• dataset_name – the name of the dataset

• dry_run (False) – whether to log the actions that would be taken but not perform them

fiftyone.core.odm.database.patch_runs(dataset_name, dry_run=False)

Ensures that the runs in the runs collection for the given dataset exactly match the values in its dataset document.

Parameters

• dataset_name – the name of the dataset

• dry_run (False) – whether to log the actions that would be taken but not perform them

fiftyone.core.odm.database.delete_dataset(name, dry_run=False)

Deletes the dataset with the given name.

This is a low-level implementation of deletion that does not call fiftyone.core.dataset.load_dataset(), which is helpful if a dataset’s backing document or collections are corrupted and cannot be loaded via the normal pathways.

Parameters

• name – the name of the dataset

• dry_run (False) – whether to log the actions that would be taken but not perform them

fiftyone.core.odm.database.delete_saved_view(dataset_name, view_name, dry_run=False)

Deletes the saved view with the given name from the dataset with the given name.

This is a low-level implementation of deletion that does not call fiftyone.core.dataset.load_dataset() or fiftyone.core.collections.SampleCollection.load_saved_view(), which is helpful if a dataset’s backing document or collections are corrupted and cannot be loaded via the normal pathways.

Parameters

• dataset_name – the name of the dataset

• view_name – the name of the saved view

• dry_run (False) – whether to log the actions that would be taken but not perform them

fiftyone.core.odm.database.delete_saved_views(dataset_name, dry_run=False)

Deletes all saved views from the dataset with the given name.

This is a low-level implementation of deletion that does not call fiftyone.core.dataset.load_dataset() or fiftyone.core.collections.SampleCollection.load_saved_view(), which is helpful if a dataset’s backing document or collections are corrupted and cannot be loaded via the normal pathways.

Parameters

• dataset_name – the name of the dataset

• dry_run (False) – whether to log the actions that would be taken but not perform them

fiftyone.core.odm.database.delete_annotation_run(name, anno_key, dry_run=False)

Deletes the annotation run with the given key from the dataset with the given name.

This is a low-level implementation of deletion that does not call fiftyone.core.dataset.load_dataset() or fiftyone.core.collections.SampleCollection.delete_annotation_run(), which is helpful if a dataset’s backing document or collections are corrupted and cannot be loaded via the normal pathways.

Note that, as this method does not load fiftyone.core.runs.Run instances, it does not call fiftyone.core.runs.Run.cleanup().

Parameters

• name – the name of the dataset

• anno_key – the annotation key

• dry_run (False) – whether to log the actions that would be taken but not perform them

fiftyone.core.odm.database.delete_annotation_runs(name, dry_run=False)

Deletes all annotation runs from the dataset with the given name.

This is a low-level implementation of deletion that does not call fiftyone.core.dataset.load_dataset() or fiftyone.core.collections.SampleCollection.delete_annotation_runs(), which is helpful if a dataset’s backing document or collections are corrupted and cannot be loaded via the normal pathways.

Note that, as this method does not load fiftyone.core.runs.Run instances, it does not call fiftyone.core.runs.Run.cleanup().

Parameters

• name – the name of the dataset

• dry_run (False) – whether to log the actions that would be taken but not perform them

fiftyone.core.odm.database.delete_brain_run(name, brain_key, dry_run=False)

Deletes the brain method run with the given key from the dataset with the given name.

This is a low-level implementation of deletion that does not call fiftyone.core.dataset.load_dataset() or fiftyone.core.collections.SampleCollection.delete_brain_run(), which is helpful if a dataset’s backing document or collections are corrupted and cannot be loaded via the normal pathways.

Note that, as this method does not load fiftyone.core.runs.Run instances, it does not call fiftyone.core.runs.Run.cleanup().

Parameters

• name – the name of the dataset

• brain_key – the brain key

• dry_run (False) – whether to log the actions that would be taken but not perform them

fiftyone.core.odm.database.delete_brain_runs(name, dry_run=False)

Deletes all brain method runs from the dataset with the given name.

This is a low-level implementation of deletion that does not call fiftyone.core.dataset.load_dataset() or fiftyone.core.collections.SampleCollection.delete_brain_runs(), which is helpful if a dataset’s backing document or collections are corrupted and cannot be loaded via the normal pathways.

Note that, as this method does not load fiftyone.core.runs.Run instances, it does not call fiftyone.core.runs.Run.cleanup().

Parameters

• name – the name of the dataset

• dry_run (False) – whether to log the actions that would be taken but not perform them

fiftyone.core.odm.database.delete_evaluation(name, eval_key, dry_run=False)

Deletes the evaluation run with the given key from the dataset with the given name.

This is a low-level implementation of deletion that does not call fiftyone.core.dataset.load_dataset() or fiftyone.core.collections.SampleCollection.delete_evaluation(), which is helpful if a dataset’s backing document or collections are corrupted and cannot be loaded via the normal pathways.

Note that, as this method does not load fiftyone.core.runs.Run instances, it does not call fiftyone.core.runs.Run.cleanup().

Parameters

• name – the name of the dataset

• eval_key – the evaluation key

• dry_run (False) – whether to log the actions that would be taken but not perform them

fiftyone.core.odm.database.delete_evaluations(name, dry_run=False)

Deletes all evaluations from the dataset with the given name.

This is a low-level implementation of deletion that does not call fiftyone.core.dataset.load_dataset() or fiftyone.core.collections.SampleCollection.delete_evaluations(), which is helpful if a dataset’s backing document or collections are corrupted and cannot be loaded via the normal pathways.

Note that, as this method does not load fiftyone.core.runs.Run instances, it does not call fiftyone.core.runs.Run.cleanup().

Parameters

• name – the name of the dataset

• dry_run (False) – whether to log the actions that would be taken but not perform them

fiftyone.core.odm.database.delete_run(name, run_key, dry_run=False)

Deletes the run with the given key from the dataset with the given name.

This is a low-level implementation of deletion that does not call fiftyone.core.dataset.load_dataset() or fiftyone.core.collections.SampleCollection.delete_run(), which is helpful if a dataset’s backing document or collections are corrupted and cannot be loaded via the normal pathways.

Note that, as this method does not load fiftyone.core.runs.Run instances, it does not call fiftyone.core.runs.Run.cleanup().

Parameters

• name – the name of the dataset

• run_key – the run key

• dry_run (False) – whether to log the actions that would be taken but not perform them

fiftyone.core.odm.database.delete_runs(name, dry_run=False)

Deletes all runs from the dataset with the given name.

This is a low-level implementation of deletion that does not call fiftyone.core.dataset.load_dataset() or fiftyone.core.collections.SampleCollection.delete_runs(), which is helpful if a dataset’s backing document or collections are corrupted and cannot be loaded via the normal pathways.

Note that, as this method does not load fiftyone.core.runs.Run instances, it does not call fiftyone.core.runs.Run.cleanup().

Parameters

• name – the name of the dataset

• dry_run (False) – whether to log the actions that would be taken but not perform them