# Pgvector Vector Search Integration
Available in:
Open SourceEnterprise
[Pgvector](https://github.com/pgvector/pgvector) is a vector search extension
to PostgreSQL, one of the most popular open source databases, and we’ve made it
easy to use Pgvector on your computer vision data directly from FiftyOne!
Follow these [simple instructions](#pgvector-setup) to get started
using Pgvector + FiftyOne.
FiftyOne provides an API to create Pgvector indexes, upload vectors, and
run similarity queries, both [programmatically](#pgvector-query) in
Python and via point-and-click in the App.
#### NOTE
Did you know? You can
[search by natural language](../user_guide/similarity.md#brain-similarity-text) using
Pgvector similarity indexes!
## Basic recipe
The basic workflow to use Pgvector to create a similarity index on your
FiftyOne datasets and use this to query your data is as follows:
1. Connect to or start a PostgreSQL server with the pgvector extension
2. [Load a dataset](../user_guide/import_datasets.md#importing-datasets) into FiftyOne
3. Compute embedding vectors for samples or patches in your dataset, or select
a model to use to generate embeddings
4. Use the [`compute_similarity()`](../api/fiftyone.brain.md#fiftyone.brain.compute_similarity)
method to generate a Pgvector similarity index for the samples or
object patches in a dataset by setting the parameter
`backend="pgvector"` and specifying a `brain_key` of your choice
5. Use this Pgvector similarity index to query your data with
[`sort_by_similarity()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.sort_by_similarity)
6. If desired, delete the index
The example below demonstrates this workflow.
#### NOTE
You must have access to
[a PostgreSQL instance with pgvector extension](https://github.com/pgvector/pgvector?tab=readme-ov-file#additional-installation-methods)
and install the
[psycopg2 Python module](https://pypi.org/project/psycopg2/)
to run this example:
```shell
pip install psycopg2
```
You can store credentials for your Postgres instance
as described in [this section](#pgvector-setup)
to avoid entering them manually each time you interact with your
Pgvector index.
First let’s load a dataset into FiftyOne and compute embeddings for the samples:
```python
import fiftyone as fo
import fiftyone.brain as fob
import fiftyone.zoo as foz
# Step 1: Load your data into FiftyOne
dataset = foz.load_zoo_dataset("quickstart")
# Steps 2 and 3: Compute embeddings and create a similarity index
pgvector_index = fob.compute_similarity(
dataset,
brain_key="pgvector_index",
backend="pgvector",
)
```
Once the similarity index has been generated, we can query our data in FiftyOne
by specifying the `brain_key`:
```python
# Step 4: Query your data
query = dataset.first().id # query by sample ID
view = dataset.sort_by_similarity(
query,
brain_key="pgvector_index",
k=10, # limit to 10 most similar samples
)
# Step 5 (optional): Cleanup
# Delete the pgvector index
pgvector_index.cleanup()
# Delete run record from FiftyOne
dataset.delete_brain_run("pgvector_index")
```
#### NOTE
Skip to [this section](#pgvector-examples) for a variety of
common pgvector query patterns.
## Setup
The easiest way to get started with pgvector is to
[install locally via Docker](https://github.com/pgvector/pgvector?tab=readme-ov-file#docker).
### Installing the psycopg2 client
In order to use the pgvector backend, you must also install the
[psycopg2 Python module](https://pypi.org/project/psycopg2/):
```shell
pip install psycopg2
```
### Using the pgvector backend
By default, calling
[`compute_similarity()`](../api/fiftyone.brain.md#fiftyone.brain.compute_similarity) or
[`sort_by_similarity()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.sort_by_similarity)
will use an sklearn backend.
To use the pgvector backend, simply set the optional `backend` parameter of
[`compute_similarity()`](../api/fiftyone.brain.md#fiftyone.brain.compute_similarity) to
`"pgvector"`:
```python
import fiftyone.brain as fob
fob.compute_similarity(..., backend="pgvector", ...)
```
Alternatively, you can permanently configure FiftyOne to use the pgvector
backend by setting the following environment variable:
```shell
export FIFTYONE_BRAIN_DEFAULT_SIMILARITY_BACKEND=pgvector
```
or by setting the `default_similarity_backend` parameter of your
[brain config](../brain/index.md#brain-config) located at `~/.fiftyone/brain_config.json`:
```json
{
"default_similarity_backend": "pgvector"
}
```
### Authentication
If you are using a custom pgvector server, you can provide your
credentials in a
[variety of ways](https://www.psycopg.org/docs/module.html#module-psycopg2).
**Environment variables (recommended)**
The recommended way to configure your pgvector credentials is to store
them in the environment variables shown below, which are automatically accessed
by FiftyOne whenever a connection to pgvector is made.
```shell
export FIFTYONE_BRAIN_SIMILARITY_PGVECTOR_CONNECTION_STRING=postgresql://postgres:mysecretpassword@localhost:5432/postgres
```
This is only one example of variables that can be used to authenticate a
pgvector client. Find more information
[here.](https://www.psycopg.org/docs/module.html#module-psycopg2)
**FiftyOne Brain config**
You can also store your credentials in your [brain config](../brain/index.md#brain-config)
located at `~/.fiftyone/brain_config.json`:
```json
{
"similarity_backends": {
"pgvector": {
"connection_string": "postgresql://postgres:mysecretpassword@localhost:5432/postgres"
}
}
}
```
Note that this file will not exist until you create it.
**Keyword arguments**
You can manually provide credentials as keyword arguments each time you call
methods like [`compute_similarity()`](../api/fiftyone.brain.md#fiftyone.brain.compute_similarity)
that require connections to pgvector:
```python
import fiftyone.brain as fob
pgvector_index = fob.compute_similarity(
...
backend="pgvector",
brain_key="pgvector_index",
connection_string="postgresql://postgres:mysecretpassword@localhost:5432/postgres",
)
```
Note that, when using this strategy, you must manually provide the credentials
when loading an index later via
[`load_brain_results()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.load_brain_results):
```python
pgvector_index = dataset.load_brain_results(
"pgvector_index",
connection_string="postgresql://postgres:mysecretpassword@localhost:5432/postgres",
)
```
### pgvector config parameters
The pgvector backend supports a variety of query parameters that can be
used to customize your similarity queries. These parameters include:
- **index_name** (*None*): the name of the pgvector vector search index
to use or create. If not specified, a new unique name is generated automatically
- **table_name** (*None*): the name of the postgres table to use or create
for storing vectors. If not specified, a new unique name is generated automatically
- **metric** ( *“cosine”*): the distance/similarity metric to use when
creating a new index. The supported values are
`("cosine", "dotproduct", "euclidean", "l1")`
- **work_mem** ( *“64MB”*): the base maximum amount of memory to be used by a query operation
(such as a sort or hash table) before writing to temporary disk files
- **maintenance_work_mem** (*None*): an optional maximum amount of memory
available to index builds. If not provided, the server default (typically
64MB) is used. Increase this for high-dimensional embeddings or IVFFlat
indexes with many lists
- **vector_type** ( *“vector”*): the pgvector column type used to store
embeddings. The supported values are `("vector", "halfvec")`.
`"halfvec"` stores half-precision (float16) vectors and supports
indexes with up to 4000 dimensions, versus 2000 for `"vector"`, and
requires pgvector >= 0.7.0
- **index_type** ( *“hnsw”*): the type of vector index to create. The
supported values are `("hnsw", "ivfflat")`. Note that IVFFlat indexes
only support the `("cosine", "dotproduct", "euclidean")` metrics
- **hnsw_m** (*16*): The max number of connections per layer in the HNSW index
- **hnsw_ef_construction** (*64*): the size of the dynamic candidate list for constructing
the graph for the HNSW index
- **hnsw_ef_search** (*None*): an optional size of the dynamic candidate
list for HNSW searches. If not provided, the server default (40) is used.
Larger values improve recall at the cost of speed
- **ivfflat_lists** (*100*): the number of inverted lists in the IVFFlat
index
- **ivfflat_probes** (*1*): the number of lists to probe during IVFFlat
searches. Larger values improve recall at the cost of speed
For detailed information on these parameters, see the
[pgvector index options documentation](https://github.com/pgvector/pgvector/?tab=readme-ov-file#index-options).
You can specify these parameters via any of the strategies described in the
previous section. Here’s an example of a [brain config](../brain/index.md#brain-config)
that includes all of the available parameters:
```json
{
"similarity_backends": {
"pgvector": {
"index_name": "your-index",
"table_name": "your-table",
"metric": "cosine",
"work_mem": "64MB",
"maintenance_work_mem": "256MB",
"vector_type": "vector",
"index_type": "hnsw",
"hnsw_m": 16,
"hnsw_ef_construction": 64,
"hnsw_ef_search": 40,
"ivfflat_lists": 100,
"ivfflat_probes": 1
}
}
}
```
However, typically these parameters are directly passed to
[`compute_similarity()`](../api/fiftyone.brain.md#fiftyone.brain.compute_similarity) to configure
a specific new index:
```python
# A customized HNSW index
pgvector_index = fob.compute_similarity(
...
backend="pgvector",
brain_key="pgvector_index",
index_name="your-index",
table_name="your-table",
metric="cosine",
work_mem="64MB",
index_type="hnsw",
hnsw_m=16,
hnsw_ef_construction=64,
hnsw_ef_search=40,
)
# A customized IVFFlat index
pgvector_index = fob.compute_similarity(
...
backend="pgvector",
brain_key="pgvector_index",
index_name="your-index",
table_name="your-table",
metric="cosine",
work_mem="64MB",
index_type="ivfflat",
ivfflat_lists=100,
ivfflat_probes=1,
)
```
## Managing brain runs
FiftyOne provides a variety of methods that you can use to manage brain runs.
For example, you can call
[`list_brain_runs()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.list_brain_runs)
to see the available brain keys on a dataset:
```python
import fiftyone.brain as fob
# List all brain runs
dataset.list_brain_runs()
# Only list similarity runs
dataset.list_brain_runs(type=fob.Similarity)
# Only list specific similarity runs
dataset.list_brain_runs(
type=fob.Similarity,
patches_field="ground_truth",
supports_prompts=True,
)
```
Or, you can use
[`get_brain_info()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.get_brain_info)
to retrieve information about the configuration of a brain run:
```python
info = dataset.get_brain_info(brain_key)
print(info)
```
Use [`load_brain_results()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.load_brain_results)
to load the [`SimilarityIndex`](../api/fiftyone.brain.similarity.md#fiftyone.brain.similarity.SimilarityIndex) instance for a brain run.
You can use
[`rename_brain_run()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.rename_brain_run)
to rename the brain key associated with an existing similarity results run:
```python
dataset.rename_brain_run(brain_key, new_brain_key)
```
Finally, you can use
[`delete_brain_run()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.delete_brain_run)
to delete the record of a similarity index computation from your FiftyOne
dataset:
```python
dataset.delete_brain_run(brain_key)
```
#### NOTE
Calling
[`delete_brain_run()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.delete_brain_run)
only deletes the **record** of the brain run from your FiftyOne dataset; it
will not delete any associated pgvector index, which you can do as
follows:
```python
# Delete the pgvector index
pgvector_index = dataset.load_brain_results(brain_key)
pgvector_index.cleanup()
```
## Examples
This section demonstrates how to perform some common vector search workflows on
a FiftyOne dataset using the pgvector backend.
#### NOTE
All of the examples below assume you have configured your pgvector
server as described in [this section](#pgvector-setup).
### Create a similarity index
In order to create a new pgvector similarity index, you need to specify
either the `embeddings` or `model` argument to
[`compute_similarity()`](../api/fiftyone.brain.md#fiftyone.brain.compute_similarity). Here’s a few
possibilities:
```python
import fiftyone as fo
import fiftyone.brain as fob
import fiftyone.zoo as foz
dataset = foz.load_zoo_dataset("quickstart")
model_name = "clip-vit-base32-torch"
model = foz.load_zoo_model(model_name)
brain_key = "pgvector_index"
# Option 1: Compute embeddings on the fly from model name
fob.compute_similarity(
dataset,
model=model_name,
backend="pgvector",
brain_key=brain_key,
)
# Option 2: Compute embeddings on the fly from model instance
fob.compute_similarity(
dataset,
model=model,
backend="pgvector",
brain_key=brain_key,
)
# Option 3: Pass precomputed embeddings as a numpy array
embeddings = dataset.compute_embeddings(model)
fob.compute_similarity(
dataset,
embeddings=embeddings,
backend="pgvector",
brain_key=brain_key,
)
# Option 4: Pass precomputed embeddings by field name
dataset.compute_embeddings(model, embeddings_field="embeddings")
fob.compute_similarity(
dataset,
embeddings="embeddings",
backend="pgvector",
brain_key=brain_key,
)
```
### Create a patch similarity index
You can also create a similarity index for
[object patches](../user_guide/similarity.md#brain-object-similarity) within your dataset by
including the `patches_field` argument to
[`compute_similarity()`](../api/fiftyone.brain.md#fiftyone.brain.compute_similarity):
```python
import fiftyone as fo
import fiftyone.brain as fob
import fiftyone.zoo as foz
dataset = foz.load_zoo_dataset("quickstart")
fob.compute_similarity(
dataset,
patches_field="ground_truth",
model="clip-vit-base32-torch",
backend="pgvector",
brain_key="pgvector_patches",
)
```
### Connect to an existing index
If you have already created a pgvector index storing the embedding vectors
for the samples or patches in your dataset, you can connect to it by passing
the `index_name` to
[`compute_similarity()`](../api/fiftyone.brain.md#fiftyone.brain.compute_similarity):
```python
import fiftyone as fo
import fiftyone.brain as fob
import fiftyone.zoo as foz
dataset = foz.load_zoo_dataset("quickstart")
fob.compute_similarity(
dataset,
model="clip-vit-base32-torch", # zoo model used (if applicable)
embeddings=False, # don't compute embeddings
index_name="your-index", # the existing pgvector index
brain_key="pgvector_index",
backend="pgvector",
)
```
### Add/remove embeddings from an index
You can use
[`add_to_index()`](../api/fiftyone.brain.similarity.md#fiftyone.brain.similarity.SimilarityIndex.add_to_index)
and
[`remove_from_index()`](../api/fiftyone.brain.similarity.md#fiftyone.brain.similarity.SimilarityIndex.remove_from_index)
to add and remove embeddings from an existing pgvector index.
These methods can come in handy if you modify your FiftyOne dataset and need
to update the pgvector index to reflect these changes:
```python
import numpy as np
import fiftyone as fo
import fiftyone.brain as fob
import fiftyone.zoo as foz
dataset = foz.load_zoo_dataset("quickstart")
pgvector_index = fob.compute_similarity(
dataset,
model="clip-vit-base32-torch",
brain_key="pgvector_index",
backend="pgvector",
)
print(pgvector_index.total_index_size) # 200
view = dataset.take(10)
ids = view.values("id")
# Delete 10 samples from a dataset
dataset.delete_samples(view)
# Delete the corresponding vectors from the index
pgvector_index.remove_from_index(sample_ids=ids)
# Add 20 samples to a dataset
samples = [fo.Sample(filepath="tmp%d.jpg" % i) for i in range(20)]
sample_ids = dataset.add_samples(samples)
# Add corresponding embeddings to the index
embeddings = np.random.rand(20, 512)
pgvector_index.add_to_index(embeddings, sample_ids)
print(pgvector_index.total_index_size) # 210
```
### Retrieve embeddings from an index
You can use
[`get_embeddings()`](../api/fiftyone.brain.similarity.md#fiftyone.brain.similarity.SimilarityIndex.get_embeddings)
to retrieve embeddings from a pgvector index by ID:
```python
import fiftyone as fo
import fiftyone.brain as fob
import fiftyone.zoo as foz
dataset = foz.load_zoo_dataset("quickstart")
pgvector_index = fob.compute_similarity(
dataset,
model="clip-vit-base32-torch",
brain_key="pgvector_index",
backend="pgvector",
)
# Retrieve embeddings for the entire dataset
ids = dataset.values("id")
embeddings, sample_ids, _ = pgvector_index.get_embeddings(sample_ids=ids)
print(embeddings.shape) # (200, 512)
print(sample_ids.shape) # (200,)
# Retrieve embeddings for a view
ids = dataset.take(10).values("id")
embeddings, sample_ids, _ = pgvector_index.get_embeddings(sample_ids=ids)
print(embeddings.shape) # (10, 512)
print(sample_ids.shape) # (10,)
```
### Querying a pgvector index
You can query a pgvector index by appending a
[`sort_by_similarity()`](../api/fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection.sort_by_similarity)
stage to any dataset or view. The query can be any of the following:
* An ID (sample or patch)
* A query vector of same dimension as the index
* A list of IDs (samples or patches)
* A text prompt (if [supported by the model](../user_guide/similarity.md#brain-similarity-text))
```python
import numpy as np
import fiftyone as fo
import fiftyone.brain as fob
import fiftyone.zoo as foz
dataset = foz.load_zoo_dataset("quickstart")
fob.compute_similarity(
dataset,
model="clip-vit-base32-torch",
brain_key="pgvector_index",
backend="pgvector",
)
# Query by vector
query = np.random.rand(512) # matches the dimension of CLIP embeddings
view = dataset.sort_by_similarity(query, k=10, brain_key="pgvector_index")
# Query by sample ID
query = dataset.first().id
view = dataset.sort_by_similarity(query, k=10, brain_key="pgvector_index")
# Query by a list of IDs
query = [dataset.first().id, dataset.last().id]
view = dataset.sort_by_similarity(query, k=10, brain_key="pgvector_index")
# Query by text prompt
query = "a photo of a dog"
view = dataset.sort_by_similarity(query, k=10, brain_key="pgvector_index")
```
#### NOTE
Performing a similarity search on a [`DatasetView`](../api/fiftyone.core.view.md#fiftyone.core.view.DatasetView) will **only** return
results from the view; if the view contains samples that were not included
in the index, they will never be included in the result.
This means that you can index an entire [`Dataset`](../api/fiftyone.core.dataset.md#fiftyone.core.dataset.Dataset) once and then perform
searches on subsets of the dataset by
[constructing views](../user_guide/using_views.md#using-views) that contain the images of
interest.
### Advanced usage
As [previously mentioned](#pgvector-config-parameters), you can
customize your pgvector indexes by providing optional parameters to
[`compute_similarity()`](../api/fiftyone.brain.md#fiftyone.brain.compute_similarity).
Here’s an example of creating a similarity index backed by a customized
pgvector index. Just for fun, we’ll specify a custom index name, use dot
product similarity, and populate the index for only a subset of our dataset:
```python
import fiftyone as fo
import fiftyone.brain as fob
import fiftyone.zoo as foz
dataset = foz.load_zoo_dataset("quickstart")
# Create a custom pgvector index
pgvector_index = fob.compute_similarity(
dataset,
model="clip-vit-base32-torch",
embeddings=False, # we'll add embeddings below
metric="dotproduct",
brain_key="pgvector_index",
backend="pgvector",
index_name="custom-quickstart-index",
)
# Add embeddings for a subset of the dataset
view = dataset.take(10)
embeddings, sample_ids, _ = pgvector_index.compute_embeddings(view)
pgvector_index.add_to_index(embeddings, sample_ids)
```
By default, the pgvector backend creates an
[HNSW index](https://github.com/pgvector/pgvector?tab=readme-ov-file#hnsw),
which offers the best query performance/recall tradeoff for most use cases.
Alternatively, you can create an
[IVFFlat index](https://github.com/pgvector/pgvector?tab=readme-ov-file#ivfflat),
which is faster to build and uses less memory, by setting the `index_type`
parameter:
```python
import fiftyone as fo
import fiftyone.brain as fob
import fiftyone.zoo as foz
dataset = foz.load_zoo_dataset("quickstart")
# Create an IVFFlat-backed similarity index
pgvector_index = fob.compute_similarity(
dataset,
model="clip-vit-base32-torch",
backend="pgvector",
brain_key="pgvector_index",
index_type="ivfflat",
ivfflat_lists=100, # number of inverted lists in the index
ivfflat_probes=10, # number of lists to search during queries
)
```
#### NOTE
IVFFlat indexes only support the `("cosine", "dotproduct", "euclidean")`
metrics. Increase `ivfflat_probes` to improve recall at the cost of query
speed.
### High-dimensional embeddings
pgvector limits both HNSW and IVFFlat indexes to at most 2,000 dimensions on
regular `vector` columns. If your embeddings exceed this limit (for example,
2048-dimensional Qwen embeddings), set `vector_type="halfvec"` to store them
as half-precision (float16) vectors, which support indexes with up to 4,000
dimensions:
```python
import numpy as np
import fiftyone as fo
import fiftyone.brain as fob
import fiftyone.zoo as foz
dataset = foz.load_zoo_dataset("quickstart")
embeddings = np.random.randn(len(dataset), 2048) # >2000 dimensions
pgvector_index = fob.compute_similarity(
dataset,
embeddings=embeddings,
backend="pgvector",
brain_key="pgvector_index",
vector_type="halfvec",
maintenance_work_mem="256MB", # high-dim index builds need more memory
)
```
#### NOTE
The `halfvec` type requires pgvector >= 0.7.0 and stores vectors at
reduced (float16) precision.