ivf_flat_index

vector_search.ivf_flat_index

IVFFlat Index implementation.

IVFFlatIndex is based on k-means clustering and shuffling of the dataset vectors.

During ingestion, TileDB computes the k-means clusters and shuffles the vectors into partitions. The vectors are stored grouped by partition in a 2D TileDB array allowing for partitions to be read with minimal I/O overhead.

To answer a query, the search focuses only on a small number of partitions, based on the query’s proximity to the k-means centroids. This is specified with a parameter called nprobe controlling how many partitions are checked for each query.

IVFFlatIndex provides a vector search implementation that can trade-off accuracy for performance.

Queries can be run in multiple modes:

  • Local main memory:
    • Loads the entire index in memory during initialization and uses it to answer queries.
  • Local out of core:
    • Avoids loading index data in memory by interleaving I/O and query execution, respecting the memory budget defined by the user.
  • Distributed execution:
    • Executes the queries using multiple workers in TileDB Cloud.

Classes

Name Description
IVFFlatIndex Opens an IVFFlatIndex.

IVFFlatIndex

vector_search.ivf_flat_index.IVFFlatIndex(self, uri, config=None, timestamp=None, memory_budget=-1, open_for_remote_query_execution=False, group=None, **kwargs)

Opens an IVFFlatIndex.

Parameters

Name Type Description Default
uri str URI of the index. required
config Optional[Mapping[str, Any]] TileDB config dictionary. None
timestamp If int, open the index at a given timestamp. If tuple, open at the given start and end timestamps. None
memory_budget int Main memory budget, in number of vectors, for query execution. If not provided, all index data are loaded in main memory. Otherwise, no index data are loaded in main memory and this memory budget is applied during queries. -1
open_for_remote_query_execution bool If True, do not load any index data in main memory locally, and instead load index data in the TileDB Cloud taskgraph created when a non-None driver_mode is passed to query(). We then load index data in the taskgraph based on memory_budget. If False, load index data in main memory locally according to memory_budget. Note that you can still use a taskgraph for query execution, you’ll just end up loading the data both on your local machine and in the cloud taskgraph.. False

Methods

Name Description
get_dimensions Returns the dimension of the vectors in the index.
query_internal Queries an IVFFlatIndex.
vacuum The vacuuming process permanently deletes index files that are consolidated through the consolidation
get_dimensions

vector_search.ivf_flat_index.IVFFlatIndex.get_dimensions()

Returns the dimension of the vectors in the index.

query_internal

vector_search.ivf_flat_index.IVFFlatIndex.query_internal(queries, k=10, nprobe=1, nthreads=-1, use_nuv_implementation=False, mode=None, resource_class=None, resources=None, num_partitions=-1, num_workers=-1, **kwargs)

Queries an IVFFlatIndex.

Parameters
Name Type Description Default
queries np.ndarray 2D array of query vectors. This can be used as a batch query interface by passing multiple queries in one call. required
k int Number of results to return per query vector. 10
nprobe int Number of partitions to check per query. Use this parameter to trade-off accuracy for latency and cost. As a rule of thumb, configuring nprobe to be the square root of partitions should result in accuracy close to 100%. 1
nthreads int Number of threads to use for local query execution. -1
use_nuv_implementation bool Whether to use the nuv query implementation. Default: False False
mode Optional[Mode] If provided the query will be executed using TileDB cloud taskgraphs. For distributed execution you can use REALTIME or BATCH mode. For local execution you can use LOCAL mode. None
resource_class Optional[str] The name of the resource class to use (“standard” or “large”). Resource classes define maximum limits for cpu and memory usage. Can only be used in REALTIME or BATCH mode. Cannot be used alongside resources. In REALTIME or BATCH mode if neither resource_class nor resources are provided, we default to the “large” resource class. None
resources Optional[Mapping[str, Any]] A specification for the amount of resources to use when executing using TileDB cloud taskgraphs, of the form: {“cpu”: “6”, “memory”: “12Gi”, “gpu”: 1}. Can only be used in BATCH mode. Cannot be used alongside resource_class. None
num_partitions int Only relevant for taskgraph based execution. If provided, we split the query execution in that many partitions. -1
num_workers int Only relevant for taskgraph based execution. If provided, this is the number of workers to use for the query execution. -1
vacuum

vector_search.ivf_flat_index.IVFFlatIndex.vacuum()

The vacuuming process permanently deletes index files that are consolidated through the consolidation process. TileDB separates consolidation from vacuuming, in order to make consolidation process-safe in the presence of concurrent reads and writes.

Note:

  1. Vacuuming is not process-safe and you should take extra care when invoking it.
  2. Vacuuming may affect the granularity of the time traveling functionality.

The IVFFlat class vacuums consolidated fragment, array metadata and commits for the db and ids arrays.

Functions

Name Description
create Creates an empty IVFFlatIndex.

create

vector_search.ivf_flat_index.create(uri, dimensions, vector_type, group_exists=False, config=None, storage_version=STORAGE_VERSION, distance_metric=vspy.DistanceMetric.SUM_OF_SQUARES, group=None, asset_creation_threads=None, **kwargs)

Creates an empty IVFFlatIndex.

Parameters

Name Type Description Default
uri str URI of the index. required
dimensions int Number of dimensions for the vectors to be stored in the index. required
vector_type np.dtype Datatype of vectors. Supported values (uint8, int8, float32). required
group_exists bool If False it creates the TileDB group for the index. If True the method expects the TileDB group to be already created. False
config Optional[Mapping[str, Any]] TileDB config dictionary. None
storage_version str The TileDB vector search storage version to use. If not provided, use the latest stable storage version. STORAGE_VERSION
group tiledb.Group TileDB group open in write mode. Internal, this is used to avoid opening the group multiple times during ingestion. None
asset_creation_threads Sequence[Thread] List of asset creation threads to append new threads. Internal, this is used to parallelize all asset creation during ingestion. None