ivf_flat_index
vector_search.ivf_flat_index
IVFFlat Index implementation.
IVFFlatIndex is based on k-means
clustering and shuffling of the dataset vectors.
During ingestion, TileDB computes the k-means
clusters and shuffles the vectors into partitions. The vectors are stored grouped by partition in a 2D TileDB array allowing for partitions to be read with minimal I/O overhead.
To answer a query, the search focuses only on a small number of partitions, based on the query’s proximity to the k-means
centroids. This is specified with a parameter called nprobe
controlling how many partitions are checked for each query.
IVFFlatIndex provides a vector search implementation that can trade-off accuracy for performance.
Queries can be run in multiple modes:
- Local main memory:
- Loads the entire index in memory during initialization and uses it to answer queries.
- Local out of core:
- Avoids loading index data in memory by interleaving I/O and query execution, respecting the memory budget defined by the user.
- Distributed execution:
- Executes the queries using multiple workers in TileDB Cloud.
Classes
Name | Description |
---|---|
IVFFlatIndex | Opens an IVFFlatIndex . |
IVFFlatIndex
vector_search.ivf_flat_index.IVFFlatIndex(self, uri, config=None, timestamp=None, memory_budget=-1, open_for_remote_query_execution=False, group=None, **kwargs)
Opens an IVFFlatIndex
.
Parameters
Name | Type | Description | Default |
---|---|---|---|
uri |
str | URI of the index. | required |
config |
Optional[Mapping[str, Any]] | TileDB config dictionary. | None |
timestamp |
If int, open the index at a given timestamp. If tuple, open at the given start and end timestamps. | None |
|
memory_budget |
int | Main memory budget, in number of vectors, for query execution. If not provided, all index data are loaded in main memory. Otherwise, no index data are loaded in main memory and this memory budget is applied during queries. | -1 |
open_for_remote_query_execution |
bool | If True , do not load any index data in main memory locally, and instead load index data in the TileDB Cloud taskgraph created when a non-None driver_mode is passed to query() . We then load index data in the taskgraph based on memory_budget . If False , load index data in main memory locally according to memory_budget . Note that you can still use a taskgraph for query execution, you’ll just end up loading the data both on your local machine and in the cloud taskgraph.. |
False |
Methods
Name | Description |
---|---|
get_dimensions | Returns the dimension of the vectors in the index. |
query_internal | Queries an IVFFlatIndex . |
vacuum | The vacuuming process permanently deletes index files that are consolidated through the consolidation |
get_dimensions
vector_search.ivf_flat_index.IVFFlatIndex.get_dimensions()
Returns the dimension of the vectors in the index.
query_internal
vector_search.ivf_flat_index.IVFFlatIndex.query_internal(queries, k=10, nprobe=1, nthreads=-1, use_nuv_implementation=False, mode=None, resource_class=None, resources=None, num_partitions=-1, num_workers=-1, **kwargs)
Queries an IVFFlatIndex
.
Parameters
Name | Type | Description | Default |
---|---|---|---|
queries |
np.ndarray | 2D array of query vectors. This can be used as a batch query interface by passing multiple queries in one call. | required |
k |
int | Number of results to return per query vector. | 10 |
nprobe |
int | Number of partitions to check per query. Use this parameter to trade-off accuracy for latency and cost. As a rule of thumb, configuring nprobe to be the square root of partitions should result in accuracy close to 100%. |
1 |
nthreads |
int | Number of threads to use for local query execution. | -1 |
use_nuv_implementation |
bool | Whether to use the nuv query implementation. Default: False | False |
mode |
Optional[Mode] | If provided the query will be executed using TileDB cloud taskgraphs. For distributed execution you can use REALTIME or BATCH mode. For local execution you can use LOCAL mode. | None |
resource_class |
Optional[str] | The name of the resource class to use (“standard” or “large”). Resource classes define maximum limits for cpu and memory usage. Can only be used in REALTIME or BATCH mode. Cannot be used alongside resources. In REALTIME or BATCH mode if neither resource_class nor resources are provided, we default to the “large” resource class. | None |
resources |
Optional[Mapping[str, Any]] | A specification for the amount of resources to use when executing using TileDB cloud taskgraphs, of the form: {“cpu”: “6”, “memory”: “12Gi”, “gpu”: 1}. Can only be used in BATCH mode. Cannot be used alongside resource_class. | None |
num_partitions |
int | Only relevant for taskgraph based execution. If provided, we split the query execution in that many partitions. | -1 |
num_workers |
int | Only relevant for taskgraph based execution. If provided, this is the number of workers to use for the query execution. | -1 |
vacuum
vector_search.ivf_flat_index.IVFFlatIndex.vacuum()
The vacuuming process permanently deletes index files that are consolidated through the consolidation process. TileDB separates consolidation from vacuuming, in order to make consolidation process-safe in the presence of concurrent reads and writes.
Note:
- Vacuuming is not process-safe and you should take extra care when invoking it.
- Vacuuming may affect the granularity of the time traveling functionality.
The IVFFlat class vacuums consolidated fragment, array metadata and commits for the db
and ids
arrays.
Functions
Name | Description |
---|---|
create | Creates an empty IVFFlatIndex. |
create
vector_search.ivf_flat_index.create(uri, dimensions, vector_type, group_exists=False, config=None, storage_version=STORAGE_VERSION, distance_metric=vspy.DistanceMetric.SUM_OF_SQUARES, group=None, asset_creation_threads=None, **kwargs)
Creates an empty IVFFlatIndex.
Parameters
Name | Type | Description | Default |
---|---|---|---|
uri |
str | URI of the index. | required |
dimensions |
int | Number of dimensions for the vectors to be stored in the index. | required |
vector_type |
np.dtype | Datatype of vectors. Supported values (uint8, int8, float32). | required |
group_exists |
bool | If False it creates the TileDB group for the index. If True the method expects the TileDB group to be already created. | False |
config |
Optional[Mapping[str, Any]] | TileDB config dictionary. | None |
storage_version |
str | The TileDB vector search storage version to use. If not provided, use the latest stable storage version. | STORAGE_VERSION |
group |
tiledb.Group | TileDB group open in write mode. Internal, this is used to avoid opening the group multiple times during ingestion. | None |
asset_creation_threads |
Sequence[Thread] | List of asset creation threads to append new threads. Internal, this is used to parallelize all asset creation during ingestion. | None |