ivf_pq_index

vector_search.ivf_pq_index

IVFPQ Index implementation.

Classes

Name	Description
IVFPQIndex	Opens a `IVFPQIndex`.

IVFPQIndex

vector_search.ivf_pq_index.IVFPQIndex(self, uri, config=None, timestamp=None, memory_budget=-1, preload_k_factor_vectors=False, open_for_remote_query_execution=False, group=None, **kwargs)

Opens a IVFPQIndex.

Parameters

Name	Type	Description	Default
`uri`	str	URI of the index.	required
`config`	Optional[Mapping[str, Any]]	TileDB config dictionary.	`None`
`timestamp`		If int, open the index at a given timestamp. If tuple, open at the given start and end timestamps.	`None`
`memory_budget`	int	Main memory budget, in number of vectors, for query execution. If not provided, all index data are loaded in main memory. Otherwise, no index data are loaded in main memory and this memory budget is applied during queries.	`-1`
`preload_k_factor_vectors`	bool	When using `k_factor` in a query, we first query for `k_factor * k` pq-encoded vectors, and then do a re-ranking step using the original input vectors for the top `k` vectors. If `True`, we will load all the input vectors in main memory. This can only be used with `memory_budget` set to `-1`, and is useful when the input vectors are small enough to fit in memory and you want to speed up re-ranking.	`False`
`open_for_remote_query_execution`	bool	If `True`, do not load any index data in main memory locally, and instead load index data in the TileDB Cloud taskgraph created when a non-`None` `driver_mode` is passed to `query()`. If `False`, load index data in main memory locally. Note that you can still use a taskgraph for query execution, you’ll just end up loading the data both on your local machine and in the cloud taskgraph.	`False`

Methods

Name	Description
get_dimensions	Returns the dimension of the vectors in the index.
query_internal	Queries a `IVFPQIndex`.

get_dimensions

vector_search.ivf_pq_index.IVFPQIndex.get_dimensions()

Returns the dimension of the vectors in the index.

query_internal

vector_search.ivf_pq_index.IVFPQIndex.query_internal(queries, k=10, k_factor=1.0, nprobe=100, **kwargs)

Queries a IVFPQIndex.

Parameters

Name	Type	Description	Default
`queries`	np.ndarray	2D array of query vectors. This can be used as a batch query interface by passing multiple queries in one call.	required
`k`	int	Number of results to return per query vector.	`10`
`k_factor`	float	To improve accuracy, IVF_PQ can search for more vectors than requested and then perform re-ranking using the original non-PQ-encoded vectors. This can be slightly slower, but is more accurate. k_factor is the factor by which to increase the number of vectors searched. 1 means we search for exactly `k` vectors. 10 means we search for `10*k` vectors. Defaults to 1.	`1.0`
`nprobe`	Optional[int]	Number of partitions to check per query. Use this parameter to trade-off accuracy for latency and cost.	`100`

Functions

Name	Description
create	Creates an empty IVFPQIndex.

create

vector_search.ivf_pq_index.create(uri, dimensions, vector_type, num_subspaces, config=None, storage_version=STORAGE_VERSION, distance_metric=vspy.DistanceMetric.SUM_OF_SQUARES, **kwargs)

Creates an empty IVFPQIndex.

Parameters

Name	Type	Description	Default
`uri`	str	URI of the index.	required
`dimensions`	int	Number of dimensions for the vectors to be stored in the index.	required
`vector_type`	np.dtype	Datatype of vectors. Supported values (uint8, int8, float32).	required
`num_subspaces`	int	Number of subspaces to use in the PQ encoding. We will divide the dimensions into num_subspaces parts, and PQ encode each part separately. This means dimensions must be divisible by num_subspaces.	required
`config`	Optional[Mapping[str, Any]]	TileDB config dictionary.	`None`
`storage_version`	str	The TileDB vector search storage version to use. If not provided, use the latest stable storage version.	`STORAGE_VERSION`