Indexing
Weaviate uses two types of indexes to facilitate fast, scalable search and filtering. It uses vector indexes for vector searches and inverted indexes for filtering and keyword searches.
This page introduces the vector indexes and inverted indexes available in Weaviate, so you can arrive at an informed indexing strategy.
Vector indexes can use hot or warm resources, depending on the index type. Inverted indexes use warm resources. For more on resource types, see Managing resources.
- For multi-tenant configurations, use a dynamic index.
- Consider the size of your data set when choosing an index type:
- If your object count is low, consider a flat index.
- For larger data sets, use an HNSW index.
- If you are unsure, use a dynamic index.
- If you can trade off some accuracy, consider using compression:
- For HNSW indexes, PQ/SQ/BQ can reduce memory requirements.
- For flat indexes, binary quantization (BQ) can improve search speeds.
- Consider your query needs when configuring inverted indexes:
- If disk space is not a concern, index all properties for searching & filtering.
- If metadata filtering is important, index it at the collection level.
Vector indexes
Weaviate offers three types of vector indexes, Hierarchical Navigable Small World (HNSW) indexes, flat indexes and dynamic indexes.
- HNSW indexes enable fast, scalable vector searching that works well even with very large data sets.
- Flat indexes are memory-efficient indexes that work best with small data sets.
- Dynamic indexes switch from a flat index to an HNSW index when a collection or tenant reaches a threshold size.
HNSW indexes
HNSW are high-performance, in-memory indexes. HNSW indexes scale well, meaning that vector searches remain fast even for very large data sets.
HNSW indexes achieve this by building a multi-layered graph of objects, allowing for fast, approximate nearest neighbor searches.
While HNSW indexes enable fast searches, they use a lot of hot resources, as they load the graph structure and vectors into memory.
Consider using compression to reduce the size of for your HNSW indexes. Weaviate offers several ways to compress your data:
Product Quantization (PQ) PQ reduces the size of the vector embedding in two ways. PQ trains on your data to create custom segments. PQ creates segments to reduce the number of dimensions, and segments are stored as 8 bit integers instead of 32 bit floats. Compared to dimensions, there are fewer segments and each segment is much smaller than a single dimension.
The PQ compression algorithm is configurable. You control the number of segments, segment granularity, and the size of the training set.
Scalar Quantization (SQ) SQ reduces the size of each vector dimension from 32 bits to 8 bits. SQ trains on your data to create custom buckets for each dimension. This training helps SQ to preserve data characteristics when it maps information from the 32 bit dimensions into 8 bit buckets.
Binary Quantization (BQ) BQ reduces the size of each vector dimension to a single bit. This compression algorithm works best for vectors with high dimensionality.
Flat indexes
Flat indexes are memory-efficient. They are disk based indexes that perform brute-force vector searches. These searches are fast for small data sets, but the search speed increases linearly as the number of indexed objects grows.
Flat indexes are best suited for cases where the number of objects is low and will not grow significantly.
Binary quantization (BQ) can improve flat indexes' search speeds. BQ improves search time by reducing the amount of data to read, and speeding up time taken to calculate the distance between vectors.
Dynamic indexes
v1.25ASYNC_INDEXINGDynamic indexes require asynchronous indexing. To enable asynchronous indexing in a self-hosted Weaviate instance, set the ASYNC_INDEXING environment variable to true. If your instance is hosted in Weaviate Cloud, use the Weaviate Cloud console to enable asynchronous indexing.
Dynamic indexes offer a flexible approach to indexing. A dynamic index starts as a flat index and converts automatically to an HNSW index when the object count reaches a threshold value.
In multi-tenant configurations where different tenants have different object counts, dynamic indexes are a good index choice. Collections with dynamic indexes have less overhead since tenants can use flat indexes when the HNSW index isn't needed.
The default index conversion threshold is 10,000 objects. You can configure the threshold value when you create the dynamic index.
This table shows how a dynamic index changes as the number of objects in a collection grows. The example configuration is for a dynamic index with the following properties:
- A conversion threshold of 10,000 objects.
- Flat index with BQ configured.
- HNSW index with SQ configured
- The training threshold for SQ is 100,000 objects.
| Number of objects | Index type | Compression | Notes |
|---|---|---|---|
| 0 - 9,999 | Flat index | BQ | Flat index and BQ are active. |
| 10,000 | Flat -> HNSW | None | The index converts to HNSW. The index is stored in RAM. |
| 100,000 | HNSW | Training | The collection's object count reaches the SQ training threshold. |
| 100,001 | HNSW | SQ | HNSW and SQ are active. |
Configure the flat index and the HNSW index when you define the dynamic index. The dynamic index uses the flat index initially, then switches to the HNSW index. Both indexes should be configured before they are used.
Asynchronous vector indexing
v1.22Building an HNSW index can be resource-intensive and slow down the time it takes to import objects into Weaviate.
Asynchronous vector indexing allows you to import objects into Weaviate without waiting for the HNSW index to be built.
This allows decoupling of data ingestion from vector index creation.
With asynchronous vector indexing, you can import objects into Weaviate without waiting for the HNSW index to finish building. You should be aware that vector searches during this time may be based on an incomplete index, which can affect search results.
Inverted indexes
Weaviate uses inverted indexes for keyword searches as well as filtering.
There are multiple types of inverted indexes, including:
indexSearchable: for keyword searchesindexFilterable: for faster filteringindexRangeFilters: for faster filtering on numerical ranges
Aspects of these indexes can be configured at the collection and property level.
Collection level configuration
The inverted index is configurable on a collection level. The collection level settings determine BM25 parameters, and what metadata is indexed for filtering. For example, you can configure whether timestamps, null state, or property lengths are indexed.
For details, see set inverted index parameters
Property level configuration
Property level configuration is more limited. Individual indexes can be turned on or off at the property level, and keyword tokenization options can be set.
indexSearchable determines whether a property is indexed for keyword searches. indexFilterable determines whether a property is indexed to speed up match-based filtering. indexRangeFilters determines whether a property is indexed for numerical range filters.
For more on filters, see Filtering.
Related pages
For more information, see these documentation pages and blog posts.
Documentation pages
To configure indexing, follow the steps on these pages:
For more documentation details, see:
Weaviate academy
For a short course on vector indexes, see:
Questions and feedback
If you have any questions or feedback, let us know in the user forum.