Weaviate 1.24 is here!
Here are the release ⭐️highlights⭐️!
- Named vectors. A single object can have multiple vectors. Create vectors for properties, use different vectorization models, and apply different metrics to fine tune interactions with your data.
- HNSW and binary quantization (BQ) HNSW indexes and BQ combine for serious compression and blazing speed.
- Simplified Docker configuration. A new Docker image that needs no configuration.
- Backend improvements. Numerous improvements to make updates, imports, and deletions faster.
- Python client update. General availability, support for all the latest features.
Named vectors
Named vectors make your collections richer and more versatile.
Starting in 1.24, collections can have multiple named vectors. Each vector is independent. Each vector has its own index, its own compression, and its own vectorizer. This means you can create vectors for properties, use different vectorization models, and apply different metrics to fine tune interactions with your data.
In earlier versions, objects in your collections are limited to one vector. Now, you can vectorize meta data like titles and descriptions so that it is available for vector search. You can group things that are logically related, like a music video and a lyric sheet, in the same object. You don't have to create cross-references or manage shared metadata. A single object in a collection can have multiple named vectors.
You don't have to use multiple vectors in your collections, but if you do, adjust your queries to specify which vector you want to use.
For details, see Multiple vectors.
HNSW and binary quantization
Weaviate improves binary quantization (BQ) in 1.24 to be faster, more memory efficient, and more cost-effective. Use BQ vector compression with HNSW indexes to dramatically improve your query speed.
BQ compresses vector representations while preserving essential information. Uncompressed, Weaviate uses a float32 to store each dimension. BQ uses one bit per dimension to encode Vector directionality. This means BQ compresses vectors from 32 bits per dimension to 1 bit per dimension - a savings of 32 times the space. This compression significantly reduces storage requirements.
Comparing BQ compressed vectors is fast. To calculate the distance between to vectors, compare their dimensional agreement. The bitwise comparison ensures optimal resource utilization without compromising accuracy. For instance, consider two vectors encoded as [1,1,0,1,0,1,0,1] and [1,1,0,0,1,1,0,1]. Their distance is 2.
Bitwise comparison details
Compare each value in the first vector one with the corresponding value in the second vector. If they are the same, score 0. If they are different, score 1. The distance between the vectors is the sum of the bitwise comparisons.
[1,1,0,1,0,1,0,1]
[1,1,0,0,1,1,0,1]
-----------------
0+0+0+1+1+0+0+0 = 2
Efficient BQ processing compliments our HNSW index to enable mind dizzyingly fast vector search and throughput!
Let’s talk numbers! You might know from our v1.23 release that combining a flat index with BQ enables real-time search capabilities. Even so, a brute force a search on 1 million vectors, where each vector has 768 dimensions, takes about 23 milliseconds with a flat index. When you use HNSW and BQ, even with vectors six times as big (4608 dimensions), the same brute force search only takes about 90 milliseconds.
A flat index combined with BQ is fast, but compare a HNSW index combined with BQ. Together HNSW and BQ enable vector search capabilities that reach nearly 10,000 queries per second at 85% recall!
Keep an eye out for an upcoming blog post with the nitty-gritty details of how HNSW+BQ compares to other index and compression algorithms.
Simplified Docker configuration
Starting in v1.24, there is a new Docker image that runs using common, default values. This image lets you start development and exploration quickly. Just enter a simple command line:
docker run -p 8080:8080 -p 50051:50051 cr.weaviate.io/semitechnologies/weaviate:1.24.1
Alternatively, you can edit the docker-compose.yml file to customize your instance.
For details, see Docker configuration
Backend improvements
Weaviate is a robust platform that handles many kinds of workloads. This release brings major improvements for imports, updates, and deletions. You don't have to do anything to enable these fixes, but you may notice improved upload times, especially if you make frequent changes to your collections.
Python client
The new Weaviate Python client v4 is available. The latest release adds support for named vectors and brings numerous other updates. If you have feedback on the new client, please let us know.
- To learn more about the new client, read the client release blog post.
- To learn how to upgrade your code, see the v3 to v4 migration guide.
Summary
Enjoy the new features and improvements in Weaviate 1.24. This release is available as a docker image and on WCS. Try it out in a free, WCS sandbox, or download a copy and try it out locally.
Thanks for reading, see you next time 👋!
Ready to start building?
Check out the Quickstart tutorial, and begin building amazing apps with the free trial of Weaviate Cloud Services (WCS).