Skip to main content

· 17 min read
Connor Shorten
Erika Cardenas
Laura Ham
Sebastian Witalec

Weaviate 1.15 release

We are happy to announce the release of Weaviate 1.15, which is packed with great features, significant performance improvements, new distance metrics and modules, and many smaller improvements and fixes.

The brief

If you like your content brief and to the point, here is the TL;DR of this release:

  1. ☁️Cloud-native backups - allows you to configure your environment to create backups - of selected classes or the whole database - straight into AWS S3, GCS or local filesystem
  2. Reduced memory usage - we found new ways to optimize memory usage, reducing RAM usage by 10-30%.
  3. Better control over Garbage Collector - with the introduction of GOMEMLIMIT we gained more control over the garbage collector, which significantly reduced the chances of OOM kills for your Weaviate setups.
  4. Faster imports for ordered data - by extending the Binary Search Tree structure with a self-balancing Red-black tree, we were able to speed up imports from O(n) to O(log n)
  5. More efficient filtered aggregations - thanks to optimization to a library reading binary data, filtered aggregations are now 10-20 faster and require a lot less memory.
  6. Two new distance metrics - with the addition of Hamming and Manhattan distance metrics, you can choose the metric (or a combination of) to best suit your data and use case.
  7. Two new Weaviate modules - with the Summarization module, you can summarize any text on the fly, while with the HuggingFace module, you can use compatible transformers from the HuggingFace
  8. Other improvements and bug fixes - it goes without saying that with every Weaviate release, we strive to make Weaviate more stable - through bug fixes - and more efficient - through many optimizations.

· 18 min read

GOMEMLIMIT Gopher

Intro

Running out of memory is never fun, but it's incredibly frustrating when you've already taken some precautions and calculated your exact memory needs. "My application requires 4GB of memory. How is it possible I ran out of memory on my 6GB machine!?". As it turns out, this was a real possibility in a garbage collected ("GC") language like Golang. The emphasis is on the word "was" because Go 1.19 changes everything: The new GOMEMLIMIT feature can help you both increase GC-related performance as well as avoid GC-related out-of-memory ("OOM") situations.

· 6 min read
Laura Ham

Using Cross-Encoders as reranker in multistage vector search

Semantic search overcomes the shortcomings of keyword-based search. If you search for "programming for data scientists", a keyword search will find "An intro to programming for data scientists", but it won't return "Python for Machine Learning use-cases".

· 5 min read

The AI-First Database Ecosystem

A new ecosystem of smaller companies is ushering in a "third wave" of AI-first database technology. New search engines and databases brilliantly answer queries posed in natural language, but their machine-learning models are not limited to text searches. The same approach can also be used to search anything from images to DNA.

· 8 min read

Wikipedia and Weaviate

To conduct semantic search queries on a large scale, one needs a vector search engine to search through the large number of vector representations that represent the data. To show you how this can be done, we have open-sourced the complete English language Wikipedia corpus backup in Weaviate. In this article, I will outline how we've created the dataset, show you how you can run the dataset yourself, and present search strategies on how to implement similar vector and semantic search solutions in your own projects and how to bring them to production.

The Wikipedia dataset used is the "truthy" version of October 9th, 2021. After processing it contains 11.348.257 articles, 27.377.159 paragraphs, and 125.447.595 graph cross-references. Although a bigger machine (see below) is needed for importing the data, the serving is done on a 12 CPU, 100 GB RAM, 250Gb SSD Google Cloud VM with 1 x NVIDIA Tesla P4. The ML-models used are multi-qa-MiniLM-L6-cos-v1 and bert-large-uncased-whole-word-masking-finetuned-squad both are available as pre-built modules in Weaviate.

· 6 min read

Docker and Containers with Weaviate

Weaviate attracts different users with various backgrounds. Some have been working with containers for years, but we understand that not everyone has. Inspired by a few recent questions and comments about Docker on the Weaviate Slack, I've set out to write an article to provide a better background on Docker and containers in general. After reading this article, your most common questions about these technologies should be answered and there should be nothing in the way of building amazing use cases with Weaviate.

· 2 min read

Weaviate 1.2 release - transformer models

In the v1.0 release of Weaviate (docsGithub) we introduced the concept of modules. Weaviate modules are used to extend the vector search engine with vectorizers or functionality that can be used to query your dataset. With the release of Weaviate v1.2, we have introduced the use of transformers (DistilBERT, BERT, RoBERTa, Sentence-BERT, etc) to vectorize and semantically search through your data.