Weaviate is completely modularized. The Core of Weaviate, without any modules attached, is a pure vector-native database and search engine. Data is stored as vectors, and these vectors are searchable by the provide vector index algorithm. Without any modules attached, Weaviate does not know how to vectorize data, i.e. how to calculate the vectors from a data item. Depending on the type of data you want to store and search (text, images, etc), and depending on the use case (like search, question answering, etc, depending on language, classification, ML model, training set, etc), you can choose and attach a module that best fits your use case.
Modules can be “vectorizers” (defines how the numbers in the vectors are chosen from the data) or other modules providing additional functions like question answering, custom classification, etc. Modules have the following characteristics:
- Naming convention:
<media>2vec-<name>-<optional>, for example
- Other modules:
- A module name must be url-safe, meaning it must not contain any characters which would require url-encoding.
- A module name is not case-sensitive.
text2vec-bertwould be the same module as
- Module information is accessible through the
- General module information (which modules are attached, version, etc.) is accessible through Weaviate’s
- Modules can add
additionalproperties in the RESTful API and
_additionalproperties in the GraphQL API.
- A module can add filters in GraphQL queries.
- Which vectorizer and other modules are applied to which data classes is configured in the schema.
One vectorizer that is provided is
text2vec-contextionary is a text vectorizer that gives context to the textual data using a language model trained using fasttext on Wiki data and CommonCrawl. More information can be found here.
Weaviate also supports any transformers model as vectorization module. Check
text2vec-transformers for the details
Note: at the moment, text vectorization modules cannot be combined in a single setup. This means that you can either enable the
text2vec-transformers or no text vectorization module.
Custom modules will soon be supported, more information can be found here. Stay tuned!
If you can’t find the answer to your question here, please look at the: