Skip to main content

Generative Search - Mistral

In short

Added in v1.24.2
  • The Generative Mistral (generative-mistral) module performs retrieval augmented generation, or RAG, based on the data stored in your Weaviate instance.
  • The module can generate a response for each object returned from Weaviate, or a combined response for a group of objects.
  • The module enables generative search operations on the Weaviate instance.
  • The default model is open-mistral-7b.
  • The module requires an API key for Mistral inference endpoints to perform the generation task.

Introduction

generative-mistral performs retrieval augmented generation, or RAG, based on the data stored in your Weaviate instance.

The module works in two steps:

  1. (Weaviate) Run a search query in Weaviate to find relevant objects.
  2. (Mistral Inference API) Use a Large Language Model to generate a response based on the results (from the previous step) and the provided prompt or task.
note

You can use the Generative Mistral module with any other upstream modules. For example, you could use text2vec-cohere, text2vec-huggingface or text2vec-openai to vectorize and query your data, but then rely on the generative-mistral module to generate a response.

The generative module can perform RAG for:

  • each returned object - singlePrompt
  • the group of all results together – groupedTask

You need to input both a query and a prompt (for individual responses) or a task (for all responses).

Weaviate instance configuration

tip

If you use Weaviate Cloud Services (WCS), this module is already enabled and pre-configured. You cannot edit the configuration in WCS.

Docker Compose file

To use generative-mistral, you must enable it in your Docker Compose file (docker-compose.yml). You can do so manually, or create one using the Weaviate configuration tool.

Parameters

  • ENABLE_MODULES (Required): The modules to enable. Include generative-mistral to enable the module.
  • MISTRAL_APIKEY Your Mistral API key. You can also provide the key at query time.

Example

This configuration enables generative-mistral and sets the Mistral authentication credentials.

---
version: '3.4'
services:
weaviate:
command:
- --host
- 0.0.0.0
- --port
- '8080'
- --scheme
- http
image: cr.weaviate.io/semitechnologies/weaviate:1.24.10
ports:
- 8080:8080
- 50051:50051
volumes:
- weaviate_data:/var/lib/weaviate
restart: on-failure:0
environment:
QUERY_DEFAULTS_LIMIT: 25
AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
MISTRAL_APIKEY: escecret-foobar # Optional. Can be set at query time.
ENABLE_MODULES: 'text2vec-cohere,generative-mistral' # Can include any modules
CLUSTER_HOSTNAME: 'node1'
volumes:
weaviate_data:
...

Class configuration

You can configure how the module will behave in each class through the Weaviate schema.

API settings

Parameters

ParameterRequiredDefaultPurpose
modelNo"open-mistral-7b"The model to use.
temperatureNo0Control of LLM stochasticity.
maxTokensNo2048Maximum number of tokens to generate.

Supported models

You can use any of the following models with generative-mistral:

  • open-mistral-7b (aka mistral-tiny-2312)
  • open-mixtral-8x7b (aka mistral-small-2312)
  • mistral-tiny
  • mistral-small
  • mistral-small-latest (aka mistral-small-2402)
  • mistral-medium
  • mistral-medium-latest (aka mistral-medium-2312)
  • mistral-large
  • mistral-large-latest (aka mistral-large-2402)

Example

The following example configures the Document class to use the generative-mistral module with the Document class, with the mistral-medium-latest model.

{
"classes": [
{
"class": "Document",
"description": "A class called document",
...,
"moduleConfig": {
"generative-mistral": {
"model": "mistral-medium-latest",
},
}
}
]
}
New to Weaviate Schemas?

If you are new to Weaviate, check out the Weaviate schema tutorial.

How to use

Query-time parameters

You can supply parameters at query time by adding them to the HTTP header.

HTTP HeaderValuePurposeNote
"X-Mistral-Api-Key"Your Mistral API key.AuthenticationLearn more

Queries

This module enables generative search queries.

generate takes the following arguments:

FieldData TypeRequiredExampleDescription
singleResult {prompt}stringnoSummarize the following in a tweet: {summary}Generates a response for each individual search result. You need to include at least one result field in the prompt, between braces.
groupedResult {task}stringnoExplain why these results are similar to each otherGenerates a single response for all search results

Example of properties in the prompt

When piping the results to the prompt, at least one field returned by the query must be added to the prompt. If you don't add any fields, Weaviate will throw an error.

For example, assume your schema looks like this:

{
Article {
title
summary
}
}

You can add both title and summary to the prompt by enclosing them in curly brackets:

{
Get {
Article {
title
summary
_additional {
generate(
singleResult: {
prompt: """
Summarize the following in a tweet:

{title} - {summary}
"""
}
) {
singleResult
error
}
}
}
}
}

Example - single result

Here is an example of a query where:

  • we get a podcast clip (with limit 1)
  • then we ask the generator module to summarize the content into one sentence.
    • the query asks for the speaker and content fields, which are then included in the prompt argument of the generate operator.
{
Get {
PodClip (limit:1){
speaker
content
_additional {
generate(
singleResult:{
prompt: """
Please summarize the following podcast clip into one sentence.
Speaker: {speaker} said {content}.
"""
}
){
singleResult
error
}
}
}
}
}

Example - grouped result

Here is an example of a query where:

  • we run a vector search (with nearText) to find podcast clips semantically similar to "What is ref2vec?"
  • then we ask the generator module to answer the question: "What is ref2vec?" based on the search results.
{
Get {
PodClip(
nearText: {
concepts: ["What is ref2vec?"]
},
limit: 5
) {
speaker
content
_additional {
generate(
groupedResult: {
task: "Please answer the question: What is ref2vec? based on the following search results. IMPORTANT! Please make sure your answer is supported by the information in the search results and provide references to the provided content when possible."
}
) {
groupedResult
error
}
}
}
}
}

Further examples

For further usage examples, please see the how-to search: generative page.

Questions and feedback

If you have any questions or feedback, please let us know on our forum. For example, you can: