Generative searches

Overview

The core concept behind generative search is that instead of just simply fetching data from the database, Weaviate can transform the data before delivering it to you.

This ability to transform data makes generative search a powerful tool that can transform your relationship with data. Instead of a database simply being a store of information, it can work with your data to deliver outputs based on the data and a set of instructions.

Configuration

Enable modules

To use generative search, a generative-xxx module must be enabled in the Weaviate instance.

If you are using WCD, generative modules are enabled by default (see docs. Otherwise, you must configure your Weaviate instance to make sure that a generative module is enabled.

Configure classes

If only one generative module is enabled for the Weaviate instance, Weaviate will automatically use that module for all generative tasks.

On the other hand, if multiple generative modules are configured, you must define for each class which generative model to use, such as shown below.

{
  "classes": [
    {
      "class": "JeopardyQuestion",
      "moduleConfig": {
        "generative-openai": {
          "model": "gpt-3.5-turbo"
        }
      }
    }
  ]
}

About generative searches

How it works

A generative search can be best thought of as two steps that are conveniently combined to one process. The two steps are to:

1. Perform a search; and then
2. Call a generative model using the search results and a user-provided prompt.

All generated outputs are then added to the search results before being returned to the user.

There are two different types of generative searches available, which are single prompt and grouped task.

• A single prompt search generates a response for each result in the results set, using each corresponding result with the user-provided prompt.
• A grouped task search generates one response for the whole result set, by using the entire results set with the user-provided prompt.

Generative search syntax

A generative search adds a step to the search process. Accordingly, the syntax for a generative search requires specifying the prompt type (single prompt or grouped task) as well as a search query.

Single prompt

To carry out a single prompt search, you must provide a prompt that contains at least one object property. The provided properties will be populated by Weaviate based on the search results.

A single prompt search can be carried out as follows:

Python

response = (
    client.query
    .get("Article", ["title", "summary"])
    .with_near_text({"concepts": ["housing prices"]})
    .with_limit(3)
    .with_generate(
        single_prompt="Provide a two-bullet point summary of the article, whose title is {title} and body {summary}"
    )
    .do()
)

print(json.dumps(response, indent=2))

  API docs

The above query searches for top 3 Article objects using a nearText similarity search. Then, Weaviate sends each result to a generative model with the provided prompt, whose outputs are returned to the user.

A single prompt search will generate an output for each object found in search. Accordingly, each object in the response will include the generated output in the _additional properties set.

See the JSON response

{
  "data": {
    "Get": {
      "Article": [
        {
          "_additional": {
            "generate": {
              "error": null,
              "singleResult": "- Real house prices have risen by 5% on average in the latest 12-month period among the 25 countries tracked by The Economist, the quickest in over a decade.\n- House prices in suburban locations are rising faster than in cities due to the expectation that commuting may no longer be daily, reversing a decade-long trend."
            }
          },
          "summary": "As restrictions have eased, house prices have started to go through the roof. Among the 25 countries that The Economist tracks, real house prices rose by 5% on average in the latest 12-month period, the quickest in over a decade. The expectation that commuting may no longer be daily has caused house prices in suburban locations to rise faster than in cities\u2014reversing a decade-long trend. One reason is that house prices do not look as if they threaten financial stability. If house prices in America fell by one-quarter, its 33 biggest banks would still have 50% more capital than they had going into the crisis of 2007-09.",
          "title": "House prices are going ballistic"
        },
        {
          "_additional": {
            "generate": {
              "error": null,
              "singleResult": "- Politicians are taking action to address housing shortages, and consultants in Auckland have detected a genuine interest in boosting housing supply.\n- Evidence suggests that autocratic planning systems may be more effective at increasing housing supply, as seen in Switzerland where house prices have risen less than in any other rich country."
            }
          },
          "summary": "Consultants to the government in Auckland detect a genuine interest in boosting housing supply. The part of the country with the most elastic housing supply, Pine Bluff, a midsized city in Arkansas, has an average house price of $90,000. Some evidence seems to back up the view that economists\u2019 obsession with housing supply is misguided. Autocratic planning systems do a better job of boosting housing supply. In the past century Swiss house prices have risen by less than those in any other rich country.",
          "title": "Supply - Politicians are finally doing something about housing shortages"
        },
        {
          "_additional": {
            "generate": {
              "error": null,
              "singleResult": "- American house prices rose by 11% in the year to January, the fastest pace for 15 years, and real house prices have risen by an average of 5% across the 25 countries tracked by The Economist.\n- House prices outside Germany's seven biggest cities rose by 11% last year, compared with 6% within them, while house prices in central London and Sydney rose by just 4% and 3% last year, respectively, and those in Manhattan fell by 4%."
            }
          },
          "summary": "American house prices rose by 11% in the year to January, the fastest pace for 15 years. Across the 25 countries tracked by The Economist, real house prices have risen by an average of 5% in the latest 12-month period. At first glance, the robustness of house prices in the face of the economic turmoil inflicted by covid-19 might seem baffling: property prices typically move in tandem with the economy. House prices outside Germany\u2019s seven biggest cities rose by 11% last year, compared with 6% within them. By contrast, house prices in central London and Sydney rose by just 4% and 3% last year, respectively; those in Manhattan fell by 4%.",
          "title": "House prices in the rich world are booming"
        }
      ]
    }
  }
}

  API docs

Grouped Task

A grouped task search does not require any properties to be specified. It can be carried out as follows:

Python

response = (
    client.query
    .get("Article", ["title", "summary"])
    .with_near_text({"concepts": ["housing prices"]})
    .with_limit(3)
    .with_generate(
        grouped_task="Provide any common threads between these articles, if any"
    )
    .do()
)

print(json.dumps(response, indent=2))

  API docs

The above query searches for top 3 Article objects using a nearText similarity search, just as we did above. In this case, however, the search results are concatenated and sent to the generative model along with the user-provided prompt.

In other words, each grouped task search will invoke one generative output per task.

Thus, a grouped task search will generate only one output for the entire task. The generated output is returned as a part of the first object in the _additional properties set.

See the JSON response

{
  "data": {
    "Get": {
      "Article": [
        {
          "_additional": {
            "generate": {
              "error": null,
              "groupedResult": "All three articles discuss the recent rise in house prices in various countries, with a focus on the impact of the COVID-19 pandemic on the housing market. The articles also touch on the factors driving the increase in prices, such as changes in commuting patterns and supply shortages. Additionally, the articles mention the potential risks and concerns associated with the rapid rise in house prices, including the threat to financial stability and the impact on affordability for buyers."
            }
          },
          "summary": "As restrictions have eased, house prices have started to go through the roof. Among the 25 countries that The Economist tracks, real house prices rose by 5% on average in the latest 12-month period, the quickest in over a decade. The expectation that commuting may no longer be daily has caused house prices in suburban locations to rise faster than in cities\u2014reversing a decade-long trend. One reason is that house prices do not look as if they threaten financial stability. If house prices in America fell by one-quarter, its 33 biggest banks would still have 50% more capital than they had going into the crisis of 2007-09.",
          "title": "House prices are going ballistic"
        },
        {
          "_additional": {
            "generate": null
          },
          "summary": "Consultants to the government in Auckland detect a genuine interest in boosting housing supply. The part of the country with the most elastic housing supply, Pine Bluff, a midsized city in Arkansas, has an average house price of $90,000. Some evidence seems to back up the view that economists\u2019 obsession with housing supply is misguided. Autocratic planning systems do a better job of boosting housing supply. In the past century Swiss house prices have risen by less than those in any other rich country.",
          "title": "Supply - Politicians are finally doing something about housing shortages"
        },
        {
          "_additional": {
            "generate": null
          },
          "summary": "American house prices rose by 11% in the year to January, the fastest pace for 15 years. Across the 25 countries tracked by The Economist, real house prices have risen by an average of 5% in the latest 12-month period. At first glance, the robustness of house prices in the face of the economic turmoil inflicted by covid-19 might seem baffling: property prices typically move in tandem with the economy. House prices outside Germany\u2019s seven biggest cities rose by 11% last year, compared with 6% within them. By contrast, house prices in central London and Sydney rose by just 4% and 3% last year, respectively; those in Manhattan fell by 4%.",
          "title": "House prices in the rich world are booming"
        }
      ]
    }
  }
}

  API docs

Object properties

You already saw that single prompt generative searches require properties to be specified. You can also specify specific properties to be used with grouped task searches.

Why specify properties?

In single prompt examples, each property serves to form the model prompt, replacing placeholder text like ({summary}) with the retrieve summary text.

You can also specify the properties to be used for each grouped task, so that the generative model only receives the data you want to pass to it.

Context window length

Another reason for specifying the properties is to reduce the chance of exceeding the model context length. Generative models are typically transformer-based, and many have a limited context window.

As a result, only passing the required properties to the generative module may allow you to include results from more objects without exceeding the context window limit.

How to specify properties

Generative search properties unrelated to returned properties

In both single prompt and grouped task searches, the properties to be used in the generative search do not need to be specified in the properties to be returned.

You saw earlier that for single prompt searches, you can specify in the prompt the specific properties to be used.

For grouped task searches, the properties to be used must be passed as an additional parameter. Take a look at the following example, where we ask the generative model to simply repeat the provided prompt:

Python

response = (
    client.query
    .get("Article", ["title"])
    .with_near_text({"concepts": ["housing prices"]})
    .with_limit(3)
    .with_generate(
        grouped_task="Repeat the provided prompt, exactly",
        grouped_properties=["title"]
    )
    .do()
)

print(json.dumps(response, indent=2))

  API docs

See the JSON response

{
  "data": {
    "Get": {
      "Article": [
        {
          "_additional": {
            "generate": {
              "error": null,
              "groupedResult": "[{\"title\":\"House prices are going ballistic\"},{\"title\":\"Supply - Politicians are finally doing something about housing shortages\"},{\"title\":\"House prices in the rich world are booming\"}]"
            }
          },
          "title": "House prices are going ballistic"
        },
        {
          "_additional": {
            "generate": null
          },
          "title": "Supply - Politicians are finally doing something about housing shortages"
        },
        {
          "_additional": {
            "generate": null
          },
          "title": "House prices in the rich world are booming"
        }
      ]
    }
  }
}

  API docs

Generative search parameters

Optional model parameters

Generative modules use what are called "large language models", or LLMs, to produce these outputs from search results and prompts.

Weaviate allows you to tune the behavior of these models through additional, optional parameters made available by their providers.

While we cannot cover every single parameter, some groups of commonly available parameters are described below.

Parameter names will vary

Each model will use different parameter names, so you should consult the exact module documentation and the corresponding model documentation.

• model: Determines the actual language model to be used.
• temperature, k and p: Determine how "random" the model will behave. At one extreme, the model will behave deterministically, whereas at the other extreme, it will produce more unpredictable (potentially incoherent) outputs. Some providers such as OpenAI recommend using only one of these settings.
• max_tokens: Determines the maximum length of a generated output. The longer the value, the more likely you are to potentially exceed the context limit in combination with the input length.
• xxx_penalty: Determines how much to penalise certain aspects such as the same tokens appearing again, or the number of times that the same tokens appear.

This example specifies various parameters for the generative-openai module:

{
  "classes": [
    {
      "class": "JeopardyQuestion",
      "moduleConfig": {
        "generative-openai": {
          "model": "gpt-3.5-turbo",
          "temperatureProperty": 0.3,
          "maxTokensProperty": 512,
          "frequencyPenaltyProperty": 0.1,
          "presencePenaltyProperty": -0.1
        }
      }
    }
  ]
}

As a starting point, we recommend that you try using the default options, including the model, if possible. Then if something is not working to your satisfaction, you could try a different model parameter or approach.

Review

Key takeaways

• Generative search transforms data before delivery, turning a database into a more active participant in data processing.
• To use generative search, a generative module must be enabled in the Weaviate instance; generative modules are enabled by default in WCD.
• A generative search involves performing a search and then calling a generative model using the search results and a user-provided prompt.
• Single prompt search generates a response for each result, while grouped task search generates one response for the whole result set.
• Object properties are used in generative searches; they form the model prompt in single prompt examples and can be specified in grouped tasks.
• Generative modules use large language models (LLMs), and Weaviate exposes optional parameters for tuning their behavior.

Questions and feedback

If you have any questions or feedback, let us know in the user forum.

• Technical questions
• Documentation feedback