Generative Search - OpenAI

     

In short

• The Generative OpenAI (generative-openai) module generates responses based on the data stored in your Weaviate instance.
• The module can generate a response for each returned object, or a single response for a group of objects.
• The module adds a generate {} operator to the GraphQL _additional {} property of the Get {} queries.
• Added in Weaviate v1.17.3.
• The default OpenAI model is gpt-3.5-turbo, but other models (e.g. gpt-4) are supported.
• For Azure OpenAI, a model must be specified.

Azure OpenAI or OpenAI?

The module usage instructions may vary based on whether you are using OpenAI directly or Azure OpenAI. Please make sure that you are following the right instructions for your service provider.

For example, the following may vary:

• Parameter names used in the schema, and
• Names of the API key to be used.

Introduction

generative-openai generates responses based on the data stored in your Weaviate instance.

The module works in two steps:

1. (Weaviate) Run a search query in Weaviate to find relevant objects.
2. (OpenAI) Use an OpenAI model to generate a response based on the results (from the previous step) and the provided prompt or task.

note

You can use the Generative OpenAI module with non-OpenAI upstream modules. For example, you could use text2vec-cohere or text2vec-huggingface to vectorize and query your data, but then rely on the generative-openai module to generate a response.

The generative module can provide results for:

• each returned object - singleResult{ prompt }
• the group of all results together – groupedResult{ task }

You need to input both a query and a prompt (for individual responses) or a task (for all responses).

Inference API key

generative-openai requires an API key from OpenAI or Azure OpenAI.

tip

You only need to provide one of the two keys, depending on which service (OpenAI or Azure OpenAI) you are using.

Providing the key to Weaviate

You can provide your API key in two ways:

1. During the configuration of your Docker instance, by adding OPENAI_APIKEY or AZURE_APIKEY as appropriate under environment to your Docker Compose file, like this:

   environment:
     OPENAI_APIKEY: 'your-key-goes-here'  # For use with OpenAI. Setting this parameter is optional; you can also provide the key at runtime.
     AZURE_APIKEY: 'your-key-goes-here'  # For use with Azure OpenAI. Setting this parameter is optional; you can also provide the key at runtime.
     ...

2. At run-time (recommended), by providing "X-OpenAI-Api-Key" or "X-Azure-Api-Key" through the request header. You can provide it using the Weaviate client, like this:

Python

import weaviate

client = weaviate.Client(
  url = "https://some-endpoint.weaviate.network/",
  additional_headers = {
    "X-OpenAI-Api-Key": "YOUR-OPENAI-API-KEY",  # Replace with your API key
    "X-Azure-Api-Key": "YOUR-AZURE-API-KEY",  # Replace with your API key
  }
)

JavaScript/TypeScript

import weaviate from 'weaviate-ts-client';

const client = weaviate.client({
  scheme: 'https',
  host: 'some-endpoint.weaviate.network',
  // Replace with your API key
  headers: {
    'X-OpenAI-Api-Key': 'YOUR-OPENAI-API-KEY',
    'X-Azure-Api-Key': 'YOUR-AZURE-API-KEY',
  },
});

Go

package main

import (
    "context"
    "fmt"
    "github.com/weaviate/weaviate-go-client/v4/weaviate"
  "github.com/weaviate/weaviate/entities/models"
)

func main() {
    cfg := weaviate.Config{
        Host:   "some-endpoint.weaviate.network/",  // Replace with your endpoint
        Scheme: "https",
        // Replace with your API key
        Headers: map[string]string{
          "X-OpenAI-Api-Key": "YOUR-OPENAI-API-KEY",  // Replace with your API key
          "X-Azure-Api-Key": "YOUR-AZURE-API-KEY", // Replace with your API key
        }
    }

    client, err := weaviate.NewClient(cfg)
    if err != nil {
      panic(err)
    }
}

Java

package io.weaviate;
import java.util.ArrayList;
import io.weaviate.client.Config;
import io.weaviate.client.WeaviateClient;
import io.weaviate.client.base.Result;

public class App {
  public static void main(String[] args) {
    Map<String, String> headers = new HashMap<String, String>() { {
      // Replace with your API key
      put("X-OpenAI-Api-Key", "YOUR-OPENAI-API-KEY");  // Replace with your API key
      put("X-Azure-Api-Key", "YOUR-AZURE-API-KEY"); // Replace with your API key
    } };

    Config config = new Config("https", "some-endpoint.weaviate.network/", headers);
    WeaviateClient client = new WeaviateClient(config);
  }
}

Organization name

Available from version v1.21.1

For requests that require the OpenAI organization name, you can provide it at query time by adding it to the HTTP header:

• "X-OpenAI-Organization": "YOUR-OPENAI-ORGANIZATION" for OpenAI

Module configuration

Not applicable to WCS

This module is enabled and pre-configured on Weaviate Cloud Services.

Docker Compose file (Weaviate open source only)

You can enable the Generative OpenAI module in your Docker Compose file (e.g. docker-compose.yml). Add the generative-openai module (alongside any other module you may need) to the ENABLE_MODULES property, like this:

ENABLE_MODULES: 'text2vec-openai,generative-openai'

Here is a full example of a Docker configuration, which uses the generative-openai module in combination with text2vec-openai:

---
version: '3.4'
services:
  weaviate:
    command:
      - --host
      - 0.0.0.0
      - --port
      - '8080'
      - --scheme
      - http
    image:
      semitechnologies/weaviate:1.21.3
    ports:
      - 8080:8080
    restart: on-failure:0
    environment:
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
      PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
      DEFAULT_VECTORIZER_MODULE: 'text2vec-openai'
      ENABLE_MODULES: 'text2vec-openai,generative-openai'
      OPENAI_APIKEY: sk-foobar  # For use with OpenAI. Setting this parameter is optional; you can also provide the key at runtime.
      OPENAI_ORGANIZATION: your-orgname  # For use with OpenAI. Setting this parameter is optional; you can also provide the key at runtime.
      AZURE_APIKEY: sk-foobar  # For use with Azure OpenAI. Setting this parameter is optional; you can also provide the key at runtime.
      CLUSTER_HOSTNAME: 'node1'

Schema configuration

You can define settings for this module in the schema.

OpenAI vs Azure OpenAI

• OpenAI users can optionally set the model parameter.
• Azure OpenAI users must set the parameters resourceName and deploymentId.

Model parameters

You can also configure additional parameters for the generative model through the xxxProperty parameters shown below.

Example schema

For example, the following schema configuration will set Weaviate to use the generative-openai model with the Document class.

{
  "classes": [
    {
      "class": "Document",
      "description": "A class called document",
      ...,
      "moduleConfig": {
        "generative-openai": {
          "model": "gpt-3.5-turbo",  // Optional - Defaults to `gpt-3.5-turbo`
          "resourceName": "<YOUR-RESOURCE-NAME>",  // For Azure OpenAI - Required
          "deploymentId": "<YOUR-MODEL-NAME>",  // For Azure OpenAI - Required
          "temperatureProperty": <temperature>,  // Optional, applicable to both OpenAI and Azure OpenAI
          "maxTokensProperty": <max_tokens>,  // Optional, applicable to both OpenAI and Azure OpenAI
          "frequencyPenaltyProperty": <frequency_penalty>,  // Optional, applicable to both OpenAI and Azure OpenAI
          "presencePenaltyProperty": <presence_penalty>,  // Optional, applicable to both OpenAI and Azure OpenAI
          "topPProperty": <top_p>,  // Optional, applicable to both OpenAI and Azure OpenAI
        },
      }
    }
  ]
}

New to Weaviate Schemas?

If you are new to Weaviate, check out the Weaviate schema tutorial.

How to use

This module extends the _additional {...} property with a generate operator.

generate takes the following arguments:

Field	Data Type	Required	Example	Description
singleResult {prompt}	string	no	Summarize the following in a tweet: {summary}	Generates a response for each individual search result. You need to include at least one result field in the prompt, between braces.
groupedResult {task}	string	no	Explain why these results are similar to each other	Generates a single response for all search results

Example of properties in the prompt

When piping the results to the prompt, at least one field returned by the query must be added to the prompt. If you don't add any fields, Weaviate will throw an error.

For example, assume your schema looks like this:

{
  Article {
    title
    summary
  }
}

You can add both title and summary to the prompt by enclosing them in curly brackets:

{
  Get {
    Article {
      title
      summary
      _additional {
        generate(
          singleResult: {
            prompt: """
            Summarize the following in a tweet:

            {title} - {summary}
            """
          }
        ) {
          singleResult
          error
        }
      }
    }
  }
}

Example - single result

Here is an example of a query where:

• we run a vector search (with nearText) to find articles about "Italian food"
• then we ask the generator module to describe each result as a Facebook ad.
  • the query asks for the summary field, which it then includes in the prompt argument of the generate operator.

GraphQL

{
  Get {
    Article(
      nearText: {
        concepts: ["Italian food"]
      }
      limit: 1
    ) {
      title
      summary
      _additional {
        generate(
          singleResult: {
            prompt: """
              Describe the following as a Facebook Ad: {summary}
            """
          }
        ) {
          singleResult
          error
        }
      }
    }
  }
}

Python

import weaviate

client  = weaviate.Client(
  url = "https://some-endpoint.weaviate.network/",
  additional_headers={
      "X-OpenAI-Api-Key": "YOUR-OPENAI-API-KEY"
  }
)

# instruction for the generative module
generatePrompt = "Describe the following as a Facebook Ad: {summary}"

result = (
  client.query
  .get("Article", ["title", "summary"])
  .with_generate(single_prompt=generatePrompt)
  .with_near_text({
    "concepts": ["Italian food"]
  })
  .with_limit(5)
).do()

print(result)

JavaScript/TypeScript

import weaviate, { ApiKey } from 'weaviate-ts-client';

const client = weaviate.client({
  scheme: 'https',
  host: 'edu-demo.weaviate.network',
  apiKey: new ApiKey('learn-weaviate'),
  headers: { 'X-OpenAI-Api-Key': process.env['OPENAI_API_KEY'] },  // Replace with your API key
});

// instruction for the generative module
const generatePrompt = 'Describe the following as a Facebook Ad: {summary}';

const response = await client.graphql
  .get()
  .withClassName('Article')
  .withFields('title summary')
  .withNearText({
    concepts: ['Italian food'],
  })
  .withGenerate({
    singlePrompt: generatePrompt,
  })
  .withLimit(5)
  .do();
console.log(JSON.stringify(response, null, 2));

Go

package main

import (
  "context"
  "fmt"
  "github.com/weaviate/weaviate-go-client/v4/weaviate"
  "github.com/weaviate/weaviate-go-client/v4/weaviate/graphql"
)

func main() {
  cfg := weaviate.Config{
    Host:    "some-endpoint.weaviate.network",
    Scheme:  "https",
    Headers: map[string]string{"X-OpenAI-Api-Key": "YOUR-OPENAI-API-KEY"},
  }
  client, err := weaviate.NewClient(cfg)
  if err != nil {
    panic(err)
  }
  ctx := context.Background()

  fields := []graphql.Field{
    {Name: "title"},
    {Name: "summary"},
  }

  concepts := []string{"Italian food"}
  nearText := client.GraphQL().NearTextArgBuilder().
    WithConcepts(concepts)

  gs := graphql.NewGenerativeSearch().SingleResult("\"Describe the following as a Facebook Ad: {summary}\"")

  result, err := client.GraphQL().Get().
    WithClassName("Article").
    WithFields(fields...).
    WithNearText(nearText).
    withGenerativeSearch(generativeSearch).
    WithLimit(5).
    Do(ctx)

  if err != nil {
    panic(err)
  }
  fmt.Printf("%v", result)
}

Java

package io.weaviate;

import java.util.HashMap;
import java.util.Map;
import io.weaviate.client.Config;
import io.weaviate.client.WeaviateClient;
import io.weaviate.client.base.Result;
import io.weaviate.client.v1.graphql.model.GraphQLResponse;
import io.weaviate.client.v1.graphql.query.argument.NearTextArgument;
import io.weaviate.client.v1.graphql.query.fields.Field;

public class App {
  public static void main(String[] args) {
    Map<String, String> headers = new HashMap<String, String>() {
      {put("X-OpenAI-Api-Key", "YOUR-OPENAI-API-KEY");}
    };
    Config config = new Config("https", "some-endpoint.weaviate.network", headers);
    WeaviateClient client = new WeaviateClient(config);

    // instruction for the generative module
    GenerativeSearchBuilder generativeSearch = GenerativeSearchBuilder.builder()
      .singleResultPrompt("\"Describe the following as a Facebook Ad: {summary}\"")
      .build();

    Field title = Field.builder().name("title").build();
    Field summary = Field.builder().name("summary").build();

    NearTextArgument nearText = client.graphQL().arguments().nearTextArgBuilder()
      .concepts(new String[]{ "Italian food" })
      .build();

    Result<GraphQLResponse> result = client.graphQL().get()
      .withClassName("Article")
      .withFields(title, summary)
      .withGenerativeSearch(generativeSearch)
      .withNearText(nearText)
      .withLimit(5)
      .run();

    if (result.hasErrors()) {
      System.out.println(result.getError());
      return;
    }
    System.out.println(result.getResult());
  }
}

Curl

echo '{
  "query": "{
    Get {
      Article(
        nearText: {
          concepts: [\"Italian food\"]
        }
        limit: 5
      ) {
        title
        summary
        _additional {
          generate(
            singleResult: {
              prompt: \"\"\"
                Describe the following as a Facebook Ad: {summary}
              \"\"\"
            }
          ) {
            singleResult
            error
          }
        }
      }
    }
  }
  "
}' | curl \
    -X POST \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer learn-weaviate' \
    -H "X-OpenAI-Api-Key: $OPENAI_API_KEY" \
    -d @- \
    https://edu-demo.weaviate.network/v1/graphql

Example response - single result

{
  "data": {
    "Get": {
      "Article": [
        {
          "_additional": {
            "generate": {
              "error": null,
              "singleResult": "This Facebook Ad will explore the fascinating history of Italian food and how it has evolved over time. Learn from Dr Eva Del Soldato and Diego Zancani, two experts in Italian food history, about how even the emoji for pasta isn't just pasta -- it's a steaming plate of spaghetti heaped with tomato sauce on top. Discover how Italy's complex history has shaped the Italian food we know and love today."
            }
          },
          "summary": "Even the emoji for pasta isn't just pasta -- it's a steaming plate of spaghetti heaped with tomato sauce on top. But while today we think of tomatoes as inextricably linked to Italian food, that hasn't always been the case. \"People tend to think Italian food was always as it is now -- that Dante was eating pizza,\" says Dr Eva Del Soldato , associate professor of romance languages at the University of Pennsylvania, who leads courses on Italian food history. In fact, she says, Italy's complex history -- it wasn't unified until 1861 -- means that what we think of Italian food is, for the most part, a relatively modern concept. Diego Zancani, emeritus professor of medieval and modern languages at Oxford University and author of \"How We Fell in Love with Italian Food,\" agrees.",
          "title": "How this fruit became the star of Italian cooking"
        }
      ]
    }
  }
}

Example - grouped result

Here is an example of a query where:

• we run a vector search (with nearText) to find publications about finance,
• then we ask the generator module to explain why these articles are about finance.

GraphQL

{
  Get {
    Publication(
      nearText: {
        concepts: ["magazine or newspaper about finance"]
        certainty: 0.75
      }
    ) {
      name
      _additional {
        generate(
          groupedResult: {
            task: "Explain why these magazines or newspapers are about finance"
          }
        ) {
          groupedResult
          error
        }
      }
    }
  }
}

Python

import weaviate

client  = weaviate.Client(
  url = "https://some-endpoint.weaviate.network/",
  additional_headers={
      "X-OpenAI-Api-Key": "YOUR-OPENAI-API-KEY"
  }
)

# instruction for the generative module
generateTask = "Explain why these magazines or newspapers are about finance"

result = (
  client.query
  .get("Publication", ["name"])
  .with_generate(grouped_task=generateTask)
  .with_near_text({
    "concepts": ["magazine or newspaper about finance"]
  })
  .with_limit(5)
).do()

print(result)

JavaScript/TypeScript

import weaviate, { ApiKey } from 'weaviate-ts-client';

const client= weaviate.client({
  scheme: 'https',
  host: 'edu-demo.weaviate.network',
  apiKey: new ApiKey('learn-weaviate'),
  headers: { 'X-OpenAI-Api-Key': process.env['OPENAI_API_KEY'] },
});

// instruction for the generative module
const generateTask = 'Explain why these magazines or newspapers are about finance';

const response = await client.graphql
  .get()
  .withClassName('Article')
  .withFields('title')
  .withNearText({
    concepts: ['magazine or newspaper about finance'],
  })
  .withGenerate({
    groupedTask: generateTask,
  })
  .withLimit(5)
  .do();
console.log(JSON.stringify(response, null, 2));

Go

package main

import (
  "context"
  "fmt"

  "github.com/weaviate/weaviate-go-client/v4/weaviate"
  "github.com/weaviate/weaviate-go-client/v4/weaviate/graphql"
)

func main() {
  cfg := weaviate.Config{
    Host:    "some-endpoint.weaviate.network",
    Scheme:  "https",
    Headers: map[string]string{"X-OpenAI-Api-Key": "YOUR-OPENAI-API-KEY"},
  }
  client, err := weaviate.NewClient(cfg)
  if err != nil {
    panic(err)
  }
  ctx := context.Background()

  name := graphql.Field{Name: "name"}

  concepts := []string{"magazine or newspaper about finance"}
  nearText := client.GraphQL().NearTextArgBuilder().
    WithConcepts(concepts)

  gs := graphql.NewGenerativeSearch().GroupedResult("Explain why these magazines or newspapers are about finance")

  result, err := client.GraphQL().Get().
    WithClassName("Publication").
    WithFields(name).
    WithGenerativeSearch(gs).
    WithNearText(nearText).
    WithLimit(5).
    Do(ctx)

  if err != nil {
    panic(err)
  }
  fmt.Printf("%v", result)
}

Java

package io.weaviate;

import java.util.HashMap;
import java.util.Map;
import io.weaviate.client.Config;
import io.weaviate.client.WeaviateClient;
import io.weaviate.client.base.Result;
import io.weaviate.client.v1.graphql.model.GraphQLResponse;
import io.weaviate.client.v1.graphql.query.argument.NearTextArgument;
import io.weaviate.client.v1.graphql.query.fields.Field;

public class App {
  public static void main(String[] args) {
    Map<String, String> headers = new HashMap<String, String>() { {
      put("X-OpenAI-Api-Key", "YOUR-OPENAI-API-KEY");
    } };
    Config config = new Config("https", "some-endpoint.weaviate.network", headers);
    WeaviateClient client = new WeaviateClient(config);

    // instruction for the generative module
    GenerativeSearchBuilder generativeSearch = GenerativeSearchBuilder.builder()
      .groupedResultTask("Explain why these magazines or newspapers are about finance")
      .build();

    Field name = Field.builder().name("name").build();

    NearTextArgument nearText = client.graphQL().arguments().nearTextArgBuilder()
      .concepts(new String[]{ "magazine or newspaper about finance" })
      .build();

    Result<GraphQLResponse> result = client.graphQL().get()
      .withClassName("Publication")
      .withFields(name)
      .withGenerativeSearch(generativeSearch)
      .withNearText(nearText)
      .withLimit(5)
      .run();

    if (result.hasErrors()) {
      System.out.println(result.getError());
      return;
    }
    System.out.println(result.getResult());
  }
}

Curl

echo '{
  "query": "{
    Get {
      Publication(
        nearText: {
          concepts: [\"magazine or newspaper about finance\"]
        }
        limit: 5
      ) {
        name
        _additional {
          generate(
            groupedResult: {
              task: \"Explain why these magazines or newspapers are about finance\"
            }
          ) {
            groupedResult
            error
          }
        }
      }
    }
  }
  "
}' | curl \
    -X POST \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer learn-weaviate' \
    -H "X-OpenAI-Api-Key: $OPENAI_API_KEY" \
    -d @- \
    https://edu-demo.weaviate.network/v1/graphql

Example response - grouped result

{
  "data": {
    "Get": {
      "Publication": [
        {
          "_additional": {
            "generate": {
              "error": null,
              "groupedResult": "The Financial Times, Wall Street Journal, and The New York Times Company are all about finance because they provide news and analysis on the latest financial markets, economic trends, and business developments. They also provide advice and commentary on personal finance, investments, and other financial topics."
            }
          },
          "name": "Financial Times"
        },
        {
          "_additional": {
            "generate": null
          },
          "name": "Wall Street Journal"
        },
        {
          "_additional": {
            "generate": null
          },
          "name": "The New York Times Company"
        }
      ]
    }
  }
}

Additional information

Supported models (OpenAI)

You can use any of

• gpt-3.5-turbo (default)
• gpt-3.5-turbo-16k
• gpt-4
• gpt-4-32k

The module also supports these legacy models (not recommended)

• davinci 002
• davinci 003

More resources

For additional information, try these sources.

1. Frequently Asked Questions
2. Weaviate Community Forum
3. Knowledge base of old issues
4. Stackoverflow
5. Weaviate slack channel