multi2vec-palm

Added in v1.24.3

The multi2vec-palm module uses a Google multimodal embedding model to create vectors from text or images

Considerations

• This module enables the nearText and nearImage search operators.

• multi2vec-palm uses an external API.

  • Check vendor pricing before you vectorize data.
  • Obtain an API key from the vendor.

• This module is only compatible with Google Vertex AI. It is not compatible with Google AI Studio.

• The module s not compatible with Auto-schema. Define your collections manually.

Weaviate instance configuration

tip

If you use Weaviate Cloud Services (WCS), this module is already enabled and pre-configured. You cannot edit the configuration in WCS.

Docker Compose file

To use the multi2vec-palm module, enable it in your Docker Compose file. Edit your docker-compose.yml manually or use the Weaviate configuration tool to generate the file.

Parameters

Parameter	Required	Default	Description
location	Yes	None	Where the model runs (e.g. "us-central1").
projectId	Yes	"<Your GCP project>"	The name of your GCP project.
modelId	No	"multimodalembedding@001"	Current the only model available.
dimensions	No	1408	Must be one of: 128, 256, 512, 1408.

Specify the API key as a request header or an environment variable.

• Request header: X-Palm-Api-Key
• Environment variable: PALM_APIKEY

Configure multi2vec-palm for VertexAI

This module is only supported in Google Vertex AI. It is not supported in Google AI Studio.

To enable the Vertex AI API on your Google Cloud project, follow Google's instructions.

Vertex AI API key

The Vertex AI API key is called an access token in Google Cloud.

To retrieve your token, install the Google Cloud CLI tool and run this command:

gcloud auth print-access-token

Token expiration

Important

By default, Google Cloud's OAuth 2.0 access tokens have a lifetime of 1 hour. You can create tokens that last up to 12 hours. To create longer lasting tokens, follow the instructions in the Google Cloud IAM Guide.

Since the OAuth token is only valid for a limited time, you must periodically replace the token with a new one. After you generate the new token, you have to re-instantiate your Weaviate client to use it.

You can update the OAuth token manually, but manual updates may not be appropriate for your use case.

You can also automate the OAth token update. Weaviate does not control the OAth token update procedure. However, here are some automation options:

With Google Cloud CLI

If you are using the Google Cloud CLI, write a script to periodically update the token and extract the results.

Python code to extract the token looks like this:

client = re_instantiate_weaviate()

This is the re_instantiate_weaviate function:

import subprocess
import weaviate

def refresh_token() -> str:
    result = subprocess.run(["gcloud", "auth", "print-access-token"], capture_output=True, text=True)
    if result.returncode != 0:
        print(f"Error refreshing token: {result.stderr}")
        return None
    return result.stdout.strip()

def re_instantiate_weaviate() -> weaviate.Client:
    token = refresh_token()

    client = weaviate.Client(
      url = "https://WEAVIATE_INSTANCE_URL",  # Replace WEAVIATE_INSTANCE_URL with the URL
      additional_headers = {
        "X-PaLM-Api-Key": token,
      }
    )
    return client

# Run this every ~60 minutes
client = re_instantiate_weaviate()

With google-auth

Another way is through Google's own authentication library google-auth.

See the links to google-auth in Python and Node.js libraries.

You can, then, periodically the refresh function (see Python docs) to obtain a renewed token, and re-instantiate the Weaviate client.

For example, you could periodically run:

client = re_instantiate_weaviate()

Where re_instantiate_weaviate is something like:

from google.auth.transport.requests import Request
from google.oauth2.service_account import Credentials
import weaviate
import os


def get_credentials() -> Credentials:
    credentials = Credentials.from_service_account_file(
        "path/to/your/service-account.json",
        scopes=[
            "https://www.googleapis.com/auth/generative-language",
            "https://www.googleapis.com/auth/cloud-platform",
        ],
    )
    request = Request()
    credentials.refresh(request)
    return credentials


def re_instantiate_weaviate() -> weaviate.Client:
    credentials = get_credentials()
    token = credentials.token

    client = weaviate.connect_to_wcs(  # e.g. if you use the Weaviate Cloud Service
        cluster_url="https://WEAVIATE_INSTANCE_URL",  # Replace WEAVIATE_INSTANCE_URL with the URL
        auth_credentials=weaviate.auth.AuthApiKey(os.getenv("WCS_DEMO_RO_KEY")),  # Replace with your WCS key
        headers={
            "X-PaLM-Api-Key": token,
        },
    )
    return client


# Run this every ~60 minutes
client = re_instantiate_weaviate()

The service account key shown above can be generated by following this guide.

Example

This configuration does the following:

• enables multi2vec-palm
• sets multi2vec-palm as the default vectorizer
• uses an environment variable to set the PaLM API key

...
version: '3.4'
services:
  weaviate:
    image: cr.weaviate.io/semitechnologies/weaviate:1.24.10
    restart: on-failure:0
    ports:
     - 8080:8080
     - 50051:50051
    environment:
      QUERY_DEFAULTS_LIMIT: 20
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
      PERSISTENCE_DATA_PATH: "./data"
      ENABLE_MODULES: multi2vec-palm
      DEFAULT_VECTORIZER_MODULE: multi2vec-palm
      PALM_APIKEY: sk-replace-with-your-api-key  # Or provide the key at query time.
      CLUSTER_HOSTNAME: 'node1'
...

Collection configuration

To specify module behavior in a collection, edit the Weaviate schema.

Vectorization settings

Set vectorizer behavior in the moduleConfig section for each collection and property.

Collection-level settings

Parameter	Description
vectorizer	The module to use to vectorize the data.
vectorizeClassName	When true, vectorize the collection name. Defaults to true.
<media>Fields	Map property names to modalities (under moduleConfig.multi2vec-palm). One of: textFields, imageFields
weights	Change the contribution of the different modalities when calculating the vector.

Property-level settings

Parameter	Description
skip	When true, do not vectorize the property. Defaults to false
vectorizePropertyName	When true, vectorize the property name. Defaults to true.
dataType	The property's data type. Use in <media>Fields. One of: text,blob

Example

This collection definition sets the following:

• The multi2vec-palm module is the vectorizer for the collection MultimodalExample.
• The name property is text datatype and is a text field.
• The image property is a blob datatype and is an image field.

{
  "classes": [
    {
      "class": "MultimodalExample",
      "description": "An example collection for multi2vec-palm",
      "vectorizer": "multi2vec-palm",
      "moduleConfig": {
        "multi2vec-palm": {
          "textFields": ["name"],
          "imageFields": ["image"],
        }
      },
      "properties": [
        {
          "dataType": ["text"],
          "name": "name"
        },
        {
          "dataType": ["blob"],
          "name": "image"
        }
      ],
    }
  ]
}

Example with weights

The following example adds weights:

• textFields is 0.7
• imageFields is 0.3

{
  "classes": [
    {
      "class": "MultimodalExample",
      "moduleConfig": {
        "multi2vec-palm": {
          ...
          "weights": {
            "textFields": [0.7],
            "imageFields": [0.3],
          }
        }
      }
    }
  ]
}

blob data objects

Data that has the blob property type must be base64 encoded. To get the base64-encoded value of an image, use the helper methods in the Weaviate clients or run the following command:

cat my_image.png | base64

Additional information

Available models

Currently, the only available model is multimodalembedding@001.

Additional search operators

The multi2vec-palm vectorizer module enables the nearText and nearImage search operators.

These operators can do cross-modal search and retrieval.

All objects are encoded into a single vector space. This means, a query that use one modality, such as text, returns results from all available modalities.

Usage example

NearText

Python (v4)

import weaviate
import weaviate.classes as wvc
from weaviate.collections.classes.grpc import Move
import os

client = weaviate.connect_to_local()

try:
    publications = client.collections.get("Publication")

    response = publications.query.near_text(
        query="fashion",
        distance=0.6,
        move_to=Move(force=0.85, concepts="haute couture"),
        move_away=Move(force=0.45, concepts="finance"),
        return_metadata=wvc.query.MetadataQuery(distance=True),
        limit=2
    )

    for o in response.objects:
        print(o.properties)
        print(o.metadata)

finally:
    client.close()

Python (v3)

import weaviate

client = weaviate.Client("http://localhost:8080")

nearText = {
  "concepts": ["fashion"],
  "distance": 0.6, # prior to v1.14 use "certainty" instead of "distance"
  "moveAwayFrom": {
    "concepts": ["finance"],
    "force": 0.45
  },
  "moveTo": {
    "concepts": ["haute couture"],
    "force": 0.85
  }
}

result = (
  client.query
  .get("Publication", "name")
  .with_additional(["certainty OR distance"]) # note that certainty is only supported if distance==cosine
  .with_near_text(nearText)
  .do()
)

print(result)

JavaScript/TypeScript

import weaviate from 'weaviate-ts-client';

const client = weaviate.client({
  scheme: 'http',
  host: 'localhost:8080',
});

const response = await client.graphql
  .get()
  .withClassName('Publication')
  .withFields('name _additional{certainty distance}') // note that certainty is only supported if distance==cosine
  .withNearText({
    concepts: ['fashion'],
    distance: 0.6,  // prior to v1.14 use certainty instead of distance
    moveAwayFrom: {
      concepts: ['finance'],
      force: 0.45,
    },
    moveTo: {
      concepts: ['haute couture'],
      force: 0.85,
    },
  })
  .do();
console.log(response);

Go

package main

import (
  "context"
  "fmt"

  "github.com/weaviate/weaviate-go-client/v4/weaviate"
  "github.com/weaviate/weaviate-go-client/v4/weaviate/graphql"
)

func main() {
  cfg := weaviate.Config{
    Host:   "localhost:8080",
    Scheme: "http",
  }
  client, err := weaviate.NewClient(cfg)
  if err != nil {
    panic(err)
  }

  className := "Publication"

  name := graphql.Field{Name: "name"}
  _additional := graphql.Field{
    Name: "_additional", Fields: []graphql.Field{
      {Name: "certainty"}, // only supported if distance==cosine
      {Name: "distance"},  // always supported
    },
  }

  concepts := []string{"fashion"}
  distance := float32(0.6)
  moveAwayFrom := &graphql.MoveParameters{
    Concepts: []string{"finance"},
    Force:    0.45,
  }
  moveTo := &graphql.MoveParameters{
    Concepts: []string{"haute couture"},
    Force:    0.85,
  }
  nearText := client.GraphQL().NearTextArgBuilder().
    WithConcepts(concepts).
    WithDistance(distance). // use WithCertainty(certainty) prior to v1.14
    WithMoveTo(moveTo).
    WithMoveAwayFrom(moveAwayFrom)

  ctx := context.Background()

  result, err := client.GraphQL().Get().
    WithClassName(className).
    WithFields(name, _additional).
    WithNearText(nearText).
    Do(ctx)

  if err != nil {
    panic(err)
  }
  fmt.Printf("%v", result)
}

Java

package io.weaviate;

import io.weaviate.client.Config;
import io.weaviate.client.WeaviateClient;
import io.weaviate.client.base.Result;
import io.weaviate.client.v1.graphql.model.GraphQLResponse;
import io.weaviate.client.v1.graphql.query.argument.NearTextArgument;
import io.weaviate.client.v1.graphql.query.argument.NearTextMoveParameters;
import io.weaviate.client.v1.graphql.query.fields.Field;

public class App {
  public static void main(String[] args) {
    Config config = new Config("http", "localhost:8080");
    WeaviateClient client = new WeaviateClient(config);

    NearTextMoveParameters moveTo = NearTextMoveParameters.builder()
      .concepts(new String[]{ "haute couture" }).force(0.85f).build();

    NearTextMoveParameters moveAway = NearTextMoveParameters.builder()
      .concepts(new String[]{ "finance" }).force(0.45f)
      .build();

    NearTextArgument nearText = client.graphQL().arguments().nearTextArgBuilder()
      .concepts(new String[]{ "fashion" })
      .distance(0.6f) // use .certainty(0.7f) prior to v1.14
      .moveTo(moveTo)
      .moveAwayFrom(moveAway)
      .build();

    Field name = Field.builder().name("name").build();
    Field _additional = Field.builder()
      .name("_additional")
      .fields(new Field[]{
        Field.builder().name("certainty").build(), // only supported if distance==cosine
        Field.builder().name("distance").build(),  // always supported
      }).build();

    Result<GraphQLResponse> result = client.graphQL().get()
      .withClassName("Publication")
      .withFields(name, _additional)
      .withNearText(nearText)
      .run();

    if (result.hasErrors()) {
      System.out.println(result.getError());
      return;
    }
    System.out.println(result.getResult());
  }
}

Curl

# Note: Under nearText, use `certainty` instead of distance prior to v1.14
# Under _additional, `certainty` is only supported if distance==cosine, but `distance` is always supported
echo '{
  "query": "{
    Get {
      Publication(
        nearText: {
          concepts: [\"fashion\"],
          distance: 0.6,
          moveAwayFrom: {
            concepts: [\"finance\"],
            force: 0.45
          },
          moveTo: {
            concepts: [\"haute couture\"],
            force: 0.85
          }
        }
      ) {
        name
        _additional {
          certainty
          distance
        }
      }
    }
  }"
}' | curl \
    -X POST \
    -H 'Content-Type: application/json' \
    -H 'Authorization: Bearer learn-weaviate' \
    -H "X-OpenAI-Api-Key: $OPENAI_API_KEY" \
    -d @- \
    https://edu-demo.weaviate.network/v1/graphql

GraphQL

{
  Get{
    Publication(
      nearText: {
        concepts: ["fashion"],
        distance: 0.6 # prior to v1.14 use "certainty" instead of "distance"
        moveAwayFrom: {
          concepts: ["finance"],
          force: 0.45
        },
        moveTo: {
          concepts: ["haute couture"],
          force: 0.85
        }
      }
    ){
      name
      _additional {
        certainty # only supported if distance==cosine.
        distance  # always supported
      }
    }
  }
}

NearImage

Python

import weaviate

client = weaviate.Client("http://localhost:8080")

nearImage = {"image": "/9j/4AAQSkZJRgABAgE..."}

result = (
  client.query
  .get("FashionItem", "image")
  .with_near_image(nearImage)
  .do()
)

print(result)

JavaScript/TypeScript

import weaviate from 'weaviate-ts-client';

const client = weaviate.client({
  scheme: 'http',
  host: 'localhost:8080',
});

const response = await client.graphql
  .get()
  .withClassName('FashionItem')
  .withFields('image')
  .withNearImage({ image: '/9j/4AAQSkZJRgABAgE...' })
  .do();

console.log(JSON.stringify(response, null, 2));

Go

package main

import (
  "context"
  "fmt"

  "github.com/weaviate/weaviate-go-client/v4/weaviate"
  "github.com/weaviate/weaviate-go-client/v4/weaviate/graphql"
)

func main() {
  cfg := weaviate.Config{
    Host:   "localhost:8080",
    Scheme: "http",
  }
  client, err := weaviate.NewClient(cfg)
  if err != nil {
    panic(err)
  }

  className := "FashionItem"
  image := graphql.Field{Name: "image"}

  nearImage := client.GraphQL().NearImageArgBuilder().WithImage("/9j/4AAQSkZJRgABAgE...")
  ctx := context.Background()

  result, err := client.GraphQL().Get().
    WithClassName(className).
    WithFields(image).
    WithNearImage(nearImage).
    Do(ctx)

  if err != nil {
    panic(err)
  }
  fmt.Printf("%v", result)
}

Java

package io.weaviate;

import io.weaviate.client.Config;
import io.weaviate.client.WeaviateClient;
import io.weaviate.client.base.Result;
import io.weaviate.client.v1.graphql.model.GraphQLResponse;
import io.weaviate.client.v1.graphql.query.argument.NearImageArgument;
import io.weaviate.client.v1.graphql.query.fields.Field;

public class App {
  public static void main(String[] args) {
    Config config = new Config("http", "localhost:8080");
    WeaviateClient client = new WeaviateClient(config);

    String className = "FashionItem";
    Field image = Field.builder().name("image").build();

    NearImageArgument nearImage = client.graphQL().arguments().nearImageArgBuilder()
      .image("/9j/4AAQSkZJRgABAgE...")
      .build();

    Result<GraphQLResponse> result = client.graphQL().get()
      .withClassName(className)
      .withFields(image)
      .withNearImage(nearImage)
      .run();

    if (result.hasErrors()) {
      System.out.println(result.getError());
      return;
    }
    System.out.println(result.getResult());
  }
}

Curl

echo '{
  "query": "{
    Get {
      FashionItem(nearImage: {
        image: "/9j/4AAQSkZJRgABAgE..."
      }) {
        image
      }
    }
  }"
}' | curl \
    -X POST \
    -H 'Content-Type: application/json' \
    -d @- \
    http://localhost:8080/v1/graphql

GraphQL

{
  Get {
    FashionItem(nearImage: {
      image: "/9j/4AAQSkZJRgABAgE..."
    }) {
      image
    }
  }
}

Questions and feedback

If you have any questions or feedback, let us know in our user forum.

• Technical questions
• Documentation feedback