img2vec-neural

     

Introduction

This module vectorizes images using neural networks. Pre-trained modules can be used. resnet50 is the first model that is supported, other models will be released soon. resnet50 is a residual convolutional neural network with 25.5 million parameters trained on more than a million images from the ImageNet database. As the name suggests, it has a total of 50 layers: 48 convolution layers, 1 MaxPool layer and 1 Average Pool layer.

Available img2vec-neural models

There are two different inference models you can choose from. Depending on your machine (arm64 or other) and whether you prefer to use multi-threading to extract feature vectors or not, you can choose between keras and pytorch. There are no other differences between the two models.

• resnet50 (keras):
  • Supports amd64, but not arm64.
  • Does not support CUDA (yet)
  • Supports multi-threaded inference
• resnet50 (pytorch):
  • Supports both amd64 and arm64.
  • Supports CUDA
  • Does not support multi-threaded inference

How to enable in Weaviate

Note: you can also use the Weaviate configuration tool.

Docker-compose file

You can find an example Docker-compose file below, which will spin up Weaviate with the image vectorization module. This example spins up a Weaviate with only one vectorization module, the img2vec-neural module of pytorch with the resnet50 model.

---
version: '3.4'
services:
  weaviate:
    command:
    - --host
    - 0.0.0.0
    - --port
    - '8080'
    - --scheme
    - http
    image: semitechnologies/weaviate:1.19.6
    ports:
    - 8080:8080
    restart: on-failure:0
    environment:
      IMAGE_INFERENCE_API: "http://i2v-neural:8080"
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
      PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
      DEFAULT_VECTORIZER_MODULE: 'img2vec-neural'
      ENABLE_MODULES: 'img2vec-neural'
      CLUSTER_HOSTNAME: 'node1'
  i2v-neural:
    image: semitechnologies/img2vec-pytorch:resnet50

You can substitute semitechnologies/img2vec-pytorch:resnet50 with semitechnologies/img2vec-keras:resnet50 in case you want to use the keras module.

You can combine the image vectorization module with a text vectorization module. In the following example, we use both the text2vec-contextionary module as well as the img2vec-neural module. We set text2vec-contextionary to default vectorizer module, which means we need to specify in a data schema when we want a class to be vectorized as with the img2vec-neural module instead of the text2vec-contextionary module.

Other methods

If you prefer not to use Docker-compose (but instead for example Kubernetes in a production setup), make you can use the img2vec-neural module after taking the following steps:

1. Enable the img2vec-neural module. Make sure you set the ENABLE_MODULES=img2vec-neural environment variable. This can be combined with a text vectorization module: ENABLE_MODULES: 'text2vec-contextionary,img2vec-neural'. Additionally, you can make one of the modules the default vectorizer, so you don't have to specify it on each schema class: DEFAULT_VECTORIZER_MODULE=text2vec-contextionary
2. Run the img2vec-neural module, for example using docker run -itp "8000:8080" semitechnologies/img2vec-neural:resnet50-61dcbf8.
3. Tell Weaviate where to find the inference module. Set the Weaviate environment variable IMAGE_INFERENCE_APIto where your inference container is running, for example IMAGE_INFERENCE_API="http://localhost:8000"
4. You can now use Weaviate normally and all vectorization of images during import and search time will be done with the selected image vectorization model (given that the schema is configured correctly).

Schema configuration

You can specify to use the image vectorizer per class in the schema. To find details on how to configure a data schema, go here. When you set the vectorizer of a class to img2vec-neural, only the property fields that are specified in the moduleConfig will be taken into the computation of the vector.

When adding a class with vectorizer type img2vec-neural, the configuration must contain information about which field holds the image. The dataType of the imageFields should be blob. This can be achieved with the following config in a class object:

  "moduleConfig": {
    "img2vec-neural": {
      "imageFields": [
        "image"
      ]
    }
  }

If multiple fields are specified, the module will vectorize them separately and use the mean vector of both.

A full example of a class using the img2vec-neural module is shown below. This module makes use of the blob dataType.

{
  "classes": [
    {
      "class": "FashionItem",
      "description": "Each example is a 28x28 grayscale image, associated with a label from 10 classes.",
      "moduleConfig": {
        "img2vec-neural": {
          "imageFields": [
            "image"
          ]
        }
      },
      "properties": [
        {
          "dataType": [
            "blob"
          ],
          "description": "Grayscale image",
          "name": "image"
        },
        {
          "dataType": [
            "number"
          ],
          "description": "Label number for the given image.",
          "name": "labelNumber"
        },
        {
          "dataType": [
            "text"
          ],
          "description": "label name (description) of the given image.",
          "name": "labelName"
        }
      ],
      "vectorIndexType": "hnsw",
      "vectorizer": "img2vec-neural"
    }
  ]
}

note

Other properties, for example the name of the image that is given in another field, will not be taken into consideration. This means that you can only find the image with semantic search by another image, data object, or raw vector. Semantic search of images by text field (using nearText) is not possible, because this requires a text2vec vectorization module. Multiple modules cannot be combined at class level yet (might become possible in the future, since image-text-combined transformers exists). We recommend to use one of the following workarounds:

1. Best practice for multi-module search: create an image class and a text class in which you refer to each other by cross-reference. This way you can always hop along the reference and search either by "text labels" (using a text2vec-.... module) or by image (using a img2vec-... module).
2. If you don't want to create multiple classes, you are limited to using a where filter to find images by other search terms than an image, data object, or vector. A where filter does not use semantic features of a module.

Adding image data objects

When adding data, make sure that the specified fields are filled with valid image data (e.g. jpg, png, etc.), encoded as base64 (string) value in the property that has a blob dataType. The blob type itself (see below) requires that all blobs are base64 encoded. To obtain the base64-encoded value of an image, you can run the following command - or use the helper methods in the Weaviate clients - to do so:

cat my_image.png | base64

You can then import data with blob dataType in to Weaviate as follows:

Python

import weaviate

client = weaviate.Client("http://localhost:8080")

data_properties = {
    "labelName": "Mickey Mouse T-shirt",
    "image": "iVBORw0KGgoAAAANS..."
}

result = (
  client.data_object
  .create(data_properties, "FashionItem", "36ddd591-2dee-4e7e-a3cc-eb86d30a4302")
)

print(result)

JavaScript

const weaviate = require('weaviate-client');

const client = weaviate.client({
  scheme: 'http',
  host: 'localhost:8080',
});

client.data
      .creator()
      .withClassName('FashionItem')
      .withId('36ddd591-2dee-4e7e-a3cc-eb86d30a4302')
      .withProperties({
        'labelName': 'Mickey Mouse T-shirt'
        'image': 'iVBORw0KGgoAAAANS...'
    })
      .do()
      .then(res => {
          console.log(res)
      })
      .catch(err => {
          console.error(err)
      });

Go

package main

import (
  "context"
  "fmt"

  "github.com/weaviate/weaviate-go-client/v4/weaviate"
)

func main() {
  cfg := weaviate.Config{
    Host:   "localhost:8080",
    Scheme: "http",
  }
  client, err := weaviate.NewClient(cfg)
  if err != nil {
    panic(err)
  }

  dataSchema := map[string]interface{}{
    "labelName": "Mickey Mouse T-shirt",
    "image":     "iVBORw0KGgoAAAANS...",
  }

  created, err := client.Data().Creator().
    WithClassName("FashionItem").
    WithID("36ddd591-2dee-4e7e-a3cc-eb86d30a4302").
    WithProperties(dataSchema).
    Do(context.Background())
  if err != nil {
    panic(err)
  }
  fmt.Printf("%v", created)
}

Java

package io.weaviate;

import java.util.HashMap;
import java.util.Map;
import io.weaviate.client.Config;
import io.weaviate.client.WeaviateClient;
import io.weaviate.client.base.Result;
import io.weaviate.client.v1.data.model.WeaviateObject;

public class App {
  public static void main(String[] args) {
    Config config = new Config("http", "localhost:8080");
    WeaviateClient client = new WeaviateClient(config);

    Map<String, Object> dataSchema = new HashMap<>();
    dataSchema.put("labelName", "Mickey Mouse T-shirt");
    dataSchema.put("image", "iVBORw0KGgoAAAANS...");

    Result<WeaviateObject> result = client.data().creator()
      .withClassName("FashionItem")
      .withID("36ddd591-2dee-4e7e-a3cc-eb86d30a4302")
      .withProperties(dataSchema)
      .run();

    if (result.hasErrors()) {
      System.out.println(result.getError());
      return;
    }
    System.out.println(result.getResult());
  }
}

Curl

$ curl \
    -X POST \
    -H "Content-Type: application/json" \
    -d '{
      "class": "FashionItem",
      "id": "36ddd591-2dee-4e7e-a3cc-eb86d30a4302",
      "properties": {
        "labelName": "Mickey Mouse T-shirt"
        "image": "iVBORw0KGgoAAAANS..."
    }
  }' \
    http://localhost:8080/v1/objects

Additional GraphQL API parameters

nearImage search

At search time you can use the "standard" vector-search operators, such as nearVector and nearObject. But in addition, you can also vectorize a new image at search time and search by using the image's vector. To do so, you can use the nearImage search operator like so:

GraphQL

{
  Get {
    FashionItem(nearImage: {
      image: "/9j/4AAQSkZJRgABAgE..."
    }) {
      image
    }
  }
}

Python

import weaviate

client = weaviate.Client("http://localhost:8080")

nearImage = {"image": "/9j/4AAQSkZJRgABAgE..."}

result = (
  client.query
  .get("FashionItem", "image")
  .with_near_image(nearImage)
  .do()
)

print(result)

JavaScript

const weaviate = require('weaviate-client');

const client = weaviate.client({
  scheme: 'http',
  host: 'localhost:8080',
});

client.graphql
      .get()
      .withClassName('FashionItem')
      .withFields('image')
      .withNearImage({image: '/9j/4AAQSkZJRgABAgE...'})
      .do()
      .then(res => {
        console.log(res)
      })
      .catch(err => {
        console.error(err)
      });

Go

package main

import (
  "context"
  "fmt"

  "github.com/weaviate/weaviate-go-client/v4/weaviate"
  "github.com/weaviate/weaviate-go-client/v4/weaviate/graphql"
)

func main() {
  cfg := weaviate.Config{
    Host:   "localhost:8080",
    Scheme: "http",
  }
  client, err := weaviate.NewClient(cfg)
  if err != nil {
    panic(err)
  }

  className := "FashionItem"
  image := graphql.Field{Name: "image"}

  nearImage := client.GraphQL().NearImageArgBuilder().WithImage("/9j/4AAQSkZJRgABAgE...")
  ctx := context.Background()

  result, err := client.GraphQL().Get().
    WithClassName(className).
    WithFields(image).
    WithNearImage(nearImage).
    Do(ctx)

  if err != nil {
    panic(err)
  }
  fmt.Printf("%v", result)
}

Java

package io.weaviate;

import io.weaviate.client.Config;
import io.weaviate.client.WeaviateClient;
import io.weaviate.client.base.Result;
import io.weaviate.client.v1.graphql.model.GraphQLResponse;
import io.weaviate.client.v1.graphql.query.argument.NearImageArgument;
import io.weaviate.client.v1.graphql.query.fields.Field;

public class App {
  public static void main(String[] args) {
    Config config = new Config("http", "localhost:8080");
    WeaviateClient client = new WeaviateClient(config);

    String className = "FashionItem";
    Field image = Field.builder().name("image").build();

    NearImageArgument nearImage = client.graphQL().arguments().nearImageArgBuilder()
      .image("/9j/4AAQSkZJRgABAgE...")
      .build();

    Result<GraphQLResponse> result = client.graphQL().get()
      .withClassName(className)
      .withFields(image)
      .withNearImage(nearImage)
      .run();

    if (result.hasErrors()) {
      System.out.println(result.getError());
      return;
    }
    System.out.println(result.getResult());
  }
}

Curl

$ echo '{
  "query": "{
    Get {
      FashionItem(nearImage: {
        image: "/9j/4AAQSkZJRgABAgE..."
      }) {
        image
      }
    }
  }"
}' | curl \
    -X POST \
    -H 'Content-Type: application/json' \
    -d @- \
    http://localhost:8080/v1/graphql

Alternatively, you can use a helper function in the Python, Java or Go client (not with the JavaScript client). With an encoder function, you can input your image as png file, and the helper function encodes this to a base64 encoded value.

GraphQL

# GraphQL doesn't support png->base64 encoding, so please use a base64 encoded image in your query

{
  Get {
    FashionItem(nearImage: {
      image: "/9j/4AAQSkZJRgABAgE..."
    }) {
      image
    }
  }
}

Python

import weaviate

client = weaviate.Client("http://localhost:8080")

nearImage = {"image": "my_image_path.png"}

result = (
  client.query
  .get("FashionItem", "image")
  .with_near_image(nearImage, encode=True)
  .do()
)

print(result)

## OR use the weaviate.utils function:

client = weaviate.Client("http://localhost:8080")

encoded_image = weaviate.util.image_encoder_b64("my_image_path.png")

nearImage = {"image": "encoded_image"}

result = (
  client.query
  .get("FashionItem", "image")
  .with_near_image(nearImage)
  .do()
)

print(result)

JavaScript

// The JavaScript client doesn't support image encoding via a helper function, you need to give the input image as base64 format yourself

const weaviate = require('weaviate-client');

const client = weaviate.client({
  scheme: 'http',
  host: 'localhost:8080',
});

client.graphql
      .get()
      .withClassName('FashionItem')
      .withFields('image')
      .withNearImage({image: '/9j/4AAQSkZJRgABAgE...'})
      .do()
      .then(res => {
        console.log(res)
      })
      .catch(err => {
        console.error(err)
      });

Go

package main

import (
  "context"
  "fmt"
  "os"

  "github.com/weaviate/weaviate-go-client/v4/weaviate"
  "github.com/weaviate/weaviate-go-client/v4/weaviate/graphql"
)

func main() {
  cfg := weaviate.Config{
    Host:   "localhost:8080",
    Scheme: "http",
  }
  client, err := weaviate.NewClient(cfg)
  if err != nil {
    panic(err)
  }

  className := "FashionItem"
  image := graphql.Field{Name: "image"}

  filename := "my_image_path.png"
  file, err := os.Open(filename)
  if err != nil {
    panic(err)
  }

  nearImage := client.GraphQL().NearImageArgBuilder().WithReader(file)
  ctx := context.Background()

  result, err := client.GraphQL().Get().
    WithClassName(className).
    WithFields(image).
    WithNearImage(nearImage).
    Do(ctx)

  if err != nil {
    panic(err)
  }
  fmt.Printf("%v", result)
}

Java

package io.weaviate;

import java.io.File;
import io.weaviate.client.Config;
import io.weaviate.client.WeaviateClient;
import io.weaviate.client.base.Result;
import io.weaviate.client.v1.graphql.model.GraphQLResponse;
import io.weaviate.client.v1.graphql.query.argument.NearImageArgument;
import io.weaviate.client.v1.graphql.query.fields.Field;

public class App {
  public static void main(String[] args) {
    Config config = new Config("http", "localhost:8080");
    WeaviateClient client = new WeaviateClient(config);

    File imageFile = new File("my_image_path.png");

    String className = "FashionItem";
    Field image = Field.builder().name("image").build();

    NearImageArgument nearImage = client.graphQL().arguments().nearImageArgBuilder()
      .imageFile(imageFile)
      .build();

    Result<GraphQLResponse> result = client.graphQL().get()
      .withClassName(className)
      .withFields(image)
      .withNearImage(nearImage)
      .run();

    if (result.hasErrors()) {
      System.out.println(result.getError());
      return;
    }
    System.out.println(result.getResult());
  }
}

Curl

$ echo '{
  "query": "{
    Get {
      FashionItem(nearImage: {
        image: "/9j/4AAQSkZJRgABAgE..."
      }) {
        image
      }
    }
  }"
}' | curl \
    -X POST \
    -H 'Content-Type: application/json' \
    -d @- \
    http://localhost:8080/v1/graphql

More resources

If you can't find the answer to your question here, please look at the:

1. Frequently Asked Questions. Or,
2. Knowledge base of old issues. Or,
3. For questions: Stackoverflow. Or,
4. For more involved discussion: Weaviate Community Forum. Or,
5. We also have a Slack channel.