Skip to main content

ResNet Image Vectorizer

CLIP recommended for new projects

For new projects, we recommend using the Transformers multi-modal integration module instead of img2vec-neural. This uses CLIP models, which uses a more modern model architecture than resnet models used in img2vec-neural. CLIP models are also multi-modal, meaning they can handle both images and text and therefore applicable to a wider range of use cases.

Overview

The img2vec-neural module enables Weaviate to obtain vectors locally images using a resnet50 model.

img2vec-neural encapsulates the model in a Docker container, which allows independent scaling on GPU-enabled hardware while keeping Weaviate on CPU-only hardware, as Weaviate is CPU-optimized.

Key notes:

  • This module is not available on Weaviate Cloud (WCD).
  • Enabling this module will enable the nearImage search operator.
  • Model encapsulated in a Docker container.
  • This module is not compatible with Auto-schema. You must define your classes manually as shown below.

Weaviate instance configuration

Not applicable to WCD

This module is not available on Weaviate Cloud.

Docker Compose file

To use img2vec-neural, you must enable it in your Docker Compose file (e.g. docker-compose.yml).

Use the configuration tool

While you can do so manually, we recommend using the Weaviate configuration tool to generate the Docker Compose file.

Parameters

Weaviate:

  • ENABLE_MODULES (Required): The modules to enable. Include img2vec-neural to enable the module.
  • DEFAULT_VECTORIZER_MODULE (Optional): The default vectorizer module. You can set this to img2vec-neural to make it the default for all classes.
  • IMAGE_INFERENCE_API (Required): The URL of the inference container.

Inference container:

  • image (Required): The image name of the inference container. (e.g. semitechnologies/img2vec-pytorch:resnet50 or semitechnologies/img2vec-keras:resnet50)

Example

This configuration enables img2vec-neural, sets it as the default vectorizer, and sets the parameters for the Docker container, including setting it to use img2vec-pytorch:resnet50 image.

services:
weaviate:
image: cr.weaviate.io/semitechnologies/weaviate:1.26.6
restart: on-failure:0
ports:
- 8080:8080
- 50051:50051
environment:
QUERY_DEFAULTS_LIMIT: 20
AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
PERSISTENCE_DATA_PATH: "./data"
ENABLE_MODULES: 'img2vec-neural'
DEFAULT_VECTORIZER_MODULE: 'img2vec-neural'
IMAGE_INFERENCE_API: "http://i2v-neural:8080"
CLUSTER_HOSTNAME: 'node1'
i2v-neural:
image: cr.weaviate.io/semitechnologies/img2vec-pytorch:resnet50
...

Alternative: Run a separate container

As an alternative, you can run the inference container independently from Weaviate. To do so, you can:

  • Enable img2vec-neural in your Docker Compose file,
  • Omit img2vec-neural parameters,
  • Run the inference container separately, e.g. using Docker, and
  • Set IMAGE_INFERENCE_API to the URL of the inference container.

Then, for example if Weaviate is running outside of Docker, set IMAGE_INFERENCE_API="http://localhost:8000". Alternatively if Weaviate is part of the same Docker network, e.g. because they are part of the same docker-compose.yml file, you can use Docker networking/DNS, such as IMAGE_INFERENCE_API=http://i2v-neural:8080.

For example, can spin up an inference container with the following command:

docker run -itp "8000:8080" semitechnologies/img2vec-neural:resnet50-61dcbf8

Class configuration

You can configure how the module will behave in each class through the Weaviate schema.

Vectorization settings

You can set vectorizer behavior using the moduleConfig section under each class and property:

Class-level

  • vectorizer - what module to use to vectorize the data.
  • imageFields - property names for images to be vectorized

Property-level

  • dataType - the data type of the property. For use in imageFields, must be set to blob.

Example

The following example class definition sets the img2vec-neural module as the vectorizer for the class FashionItem. It also sets:

  • image property as a blob datatype and as the image field,
{
"classes": [
{
"class": "FashionItem",
"description": "Each example is a 28x28 grayscale image, associated with a label from 10 classes.",
"vectorizer": "img2vec-neural",
"moduleConfig": {
"img2vec-neural": {
"imageFields": [
"image"
]
}
},
"properties": [
{
"dataType": [
"blob"
],
"description": "Grayscale image",
"name": "image"
},
{
"dataType": [
"number"
],
"description": "Label number for the given image.",
"name": "labelNumber"
},
{
"dataType": [
"text"
],
"description": "label name (description) of the given image.",
"name": "labelName"
}
],
}
]
}
All blob properties must be in base64-encoded data.

Adding blob data objects

Any blob property type data must be base64 encoded. To obtain the base64-encoded value of an image for example, you can use the helper methods in the Weaviate clients or run the following command:

cat my_image.png | base64

Additional search operator

The img2vec-neural vectorizer module will enable the nearImage search operator.

Usage example

NearImage

import weaviate

client = weaviate.Client("http://localhost:8080")

nearImage = {"image": "/9j/4AAQSkZJRgABAgE..."}

result = (
client.query
.get("FashionItem", "image")
.with_near_image(nearImage)
.do()
)

print(result)

About the model

resnet50 is a residual convolutional neural network with 25.5 million parameters trained on more than a million images from the ImageNet database. As the name suggests, it has a total of 50 layers: 48 convolution layers, 1 MaxPool layer and 1 Average Pool layer.

Available img2vec-neural models

There are two different inference models you can choose from. Depending on your machine (arm64 or other) and whether you prefer to use multi-threading to extract feature vectors or not, you can choose between keras and pytorch. There are no other differences between the two models.

  • resnet50 (keras):
    • Supports amd64, but not arm64.
    • Does not currently support CUDA
    • Supports multi-threaded inference
  • resnet50 (pytorch):
    • Supports both amd64 and arm64.
    • Supports CUDA
    • Does not support multi-threaded inference

Model license(s)

The img2vec-neural module uses the resnet50 model.

It is your responsibility to evaluate whether the terms of its license(s), if any, are appropriate for your intended use.

Questions and feedback

If you have any questions or feedback, let us know in the user forum.