img2vec-neural
Introductionโ
This module vectorizes images using neural networks. Pre-trained modules can be used. resnet50
is the first model that is supported, other models will be released soon. resnet50
is a residual convolutional neural network with 25.5 million parameters trained on more than a million images from the ImageNet database. As the name suggests, it has a total of 50 layers: 48 convolution layers, 1 MaxPool layer and 1 Average Pool layer.
Available img2vec-neural modelsโ
There are two different inference models you can choose from. Depending on your machine (arm64
or other) and whether you prefer to use multi-threading to extract feature vectors or not, you can choose between keras
and pytorch
. There are no other differences between the two models.
resnet50
(keras
):- Supports
amd64
, but notarm64
. - Does not support
CUDA
(yet) - Supports multi-threaded inference
- Supports
resnet50
(pytorch
):- Supports both
amd64
andarm64
. - Supports
CUDA
- Does not support multi-threaded inference
- Supports both
How to enable in Weaviateโ
Note: you can also use the Weaviate configuration tool.
Docker-compose fileโ
You can find an example Docker-compose file below, which will spin up Weaviate with the image vectorization module. This example spins up a Weaviate with only one vectorization module, the img2vec-neural
module of pytorch
with the resnet50
model.
---
version: '3.4'
services:
weaviate:
command:
- --host
- 0.0.0.0
- --port
- '8080'
- --scheme
- http
image: semitechnologies/weaviate:1.19.6
ports:
- 8080:8080
restart: on-failure:0
environment:
IMAGE_INFERENCE_API: "http://i2v-neural:8080"
QUERY_DEFAULTS_LIMIT: 25
AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
DEFAULT_VECTORIZER_MODULE: 'img2vec-neural'
ENABLE_MODULES: 'img2vec-neural'
CLUSTER_HOSTNAME: 'node1'
i2v-neural:
image: semitechnologies/img2vec-pytorch:resnet50
You can substitute semitechnologies/img2vec-pytorch:resnet50
with semitechnologies/img2vec-keras:resnet50
in case you want to use the keras
module.
You can combine the image vectorization module with a text vectorization module. In the following example, we use both the text2vec-contextionary
module as well as the img2vec-neural
module. We set text2vec-contextionary
to default vectorizer module, which means we need to specify in a data schema when we want a class to be vectorized as with the img2vec-neural
module instead of the text2vec-contextionary
module.
Other methodsโ
If you prefer not to use Docker-compose (but instead for example Kubernetes in a production setup), make you can use the img2vec-neural
module after taking the following steps:
- Enable the
img2vec-neural
module. Make sure you set theENABLE_MODULES=img2vec-neural
environment variable. This can be combined with a text vectorization module:ENABLE_MODULES: 'text2vec-contextionary,img2vec-neural'
. Additionally, you can make one of the modules the default vectorizer, so you don't have to specify it on each schema class:DEFAULT_VECTORIZER_MODULE=text2vec-contextionary
- Run the
img2vec-neural
module, for example usingdocker run -itp "8000:8080" semitechnologies/img2vec-neural:resnet50-61dcbf8
. - Tell Weaviate where to find the inference module. Set the Weaviate environment variable
IMAGE_INFERENCE_API
to where your inference container is running, for exampleIMAGE_INFERENCE_API="http://localhost:8000"
- You can now use Weaviate normally and all vectorization of images during import and search time will be done with the selected image vectorization model (given that the schema is configured correctly).
Schema configurationโ
You can specify to use the image vectorizer per class in the schema. To find details on how to configure a data schema, go here. When you set the vectorizer
of a class to img2vec-neural
, only the property fields that are specified in the moduleConfig
will be taken into the computation of the vector.
When adding a class with vectorizer type img2vec-neural
, the configuration must contain information about which field holds the image. The dataType of the imageFields
should be blob
. This can be achieved with the following config in a class object:
"moduleConfig": {
"img2vec-neural": {
"imageFields": [
"image"
]
}
}
If multiple fields are specified, the module will vectorize them separately and use the mean vector of both.
A full example of a class using the img2vec-neural
module is shown below. This module makes use of the blob
dataType.
{
"classes": [
{
"class": "FashionItem",
"description": "Each example is a 28x28 grayscale image, associated with a label from 10 classes.",
"moduleConfig": {
"img2vec-neural": {
"imageFields": [
"image"
]
}
},
"properties": [
{
"dataType": [
"blob"
],
"description": "Grayscale image",
"name": "image"
},
{
"dataType": [
"number"
],
"description": "Label number for the given image.",
"name": "labelNumber"
},
{
"dataType": [
"text"
],
"description": "label name (description) of the given image.",
"name": "labelName"
}
],
"vectorIndexType": "hnsw",
"vectorizer": "img2vec-neural"
}
]
}
Other properties, for example the name of the image that is given in another field, will not be taken into consideration. This means that you can only find the image with semantic search by another image, data object, or raw vector. Semantic search of images by text field (using nearText
) is not possible, because this requires a text2vec
vectorization module. Multiple modules cannot be combined at class level yet (might become possible in the future, since image-text-combined transformers
exists). We recommend to use one of the following workarounds:
- Best practice for multi-module search: create an image class and a text class in which you refer to each other by cross-reference. This way you can always hop along the reference and search either by "text labels" (using a
text2vec-....
module) or by image (using aimg2vec-...
module). - If you don't want to create multiple classes, you are limited to using a
where
filter to find images by other search terms than animage
,data object
, orvector
. Awhere
filter does not use semantic features of a module.
Adding image data objectsโ
When adding data, make sure that the specified fields are filled with valid image data (e.g. jpg, png, etc.), encoded as base64
(string) value in the property that has a blob
dataType. The blob type itself (see below) requires that all blobs are base64 encoded. To obtain the base64-encoded value of an image, you can run the following command - or use the helper methods in the Weaviate clients - to do so:
cat my_image.png | base64
You can then import data with blob
dataType in to Weaviate as follows:
- Python
- JavaScript
- Go
- Java
- Curl
import weaviate
client = weaviate.Client("http://localhost:8080")
data_properties = {
"labelName": "Mickey Mouse T-shirt",
"image": "iVBORw0KGgoAAAANS..."
}
result = (
client.data_object
.create(data_properties, "FashionItem", "36ddd591-2dee-4e7e-a3cc-eb86d30a4302")
)
print(result)
const weaviate = require('weaviate-client');
const client = weaviate.client({
scheme: 'http',
host: 'localhost:8080',
});
client.data
.creator()
.withClassName('FashionItem')
.withId('36ddd591-2dee-4e7e-a3cc-eb86d30a4302')
.withProperties({
'labelName': 'Mickey Mouse T-shirt'
'image': 'iVBORw0KGgoAAAANS...'
})
.do()
.then(res => {
console.log(res)
})
.catch(err => {
console.error(err)
});
package main
import (
"context"
"fmt"
"github.com/weaviate/weaviate-go-client/v4/weaviate"
)
func main() {
cfg := weaviate.Config{
Host: "localhost:8080",
Scheme: "http",
}
client, err := weaviate.NewClient(cfg)
if err != nil {
panic(err)
}
dataSchema := map[string]interface{}{
"labelName": "Mickey Mouse T-shirt",
"image": "iVBORw0KGgoAAAANS...",
}
created, err := client.Data().Creator().
WithClassName("FashionItem").
WithID("36ddd591-2dee-4e7e-a3cc-eb86d30a4302").
WithProperties(dataSchema).
Do(context.Background())
if err != nil {
panic(err)
}
fmt.Printf("%v", created)
}
package io.weaviate;
import java.util.HashMap;
import java.util.Map;
import io.weaviate.client.Config;
import io.weaviate.client.WeaviateClient;
import io.weaviate.client.base.Result;
import io.weaviate.client.v1.data.model.WeaviateObject;
public class App {
public static void main(String[] args) {
Config config = new Config("http", "localhost:8080");
WeaviateClient client = new WeaviateClient(config);
Map<String, Object> dataSchema = new HashMap<>();
dataSchema.put("labelName", "Mickey Mouse T-shirt");
dataSchema.put("image", "iVBORw0KGgoAAAANS...");
Result<WeaviateObject> result = client.data().creator()
.withClassName("FashionItem")
.withID("36ddd591-2dee-4e7e-a3cc-eb86d30a4302")
.withProperties(dataSchema)
.run();
if (result.hasErrors()) {
System.out.println(result.getError());
return;
}
System.out.println(result.getResult());
}
}
$ curl \
-X POST \
-H "Content-Type: application/json" \
-d '{
"class": "FashionItem",
"id": "36ddd591-2dee-4e7e-a3cc-eb86d30a4302",
"properties": {
"labelName": "Mickey Mouse T-shirt"
"image": "iVBORw0KGgoAAAANS..."
}
}' \
http://localhost:8080/v1/objects
Additional GraphQL API parametersโ
nearImage searchโ
At search time you can use the "standard" vector-search operators, such as nearVector
and nearObject
. But in addition, you can also vectorize a new image at search time and search by using the image's vector. To do so, you can use the nearImage search operator like so:
- GraphQL
- Python
- JavaScript
- Go
- Java
- Curl
{
Get {
FashionItem(nearImage: {
image: "/9j/4AAQSkZJRgABAgE..."
}) {
image
}
}
}
import weaviate
client = weaviate.Client("http://localhost:8080")
nearImage = {"image": "/9j/4AAQSkZJRgABAgE..."}
result = (
client.query
.get("FashionItem", "image")
.with_near_image(nearImage)
.do()
)
print(result)
const weaviate = require('weaviate-client');
const client = weaviate.client({
scheme: 'http',
host: 'localhost:8080',
});
client.graphql
.get()
.withClassName('FashionItem')
.withFields('image')
.withNearImage({image: '/9j/4AAQSkZJRgABAgE...'})
.do()
.then(res => {
console.log(res)
})
.catch(err => {
console.error(err)
});
package main
import (
"context"
"fmt"
"github.com/weaviate/weaviate-go-client/v4/weaviate"
"github.com/weaviate/weaviate-go-client/v4/weaviate/graphql"
)
func main() {
cfg := weaviate.Config{
Host: "localhost:8080",
Scheme: "http",
}
client, err := weaviate.NewClient(cfg)
if err != nil {
panic(err)
}
className := "FashionItem"
image := graphql.Field{Name: "image"}
nearImage := client.GraphQL().NearImageArgBuilder().WithImage("/9j/4AAQSkZJRgABAgE...")
ctx := context.Background()
result, err := client.GraphQL().Get().
WithClassName(className).
WithFields(image).
WithNearImage(nearImage).
Do(ctx)
if err != nil {
panic(err)
}
fmt.Printf("%v", result)
}
package io.weaviate;
import io.weaviate.client.Config;
import io.weaviate.client.WeaviateClient;
import io.weaviate.client.base.Result;
import io.weaviate.client.v1.graphql.model.GraphQLResponse;
import io.weaviate.client.v1.graphql.query.argument.NearImageArgument;
import io.weaviate.client.v1.graphql.query.fields.Field;
public class App {
public static void main(String[] args) {
Config config = new Config("http", "localhost:8080");
WeaviateClient client = new WeaviateClient(config);
String className = "FashionItem";
Field image = Field.builder().name("image").build();
NearImageArgument nearImage = client.graphQL().arguments().nearImageArgBuilder()
.image("/9j/4AAQSkZJRgABAgE...")
.build();
Result<GraphQLResponse> result = client.graphQL().get()
.withClassName(className)
.withFields(image)
.withNearImage(nearImage)
.run();
if (result.hasErrors()) {
System.out.println(result.getError());
return;
}
System.out.println(result.getResult());
}
}
$ echo '{
"query": "{
Get {
FashionItem(nearImage: {
image: "/9j/4AAQSkZJRgABAgE..."
}) {
image
}
}
}"
}' | curl \
-X POST \
-H 'Content-Type: application/json' \
-d @- \
http://localhost:8080/v1/graphql
Alternatively, you can use a helper function in the Python, Java or Go client (not with the JavaScript client). With an encoder function, you can input your image as png
file, and the helper function encodes this to a base64
encoded value.
- GraphQL
- Python
- JavaScript
- Go
- Java
- Curl
# GraphQL doesn't support png->base64 encoding, so please use a base64 encoded image in your query
{
Get {
FashionItem(nearImage: {
image: "/9j/4AAQSkZJRgABAgE..."
}) {
image
}
}
}
import weaviate
client = weaviate.Client("http://localhost:8080")
nearImage = {"image": "my_image_path.png"}
result = (
client.query
.get("FashionItem", "image")
.with_near_image(nearImage, encode=True)
.do()
)
print(result)
## OR use the weaviate.utils function:
client = weaviate.Client("http://localhost:8080")
encoded_image = weaviate.util.image_encoder_b64("my_image_path.png")
nearImage = {"image": "encoded_image"}
result = (
client.query
.get("FashionItem", "image")
.with_near_image(nearImage)
.do()
)
print(result)
// The JavaScript client doesn't support image encoding via a helper function, you need to give the input image as base64 format yourself
const weaviate = require('weaviate-client');
const client = weaviate.client({
scheme: 'http',
host: 'localhost:8080',
});
client.graphql
.get()
.withClassName('FashionItem')
.withFields('image')
.withNearImage({image: '/9j/4AAQSkZJRgABAgE...'})
.do()
.then(res => {
console.log(res)
})
.catch(err => {
console.error(err)
});
package main
import (
"context"
"fmt"
"os"
"github.com/weaviate/weaviate-go-client/v4/weaviate"
"github.com/weaviate/weaviate-go-client/v4/weaviate/graphql"
)
func main() {
cfg := weaviate.Config{
Host: "localhost:8080",
Scheme: "http",
}
client, err := weaviate.NewClient(cfg)
if err != nil {
panic(err)
}
className := "FashionItem"
image := graphql.Field{Name: "image"}
filename := "my_image_path.png"
file, err := os.Open(filename)
if err != nil {
panic(err)
}
nearImage := client.GraphQL().NearImageArgBuilder().WithReader(file)
ctx := context.Background()
result, err := client.GraphQL().Get().
WithClassName(className).
WithFields(image).
WithNearImage(nearImage).
Do(ctx)
if err != nil {
panic(err)
}
fmt.Printf("%v", result)
}
package io.weaviate;
import java.io.File;
import io.weaviate.client.Config;
import io.weaviate.client.WeaviateClient;
import io.weaviate.client.base.Result;
import io.weaviate.client.v1.graphql.model.GraphQLResponse;
import io.weaviate.client.v1.graphql.query.argument.NearImageArgument;
import io.weaviate.client.v1.graphql.query.fields.Field;
public class App {
public static void main(String[] args) {
Config config = new Config("http", "localhost:8080");
WeaviateClient client = new WeaviateClient(config);
File imageFile = new File("my_image_path.png");
String className = "FashionItem";
Field image = Field.builder().name("image").build();
NearImageArgument nearImage = client.graphQL().arguments().nearImageArgBuilder()
.imageFile(imageFile)
.build();
Result<GraphQLResponse> result = client.graphQL().get()
.withClassName(className)
.withFields(image)
.withNearImage(nearImage)
.run();
if (result.hasErrors()) {
System.out.println(result.getError());
return;
}
System.out.println(result.getResult());
}
}
$ echo '{
"query": "{
Get {
FashionItem(nearImage: {
image: "/9j/4AAQSkZJRgABAgE..."
}) {
image
}
}
}"
}' | curl \
-X POST \
-H 'Content-Type: application/json' \
-d @- \
http://localhost:8080/v1/graphql
More resourcesโ
If you can't find the answer to your question here, please look at the:
- Frequently Asked Questions. Or,
- Knowledge base of old issues. Or,
- For questions: Stackoverflow. Or,
- For more involved discussion: Weaviate Community Forum. Or,
- We also have a Slack channel.