Skip to main content

(Bonus) Generate vectors

Since we are using custom vectors, we need to generate them ourselves.

This step is optional, as the next section shows you how to download and use the pre-generated vectors. But if you are interested in how to generate vectors, read on.

Code

This example creates embeddings for the movie dataset:

import requests
import pandas as pd


# Define a function to call the endpoint and obtain embeddings
def query(texts):
import requests
import os

model_id = "sentence-transformers/all-MiniLM-L6-v2"
hf_token = os.getenv("HUGGINGFACE_APIKEY")

api_url = f"https://api-inference.huggingface.co/pipeline/feature-extraction/{model_id}"
headers = {"Authorization": f"Bearer {hf_token}"}

response = requests.post(
api_url,
headers=headers,
json={"inputs": texts, "options": {"wait_for_model": True}},
)
return response.json()


# Get the source data
data_url = "https://raw.githubusercontent.com/weaviate-tutorials/edu-datasets/main/movies_data_1990_2024.json"
resp = requests.get(data_url)
df = pd.DataFrame(resp.json())

# Loop through the dataset to generate vectors in batches
emb_dfs = list()
src_texts = list()
for i, row in enumerate(df.itertuples(index=False)):
# Concatenate text to create a source string
src_text = "Title" + row.title + "; Overview: " + row.overview
# Add to the buffer
src_texts.append(src_text)
if (len(src_texts) == 50) or (i + 1 == len(df)): # Get embeddings in batches of 50
# Get a batch of embeddings
output = query(src_texts)
emb_df = pd.DataFrame(output)
# Add the batch of embeddings to a list
emb_dfs.append(emb_df)
# Reset the buffer
src_texts = list()


emb_df = pd.concat(emb_dfs) # Create a combined dataset

# Save the data as a CSV
emb_df.to_csv(
f"scratch/movies_data_1990_2024_embeddings.csv",
index=False,
)

This will generate a vector for each movie in the dataset, which we can use when adding the movies to Weaviate.

Explain the code

Model

We use the sentence-transformers/all-MiniLM-L6-v2 model to generate the vectors. We access it here through the Hugging Face API for convenience. You could also use the transformers library, if you would like to perform the generation locally.

Source text

We combine the movie title and overview to create a source string for the model. This is the text that the model will "translate" into a vector.

    src_text = "Title" + row.title + "; Overview: " + row.overview

Get embeddings in batches

We use a buffer to store the concatenated strings, and then get the embeddings in batches. This is a good practice to limit the number of requests to the model, and to avoid timeouts.

        output = query(src_texts)
emb_df = pd.DataFrame(output)
# Add the batch of embeddings to a list
emb_dfs.append(emb_df)

Export the embeddings

The embeddings are then saved to a file so that we can use when adding the movies to Weaviate.

emb_df.to_csv(
f"scratch/movies_data_1990_2024_embeddings.csv",
index=False,
)

Questions and feedback

If you have any questions or feedback, please let us know on our forum. For example, you can: