Weaviate Transformation Agent: Usage
Sign up here for notifications on Weaviate Agents, or visit this page to see the latest updates and provide feedback.
The Weaviate Transformation Agent is designed to modify data in Weaviate in place. While the Agent is in technical preview, do not use it in a production environment. The Agent may not work as expected, and the data in your Weaviate instance may be affected in unexpected ways.
The Weaviate Transformation Agent is an agentic service designed to augment and transform data using generative models. Use the Transformation Agent to append new properties and/or update existing properties of data on existing objects in Weaviate.
This can help you to improve the quality of your objects in your Weaviate collections, ready for further use in your applications.
This page describes how to use the Weaviate Transformation Agent to transform and augment your data in Weaviate.
The official changelog for Weaviate Agents can be found here. If you have feedback, such as feature requests, bug reports or questions, please submit them here, where you will be able to see the status of your feedback and vote on others' feedback.
Prerequisites
Weaviate instance
This Agent is available exclusively for use with a Weaviate Cloud instance.
Refer to the Weaviate Cloud documentation for more information on how to set up a Weaviate Cloud instance.
You can try this Weaviate Agent with a free Sandbox instance on Weaviate Cloud.
Client library
At this time, this Agent is available only for Python. Support for other languages will be added in the future.
You can install the Weaviate client library with the optional agents extras to use Weaviate Agents. This will install the weaviate-agents package along with the weaviate-client package.
Install the client library using the following command:
- Python
pip install -U weaviate-client[agents]
Troubleshooting: Force pip to install the latest version
For existing installations, even pip install -U "weaviate-client[agents]" may not upgrade weaviate-agents to the latest version. If this occurs, additionally try to explicitly upgrade the weaviate-agents package:
pip install -U weaviate-agents
Or install a specific version:
pip install -U weaviate-agents==0.5.0
Usage
To use the Transformation Agent, instantiate it with the required inputs and start the operations.
Transformation operations are asynchronous. Each operation will return a workflow ID to the user. Use this ID to check its status.
Example usage is shown below.
Prerequisites
The Transformation Agent is tightly integrated with Weaviate Cloud. As a result, the Transformation Agent is available exclusively for use with a Weaviate Cloud instance, and a supported version of the client library.
Connect to Weaviate
You must connect to the Weaviate Cloud instance to use the Transformation Agent. Connect to the Weaviate Cloud instance using the Weaviate client library.
- Python
# [🚧 CAUTION 🚧] DO NOT USE ON PRODUCTION DATA.
# The Transformation Agent will modify your data in place.
# While the Transformation Agent is in technical preview,
# it is recommended to only use it on test data.
import os
import weaviate
from weaviate.classes.init import Auth
headers = {
# END SimpleTransformationAgentExample
"X-Cohere-API-Key": os.environ.get("COHERE_API_KEY"),
# START SimpleTransformationAgentExample
# Provide your required API key(s), e.g. Cohere, OpenAI, etc. for the configured vectorizer(s)
"X-INFERENCE-PROVIDER-API-KEY": os.environ.get("YOUR_INFERENCE_PROVIDER_KEY", ""),
}
client = weaviate.connect_to_weaviate_cloud(
cluster_url=os.environ.get("WEAVIATE_URL"),
auth_credentials=Auth.api_key(os.environ.get("WEAVIATE_API_KEY")),
headers=headers,
)
Define transformation operations
You must define transformation operations before starting the Transformation Agent. Define operations with the following information:
- Operation type
- Target property name
- The properties to be used as context
- Instructions
- (For new properties) The data type of the new property
Example operations are shown below.
Append new properties to data
New properties can be appended to objects based on existing property values and user instructions.
- Python
# [🚧 CAUTION 🚧] DO NOT USE ON PRODUCTION DATA.
# The Transformation Agent will modify your data in place.
# While the Transformation Agent is in technical preview,
# it is recommended to only use it on test data.
from weaviate.agents.classes import Operations
add_french_abstract = Operations.append_property(
property_name="french_abstract",
data_type=DataType.TEXT,
view_properties=["abstract"],
instruction="Translate the abstract to French",
)
add_nlp_relevance = Operations.append_property(
property_name="nlp_relevance",
data_type=DataType.INT,
view_properties=["abstract"],
instruction="""Give a score from 0-10 based on how relevant the abstract is to Natural Language Processing.
The scale is from 0 (not relevant at all) to 10 (very relevant)""",
)
is_survey_paper = Operations.append_property(
property_name="is_survey_paper",
data_type=DataType.BOOL,
view_properties=["abstract"],
instruction="""Determine if the paper is a "survey".
A paper is considered survey it's a surveys existing techniques, and not if it presents novel techniques""",
)
Update existing properties
Values of existing properties can be updated on objects based on existing property values and user instructions.
- Python
# [🚧 CAUTION 🚧] DO NOT USE ON PRODUCTION DATA.
# The Transformation Agent will modify your data in place.
# While the Transformation Agent is in technical preview,
# it is recommended to only use it on test data.
from weaviate.agents.classes import Operations
update_topics = Operations.update_property(
property_name="topics",
view_properties=["abstract", "title"],
instruction="""Create a list of topic tags based on the title and abstract.
Topics should be distinct from eachother. Provide a maximum of 3 topics.
Group similar topics under one topic tag.""",
)
Start the transformation operations
To start the transformation operations, instantiate the Transformation Agent with the required inputs and start the operations.
Instantiate the Transformation Agent with the Weaviate client, the target collection name, and the list of transformation operations.
- Python
# [🚧 CAUTION 🚧] DO NOT USE ON PRODUCTION DATA.
# The Transformation Agent will modify your data in place.
# While the Transformation Agent is in technical preview,
# it is recommended to only use it on test data.
from weaviate.agents.transformation import TransformationAgent
agent = TransformationAgent(
client=client,
collection="ArxivPapers",
operations=[
add_french_abstract,
add_nlp_relevance,
is_survey_paper,
update_topics,
],
)
response = agent.update_all()
print(response) # The response is a TransformationResponse object, including the workflow_id
Monitor job status
You can use the workflow ID to monitor the status of each transformation operation.
- Python
# [🚧 CAUTION 🚧] DO NOT USE ON PRODUCTION DATA.
# The Transformation Agent will modify your data in place.
# While the Transformation Agent is in technical preview,
# it is recommended to only use it on test data.
print(agent.get_status(workflow_id=response.workflow_id)) # Use the workflow_id to check the status of each operation
Limitations & Troubleshooting
Sign up here for notifications on Weaviate Agents, or visit this page to see the latest updates and provide feedback.
Usage limits
At this stage, there is a limit of 10,000 Transformation Agent operations per day per Weaviate Cloud organization.
Note that this limit is per individual operations. In other words, running a Transformation Agent with 4 operations on a collection of 2,500 objects would max out the limit for that day.
This limit may change in future versions of the Transformation Agent.
Race condition on multiple operations
When multiple transformation operations are initiated on the same collection, it is possible for a race condition to occur, overwriting the results of one operation with the results of another.
This can be avoided by ensuring that only one operation is performed on a collection at a time. If you need to perform multiple operations on the same collection, ensure that the operations are performed sequentially.
You can do this by using the workflow ID of the previous operation to monitor its status before starting the next operation.
This will be addressed in future versions of the Transformation Agent.
Questions and feedback
The official changelog for Weaviate Agents can be found here. If you have feedback, such as feature requests, bug reports or questions, please submit them here, where you will be able to see the status of your feedback and vote on others' feedback.
If you have any questions or feedback, let us know in the user forum.