Skip to main content

Better automated testing with Embedded Weaviate

· 10 min read
Dan Dascalescu

Automated testing for Weaviate applications

As a software engineer with experience in test automation, I firmly believe in Test-Driven Development, and more specifically, incorporating integration testing from the very early stages of developing an application.

But you probably know that writing tests is quite a task in itself, and in many cases running them can also be a chore. For example, the test suite may need to set up and tear down a separate service such as a database, which can be time-consuming and error-prone.

I'm here to tell you that it doesn't have to be that way. In this article, I'll show you how to make testing easier with Embedded Weaviate, and other tips for better automated testing.

And doing so, you might just discover the hidden gem that is the value provided by adding tests to your applications.

Testing and Weaviate​

In this article, we will focus on integration tests. Integrated testing is an important part of the development process, and especially so for complex applications. Weaviate-based apps usually fall in this category.

For one, Weaviate must interact with the application in a variety of ways. And additionally, Weaviate often interacts with external services such as vectorizers or LLMs.

Such complexity makes it important to test the application as a whole, and not just its individual components. This complexity also means that arranging the test suite can be cumbersome with a variety of moving parts that need to be set up and torn down.

Embedded Weaviate makes one part of this puzzle much easier, since Weaviate can be instantiated directly from the client. The following is all you need to do to start a Weaviate server:

import weaviate
import os

# Instantiate the embedded Weaviate client and pass the OpenAI API key
client = weaviate.Client(
    embedded_options=weaviate.embedded.EmbeddedOptions(),
    additional_headers={
        'X-OpenAI-Api-Key': os.environ['OPENAI_API_KEY']  # Replace with your OpenAI API key
    }
)

This is not only useful for new contributors to the project, but also for experienced developers. Starting anew as a new contributor, or working from a different machine on occasion, can be a hassle. With Embedded Weaviate, you can just run the test suite and be done with it.

But Embedded Weaviate is not the only way to make testing easier. In the following sections, we'll look at other ways to make testing easier, and how to make the most of Embedded Weaviate.

Scoping tests​

While you may be familiar with tests and integration tests in general, here are some specific suggestions for Weaviate-powered applications:

  • Whether to test search quality: This depends primarily on the model used for vectorization, such as by a Weaviate vectorizer module. We suggest evaluating models separately, but not tested as a part of the application.
  • Focus on interactions with the inference provider: Search itself is a core Weaviate functionality that we can trust. So, we suggest any integration tests focus on the interaction with the inference provider. For example,
    • is the vectorization model the expected one?
    • if switching to a different inference provider or model, does the application still function as expected?
  • Other common issues to test include:
    • Connection or authentication issues with the inference provider
    • Incomplete or incorrect data imports
    • Specifying the vector correctly when bringing your own vectors
    • Data definition issues, like invalid class names, properties, or data types

Testing with embedded Weaviate​

Set up​

Embedded Weaviate lets us spawn a Weaviate server instance from the client, and automatically tear it down when the client terminates. The data is persisted between sessions, so we recommend deleting your data before each test.

Here's how to instantiate an embedded Weaviate server and perform this cleanup:

Install dependencies

If you have yet to install the required dependencies, run the following command:

pip install -U weaviate-client pytest

Then, save the code as embedded_test.py and run pytest.

import weaviate
import os

# Instantiate the embedded Weaviate client and pass the OpenAI API key
client = weaviate.Client(
    embedded_options=weaviate.embedded.EmbeddedOptions(),
    additional_headers={
        'X-OpenAI-Api-Key': os.environ['OPENAI_API_KEY']  # Replace with your OpenAI API key
    }
)
# Client is now ready to accept requests

# Clean slate for local testing (GitHub Actions VMs start fresh) because Weaviate data is persisted in embedded mode.
client.schema.delete_all()

Now, let's walk through some examples of how you might construct integration tests with Embedded Weaviate.

Our first test​

As a simple example, let's create a class and then test that it was created correctly.

We'll create a class for question & answer objects from the game show Jeopardy!, by specifying its name and the vectorizer (text2vec-openai).

Here, the integration test will consist of checking that the class was created with the expected default OpenAI vectorization model type, ada.

class_name = 'JeopardyQuestion'
class_definition = {
    'class': class_name,
    'vectorizer': 'text2vec-openai',
}

client.schema.create_class(class_definition)

# Test
retrieved_definition = client.schema.get(class_name)
assert retrieved_definition['moduleConfig']['text2vec-openai']['model'] == 'ada'
When might I use tests like these?

Although this is a simple test, you can imagine that if you have tens, or even hundreds of classes, tests like these can save you a lot of time and effort. And, if you're working with a team, you can be sure that everyone is on the same page about the expected schema.

All we've done so far (instantiate and connect to Weaviate and OpenAI, perform cleanup, create the class and test creation), was done with very little code, thanks to Embedded Weaviate. Next, let's see how we might test imports.

Testing imports​

One particularly common issue we see is skipped objects during import due to rate limits from the vectorization provider. So, let's see how we might test that all objects were imported correctly.

In this section, we'll import a small subset (100 objects) of the original Jeopardy dataset. As always, we'll use batching for optimal speed.

For very large files

While we load the JSON into memory here, you can use other methods such as streaming for very large JSON or CSV files.

The test is simple; it verifies that all specified objects have been imported by performing an object count and checking it is 100.

Download jeopardy_100.json

import json

with open('jeopardy_100.json') as f:
    data = json.load(f)

# Context manager for batch import
with client.batch as batch:
    # Build data objects & add to batch
    for i, obj in enumerate(data):
        batch.add_data_object(
            data_object={
                'question': obj['Question'],
                'answer': obj['Answer'],
            },
            class_name=class_name,
        )
    if i % 20 == 0:
        print(f'Imported {i} objects...')

# Test all objects were imported
response = client.query.aggregate(class_name).with_meta_count().do()
actual_count = response['data']['Aggregate'][class_name][0]['meta']['count']
assert actual_count == 100, f'Expected 100 imported objects but got {actual_count}'
When might I use tests like these?

Such a test would provide a simple, repeatable, way of ensuring that all objects were imported correctly.

And now that all data has been imported, let's see how we might test queries.

Testing queries​

Semantic (nearText) searches may be one of the most common (if not the most common) searches our users perform.

So let's see how we might test semantic searches. A semantic search requires vectorizing the query, so a test will validate the integration with the vectorizer (text2vec-openai in this case).

We'll run a query for "chemistry" and check that the top result is about "sodium".

Will the top result always be the same?

Depending on your application and usage, you may find that the top result changes over time - for example, if your data changes over time. In this case, we will assume that the top result is immutable, or that you will update the specific test over time.

result = (
    client.query
    .get('JeopardyQuestion', ['question', 'answer'])
    .with_near_text({'concepts': 'chemistry'})
    .with_limit(1)
    .do()
)

# Test
assert 'sodium' in result['data']['Get'][class_name][0]['answer']
When might I use tests like these?

A test like this can be used to ensure that the vectorizer is working as expected, and that the data has been imported correctly. For example - if the top result was something completely unintuitive and wildly dissimilar to the concept of "chemistry" - this might be cause to investigate further.

You could also test additional aspects, like the number of results returned, or the order of results.

End-to-end code​

So far, we've seen how to test the following:

  • Create the collection
  • Import data
  • Semantic search functionality

The code below brings together the setup and tests we've implemented so far - if you haven't done so yet, try running it yourself 😉.

End-to-end code
import weaviate
import os

# Instantiate the embedded Weaviate client and pass the OpenAI API key
client = weaviate.Client(
    embedded_options=weaviate.embedded.EmbeddedOptions(),
    additional_headers={
        'X-OpenAI-Api-Key': os.environ['OPENAI_API_KEY']  # Replace with your OpenAI API key
    }
)
# Client is now ready to accept requests

# Clean slate for local testing (GitHub Actions VMs start fresh) because Weaviate data is persisted in embedded mode.
client.schema.delete_all()
# Client connected and schema cleared


# Create the class
class_name = 'JeopardyQuestion'
class_definition = {
    'class': class_name,
    'vectorizer': 'text2vec-openai',
}

client.schema.create_class(class_definition)

# Test
retrieved_definition = client.schema.get(class_name)
assert retrieved_definition['moduleConfig']['text2vec-openai']['model'] == 'ada'
# Class created successfully


# Import objects from the JSON file
import json

with open('jeopardy_100.json') as f:
    data = json.load(f)

# Context manager for batch import
with client.batch as batch:
    # Build data objects & add to batch
    for i, obj in enumerate(data):
        batch.add_data_object(
            data_object={
                'question': obj['Question'],
                'answer': obj['Answer'],
            },
            class_name=class_name,
        )
    if i % 20 == 0:
        print(f'Imported {i} objects...')

# Test all objects were imported
response = client.query.aggregate(class_name).with_meta_count().do()
actual_count = response['data']['Aggregate'][class_name][0]['meta']['count']
assert actual_count == 100, f'Expected 100 imported objects but got {actual_count}'
# Import completed successfully


# Run a test query
result = (
    client.query
    .get('JeopardyQuestion', ['question', 'answer'])
    .with_near_text({'concepts': 'chemistry'})
    .with_limit(1)
    .do()
)

# Test
assert 'sodium' in result['data']['Get'][class_name][0]['answer']
# Query test completed

CI / CD​

For many of us, Continuous Integration and Continuous Deployment (CI/CD) is a critical part of our development process. It allows us to automate the testing and deployment of our code, and to ensure that our code is always in a deployable state.

We use GitHub, and they offer GitHub Actions which we like at Weaviate. While we don't have space to cover it in detail (check out their docs if you're interested), we want to highlight that you can set up a YAML file to automate running of tests on GitHub. The YAML file could look something like this:

name: Run Python Automated Tests
on: [push]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Set up Python
      uses: actions/setup-python@v2
      with:
        python-version: 3.11
    - name: Install dependencies
      run: pip install weaviate-client pytest
    - name: Run tests
      run: pytest

With this step, you'll be able to run your tests on every push to the main branch.

And did you notice that we didn't have to spin up an instance of Weaviate through GitHub Actions? That's because we're using Embedded Weaviate, which means we can run our tests without worrying about the Weaviate server instance.

This might not seem like a big deal, but it lowers the barrier to entry for running tests, and makes it easier to run tests locally, or in a CI/CD pipeline.

As a result, your application will be more robust, and you'll be able to deploy with confidence.

Closing thoughts​

In this post, we've seen how to write an integration test for an application using Weaviate — Embedded Weaviate in particular.

With just a few lines of code, we are able to verify how we import a data set, vectorize it, then export the vectorized objects. The test can be extended with search, insertion, updates, deletes, and other operations that are part of the user journey.

What's more - because we were using Embedded Weaviate, the journey from the start to finish was far easier, not to mention portable.

So what are you waiting for? Try out Embedded Weaviate - and add those tests to your application that you've been putting off 😉.

What other aspects of integration testing would you like to learn about? Let us know in the comments below!

Ready to start building?​

Check out the Quickstart tutorial, and begin building amazing apps with the free trial of Weaviate Cloud Services (WCS).

GitHub

Forum

Slack

X (Twitter)

Don't want to miss another blog post?

Sign up for our bi-weekly newsletter to stay updated!
By submitting, I agree to the Terms of Service and Privacy Policy.