Database setup
Let's see we can build a Weaviate instance to support MyPrivateJournal
's requirements.
Project requirements
As a SaaS application, MyPrivateJournal
has determined that they need the following features:
- Data isolation: Each user's data should be private.
- Efficient scalability:
- As
MyPrivateJournal
grows, it should be able to handle tens or hundreds of thousands users. - It must be fast for active users, but inactive users should not consume resources.
- As
- Ease of management:
- Adding new users should be simple and fast.
- Removing users should be straightforward.
- Flexibility:
- It should be efficient for low-volume users as well as high-volume users.
- Fault tolerance:
- A node failure should not lead to complete downtime for a user.
- Developer experience:
- These features should be easy to implement and maintain.
After reviewing potential solutions, we determined that Weaviate with multi-tenant collections can meet these challenges. Let's see how we can implement a proof-of-concept (PoC) solution for MyPrivateJournal
.
Weaviate configuration
As a development setup, we will use a local Weaviate instance with Docker. This will allow us to quickly set up a Weaviate instance for development and testing.
Here is our docker-compose.yml
file:
---
services:
weaviate_anon:
command:
- --host
- 0.0.0.0
- --port
- '8080'
- --scheme
- http
image: cr.weaviate.io/semitechnologies/weaviate:1.27.4
ports:
- 8080:8080
- 50051:50051
restart: on-failure:0
environment:
QUERY_DEFAULTS_LIMIT: 25
AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
DEFAULT_VECTORIZER_MODULE: 'none'
ENABLE_API_BASED_MODULES: 'true'
ASYNC_INDEXING: 'true'
ENABLE_MODULES: 'backup-filesystem,offload-s3'
AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY:-}
AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_KEY:-}
OFFLOAD_S3_BUCKET_AUTO_CREATE: 'true'
BACKUP_FILESYSTEM_PATH: '/var/lib/weaviate/backups'
CLUSTER_HOSTNAME: 'node1'
...
What about a multi-node setup?
Great question! As you probably noticed, we are using a single-node setup here for simplicity.
But we can easily extend this to a multi-node setup by adding additional services. This will allow you to scale your Weaviate instance horizontally, and provide fault tolerance with replication.
For example, here is a multi-node setup with three nodes.
---
services:
weaviate-node-1: # Founding member service name
command:
- --host
- 0.0.0.0
- --port
- '8080'
- --scheme
- http
image: cr.weaviate.io/semitechnologies/weaviate:1.27.4
restart: on-failure:0
ports:
- "8180:8080"
- 50151:50051
environment:
AUTOSCHEMA_ENABLED: 'false'
OPENAI_APIKEY: $OPENAI_APIKEY
COHERE_APIKEY: $COHERE_APIKEY
QUERY_DEFAULTS_LIMIT: 25
QUERY_MAXIMUM_RESULTS: 10000
AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
DEFAULT_VECTORIZER_MODULE: 'none'
ASYNC_INDEXING: 'true'
ENABLE_MODULES: 'text2vec-ollama,generative-ollama,backup-filesystem,offload-s3'
ENABLE_API_BASED_MODULES: 'true'
AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY:-}
AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_KEY:-}
OFFLOAD_S3_BUCKET_AUTO_CREATE: 'true'
BACKUP_FILESYSTEM_PATH: '/var/lib/weaviate/backups'
CLUSTER_HOSTNAME: 'node1'
CLUSTER_GOSSIP_BIND_PORT: '7100'
CLUSTER_DATA_BIND_PORT: '7101'
weaviate-node-2: # Founding member service name
command:
- --host
- 0.0.0.0
- --port
- '8080'
- --scheme
- http
image: cr.weaviate.io/semitechnologies/weaviate:1.27.4
restart: on-failure:0
ports:
- "8181:8080"
- 50152:50051
environment:
AUTOSCHEMA_ENABLED: 'false'
OPENAI_APIKEY: $OPENAI_APIKEY
COHERE_APIKEY: $COHERE_APIKEY
QUERY_DEFAULTS_LIMIT: 25
QUERY_MAXIMUM_RESULTS: 10000
AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
DEFAULT_VECTORIZER_MODULE: 'none'
ASYNC_INDEXING: 'true'
ENABLE_MODULES: 'text2vec-ollama,generative-ollama,backup-filesystem,offload-s3'
ENABLE_API_BASED_MODULES: 'true'
AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY:-}
AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_KEY:-}
OFFLOAD_S3_BUCKET_AUTO_CREATE: 'true'
BACKUP_FILESYSTEM_PATH: '/var/lib/weaviate/backups'
CLUSTER_HOSTNAME: 'node2'
CLUSTER_GOSSIP_BIND_PORT: '7102'
CLUSTER_DATA_BIND_PORT: '7103'
CLUSTER_JOIN: 'weaviate-node-1:7100'
weaviate-node-3: # Founding member service name
command:
- --host
- 0.0.0.0
- --port
- '8080'
- --scheme
- http
image: cr.weaviate.io/semitechnologies/weaviate:1.27.4
restart: on-failure:0
ports:
- "8182:8080"
- 50153:50051
environment:
AUTOSCHEMA_ENABLED: 'false'
OPENAI_APIKEY: $OPENAI_APIKEY
COHERE_APIKEY: $COHERE_APIKEY
QUERY_DEFAULTS_LIMIT: 25
QUERY_MAXIMUM_RESULTS: 10000
AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
DEFAULT_VECTORIZER_MODULE: 'none'
ASYNC_INDEXING: 'true'
image: cr.weaviate.io/semitechnologies/weaviate:1.27.4
ENABLE_API_BASED_MODULES: 'true'
AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY:-}
AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_KEY:-}
OFFLOAD_S3_BUCKET_AUTO_CREATE: 'true'
BACKUP_FILESYSTEM_PATH: '/var/lib/weaviate/backups'
CLUSTER_HOSTNAME: 'node3'
CLUSTER_GOSSIP_BIND_PORT: '7104'
CLUSTER_DATA_BIND_PORT: '7105'
CLUSTER_JOIN: 'weaviate-node-1:7100'
...
Configuration highlights
You may have seen Docker configurations elsewhere (e.g. Docs, Academy). But these highlighted configurations may be new to you:
ASYNC_INDEXING
: This will enable asynchronous indexing. This is useful for high-volume data insertion, and enables us to use thedynamic
index type, which you will learn about later on.ENABLE_MODULES
: We enableoffload-s3
to demonstrate tenant offloading later on. Offloading helps us to manage inactive users' data efficiently.AWS_ACCESS_KEY_ID
andAWS_SECRET_ACCESS_KEY
: These are the AWS credentials that Weaviate will use to access the S3 bucket.OFFLOAD_S3_BUCKET_AUTO_CREATE
: This will automatically create the S3 bucket if it does not exist.
Save the file to docker-compose.yaml
, and run the following command to start Weaviate:
docker compose up
As of Weaviate v1.26.0
, tenants can only be offloaded to cold storage in AWS S3. Additional storage options may be added in future releases.
To offload a tenant, use the offload-s3
module.
Your Weaviate instance details
Once the instance is created, you can access it at http://localhost:8080
.
Now, we are ready to create a collection for MyPrivateJournal
.
Create a collection
Enable multi-tenancy
A collection must be specified as multi-tenant when it is created. So, we enable multi-tenancy in the collection configuration.
from weaviate.classes.config import Configure, Property, DataType
mt_collection = client.collections.create(
name=mt_collection_name, # e.g. "JournalEntry"
multi_tenancy_config=Configure.multi_tenancy(
enabled=True,
auto_tenant_creation=True,
auto_tenant_activation=True,
),
)
We also set auto_tenant_creation
and auto_tenant_activation
here to true
. You'll learn more about these features later on. But here is a brief overview:
auto_tenant_activation
: iftrue
, activate any deactivated (INACTIVE
orOFFLOADED
) tenants when they are accessed.auto_tenant_creation
: iftrue
, automatically create the tenant when an object is inserted against a non-existent tenant.
You will see these features in action later on.
More about auto_tenant_creation
v1.25
The auto tenant creation feature is available from v1.25.0
for batch imports, and from v1.25.2
for single object insertions.
Enabling auto_tenant_creation
will cause Weaviate to automatically create the tenant when an object is inserted against a non-existent tenant.
This option is particularly useful for bulk data ingestion, as it removes the need to create the tenant prior to object insertion. Instead, auto_tenant_creation
will allow the object insertion process to continue without interruption.
A risk of using auto_tenant_creation
is that an error in the source data will not be caught during import. For example, a source object with erroneously spelt "TenntOn"
instead of "TenantOne"
will create a new tenant for "TenntOne"
instead of raising an error.
The server-side default for auto_tenant_creation
is false
.
More about auto_tenant_activation
v1.25.2
The auto tenant activation feature is available from v1.25.2
.
If auto_tenant_activation
is enabled, Weaviate will automatically activate any deactivated (INACTIVE
or OFFLOADED
) tenants when they are accessed.
This option is particularly useful for scenarios where you have a large number of tenants, but only a subset of them are active at any given time. An example is a SaaS app where some tenants may be unlikely due to their local time zone, or their recent activity level.
By enabling auto_tenant_activation
, you can safely set those less active users to be inactive, knowing that they will be loaded onto memory once requested.
This can help to reduce the memory footprint of your Weaviate instance, as only the active tenants are loaded into memory.
The server-side default for auto_tenant_activation
is false
.
Configure vector index
From what we know about other journal use cases, a majority of users will only have a small number of entries. But, a few of those users may have a large number of entries.
This is a tricky situation to balance. If we use a hnsw
index, it will be fast for users with many entries, but it will require a lot of memory. If we use a flat
index, it will require less memory, but potentially slower for users with many entries.
What we can do here is to choose a dynamic
index. A dynamic
index will automatically switch from flat
to hnsw
once it passes a threshold count. This way, we can balance the memory usage and speed for our users.
Here is an example code snippet, configuring a "note" named vector with a dynamic
index.
vectorizer_config=[
Configure.NamedVectors.text2vec_cohere(
name="text",
source_properties=["text"],
vector_index_config=Configure.VectorIndex.dynamic(
hnsw=Configure.VectorIndex.hnsw(
quantizer=Configure.VectorIndex.Quantizer.sq(training_limit=50000)
),
flat=Configure.VectorIndex.flat(
quantizer=Configure.VectorIndex.Quantizer.bq()
),
threshold=10000
)
)
],
Note (no pun intended) that the dynamic
index configuration accepts both flat
and hnsw
index configurations. Each index configuration is used when the dynamic
index is in that state.
Full code snippet
Here is the complete code snippet to create the collection. Take a look at the configuration and see if you agree with our choices.
In our PoC, we create just the one collection ("JournalEntry"
), with "text"
, "date"
, and "tags"
properties to keep things simple.
from weaviate.classes.config import Configure, Property, DataType
mt_collection = client.collections.create(
name=mt_collection_name, # e.g. "JournalEntry"
multi_tenancy_config=Configure.multi_tenancy(
enabled=True,
auto_tenant_creation=True,
auto_tenant_activation=True,
),
properties=[
Property(name="text", data_type=DataType.TEXT),
Property(name="date", data_type=DataType.DATE),
Property(name="tags", data_type=DataType.TEXT_ARRAY),
],
vectorizer_config=[
Configure.NamedVectors.text2vec_cohere(
name="text",
source_properties=["text"],
vector_index_config=Configure.VectorIndex.dynamic(
hnsw=Configure.VectorIndex.hnsw(
quantizer=Configure.VectorIndex.Quantizer.sq(training_limit=50000)
),
flat=Configure.VectorIndex.flat(
quantizer=Configure.VectorIndex.Quantizer.bq()
),
threshold=10000
)
)
],
generative_config=Configure.Generative.cohere(model="command-r-plus")
# MTConfig
)
Summary
In this section, we set up a Weaviate instance with multi-tenancy enabled. We also created a collection for MyPrivateJournal
, with multi-tenancy and a dynamic index configuration.
Now, the MyPrivateJournal
team can start onboarding users and storing their journal entries in the collection.
Questions and feedback
If you have any questions or feedback, let us know in the user forum.