Skip to main content

HNSW Snapshots

Added in v1.31

HNSW (Hierarchical Navigable Small World) snapshots can significantly reduce startup times for instances with large vector indexes.

By default, HNSW snapshotting is disabled. To use this feature, configure the environment variable shown below.

Concepts: HNSW snapshots

See this concepts page for a detailed description.

1. Enabling HNSW snapshots​

Set PERSISTENCE_HNSW_DISABLE_SNAPSHOTS to false to enable HNSW snapshotting. (Default: true)

2. Configuring snapshot creation​

Set the following optional environment variables to configure the snapshotting behavior.

note

Before creating a new snapshot, the previous snapshot and the commit log difference need to be loaded into memory. Make sure you have enough memory to accommodate this process.

Snapshot on startup​

Enable or disable snapshot creation on startup:

  • PERSISTENCE_HNSW_SNAPSHOT_ON_STARTUP: If true, Weaviate will try to create a new snapshot during startup if there are changes in the commit log since the last snapshot. If there are no changes, then the existing snapshot will be loaded.
    • Default: true

Periodic snapshots​

Set the following to configure periodic snapshot creation. Note all of the following conditions must be met to trigger a snapshot:

  1. A time interval has passed:

    • PERSISTENCE_HNSW_SNAPSHOT_INTERVAL_SECONDS: The minimum time in seconds since the previous snapshot.
      • Default: 21600 seconds (6 hours)

  2. Sufficient new commit logs (by number):

    • PERSISTENCE_HNSW_SNAPSHOT_MIN_DELTA_COMMITLOGS_NUMBER: The minimum number of new commit log files created since the last snapshot.
      • Default: 1

  3. Sufficient new commit logs (by size percentage):

    • PERSISTENCE_HNSW_SNAPSHOT_MIN_DELTA_COMMITLOGS_SIZE_PERCENTAGE: The minimum total size of new commit logs (as a percentage of the previous snapshot's size) required to trigger a new snapshot.
      • Default: 5 (meaning 5% of the previous snapshot's size in new commit logs). For example, if the previous snapshot was 1000MB, at least 50MB of new commit log data is required.

Further resources​

Questions and feedback​

If you have any questions or feedback, let us know in the user forum.