Weaviate Knowledge Cards

Semantic Chunking

In this technique, the text is divided into meaningful units, such as sentences or paragraphs, which are then vectorized. These units are then combined into chunks based on the cosine distance between their embeddings, with a new chunk formed whenever a significant context shift is detected.