Question Answering - transfomers

     

In short

• The Question and Answer (Q&A) module is a Weaviate module for answer extraction from data.
• The module depends on a text vectorization module that should be running with Weaviate.
• The module adds an ask {} parameter to the GraphQL Get {} queries
• The module returns a max. of 1 answer in the GraphQL _additional {} field.
• The answer with the highest certainty (confidence level) will be returned.

Introduction

The Question and Answer (Q&A) module is a Weaviate module for answer extraction from data. It uses BERT-related models for finding and extracting answers. This module can be used in GraphQL Get{...} queries, as a search operator. The qna-transformers module tries to find an answer in the data objects of the specified class. If an answer is found within the given certainty range, it will be returned in the GraphQL _additional { answer { ... } } field. There will be a maximum of 1 answer returned, if this is above the optionally set certainty. The answer with the highest certainty (confidence level) will be returned.

There are currently five different Question Answering modules available (taken from Hugging Face): distilbert-base-uncased-distilled-squad (uncased), bert-large-uncased-whole-word-masking-finetuned-squad (uncased), distilbert-base-cased-distilled-squad (cased), deepset/roberta-base-squad2, and deepset/bert-large-uncased-whole-word-masking-squad2 (uncased). Note that not all models perform well on every dataset and use case. We recommend to use bert-large-uncased-whole-word-masking-finetuned-squad (uncased), which performs best on most datasets (although it's quite heavyweighted).

Starting with v1.10.0, the answer score can be used as a reranking factor for the search results.

How to enable (module configuration)

Docker-compose

The Q&A module can be added as a service to the Docker-compose file. You must have a text vectorizer like text2vec-contextionary or text2vec-transformers running. An example Docker-compose file for using the qna-transformers module (bert-large-uncased-whole-word-masking-finetuned-squad (uncased)) in combination with the text2vec-transformersis as follows:

---
version: '3.4'
services:
  weaviate:
    command:
    - --host
    - 0.0.0.0
    - --port
    - '8080'
    - --scheme
    - http
    image: semitechnologies/weaviate:1.19.6
    ports:
    - 8080:8080
    restart: on-failure:0
    environment:
      TRANSFORMERS_INFERENCE_API: 'http://t2v-transformers:8080'
      QNA_INFERENCE_API: "http://qna-transformers:8080"
      QUERY_DEFAULTS_LIMIT: 25
      AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
      PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
      DEFAULT_VECTORIZER_MODULE: 'text2vec-transformers'
      ENABLE_MODULES: 'text2vec-transformers,qna-transformers'
      CLUSTER_HOSTNAME: 'node1'
  t2v-transformers:
    image: semitechnologies/transformers-inference:sentence-transformers-msmarco-distilbert-base-v2
    environment:
      ENABLE_CUDA: '1'
      NVIDIA_VISIBLE_DEVICES: all
    deploy:
      resources:
        reservations:
          devices:
          - capabilities: [gpu]
  qna-transformers:
    image: semitechnologies/qna-transformers:bert-large-uncased-whole-word-masking-finetuned-squad
    environment:
      ENABLE_CUDA: '1'
      NVIDIA_VISIBLE_DEVICES: all
    deploy:
      resources:
        reservations:
          devices:
          - capabilities: [gpu]
...

Variable explanations:

• QNA_INFERENCE_API: where the qna module is running
• ENABLE_CUDA: if set to 1 it uses GPU (if available on the host machine)

Note: at the moment, text vectorization modules cannot be combined in a single setup. This means that you can either enable the text2vec-contextionary, the text2vec-transformers or no text vectorization module.

How to use (GraphQL)

GraphQL Ask search

This module adds a search parameter to GraphQL Get{...} queries: ask{}. This new search parameter takes the following arguments:

Field	Data Type	Required	Example value	Description
question	string	yes	"What is the name of the Dutch king?"	The question to be answered.
certainty	float	no	0.75	Desired minimal certainty or confidence of answer to the question. The higher the value, the stricter the search becomes. The lower the value, the fuzzier the search becomes. If no certainty is set, any answer that could be extracted will be returned
properties	list of strings	no	["summary"]	The properties of the queries Class which contains text. If no properties are set, all are considered.
rerank	bool	no	true	If enabled, the qna module will rerank the result based on the answer score. For example, if the 3rd result - as determined by the previous (semantic) search contained the most likely answer, result 3 will be pushed to position 1, etc. Not supported prior to v1.10.0

Notes:

• The GraphQL Explore { } function does support the ask searcher, but the result is only a beacon to the object containing the answer. It is thus not any different from performing a nearText semantic search with the question. No extraction is happening.
• You cannot use the 'ask' parameter along with a 'near' parameter!

Example query

Python

import weaviate

client = weaviate.Client("http://localhost:8080")

ask = {
  "question": "Who is the king of the Netherlands?",
  "properties": ["summary"]
}

result = (
  client.query
  .get("Article", ["title", "_additional {answer {hasAnswer certainty property result startPosition endPosition} }"])
  .with_ask(ask)
  .with_limit(1)
  .do()
)

print(result)

JavaScript

const weaviate = require('weaviate-client');

const client = weaviate.client({
  scheme: 'http',
  host: 'localhost:8080',
});

client.graphql
      .get()
      .withClassName('Article')
      .withAsk({
        question: 'Who is the king of the Netherlands?',
        properties: ['summary'],
      })
      .withFields('title _additional { answer { hasAnswer certainty property result startPosition endPosition } }')
      .withLimit(1)
      .do()
      .then(res => {
        console.log(res)
      })
      .catch(err => {
        console.error(err)
      });

Go

package main

import (
  "context"
  "fmt"

  "github.com/weaviate/weaviate-go-client/v4/weaviate"
  "github.com/weaviate/weaviate-go-client/v4/weaviate/graphql"
)

func main() {
  cfg := weaviate.Config{
    Host:   "localhost:8080",
    Scheme: "http",
  }
  client, err := weaviate.NewClient(cfg)
  if err != nil {
    panic(err)
  }

  className := "Article"
  fields := []graphql.Field{
    {Name: "title"},
    {Name: "_additional", Fields: []graphql.Field{
      {Name: "answer", Fields: []graphql.Field{
        {Name: "hasAnswer"},
        {Name: "certainty"},
        {Name: "property"},
        {Name: "result"},
        {Name: "startPosition"},
        {Name: "endPosition"},
      }},
    }},
  }
  ask := client.GraphQL().AskArgBuilder().
    WithQuestion("Who is the king of the Netherlands?").
    WithProperties([]string{"summary"})
  ctx := context.Background()

  result, err := client.GraphQL().Get().
    WithClassName(className).
    WithFields(fields...).
    WithAsk(ask).
    WithLimit(1).
    Do(ctx)

  if err != nil {
    panic(err)
  }
  fmt.Printf("%v", result)
}

Java

package io.weaviate;

import io.weaviate.client.Config;
import io.weaviate.client.WeaviateClient;
import io.weaviate.client.base.Result;
import io.weaviate.client.v1.graphql.model.GraphQLResponse;
import io.weaviate.client.v1.graphql.query.argument.AskArgument;
import io.weaviate.client.v1.graphql.query.fields.Field;

public class App {
  public static void main(String[] args) {
    Config config = new Config("http", "localhost:8080");
    WeaviateClient client = new WeaviateClient(config);

    Field title = Field.builder().name("title").build();
    Field _additional = Field.builder()
      .name("_additional")
      .fields(new Field[]{
        Field.builder()
          .name("answer")
          .fields(new Field[]{
            Field.builder().name("hasAnswer").build(),
            Field.builder().name("certainty").build(),
            Field.builder().name("property").build(),
            Field.builder().name("result").build(),
            Field.builder().name("startPosition").build(),
            Field.builder().name("endPosition").build()
          }).build()
      }).build();

    AskArgument ask = AskArgument.builder()
      .question("Who is the king of the Netherlands?")
      .properties(new String[]{ "summary" })
      .build();

    Result<GraphQLResponse> result = client.graphQL().get()
      .withClassName("Article")
      .withFields(title, _additional)
      .withAsk(ask)
      .withLimit(1)
      .run();

    if (result.hasErrors()) {
      System.out.println(result.getError());
      return;
    }
    System.out.println(result.getResult());
  }
}

Curl

$ echo '{
  "query": "{
    Get {
      Article(
        ask: {
          question: \"Who is the king of the Netherlands?\",
          properties: [\"summary\"]
        },
        limit: 1
      ) {
        title
        _additional {
          answer {
            hasAnswer
            certainty
            property
            result
            startPosition
            endPosition
          }
        }
      }
    }
  }
  "
}' | curl \
    -X POST \
    -H 'Content-Type: application/json' \
    -d @- \
    http://localhost:8080/v1/graphql

GraphQL

{
  Get {
    Article(
      ask: {
        question: "Who is the king of the Netherlands?",
        properties: ["summary"],
        rerank: false # supported from v1.10.0 on
      },
      limit: 1
    ) {
      title
      _additional {
        answer {
          hasAnswer
          certainty
          property
          result
          startPosition
          endPosition
        }
      }
    }
  }
}

GraphQL response

The answer is contained in a new GraphQL _additional property called answer. It contains the following fields:

• hasAnswer (boolean): could an answer be found?
• result (nullable string): An answer if one could be found. null if hasAnswer==false
• certainty (nullable float): The certainty of the answer returned. null if hasAnswer==false
• property (nullable string): The property which contains the answer. null if hasAnswer==false
• startPosition (int): The character offset where the answer starts. 0 if hasAnswer==false
• endPosition (int): The character offset where the answer ends 0 if hasAnswer==false

Note: startPosition, endPosition and property in the response are not guaranteed to be present. They are calculated by a case-insensitive string matching function against the input text. If the transformer model formats the output differently (e.g. by introducing spaces between tokens which were not present in the original input), the calculation of the position and determining the property fails.

Example response

{
  "data": {
    "Get": {
      "Article": [
        {
          "_additional": {
            "answer": {
              "certainty": 0.73,
              "endPosition": 26,
              "hasAnswer": true,
              "property": "summary",
              "result": "king willem - alexander",
              "startPosition": 48
            }
          },
          "title": "Bruised Oranges - The Dutch royals are botching covid-19 etiquette"
        }
      ]
    }
  },
  "errors": null
}

Custom Q&A Transformer module

You can use the same approach as for text2vec-transformers, see here, i.e. either pick one of the pre-built containers or build your own container from your own model using the semitechnologies/qna-transformers:custom base image. Make sure that your model is compatible with Hugging Face's transformers.AutoModelForQuestionAnswering.

How it works (under the hood)

Under the hood, the model uses a two-step approach. First it performs a semantic search with k=1 to find the document (e.g. a Sentence, Paragraph, Article, etc.) which is most likely to contain the answer. This step has no certainty threshold and as long as at least one document is present, it will be fetched and selected as the one most likely containing the answer. In a second step, a BERT-style answer extraction is performed on all text and string properties of the document. There are now three possible outcomes:

1. No answer was found because the question can not be answered,
2. An answer was found, but did not meet the user-specified minimum certainty, so it was discarded (typically the case when the document is on topic, but does not contain an actual answer to the question), and
3. An answer was found that matches the desired certainty. It is returned to the user.

The module performs a semantic search under the hood, so a text2vec-... module is required. It does not need to be transformers-based and you can also combine it with text2vec-contextionary. However, we expect that you will receive the best results by combining it with a well-fitting transformers model by using the appropriate configured text2vec-transformers module.

Automatic sliding window for long documents

If a text value in a data object is longer than 512 tokens, the Q&A Transformer module automatically splits the text into smaller texts. The module uses a sliding window, i.e. overlapping pieces of text, to avoid a scenario that an answer cannot be found if it lies on a boundary. If an answer lies on the boundary, the Q&A module returns the result (answer) with the highest score (as the sliding mechanism could lead to duplicates).

More resources

If you can't find the answer to your question here, please look at the:

1. Frequently Asked Questions. Or,
2. Knowledge base of old issues. Or,
3. For questions: Stackoverflow. Or,
4. For more involved discussion: Weaviate Community Forum. Or,
5. We also have a Slack channel.