Member-only story
An Introduction to Open Domain Question-Answering
Unlocking the treasure trove of unstructured data with deep learning
Search is a crucial functionality in many applications and companies globally. Whether in manufacturing, finance, healthcare, or almost any other industry, organizations have vast internal information and document repositories.
Unfortunately, the scale of many companies’ data means that the organization and accessibility of information can become incredibly inefficient. The problem is exacerbated for language-based information.
Language is a tool for people to communicate often abstract ideas and concepts. Naturally, ideas and concepts are harder for a computer to comprehend and store in a meaningful way.
Most organizations rely on a cluster of keyword-based search interfaces hosted on various ‘internal portals’ to deal with language data. This can satisfy business requirements for some of that data if done well.
A keyword-based search is ideal if a person knows what they’re looking for and the keywords and terminology of the information they need. When the keywords and terminology of the answer are unknown, keyword search is inadequate. People searching for unknown answers in large repositories of documents is a drain on productivity.
How do we minimize this problem? The answer lies with semantic search, specifically with the question-answering (QA) flavor of semantic search.
Semantic search allows us to search based on concepts and ideas rather than keywords. Given a phrase, a semantic search tool returns the most semantically similar phrases from a repository.
Question-answering takes this idea further by searching using a natural language question and returning relevant documents and specific answers. QA aims to mimic natural language as much as possible.
If we asked a shop assistant, “where are those tasty, freshly baked things that are not cookies but look…
