Sentence extraction is a technique used for
automatic summarization
Automatic summarization is the process of shortening a set of data computationally, to create a subset (a summary) that represents the most important or relevant information within the original content. Artificial intelligence algorithms are commo ...
of a text.
In this shallow approach,
statistical heuristics are used to identify the most salient sentences of a text. Sentence extraction is a low-cost approach compared to more knowledge-intensive deeper approaches which require additional knowledge bases such as
ontologies
In computer science and information science, an ontology encompasses a representation, formal naming, and definition of the categories, properties, and relations between the concepts, data, and entities that substantiate one, many, or all domains ...
or
linguistic knowledge. In short "sentence extraction" works as a filter which allows only important sentences to pass.
The major downside of applying sentence-extraction techniques to the task of summarization is the loss of
coherence
Coherence, coherency, or coherent may refer to the following:
Physics
* Coherence (physics), an ideal property of waves that enables stationary (i.e. temporally and spatially constant) interference
* Coherence (units of measurement), a deriv ...
in the resulting summary.
Nevertheless, sentence extraction summaries can give valuable clues to the main points of a document and are frequently sufficiently intelligible to human readers.
Procedure
Usually, a combination of heuristics is used to determine the most important sentences within the document. Each heuristic assigns a (positive or negative) score to the sentence. After all heuristics have been applied, the highest-scoring sentences are included in the summary.
The individual heuristics are weighted according to their importance.
Early approaches and some sample heuristics
Seminal papers which laid the foundations for many techniques used today have been published by
Hans Peter Luhn
Hans Peter Luhn (July 1, 1896 – August 19, 1964) was a German researcher in the field of computer science and Library & Information Science for IBM, and creator of the Luhn algorithm, KWIC (Key Words In Context) indexing, and Selective ...
in 1958 and
H. P Edmundson in 1969.
Luhn proposed to assign more weight to sentences at the beginning of the document or a paragraph.
Edmundson stressed the importance of title-words for summarization and was the first to employ stop-lists in order to filter uninformative words of low semantic content (e.g. most grammatical words such as "of", "the", "a"). He also distinguished between ''bonus words'' and ''stigma words'', i.e. words that probably occur together with important (e.g. the word form "significant") or unimportant information.
His idea of using key-words, i.e. words which occur significantly frequently in the document, is still one of the core heuristics of today's summarizers. With large linguistic corpora available today, the
tf–idf
In information retrieval, tf–idf (also TF*IDF, TFIDF, TF–IDF, or Tf–idf), short for term frequency–inverse document frequency, is a numerical statistic that is intended to reflect how important a word is to a document in a collection or co ...
value which originated in
information retrieval
Information retrieval (IR) in computing and information science is the process of obtaining information system resources that are relevant to an information need from a collection of those resources. Searches can be based on full-text or other co ...
, can be successfully applied to identify the key words of a text: If for example the word "cat" occurs significantly more often in the text to be summarized (TF = "term frequency") than in the corpus (IDF means "inverse document frequency"; here the corpus is meant by "document"), then "cat" is likely to be an important word of the text; the text may in fact be a text about cats.
See also
*
Text segmentation
Text segmentation is the process of dividing written text into meaningful units, such as words, sentences, or topics. The term applies both to mental processes used by humans when reading text, and to artificial processes implemented in comput ...
*
Sentence boundary disambiguation Sentence boundary disambiguation (SBD), also known as sentence breaking, sentence boundary detection, and sentence segmentation, is the problem in natural language processing of deciding where sentences begin and end. Natural language processing too ...
References
{{Natural Language Processing
Computational linguistics
Natural language processing