The Google Ngram Viewer or Google Books Ngram Viewer is an
online search engine that charts the frequencies of any set of search strings using a yearly count of
n-grams found in printed sources published between 1500 and 2019
[ in ]Google
Google LLC () is an American multinational technology company focusing on search engine technology, online advertising, cloud computing, computer software, quantum computing, e-commerce, artificial intelligence, and consumer electronics. ...
's text corpora
In linguistics, a corpus (plural ''corpora'') or text corpus is a language resource consisting of a large and structured set of texts (nowadays usually electronically stored and processed). In corpus linguistics, they are used to do statistical ...
in English, Chinese (simplified), French, German, Hebrew, Italian, Russian, or Spanish.[ There are also some specialized English corpora, such as American English, British English, and English Fiction.]
The program can search for a word or a phrase, including misspellings or gibberish.[ The n-grams are matched with the text within the selected corpus, optionally using ]case-sensitive In computers, case sensitivity defines whether uppercase and lowercase letters are treated as distinct (case-sensitive) or equivalent (case-insensitive). For instance, when users interested in learning about dogs search an e-book, "dog" and "Dog" a ...
spelling (which compares the exact use of uppercase letters),[ and, if found in 40 or more books, are then displayed as a graph.][
The Google Ngram Viewer supports searches for ]parts of speech
In grammar, a part of speech or part-of-speech (abbreviated as POS or PoS, also known as word class or grammatical category) is a category of words (or, more generally, of lexical items) that have similar grammatical properties. Words that are ass ...
and wildcards. It is routinely used in research.
History
The program was developed by Jon Orwant and Will Brockman and released in mid-December 2010.[ It was inspired by a prototype called "Bookworm" created by Jean-Baptiste Michel and ]Erez Aiden
Erez Lieberman Aiden (born 1980, né Erez Lieberman) is an American research scientist active in multiple fields related to applied mathematics. He is an assistant professor at the Baylor College of Medicine, and formerly a fellow at the Harvard ...
from Harvard's Cultural Observatory, Yuan Shen from MIT
The Massachusetts Institute of Technology (MIT) is a private land-grant research university in Cambridge, Massachusetts. Established in 1861, MIT has played a key role in the development of modern technology and science, and is one of the m ...
, and Steven Pinker
Steven Arthur Pinker (born September 18, 1954) is a Canadian-American cognitive psychologist, psycholinguist, popular science author, and public intellectual. He is an advocate of evolutionary psychology and the computational theory of mind.
...
.
The Ngram Viewer was initially based on the 2009 edition of the Google Books Ngram Corpus. , the program supports 2009, 2012, and 2019 corpora.
Operation and restrictions
Commas delimit user-entered search terms, indicating each separate word or phrase to find.[ The Ngram Viewer returns a plotted ]line chart
A line chart or line graph or curve chart is a type of chart which displays information as a series of data points called 'markers' connected by straight line segments. It is a basic type of chart common in many fields. It is similar to a ...
within seconds of the user pressing the Enter key
On computer keyboards, the enter key and return key are two closely related keys with overlapping and distinct functions dependent on operating system and application.
Functions
The return key has its origins in two typewriter functions: ca ...
or the "Search" button on the screen.
As an adjustment for more books having been published during some years, the data are normalized, as a relative level, by the number of books published in each year.[
Due to limitations on the size of the Ngram database, only matches found in at least 40 books are indexed in the database; otherwise, the database could not have stored all possible combinations.][
Typically, search terms cannot end with punctuation, although a separate full stop (a period) can be searched.][ Also, an ending ]question mark
The question mark (also known as interrogation point, query, or eroteme in journalism) is a punctuation mark that indicates an interrogative clause or phrase in many languages.
History
In the fifth century, Syriac Bible manuscripts used ques ...
(as in "Why?") will cause a second search for the question mark separately.[
Omitting the periods in abbreviations will allow a form of matching, such as using "R M S" to search for "R.M.S." versus "RMS".
]
Corpora
The corpora
Corpus is Latin for "body". It may refer to:
Linguistics
* Text corpus, in linguistics, a large and structured set of texts
* Speech corpus, in linguistics, a large set of speech audio files
* Corpus linguistics, a branch of linguistics
Music
* ...
used for the search are composed of total_counts, 1-grams, 2-grams, 3-grams, 4-grams, and 5-grams files for each language. The file format of each of the files is tab-separated data. Each line has the following format:
* total_counts file
*: year TAB match_count TAB page_count TAB volume_count NEWLINE
* Version 1 ngram file (generated in July 2009)
*: ngram TAB year TAB match_count TAB page_count TAB volume_count NEWLINE
* Version 2 ngram file (generated in July 2012)
*: ngram TAB year TAB match_count TAB volume_count NEWLINE
The Google Ngram Viewer uses match_count to plot the graph.
As an example, a word "Wikipedia" from the Version 2 file of the English 1-grams is stored as follows:
The graph plotted by the Google Ngram Viewer using the above data is here:
Limitations
The data set has been criticized for its reliance upon inaccurate OCR, an overabundance of scientific literature, and for including large numbers of incorrectly dated and categorized texts. Because of these errors, and because it is uncontrolled for bias (such as the increasing amount of scientific literature, which causes other terms to appear to decline in popularity), it is risky to use this corpus to study language or test theories. Since the data set does not include metadata
Metadata is "data that provides information about other data", but not the content of the data, such as the text of a message or the image itself. There are many distinct types of metadata, including:
* Descriptive metadata – the descriptive ...
, it may not reflect general linguistic or cultural change and can only hint at such an effect.
Guidelines for doing research with data from Google Ngram have been proposed that address many of the issues discussed above.
OCR issues
Optical character recognition, or OCR, is not always reliable, and some characters may not be scanned correctly. In particular, systemic errors like the confusion of "s" and "f" in pre-19th century texts (due to the use of the long s
The long s , also known as the medial s or initial s, is an archaism, archaic form of the lowercase letter . It replaced the single ''s'', or one or both of the letters ''s'' in a 'double ''s sequence (e.g., "ſinfulneſs" for "sinfulness" ...
which was similar in appearance to "f") can cause systemic bias. Although Google Ngram Viewer claims that the results are reliable from 1800 onwards, poor OCR and insufficient data mean that frequencies given for languages such as Chinese may only be accurate from 1970 onward, with earlier parts of the corpus showing no results at all for common terms, and data for some years containing more than 50% noise.When n-grams go bad
digitalsinology.org.
See also
* Culturomics
* Google Trends
Google Trends is a website by Google that analyzes the popularity of top web search query, search queries in Google Search across various regions and languages. The website uses graphs to compare the search volume of different queries over time. ...
* Lexical analysis
In computer science, lexical analysis, lexing or tokenization is the process of converting a sequence of characters (such as in a computer program or web page) into a sequence of ''lexical tokens'' ( strings with an assigned and thus identified ...
References
Bibliography
*
External links
*
{{Google Inc.
Ngram Viewer
The Google Ngram Viewer or Google Books Ngram Viewer is an online search engine that charts the frequencies of any set of search strings using a yearly count of n-grams found in printed sources published between 1500 and 2019 in Google's text co ...
Ngram Viewer
The Google Ngram Viewer or Google Books Ngram Viewer is an online search engine that charts the frequencies of any set of search strings using a yearly count of n-grams found in printed sources published between 1500 and 2019 in Google's text co ...
2010 software
Natural language processing
Computational linguistics
Language modeling
Speech recognition
Corpus linguistics
Probabilistic models