In
machine learning
Machine learning (ML) is a field of inquiry devoted to understanding and building methods that 'learn', that is, methods that leverage data to improve performance on some set of tasks. It is seen as a part of artificial intelligence.
Machine ...
and
natural language processing, the pachinko allocation model (PAM) is a
topic model
In statistics and natural language processing, a topic model is a type of statistical model for discovering the abstract "topics" that occur in a collection of documents. Topic modeling is a frequently used text-mining tool for discovery of hidden ...
. Topic models are a suite of algorithms to uncover the hidden thematic structure of a collection of documents.
The algorithm improves upon earlier topic models such as
latent Dirichlet allocation
In natural language processing, Latent Dirichlet Allocation (LDA) is a generative statistical model that explains a set of observations through unobserved groups, and each group explains why some parts of the data are similar. The LDA is an ex ...
(LDA) by modeling correlations between topics in addition to the word correlations which constitute topics. PAM provides more flexibility and greater expressive power
than latent Dirichlet allocation.
While first described and implemented in the context of natural language processing, the algorithm may have applications in other fields such as
bioinformatics. The
model is named for
pachinko
is a mechanical game originating in Japan that is used as an arcade game, and much more frequently for gambling. Pachinko fills a niche in Japanese gambling comparable to that of the slot machine in the West as a form of low-stakes, low-st ...
machines—a game popular in Japan, in which metal balls bounce down around
a complex collection of pins until they land in various
bins at the bottom.
History
Pachinko allocation was first described by Wei Li and
Andrew McCallum
Andrew McCallum is a professor in the computer science department at University of Massachusetts Amherst. His primary specialties are in machine learning, natural language processing, information extraction, information integration, and social ...
in 2006.
The idea was extended with hierarchical Pachinko allocation by Li, McCallum, and David Mimno in 2007.
In 2007, McCallum and his colleagues proposed a nonparametric Bayesian prior for PAM based
on a variant of the hierarchical Dirichlet process (HDP).
The algorithm has been implemented in the
MALLET software package published by McCallum's group at the
University of Massachusetts Amherst
The University of Massachusetts Amherst (UMass Amherst, UMass) is a public research university in Amherst, Massachusetts and the sole public land-grant university in Commonwealth of Massachusetts. Founded in 1863 as an agricultural college, ...
.
Model
PAM connects words in V and topics in T with an arbitrary
directed acyclic graph
In mathematics, particularly graph theory, and computer science, a directed acyclic graph (DAG) is a directed graph with no directed cycles. That is, it consists of vertices and edges (also called ''arcs''), with each edge directed from one v ...
(DAG), where topic nodes occupy the interior levels and the leaves are words.
The probability of generating a whole corpus is the product of the probabilities for every document:
See also
*
Probabilistic latent semantic indexing Probabilistic latent semantic analysis (PLSA), also known as probabilistic latent semantic indexing (PLSI, especially in information retrieval circles) is a statistical technique for the analysis of two-mode and co-occurrence data. In effect, one ca ...
(PLSI), an early topic model from Thomas Hofmann in 1999.
*
Latent Dirichlet allocation
In natural language processing, Latent Dirichlet Allocation (LDA) is a generative statistical model that explains a set of observations through unobserved groups, and each group explains why some parts of the data are similar. The LDA is an ex ...
, a generalization of PLSI developed by
David Blei
David Meir Blei is a professor in the Statistics and Computer Science departments at Columbia University. Prior to fall 2014 he was an associate professor in the Department of Computer Science at Princeton University. His work is primarily in mach ...
,
Andrew Ng
Andrew Yan-Tak Ng (; born 1976) is a British-born American computer scientist and technology entrepreneur focusing on machine learning and AI. Ng was a co-founder and head of Google Brain and was the former Chief Scientist at Baidu, building ...
, and
Michael Jordan in 2002, allowing documents to have a mixture of topics.
*
MALLET, an open-source Java library that implements Pachinko allocation.
References
External links
Mixtures of Hierarchical Topics with Pachinko Allocation a video recording of David Mimno presenting HPAM in 2007.
Statistical natural language processing
Latent variable models
{{comp-sci-stub