Neuro-symbolic AI integrates
neural and
symbolic AI architectures to address complementary strengths and weaknesses of each, providing a robust AI capable of
reasoning,
learning, and
cognitive modeling A cognitive model is an approximation of one or more cognitive processes in humans or other animals for the purposes of comprehension and prediction. There are many types of cognitive models, and they can range from box-and-arrow diagrams to a set o ...
. As argued by
Valiant and many others, the effective construction of rich computational
cognitive model A cognitive model is an approximation of one or more cognitive processes in humans or other animals for the purposes of comprehension and prediction. There are many types of cognitive models, and they can range from box-and-arrow diagrams to a set ...
s demands the combination of sound
symbolic reasoning
In mathematics and computer science, computer algebra, also called symbolic computation or algebraic computation, is a scientific area that refers to the study and development of algorithms and software for manipulating expression (mathematics), ...
and efficient
machine learning
Machine learning (ML) is a field of inquiry devoted to understanding and building methods that 'learn', that is, methods that leverage data to improve performance on some set of tasks. It is seen as a part of artificial intelligence.
Machine ...
models.
Gary Marcus
Gary F. Marcus (born February 8, 1970) is a professor emeritus of psychology and neural science at New York University. In 2014 he founded Geometric Intelligence, a machine-learning company later acquired by Uber. Marcus's books include '' Gui ...
, argues that: "We cannot construct rich
cognitive models in an adequate, automated way without the triumvirate of hybrid architecture, rich prior knowledge, and sophisticated techniques for reasoning.". Further, "To build a robust, knowledge-driven approach to AI we must have the machinery of symbol-manipulation in our toolkit. Too much of useful knowledge is abstract to make do without tools that represent and manipulate abstraction, and to date, the only machinery that we know of that can manipulate such abstract knowledge reliably is the apparatus of symbol-manipulation."
Henry Kautz
Henry A. Kautz (born 1956) is a computer scientist, Founding Director of Institute for Data Science and Professor at University of Rochester. He is interested in knowledge representation, artificial intelligence, data science and pervasive comp ...
,
Francesca Rossi
Francesca Rossi (born December 7, 1962) is an Italian computer scientist, currently working at the IBM T.J. Watson Research Lab (New York, USA) as an IBM Fellow and the IBM AI Ethics Global Leader.
Education and career
She received her bachelor ...
, and
Bart Selman
Bart Selman is a Dutch-American professor of computer science at Cornell University. He has previously worked at AT&T Bell Laboratories. He is also co-founder and principal investigator of the Center for Human-Compatible Artificial Intelligence ( ...
have also argued for a synthesis. Their arguments are based on a need to address the two kinds of thinking discussed in
Daniel Kahneman's book,
Thinking Fast and Slow.
Kahneman
Daniel Kahneman (; he, דניאל כהנמן; born March 5, 1934) is an Israeli-American psychologist and economist notable for his work on the psychology of judgment and decision-making, as well as behavioral economics, for which he was award ...
describes human thinking as having two components,
System 1 and System 2. System 1 is fast, automatic, intuitive and unconscious. System 2 is slower, step-by-step, and explicit. System 1 is used for pattern recognition. System 2 handles planning, deduction, and deliberative thinking. In this view,
deep learning best handles the first kind of cognition while symbolic reasoning best handles the second kind. Both are needed for a robust, reliable AI that can learn, reason, and interact with humans to accept advice and answer questions. In fact, such dual-process models with explicit references to the two contrasting systems have been worked on since the 1990s, both in AI and in Cognitive Science, by a number of researchers (e.g.,).
Types of approaches
Approaches for integration are varied.
Henry Kautz
Henry A. Kautz (born 1956) is a computer scientist, Founding Director of Institute for Data Science and Professor at University of Rochester. He is interested in knowledge representation, artificial intelligence, data science and pervasive comp ...
's taxonomy of neuro-symbolic architectures, along with some examples, follows:
* Symbolic Neural symbolic—is the current approach of many neural models in natural language processing, where words or subword tokens are both the ultimate input and output of large language models. Examples include
BERT
Bert or BERT may refer to:
Persons, characters, or animals known as Bert
*Bert (name), commonly an abbreviated forename and sometimes a surname
*Bert, a character in the poem "Bert the Wombat" by The Wiggles; from their 1992 album Here Comes a Son ...
, RoBERTa, and
GPT-3
Generative Pre-trained Transformer 3 (GPT-3) is an autoregressive language model that uses deep learning to produce human-like text. Given an initial text as prompt, it will produce text that continues the prompt.
The architecture is a standa ...
.
* Symbolic
eural''—is exemplified by
AlphaGo
AlphaGo is a computer program that plays the board game Go. It was developed by DeepMind Technologies a subsidiary of Google (now Alphabet Inc.). Subsequent versions of AlphaGo became increasingly powerful, including a version that competed u ...
, where symbolic techniques are used to call neural techniques. In this case the symbolic approach is
Monte Carlo tree search
In computer science, Monte Carlo tree search (MCTS) is a heuristic search algorithm for some kinds of decision processes, most notably those employed in software that plays board games. In that context MCTS is used to solve the game tree.
MCT ...
and the neural techniques learn how to evaluate game positions.
* Neural, Symbolic—uses a neural architecture to interpret perceptual data as symbols and relationships that are then reasoned about symbolically. The Neural-Concept Learner is an example.
* Neural:Symbolic → Neural—relies on symbolic reasoning to generate or label training data that is subsequently learned by a deep learning model, e.g., to train a neural model for symbolic computation by using a
Macsyma
Macsyma (; "Project MAC's SYmbolic MAnipulator") is one of the oldest general-purpose computer algebra systems still in wide use. It was originally developed from 1968 to 1982 at MIT's Project MAC.
In 1982, Macsyma was licensed to Symbolics a ...
-like symbolic mathematics system to create or label examples.
* Neural_—uses a neural net that is generated from symbolic rules. An example is the Neural Theorem Prover, which constructs a neural network from an
AND-OR proof tree generated from knowledge base rules and terms. Logic Tensor Networks also fall into this category.
* Neural
ymbolic''—allows a neural model to directly call a symbolic reasoning engine, e.g., to perform an action or evaluate a state.
These categories are not exhaustive, for example, as they do not consider multi-agent systems. In 2005, Bader and Hitzler presented a more fine-grained categorization that considered, e.g., whether the use of symbols included logic or not, and if it did, whether the logic was propositional or first-order logic. The 2005 categorization and Kautz' taxonomy above are compared and contrasted in a 2021 article. Recently,
Sepp Hochreiter
Josef "Sepp" Hochreiter (born 14 February 1967) is a German computer scientist. Since 2018 he has led the Institute for Machine Learning at the Johannes Kepler University of Linz after having led the Institute of Bioinformatics from 2006 to 2018 ...
argued that
Graph Neural Networks "...are the predominant models of neural-symbolic computing
[L.C. Lamb, A.S. d'Avila Garcez, M.Gori, M.O.R. Prates, P.H.C. Avelar, M.Y. Vardi (2020). "Graph Neural Networks Meet Neural-Symbolic Computing: A Survey and Perspective." CoRR abs/2003.00330 (2020)]" since "
ey describe the properties of molecules, simulate social networks, or predict future states in physical and engineering applications with particle-particle interactions."
/sup>
As a prerequisite for artificial general intelligence
Gary Marcus, Marcus argues that "...hybrid architectures that combine learning and symbol manipulation are necessary for robust intelligence, but not sufficient", and that there are:
"...four cognitive prerequisites for building robust artificial intelligence:
* hybrid architectures that combine large-scale learning with the representational and computational powers of symbol-manipulation,
* large-scale knowledge bases—likely leveraging innate frameworks—that incorporate symbolic knowledge along with other forms of knowledge,
* reasoning mechanisms capable of leveraging those knowledge bases in tractable ways, and
* rich cognitive models that work together with those mechanisms and knowledge bases
A knowledge base (KB) is a technology used to store complex structured and unstructured information used by a computer system. The initial use of the term was in connection with expert systems, which were the first knowledge-based systems.
Or ...
."
This echoes the much earlier calls for hybrid models in as early as the 1990s.
History
Garcez and Lamb described research in this area as being ongoing for at least the past twenty years (actually more than thirty years by now). A series of workshops on neuro-symbolic AI has been held every year since 200
In the early 1990s, an initial set of workshops on this topic were organized.
Open research questions
Many key research questions remain, such as:
* What is the best way to integrate neural and symbolic architectures?
* How should symbolic structures be represented within neural networks and extracted from them?
* How should common-sense knowledge be learned and reasoned about?
* How can abstract knowledge that is hard to encode logically be handled?
Implementations
Some specific implementations of neuro-symbolic approaches are:
* Logic Tensor Networks—these encode logical formulas as neural networks and simultaneously learn term neural encodings, term weights, and formula weights from data.
* DeepProbLog—which combines neural networks with the probabilistic reasoning of ProbLog.
Citations
References
*
*
*
*
*
* Hochreiter, Sepp. "Toward a Broad AI." Commun. ACM 65(4): 56-57 (2022). https://dl.acm.org/doi/pdf/10.1145/3512715
*
*
*
*
*
*
*
*
*
*
*
* Sun, Ron; Alexandre, Frederic (1997). ''Connectionist Symbolic Integration''. Lawrence Erlbaum Associates.
* {{Cite journal , pages= , last=Valiant , first=Leslie G , title=Knowledge Infusion: In Pursuit of Robustness in Artificial Intelligence , journal= , date=2008
See also
* Symbolic AI
* Connectionist AI
* Hybrid intelligent systems Hybrid intelligent system denotes a software system which employs, in parallel, a combination of methods and techniques from artificial intelligence subfields, such as:
* Neuro-symbolic systems
* Neuro-fuzzy systems
* Hybrid connectionist-symbolic ...
External links
Artificial Intelligence: Workshop series on Neural-Symbolic Learning and Reasoning
Artificial intelligence