Intrinsic Motivation (artificial Intelligence)

Intrinsic motivation in the study of artificial intelligence and robotics is a mechanism for enabling artificial agents (including robots) to exhibit inherently rewarding behaviours such as exploration and curiosity, grouped under the same term in the study of psychology. Psychologists consider intrinsic motivation in humans to be the drive to perform an activity for inherent satisfaction – just for the fun or challenge of it.

Definition

An intelligent agent is intrinsically motivated to act if the information content alone, or the experience resulting from the action, is the motivating factor.

Information content in this context is measured in the information-theoretic sense of quantifying uncertainty. A typical intrinsic motivation is to search for unusual, surprising situations (exploration), in contrast to a typical extrinsic motivation such as the search for food (homeostasis). Extrinsic motivations are typically described in artificial intelligence as ''task-dependent'' or ''goal-directed''.

Origins in psychology

The study of intrinsic motivation in psychology and neuroscience began in the 1950s with some psychologists explaining exploration through drives to manipulate and explore, however, this homeostatic view was criticised by White. An alternative explanation from Berlyne in 1960 was the pursuit of an optimal balance between novelty and familiarity. Festinger described the difference between internal and external view of the world as dissonance that organisms are motivated to reduce. A similar view was expressed in the '70s by Kagan as the desire to reduce the incompatibility between cognitive structure and experience. In contrast to the idea of optimal incongruity, Deci and Ryan identified in the mid 80's an intrinsic motivation based on competence and self-determination.

Computational models

An influential early computational approach to implement artificial curiosity in the early 1990s by Schmidhuber, has since been developed into a "Formal theory of creativity, fun, and intrinsic motivation”.

Intrinsic motivation is often studied in the framework of computational reinforcement learning (introduced by Sutton and Barto), where the rewards that drive agent behaviour are intrinsically derived rather than externally imposed and must be learnt from the environment. Reinforcement learning is agnostic to how the reward is generated - an agent will learn a policy (action strategy) from the distribution of rewards afforded by actions and the environment. Each approach to intrinsic motivation in this scheme is essentially a different way of generating the reward function for the agent.

Curiosity vs. exploration

Intrinsically motivated artificial agents exhibit behaviour that resembles curiosity or exploration. Exploration in artificial intelligence and robotics has been extensively studied in reinforcement learning models, usually by encouraging the agent to explore as much of the environment as possible, to reduce uncertainty about the dynamics of the environment (learning the transition function) and how best to achieve its goals (learning the reward function). Intrinsic motivation, in contrast, encourages the agent to first explore aspects of the environment that confer more information, to seek out novelty. Recent work unifying state visit count exploration and intrinsic motivation has shown faster learning in a video game setting.

Types of models

Ouedeyer and Kaplan have made a substantial contribution to the study of intrinsic motivation. They define intrinsic motivation based on Berlyne's theory, and divide approaches to the implementation of intrinsic motivation into three categories that broadly follow the roots in psychology: "knowledge-based models", "competence-based models" and "morphological models". Knowledge-based models are further subdivided into "information-theoretic" and "predictive". Baldassare and Mirolli present a similar typology, differentiating knowledge-based models between prediction-based and novelty-based.

Information-theoretic intrinsic motivation

The quantification of prediction and novelty to drive behaviour is generally enabled through the application of information-theoretic models, where agent state and strategy (policy) over time are represented by probability distributions describing a markov decision process and the cycle of perception and action treated as an information channel. These approaches claim biological feasibility as part of a family of bayesian approaches to brain function. The main criticism and difficulty of these models is the intractability of computing probability distributions over large discrete or continuous state spaces. Nonetheless a considerable body of work has built up modelling the flow of information around the sensorimotor cycle, leading to de facto reward functions derived from the reduction of uncertainty, including most notably active inference, but also infotaxis, predictive information, and empowerment.

Competence-based models

Steels' autotelic principle is an attempt to formalise flow (psychology).

Achievement, affiliation and power models

Other intrinsic motives that have been modelled computationally include achievement, affiliation and power motivation. These motives can be implemented as functions of probability of success or incentive. Populations of agents can include individuals with different profiles of achievement, affiliation and power motivation, modelling population diversity and explaining why different individuals take different actions when faced with the same situation.

Beyond achievement, affiliation and power

A more recent computational theory of intrinsic motivation attempts to explain a large variety of psychological findings based on such motives. Notably this model of intrinsic motivation goes beyond just achievement, affiliation and power, by taking into consideration other important human motives. Empirical data from psychology were computationally simulated and accounted for using this model.

Intrinsically Motivated Learning

Intrinsically motivated (or curiosity-driven) learning is an emerging research topic in artificial intelligence and developmental robotics that aims to develop agents that can learn general skills or behaviours, that can be deployed to improve performance in extrinsic tasks, such as acquiring resources. Intrinsically motivated learning has been studied as an approach to autonomous lifelong learning in machines and open-ended learning in computer game characters. In particular, when the agent learns a meaningful abstract representation, a notion of distance between two representations can be used to gauge novelty, hence allowing for an efficient exploration of its environment. Despite the impressive success of deep learning in specific domains (e.g. AlphaGo), many in the field (e.g. Gary Marcus) have pointed out that the ability to generalise remains a fundamental challenge in artificial intelligence. Intrinsically motivated learning, although promising in terms of being able to generate goals from the structure of the environment without externally imposed tasks, faces the same challenge of generalisation – how to reuse policies or action sequences, how to compress and represent continuous or complex state spaces and retain and reuse the salient features that have been learnt.

See also

* reinforcement learning
* markov decision process
* motivation
* predictive coding
* perceptual control theory

References

{{reflist, refs=

{{cite journal
, last1=Ryan
, first1=Richard M
, last2=Deci
, first2=Edward L
, date=2000
, title=Intrinsic and extrinsic motivations: Classic definitions and new directions
, journal=Contemporary Educational Psychology
, volume=25
, issue=1
, pages=54–67, doi=10.1006/ceps.1999.1020
, pmid=10620381
, s2cid=1098145
, hdl=20.500.12799/2958
, hdl-access=free

{{cite book
, last1=Oudeyer
, first1=Pierre-Yves
, last2=Kaplan
, first2=Frederic
, date=2008
, chapter=How can we define intrinsic motivation?
, title=Proc. of the 8th Conf. on Epigenetic Robotics
, volume=5
, pages=29–31

{{cite book , last1=Baldassarre , first1=Gianluca , last2=Mirolli , first2=Marco , title=Intrinsically Motivated Learning in Natural 1 and Artificial Systems , date=2013 , publisher=Springer , location=Rome, Italy , pages=1–14 , chapter=Intrinsically Motivated Learning Systems: An Overview

{{cite journal , last1=Schmidhuber , first1=J , title=Formal theory of creativity, fun, and intrinsic motivation (1990-2010) , journal=IEEE Trans. Auton. Mental Dev. , date=2010 , volume=2 , issue=3 , pages=230–247, doi=10.1109/TAMD.2010.2056368 , s2cid=234198

{{cite journal , last1=White , first1=R. , title=Motivation reconsidered: The concept of competence. , journal=Psychological Review , date=1959 , volume=66 , issue=5 , pages=297–333, doi=10.1037/h0040934 , pmid=13844397 , s2cid=37385966

Berlyne, D.: Conflict, Arousal and Curiosity. McGraw-Hill, New York (1960)

Festinger, L.: A theory of cognitive dissonance. Evanston, Row, Peterson (1957)

Kagan, J.: Motives and development. Journal of Personality and Social Psychology 22, 51–66

Deci, E.L., Ryan, R.M.: Intrinsic motivation and self-determination in human behavior. Plenum,
New York (1985)

Barto, A., Singh, S., Chentanez, N.: Intrinsically motivated learn- ing of hierarchical collections of skills. In: ICDL 2004. Proceedings of the 3rd International Conference on Development and Learning, Salk Institute, San Diego (2004)

{{cite journal , last1=Friston , first1=Karl , last2=Kilner , first2=James , last3=Harrison , first3=Lee , title=A free energy principle for the brain , journal=Journal of Physiology-Paris , publisher=Elsevier BV , volume=100 , issue=1–3 , year=2006 , issn=0928-4257 , doi=10.1016/j.jphysparis.2006.10.001 , pmid=17097864 , pages=70–87, s2cid=637885 , url=http://www.fil.ion.ucl.ac.uk/~karl/A%20free%20energy%20principle%20for%20the%20brain.pdf

{{cite book
, last1=Salge
, first1=C
, last2=Glackin
, first2=C
, last3=Polani
, first3=D
, date=2014
, chapter=Empowerment -- An Introduction
, editor-last=Prokopenko
, editor-first=M
, title=Guided Self-Organization: Inception. Emergence, Complexity and Computation
, volume=9
, publisher=Springer
, pages=67–114
, doi=10.1007/978-3-642-53734-9_4
, arxiv=1310.1863
, isbn=978-3-642-53733-2, s2cid=9662065

Barto, A.G.: Intrinsic motivation and reinforcement learning. In: Baldassarre, G., Mirolli, M. (eds.) Intrinsically Motivated Learning in Natural and Artificial Systems. Springer, Berlin (2012)

Steels, Luc: The autotelic principle. In: Iida, F., Pfeifer, R., Steels, L., Kuniyoshi, Y. (eds.) Embodied Artificial Intelligence. LNCS (LNAI), vol. 3139, pp. 231–242. Springer, Heidelberg (2004)

Ay, N., Bertschinger, N., Der, R., Güttler, F. and Olbrich, E. (2008), ‘Predictive information and explorative behavior of autonomous robots’, The European Physical Journal B 63(3), 329–339.

Oudeyer, P. Y., & Kaplan, F. (2009). What is intrinsic motivation? A typology of computational approaches. Frontiers in Neurorobotics, 3(NOV). https://doi.org/10.3389/neuro.12.006.2007

Vergassola, M., Villermaux, E., & Shraiman, B. I. (2007). ‘Infotaxis’ as a strategy for searching without gradients. Nature, 445(7126), 406–409. https://doi.org/10.1038/nature05464

Kaplan, F. and Oudeyer, P. (2004). Maximizing learning progress: an internal reward system for development. Embodied artificial intelligence, pages 629–629.

Singh, S., Barto, A. G., and Chentanez, N. (2005). Intrinsically motivated reinforcement learning. In Proceedings of the 18th Annual Conference on Neural Information Processing Systems (NIPS), Vancouver, B.C., Canada.

Klyubin, A., Polani, D., and Nehaniv, C. (2008). Keep your options open: an information-based driving principle for sensorimotor systems. PLOS ONE, 3(12):e4018. https://dx.doi.org/10.1371%2Fjournal.pone.0004018

{{cite journal,
last1=Biehl,
first1=Martin,
last2=Guckelsberger,
first2=Christian,
last3=Salge,
first3=Christoph,
last4=Smith,
first4=Simón C.,
last5=Polani,
first5=Daniel,
title=Expanding the Active Inference Landscape: More Intrinsic Motivations in the Perception-Action Loop ,
journal=Frontiers in Neurorobotics ,
volume=12 ,
year=2018 ,
pages=45 ,
doi=10.3389/fnbot.2018.00045 ,
pmid=30214404,
pmc=6125413,
arxiv=1806.08083,
issn=1662-5218 ,
doi-access=free

Csikszentmihalyi, M. (2000). Beyond boredom and anxiety. Jossey-Bass.

Lungarella, M., Metta, G., Pfeifer, R., and Sandini, G. (2003). Developmental robotics: a survey. Connect. Sci. 15, 151–190. doi: 10.1080/09540090310001655110

Barto, A. G. (2013). “Intrinsic motivation and reinforcement learning,” in Intrinsically Motivated Learning in Natural and Artificial Systems (Berlin; Heidelberg: Springer), 17–47

Martius, G., Der, R., and Ay, N. (2013). Information driven self-organization of complex robotic behaviors. PLOS ONE 8:e63400. doi: 10.1371/journal.pone.0063400

Mirolli, M., and Baldassarre, G. (2013). “Functions and mechanisms of intrinsic motivations,” in Intrinsically Motivated Learning in Natural and Artificial Systems, eds G. Baldassarre and M. Mirolli (Berlin; Heidelberg: Springer), 49–72

Santucci, V. G., Oudeyer, P. Y., Barto, A., & Baldassarre, G. (2020). Editorial: Intrinsically motivated open-ended learning in autonomous robots. Frontiers in Neurorobotics, 13(January), 2019–2021. https://doi.org/10.3389/fnbot.2019.00115

Sun, R., Bugrov, S, and Dai, D. (2022). A unified framework for interpreting a range of motivation-performance phenomena. Cognitive Systems Research, 71, 24–40.

Tao, Ruo Yu and Francois-Lavet, Vincent and Pineau, Joelle (2020). Novelty search in representational space for sample efficient exploration. Neural Information Processing Systems, 2020. https://arxiv.org/abs/2009.13579

Bellemare, M. G., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., & Munos, R. (2016). Unifying count-based exploration and intrinsic motivation. Advances in Neural Information Processing Systems, 1479–1487.
Thrun, S. B. (1992). Efficient Exploration in Reinforcement Learning. https://doi.org/10.1007/978-1-4899-7687-1_244

Merrick, K. E., Maher, M-L (2009). Motivated Reinforcement Learning: Curious Characters for Multiuser Games. Springer-Verlag Berlin Heidelberg, https://doi.org/10.1007/978-3-540-89187-1.
Merrick, K. E. (2016). Computational Models of Motivation for Game-Playing Agents. Springer International Publishing, https://doi.org/10.1007/978-3-319-33459-2.

Artificial intelligence
Robotics
Cognitive science