Spiking neural networks (SNNs) are
artificial neural network
Artificial neural networks (ANNs), usually simply called neural networks (NNs) or neural nets, are computing systems inspired by the biological neural networks that constitute animal brains.
An ANN is based on a collection of connected unit ...
s that more closely mimic natural neural networks.
In addition to
neuronal
A neuron, neurone, or nerve cell is an electrically excitable cell that communicates with other cells via specialized connections called synapses. The neuron is the main component of nervous tissue in all animals except sponges and placozoa. No ...
and
synaptic state, SNNs incorporate the concept of time into their
operating model
An operating model is both an abstract and visual representation (model) of how an organization delivers value to its customers or beneficiaries as well as how an organization actually runs itself.
Definition
There are different ways of defining t ...
. The idea is that
neurons
A neuron, neurone, or nerve cell is an electrically excitable cell that communicates with other cells via specialized connections called synapses. The neuron is the main component of nervous tissue in all animals except sponges and placozoa. N ...
in the SNN do not transmit information at each propagation cycle (as it happens with typical multi-layer
perceptron networks), but rather transmit information only when a
membrane potential
Membrane potential (also transmembrane potential or membrane voltage) is the difference in electric potential between the interior and the exterior of a biological cell. That is, there is a difference in the energy required for electric charges ...
– an intrinsic quality of the neuron related to its membrane electrical charge – reaches a specific value, called the threshold. When the membrane potential reaches the threshold, the neuron fires, and generates a signal that travels to other neurons which, in turn, increase or decrease their potentials in response to this signal. A neuron model that fires at the moment of threshold crossing is also called a
spiking neuron model
Biological neuron models, also known as a spiking neuron models, are mathematical descriptions of the properties of certain cells in the nervous system that generate sharp electrical potentials across their cell membrane, roughly one millisecon ...
.
The most prominent spiking neuron model is the
leaky integrate-and-fire model. In the integrate-and-fire model, the momentary activation level (modeled as a
differential equation
In mathematics, a differential equation is an equation that relates one or more unknown functions and their derivatives. In applications, the functions generally represent physical quantities, the derivatives represent their rates of change, an ...
) is normally considered to be the neuron's state, with incoming spikes pushing this value higher or lower, until the state eventually either decays or - if the firing threshold is reached - the neuron fires. After firing the state variable is reset to a lower value.
Various decoding methods exist for interpreting the outgoing ''
spike train
An action potential occurs when the membrane potential of a specific cell location rapidly rises and falls. This depolarization then causes adjacent locations to similarly depolarize. Action potentials occur in several types of animal cells, c ...
'' as a real-value number, relying on either the frequency of spikes (rate-code), the time-to-first-spike after stimulation, or the interval between spikes.
History
![Artificial synapses based on FTJs](https://upload.wikimedia.org/wikipedia/commons/e/e2/Artificial_synapses_based_on_FTJs.png)
Many multi-layer artificial neural networks are
fully connected
Fully () is a municipality in the district of Martigny in the canton of Valais in Switzerland.
History
Fully is first mentioned in the 11th Century as ''Fuliacum''.
Geography
Fully has an area, , of . Of this area, 30.5% is used for agricult ...
, receiving input from every neuron in the previous layer and signalling every neuron in the subsequent layer. Although these networks have achieved breakthroughs in many fields, they are biologically inaccurate and do not mimic the operation mechanism of neurons in the brain of a living thing.
The biologically inspired
Hodgkin–Huxley model
The Hodgkin–Huxley model, or conductance-based model, is a mathematical model that describes how action potentials in neurons are initiated and propagated. It is a set of nonlinear differential equations that approximates the electrical charact ...
of a spiking neuron was proposed in 1952. This model describes how
action potential
An action potential occurs when the membrane potential of a specific cell location rapidly rises and falls. This depolarization then causes adjacent locations to similarly depolarize. Action potentials occur in several types of animal cells, ...
s are initiated and propagated. Communication between neurons, which requires the exchange of chemical
neurotransmitter
A neurotransmitter is a signaling molecule secreted by a neuron to affect another cell across a synapse. The cell receiving the signal, any main body part or target cell, may be another neuron, but could also be a gland or muscle cell.
Neuro ...
s in the
synaptic gap, is described in various models, such as the
integrate-and-fire
Biological neuron models, also known as a spiking neuron models, are mathematical descriptions of the properties of certain cells in the nervous system that generate sharp electrical potentials across their cell membrane, roughly one millisecon ...
model,
FitzHugh–Nagumo model
The FitzHugh–Nagumo model (FHN), named after Richard FitzHugh (1922–2007) who suggested the system in 1961 and J. Nagumo ''et al''. who created the equivalent circuit the following year, describes a prototype of an excitable system (e.g., a n ...
(1961–1962), and
Hindmarsh–Rose model
The Hindmarsh–Rose model of neuronal activity is aimed to study the spiking-bursting behavior of the membrane potential observed in experiments made with a single neuron. The relevant variable is the membrane potential, ''x''(''t''), which is ...
(1984). The leaky integrate-and-fire model (or a derivative) is commonly used as it is easier to compute than the Hodgkin–Huxley model.
Brainchip Holdings Ltd announced on 21 October 2021 that it was taking orders for its Akida AI Processor Development Kits, making it the world's first commercially available neuromorphic processor operating on a spiking neural network.
Underpinnings
Information in the brain is represented as action potentials (neuron spikes), which may be grouped into spike trains or even coordinated waves of brain activity. A fundamental question of neuroscience is to determine whether neurons communicate by a
rate or temporal code.
Temporal coding
Neural coding (or Neural representation) is a neuroscience field concerned with characterising the hypothetical relationship between the stimulus and the individual or ensemble neuronal responses and the relationship among the electrical activity o ...
suggests that a single spiking neuron can replace hundreds of hidden units on a sigmoidal neural net''.''
An SNN computes in the continuous rather than the discrete domain. The idea is that neurons may not test for activation in every iteration of propagation (as is the case in a typical multilayer
perceptron
In machine learning, the perceptron (or McCulloch-Pitts neuron) is an algorithm for supervised learning of binary classifiers. A binary classifier is a function which can decide whether or not an input, represented by a vector of numbers, belon ...
network), but only when their membrane potentials reach a certain value. When a neuron is activated, it produces a signal that is passed to connected neurons, raising or lowering their membrane potential.
In a spiking neural network, a neuron's current state is defined as its membrane potential (possibly modeled as a differential equation). An input pulse causes the membrane potential to rise for a period of time and then gradually decline. Encoding schemes have been constructed to interpret these pulse sequences as a number, taking into account both pulse frequency and pulse interval. A neural network model based on pulse generation time can be established. Using the exact time of pulse occurrence, a neural network can employ more information and offer better computing properties.
The SNN approach produces a continuous output instead of the binary output of traditional ANNs. Pulse trains are not easily interpretable, hence the need for encoding schemes as above. However, a pulse train representation may be more suited for processing spatiotemporal data (or continual real-world sensory data classification). SNNs consider space by connecting neurons only to nearby neurons so that they process input blocks separately (similar to
CNN
CNN (Cable News Network) is a multinational cable news channel headquartered in Atlanta, Georgia, U.S. Founded in 1980 by American media proprietor Ted Turner and Reese Schonfeld as a 24-hour cable news channel, and presently owned by ...
using filters). They consider time by encoding information as pulse trains so as not to lose information in a binary encoding. This avoids the additional complexity of a
recurrent neural network
A recurrent neural network (RNN) is a class of artificial neural networks where connections between nodes can create a cycle, allowing output from some nodes to affect subsequent input to the same nodes. This allows it to exhibit temporal dynamic ...
(RNN). It turns out that impulse neurons are more powerful computational units than traditional artificial neurons.
SNNs are theoretically more powerful than second-generation networks
erm undefined: what ''are'' 2nd-gen networks?/sup>; however, SNN training issues and hardware requirements limit their use. Although unsupervised biologically inspired learning methods are available such as Hebbian learning
Hebbian theory is a neuroscientific theory claiming that an increase in synaptic efficacy arises from a presynaptic cell's repeated and persistent stimulation of a postsynaptic cell. It is an attempt to explain synaptic plasticity, the adaptation ...
and STDP
Spike-timing-dependent plasticity (STDP) is a biological process that adjusts the strength of connections between neurons in the brain. The process adjusts the connection strengths based on the relative timing of a particular neuron's output and in ...
, no effective supervised training method is suitable for SNNs that can provide better performance than second-generation networks. Spike-based activation of SNNs is not differentiable thus making it hard to develop gradient descent
In mathematics, gradient descent (also often called steepest descent) is a first-order iterative optimization algorithm for finding a local minimum of a differentiable function. The idea is to take repeated steps in the opposite direction of the ...
based training methods to perform error backpropagation
In machine learning, backpropagation (backprop, BP) is a widely used algorithm for training feedforward neural network, feedforward artificial neural networks. Generalizations of backpropagation exist for other artificial neural networks (ANN ...
, though a few recent algorithms such as NormAD and multilayer NormAD have demonstrated good training performance through suitable approximation of the gradient of spike based activation.
SNNs have much larger computational costs for simulating realistic neural models than traditional ANNs.
Pulse-coupled neural networks (PCNN) are often confused with SNNs. A PCNN can be seen as a kind of SNN.
Currently there are a few challenges when using SNNs that researchers are actively working on. The first challenge concerns the nondifferentiability of the spiking nonlinearity. The expressions for both the forward- and backward-learning methods contain the derivative of the neural activation function which is non-differentiable because neuron's output is either 1 when it spikes, and 0 otherwise. This all-or-nothing behavior of the binary spiking nonlinearity stops gradients from “flowing” and makes LIF neurons unsuitable for gradient-based optimization. The second challenge concerns the implementation of the optimization algorithm itself. Standard BP can be expensive in terms of computation, memory, and communication and may be poorly suited to the constraints dictated by the hardware that implements it (e.g., a computer, brain, or neuromorphic device). Regarding the first challenge there are several approached in order to overcome it. A few of them are:
# resorting to entirely biologically inspired local learning rules for the hidden units
# translating conventionally trained “rate-based” NNs to SNNs
# smoothing the network model to be continuously differentiable
# defining an SG (Surogate Gradient) as a continuous relaxation of the real gradients
Applications
SNNs can in principle apply to the same applications as traditional ANNs. In addition, SNNs can model the central nervous system
The central nervous system (CNS) is the part of the nervous system consisting primarily of the brain and spinal cord. The CNS is so named because the brain integrates the received information and coordinates and influences the activity of all par ...
of biological organisms, such as an insect seeking food without prior knowledge of the environment. Due to their relative realism, they can be used to study the operation of biological neural circuits. Starting with a hypothesis about the topology of a biological neuronal circuit and its function, recordings
A record, recording or records may refer to:
An item or collection of data Computing
* Record (computer science), a data structure
** Record, or row (database), a set of fields in a database related to one entity
** Boot sector or boot record, r ...
of this circuit can be compared to the output of the corresponding SNN, evaluating the plausibility of the hypothesis. However, there is a lack of effective training mechanisms for SNNs, which can be inhibitory for some applications, including computer vision tasks.
As of 2019 SNNs lag behind ANNs in terms of accuracy, but the gap is decreasing, and has vanished on some tasks.
When using SNNs for image based data we need to convert static images into binary spike trains coding. Types of encodings:
* Temporal coding generates one spike per neuron in which spike latency is inversely proportional to the pixel intensity.
* Rate coding converts pixel intensity into a spike train where the number of spikes is proportional to the pixel intensity.
* Direct coding uses a trainable layer to generate float value for each time-step. We have a learnable layer which converts each pixel at certain time step in float number and then threshold is used on the generated floating numbers to see if they will be 1 or 0.
* Phase coding encodes temporal information into spike patterns based on a global oscillator.
* Burst coding transmits the burst of spikes in a small-time duration, increasing the reliability of synaptic communication between neurons.
Software
A diverse range of application software
Application may refer to:
Mathematics and computing
* Application software, computer software designed to help the user to perform specific tasks
** Application layer, an abstraction layer that specifies protocols and interface methods used in a c ...
can simulate SNNs. This software can be classified according to its uses:
SNN simulation
These simulate complex neural models with a high level of detail and accuracy. Large networks usually require lengthy processing. Candidates include:
* Brian
Brian (sometimes spelled Bryan in English) is a male given name of Irish and Breton origin, as well as a surname of Occitan origin. It is common in the English-speaking world.
It is possible that the name is derived from an Old Celtic word me ...
– developed by Romain Brette and Dan Goodman at the École Normale Supérieure
École may refer to:
* an elementary school in the French educational stages normally followed by secondary education establishments (collège and lycée)
* École (river), a tributary of the Seine flowing in région Île-de-France
* École, Savoi ...
;
* GENESIS
Genesis may refer to:
Bible
* Book of Genesis, the first book of the biblical scriptures of both Judaism and Christianity, describing the creation of the Earth and of mankind
* Genesis creation narrative, the first several chapters of the Book of ...
(the GEneral NEural SImulation System) – developed in James Bower's laboratory at Caltech
The California Institute of Technology (branded as Caltech or CIT)The university itself only spells its short form as "Caltech"; the institution considers other spellings such a"Cal Tech" and "CalTech" incorrect. The institute is also occasional ...
;
* NEST
A nest is a structure built for certain animals to hold eggs or young. Although nests are most closely associated with birds, members of all classes of vertebrates and some invertebrates construct nests. They may be composed of organic materia ...
– developed by the NEST Initiative;
* NEURON
A neuron, neurone, or nerve cell is an electrically excitable cell that communicates with other cells via specialized connections called synapses. The neuron is the main component of nervous tissue in all animals except sponges and placozoa. N ...
– mainly developed by Michael Hines, John W. Moore and Ted Carnevale in Yale University
Yale University is a private research university in New Haven, Connecticut. Established in 1701 as the Collegiate School, it is the third-oldest institution of higher education in the United States and among the most prestigious in the wo ...
and Duke University
Duke University is a private research university in Durham, North Carolina. Founded by Methodists and Quakers in the present-day city of Trinity in 1838, the school moved to Durham in 1892. In 1924, tobacco and electric power industrialist James ...
;
Hardware
Future neuromorphic architectures will comprise billions of such nanosynapses, which require a clear understanding of the physical mechanisms responsible for plasticity. Experimental systems based on ferroelectric tunnel junctions have been used to show that STDP can be harnessed from heterogeneous polarization switching. Through combined scanning probe imaging, electrical transport and atomic-scale molecular dynamics, conductance variations can be modelled by nucleation-dominated reversal of domains. Simulations show that arrays of ferroelectric nanosynapses can autonomously learn to recognize patterns in a predictable way, opening the path towards unsupervised learning
Unsupervised learning is a type of algorithm that learns patterns from untagged data. The hope is that through mimicry, which is an important mode of learning in people, the machine is forced to build a concise representation of its world and t ...
.
*Neurogrid
Neurogrid is a piece of computer hardware that is designed specifically for simulation of biological brains. It uses analog computation to emulate ion channel activity, and digital communication to softwire structured connectivity patterns. Neurog ...
is a board that can simulate spiking neural networks directly in hardware. (Stanford University)
*SpiNNaker
A spinnaker is a sail designed specifically for sailing off the wind on courses between a reach (wind at 90° to the course) to downwind (course in the same direction as the wind). Spinnakers are constructed of lightweight fabric, usually n ...
(Spiking Neural Network Architecture) uses ARM
In human anatomy, the arm refers to the upper limb in common usage, although academically the term specifically means the upper arm between the glenohumeral joint (shoulder joint) and the elbow joint. The distal part of the upper limb between th ...
processors as the building blocks of a massively parallel
Massively parallel is the term for using a large number of computer processors (or separate computers) to simultaneously perform a set of coordinated computations in parallel. GPUs are massively parallel architecture with tens of thousands of t ...
computing platform based on a six-layer thalamocortical model. (University of Manchester
, mottoeng = Knowledge, Wisdom, Humanity
, established = 2004 – University of Manchester Predecessor institutions: 1956 – UMIST (as university college; university 1994) 1904 – Victoria University of Manchester 1880 – Victoria Univer ...
) The SpiNNaker system is based on numerical models running in real time on custom digital multicore chips using the ARM architecture
ARM (stylised in lowercase as arm, formerly an acronym for Advanced RISC Machines and originally Acorn RISC Machine) is a family of reduced instruction set computer (RISC) instruction set architectures for computer processors, configured ...
. It provides custom digital chips, each with eighteen cores and a shared local 128 Mbyte RAM, with a total of over 1,000,000 cores.[https://www.humanbrainproject.eu/en/silicon-brains/ Neuromorphic Computing] A single chip can simulate 16,000 neurons with eight million plastic synapses running in real time.
*TrueNorth A cognitive computer is a computer that hardwires artificial intelligence and machine-learning algorithms into an integrated circuit (printed circuit board) that closely reproduces the behavior of the human brain. It generally adopts a neuromorphic ...
is a processor that contains 5.4 billion transistors that consumes only 70 milliwatt
The watt (symbol: W) is the unit of power or radiant flux in the International System of Units (SI), equal to 1 joule per second or 1 kg⋅m2⋅s−3. It is used to quantify the rate of energy transfer. The watt is named after James Watt ...
s; most processors in personal computers contain about 1.4 billion transistors and require 35 watts or more. IBM refers to the design principle behind TrueNorth as neuromorphic computing
Neuromorphic engineering, also known as neuromorphic computing, is the use of electronic circuits to mimic neuro-biological architectures present in the nervous system. A neuromorphic computer/chip is any device that uses physical artificial ne ...
. Its primary purpose is pattern recognition. While critics say the chip isn't powerful enough, its supporters point out that this is only the first generation, and the capabilities of improved iterations will become clear. (IBM)
* AKD1000 is BrainChip's commercially available neuromorphic processor running an event-based development environment, it has 1.2 million neurons in clustered in nodes within the NPU network mesh. BrainChip describes it as having "unparalleled efficiency, precision, and economy of energy". It is designed with neural network learning capability on-chip. it has an ARM Cortex-M4 32-bit @ 300MHz subblock processor and built-in 256M x 16 bytes LPDDR4 SDRAM
Benchmarks
Classification capabilities of spiking networks trained according to unsupervised learning methods have been tested on the common benchmark datasets, such as, Iris, Wisconsin Breast Cancer or Statlog Landsat dataset. Various approaches to information encoding and network design have been used. For example, a 2-layer feedforward network for data clustering and classification. Based on the idea proposed in Hopfield (1995) the authors implemented models of local receptive fields combining the properties of radial basis functions (RBF) and spiking neurons to convert input signals (classified data) having a floating-point representation into a spiking representation.[Bohte, et al. (2002b)]
See also
*CoDi
CoDi is a cellular automaton (CA) model for spiking neural networks (SNNs). CoDi is an acronym for Collect and Distribute, referring to the signals and spikes in a neural network.
CoDi uses a von Neumann neighborhood modified for a three-dimen ...
*Cognitive architecture A cognitive architecture refers to both a theory about the structure of the human mind and to a computational instantiation of such a theory used in the fields of artificial intelligence (AI) and computational cognitive science. The formalized mod ...
*Cognitive map
A cognitive map is a type of mental representation which serves an individual to acquire, code, store, recall, and decode information about the relative locations and attributes of phenomena in their everyday or metaphorical spatial environment. T ...
* Cognitive computer
*Computational neuroscience
Computational neuroscience (also known as theoretical neuroscience or mathematical neuroscience) is a branch of neuroscience which employs mathematical models, computer simulations, theoretical analysis and abstractions of the brain to u ...
*Neural coding
Neural coding (or Neural representation) is a neuroscience field concerned with characterising the hypothetical relationship between the stimulus and the individual or ensemble neuronal responses and the relationship among the electrical activity o ...
*Neural correlate
The neural correlates of consciousness (NCC) refer to the relationships between mental states and neural states and constitute the minimal set of neuronal events and mechanisms sufficient for a specific conscious Perception#Process and terminology ...
*Neural decoding
Neural decoding is a neuroscience field concerned with the hypothetical reconstruction of sensory and other stimuli from information that has already been encoded and represented in the brain by networks of neurons. Reconstruction refers to the abi ...
*Neuroethology
Neuroethology is the evolutionary and comparative approach to the study of animal behavior and its underlying mechanistic control by the nervous system. It is an interdisciplinary science that combines both neuroscience (study of the nervous syste ...
*Neuroinformatics
Neuroinformatics is the field that combines informatics and neuroscience. Neuroinformatics is related with neuroscience data and information processing by artificial neural networks. There are three main directions where neuroinformatics has to be ...
*Models of neural computation
Models of neural computation are attempts to elucidate, in an abstract and mathematical fashion, the core principles that underlie information processing in biological nervous systems, or functional components thereof. This article aims to provide ...
*Motion perception
Motion perception is the process of inferring the speed and direction of elements in a scene based on visual, vestibular and proprioceptive inputs. Although this process appears straightforward to most observers, it has proven to be a difficult pr ...
*Systems neuroscience
Systems neuroscience is a subdiscipline of neuroscience and systems biology that studies the structure and function of neural circuits and systems. Systems neuroscience encompasses a number of areas of study concerned with how nerve cells behave ...
References
External links
* Full text of the boo
Spiking Neuron Models. Single Neurons, Populations, Plasticity
by Wulfram Gerstner and Werner M. Kistler ({{ISBN, 0-521-89079-9)
Computational statistics
Artificial neural networks
Articles containing video clips
Computational neuroscience