Image Signal Processor
   HOME

TheInfoList



OR:

An image processor, also known as an image processing engine, image processing unit (IPU), or image signal processor (ISP), is a type of
media processor A media processor, mostly used as an image/video processor, is a microprocessor-based system-on-a-chip which is designed to deal with digital streaming data in real-time (e.g. display refresh) rates. These devices can also be considered a class of ...
or specialized
digital signal processor A digital signal processor (DSP) is a specialized microprocessor chip, with its architecture optimized for the operational needs of digital signal processing. DSPs are fabricated on MOS integrated circuit chips. They are widely used in audio si ...
(DSP) used for
image processing An image is a visual representation of something. It can be two-dimensional, three-dimensional, or somehow otherwise feed into the visual system to convey information. An image can be an artifact, such as a photograph or other two-dimensiona ...
, in
digital camera A digital camera is a camera that captures photographs in digital memory. Most cameras produced today are digital, largely replacing those that capture images on photographic film. Digital cameras are now widely incorporated into mobile device ...
s or other devices. Image processors often employ
parallel computing Parallel computing is a type of computation in which many calculations or processes are carried out simultaneously. Large problems can often be divided into smaller ones, which can then be solved at the same time. There are several different fo ...
even with
SIMD Single instruction, multiple data (SIMD) is a type of parallel processing in Flynn's taxonomy. SIMD can be internal (part of the hardware design) and it can be directly accessible through an instruction set architecture (ISA), but it should ...
or
MIMD In computing, multiple instruction, multiple data (MIMD) is a technique employed to achieve parallelism. Machines using MIMD have a number of processors that function asynchronously and independently. At any time, different processors may be exe ...
technologies to increase speed and efficiency. The
digital image A digital image is an image composed of picture elements, also known as ''pixels'', each with ''finite'', '' discrete quantities'' of numeric representation for its intensity or gray level that is an output from its two-dimensional functions ...
processing engine can perform a range of tasks. To increase the system integration on embedded devices, often it is a
system on a chip A system on a chip or system-on-chip (SoC ; pl. ''SoCs'' ) is an integrated circuit that integrates most or all components of a computer or other electronic system. These components almost always include a central processing unit (CPU), memory ...
with
multi-core processor A multi-core processor is a microprocessor on a single integrated circuit with two or more separate processing units, called cores, each of which reads and executes program instructions. The instructions are ordinary CPU instructions (such a ...
architecture.


Function


Bayer transformation

The
photodiode A photodiode is a light-sensitive semiconductor diode. It produces current when it absorbs photons. The package of a photodiode allows light (or infrared or ultraviolet radiation, or X-rays) to reach the sensitive part of the device. The packag ...
s employed in an
image sensor An image sensor or imager is a sensor that detects and conveys information used to make an image. It does so by converting the variable attenuation of light waves (as they pass through or reflect off objects) into signals, small bursts of curr ...
are color-blind by nature: they can only record shades of grey. To get color into the picture, they are covered with different color filters: red, green and blue (
RGB The RGB color model is an additive color model in which the red, green and blue primary colors of light are added together in various ways to reproduce a broad array of colors. The name of the model comes from the initials of the three additiv ...
) according to the pattern designated by the
Bayer filter A Bayer filter mosaic is a color filter array (CFA) for arranging RGB color filters on a square grid of photosensors. Its particular arrangement of color filters is used in most single-chip digital image sensors used in digital cameras, camco ...
- named after its inventor. As each photodiode records the color information for exactly one
pixel In digital imaging, a pixel (abbreviated px), pel, or picture element is the smallest addressable element in a raster image, or the smallest point in an all points addressable display device. In most digital display devices, pixels are the smal ...
of the image, without an image processor there would be a green pixel next to each red and blue pixel. (Actually, with most sensors there are two green for each blue and red diodes.) This process, however, is quite complex and involves a number of different operations. Its quality depends largely on the effectiveness of the algorithms applied to the raw data coming from the sensor. The mathematically manipulated data becomes the photo file recorded.


Demosaicing

As stated above, the image processor evaluates the color and brightness data of a given pixel, compares them with the data from neighboring pixels and then uses a
demosaicing A demosaicing (also de-mosaicing, demosaicking or debayering) algorithm is a digital image process used to reconstruct a full color image from the incomplete color samples output from an image sensor overlaid with a color filter array (CFA). It is ...
algorithm to produce an appropriate colour and brightness value for the pixel. The image processor also assesses the whole picture to guess at the correct distribution of contrast. By adjusting the
gamma Gamma (uppercase , lowercase ; ''gámma'') is the third letter of the Greek alphabet. In the system of Greek numerals it has a value of 3. In Ancient Greek, the letter gamma represented a voiced velar stop . In Modern Greek, this letter re ...
value (heightening or lowering the contrast range of an image's mid-tones) subtle tonal gradations, such as in human skin or the blue of the sky, become much more realistic.


Noise reduction

Noise Noise is unwanted sound considered unpleasant, loud or disruptive to hearing. From a physics standpoint, there is no distinction between noise and desired sound, as both are vibrations through a medium, such as air or water. The difference arise ...
is a phenomenon found in any electronic circuitry. In digital photography its effect is often visible as random spots of obviously wrong colour in an otherwise smoothly-coloured area. Noise increases with temperature and exposure times. When higher
ISO ISO is the most common abbreviation for the International Organization for Standardization. ISO or Iso may also refer to: Business and finance * Iso (supermarket), a chain of Danish supermarkets incorporated into the SuperBest chain in 2007 * Iso ...
settings are chosen the electronic signal in the image sensor is amplified, which at the same time increases the noise level, leading to a lower
signal-to-noise ratio Signal-to-noise ratio (SNR or S/N) is a measure used in science and engineering that compares the level of a desired signal to the level of background noise. SNR is defined as the ratio of signal power to the noise power, often expressed in deci ...
. The image processor attempts to separate the noise from the image information and to remove it. This can be quite a challenge, as the image may contain areas with fine textures which, if treated as noise, may lose some of their definition.


Image sharpening

As the color and brightness values for each pixel are
interpolated In the mathematical field of numerical analysis, interpolation is a type of estimation, a method of constructing (finding) new data points based on the range of a discrete set of known data points. In engineering and science, one often has a n ...
some
image softening An image is a visual representation of something. It can be two-dimensional, three-dimensional, or somehow otherwise feed into the visual system to convey information. An image can be an artifact, such as a photograph or other two-dimensiona ...
is applied to even out any fuzziness that has occurred. To preserve the impression of depth, clarity and fine details, the image processor must sharpen edges and contours. It therefore must detect edges correctly and reproduce them smoothly and without over-sharpening.


Models

Image processor users are using industry standard products, application-specific standard products (ASSP) or even
application-specific integrated circuit An application-specific integrated circuit (ASIC ) is an integrated circuit (IC) chip customized for a particular use, rather than intended for general-purpose use, such as a chip designed to run in a digital voice recorder or a high-efficie ...
s (ASIC) with trade names: Canon's is called
DIGIC Digital Imaging Integrated Circuit (often styled as "DiG!C") is Canon Inc.'s name for a family of signal processing and control units for digital cameras and camcorders. DIGIC units are used as image processors by Canon in its own digital imagin ...
, Nikon's
Expeed The Nikon Expeed image processor, image/video processors (often styled ''EXPEED'') are media processors for Nikon dslr, Nikon's digital cameras. They perform a large number of tasks: Bayer filtering, demosaicing, image sensor corrections/dark-fram ...
, Olympus' TruePic, Panasonic's
Venus Engine The Venus Engine is an image processing engine for digital cameras. It is developed by Panasonic, and almost all of their Lumix cameras use a version of it. It is based on the Panasonic MN103/MN103S. Image processors operate in four steps: rec ...
and Sony's
Bionz BIONZ is a line of image processors used in Sony digital cameras. It is currently used in many Sony α DSLR and mirrorless cameras. Image processing in the camera converts the raw data from a CCD or CMOS image sensor into the format that is sto ...
. Some are known to be based on the
Fujitsu is a Japanese multinational information and communications technology equipment and services corporation, established in 1935 and headquartered in Tokyo. Fujitsu is the world's sixth-largest IT services provider by annual revenue, and the la ...
Milbeaut The Socionext Milbeaut image/video processors are media processors in multi-processor system on a chip architecture. Started by Fujitsu with the M-1 Series in 2000 each generation (2013: 7th) has several variants regarding included modules and proc ...
, the
Texas Instruments Texas Instruments Incorporated (TI) is an American technology company headquartered in Dallas, Texas, that designs and manufactures semiconductors and various integrated circuits, which it sells to electronics designers and manufacturers globall ...
OMAP The OMAP (Open Multimedia Applications Platform) family, developed by Texas Instruments, was a series of image/video processors. They are proprietary system on chips (SoCs) for portable and mobile multimedia applications. OMAP devices generally i ...
,
Panasonic formerly between 1935 and 2008 and the first incarnation of between 2008 and 2022, is a major Japanese multinational corporation, multinational Conglomerate (company), conglomerate corporation, headquartered in Kadoma, Osaka, Kadoma, Osaka P ...
MN103 The MN103 is a 32-bit microprocessor series developed by Matsushita Electric Industrial, now Panasonic Corporation. Most variants include a media processor, working as an image processor or video processor. It is used in digital cameras, set-top box ...
,
Zoran Zoran ( sr-Cyrl, Зоран) is a common South Slavic name, the masculine form of Zora, which means ''dawn, daybreak''. The name is especially common in Serbia, North Macedonia, Croatia and a little in Slovenia. Notable people with this given na ...
Coach, Altek Sunny or
Sanyo , stylized as SANYO, is a Japanese electronics company and formerly a member of the Fortune Global 500, ''Fortune'' Global 500 whose headquarters was located in Moriguchi, Osaka, Moriguchi, Osaka prefecture, Japan. Sanyo had over 230 subsidiari ...
image/video processors.
ARM architecture ARM (stylised in lowercase as arm, formerly an acronym for Advanced RISC Machines and originally Acorn RISC Machine) is a family of reduced instruction set computer (RISC) instruction set architectures for computer processors, configured ...
processors with its NEON SIMD
Media Processing Engine A media processor, mostly used as an image/video processor, is a microprocessor-based system-on-a-chip which is designed to deal with digital streaming data in real-time (e.g. display refresh) rates. These devices can also be considered a class of ...
s (MPE) are often used in
mobile phone A mobile phone, cellular phone, cell phone, cellphone, handphone, hand phone or pocket phone, sometimes shortened to simply mobile, cell, or just phone, is a portable telephone that can make and receive calls over a radio frequency link whil ...
s.


Processor brand names

*ATI -
Imageon Imageon (previously ATI Imageon) was a series of media coprocessors and mobile chipsets produced by ATI (later AMD) in 2002–2008, providing graphics acceleration and other multimedia features for handheld devices such as mobile phones and Person ...
(graphics co-processor used in many early mobile photos to offer camera image signal processing) *Canon -
DIGIC Digital Imaging Integrated Circuit (often styled as "DiG!C") is Canon Inc.'s name for a family of signal processing and control units for digital cameras and camcorders. DIGIC units are used as image processors by Canon in its own digital imagin ...
(based on Texas Instruments
OMAP The OMAP (Open Multimedia Applications Platform) family, developed by Texas Instruments, was a series of image/video processors. They are proprietary system on chips (SoCs) for portable and mobile multimedia applications. OMAP devices generally i ...
) *Casio - EXILIM engine *Epson - EDiART *Fujifilm - EXR III or X Processor Pro *Google -
Pixel Visual Core The Pixel Visual Core (PVC) is a series of ARM-based system in package (SiP) image processors designed by Google. The PVC is a fully programmable image, vision and AI multi-core domain-specific architecture (DSA) for mobile devices and in futur ...
*HTC - ImageSense *MediaTek - Imagiq *Minolta / Konica Minolta -
SUPHEED CxProcess is the trademark of an image processing technology used in Minolta and Konica Minolta digital cameras. Image processing in a camera converts the raw image data from a CCD image sensor into the format that is stored on the memory card. ...
with CxProcess *Leica - MAESTRO (based on Fujitsu
Milbeaut The Socionext Milbeaut image/video processors are media processors in multi-processor system on a chip architecture. Started by Fujitsu with the M-1 Series in 2000 each generation (2013: 7th) has several variants regarding included modules and proc ...
) *Nikon -
Expeed The Nikon Expeed image processor, image/video processors (often styled ''EXPEED'') are media processors for Nikon dslr, Nikon's digital cameras. They perform a large number of tasks: Bayer filtering, demosaicing, image sensor corrections/dark-fram ...
(based on Fujitsu
Milbeaut The Socionext Milbeaut image/video processors are media processors in multi-processor system on a chip architecture. Started by Fujitsu with the M-1 Series in 2000 each generation (2013: 7th) has several variants regarding included modules and proc ...
)Milbeaut and EXPEED
byThom *Olympus - TruePic (based on Panasonic
MN103 The MN103 is a 32-bit microprocessor series developed by Matsushita Electric Industrial, now Panasonic Corporation. Most variants include a media processor, working as an image processor or video processor. It is used in digital cameras, set-top box ...
/MN103S) *OPPO - MariSilicon X *Panasonic -
Venus Engine The Venus Engine is an image processing engine for digital cameras. It is developed by Panasonic, and almost all of their Lumix cameras use a version of it. It is based on the Panasonic MN103/MN103S. Image processors operate in four steps: rec ...
(based on Panasonic
MN103 The MN103 is a 32-bit microprocessor series developed by Matsushita Electric Industrial, now Panasonic Corporation. Most variants include a media processor, working as an image processor or video processor. It is used in digital cameras, set-top box ...
/MN103S) *Pentax - PRIME (Pentax Real IMage Engine) (newer variants based on Fujitsu
Milbeaut The Socionext Milbeaut image/video processors are media processors in multi-processor system on a chip architecture. Started by Fujitsu with the M-1 Series in 2000 each generation (2013: 7th) has several variants regarding included modules and proc ...
) *Qualcomm -
Qualcomm Spectra Qualcomm () is an American multinational corporation headquartered in San Diego, California, and Delaware General Corporation Law, incorporated in Delaware. It creates semiconductors, software, and services related to wireless technology. It ow ...
*Ricoh - GR engine (GR digital), Smooth Imaging Engine *Samsung - DRIMe (based on
Samsung The Samsung Group (or simply Samsung) ( ko, 삼성 ) is a South Korean multinational manufacturing conglomerate headquartered in Samsung Town, Seoul, South Korea. It comprises numerous affiliated businesses, most of them united under the ...
Exynos Exynos, formerly Hummingbird (), is a series of ARM-based system-on-chips developed by Samsung Electronics' System LSI division and manufactured by Samsung Foundry. It is a continuation of Samsung's earlier S3C, S5L and S5P line of SoCs. Exy ...
) *Sanyo - Platinum engine *Sigma - True *Sharp - ProPix *Socionext
Milbeaut
Family of ISPs - SC2000 (M-10V), SC2002 (M-11S) *Sony -
Bionz BIONZ is a line of image processors used in Sony digital cameras. It is currently used in many Sony α DSLR and mirrorless cameras. Image processing in the camera converts the raw data from a CCD or CMOS image sensor into the format that is sto ...
*THine - THP serie

with compatible SDK Kit for developing firmwar

*UNISOC - Vivimagic


Speed

With the ever-higher pixel count in image sensors, the image processor's speed becomes more critical: photographers don't want to wait for the camera's image processor to complete its job before they can carry on shooting - they don't even want to notice some processing is going on inside the camera. Therefore, image processors must be optimised to cope with more data in the same or even a shorter period of time.


See also

*
Color image pipeline An image pipeline or video pipeline is the set of components commonly used between an image source (such as a camera, a scanner, or the rendering engine in a computer game), and an image renderer (such as a television set, a computer screen, a compu ...
*
Image processing An image is a visual representation of something. It can be two-dimensional, three-dimensional, or somehow otherwise feed into the visual system to convey information. An image can be an artifact, such as a photograph or other two-dimensiona ...
*
Digital image processing Digital image processing is the use of a digital computer to process digital images through an algorithm. As a subcategory or field of digital signal processing, digital image processing has many advantages over analog image processing. It allo ...
*
Digital image editing Image editing encompasses the processes of altering images, whether they are Digital photography, digital photographs, traditional Photographic processing, photo-chemical photographs, or illustrations. Traditional analog image editing is kno ...
*
Demosaicing A demosaicing (also de-mosaicing, demosaicking or debayering) algorithm is a digital image process used to reconstruct a full color image from the incomplete color samples output from an image sensor overlaid with a color filter array (CFA). It is ...


References

{{System on a chip Digital signal processors Photography equipment