In scientific visualization and

computer graphics Computer graphics deals with generating images with the aid of computers. Today, computer graphics is a core technology in digital photography, film, video games, cell phone and computer displays, and many specialized applications. A great de ...

, volume rendering is a set of techniques used to display a 2D projection of a 3D discretely sampled data set, typically a 3D

scalar field In mathematics and physics, a scalar field is a function associating a single number to every point in a space – possibly physical space. The scalar may either be a pure mathematical number ( dimensionless) or a scalar physical quantity ...

. A typical 3D data set is a group of 2D slice images acquired by a CT, MRI, or MicroCT scanner. Usually these are acquired in a regular pattern (e.g., one slice for each millimeter of depth) and usually have a regular number of image

pixel In digital imaging, a pixel (abbreviated px), pel, or picture element is the smallest addressable element in a raster image, or the smallest point in an all points addressable display device. In most digital display devices, pixels are the ...

s in a regular pattern. This is an example of a regular volumetric grid, with each volume element, or

voxel In 3D computer graphics, a voxel represents a value on a regular grid in three-dimensional space. As with pixels in a 2D bitmap, voxels themselves do not typically have their position (i.e. coordinates) explicitly encoded with their values. I ...

represented by a single value that is obtained by sampling the immediate area surrounding the voxel. To render a 2D projection of the 3D data set, one first needs to define a

camera A camera is an optical instrument that can capture an image. Most cameras can capture 2D images, with some more advanced models being able to capture 3D images. At a basic level, most cameras consist of sealed boxes (the camera body), with ...

in space relative to the volume. Also, one needs to define the opacity and color of every voxel. This is usually defined using an

RGBA RGBA stands for red green blue alpha. While it is sometimes described as a color space, it is actually a three-channel RGB color model supplemented with a fourth ''alpha channel''. Alpha indicates how opaque each pixel is and allows an image to ...

(for red, green, blue, alpha) transfer function that defines the RGBA value for every possible voxel value. For example, a volume may be viewed by extracting isosurfaces (surfaces of equal values) from the volume and rendering them as polygonal meshes or by rendering the volume directly as a block of data. The marching cubes algorithm is a common technique for extracting an isosurface from volume data. Direct volume rendering is a computationally intensive task that may be performed in several ways. Another method of volume rendering is Ray marching.

Scope

CT presentation as thin slice, projection and volume rendering

Volume rendering is distinguished from thin slice

tomography Tomography is imaging by sections or sectioning that uses any kind of penetrating wave. The method is used in radiology, archaeology, biology, atmospheric science, geophysics, oceanography, plasma physics, materials science, astrophysics, ...

presentations, and is also generally distinguished from projections of 3D models, including maximum intensity projection. Still, technically, all volume renderings become projections when viewed on a 2-dimensional display, making the distinction between projections and volume renderings a bit vague. Nevertheless, the epitomes of volume rendering models feature a mix of for example coloring and shading in order to create realistic and/or observable representations.

Direct volume rendering

A direct volume rendererMarc Levoy, "Display of Surfaces from Volume Data",

IEEE The Institute of Electrical and Electronics Engineers (IEEE) is a 501(c)(3) professional association for electronic engineering and electrical engineering (and associated disciplines) with its corporate office in New York City and its operati ...

CG&A, May 1988
Archive of Paper
/ref> requires every sample value to be mapped to opacity and a color. This is done with a "

transfer function In engineering, a transfer function (also known as system function or network function) of a system, sub-system, or component is a mathematical function that theoretically models the system's output for each possible input. They are widely used ...

" which can be a simple ramp, a piecewise linear function or an arbitrary table. Once converted to an RGBA color model (for red, green, blue, alpha) value, the composed RGBA result is projected on the corresponding pixel of the frame buffer. The way this is done depends on the rendering technique. A combination of these techniques is possible. For instance, a shear warp implementation could use texturing hardware to draw the aligned slices in the off-screen buffer.

Volume ray casting

The technique of volume ray casting can be derived directly from the

rendering equation In computer graphics, the rendering equation is an integral equation in which the equilibrium radiance leaving a point is given as the sum of emitted plus reflected radiance under a geometric optics approximation. It was simultaneously introduc ...

. It provides results of very high quality, usually considered to provide the best image quality. Volume ray casting is classified as image based volume rendering technique, as the computation emanates from the output image, not the input volume data as is the case with object based techniques. In this technique, a ray is generated for each desired image pixel. Using a simple camera model, the ray starts at the center of projection of the camera (usually the eye point) and passes through the image pixel on the imaginary image plane floating in between the camera and the volume to be rendered. The ray is clipped by the boundaries of the volume in order to save time. Then the ray is sampled at regular or adaptive intervals throughout the volume. The data is interpolated at each sample point, the transfer function applied to form an RGBA sample, the sample is composited onto the accumulated RGBA of the ray, and the process repeated until the ray exits the volume. The RGBA color is converted to an RGB color and deposited in the corresponding image pixel. The process is repeated for every pixel on the screen to form the completed image.

Splatting

This is a technique which trades quality for speed. Here, every volume element is splatted, as Lee Westover said, like a snow ball, on to the viewing surface in back to front order. These splats are rendered as disks whose properties (color and transparency) vary diametrically in normal (

Gaussian Carl Friedrich Gauss (1777–1855) is the eponym of all of the topics listed below. There are over 100 topics all named after this German mathematician and scientist, all in the fields of mathematics, physics, and astronomy. The English eponym ...

) manner. Flat disks and those with other kinds of property distribution are also used depending on the application.

Shear warp

The shear warp approach to volume rendering was developed by Cameron and Undrill, popularized by Philippe Lacroute and

Marc Levoy Marc Levoy is a computer graphics researcher and Professor Emeritus of Computer Science and Electrical Engineering at Stanford University, a vice president and Fellow at Adobe Inc., and (until 2020) a Distinguished Engineer at Google. He is noted ...

. In this technique, the viewing transformation is transformed such that the nearest face of the volume becomes axis aligned with an off-screen image

data buffer In computer science, a data buffer (or just buffer) is a region of a memory used to temporarily store data while it is being moved from one place to another. Typically, the data is stored in a buffer as it is retrieved from an input device (such a ...

with a fixed scale of voxels to pixels. The volume is then rendered into this buffer using the far more favorable memory alignment and fixed scaling and blending factors. Once all slices of the volume have been rendered, the buffer is then warped into the desired orientation and scaled in the displayed image. This technique is relatively fast in software at the cost of less accurate sampling and potentially worse image quality compared to ray casting. There is memory overhead for storing multiple copies of the volume, for the ability to have near axis aligned volumes. This overhead can be mitigated using run length encoding.

Texture-based volume rendering

Many 3D graphics systems use

texture mapping Texture mapping is a method for mapping a texture on a computer-generated graphic. Texture here can be high frequency detail, surface texture, or color. History The original technique was pioneered by Edwin Catmull in 1974. Texture mappi ...

to apply images, or textures, to geometric objects. Commodity PC graphics cards are fast at texturing and can efficiently render slices of a 3D volume, with real time interaction capabilities.

Workstation A workstation is a special computer designed for technical or scientific applications. Intended primarily to be used by a single user, they are commonly connected to a local area network and run multi-user operating systems. The term ''workst ...

GPU A graphics processing unit (GPU) is a specialized electronic circuit designed to manipulate and alter memory to accelerate the creation of images in a frame buffer intended for output to a display device. GPUs are used in embedded systems, mobi ...

s are even faster, and are the basis for much of the production volume visualization used in

medical imaging Medical imaging is the technique and process of imaging the interior of a body for clinical analysis and medical intervention, as well as visual representation of the function of some organs or tissues (physiology). Medical imaging seeks to re ...

, oil and gas, and other markets (2007). In earlier years, dedicated 3D texture mapping systems were used on graphics systems such as

Silicon Graphics Silicon Graphics, Inc. (stylized as SiliconGraphics before 1999, later rebranded SGI, historically known as Silicon Graphics Computer Systems or SGCS) was an American high-performance computing manufacturer, producing computer hardware and soft ...

InfiniteReality InfiniteReality refers to a 3D graphics hardware architecture and a family of graphics systems that implemented the aforementioned hardware architecture that was developed and manufactured by Silicon Graphics from 1996 to 2005. The InfiniteRealit ...

, HP Visualize FX

graphics accelerator A graphics processing unit (GPU) is a specialized electronic circuit designed to manipulate and alter memory to accelerate the creation of images in a frame buffer intended for output to a display device. GPUs are used in embedded systems, mobil ...

, and others. This technique was first described by

Bill Hibbard Bill Hibbard is a scientist at the University of Wisconsin–Madison Space Science and Engineering Center working on visualization and machine intelligence. He is principal author of the Vis5D, Cave5D, and VisAD open-source visualization system ...

and Dave Santek.Hibbard W., Santek D.
"Interactivity is the key"
''Chapel Hill Workshop on Volume Visualization'', University of North Carolina, Chapel Hill, 1989, pp. 39–43. These slices can either be aligned with the volume and rendered at an angle to the viewer, or aligned with the viewing plane and sampled from unaligned slices through the volume. Graphics hardware support for 3D textures is needed for the second technique. Volume aligned texturing produces images of reasonable quality, though there is often a noticeable transition when the volume is rotated.

Hardware-accelerated volume rendering

Due to the extremely parallel nature of direct volume rendering, special purpose volume rendering hardware was a rich research topic before

volume rendering became fast enough. The most widely cited technology was the VolumePro real-time ray-casting system, developed by Hanspeter Pfister and scientists at

Mitsubishi Electric Research Laboratories Mitsubishi Electric Research Laboratories (MERL) is a subsidiary of Mitsubishi Electric US Holdings, Inc., which, in its turn, is the principal subsidiary of Mitsubishi Electric in the United States. MERL is the North American arm of the Corpora ...

, which used high memory bandwidth and brute force to render using the ray casting algorithm. The technology was transferred to TeraRecon, Inc. and two generations of ASICs were produced and sold. The VP1000 was released in 2002 and the VP2000 in 2007. A recently exploited technique to accelerate traditional volume rendering algorithms such as ray-casting is the use of modern graphics cards. Starting with the programmable

pixel shader In computer graphics, a shader is a computer program that calculates the appropriate levels of light, darkness, and color during the Rendering (computer graphics), rendering of a 3D scene - a process known as ''shading''. Shaders have evolved ...

s, people recognized the power of parallel operations on multiple pixels and began to perform ''general-purpose computing on (the) graphics processing units'' (GPGPU). The

s are able to read and write randomly from video memory and perform some basic mathematical and logical calculations. These

SIMD Single instruction, multiple data (SIMD) is a type of parallel processing in Flynn's taxonomy. SIMD can be internal (part of the hardware design) and it can be directly accessible through an instruction set architecture (ISA), but it shoul ...

processors were used to perform general calculations such as rendering polygons and signal processing. In recent

generations, the pixel shaders now are able to function as

MIMD In computing, multiple instruction, multiple data (MIMD) is a technique employed to achieve parallelism. Machines using MIMD have a number of processors that function asynchronously and independently. At any time, different processors may be exe ...

processors (now able to independently branch) utilizing up to 1 GB of texture memory with floating point formats. With such power, virtually any algorithm with steps that can be performed in parallel, such as

volume ray casting Volume ray casting, sometimes called volumetric ray casting, volumetric ray tracing, or volume ray marching, is an image-based volume rendering technique. It computes 2D images from 3D volumetric data sets (3D scalar fields). Volume ray casting, ...

tomographic reconstruction Tomographic reconstruction is a type of multidimensional inverse problem where the challenge is to yield an estimate of a specific system from a finite number of projections. The mathematical basis for tomographic imaging was laid down by Johann ...

, can be performed with tremendous acceleration. The programmable

pixel shaders In computer graphics, a shader is a computer program that calculates the appropriate levels of light, darkness, and color during the rendering of a 3D scene - a process known as '' shading''. Shaders have evolved to perform a variety of ...

can be used to simulate variations in the characteristics of lighting, shadow, reflection, emissive color and so forth. Such simulations can be written using high level shading languages.

Optimization techniques

The primary goal of optimization is to skip as much of the volume as possible. A typical medical data set can be 1 GB in size. To render that at 30 frame/s requires an extremely fast memory bus. Skipping voxels means that less information needs to be processed.

Empty space skipping

Often, a volume rendering system will have a system for identifying regions of the volume containing no visible material. This information can be used to avoid rendering these transparent regions.Sherbondy A., Houston M., Napel S.: ''Fast volume segmentation with simultaneous visualization using programmable graphics hardware.'' In Proceedings of

Visualization (2003), pp. 171–176.

Early ray termination

This is a technique used when the volume is rendered in front to back order. For a ray through a pixel, once sufficient dense material has been encountered, further samples will make no significant contribution to the pixel and so may be neglected.

Octree and BSP space subdivision

The use of hierarchical structures such as octree and BSP-tree could be very helpful for both compression of volume data and speed optimization of volumetric ray casting process.

Volume segmentation

CT angiography of the head without and with bone removal

Image segmentation is a manual or automatic procedure that can be used to section out large portions of the volume that one considers uninteresting before rendering, the amount of calculations that have to be made by ray casting or texture blending can be significantly reduced. This reduction can be as much as from O(n) to O(log n) for n sequentially indexed voxels. Volume segmentation also has significant performance benefits for other ray tracing algorithms. Volume segmentation can subsequently be used to highlight structures of interest.

Multiple and adaptive resolution representation

By representing less interesting regions of the volume in a coarser resolution, the data input overhead can be reduced. On closer observation, the data in these regions can be populated either by reading from memory or disk, or by

interpolation In the mathematical field of numerical analysis, interpolation is a type of estimation, a method of constructing (finding) new data points based on the range of a discrete set of known data points. In engineering and science, one often has ...

. The coarser resolution volume is resampled to a smaller size in the same way as a 2D mipmap image is created from the original. These smaller volume are also used by themselves while rotating the volume to a new orientation.

Pre-integrated volume rendering

Pre-integrated volume renderingMax N., Hanrahan P., Crawfis R.:
Area and volume coherence for efficient visualization of 3D scalar functions
'' In Computer Graphics (San Diego Workshop on Volume Visualization, 1990) vol. 24, pp. 27–33. is a method that can reduce sampling artifacts by pre-computing much of the required data. It is especially useful in hardware-accelerated applicationsLum E., Wilson B., Ma K.:
High-Quality Lighting and Efficient Pre-Integration for Volume Rendering
'' In Eurographics/

Symposium on Visualization 2004. because it improves quality without a large performance impact. Unlike most other optimizations, this does not skip voxels. Rather it reduces the number of samples needed to accurately display a region of voxels. The idea is to render the intervals between the samples instead of the samples themselves. This technique captures rapidly changing material, for example the transition from muscle to bone with much less computation.

Image-based meshing

Image-based meshing is the automated process of creating computer models from 3D image data (such as MRI, CT, Industrial CT or

microtomography X-ray microtomography, like tomography and X-ray computed tomography, uses X-rays to create cross-sections of a physical object that can be used to recreate a virtual model (3D model) without destroying the original object. The prefix ''micro-'' ...

) for computational analysis and design, e.g. CAD, CFD, and FEA.

Temporal reuse of voxels

For a complete display view, only one voxel per pixel (the front one) is required to be shown (although more can be used for smoothing the image), if animation is needed, the front voxels to be shown can be cached and their location relative to the camera can be recalculated as it moves. Where display voxels become too far apart to cover all the pixels, new front voxels can be found by ray casting or similar, and where two voxels are in one pixel, the front one can be kept.

List of related software

; Open source *

3D Slicer 3D Slicer (Slicer) is a Free software, free and Open-source software, open source software package for image analysis and scientific visualization. Slicer is used in a variety of medical applications, including autism, multiple sclerosis, syste ...

– a software package for scientific visualization and image analysis * ClearVolume – a GPU ray-casting based, live 3D visualization library designed for high-end volumetric light sheet microscopes. * ParaView – a cross-platform, large data analysis and visualization application. ParaView users can quickly build visualizations to analyze their data using qualitative and quantitative techniques. ParaView is built on VTK (below). * Studierfenster (StudierFenster) – a free, non-commercial Open Science client/server-based Medical Imaging Processing (MIP) online framework. * Vaa3D – a 3D, 4D and 5D volume rendering and image analysis platform for gigabytes and terabytes of large images (based on OpenGL) especially in the microscopy image field. Also cross-platform with Mac, Windows, and Linux versions. Include a comprehensive plugin interface and 100 plugins for image analysis. Also render multiple types of surface objects. *

VisIt Visit refer as go to see and spend time with socially. Visit may refer to: *State visit, a formal visit by a head of state to a foreign country *Conjugal visit, in which a prisoner is permitted to spend several hours or days in private with a visit ...

– a cross-platform interactive parallel visualization and graphical analysis tool for viewing scientific data. *

Volume cartography Volume Cartography is the name of a computer program for locating and mapping 2-dimensional surfaces within a 3-dimensional object. X-rays can reveal minute details of what is in an object, and computer program such as Volume Cartography can organ ...

– an open source software used in recovering the En-Gedi Scroll. * Voreen – a cross-platform rapid application development framework for the interactive visualization and analysis of multi-modal volumetric data sets. It provides GPU-based volume rendering and data analysis techniques *

VTK The Visualization Toolkit (VTK) is an open-source software system for 3D computer graphics, image processing and scientific visualization.''Visualization Handbook'', Academic Press, 2005, Chapter 30: the Visualization Toolkit/ref> VTK is distribu ...

– a general-purpose C++ toolkit for data processing, visualization, 3D interaction, computational geometry, with Python and Java bindings. Also, VTK.js provides a JavaScript implementation. ; Commercial * Ambivu 3D Workstation – a medical imaging workstation that offers a range of volume rendering modes (based on OpenGL) * Amira – a 3D visualization and analysis software for scientists and researchers (in life sciences and biomedical) * Imaris – a scientific software module that delivers all the necessary functionality for data management, visualization, analysis, segmentation and interpretation of 3D and 4D microscopy datasets *

MeVisLab MeVisLab is a cross-platform application framework for medical image processing and scientific visualization. It includes advanced algorithms for image registration, segmentation, and quantitative morphological and functional image analysis. An ...

– cross-platform software for medical image processing and visualization (based on OpenGL and Open Inventor) * Open Inventor – a high-level 3D API for 3D graphics software development (C++, .NET, Java) * ScanIP – an image processing and image-based meshing platform that can render scan data (MRI, CT, Micro-CT...) in 3D directly after import. * tomviz – a 3D visualization platform for scientists and researchers that can utilize Python scripts for advanced 3D data processing. *

VoluMedic VoluMedic(TM) by VoluMedic LLis a 3d- volume rendering software based on LightWave3d by NewTek (plugin/addon). It offers software volume rendering as well as hardware volume rendering (for interactive previews) of volumetric datasets. Image:Volumed ...

– a volume slicing and rendering software