HOME

TheInfoList



OR:

In
numerical analysis Numerical analysis is the study of algorithms that use numerical approximation (as opposed to symbolic computation, symbolic manipulations) for the problems of mathematical analysis (as distinguished from discrete mathematics). It is the study of ...
and
scientific computing Computational science, also known as scientific computing or scientific computation (SC), is a field in mathematics that uses advanced computing capabilities to understand and solve complex problems. It is an area of science that spans many disc ...
, a sparse matrix or sparse array is a
matrix Matrix most commonly refers to: * ''The Matrix'' (franchise), an American media franchise ** ''The Matrix'', a 1999 science-fiction action film ** "The Matrix", a fictional setting, a virtual reality environment, within ''The Matrix'' (franchis ...
in which most of the elements are zero. There is no strict definition regarding the proportion of zero-value elements for a matrix to qualify as sparse but a common criterion is that the number of non-zero elements is roughly equal to the number of rows or columns. By contrast, if most of the elements are non-zero, the matrix is considered dense. The number of zero-valued elements divided by the total number of elements (e.g., ''m'' × ''n'' for an ''m'' × ''n'' matrix) is sometimes referred to as the sparsity of the matrix. Conceptually, sparsity corresponds to systems with few pairwise interactions. For example, consider a line of balls connected by springs from one to the next: this is a sparse system as only adjacent balls are coupled. By contrast, if the same line of balls were to have springs connecting each ball to all other balls, the system would correspond to a dense matrix. The concept of sparsity is useful in
combinatorics Combinatorics is an area of mathematics primarily concerned with counting, both as a means and an end in obtaining results, and certain properties of finite structures. It is closely related to many other areas of mathematics and has many appl ...
and application areas such as
network theory Network theory is the study of graphs as a representation of either symmetric relations or asymmetric relations between discrete objects. In computer science and network science, network theory is a part of graph theory: a network can be defi ...
and
numerical analysis Numerical analysis is the study of algorithms that use numerical approximation (as opposed to symbolic computation, symbolic manipulations) for the problems of mathematical analysis (as distinguished from discrete mathematics). It is the study of ...
, which typically have a low density of significant data or connections. Large sparse matrices often appear in
scientific Science is a systematic endeavor that builds and organizes knowledge in the form of testable explanations and predictions about the universe. Science may be as old as the human species, and some of the earliest archeological evidence for ...
or
engineering Engineering is the use of scientific method, scientific principles to design and build machines, structures, and other items, including bridges, tunnels, roads, vehicles, and buildings. The discipline of engineering encompasses a broad rang ...
applications when solving
partial differential equation In mathematics, a partial differential equation (PDE) is an equation which imposes relations between the various partial derivatives of a Multivariable calculus, multivariable function. The function is often thought of as an "unknown" to be sol ...
s. When storing and manipulating sparse matrices on a
computer A computer is a machine that can be programmed to Execution (computing), carry out sequences of arithmetic or logical operations (computation) automatically. Modern digital electronic computers can perform generic sets of operations known as C ...
, it is beneficial and often necessary to use specialized
algorithm In mathematics and computer science, an algorithm () is a finite sequence of rigorous instructions, typically used to solve a class of specific Computational problem, problems or to perform a computation. Algorithms are used as specificat ...
s and
data structure In computer science, a data structure is a data organization, management, and storage format that is usually chosen for efficient access to data. More precisely, a data structure is a collection of data values, the relationships among them, a ...
s that take advantage of the sparse structure of the matrix. Specialized computers have been made for sparse matrices, as they are common in the machine learning field. Operations using standard dense-matrix structures and algorithms are slow and inefficient when applied to large sparse matrices as processing and
memory Memory is the faculty of the mind by which data or information is encoded, stored, and retrieved when needed. It is the retention of information over time for the purpose of influencing future action. If past events could not be remembered, ...
are wasted on the zeros. Sparse data is by nature more easily compressed and thus requires significantly less storage. Some very large sparse matrices are infeasible to manipulate using standard dense-matrix algorithms.


Storing a sparse matrix

A matrix is typically stored as a two-dimensional array. Each entry in the array represents an element of the matrix and is accessed by the two indices and . Conventionally, is the row index, numbered from top to bottom, and is the column index, numbered from left to right. For an matrix, the amount of memory required to store the matrix in this format is proportional to (disregarding the fact that the dimensions of the matrix also need to be stored). In the case of a sparse matrix, substantial memory requirement reductions can be realized by storing only the non-zero entries. Depending on the number and distribution of the non-zero entries, different data structures can be used and yield huge savings in memory when compared to the basic approach. The trade-off is that accessing the individual elements becomes more complex and additional structures are needed to be able to recover the original matrix unambiguously. Formats can be divided into two groups: * Those that support efficient modification, such as DOK (Dictionary of keys), LIL (List of lists), or COO (Coordinate list). These are typically used to construct the matrices. * Those that support efficient access and matrix operations, such as CSR (Compressed Sparse Row) or CSC (Compressed Sparse Column).


Dictionary of keys (DOK)

DOK consists of a
dictionary A dictionary is a listing of lexemes from the lexicon of one or more specific languages, often arranged alphabetically (or by radical and stroke for ideographic languages), which may include information on definitions, usage, etymologies ...
that maps -
pairs Concentration, also known as Memory, Shinkei-suijaku (Japanese meaning "nervous breakdown"), Matching Pairs, Match Match, Match Up, Pelmanism, Pexeso or simply Pairs, is a card game in which all of the cards are laid face down on a surface and tw ...
to the value of the elements. Elements that are missing from the dictionary are taken to be zero. The format is good for incrementally constructing a sparse matrix in random order, but poor for iterating over non-zero values in lexicographical order. One typically constructs a matrix in this format and then converts to another more efficient format for processing.


List of lists (LIL)

LIL stores one list per row, with each entry containing the column index and the value. Typically, these entries are kept sorted by column index for faster lookup. This is another format good for incremental matrix construction.


Coordinate list (COO)

COO stores a list of tuples. Ideally, the entries are sorted first by row index and then by column index, to improve random access times. This is another format that is good for incremental matrix construction.


Compressed sparse row (CSR, CRS or Yale format)

The ''compressed sparse row'' (CSR) or ''compressed row storage'' (CRS) or Yale format represents a matrix by three (one-dimensional) arrays, that respectively contain nonzero values, the extents of rows, and column indices. It is similar to COO, but compresses the row indices, hence the name. This format allows fast row access and matrix-vector multiplications (). The CSR format has been in use since at least the mid-1960s, with the first complete description appearing in 1967. The CSR format stores a sparse matrix in row form using three (one-dimensional) arrays . Let denote the number of nonzero entries in . (Note that zero-based indices shall be used here.) * The arrays and are of length , and contain the non-zero values and the column indices of those values respectively. * The array is of length and encodes the index in and where the given row starts. This is equivalent to encoding the total number of nonzeros above row . The last element is , i.e., the fictitious index in immediately after the last valid index . For example, the matrix \begin 5 & 0 & 0 & 0 \\ 0 & 8 & 0 & 0 \\ 0 & 0 & 3 & 0 \\ 0 & 6 & 0 & 0 \\ \end is a matrix with 4 nonzero elements, hence V =
5 8 3 6 5 (five) is a number, numeral and digit. It is the natural number, and cardinal number, following 4 and preceding 6, and is a prime number. It has attained significance throughout history in part because typical humans have five digits on eac ...
COL_INDEX = 0 1 2 1 ROW_INDEX = 0 1 2 3 4 assuming a zero-indexed language. To extract a row, we first define: row_start = ROW_INDEX ow row_end = ROW_INDEX ow + 1 Then we take slices from V and COL_INDEX starting at row_start and ending at row_end. To extract the row 1 (the second row) of this matrix we set row_start=1 and row_end=2. Then we make the slices V :2= /code> and COL_INDEX :2= /code>. We now know that in row 1 we have one element at column 1 with value 8. In this case the CSR representation contains 13 entries, compared to 16 in the original matrix. The CSR format saves on memory only when . Another example, the matrix \begin 10 & 20 & 0 & 0 & 0 & 0 \\ 0 & 30 & 0 & 40 & 0 & 0 \\ 0 & 0 & 50 & 60 & 70 & 0 \\ 0 & 0 & 0 & 0 & 0 & 80 \\ \end is a matrix (24 entries) with 8 nonzero elements, so V =
10 20 30 40 50 60 70 80 1 (one, unit, unity) is a number representing a single or the only entity. 1 is also a numerical digit and represents a single unit (measurement), unit of counting or measurement. For example, a line segment of ''unit length'' is a line segment ...
COL_INDEX = 0 1 1 3 2 3 4 5 ROW_INDEX = 0 2 4 7 8 The whole is stored as 21 entries: 8 in , 8 in , and 5 in . * splits the array into rows: (10, 20) (30, 40) (50, 60, 70) (80), indicating the index of (and ) where each row starts and ends; * aligns values in columns: (10, 20, ...) (0, 30, 0, 40, ...)(0, 0, 50, 60, 70, 0) (0, 0, 0, 0, 0, 80). Note that in this format, the first value of is always zero and the last is always , so they are in some sense redundant (although in programming languages where the array length needs to be explicitly stored, would not be redundant). Nonetheless, this does avoid the need to handle an exceptional case when computing the length of each row, as it guarantees the formula works for any row . Moreover, the memory cost of this redundant storage is likely insignificant for a sufficiently large matrix. The (old and new) Yale sparse matrix formats are instances of the CSR scheme. The old Yale format works exactly as described above, with three arrays; the new format combines and into a single array and handles the diagonal of the matrix separately. For logical adjacency matrices, the data array can be omitted, as the existence of an entry in the row array is sufficient to model a binary adjacency relation. It is likely known as the Yale format because it was proposed in the 1977 Yale Sparse Matrix Package report from Department of Computer Science at Yale University.


Compressed sparse column (CSC or CCS)


CSC
is similar to CSR except that values are read first by column, a row index is stored for each value, and column pointers are stored. For example, CSC is , where is an array of the (top-to-bottom, then left-to-right) non-zero values of the matrix; is the row indices corresponding to the values; and, is the list of indexes where each column starts. The name is based on the fact that column index information is compressed relative to the COO format. One typically uses another format (LIL, DOK, COO) for construction. This format is efficient for arithmetic operations, column slicing, and matrix-vector products. Se

This is the traditional format for specifying a sparse matrix in MATLAB (via th

function).


Special structure


Banded

An important special type of sparse matrices is
band matrix Band or BAND may refer to: Places *Bánd, a village in Hungary *Band, Iran, a village in Urmia County, West Azerbaijan Province, Iran *Band, Mureș, a commune in Romania * Band-e Majid Khan, a village in Bukan County, West Azerbaijan Province, I ...
, defined as follows. The
lower bandwidth of a matrix Lower may refer to: *Lower (surname) *Lower Township, New Jersey *Lower Receiver (firearms) *Lower Wick Gloucestershire, England See also *Nizhny Nizhny (russian: Ни́жний; masculine), Nizhnyaya (; feminine), or Nizhneye (russian: Ни́ ...
is the smallest number such that the entry vanishes whenever . Similarly, the upper bandwidth is the smallest number such that whenever . For example, a
tridiagonal matrix In linear algebra, a tridiagonal matrix is a band matrix that has nonzero elements only on the main diagonal, the subdiagonal/lower diagonal (the first diagonal below this), and the supradiagonal/upper diagonal (the first diagonal above the main di ...
has lower bandwidth and upper bandwidth . As another example, the following sparse matrix has lower and upper bandwidth both equal to 3. Notice that zeros are represented with dots for clarity. \begin X & X & X & \cdot & \cdot & \cdot & \cdot & \\ X & X & \cdot & X & X & \cdot & \cdot & \\ X & \cdot & X & \cdot & X & \cdot & \cdot & \\ \cdot & X & \cdot & X & \cdot & X & \cdot & \\ \cdot & X & X & \cdot & X & X & X & \\ \cdot & \cdot & \cdot & X & X & X & \cdot & \\ \cdot & \cdot & \cdot & \cdot & X & \cdot & X & \\ \end Matrices with reasonably small upper and lower bandwidth are known as band matrices and often lend themselves to simpler algorithms than general sparse matrices; or one can sometimes apply dense matrix algorithms and gain efficiency simply by looping over a reduced number of indices. By rearranging the rows and columns of a matrix it may be possible to obtain a matrix with a lower bandwidth. A number of algorithms are designed for bandwidth minimization.


Diagonal

A very efficient structure for an extreme case of band matrices, the
diagonal matrix In linear algebra, a diagonal matrix is a matrix in which the entries outside the main diagonal are all zero; the term usually refers to square matrices. Elements of the main diagonal can either be zero or nonzero. An example of a 2×2 diagonal ma ...
, is to store just the entries in the
main diagonal In linear algebra, the main diagonal (sometimes principal diagonal, primary diagonal, leading diagonal, major diagonal, or good diagonal) of a matrix A is the list of entries a_ where i = j. All off-diagonal elements are zero in a diagonal matrix. ...
as a
one-dimensional array In computer science, an array is a data structure consisting of a collection of ''elements'' (values or variables), each identified by at least one ''array index'' or ''key''. An array is stored such that the position of each element can be co ...
, so a diagonal matrix requires only entries.


Symmetric

A symmetric sparse matrix arises as the
adjacency matrix In graph theory and computer science, an adjacency matrix is a square matrix used to represent a finite graph. The elements of the matrix indicate whether pairs of vertices are adjacent or not in the graph. In the special case of a finite simp ...
of an
undirected graph In discrete mathematics, and more specifically in graph theory, a graph is a structure amounting to a set of objects in which some pairs of the objects are in some sense "related". The objects correspond to mathematical abstractions called '' v ...
; it can be stored efficiently as an
adjacency list In graph theory and computer science, an adjacency list is a collection of unordered lists used to represent a finite graph. Each unordered list within an adjacency list describes the set of neighbors of a particular vertex in the graph. This is ...
.


Block diagonal

A
block-diagonal matrix In mathematics, a block matrix or a partitioned matrix is a matrix that is '' interpreted'' as having been broken into sections called blocks or submatrices. Intuitively, a matrix interpreted as a block matrix can be visualized as the original ma ...
consists of sub-matrices along its diagonal blocks. A block-diagonal matrix has the form \mathbf = \begin \mathbf_1 & 0 & \cdots & 0 \\ 0 & \mathbf_2 & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & \mathbf_n \end, where is a square matrix for all .


Reducing fill-in

The ''fill-in'' of a matrix are those entries that change from an initial zero to a non-zero value during the execution of an algorithm. To reduce the memory requirements and the number of arithmetic operations used during an algorithm, it is useful to minimize the fill-in by switching rows and columns in the matrix. The
symbolic Cholesky decomposition In the mathematical subfield of numerical analysis the symbolic Cholesky decomposition is an algorithm used to determine the non-zero pattern for the L factors of a symmetric sparse matrix when applying the Cholesky decomposition or variants. Algo ...
can be used to calculate the worst possible fill-in before doing the actual
Cholesky decomposition In linear algebra, the Cholesky decomposition or Cholesky factorization (pronounced ) is a decomposition of a Hermitian, positive-definite matrix into the product of a lower triangular matrix and its conjugate transpose, which is useful for effici ...
. There are other methods than the
Cholesky decomposition In linear algebra, the Cholesky decomposition or Cholesky factorization (pronounced ) is a decomposition of a Hermitian, positive-definite matrix into the product of a lower triangular matrix and its conjugate transpose, which is useful for effici ...
in use. Orthogonalization methods (such as QR factorization) are common, for example, when solving problems by least squares methods. While the theoretical fill-in is still the same, in practical terms the "false non-zeros" can be different for different methods. And symbolic versions of those algorithms can be used in the same manner as the symbolic Cholesky to compute worst case fill-in.


Solving sparse matrix equations

Both
iterative Iteration is the repetition of a process in order to generate a (possibly unbounded) sequence of outcomes. Each repetition of the process is a single iteration, and the outcome of each iteration is then the starting point of the next iteration. ...
and direct methods exist for sparse matrix solving. Iterative methods, such as
conjugate gradient In mathematics, the conjugate gradient method is an algorithm for the numerical solution of particular systems of linear equations, namely those whose matrix is positive-definite. The conjugate gradient method is often implemented as an iterative ...
method and
GMRES In mathematics, the generalized minimal residual method (GMRES) is an iterative method for the numerical solution of an indefinite nonsymmetric system of linear equations. The method approximates the solution by the vector in a Krylov subspace with ...
utilize fast computations of matrix-vector products A x_i, where matrix A is sparse. The use of
preconditioner In mathematics, preconditioning is the application of a transformation, called the preconditioner, that conditions a given problem into a form that is more suitable for numerical solving methods. Preconditioning is typically related to reducing ...
s can significantly accelerate convergence of such iterative methods.


Software

Many software libraries support sparse matrices, and provide solvers for sparse matrix equations. The following are open-source:
SuiteSparse
a suite of sparse matrix algorithms, geared toward the direct solution of sparse linear systems. * PETSc, a large C library, containing many different matrix solvers for a variety of matrix storage formats. *
Trilinos Trilinos is a collection of open-source software, open-source software libraries, called ''packages'', intended to be used as building blocks for the development of scientific applications. The word "Trilinos" is Greek and conveys the idea of "a st ...
, a large C++ library, with sub-libraries dedicated to the storage of dense and sparse matrices and solution of corresponding linear systems. * Eigen3 is a C++ library that contains several sparse matrix solvers. However, none of them are parallelized. *
MUMPS MUMPS ("Massachusetts General Hospital Utility Multi-Programming System"), or M, is an imperative, high-level programming language with an integrated transaction processing key–value database. It was originally developed at Massachusetts Gener ...
(MUltifrontal Massively Parallel sparse direct Solver), written in Fortran90, is a
frontal solver A frontal solver, conceived by Bruce Irons (engineer), Bruce Irons, is an approach to solving sparse matrix, sparse linear systems which is used extensively in finite element analysis. It is a variant of Gauss elimination that automatically avoids a ...
. *
deal.II deal.II is a free, open-source library to solve partial differential equations using the finite element method.  The current release is version 9.4, released in June 2022. It is one of the most widely used finite element libraries, and pro ...
, a finite element library that also has a sub-library for sparse linear systems and their solution. *
DUNE A dune is a landform composed of wind- or water-driven sand. It typically takes the form of a mound, ridge, or hill. An area with dunes is called a dune system or a dune complex. A large dune complex is called a dune field, while broad, f ...
, another finite element library that also has a sub-library for sparse linear systems and their solution.
PaStix

SuperLU
*
Armadillo Armadillos (meaning "little armored ones" in Spanish) are New World placental mammals in the order Cingulata. The Chlamyphoridae and Dasypodidae are the only surviving families in the order, which is part of the superorder Xenarthra, along wi ...
provides a user-friendly C++ wrapper for BLAS and LAPACK. *
SciPy SciPy (pronounced "sigh pie") is a free and open-source Python library used for scientific computing and technical computing. SciPy contains modules for optimization, linear algebra, integration, interpolation, special functions, FFT, signal ...
provides support for several sparse matrix formats, linear algebra, and solvers.
SPArse Matrix (spam)
R and Python package for sparse matrices.

Tools for handling sparse arrays *
ALGLIB ALGLIB is a cross-platform open source numerical analysis and data processing library. It can be used from several programming languages (C++, C#, VB.NET, Python, Delphi). ALGLIB started in 1999 and has a long history of steady development with r ...
is a C++ and C# library with sparse linear algebra support *
ARPACK ARPACK, the ARnoldi PACKage, is a numerical computation, numerical software library written in Fortran, FORTRAN 77 for solving large scale eigenvalue problems in the matrix-free methods, matrix-free fashion. The package is designed to compute a fe ...
Fortran 77 library for sparse matrix diagonalization and manipulation, using the Arnoldi algorithm
SPARSE
Reference (old)
NIST The National Institute of Standards and Technology (NIST) is an agency of the United States Department of Commerce whose mission is to promote American innovation and industrial competitiveness. NIST's activities are organized into physical sci ...
package for (real or complex) sparse matrix diagonalization *
SLEPc SLEPc is a software library for the parallel computation of eigenvalues and eigenvectors of large, sparse matrices. It can be seen as a module of Portable, Extensible Toolkit for Scientific Computation, PETSc that provides solvers for different ty ...
Library for solution of large scale linear systems and sparse matrices
Sympiler
a domain-specific code generator and library for solving linear systems and quadratic programming problems. *
scikit-learn scikit-learn (formerly scikits.learn and also known as sklearn) is a free software machine learning library for the Python programming language. It features various classification, regression and clustering algorithms including support-vector m ...
, a Python library for
machine learning Machine learning (ML) is a field of inquiry devoted to understanding and building methods that 'learn', that is, methods that leverage data to improve performance on some set of tasks. It is seen as a part of artificial intelligence. Machine ...
, provides support for sparse matrices and solvers.
sprs
implements sparse matrix data structures and linear algebra algorithms in pure Rust.
Basic Matrix Library (bml)
supports several sparse matrix formats and linear algebra algorithms with bindings for C, C++, and Fortran.


History

The term ''sparse matrix'' was possibly coined by
Harry Markowitz Harry Max Markowitz (born August 24, 1927) is an American economist who received the 1989 John von Neumann Theory Prize and the 1990 Nobel Memorial Prize in Economic Sciences. Markowitz is a professor of finance at the Rady School of Management ...
who initiated some pioneering work but then left the field.


See also


Notes


References

* * * (This book, by a professor at the State University of New York at Stony Book, was the first book exclusively dedicated to Sparse Matrices. Graduate courses using this as a textbook were offered at that University in the early 1980s). * * * Also NOAA Technical Memorandum NOS NGS-4, National Geodetic Survey, Rockville, MD.


Further reading

* *
Sparse Matrix Algorithms Research
at the Texas A&M University.
SuiteSparse Matrix Collection

SMALL project
A EU-funded project on sparse models, algorithms and dictionary learning for large-scale data. {{Numerical linear algebra Arrays