In
computer science
Computer science is the study of computation, automation, and information. Computer science spans theoretical disciplines (such as algorithms, theory of computation, information theory, and automation) to Applied science, practical discipli ...
, a sparse file is a type of
computer file
A computer file is a computer resource for recording data in a computer storage device, primarily identified by its file name. Just as words can be written to paper, so can data be written to a computer file. Files can be shared with and transfe ...
that attempts to use
file system
In computing, file system or filesystem (often abbreviated to fs) is a method and data structure that the operating system uses to control how data is stored and retrieved. Without a file system, data placed in a storage medium would be one larg ...
space more efficiently when the file itself is partially empty. This is achieved by writing brief information (
metadata
Metadata is "data that provides information about other data", but not the content of the data, such as the text of a message or the image itself. There are many distinct types of metadata, including:
* Descriptive metadata – the descriptive ...
) ''representing'' the empty blocks to the data storage media instead of the actual "empty" space which makes up the block, thus consuming less storage space. The full block size is written to the media as the actual size only when the block contains "real" (non-empty) data.
When reading sparse files, the file system transparently converts metadata representing empty blocks into "real" blocks filled with null bytes at runtime. The application is unaware of this conversion.
Most modern file systems support sparse files, including most
Unix
Unix (; trademarked as UNIX) is a family of multitasking, multiuser computer operating systems that derive from the original AT&T Unix, whose development started in 1969 at the Bell Labs research center by Ken Thompson, Dennis Ritchie, and ot ...
variants and
NTFS
New Technology File System (NTFS) is a proprietary journaling file system developed by Microsoft. Starting with Windows NT 3.1, it is the default file system of the Windows NT family. It superseded File Allocation Table (FAT) as the preferred fil ...
. Apple's
HFS+
HFS Plus or HFS+ (also known as Mac OS Extended or HFS Extended) is a journaling file system developed by Apple Inc. It replaced the Hierarchical File System (HFS) as the primary file system of Apple computers with the 1998 release of Mac OS 8.1 ...
does not provide support for sparse files, but in OS X, the
virtual file system
A virtual file system (VFS) or virtual filesystem switch is an abstract layer on top of a more concrete file system. The purpose of a VFS is to allow client applications to access different types of concrete file systems in a uniform way. A VFS ...
layer supports storing them in any supported file system, including HFS+.
Apple File System
Apple File System (APFS) is a Proprietary software, proprietary file system developed and deployed by Apple Inc. for macOS macOS Sierra, Sierra (10.12.4) and later, iOS iOS 10 , 10.3 and later, tvOS 10.2 and later, watchOS 3.2 and later, and ...
(APFS), announced in June 2016 at WWDC, also supports them. Sparse files are commonly used for
disk images
A disk image, in computing, is a computer file containing the contents and structure of a disk volume (computing), volume or of an entire data storage device, such as a hard disk drive, tape drive, floppy disk, optical disc, or USB flash drive. ...
,
database snapshots,
log files
In computing, logging is the act of keeping a log of events that occur in a computer system, such as problems, errors or just information on current operations. These events may occur in the operating system or in other software. A message or lo ...
and in scientific applications.
Advantages
The advantage of sparse files is that storage space is only allocated when actually needed: Storage capacity is conserved, and large files can occasionally be created even if insufficient free space for the original file is available on the storage media. This also reduces the time of the first write as the system does not have to allocate blocks for the "skipped" space. If the initial allocation requires writing all zeros to the space, it also keeps the system from having to write over the "skipped" space twice.
For example, a
virtual machine
In computing, a virtual machine (VM) is the virtualization/emulation of a computer system. Virtual machines are based on computer architectures and provide functionality of a physical computer. Their implementations may involve specialized hardw ...
image with max size of 100 GB that has 2 GB of files actually written would require the full 100 GB when backed by pre-allocated storage, yet only 2 GB on a sparse file. If the file system supports hole punching and the guest operating system issues
TRIM commands, deleting files on the guest will accordingly reduce the space needed.
Disadvantages
Disadvantages are that sparse files may become
fragmented; file system free space reports may be misleading; filling up file systems containing sparse files can have unexpected effects (such as disk-full or quota-exceeded errors when merely overwriting an existing portion of a file that happened to have been sparse); and copying a sparse file with a
program
Program, programme, programmer, or programming may refer to:
Business and management
* Program management, the process of managing several related projects
* Time management
* Program, a part of planning
Arts and entertainment Audio
* Progra ...
that does not explicitly support them may copy the entire, uncompressed size of the file, including the zero sections which are not allocated on the storage media—losing the benefits of the sparse property in the file. Sparse files are also not fully supported by all backup software or applications. However, the
VFS implementation sidesteps the prior two disadvantages. Loading
executables
In computing, executable code, an executable file, or an executable program, sometimes simply referred to as an executable or binary, causes a computer "to perform indicated tasks according to encoded instructions", as opposed to a data file ...
on 32 bit Windows (exe or dll) which are sparse takes a much longer time since the file cannot be memory mapped in the limited 4 GB address space, and are not cached as there is no codepath for caching 32 bit sparse executables (Windows on 64 bit architectures can map sparse executables). On NTFS sparse files (or rather their non-zero areas) cannot be compressed. NTFS implements sparseness as a special kind of compression so a file may be either sparse or compressed.
Sparse files in Unix
Sparse files are typically handled transparently to the user. But the differences between a normal file and sparse file become apparent in some situations.
Creation
The
Unix
Unix (; trademarked as UNIX) is a family of multitasking, multiuser computer operating systems that derive from the original AT&T Unix, whose development started in 1969 at the Bell Labs research center by Ken Thompson, Dennis Ritchie, and ot ...
command
dd of=sparse-file bs=5M seek=1 count=0
will create a file of five
mebibyte
The byte is a unit of digital information that most commonly consists of eight bits. Historically, the byte was the number of bits used to encode a single character of text in a computer and for this reason it is the smallest addressable unit ...
s in size, but with no data stored on the media (only
metadata
Metadata is "data that provides information about other data", but not the content of the data, such as the text of a message or the image itself. There are many distinct types of metadata, including:
* Descriptive metadata – the descriptive ...
). (
GNU
GNU () is an extensive collection of free software (383 packages as of January 2022), which can be used as an operating system or can be used in parts with other operating systems. The use of the completed GNU tools led to the family of operat ...
dd
has this behavior because it calls
ftruncate
to set the file size; other implementations may merely create an empty file.)
Similarly the truncate command may be used, if available:
truncate -s 5M
On
Linux
Linux ( or ) is a family of open-source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991, by Linus Torvalds. Linux is typically packaged as a Linux distribution, which ...
, an existing file can be converted to sparse by:
fallocate -d
There is no portable system call to punch holes; Linux provides
fallocate(FALLOC_FL_PUNCH_HOLE)
, and
Solaris
Solaris may refer to:
Arts and entertainment Literature, television and film
* ''Solaris'' (novel), a 1961 science fiction novel by Stanisław Lem
** ''Solaris'' (1968 film), directed by Boris Nirenburg
** ''Solaris'' (1972 film), directed by ...
provides
fcntl(F_FREESP)
.
Detection
The
-s
option of the
ls
command shows the occupied space in blocks.
ls -ls sparse-file
Alternatively, the
du
command prints the occupied space, while
ls
prints the apparent size.
In some non-standard versions of
du
, the option
--block-size=1
prints the occupied space in bytes instead of blocks, so that it can be compared to the
ls
output:
du --block-size=1 sparse-file
ls -l sparse-file
Note the above du usage has the abbreviated option syntax format "du -B 1 sf", itself equivalent to the shortest version "du -b sf" as stated in the du manual: -b, --bytes is equivalent to '--apparent-size --block-size=1'.
Also, the tool
filefrag
from
e2fsprogs
package can be used to show block allocation details of the file.
filefrag -v sparse-file
Copying
Normally the GNU version of
cp
is good at detecting whether a file is sparse, so
cp sparse-file new-file
creates new-file, which will be sparse. However, GNU cp does have a
--sparse
option. This is especially useful if a file containing long zero blocks is saved in a non-sparse way (i.e. the zero blocks have been written to the storage media in full). Storage space can be conserved by doing:
cp --sparse=always file1 file1_sparsed
Some cp implementations, like
FreeBSD
FreeBSD is a free and open-source Unix-like operating system descended from the Berkeley Software Distribution (BSD), which was based on Research Unix. The first version of FreeBSD was released in 1993. In 2005, FreeBSD was the most popular ...
's cp, do not support the
--sparse
option and will always expand sparse files. A partially viable alternative on those systems is to use
rsync
rsync is a utility for efficiently transferring and synchronizing files between a computer and a storage drive and across networked computers by comparing the modification times and sizes of files. It is commonly found on Unix-like operat ...
with its own
--sparse
option instead of cp. Unfortunately
--sparse
cannot be combined with
--inplace
. Newer Versions of rsync do support
--sparse
combined with
--inplace
.
Via
standard input
In computer programming, standard streams are interconnected input and output communication channels between a computer program and its environment when it begins execution. The three input/output (I/O) connections are called standard input (stdin ...
, sparse file copying is achieved as follows:
cp --sparse=always /dev/fd/0 new-sparse-file < somefile
See also
*
Comparison of file systems
The following tables compare general and technical information for a number of file systems.
General information
Limits
Metadata
Features File capabilities
Block capabilities
Note that in addition to the below table, blo ...
References
External links
NTFS Sparse Files For ProgrammersCreating sparse files in Windows Server using fsutilView the Size of the Sparse File of a Database SnapshotSEEK_HOLE or FIEMAP: Detecting holes in sparse filesvirtsync is a commercial solution to rsync's --sparse
and --inplace
issue.*
ttps://www.vanheusden.com/Linux/phantom.php Phantom - a program to convert files to sparse files to reduce storage consumptionArchLinux Wiki: Sparse file
{{DEFAULTSORT:Sparse file
Computer files