Incremental encoding, also known as front compression, back compression, or front coding, is a type of

delta encoding Delta encoding is a way of storing or transmitting data in the form of '' differences'' (deltas) between sequential data rather than complete files; more generally this is known as data differencing. Delta encoding is sometimes called delta compre ...

compression algorithm In information theory, data compression, source coding, or bit-rate reduction is the process of encoding information using fewer bits than the original representation. Any particular compression is either lossy or lossless. Lossless compressio ...

whereby common prefixes or

suffixes In linguistics, a suffix is an affix which is placed after the stem of a word. Common examples are case endings, which indicate the grammatical case of nouns, adjectives, and verb endings, which form the conjugation of verbs. Suffixes can carry g ...

and their lengths are recorded so that they need not be duplicated. This algorithm is particularly well-suited for compressing sorted

data In the pursuit of knowledge, data (; ) is a collection of discrete Value_(semiotics), values that convey information, describing quantity, qualitative property, quality, fact, statistics, other basic units of meaning, or simply sequences of sy ...

, e.g., a list of

word A word is a basic element of language that carries an objective or practical meaning, can be used on its own, and is uninterruptible. Despite the fact that language speakers often have an intuitive grasp of what a word is, there is no conse ...

s from a dictionary. For example: The encoding used to store the common prefix length itself varies from application to application. Typical techniques are storing the value as a single byte;

, which stores only the change in the common prefix length; and various universal codes. It may be combined with other general

lossless data compression Lossless compression is a class of data compression that allows the original data to be perfectly reconstructed from the compressed data with no loss of information. Lossless compression is possible because most real-world data exhibits statistic ...

techniques such as

entropy encoding In information theory, an entropy coding (or entropy encoding) is any lossless data compression method that attempts to approach the lower bound declared by Shannon's source coding theorem, which states that any lossless data compression method ...

and

dictionary coder A dictionary coder, also sometimes known as a substitution coder, is a class of lossless data compression algorithms which operate by searching for matches between the text to be compressed and a set of strings contained in a data structure (call ...

s to compress the remaining suffixes.

Applications

Incremental encoding is widely used in information retrieval to compress the lexicons used in search indexes; these list all the words found in all the documents and a pointer for each one to a list of locations. Typically, it compresses these indexes by about 40%.Ian H. Witten, Alistair Moffat, Timothy C. Bell. Managing Gigabytes. Second edition. Academic Press. . Section 4.1: Accessing the lexicon, subsection Front coding, pp.159–161. As one example, incremental encoding is used as a starting point by the

GNU locate locate is a Unix utility which serves to find files on filesystems. It searches through a prebuilt database of files generated by the updatedb command or by a daemon and compressed using incremental encoding. It operates significantly faster than ...

utility, in an index of filenames and directories. The

utility further uses

bigram A bigram or digram is a sequence of two adjacent elements from a string of tokens, which are typically letters, syllables, or words. A bigram is an ''n''-gram for ''n''=2. The frequency distribution of every bigram in a string is commonly used f ...

encoding to further shorten popular filepath prefixes.

References

Lossless compression algorithms Database index techniques {{storage-software-stub