Trellis quantization is an algorithm that can improve

data compression In information theory, data compression, source coding, or bit-rate reduction is the process of encoding information using fewer bits than the original representation. Any particular compression is either lossy or lossless. Lossless compression ...

in DCT-based encoding methods. It is used to optimize residual DCT coefficients after

motion estimation Motion estimation is the process of determining ''motion vectors'' that describe the transformation from one 2D image to another; usually from adjacent frames in a video sequence. It is an ill-posed problem as the motion is in three dimensions b ...

lossy In information technology, lossy compression or irreversible compression is the class of data compression methods that uses inexact approximations and partial data discarding to represent the content. These techniques are used to reduce data size ...

video compression encoders such as

Xvid Xvid (formerly "XviD") is a video codec library following the MPEG-4 video coding standard, specifically MPEG-4 Part 2 Advanced Simple Profile (ASP). It uses ASP features such as b-frames, global and quarter pixel motion compensation, lumi mas ...

and

x264 x264 is a free and open-source software library and a command-line utility developed by VideoLAN for encoding video streams into the H.264/MPEG-4 AVC video coding format. It is released under the terms of the GNU General Public License. History ...

. Trellis quantization reduces the size of some DCT coefficients while recovering others to take their place. This process can increase quality because coefficients chosen by Trellis have the lowest rate-distortion ratio. Trellis quantization effectively finds the optimal quantization for each block to maximize the

PSNR Peak signal-to-noise ratio (PSNR) is an engineering term for the ratio between the maximum possible power of a Signal (information theory), signal and the power of corrupting noise that affects the fidelity of its representation. Because many sign ...

relative to

bitrate In telecommunications and computing, bit rate (bitrate or as a variable ''R'') is the number of bits that are conveyed or processed per unit of time. The bit rate is expressed in the unit bit per second (symbol: bit/s), often in conjunction w ...

. It has varying effectiveness depending on the input data and compression method.

References

VirtualDub/Xvid guide mentioning Trellis quantizationTrellis explanation and pseudocode by the x264-author
MPEG Data compression Video compression {{compu-sci-stub