A software regression is a type of

software bug A software bug is an error, flaw or fault in the design, development, or operation of computer software that causes it to produce an incorrect or unexpected result, or to behave in unintended ways. The process of finding and correcting bugs i ...

where a feature that has worked before stops working. This may happen after changes are applied to the software's

source code In computing, source code, or simply code, is any collection of code, with or without comments, written using a human-readable programming language, usually as plain text. The source code of a program is specially designed to facilitate the wo ...

, including the addition of new

features Feature may refer to: Computing * Feature (CAD), could be a hole, pocket, or notch * Feature (computer vision), could be an edge, corner or blob * Feature (software design) is an intentional distinguishing characteristic of a software item ...

and bug fixes. They may also be introduced by changes to the environment in which the software is running, such as system upgrades, system patching or a change to

daylight saving time Daylight saving time (DST), also referred to as daylight savings time or simply daylight time (United States, Canada, and Australia), and summer time (United Kingdom, European Union, and others), is the practice of advancing clocks (typicall ...

. A software performance regression is a situation where the software still functions correctly, but performs more slowly or uses more memory or resources than before. Various types of software regressions have been identified in practice, including the following: * ''Local'' – a change introduces a new bug in the changed module or component. * ''Remote'' – a change in one part of the software breaks functionality in another module or component. * ''Unmasked'' – a change unmasks an already existing bug that had no effect before the change. Regressions are often caused by encompassed bug fixes included in

software patch A patch is a set of changes to a computer program or its supporting data designed to update, fix, or improve it. This includes fixing security vulnerabilities and other bugs, with such patches usually being called bugfixes or bug fixes. Patches ...

es. One approach to avoiding this kind of problem is

regression testing Regression testing (rarely, ''non-regression testing'') is re-running functional and non-functional tests to ensure that previously developed and tested software still performs as expected after a change. If not, that would be called a '' regre ...

. A properly designed

test plan A test plan is a document detailing the objectives, resources, and processes for a specific test for a software or hardware product. The plan typically contains a detailed understanding of the eventual workflow. Test plans A test plan documents the ...

aims at preventing this possibility before releasing any software.

Automated testing In software testing, test automation is the use of software separate from the software being tested to control the execution of tests and the comparison of actual outcomes with predicted outcomes. Test automation can automate some repetitive bu ...

and well-written

test case In software engineering, a test case is a specification of the inputs, execution conditions, testing procedure, and expected results that define a single test to be executed to achieve a particular software testing objective, such as to exercise ...

s can reduce the likelihood of a regression.

Prevention and detection

Techniques have been proposed that try to prevent regressions from being introduced into software at various stages of development, outlined below.

Prior to release

In order to avoid regressions being seen by the

end-user In product development, an end user (sometimes end-user) is a person who ultimately uses or is intended to ultimately use a product. The end user stands in contrast to users who support or maintain the product, such as sysops, system administrat ...

after release, developers regularly run regression tests after changes are introduced to the software. These tests can include

unit tests In computer programming, unit testing is a software testing method by which individual units of source code—sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures&mda ...

to catch local regressions as well as integration tests to catch remote regressions. Regression testing techniques often leverage existing test cases to minimize the effort involved in creating them. However, due to the volume of these existing tests, it is often necessary to select a representative subset, using techniques such as test-case prioritization. For detecting performance regressions, software performance tests are run on a regular basis, to monitor the response time and resource usage metrics of the software after subsequent changes. Unlike functional regression tests, the results of performance tests are subject to

variance In probability theory and statistics, variance is the expectation of the squared deviation of a random variable from its population mean or sample mean. Variance is a measure of dispersion, meaning it is a measure of how far a set of numbers ...

- that is, results can differ between tests due to variance in performance measurements; as a result, a decision must be made on whether a change in performance numbers constitutes a regression, based on experience and end-user demands. Approaches such as statistical significance testing and change point detection are sometimes used to aid in this decision.

Prior to commit

Since

debugging In computer programming and software development, debugging is the process of finding and resolving '' bugs'' (defects or problems that prevent correct operation) within computer programs, software, or systems. Debugging tactics can involve in ...

and localizing the root cause of a software regression can be expensive, there also exists some methods that try to prevent regressions from being committed into the

code repository In version control systems, a repository is a data structure that stores metadata for a set of files or directory structure. Depending on whether the version control system in use is distributed, like Git or Mercurial, or centralized, like Subver ...

in the first place. For example,

Git Git () is a distributed version control system: tracking changes in any set of files, usually used for coordinating work among programmers collaboratively developing source code during software development. Its goals include speed, data in ...

Hooks enable developers to run test scripts before code changes are committed or pushed to the code repository. In addition,

change impact analysis Change impact analysis (IA) or impact analysis is the analysis of changes within a deployed product or application and their potential consequences. Change impact analysis is defined by Bohnner and Arnold as "identifying the potential consequenc ...

has been applied to software to predict the impact of a code change on various components of the program, and to supplement test case selection and prioritization. Software linters are also often added to commit hooks to ensure consistent coding style, thereby minimizing stylistic issues that can make the software prone to regressions.

Localization

Many of the techniques used to find the root cause of non-regression software bugs can also be used to debug software regressions, including breakpoint debugging, print debugging, and

program slicing In computer programming, program slicing is the computation of the set of program statements, the program slice, that may affect the values at some point of interest, referred to as a slicing criterion. Program slicing can be used in debugging t ...

. The techniques described below are often used specifically to debug software regressions.

Functional regressions

A common technique used to localize functional regressions is

bisection In geometry, bisection is the division of something into two equal or congruent parts, usually by a line, which is then called a ''bisector''. The most often considered types of bisectors are the ''segment bisector'' (a line that passes through ...

, which takes both a buggy commit and a previously working commit as input, and tries to find the root cause by doing a binary search on the commits in between.

Version control In software engineering, version control (also known as revision control, source control, or source code management) is a class of systems responsible for managing changes to computer programs, documents, large web sites, or other collections o ...

systems such as Git and

Mercurial Mercurial is a distributed revision control tool for software developers. It is supported on Microsoft Windows and Unix-like systems, such as FreeBSD, macOS, and Linux. Mercurial's major design goals include high performance and scalability, d ...

provide built-in ways to perform bisection on a given pair of commits. Other options include directly associating the result of a regression test with code changes; setting divergence breakpoints; or using incremental

data-flow analysis In computing, dataflow is a broad concept, which has various meanings depending on the application and context. In the context of software architecture, data flow relates to stream processing or reactive programming. Software architecture Dataf ...

, which identifies test cases - including failing ones - that are relevant to a set of code changes, among others.

Performance regressions

Profiling measures the performance and resource usage of various components of a program, and is used to generate data useful in debugging performance issues. In the context of software performance regressions, developers often compare the call trees (also known as "timelines") generated by profilers for both the buggy version and the previously working version, and mechanisms exist to simplify this comparison.

Web development tools Web development tools (often called devtools or inspect element) allow web developers to test and debug their code. They are different from website builders and integrated development environments (IDEs) in that they do not assist in the direct c ...

typically provide developers the ability to record these performance profiles. Logging also helps with performance regression localization, and similar to call trees, developers can compare systematically-placed performance logs of multiple versions of the same software. A tradeoff exists when adding these performance logs, as adding many logs can help developers pinpoint which portions of the software are regressing at smaller granularities, while adding only a few logs will also reduce overhead when executing the program. Additional approaches include writing performance-aware unit tests to help with localization, and ranking subsystems based on performance counter deviations. Bisection can also be repurposed for performance regressions by considering commits that perform below (or above) a certain baseline value as buggy, and taking either the left or the right side of the commits based on the results of this comparison.

References

{{DEFAULTSORT:Software Regression Software bugs