Metamorphic testing (MT) is a property-based
software testing
Software testing is the act of examining the artifacts and the behavior of the software under test by validation and verification. Software testing can also provide an objective, independent view of the software to allow the business to apprecia ...
technique, which can be an effective approach for addressing the
test oracle
In computing, software engineering, and software testing, a test oracle (or just oracle) is a mechanism for determining whether a test has passed or failed. The use of oracles involves comparing the output(s) of the system under test, for a given ...
problem and test case generation problem. The test oracle problem is the difficulty of determining the expected outcomes of selected
test case
In software engineering, a test case is a specification of the inputs, execution conditions, testing procedure, and expected results that define a single test to be executed to achieve a particular software testing objective, such as to exercise ...
s or to determine whether the actual outputs agree with the expected outcomes.
''Metamorphic relations'' (MRs) are
necessary properties of the intended functionality of the software, and must involve ''multiple'' executions of the software. Consider, for example, a program that implements
sin ''x'' correct to 100
significant figures
Significant figures (also known as the significant digits, ''precision'' or ''resolution'') of a number in positional notation are digits in the number that are reliable and necessary to indicate the quantity of something.
If a number expre ...
; a metamorphic relation for sine functions is "
sin (''π − x'') = sin ''x''". Thus, even though the expected value of sin ''x''
1 for the source test case ''x''
1 = 1.234 correct to the required accuracy is not known, a follow-up test case ''x''
2 = ''π'' − 1.234 can be constructed.
We can verify whether the actual outputs produced by the program under test from the source test case and the follow-up test case are consistent with the MR in question. Any inconsistency (after taking rounding errors into consideration) indicates a
failure
Failure is the state or condition of not meeting a desirable or intended objective, and may be viewed as the opposite of success. The criteria for failure depends on context, and may be relative to a particular observer or belief system. One ...
of the program, caused by a
fault[ in the implementation.
MRs are not limited to programs with numerical inputs or equality relations. As an example, when testing a booking website, a web search for accommodation in Sydney, Australia, returns 1,671 results; are the results of this search correct and complete? This is a test oracle problem. Based on a metamorphic relation, we may filter the price range or star rating and apply the search again; it should return a subset of the previous results. A violation of this expectation would similarly reveal a failure of the system.
Metamorphic testing was invented by T.Y. Chen in the technical report in 1998.
Since then, more than 150 international researchers and practitioners have applied the technique to real-life applications.
Some examples include
web services,
]computer graphics
Computer graphics deals with generating images with the aid of computers. Today, computer graphics is a core technology in digital photography, film, video games, cell phone and computer displays, and many specialized applications. A great deal ...
,
embedded systems
An embedded system is a computer system—a combination of a computer processor, computer memory, and input/output peripheral devices—that has a dedicated function within a larger mechanical or electronic system. It is ''embedded'' ...
,
simulation
A simulation is the imitation of the operation of a real-world process or system over time. Simulations require the use of models; the model represents the key characteristics or behaviors of the selected system or process, whereas the ...
and modeling,
machine learning
Machine learning (ML) is a field of inquiry devoted to understanding and building methods that 'learn', that is, methods that leverage data to improve performance on some set of tasks. It is seen as a part of artificial intelligence.
Machine ...
,
decision support
A decision support system (DSS) is an information system that supports business or organizational decision-making activities. DSSs serve the management, operations and planning levels of an organization (usually mid and higher management) and h ...
,
bioinformatics
Bioinformatics () is an interdisciplinary field that develops methods and software tools for understanding biological data, in particular when the data sets are large and complex. As an interdisciplinary field of science, bioinformatics combin ...
,
components,
numerical analysis
Numerical analysis is the study of algorithms that use numerical approximation (as opposed to symbolic manipulations) for the problems of mathematical analysis (as distinguished from discrete mathematics). It is the study of numerical methods th ...
,
and compilers
In computing, a compiler is a computer program that translates computer code written in one programming language (the ''source'' language) into another language (the ''target'' language). The name "compiler" is primarily used for programs that ...
.
The first major survey of the field of MT was conducted in 2016.
It was followed by another major survey in 2018, which highlights the challenges and opportunities and clarifies common misunderstandings.
Although MT was initially proposed as a software verification Software verification is a discipline of software engineering whose goal is to assure that software fully satisfies all the expected requirements.
Broad scope and classification
A broad definition of verification makes it equivalent to software t ...
technique, it was later developed into a paradigm that covers verification, validation, and other types of software quality
In the context of software engineering, software quality refers to two related but distinct notions:
* Software functional quality reflects how well it complies with or conforms to a given design, based on functional requirements or specification ...
assessment. MT can be applied independently, and can also be combined with other static and dynamic software analysis techniques such as proving and debugging
In computer programming and software development, debugging is the process of finding and resolving ''bugs'' (defects or problems that prevent correct operation) within computer programs, software, or systems.
Debugging tactics can involve in ...
.
In August 2018, Google
Google LLC () is an American Multinational corporation, multinational technology company focusing on Search Engine, search engine technology, online advertising, cloud computing, software, computer software, quantum computing, e-commerce, ar ...
acquired GraphicsFuzz, a startup from Imperial College London
Imperial College London (legally Imperial College of Science, Technology and Medicine) is a public research university in London, United Kingdom. Its history began with Prince Albert, consort of Queen Victoria, who developed his vision for a ...
, to apply
metamorphic testing to graphics device drivers
In computing, a device driver is a computer program that operates or controls a particular type of device that is attached to a computer or automaton. A driver provides a software interface to hardware devices, enabling operating systems and o ...
for Android smartphones
A smartphone is a Mobile device, portable computer device that combines Mobile phone, mobile telephone and Mobile computing, computing functions into one unit. They are distinguished from feature phones by their stronger hardware capabilities ...
.
References
External links
*
*
* {{Cite journal, title=Automated testing of real-life self-driving systems and beyond: Keynote speech by Z.Q. (George) Zhou at IEEE AITest, San Francisco East Bay, 2019, doi = 10.5281/zenodo.2636979, year = 2019, last1 = Zhou, first1 = Zhi Quan
Software testing