computing Computing is any goal-oriented activity requiring, benefiting from, or creating computing machinery. It includes the study and experimentation of algorithmic processes, and development of both hardware and software. Computing has scientific, e ...

software engineering Software engineering is a systematic engineering approach to software development. A software engineer is a person who applies the principles of software engineering to design, develop, maintain, test, and evaluate computer software. The term '' ...

, and

software testing Software testing is the act of examining the artifacts and the behavior of the software under test by validation and verification. Software testing can also provide an objective, independent view of the software to allow the business to apprecia ...

, a test oracle (or just oracle) is a mechanism for determining whether a test has passed or failed. The use of oracles involves comparing the output(s) of the system under test, for a given test-case input, to the output(s) that the oracle determines that product should have. The term "test oracle" was first introduced in a paper by William E. Howden. Additional work on different kinds of oracles was explored by Elaine Weyuker. Oracles often operate separately from the system under test.Jalote, Pankaj; ''An Integrated Approach to Software Engineering'', Springer/Birkhäuser, 2005, However,

method Method ( grc, μέθοδος, methodos) literally means a pursuit of knowledge, investigation, mode of prosecuting such inquiry, or system. In recent centuries it more often means a prescribed process for completing a task. It may refer to: *Scien ...

postconditions are part of the system under test, as automated oracles in design by contract models. Determining the correct output for a given input (and a set of program or system states) is known as the oracle problem or test oracle problem, which is a much harder problem than it seems, and involves working with problems related to controllability and observability.Ammann, Paul; and Offutt, Jeff; "Introduction to Software Testing", ''Cambridge University Press'', 2008,

Categories

A research literature survey covering 1978 to 2012 found several potential categories of test oracles.

Specified

These oracles are typically associated with formalized approaches to software modeling and software code construction. They are connected to

formal specification In computer science, formal specifications are mathematically based techniques whose purpose are to help with the implementation of systems and software. They are used to describe a system, to analyze its behavior, and to aid in its design by verif ...

model-based design Model-based design (MBD) is a mathematical and visual method of addressing problems associated with designing complex control, signal processing and communication systems. It is used in many motion control, industrial equipment, aerospace, and aut ...

which may be used to generate test oracles, state transition specification for which oracles can be derived to aid

model-based testing Model-based testing is an application of model-based design for designing and optionally also executing artifacts to perform software testing or system testing. Models can be used to represent the desired behavior of a system under test (SUT), or ...

and protocol conformance testing, and design by contract for which the equivalent test oracle is an assertion. Specified Test Oracles have a number of challenges. Formal specification relies on abstraction, which in turn may naturally have an element of imprecision as all models cannot capture all behavior.

Derived

A derived test oracle differentiates correct and incorrect behavior by using information derived from artifacts of the system. These may include documentation, system execution results and characteristics of versions of the system under test. Regression test suites (or reports) are an example of a derived test oracle - they are built on the assumption that the result from a previous system version can be used as aid (oracle) for a future system version. Previously measured performance characteristics may be used as an oracle for future system versions, for example, to trigger a question about observed potential performance degradation. Textual documentation from previous system versions may be used as a basis to guide expectations in future system versions. A pseudo-oracle falls into the category of derived test oracle. A pseudo-oracle, as defined by Weyuker, is a separately written program which can take the same input as the program or system under test so that their outputs may be compared to understand if there might be a problem to investigate. A partial oracle is a hybrid between specified test oracle and derived test oracle. It specifies important (but not complete) properties of the system under test. For example,

metamorphic testing Metamorphic testing (MT) is a property-based software testing technique, which can be an effective approach for addressing the test oracle problem and test case generation problem. The test oracle problem is the difficulty of determining the expect ...

exploits such properties, called metamorphic relations, across multiple executions of the system.

Implicit

An implicit test oracle relies on implied information and assumptions. For example, there may be some implied conclusion from a program crash, i.e. unwanted behavior - an oracle to determine that there may be a problem. There are a number of ways to search and test for unwanted behavior, whether some call it negative testing, where there are specialized subsets such as

fuzzing In programming and software development, fuzzing or fuzz testing is an automated software testing technique that involves providing invalid, unexpected, or random data as inputs to a computer program. The program is then monitored for exceptions ...

. There are limitations in implicit test oracles - as they rely on implied conclusions and assumptions. For example, a program or process crash may not be a priority issue if the system is a fault-tolerant system and so operating under a form of self-healing/ self-management. Implicit test oracles may be susceptible to false positives due to environment dependencies.

Human

When specified, derived or implicit test oracles cannot be used, then human input to determine the test oracles is required. These can be thought of as quantitative and qualitative approaches. A quantitative approach aims to find the right amount of information to gather on a system under test (e.g., test results) for a stakeholder to be able to make decisions on fit-for-purpose or the release of the software. A qualitative approach aims to find the representativeness and suitability of the input test data and context of the output from the system under test. An example is using realistic and representative test data and making sense of the results (if they are realistic). These can be guided by

heuristic A heuristic (; ), or heuristic technique, is any approach to problem solving or self-discovery that employs a practical method that is not guaranteed to be optimal, perfect, or rational, but is nevertheless sufficient for reaching an immediate, ...

approaches, such as gut instincts, rules of thumb, checklist aids, and experience to help tailor the specific combination selected for the program/system under test.

Examples

Test oracles are most commonly based on

specifications A specification often refers to a set of documented requirements to be satisfied by a material, design, product, or service. A specification is often a type of technical standard. There are different types of technical or engineering specificati ...

and

documentation Documentation is any communicable material that is used to describe, explain or instruct regarding some attributes of an object, system or procedure, such as its parts, assembly, installation, maintenance and use. As a form of knowledge manageme ...

. A formal specification used as input to

and

would be an example of a ''specified test oracle''. The ''model-based oracle'' uses the same model to generate and verify system behavior. Documentation that is not a full specification of the product, such as a usage or installation guide, or a record of performance characteristics or minimum machine requirements for the software, would typically be a derived test oracle. A consistency oracle compares the results of one test execution to another for similarity. This is another example of a derived test oracle. An oracle for a software program might be a second program that uses a different

algorithm In mathematics and computer science, an algorithm () is a finite sequence of rigorous instructions, typically used to solve a class of specific Computational problem, problems or to perform a computation. Algorithms are used as specificat ...

to evaluate the same mathematical expression as the product under test. This is an example of a pseudo-oracle, which is a derived test oracle. During

Google Google LLC () is an American multinational technology company focusing on search engine technology, online advertising, cloud computing, computer software, quantum computing, e-commerce, artificial intelligence, and consumer electronics. ...

search, we do not have a complete oracle to verify whether the number of returned results is correct. We may define a metamorphic relation such that a follow-up narrowed-down search will produce fewer results. This is an example of a partial oracle, which is a hybrid between specified test oracle and derived test oracle. A statistical oracle uses probabilistic characteristics, for example with image analysis where a range of certainty and uncertainty is defined for the test oracle to pronounce a match or otherwise. This would be an example of a quantitative approach in human test oracle. A heuristic oracle provides representative or approximate results over a class of test inputs.Hoffman, Douglas
Heuristic Test Oracles
Software Testing & Quality Engineering Magazine, 1999 This would be an example of a qualitative approach in human test oracle.

References

Bibliography

* Binder, Robert V. (1999). "Chapter 18 - Oracles" in ''Testing Object-Oriented Systems: Models, Patterns, and Tools'', Addison-Wesley Professional, 7 November 1999, {{ISBN, 978-0-201-80938-1 Software testing Computation oracles